Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Changes the table's update policy. The [update policy](update-policy.md) simplif
>
> * The source table and the table for which the update policy is defined must be in the same database.
> * The update policy function schema and the target table schema must match in their column names, types, and order.
> * If the policy already defined on the table, the PolicyObjects specified in the command are added to the array of PolicyObjects in the existing policy.

## Permissions

Expand All @@ -30,17 +31,14 @@ You must have at least [Table Admin](../access-control/role-based-access-control

|Name|Type|Required|Description|
|--|--|--|--|
| *DatabaseName* | `string` | | The name of the database. When you run the command from the database context that contains the table to alter, *DatabaseName* isn't required. |
| *DatabaseName* | `string` | | The name of the database. When you run the command from the database context that contains the table to alter, *DatabaseName* is not required. |
| *TableName* | `string` | :heavy_check_mark: | The name of the table. A wildcard, `*`, denotes all tables.|
| *ArrayOfPolicyObjects* | `string` | :heavy_check_mark: | A serialized array of policy objects. For more information, see [update policy](update-policy.md).|

## Returns

Returns a JSON representation of the policy.

> [!WARNING]
> If an update policy is already defined on the table, the PolicyObjects specified in the command are added to the array of PolicyObjects in the existing policy. Entries aren't deduplicated, so if the exact same PolicyObject already exists in the current update policy, another entry will be added, potentially causing duplicates during ingestion time.

## Example

The following command changes the update policy for a table using [multi-line string literals](../query/scalar-data-types/string.md#multi-line-string-literals).
Expand All @@ -58,4 +56,4 @@ The following command changes the update policy for a table using [multi-line st
}
]
```
````
````
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,8 @@ ms.date: 08/11/2024
Use this command to change the table update policy. The [update policy](update-policy.md) simplifies the process of syncing and updating data between two tables. When new data is inserted into the source table, a transformation query runs over this data to modify and insert the data into the target table.

> [!NOTE]
>
> * The source table and the table for which the update policy is defined must be in the same database.
> * The update policy function schema and the target table schema must match in their column names, types, and order.
> The source table and the table for which the update policy is defined must be in the same database.
> The update policy function schema and the target table schema must match in their column names, types, and order.

## Permissions

Expand All @@ -38,11 +37,6 @@ You must have at least [Table Admin](../access-control/role-based-access-control

Returns a JSON representation of the policy.

> [!WARNING]
> The alter command *replaces* the current update policy with a new policy. If you would like to *add* a new entry to the array
> of policy objects, in addition to the current ones, use the
> [.alter-merge table policy update command](alter-merge-table-update-policy-command.md).

## Example

Change the update policy for a table (using [multi-line string literals](../query/scalar-data-types/string.md#multi-line-string-literals)):
Expand All @@ -60,4 +54,4 @@ Change the update policy for a table (using [multi-line string literals](../quer
}
]
```
````
````
96 changes: 3 additions & 93 deletions data-explorer/kusto/management/update-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,8 @@ Each such object is represented as a JSON property bag, with the following prope
|Property |Type | Description |
|---------|---------|----------------|
|IsEnabled |`bool` |States if update policy is *true* - enabled, or *false* - disabled|
|Source |`string` |Name of the table that triggers invocation of the update policy. |
|SourceIsWildCard |`bool` |If *true*, the `Source` property can be a wildcard pattern. See [Update policy with source table wildcard pattern](#update-policy-with-source-table-wildcard-pattern) |
|Query |`string` |A query used to produce data for the update. |
|Source |`string` |Name of the table that triggers invocation of the update policy |
|Query |`string` |A query used to produce data for the update |
|IsTransactional |`bool` |States if the update policy is transactional or not, default is *false*. If the policy is transactional and the update policy fails, the source table isn't updated. |
|PropagateIngestionProperties |`bool`|States if properties specified during ingestion to the source table, such as [extent tags](extent-tags.md) and creation time, apply to the target table. |
|ManagedIdentity | `string` | The managed identity on behalf of which the update policy runs. The managed identity can be an object ID, or the `system` reserved word. The update policy must be configured with a managed identity when the query references tables in other databases or tables with an enabled [row level security policy](row-level-security-policy.md). For more information, see [Use a managed identity to run a update policy](update-policy-with-managed-identity.md). |
Expand All @@ -106,7 +105,6 @@ Each such object is represented as a JSON property bag, with the following prope
|---------|---------|----------------|
|IsEnabled |`bool` |States if update policy is *true* - enabled, or *false* - disabled|
|Source |`string` |Name of the table that triggers invocation of the update policy |
|SourceIsWildCard |`bool` |If *true*, the `Source` property can be a wildcard pattern. |
|Query |`string` |A query used to produce data for the update |
|IsTransactional |`bool` |States if the update policy is transactional or not, default is *false*. If the policy is transactional and the update policy fails, the source table isn't updated. |
|PropagateIngestionProperties |`bool`|States if properties specified during ingestion to the source table, such as [extent tags](extent-tags.md) and creation time, apply to the target table. |
Expand Down Expand Up @@ -146,14 +144,6 @@ Update policies take effect when data is ingested or moved to a source table, or
> Data may be lost in all tables with an update policy relationship if the `replace` command is invoked.
> Consider using `.set-or-append` instead.

## Update policy with source table wildcard pattern

Update policy supports ingesting from multiple source tables that share the same pattern, while using the
same query as the update policy query. This is useful if you have several source tables, usually sharing the same schema (or a subset of columns that share a common schema), and you would like to trigger ingestion to a
single target table, when ingesting to either of those tables. In this case, instead of defining multiple
update policies, each for a single source table, you can define a single update policy with wildcard as `Source`.The `Query` of the update policy must comply with all source tables matching the pattern.
To reference the source table in the update policy query, you can use a special symbol named `$source_table`. See example in [Example of wild card update policy](#example-of-wild-card-update-policy).

## Remove data from source table

After ingesting data to the target table, you can optionally remove it from the source table. Set a soft-delete period of `0sec` (or `00:00:00`) in the source table's [retention policy](retention-policy.md), and the update policy as transactional. The following conditions apply:
Expand Down Expand Up @@ -261,87 +251,7 @@ In this example, use an update policy with a simple function to perform ETL. Fir
.alter-merge table MySourceTable policy retention softdelete = 0s
```

## Example of wild card update policy

The following example creates an update policy with a single entry on table `TargetTable`. The policy references all tables matching pattern `SourceTable*` as its source.
Any ingestion to a table which matches the pattern (in local database) will trigger the update policy, and ingest data to `TargetTable`, based on the update policy query.

1. Create two source tables:

```kusto
.create table SourceTable1(Id:long, Value:string)
```

```kusto
.create table SourceTable2(Id:long, Value:string)
```

1. Create the target table:

```kusto
.create table TargetTable(Id:long, Value:string, Source:string)
```

1. Create a function which will serve as the `Query` of the update policy. The function uses the `$source_table` symbol to reference the `Source` of the update policy. Use `skipValidation=true` to skip validation during the create function, since `$source_table` is only known during update policy execution. The function is validated during the next step, when altering the update policy.

```kusto
.create function with(skipValidation=true) IngestToTarget()
{
$source_table
| parse Value with "I'm from table " Source
| project Id, Value, Source
}
```

1. Create the update policy on `TargetTable`. The policy references all tables matching pattern `SourceTable*` as its source.

````kusto
.alter table TargetTable policy update
```[{
"IsEnabled": true,
"Source": "SourceTable*",
"SourceIsWildCard" : true,
"Query": "IngestToTarget()",
"IsTransactional": true,
"PropagateIngestionProperties": true
}]```

````

1. Ingest to source tables. Both ingestions trigger the update policy:

```kusto
.set-or-append SourceTable1 <|
datatable (Id:long, Value:string)
[
1, "I'm from table SourceTable1",
2, "I'm from table SourceTable1"
]
```

```kusto
.set-or-append SourceTable2 <|
datatable (Id:long, Value:string)
[
3, "I'm from table SourceTable2",
4, "I'm from table SourceTable2"
]
```

1. Query `TargetTable`:

```kusto
TargetTable
```

|Id|Value|Source|
|---|---|---|
|1|I'm from table SourceTable1|SourceTable1|
|2|I'm from table SourceTable1|SourceTable1|
|3|I'm from table SourceTable2|SourceTable2|
|4|I'm from table SourceTable2|SourceTable2|

## Related content

* [Common scenarios for using table update policies](update-policy-common-scenarios.md)
* [Tutorial: Route data using table update policies](update-policy-tutorial.md)
* [Tutorial: Route data using table update policies](update-policy-tutorial.md)
3 changes: 3 additions & 0 deletions data-explorer/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,9 @@ items:
- name: Apache Log4J 2
displayName: ingest data, ingestion, connector
href: apache-log4j2-connector.md
- name: Azure Monitor
displayName: ingest data, ingestion, connector
href: /azure/azure-monitor/vm/send-fabric-destination?toc=/azure/data-explorer/toc.json
- name: Apache Spark
displayName: connector, export, ingest
href: spark-connector.md
Expand Down