Repo sync for protected branch

data-explorer/kusto/functions-library/slm-embeddings-fl.md

@@ -135,25 +135,26 @@ let slm_embeddings_fl=(tbl:(*), text_col:string, embeddings_col:string, batch_si
 {
     let kwargs = bag_pack('text_col', text_col, 'embeddings_col', embeddings_col, 'batch_size', batch_size, 'model_name', model_name, 'prefix', prefix);
     let code = ```if 1:
-from sandbox_utils import Zipackage
-Zipackage.install('embedding_engine.zip')
-
-from embedding_factory import create_embedding_engine
-
-text_col = kargs["text_col"]
-embeddings_col = kargs["embeddings_col"]
-batch_size = kargs["batch_size"]
-model_name = kargs["model_name"]
-prefix = kargs["prefix"]
-
-Zipackage.install(f'{model_name}.zip')
+		from sandbox_utils import Zipackage
+		Zipackage.install('embedding_engine.zip')
+#		Zipackage.install('tokenizers-0.22.1.whl')			# redundant if tokenizers package is included in the Python image
+
+		from embedding_factory import create_embedding_engine
+
+		text_col = kargs["text_col"]
+		embeddings_col = kargs["embeddings_col"]
+		batch_size = kargs["batch_size"]
+		model_name = kargs["model_name"]
+		prefix = kargs["prefix"]
 
-engine = create_embedding_engine(model_name, cache_dir="C:\\Temp")
-embeddings = engine.encode(df[text_col].tolist(), batch_size=batch_size, prefix=prefix)
+		Zipackage.install(f'{model_name}.zip')
 
-result = df
-result[embeddings_col] = list(embeddings)
-```;
+		engine = create_embedding_engine(model_name, cache_dir="C:\\Temp")
+		embeddings = engine.encode(df[text_col].tolist(), batch_size=batch_size, prefix=prefix)		#	prefix is used only for E5
+
+		result = df
+		result[embeddings_col] = list(embeddings)
+	```;
     tbl
     | evaluate hint.distribution=per_node python(typeof(*), code, kwargs, external_artifacts = bag_pack(
     'embedding_engine.zip', 'https://artifactswestus.z22.web.core.windows.net/models/SLM/embedding_engine.zip',

data-explorer/kusto/management/alter-merge-table-update-policy-command.md

@@ -15,7 +15,6 @@ Changes the table's update policy. The [update policy](update-policy.md) simplif
 >
 > * The source table and the table for which the update policy is defined must be in the same database.
 > * The update policy function schema and the target table schema must match in their column names, types, and order.
-> * If the policy already defined on the table, the PolicyObjects specified in the command are added to the array of PolicyObjects in the existing policy.
 
 ## Permissions
 
@@ -31,14 +30,17 @@ You must have at least [Table Admin](../access-control/role-based-access-control
 
 |Name|Type|Required|Description|
 |--|--|--|--|
-| *DatabaseName* | `string` | | The name of the database. When you run the command from the database context that contains the table to alter, *DatabaseName* is not required. |
+| *DatabaseName* | `string` | | The name of the database. When you run the command from the database context that contains the table to alter, *DatabaseName* isn't required. |
 | *TableName* | `string` |  :heavy_check_mark: | The name of the table. A wildcard, `*`, denotes all tables.|
 | *ArrayOfPolicyObjects* | `string` |  :heavy_check_mark: | A serialized array of policy objects. For more information, see [update policy](update-policy.md).|
 
 ## Returns
 
 Returns a JSON representation of the policy.
 
+> [!WARNING]
+> If an update policy is already defined on the table, the PolicyObjects specified in the command are added to the array of PolicyObjects in the existing policy. Entries aren't  deduplicated, so if the exact same PolicyObject already exists in the current update policy, another entry will be added, potentially causing duplicates during ingestion time.
+
 ## Example
 
 The following command changes the update policy for a table using [multi-line string literals](../query/scalar-data-types/string.md#multi-line-string-literals).

data-explorer/kusto/management/alter-table-update-policy-command.md

@@ -12,8 +12,9 @@ ms.date: 08/11/2024
 Use this command to change the table update policy. The [update policy](update-policy.md) simplifies the process of syncing and updating data between two tables. When new data is inserted into the source table, a transformation query runs over this data to modify and insert the data into the target table.
 
 > [!NOTE]
-> The source table and the table for which the update policy is defined must be in the same database.
-> The update policy function schema and the target table schema must match in their column names, types, and order.
+>
+> * The source table and the table for which the update policy is defined must be in the same database.
+> * The update policy function schema and the target table schema must match in their column names, types, and order.
 
 ## Permissions
 
@@ -37,6 +38,11 @@ You must have at least [Table Admin](../access-control/role-based-access-control
 
 Returns a JSON representation of the policy.
 
+> [!WARNING]
+> The alter command *replaces* the current update policy with a new policy. If you would like to *add* a new entry to the array
+> of policy objects, in addition to the current ones, use the
+> [.alter-merge table policy update command](alter-merge-table-update-policy-command.md).
+
 ## Example
 
 Change the update policy for a table (using [multi-line string literals](../query/scalar-data-types/string.md#multi-line-string-literals)):

data-explorer/kusto/management/materialized-views/materialized-view-create.md

@@ -22,7 +22,7 @@ There are two possible ways to create a materialized view, as noted by the *back
 
 * See [Backfill a materialized view](#backfill-a-materialized-view).
 * Creation might take a long while to complete, depending on the number of records in the source table. The view isn't available for queries until backfill is complete.
-* When you're using this option, the create command must be `async`. You can monitor execution by using the [`.show operations`](../show-operations.md) command.
+* When you're using this option, the create command must be `async`. You can monitor execution by using the [`.show operations`](../show-operations.md) command. Query for the entire operation log to view the progress of the backfill (see [examples here](../show-operations.md#example)).
 * You can cancel the backfill process by using the [`.cancel operation`](#cancel-materialized-view-creation) command.
 
 > [!IMPORTANT]

data-explorer/kusto/management/show-operations.md

@@ -15,9 +15,7 @@ Entries are appended to the log when operations start and change their state, in
 Users can view the ongoing and past operations they started by using the `.show operations` command.
 Database administrators can view all operations that apply to the databases they administer.
 
-Users can also view the results of an operation by using the [`.show operation details`](show-operation-details.md)
-command. Normally, the results are returned as part of `.show operations` command itself. For asynchronous
-management commands, the `.show operation details` command is the only way to view the command's results.
+The `.show operations` command returns general details about all operations running on the cluster. Some of the operations also support retrieving the operation's results by using the [`.show operation details`](show-operation-details.md) command.
 
 ## Syntax
 
@@ -33,7 +31,8 @@ management commands, the `.show operation details` command is the only way to vi
 
 ## Returns
 
-When an operation ID is omitted, the command returns a table displaying all administrative operations executed in the last two weeks, whether ongoing or completed. It includes entries accessible to the user, with multiple records possible for a single operation. Only one record indicates the terminal state of 'Completed' or 'Failed.' This mode is for checking the history of operations, but note that records may take time to appear in the historic log.
+When an operation ID is omitted, the command returns a table displaying all administrative operations executed in the last two weeks, whether ongoing or completed. It includes entries accessible to the user, with multiple records possible for a single operation. Only one record indicates the terminal state of 'Completed' or 'Failed.' This mode is for checking the history of operations. Records may take a short time to appear in the historical log. 
+You can use [arg_max()](../query/arg-max-aggregation-function.md) over the results of the historical log to view the latest state for each operation ID (see [examples](#example)).
 
 When one or more operation IDs are provided, the command returns the latest update for each ID, given the user's access and the record being less than 6 hours old. This mode helps quickly check the latest status of recently executed operations.
 
@@ -69,17 +68,49 @@ The following table describes the possible values for the result table's *State*
 
 ## Example
 
+The following command returns the latest state per operation ID for operations that started after `2026-01-05`:
+
+```kusto
+.show operations 
+| where StartedOn > datetime(2026-01-05)
+| summarize arg_max(LastUpdatedOn, *) by OperationId
+| project OperationId, Operation, StartedOn, LastUpdatedOn, Duration, State
+```
+
+|OperationId|Operation|StartedOn|LastUpdatedOn|Duration|State|
+|---|---|---|---|---|---|
+|62e40fad-516c-4133-814f-f509e889d006|DataIngestPull|2026-01-05 18:57:10.4234023|2026-01-05 18:57:53.4074572|00:00:42.9840549|Completed|
+|989b527f-20da-48fe-ae22-deba91e20764|OperationsShow|2026-01-05 18:57:54.1959438|2026-01-05 18:57:54.2036051|00:00:00.0076613|Completed|
+|1671b635-b42c-45c6-928c-ad3f3436cb75|TableAppend|2026-01-05 18:58:13.3479575|2026-01-05 18:58:13.3974038|00:00:00.0494463|InProgress|
+|55bc427e-d576-40dc-bd38-58f9df34d357|DatabasesShow|2026-01-05 18:58:49.4693980|2026-01-05 18:58:49.4697805|00:00:00.0003825|Completed|
+
+The following command returns the entire log (not only latest state) for operation with ID `b152f9da-616a-40a7-8cde-f2390cfc8064`:
+
+```kusto
+.show operations 
+| where LastUpdatedOn >  ago(1h)
+| where OperationId == "b152f9da-616a-40a7-8cde-f2390cfc8064"
+| project OperationId, Operation, StartedOn, LastUpdatedOn, Duration, State, Status
+```
+
+|OperationId|Operation|StartedOn|LastUpdatedOn|Duration|State|Status|
+|---|---|---|---|---|---|---|
+|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:36.8136477|00:00:00.0000758|InProgress||
+|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:36.8143127|00:00:00.0006764|InProgress|Assigned|
+|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:36.8146015|00:00:00.0009574|InProgress|Assigned|
+|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:40.2102817|00:00:03.3966431|InProgress|Extent(s) created; metadata updated; cluster map updated|
+|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:40.2103713|00:00:03.3967237|Completed|Extent(s) created; metadata updated; cluster map updated|
+
+The following command returns only the latest state for the same operation. Operations that completed over 6 hours ago will not be returned using this method. Use the options above to query for entries that are older than 6 hours.
+
 ```kusto
-.show operations
+.show operations b152f9da-616a-40a7-8cde-f2390cfc8064
+| project OperationId, Operation, StartedOn, LastUpdatedOn, Duration, State, Status
 ```
 
-|ID |Operation |Node ID |Started On |Last Updated On |Duration |State |Status |
-|--|--|--|--|--|--|--|--|
-|3827def6-0773-4f2a-859e-c02cf395deaf |SchemaShow | |2015-01-06 08:47:01.0000000 |2015-01-06 08:47:01.0000000 |0001-01-01 00:00:00.0000000 |Completed |
-|841fafa4-076a-4cba-9300-4836da0d9c75 |DataIngestPull |Kusto.Azure.Svc_IN_1 |2015-01-06 08:47:02.0000000 |2015-01-06 08:48:19.0000000 |0001-01-01 00:01:17.0000000 |Completed |
-|e198c519-5263-4629-a158-8d68f7a1022f |OperationsShow | |2015-01-06 08:47:18.0000000 |2015-01-06 08:47:18.0000000 |0001-01-01 00:00:00.0000000 |Completed |
-|a9f287a1-f3e6-4154-ad18-b86438da0929 |ExtentsDrop | |2015-01-11 08:41:01.0000000 |0001-01-01 00:00:00.0000000 |0001-01-01 00:00:00.0000000 |InProgress |
-|9edb3ecc-f4b4-4738-87e1-648eed2bd998 |DataIngestPull | |2015-01-10 14:57:41.0000000 |2015-01-10 14:57:41.0000000 |0001-01-01 00:00:00.0000000 |Failed |Collection was modified. Enumeration operation may not execute. |
+|OperationId|Operation|StartedOn|LastUpdatedOn|Duration|State|Status|
+|---|---|---|---|---|---|---|
+|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:40.2103713|00:00:03.3967237|Completed|Extent(s) created; metadata updated; cluster map updated|
 
 ## Related content
 

data-explorer/kusto/management/update-policy.md

@@ -92,8 +92,9 @@ Each such object is represented as a JSON property bag, with the following prope
 |Property |Type | Description  |
 |---------|---------|----------------|
 |IsEnabled  |`bool` |States if update policy is *true* - enabled, or *false* - disabled|
-|Source |`string` |Name of the table that triggers invocation of the update policy |
-|Query |`string` |A query used to produce data for the update |
+|Source |`string` |Name of the table that triggers invocation of the update policy. |
+|SourceIsWildCard |`bool` |If *true*, the `Source` property can be a wildcard pattern. See [Update policy with source table wildcard pattern](#update-policy-with-source-table-wildcard-pattern) |
+|Query |`string` |A query used to produce data for the update. |
 |IsTransactional |`bool` |States if the update policy is transactional or not, default is *false*. If the policy is transactional and the update policy fails, the source table isn't updated. |
 |PropagateIngestionProperties  |`bool`|States if properties specified during ingestion to the source table, such as [extent tags](extent-tags.md) and creation time, apply to the target table. |
 |ManagedIdentity | `string` | The managed identity on behalf of which the update policy runs. The managed identity can be an object ID, or the `system` reserved word. The update policy must be configured with a managed identity when the query references tables in other databases or tables with an enabled [row level security policy](row-level-security-policy.md). For more information, see [Use a managed identity to run a update policy](update-policy-with-managed-identity.md). |
@@ -105,6 +106,7 @@ Each such object is represented as a JSON property bag, with the following prope
 |---------|---------|----------------|
 |IsEnabled  |`bool` |States if update policy is *true* - enabled, or *false* - disabled|
 |Source |`string` |Name of the table that triggers invocation of the update policy |
+|SourceIsWildCard |`bool` |If *true*, the `Source` property can be a wildcard pattern. |
 |Query |`string` |A query used to produce data for the update |
 |IsTransactional |`bool` |States if the update policy is transactional or not, default is *false*. If the policy is transactional and the update policy fails, the source table isn't updated. |
 |PropagateIngestionProperties  |`bool`|States if properties specified during ingestion to the source table, such as [extent tags](extent-tags.md) and creation time, apply to the target table. |
@@ -144,6 +146,14 @@ Update policies take effect when data is ingested or moved to a source table, or
 > Data may be lost in all tables with an update policy relationship if the `replace` command is invoked.
 > Consider using `.set-or-append` instead.
 
+## Update policy with source table wildcard pattern
+
+Update policy supports ingesting from multiple source tables that share the same pattern, while using the
+same query as the update policy query. This is useful if you have several source tables, usually sharing the same schema (or a subset of columns that share a common schema), and you would like to trigger ingestion to a
+single target table, when ingesting to either of those tables. In this case, instead of defining multiple
+update policies, each for a single source table, you can define a single update policy with wildcard as `Source`.The `Query` of the update policy must comply with all source tables matching the pattern.
+To reference the source table in the update policy query, you can use a special symbol named `$source_table`. See example in [Example of wild card update policy](#example-of-wild-card-update-policy).
+
 ## Remove data from source table
 
 After ingesting data to the target table, you can optionally remove it from the source table. Set a soft-delete period of `0sec` (or `00:00:00`) in the source table's [retention policy](retention-policy.md), and the update policy as transactional. The following conditions apply:
@@ -251,6 +261,86 @@ In this example, use an update policy with a simple function to perform ETL. Fir
      .alter-merge table MySourceTable policy retention softdelete = 0s
     ```
 
+## Example of wild card update policy
+
+The following example creates an update policy with a single entry on table `TargetTable`. The policy references all tables matching pattern `SourceTable*` as its source.
+Any ingestion to a table which matches the pattern (in local database) will trigger the update policy, and ingest data to `TargetTable`, based on the update policy query.
+
+1. Create two source tables:
+
+    ```kusto
+    .create table SourceTable1(Id:long, Value:string)
+    ```
+
+    ```kusto
+    .create table SourceTable2(Id:long, Value:string)
+    ```
+
+1. Create the target table:
+
+    ```kusto
+    .create table TargetTable(Id:long, Value:string, Source:string)
+    ```
+
+1. Create a function which will serve as the `Query` of the update policy. The function uses the `$source_table` symbol to reference the `Source` of the update policy. Use `skipValidation=true` to skip validation during the create function, since `$source_table` is only known during update policy execution. The function is validated during the next step, when altering the update policy.
+
+    ```kusto
+    .create function with(skipValidation=true) IngestToTarget()
+    {
+        $source_table 
+        | parse Value with "I'm from table " Source
+        | project Id, Value, Source
+    }
+    ```
+
+1. Create the update policy on `TargetTable`. The policy references all tables matching pattern `SourceTable*` as its source.
+
+    ````kusto
+        .alter table TargetTable policy update
+        ```[{ 
+                "IsEnabled": true, 
+                "Source": "SourceTable*", 
+                "SourceIsWildCard" : true,
+                "Query": "IngestToTarget()",
+                "IsTransactional": true,
+                "PropagateIngestionProperties": true
+        }]```
+
+    ````
+
+1. Ingest to source tables. Both ingestions trigger the update policy:
+
+    ```kusto
+    .set-or-append SourceTable1 <| 
+        datatable (Id:long, Value:string)
+        [
+            1, "I'm from table SourceTable1",
+            2, "I'm from table SourceTable1"
+        ]
+    ```
+
+    ```kusto
+    .set-or-append SourceTable2 <| 
+        datatable (Id:long, Value:string)
+        [
+            3, "I'm from table SourceTable2",
+            4, "I'm from table SourceTable2"
+        ]
+    ```
+
+1. Query `TargetTable`:
+
+    ```kusto
+     TargetTable
+    ```
+
+    |Id|Value|Source|
+    |---|---|---|
+    |1|I'm from table SourceTable1|SourceTable1|
+    |2|I'm from table SourceTable1|SourceTable1|
+    |3|I'm from table SourceTable2|SourceTable2|
+    |4|I'm from table SourceTable2|SourceTable2|
+
 ## Related content
 
 * [Common scenarios for using table update policies](update-policy-common-scenarios.md)