diff --git a/data-explorer/get-data-storage.md b/data-explorer/get-data-storage.md index 20bdee9b2b..27e3b023c3 100644 --- a/data-explorer/get-data-storage.md +++ b/data-explorer/get-data-storage.md @@ -1,17 +1,15 @@ --- -title: Get data from Azure storage +title: Get Data from Azure Storage description: Learn how to get data from Azure storage in Azure Data Explorer. ms.reviewer: sharmaanshul ms.topic: how-to -ms.date: 07/31/2025 +ms.date: 12/31/2025 --- # Get data from Azure storage Data ingestion is the process used to load data from one or more sources into a table in Azure Data Explorer. Once ingested, the data becomes available for query. In this article, you learn how to get data from Azure storage (ADLS Gen2 container, blob container, or individual blobs) into either a new or existing table. -Ingestion from an Azure storage account is a one-time operation. To ingest data continuously, see [Configure streaming ingestion](ingest-data-streaming.md). - For general information on data ingestion, see [Azure Data Explorer data ingestion overview](ingest-data-overview.md). > [!WARNING] @@ -50,29 +48,31 @@ Select the data source from the available list. In this example, you're ingestin 1. To add your source, select **Select container** or **Add URI**. - 1. If you selected **Select container**, fill in the following fields: +1. If you selected **Select container**, fill in the following fields: + + :::image type="content" source="media/get-data-storage/configure-tab.png" alt-text="Screenshot of configure tab with new table entered and one sample data file selected." lightbox="media/get-data-storage/configure-tab.png"::: - :::image type="content" source="media/get-data-storage/configure-tab.png" alt-text="Screenshot of configure tab with new table entered and one sample data file selected." lightbox="media/get-data-storage/configure-tab.png"::: + | **Setting** | **Field description** | + |--------------------------|----------| + | Subscription | The subscription ID where the storage account is located. | + | Storage account | The name that identifies your storage account. | + | Container | The storage container you want to ingest. | + | **File filters (optional)** | | + | Folder path| Filters data to ingest files with a specific folder path. | + | File extension| Filters data to ingest files with a specific file extension only.| - | **Setting** | **Field description** | - |--------------------------|----------| - | Subscription | The subscription ID where the storage account is located. | - | Storage account | The name that identifies your storage account. | - | Container | The storage container you want to ingest. | - | **File filters (optional)** | | - | Folder path| Filters data to ingest files with a specific folder path. | - | File extension| Filters data to ingest files with a specific file extension only.| +1. If you selected **Add URI**: - 1. If you selected **Add URI**, from the storage account, generate an SAS URL for the container or individual blobs you want to ingest. Set the permissions to **Read** and **List** for containers or **Read** for individual blobs. For more information, see [Generate a SAS token](/kusto/api/connection-strings/generate-sas-token?view=azure-data-explorer&preserve-view=true). + 1. From the storage account, generate an SAS URL for the container or individual blobs you want to ingest. Set the permissions to **Read** and **List** for containers or **Read** for individual blobs. For more information, see [Generate a SAS token](/kusto/api/connection-strings/generate-sas-token?view=azure-data-explorer&preserve-view=true). - 1. Paste the URL into the **URI** field, and then select plus (**+**). You can add multiple URIs for individual blobs, or a single URI for a container. + 1. Paste the URL into the **URI** field, and then select plus (**+**). You can add multiple URIs for individual blobs, or a single URI for a container. - :::image type="content" source="media/get-data-storage/add-uri.png" alt-text="Screenshot of configure tab with the connection string pasted in the URI field." lightbox="media/get-data-storage/add-uri.png"::: + :::image type="content" source="media/get-data-storage/add-uri.png" alt-text="Screenshot of configure tab with the connection string pasted in the URI field." lightbox="media/get-data-storage/add-uri.png"::: - > [!NOTE] - > * You can add up to 10 individual blobs. Each blob can be a max of 1 GB uncompressed. - > * You can ingest up to 5000 blobs from a single container. - > * You can't ingest individual blobs and containers in the same ingestion. + > [!NOTE] + > * You can add up to 10 individual blobs. Each blob can be a max of 1 GB uncompressed. + > * You can ingest up to 5,000 blobs from a single container. + > * You can't ingest individual blobs and containers in the same ingestion. 1. Select **Next** @@ -86,7 +86,6 @@ To complete the ingestion process, select **Finish**. Optionally: -* Select **Command viewer** to view and copy the automatic commands generated from your inputs. * Use the **Schema definition file** dropdown to change the file that the schema is inferred from. * Change the automatically inferred data format by selecting the desired format from the dropdown. See [Data formats supported by Azure Data Explorer for ingestion](ingestion-supported-formats.md). * [Edit columns](#edit-columns). @@ -102,7 +101,7 @@ Optionally: ## Summary -In the **Data preparation** window, all three steps are marked with green check marks when data ingestion finishes successfully. You can view the commands that were used for each step, or select a card to query, visualize, or drop the ingested data. +In the **Summary** window, all three steps are marked with green check marks when data ingestion finishes successfully. You can view the commands that were used for each step, or select a card to query, visualize, or drop the ingested data. :::image type="content" source="media/get-data-storage/summary.png" alt-text="Screenshot of summary page with successful ingestion completed." lightbox="media/get-data-storage/summary.png"::: diff --git a/data-explorer/kusto/management/batching-policy.md b/data-explorer/kusto/management/batching-policy.md index cca68d3e34..323d4f4b3a 100644 --- a/data-explorer/kusto/management/batching-policy.md +++ b/data-explorer/kusto/management/batching-policy.md @@ -28,6 +28,7 @@ The following list shows the basic batching policy triggers to seal a batch. A b * `Size`: Batch size limit reached or exceeded * `Count`: Batch file number limit reached * `Time`: Batching time has expired +* `FlushImmediatly`: A blob with ['FlushImmediately'](../api/netfx/kusto-ingest-client-reference.md#class-kustoqueuedingestionproperties) indication caused a batch to be sealed The `IngestionBatching` policy can be set on databases or tables. Default values are as follows: **5 minutes** maximum delay time, **500** items, total size of **1 GB**. @@ -40,7 +41,7 @@ The `IngestionBatching` policy can be set on databases or tables. Default values The following list shows conditions to seal batches related to single blob ingestion. A batch is sealed and ingested when the conditions are met: -* `SingleBlob_FlushImmediately`: Ingest a single blob because ['FlushImmediately'](../api/netfx/kusto-ingest-client-reference.md#class-kustoqueuedingestionproperties) was set +* `SingleBlob_FlushImmediately`: **Deprecated!** Ingest a single blob because ['FlushImmediately'](../api/netfx/kusto-ingest-client-reference.md#class-kustoqueuedingestionproperties) was set * `SingleBlob_IngestIfNotExists`: Ingest a single blob because ['IngestIfNotExists'](../ingestion-properties.md#ingestion-properties) was set * `SingleBlob_IngestByTag`: Ingest a single blob because ['ingest-by'](extent-tags.md) was set diff --git a/data-explorer/kusto/management/data-ingestion/queued-ingest-use-http.md b/data-explorer/kusto/management/data-ingestion/queued-ingest-use-http.md index 64590e68fb..d120e4709e 100644 --- a/data-explorer/kusto/management/data-ingestion/queued-ingest-use-http.md +++ b/data-explorer/kusto/management/data-ingestion/queued-ingest-use-http.md @@ -84,7 +84,7 @@ Authorization: Bearer "blobs": [ { "url": "https://example.com/blob1.csv.gz", - "sourceId": "123a6999-411e-4226-a333-w79992dd9b95", + "sourceId": "123a6999-411e-4226-a333-a79992dd9b95", "rawSize": 1048576 } ], diff --git a/data-explorer/media/get-data-storage/add-uri.png b/data-explorer/media/get-data-storage/add-uri.png index 3255c70c91..cc45fa1deb 100644 Binary files a/data-explorer/media/get-data-storage/add-uri.png and b/data-explorer/media/get-data-storage/add-uri.png differ diff --git a/data-explorer/media/get-data-storage/configure-tab.png b/data-explorer/media/get-data-storage/configure-tab.png index cc65966fa1..e9fc448cc5 100644 Binary files a/data-explorer/media/get-data-storage/configure-tab.png and b/data-explorer/media/get-data-storage/configure-tab.png differ diff --git a/data-explorer/media/get-data-storage/get-data.png b/data-explorer/media/get-data-storage/get-data.png index ad1809402f..2788866844 100644 Binary files a/data-explorer/media/get-data-storage/get-data.png and b/data-explorer/media/get-data-storage/get-data.png differ diff --git a/data-explorer/media/get-data-storage/inspect-data.png b/data-explorer/media/get-data-storage/inspect-data.png index a23fab8e66..ebc7627778 100644 Binary files a/data-explorer/media/get-data-storage/inspect-data.png and b/data-explorer/media/get-data-storage/inspect-data.png differ diff --git a/data-explorer/media/get-data-storage/select-data-source.png b/data-explorer/media/get-data-storage/select-data-source.png index f10afdd621..d0700dca31 100644 Binary files a/data-explorer/media/get-data-storage/select-data-source.png and b/data-explorer/media/get-data-storage/select-data-source.png differ diff --git a/data-explorer/media/get-data-storage/summary.png b/data-explorer/media/get-data-storage/summary.png index 9e5896d42c..274dfafb9d 100644 Binary files a/data-explorer/media/get-data-storage/summary.png and b/data-explorer/media/get-data-storage/summary.png differ