Skip to main content
Sumo Logic

Collect Logs for the Azure Blob Storage App

instructions for configuring a pipeline for shipping logs available from Azure Blob Storage to an Event Hub, on to an Azure Function, and finally to an HTTP source on an hosted collector in Sumo Logic.

This page has instructions for configuring a pipeline for shipping logs available from Azure Blob Storage to an Event Hub, on to an Azure Function, and finally to an HTTP source on an hosted collector in Sumo Logic. 

Requirements 

  • Only General-purpose v2 (GPv2) and Blob storage accounts are supported. This integration does not support General-purpose v1 (GPv1) accounts.  
  • Configure your storage account in the same location as your Azure Service.
  • This solution supports only log files from Blob storage that have file extensions of .csv, .json, .blob, or .log. 

Functional overview

  1. You configure the Azure service to export logs to a container in a storage account created for that purpose.
  2. The ARM template creates an Event Grid subscription with the storage container as publisher and the event hub (created by the Sumo-provided ARM) as subscriber. Event Grid routes block blob creation events to event hub.
  3. Event Hub streams the events to the TaskProducer Azure function, which creates tasks (a JSON object that specifies start and end byte, container name, blob path) and pushes those tasks to the service bus task queue.
  4. The TaskConsumer Azure function, which is triggered when the service bus receives a new task, reads the block blob, from start byte to stop byte, and sends that data to Sumo. 
  5. The set up also includes failure handling mechanism. For more information about the solution strategy, see Azure Blob Storage.

Step 1. Configure Azure storage account 

In this step you configure a storage account to which you will export monitoring data for your Azure service.   

If you have a storage account with a container that you want to use for this purpose, make a note of its resource group, storage account name and container name and proceed to Step 2.

To configure an Azure storage account, do the following:
  1. Create a new storage account General-purpose v2 (GPv2) storage account. For instructions, see Create a storage account in Azure help.
  2. In the Azure portal, navigate to the storage account you just created (in the previous step).
  3. Select Blobs under Blob Service.
    Make a note of the container name, you will need to supply later in this procedure. 
    1. Select + Container,
    2. Enter the Name
    3. Select Private for the Public Access Level.
    4. Click OK.

Step 2. Configure an HTTP source

 In this step, you configure an HTTP source to receive logs from the Azure function.

  1. Select a hosted collector where you want to configure the HTTP source. If desired, create a new hosted collector, as described on Configure a Hosted Collector.
  2. Configure an HTTP source, as described on HTTP Logs and Metrics Source. Make a note of the URL for the source, you will need it in the next step.

Step 3. Configure Azure resources using ARM template

In this step, you use a Sumo-provided Azure Resource Manager (ARM) template to create an Event Hub, three Azure functions, Service Bus Queue, and a Storage Account.

  1. Download the blobreaderdeploy.json ARM template.
  2. Click Create a Resource, search for Template deployment in the Azure Portal, and then click Create.
  3. On the Custom deployment blade, click Build your own template in the editor.
  4. Copy the contents of the template and paste it into the editor window.

    edit-template.png
  5. Click Save.
  6. On the Custom deployment blade, do the following:
    1. Create a new Resource Group (recommended) or select an existing one.
    2. Choose Location.
    3. Set the values of the following parameters:
  • SumoEndpointURL: URL for the HTTP source you configured in Step 2 above.
  • StorageAccountName: Name of the storage account where  you are storing logs from Azure Service, that you configured in Step 1 above.
  • StorageAccountResourceGroupName: Name of the resource group of the storage account you configured in Step 1 above.
  • Filter Prefix (Optional): If you want to filter logs from a specific container, enter the following, replacing the variable with your container name: /blobServices/default/containers/<container_name>/
  1. Select the check box to agree to the terms and conditions, and then click Purchase.

Azure_Blob_Storage_Custom_Deployment.png

  1. Verify the deployment was successful by looking at Notifications at top right corner of Azure Portal.

notification-success.png

  1. (Optional) In the same window, click Go to resource group to verify the all resources were successfully created, such as shown in the following example:

Azure_Blob_all-resources.png

  1. Go to Storage accounts and search for sumobrlogs, then select sumobrlogs<random-string>.

storage-accounts.png

  1. Under Table Service do the following:
    1. Click Tables.
    2. Click + Table.
    3. For Name, enter FileOffsetMap.
  2. Click OK.

Azure_Blob_create-table.png

Step 4. Push logs from Azure Service to Azure Blob Storage

This section describes how to push logs from an Azure service to Azure Blob Storage by configuring Diagnostic Logs. The instructions use the Azure Web Apps Service as an example. 

  1. Login to the Azure Portal.
  2. Click AppServices > Your Function App > Diagnostic Logs under Monitoring.
  3. You will see the Diagnostic Logs blade. Enable Application Logging, Web Server Logging, or both, and click Storage Settings.
  4. Select the Storage Account whose connection string you configured in Step 1
  5. In the Containers blade, select the container you created in Step 1.
  6. In the Diagnostic Logs blade, specify the Retention Period (Days), and click Save to exit Diagnostic Logs configuration.

    export-webapp-logs.png

Ingesting from Multiple Storage Accounts (Optional)

If you want to ingest data into Sumo from multiple storage accounts, perform following tasks for each storage account separately.

Step 1: Authorize App Service to list Storage Account key

This section provides instructions on authorizing the App Service to list the Storage Account key. This enables the Azure function to read from the storage account.

To authorize the App Service to list the Storage Account key, do the following:
  1. Go to Storage Account and click Access Control(IAM).

AzureBlob_AccessControl_IAM.png

  1. Click the Add + at the top of the page.

AzureBlob_IAM_Add.png

  1. In the Add Permissions window specify the following values:
  • Role: Choose “Storage Account Key Operator Service Role”
  • Assign Access To: Choose App Service
  • Subscription: Pay as you Go
  • Select:  Select SUMOBRDLQProcessor<unique_prefix> and SUMORTaskConsumer<unique_prefix> app services which are created by the ARM template

AzureBlob_AddPermissions.png

  1. Click Save.

Step 2: Create an Event Grid Subscription

This section provides instructions for creating an event grid subscription, that subscribes all blob creation events to the Event Hub created by ARM template in Step 3 above.

To create an event grid subscription, do the following:
  1. In the left pane of Azure portal click All Services, then search for and click Event Grid Subscriptions.

AzureBlob_EventGridSubscriptions.png

  1. At the top of the Event subscriptions page, click +Event Subscription

AzureBlob_EventSubscriptionsPage.png

The Create Event Subscription dialog appears.

AzureBlob_CreatEventSubscription_dialog.png

  1. Specify the following values for Event Subscription Details:
  • Name: Fill the event subscription name.
  • Event Schema: Select Event Grid Schema.
  1. Specify the following values for Topic Details:
  • Topic Type. Select Storage Accounts.
  • Subscription. Select Pay As You Go
  • Resource Group. Select the Resource Group for the Storage Account to which your Azure service will export logs, from where you want to ingest logs. 
  • Resource. Select the Storage Account you configured, from where you want to ingest logs.
  1. Specify the following details for Event Types:
  • Uncheck the Subscribe to all event types box.
  • Select Blob Created from the Define Event Types dropdown.
  1. Specify the following details for Endpoint Types: 
  • Endpoint Type. Select Event Hubs from the dropdown. 
  • Endpoint.  Click on Select an endpoint. 

The Select Event Hub dialog appears.

AzureBlob_SelectEventHub-EventGrid.png

  1. Specify the following Select Event Hub parameters, then click Confirm Selection.
  • Resource Group. Select the resource group you created Step 3 in which all the resources created by ARM template are present.
  • Event Hub Namespace. Select SUMOBREventHubNamespace<unique string>.
  • Event Hub. Select blobreadereventhub from the dropdown.
  1. Specify the following Filters tab options:
  • Check Enable subject filtering.
  • To filter events by container name, enter the following in the Subject Begins With field, replacing <container_name> with the name of the container from where you want to export logs.  /blobServices/default/containers/<container_name>/

AzureBlob_FiltersDialog.png

  1. Click Create.
  2. Verify the deployment was successful by checking Notifications in the top right corner of the Azure Portal.