Skip to main content
Sumo Logic

Collect Logs for the Azure Network Watcher App

This document provides instructions for configuring the collection of NSG Flow Logs for the Azure Network Watcher App. This process includes the following requirements and tasks:

Configuration requirements

Before you begin configuring NSG Flow Log collection, make sure the following environment prerequisites are met:

  • Your Storage Account must be of type General-purpose v2 or Blob storage.
  • Your Network Security Group and Storage Account should be in same resource location.
  • You also need to have Microsoft Authorization/role Assignments/write permissions, so they should be a "User Access Administrator" or "Owner".

Configure Azure Storage Account

In this step you configure a storage account to which you will export monitoring data for your Azure service.   

If you have a storage account with a container that you want to use for this purpose, make a note of its resource group, storage account name and container name and proceed to Step 2.

To configure an Azure storage account, do the following:
  1. Create a new storage account General-purpose v2 (GPv2) storage account. For instructions, see Create a storage account in Azure help.
  2. Create a container(Optional) all services in azure create containers automatically. This step is needed only when you are exporting custom logs in some container.
    • In the Azure portal, navigate to the storage account you just created (in the previous step).
    • Select Blobs under Blob Service.
      • Select + Container,
      • Enter the Name
      • Select Private for the Public Access Level.
      • Click OK.

 And now follow the Step below. By default, the flow logs are in insights-logs-networksecuritygroupflowevent container.

If you have a storage account that you want to use for this purpose, make a note of its resource group, storage account name, then proceed to Step below. If you want to collect only flow logs from the storage account then you can add the filter /blobServices/default/containers/insights-logs-networksecuritygroupflowevent/

Configure an HTTP source

This section demonstrates how to configure an HTTP source to receive logs from the Azure function.

To configure an HTTP source for Azure, do the following:
  1. Select a hosted collector where you want to configure the HTTP source. If desired, create a new hosted collector, as described on Configure a Hosted Collector.
  2. Configure an HTTP source, as described on HTTP Logs and Metrics Source. Make a note of the URL for the source, you will need it in the next step. I
  3. In Advanced Options for Logs, under Timestamp Format, click Specify a format and enter the following:
  • Specify Format as epoch
  • Specify Timestamp locator as \"time\": (.*),

Configure Azure Resources using ARM template

In this step, you use a Sumo-provided Azure Resource Manager (ARM) template to create an Event Hub, three Azure functions, Service Bus Queue, and a Storage Account.

  1. Download the blobreaderdeploy.json ARM template.
  2. Click Create a Resource, search for Template deployment in the Azure Portal, and then click Create.
  3. On the Custom deployment blade, click Build your own template in the editor.
  4. Copy the contents of the template and paste it into the editor window.

    edit-template.png
  5. Click Save.
  6. On the Custom deployment blade, do the following:
    1. Create a new Resource Group (recommended) or select an existing one.
    2. Choose Location.
    3. Set the values of the following parameters:
  • SumoEndpointURL: URL for the HTTP source you configured in Step 2 above.
  • StorageAccountName: Name of the storage account where  you are storing logs from Azure Service, that you configured in Step 1 above.
  • StorageAccountResourceGroupName: Name of the resource group of the storage account you configured in Step 1 above.
  • Filter Prefix (Optional): If you want to filter logs from a specific container, enter the following, replacing the variable with your container name: /blobServices/default/containers/<container_name>/
  1. Select the check box to agree to the terms and conditions, and then click Purchase.

Azure_Blob_Storage_Custom_Deployment.png

  1. Verify the deployment was successful by looking at Notifications at top right corner of Azure Portal.

notification-success.png

  1. (Optional) In the same window, click Go to resource group to verify the all resources were successfully created, such as shown in the following example:

Azure_Blob_all-resources.png

  1. Go to Storage accounts and search for sumobrlogs, then select sumobrlogs<random-string>.

storage-accounts.png

  1. Under Table Service do the following:
    1. Click Tables.
    2. Click + Table.
    3. For Name, enter FileOffsetMap.
  2. Click OK.

Azure_Blob_create-table.png

Enable NSG flow logs via the Azure Portal

In this step, you enable NSG flow logs with the Azure portal.

Troubleshooting

If logs don't start flowing into Sumo Logic after you perform the configuration above, see Troubleshoot Azure Blob Storage Log Collection.

Sample Log Message

{  
  "time":"2017-09-27 21:22:33.443+0000",
  "sys_id":"4181995a-801f-4075-a56c-30b3671148bf",
  "category":"NetworkSecurityGroupFlowEvent",
  "resource_id":"/SUBSCRIPTIONS/C088DC46-D692-42AD-A4B6-9A542D28AD2A/RESOURCEGROUPS/AZURELABS/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-AZURELABS-03",
  "event_name":"NetworkSecurityGroupFlowEvents",
  "rule_name":"All_prod_tcp",
  "mac":"000D3AF86058",
  "src_ip":"51.148.136.204",
  "dest_IP":"107.198.121.243",
  "src_port":"47676",
  "dest_port":"4367",
  "protocol":"T",
  "traffic_destination":"I",
  "traffic_a/d":"D"
}

Query Sample

Denied Traffic Flow by Source Location

_sourceCategory="security/flowlogs"
| json field=_raw "rule_name" 
| json field=_raw "resource_id"
| json field=_raw "event_name"
| json field=_raw "mac"
| json field=_raw "src_ip"
| json field=_raw "dest_IP"
| json field=_raw "dest_port"
| json field=_raw "protocol"
| json field=_raw "traffic_destination"
| json field=_raw "traffic_a/d" as traffic_a_d
| parse regex field=resource_id"(?<NSG>[\w-_.]+)$"
| json field=_raw "src_port"
| where traffic_a_d = "D"
| lookup latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code from geo://default on ip = src_ip
| count by latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code
| sort _count