Troubleshoot Azure Blob Storage Log Collection

If logs do not start flowing into Sumo Logic after you perform the Collect Logs from Azure Blob Storage procedure, see the troubleshooting tips below.

Error while deploying the ARM template

If you received the following error while deploying the ARM template: The Resource 'X' under resource group 'Y' was not found.

If you attempted to redeploy the template after getting this error and used the same storage account + resource group values in the first failed deployment, follow these steps.

Refer to this video, then redeploy the template, in the following way: 

1. Go to the resource group where the deployment failed
2. Click on Failed deployment, and then click on Microsoft.Template(last deployment) in the next window.
3. Click Redeploy, provide the required fields, and click Purchase. You should see the successful deployment notification.

If you get namespace invalid error make sure it follows the naming convention specified in this doc.

{
 "code": "DeploymentFailed",
 "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
 "details": [
 {
 "code": "BadRequest",
 "message": "{\r\n \"error\":
{\r\n \"message\": \"The specified service namespace is invalid. CorrelationId: fb6a08d9-fc78-4540-a79e-861f1d81fe2e\",\r\n \"code\": \"BadRequest\"\r\n }
\r\n}"
 },
 {
 "code": "BadRequest",
 "message": "{\r\n \"error\":
{\r\n \"message\": \"The specified service namespace is invalid. CorrelationId: fb6a08d9-fc78-4540-a79e-861f1d81fe2e\",\r\n \"code\": \"BadRequest\"\r\n }
\r\n}"
 }
 ]

For common deployment errors, refer to Troubleshoot common Azure deployment errors. 

AutoScaling producer function to handle huge load on creating tasks for consumer function

1. Go to the Producer function app.
2. Under Settings blade, select Scale out (App Service plan).
3. Select Rule Based Scaling.
4. Add your rules based scaling configuration as defined in Create your first autoscale setting.

Verify configurations

Make sure that the resources you created in the Collect Logs from Azure Blob Storage procedure were successfully created.

1. Go to Resource groups, and select the resource group you created or selected in Step 3. Configure Azure resources using ARM Template of the "Collect Logs from Azure Blob Storage" page. You should see resources you created:
   • Two App Service plans. 
   • Three App Services.
   • A Service Bus Namespace.
   • An Event Hubs Namespace.
   • A Storage account.
2. In the left pane of the Azure Portal, click AppServices, and search for “SUMOBRTaskConsumer”. You should find the “SUMOBRTaskConsumer\<random-string\>” Function App and click it. 
3. Click the Application settings link. Check that the value of the SumoLogEndpoint field matches the HTTP source URL. 

Verify Block Blob Create Events are getting published

1. Click All Services. Go to Event Grid Subscription services. 
2. Select Storage Account and region from the dropdown.
3. Select the Event Subscription created in Step 3 from the list.
4. Click Metrics.
5. On the Event Grid Metrics page, check that the Publish Succeeded and Delivery Succeeded counts are greater than zero.
   

Verify Event Hub is receiving log messages

To verify that events are appearing in your event hub:

1. In the left pane of Azure Portal, Click Eventhub.
2. Search for “SUMOBREventHubNamespace”. You should find the “SUMOBREventHubNamespace\<random-string>” Event Hub Namespace and click it. 
3. Click the Messages link.
4. Message summary information appears below the chart. Check that the Incoming Messages count is greater than zero.
   

Verify Service Bus Queue is receiving tasks

Go to Service Bus Service from the Azure portal and click on SUMOBRTaskQueueNamespace\<unique string> Service Bus Namespace. Check that the incoming messages count is greater than zero.

Verify with Live Tail

In Sumo, open a Live Tail tab and run a search to verify Sumo is receiving events. Search by the source category you assigned to the HTTP Source that receives the log data, for example: _sourceCategory="azure/ad"

For more information about using Live Tail, see Live Tail. 

Verify Azure Function is not getting Failed

1. Go to Function App.
2. Go to Application Insights.
   
3. Go to Failures section.
   
4. Click on the Function Name under Operation Name
5. In the top 3 exception types, click on the count it will open a sample exception.
6. Click on any exception it will open an end to end transaction details page where you can click on View all telemetry to view all the logs for that execution.
   

Common Azure function errors

`ExitCode C0000005

ExitCodeString NATIVE ACCESS VIOLATION

Managed Exception = System.AccessViolationException:Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

CallStack - Managed Exception`

The above error occurs in certain situations the runtime initiates a host shutdown via HostingEnvironment.InitiateShutdown, for example, when an unhandled global exception occurs, when a function TimeoutException is thrown, or when performance counter thresholds are exceeded (HostHealthMonitor).

If you're using this function for quite some time then we recommend redeploying the solution with new ARM templates.

If the error still persists, then

1. If you want to collect metrics only  you can migrate from Consumption plan to Premium plan by making changes in the ARM template.

{
  "type":"Microsoft.Web/serverfarms",
  "kind":"app",
  "name":"[parameters('serverfarms_SumoAzureLogsAppServicePlan_name')]",
  "apiVersion":"2018-02-01",
  "location":"[resourceGroup().location]",
  "sku":{
    "name":"P1v2",
    "tier":"PremiumV2",
    "size":"P1v2",
    "family":"Pv2",
    "capacity":2
  },
  "properties":{
    "maximumElasticWorkerCount":1,
    "perSiteScaling":false,
    "targetWorkerCount":0,
    "targetWorkerSizeId":0,
    "reserved":false,
    "isSpot":false,
    "isXenon":false,
    "hyperV":false
  },
  "dependsOn":[

  ]
}

2. If you want to collect only logs from Azure Monitor, we recommend switching to new Cloud-to-Cloud collection for Event hub

Exception while executing function: Functions.BlobTaskProducer
StorageError: The table specified does not exists 

RequestId: 87039a85-e002-0042-252ddb36f8000000 Time:2021-02-12T21:12:23

This error occurs when the FileOffsetMap table is not created inside Table service.