Skip to main content
Sumo Logic

Log Ingest Data Volume Index

The Data Volume Index is populated with a set of log messages every five minutes. The messages contain information on how much data (by bytes and messages count) your account is ingesting. Each log message includes information based on one of the following Index Source Categories.

Index Log Type Index Source Category
Collector collector_volume
Source source_volume
SourceName sourcename_volume
SourceCategory sourcecategory_volume
SourceHost sourcehost_volume
View view_volume

You can query the Data Volume Index just like any other message using the Sumo Logic Search page. To see the data created within the Data Volume Index, when you search, specify the _index metadata field with a value of sumologic_volume. For more information, see Search Metadata.

Known Issue

There is a known issue when searching against _sourceCategory values where Scheduled Views show up blank. This causes results to be returned with numbers as the _sourceCategory values.

For example, you would see:

"sizeInBytes":2862,
"count":353325

In this case, the _sourceCategory is returned as 2862, which is the actual size of the Default Index from the Scheduled View.

Query the Data Volume Index

  1. In the Search page, enter the query _index=sumologic_volume
     
  1. Choose the time range for the data that you'd like to review.
  2. Click Start to run the search. Results return in the Messages tab.

To further limit the search results to the Data Volume Index data for a specific volume category, you can supply the Index Source Category using the _sourceCategory metadata and one of the Index Source Categories from the previous table. For example:

_index=sumologic_volume AND _sourceCategory=collector_volume

 

Data Volume Index Message Format

The Data Volume Index messages are JSON formatted messages that contain parent objects for each source data point, and child objects that detail the message size and count for each parent.

For example, a single message for the "Collector" volume data may look similar to the following, withcollector_X representing the Collector names. The sizeInBytes and count values are the aggregated volume for that five minute time period.

{
    "collector_a":{"sizeInBytes":733296,"count":1646},
    "collector_b":{"sizeInBytes":4380031,"count":12105},
    "collector_c":{"sizeInBytes":386255,"count":843},
    "collector_d":{"sizeInBytes":10823082,"count":23923},
    .
    .
}

Examples

Volume for Each Category

This example query will return the volume for each Source Category.

_index=sumologic_volume _sourceCategory=sourcecategory_volume
| parse regex "\"(?<sourcecategory>(?:[^\"]+)|(?:\"\"))\"\:\{\"sizeInBytes\"\:(?<bytes>\d+),\"count\"\:(?<count>\d+)\}" multi
| bytes/1024/1024/1024 as gbytes
| sum(gbytes) as gbytes by sourcecategory

would produce results such as:

Volume for Each Collector

This example query will return the volume for each Collector.

_index=sumologic_volume _sourceCategory=collector_volume
| parse regex "\"(?<collector>(?:[^\"]+)|(?:\"\"))\"\:\{\"sizeInBytes\"\:(?<bytes>\d+),\"count\"\:(?<count>\d+)\}" multi
| bytes/1024/1024/1024 as gbytes
| sum(gbytes) as gbytes by collector

would produce results such as:

Volume for a Specific Source

The following query returns the message volume for a specific Source. The Source name can be supplied within a JSON operation to get the child objects for that Source.

_index=sumologic_volume _sourceCategory=source_volume

| json "my_source_name" as source

| json field=source "sizeInBytes", "count"

| sizeinbytes/1024/1024/1024 as gbytes

Volume for a Specific Collector

The following query returns the message volume for a specific Collector. The Collector name can be supplied within a JSON operation to get the child objects for that Collector.

_index=sumologic_volume _sourceCategory=collector_volume prod-receiver-10
| json "prod-receiver-10" as collector_json
| json field=collector_json "sizeInBytes", "count" as bytes, count
| sum(bytes) as bytes
| bytes/1024/1024/1024 as gbytes
| fields gbytes

Volume for a Specific Source Host

The following query returns the message volume for a specific Source Host.

_index=sumologic_volume
| where _sourceCategory="sourcehost_volume"
| parse regex "(?<sourcehost>\"[^\"]+\")\:\{\"sizeInBytes\"\:(?<bytes>\d+),\"count\"\:(?<count>\d+)\}" multi
| bytes/1024/1024/1024 as gbytes
| sum(gbytes) as gbytes by sourcehost
| sort by gbytes
| round(gbytes)

Volume for the Default Index

The following query returns the message volume for the Default Index. 

_index=sumologic_volume|where _sourceCategory="view_volume"
| parse regex "(?<view_name>\"[^\"]+\"|\"\")\:\{\"sizeInBytes\"\:(?<bytes>\d+),\"count\"\:(?<count>\d+)\}" multi
| where view_name = "\"Default Index\""
| bytes/1024/1024/1024 as gbytes
| sum(gbytes) as gbytes by view_name
| sort by view_name desc 

Which would produce results such as:

Sumo Logic App for Data Volume

Sumo Logic provides an application that utilizes the Data Volume Index to see your account's volume usage as a glance. For details, see Data Volume app.