Skip to main content
Sumo Logic

Metadata Ingest Budgets

Metadata ingest budgets provide you the ability to assign budgets to your log data by either Fields or the following built in metadata fields, _collector, _source, _sourceCategory, _sourceHost, and _sourceName for defining the scope of an ingest budget instead of using a Field Value.

This new budget assignment scheme allows you to have granular control over your ingest budgets while keeping the configuration overhead to a minimum.

Ingest budgets now have a Scope, replacing Field Value, which defines the log data to apply to a budget. For example, a scope expression like _sourceCategory=/dev/catalog/* implies that all incoming logs ingested into Sumo Logic with a matching _sourceCategory will fall under the scope of the given budget.

In this release, the scope expression is limited to a single metadata field consisting of a key and value. You can use either:

  • A Field that is enabled in the Fields table. Fields are created in many ways, see Fields for details.
  • One of the following built-in metadata fields: _collector, _source, _sourceCategory, _sourceHost, or _sourceName.

The value supports a single wildcard, such as _sourceCategory=prod*payment.

Metadata Ingest Budgets.png

Legacy Ingest Budgets

  • Once metadata ingest budget features are enabled you will not be able to create a legacy ingest budget. All new budgets have a scope instead of a field value. Your existing budgets will remain and both old and new ingest budgets will co-exist.
  • When data matches a legacy and metadata budget and either has reached its limit and set to drop data, your data will be dropped.
  • Legacy budgets will get a scope option added, however, it is ignored until you edit it to define a scope of data. You can edit the other settings and it remains a legacy budget. When you edit the scope, the budget will migrate from legacy to metadata. 
  • Only legacy budgets show the assigned collectors list. For metadata budgets, the collector list is empty.

Examples

Control ingest by team or service

You can assign Collectors and Sources with fields based on teams and services. For example, a field could be team=<name of the team> or service=<name of the service>. With these fields assigned, you can create a budget with the scope team=<name of the team> to achieve team based budgets. You can leverage Source fields for finer control over the scope of the budget. You can map a model of your deployment or organization to metadata fields and then create ingest budgets with a scope referencing them.

Match against multiple budgets

Log messages can match against multiple budgets if two or more budgets have overlapping scopes. For example, see the following two budgets:

  1. Budget #1

Scope = "_sourceCategory=/service/payment/*"
Action = "stop collecting"
capacity= 10 GB

  1. Budget #2

Scope = "_sourceCategory=/service/payment/component/oracle"
Action = "stop collecting"
capacity= 2 GB

In this case, all log messages ingested with _sourceCategory=/service/payment/component/oracle will match against both budget #1 and budget #2. This will consume capacity from both budgets. As expected, if either budget reaches its capacity data will stop being collected.

This concept can be used to manage your ingestion with sub-budgets. Let’s understand this better with an example,

To ensure the daily ingestion for the infrastructure components ALB, Kafka, JBoss, and MySQL do not exceed 1 TB in total, and a daily limit of 400 GB, 200 GB, and 100 GB is placed on the ALB, Kafka, and JBoss components respectively you'd use the following configurations.

  1. Configure all infrastructure component logs with the following fields: 

  • ALB logs: "component=ALB"
  • Kafla logs: "component=Kafka"
  • MySQL logs: "component=MySQL"
  • JBoss logs: "component=JBoss"
  1. Create the budgets

    • Budget#1

    Scope=“component=*”
    Capacity= 1 TB
    Action = “stop collecting” 

    • Budget#2

    Scope=“component=ALB”
    Capacity= 400 GB
    Action = “stop collecting” 

    • Budget#3

    Scope=“component=Kafka”
    Capacity= 200 GB
    Action = “stop collecting” 

    • Budget#4

    Scope=“component=JBoss”
    Capacity= 100 GB
    Action = “stop collecting”