Skip to main content
Sumo Logic

Ingest Budgets

availability_enterprise_only_pub.png

Ingest budgets control the capacity of daily ingestion volume sent to Sumo Logic from Collectors. Collectors can be assigned to an ingest budget that defines a daily capacity limit. The capacity is tracked based on the combined volume from Collectors, from all their Sources. When an ingest budget's capacity is reached you can have the Collectors assigned to it stop ingesting data to control costs.

Ingest budgets automatically reset their capacity utilization tracking every 24 hours based on the time and time zone you specify. For example, you can schedule an ingest budget to refresh every day at 02:00 in the America/Los_Angeles time zone. You can manually reset an ingest budget at any time.

An ingest budget's capacity usage is logged in the Audit Index once it reaches 85% and continues to be logged until the budget is reset. You have the option to have its assigned Collectors stop collecting data when the capacity has reached 100%. In the Ingest Budgets API this option is set with the action parameter. When an ingest budget triggers Collectors assigned to it to stop collecting data, the data is not automatically recovered or ingested later once the capacity tracking is reset. To track and schedule alerts on ingest budget capacity-usage and resets see audit ingest budgets.

Rules

  • A Collector can be assigned to one ingest budget.
  • Installed Collectors must be version 19.227-14 or higher.
  • There is a limit of 20 ingest budgets.
  • Use the IANA time zone database format.
  • Specify the time in HH:MM format.
  • Ingest budgets requires the Manage Ingest Budgets role capability. Enabling this will automatically enable the Manage Collectors capability. The Manage Collectors capability on its own permits the re-assignment of budgets to different Collectors, but not creating or deleting them.
  • Data is not automatically recovered or ingested later once the capacity tracking is reset.
  • Updating an ingest budget's Field Value will not propagate to assigned Collectors, instead the Collectors are unassigned as they do not match its configured Field Value.
Source type behavior

A few Sources on Hosted Collectors behave differently when the Collector is instructed to stop collecting data.

  • HTTP Sources will drop data requests yet still return a 200 response.
  • AWS S3 based Sources will skip objects. 
  • Cloud Syslog Sources will keep the connection open yet drop incoming syslog messages.

View ingest budgets

Use the Ingest Budgets page to view your ingest budgets. To access the page go to Manage Data > Collection > Ingest Budgets.

ingest budgets table read only.png

The page displays the following information:

  • Name. Name of the ingest budget.
  • Field Value. Custom value used to assign Collectors to the ingest budget.
  • Capacity. Maximum amount of data permitted.
  • Usage. Percentage of data used.
  • Collectors. Number of Collectors assigned to the ingest budget.
  • Reset Time. Time and time zone to reset the data usage tracking in HH:MM timestamp format. This is fixed at a 24-hour time interval, so the reset time is triggered every 24 hours.
  • Allocated Capacity (bottom of table). The total allocated capacity from all ingest budgets out of your account's available daily log ingest capacity is provided. If you assign all your Collectors to ingest budgets you can easily track how much data you are allowing Collectors to ingest compared to your account's available daily log ingest quota. 

At the top of the page, you can:

  • Search Ingest Budgets

For the ingest budgets listed, select a row to:

  • View Details

Set up an ingest budget

  1. Use the Ingest Budget Management API to create a new ingest budget. This API supports viewing, creating, updating, and deleting ingest budgets. To use the API, view the API's documentation by clicking one of the following links based on your organization's deployment.

    Sumo Logic has several deployments that are assigned depending on the geographic location and the date an account is created. See how to determine which endpoint to use if you are unsure.

    Select the documentation link for your deployment:

    Deployment Documentation URL
    AU https://api.au.sumologic.com/docs/#tag/ingestBudgetManagement
    DE https://api.de.sumologic.com/docs/#tag/ingestBudgetManagement
    EU https://api.eu.sumologic.com/docs/#tag/ingestBudgetManagement
    JP https://api.jp.sumologic.com/docs/#tag/ingestBudgetManagement
    US1 https://api.sumologic.com/docs/#tag/ingestBudgetManagement
    US2 https://api.us2.sumologic.com/docs/#tag/ingestBudgetManagement
  2. Now that you have created an ingest budget you can assign a Collector to it. See assign Collector to ingest budget for details.

Audit ingest budgets

The Audit Index logs events when an ingest budget has exceeded or reached 85 percent of its set capacity limit. When the utilized capacity is below 85 percent no audit logs are available.

Example audit log when an ingest budget is approaching or exceeding capacity, where:

  • budget_name is the name of the ingest budget.
  • budget_value is the ingest budget's field value.
  • Usage status is either Approaching (≥ 85%) or Exceeded (≥ 100%) its set capacity limit.
Budget budget_name with field value budget_value consumed 6330.00% of capacity since last reset at 2018-09-17T13:38:53.663 -0700.
Capacity: 200 bytes
Usage: 12660 bytes
Usage status: Exceeded
Action: drop_data
Next reset: 2018-09-18T13:35:00.000 -0700

Example audit log when an ingest budget is reset, where budget_name is the name of the ingest budget and budget_value is the ingest budget's field value:

Budget budget_name with field value budget_value consumed 0.00% of capacity and is reset at 2018-09-18T00:03:34.574 -0700.
Capacity: 1000 bytes
Usage: 0 bytes
Next reset: 2018-09-19T00:00:00.000 -0700
Audit Index queries

You can schedule the following searches to get alerts when needed, see scheduled searches for details.

Search for when approaching usage capacity:

_index=sumologic_audit _sourceName=VOLUME_QUOTA _sourceCategory=account_management "Budget" "last reset" "Approaching"

Search for all available audit logs:

_index=sumologic_audit _sourceName=VOLUME_QUOTA _sourceCategory=account_management "Budget" "reset"

Search for only reset logs:

_index=sumologic_audit _sourceName=VOLUME_QUOTA _sourceCategory=account_management "Budget" "is reset"

Search for only capacity usage logs:

_index=sumologic_audit _sourceName=VOLUME_QUOTA _sourceCategory=account_management "Budget" "last reset"