Skip to main content
Sumo Logic

Ingest Budgets

availability_enterprise_only_pub.png

Ingest budgets control the capacity of daily log ingestion volume sent to Sumo Logic from Collectors. Collectors can be assigned to an ingest budget that defines a daily log capacity limit. The capacity is tracked based on the combined volume from Collectors, from all their Sources. When an ingest budget's capacity is reached you can have the Collectors assigned to it stop ingesting data to control costs.

Ingest budgets automatically reset their capacity utilization tracking every 24 hours based on the time and time zone you specify. For example, you can schedule an ingest budget to refresh every day at 02:00 in the America/Los_Angeles time zone. You can manually reset an ingest budget at any time.

An ingest budget's capacity usage is logged in the Audit Index once it reaches 85% and continues to be logged until the budget is reset. You have the option to have its assigned Collectors stop collecting data when the capacity has reached 100%. When an ingest budget triggers Collectors assigned to it to stop collecting data, the data is not automatically recovered or ingested later once the capacity tracking is reset. To track and schedule alerts on ingest budget capacity-usage and resets see audit ingest budgets.

Rules

  • A Collector can be assigned to one ingest budget.
  • There is a limit of 100 ingest budgets.
  • Bytes are calculated in base 10 (SI-style, 1,000 based).
  • Ingest budgets requires the Manage Ingest Budgets role capability. Enabling this will automatically enable the Manage Collectors capability. The Manage Collectors capability on its own permits the re-assignment of budgets to different Collectors, but not creating or deleting them.
  • It takes around 30 seconds for Collectors to stop collecting when capacity is reached. We recommend setting a soft limit that is lower than your needed hard limit.
  • Data is not automatically recovered or ingested later once the capacity tracking is reset.
  • Updating an ingest budget's Field Value will not propagate to assigned Collectors, instead the Collectors are unassigned as they do not match its configured Field Value.
Source type behavior

A few Sources on Hosted Collectors behave differently when the Collector is instructed to stop collecting data.

  • HTTP Sources will drop data requests yet still return a 200 response.
  • AWS S3 based Sources will skip objects. 
  • Cloud Syslog Sources will keep the connection open yet drop incoming syslog messages.

Manage ingest budgets

Use the Ingest Budgets page to manage your ingest budgets. To access the page go to Manage Data > Collection > Ingest Budgets.

Ingest Budgets table complete.png

The page displays the following information:

  • Name. Name of the ingest budget.
  • Field Value. Custom value used to assign Collectors to the ingest budget.
  • Capacity. Maximum amount of data permitted. Bytes are calculated in base 10 (SI-style, 1,000 based).
  • Usage. Percentage of data used. To refresh this information, close and reopen the main Collection tab.
  • Collectors. Number of Collectors assigned to the ingest budget.
  • Reset Time. Time and time zone to reset the data usage tracking in HH:MM timestamp format. This is fixed at a 24-hour time interval, so the reset time is triggered every 24 hours. Use the IANA time zone database format.
  • Allocated Capacity (bottom of table). The total allocated capacity from all ingest budgets out of your account's available daily log ingest capacity is provided. If you assign all your Collectors to ingest budgets you can easily track how much data you are allowing Collectors to ingest compared to your account's available daily log ingest quota. 

At the top of the page, you can:

For the ingest budgets listed, select a row to view its details. A details pane appears to the right of the table.

ingest budget details panel.png

In the details pane you can do the following to the selected ingest budget:

Create ingest budget

  1. In Sumo Logic select Manage Data > Collection > Ingest Budgets.
  2. Click the + Add Budget button on the top right of the table. A panel named Create Ingest Budget appears to the right of the Ingest Budgets table.

Provide the following information, all fields are required except Description.

  1. Display Name. Enter the name you'd like to assign to the new ingest budget.
  2. Field Value. Provide a desired custom value for the ingest budget. This value is used to assign a Collector to the ingest budget.
  3. Description is optional.
  4. Allocated Capacity. Set the maximum daily ingestion volume you want for the ingest budget.
    • Amount. Enter a value up to 999.
    • Unit. Select a unit of memory. Bytes are calculated in base 10 (SI-style, 1,000 based).
  5. Reset every day at. Ingest budgets automatically reset their capacity utilization tracking every 24 hours based on the time and time zone you specify.
    • Time. Set the time of day to reset the capacity tracking.
    • Time zone. Set the time zone of the reset time.
    • Action when capacity reached. Select the action to take when the ingest budget's capacity is reached. All actions are audited.
      • Stop Collecting - It takes around 30 seconds for Collectors to stop collecting when capacity is reached. We recommend setting a soft limit that is lower than your needed hard limit.
      • Keep Collecting
  6. When you're finished configuring the ingest budget click Add.

Now that you have created an ingest budget you can assign a Collector to it. See assign Collector to ingest budget for details.

Reset ingest budget

You can manually reset a budget at any time to set its capacity utilization tracking to zero. If your ingest budget has triggered its assigned Collectors to stop collecting data you can reset it to begin collecting data again. This won't affect the next scheduled reset time and can be done as many times as needed.

  1. In Sumo Logic select Manage Data > Collection > Ingest Budgets.
  2. In the table find the ingest budget you want to reset and click the row to open its details pane.
  3. Click the Reset button.

Edit ingest budget

Editing an ingest budget's Field Value will not propagate to its assigned Collectors, instead the Collectors are unassigned as they do not match its configured Field Value.

  1. In Sumo Logic select Manage Data > Collection > Ingest Budgets.
  2. In the table find the ingest budget you want to edit and click the edit icon pencil edit icon.png on the right of the row or click the row and then click the edit icon in the details panel.
  3. Make your changes and click Update.

Delete ingest budget

When an ingest budget is deleted its assigned Collectors retain their Field Value configuration.

  1. In Sumo Logic select Manage Data > Collection > Ingest Budgets.
  2. In the table find the ingest budget you want to delete and click the delete icon delete trash icon.png on the right of the row or click the row and then click the delete icon in the details panel.
  3. You will get a confirmation prompt, ensure that you are deleting the desired ingest budget and then click Delete.

Audit ingest budgets

The Audit Index logs events when an ingest budget has reached and when exceeding 85 percent of its set capacity limit.

There are two different log formats. 

  1. Approaching or exceeding capacity
  2. Resets

Approaching or exceeding capacity example, where:

  • budget_name is the name of the ingest budget.
  • budget_value is the ingest budget's field value.
  • Usage status is either Approaching (≥ 85%) or Exceeded (≥ 100%) its set capacity limit.
Budget budget_name with field value budget_value consumed 6330.00% of capacity since last reset at 2018-09-17T13:38:53.663 -0700.
Capacity: 200 bytes
Usage: 12660 bytes
Usage status: Exceeded
Action: drop_data
Next reset: 2018-09-18T13:35:00.000 -0700

Reset example, where budget_name is the name of the ingest budget and budget_value is the ingest budget's field value:

Budget budget_name with field value budget_value consumed 0.00% of capacity and is reset at 2018-09-18T00:03:34.574 -0700.
Capacity: 1000 bytes
Usage: 0 bytes
Next reset: 2018-09-19T00:00:00.000 -0700
Audit Index queries

You can schedule the following searches to get alerts when needed, see scheduled searches for details.

Search for when approaching usage capacity:

_index=sumologic_audit _sourceName=VOLUME_QUOTA _sourceCategory=account_management "Budget" "last reset" "Approaching"

Search for all available audit logs:

_index=sumologic_audit _sourceName=VOLUME_QUOTA _sourceCategory=account_management "Budget" "reset"

Search for only reset logs:

_index=sumologic_audit _sourceName=VOLUME_QUOTA _sourceCategory=account_management "Budget" "is reset"

Search for only capacity usage logs:

_index=sumologic_audit _sourceName=VOLUME_QUOTA _sourceCategory=account_management "Budget" "last reset"