Skip to main content
Sumo Logic

Data Streams

You assign data to a Data Tier by defining a Data Stream.

Data Streams allow you to group your data using filter expressions, apply processing (ETL) to the data, and make it ready for downstream consumption. You can use  Data Streams to specify a data set and assign it to a Data Tier and a partition using a unified UI. Data Streams simplify the process of managing ETL workflows and allocating data to Data Tiers. For more information, see Data Tiers.

Create a Data Stream 

Step 1: Configure Basic Details

  1. Go to Manage Data > Settings > Data Streams.
  2. The Data Streams page displays a list of existing Data Streams.
  3. Click + Add Data Stream.
  4. The Create New Data Stream page appears. create-data-stream.png
  5. Name. Enter a name for the new Data Stream.
  6. Select Data Tier. Click the radio button for the Data Tier where you want the data to reside. For more information, see Data Tiers.
    • Continuous. You can search and analyze data in the Continuous tier in real time. You can also dashboard the data and alert on it. 
    • Frequent. You can only run interactive queries on data in the Frequent tier.  (Pricing is up front, based on ingested data volume.) 
    • Infrequent. You can only run interactive queries on data in the Infrequent tier. (Cost of data ingestion is nominal, cost of queries is based on the how much data is scanned.) 

Step 2: Configure Data Stream scope

In this step, you specify what data you want to include in the Data Stream using built-in Sumo Logic metadata fields and custom metadata fields. There are two modes for specifying the scope:

  • Basic Mode. This mode provides a picker you can use to select a built-in Sumo Logic metadata field and a value. In Basic Mode, your scope can only include a single metadata field. If you want to include multiple metadata fields, use Advanced mode. By default, the Filter section of Create New Data Stream page is in Basic Mode.
  • Advanced Mode. In this mode, you manually enter a log query. 
Configure scope in Basic Mode
  1. Click the Metadata field and choose a  field. The list includes Sumo Logic built-in metadata fields.
  2. Click the Value field to choose a value for the metadata field. 
  3. (Optional) Enter keywords to further reduce the scope of your data.
Configure scope in Advanced Mode
  1. Hover over the right side of the Scope area and click the </> icon to open Advanced Mode.
  2. Enter a query using one or more built-in metadata fields, custom metadata fields, and keywords and separated by logical operators (AND, OR). The not operator (!) and wildcards (*) are  supported.

    For example:

    _sourceCategory=x OR _sourceCategory=y AND custom-field-1=z and error

    _sourceCategory=x AND custom-field-1=z !error

    _sourceCategory=y and "hello"

    _sourceCategory=* y * and *error

Step 3: Configure load

In this step, you create a new partition for the data stream, or select an existing partition. Sumo Logic recommends you create a new partition for a new data stream.

Create a new partition

In the Load section of the page:


  1. Click Add New Partition.
  2. The Create New Partition pane appears on the right side of the page.
  3. Name. Enter a name for the partition.
  4. Retention Period. Enter the number of days you want to retain the data in the partition, or click the Apply the retention period of the Default Index option to set the period to 31 days. 
  5. Click Save to save the new partition.
  6. Click Save to save the new data stream.
Use an existing partition
  1. Click Load Into Existing <Tier-Type> Partition.
  2. The Select Partition pane appears. The right side of the pane displays details about the selected partition. 
  3. Select the desired partition and click Save.

Filtering the Data Stream list

In addition to searching for a Data Stream by name, you can filter the contents of the list.

  1. Go to Manage Data > Settings > Data Streams.
  2. The Data Streams page displays a list of existing Data Streams that are
  3. Enter all or part of a Data Stream name. You can filter the list in the following ways.
    • Click the down arrow next to Active Data Streams to display only those Data Streams that are active, or to display all Data Streams.
    • Click the funnel icon in the Data Tier column header to choose which tiers’ Data Streams you want to list. Check the desired tier or tiers and click Apply Filters.

Edit a Data Stream 

  1. Go to Manage Data > Settings > Data Streams.
  2. The Data Streams page displays a list of existing Data Streams that are active. Search for the Data Stream you want to edit. See Filtering the Data Stream list for information about filtering the list.
  3. Hover over the row for the Data Stream, and choose Edit from the three-dot more options menu. For information about the options on the edit page, see Create a Data Stream.

Delete or disable a Data Stream

The more options menu for a Data Stream has Disable and Delete options. Deleting a Data Stream is permanent. You can re-enable a disabled Data Stream using the Enable option, which appears on the more options menu for a disabled Data Stream.

After you disable or delete a Data Stream, any of the data within the scope (as defined in the Filter section of the stream configuration) of the deleted or disabled stream that isn’t within the scope of another stream, will be routed via the default Continuous stream and will be accessible through default Continuous Index.‎

You can access ‎log data that was routed through the stream before you disabled or deleted it through the partition that was mapped to the stream.