Skip to main content
Sumo Logic

Data Tiers

Data Tiers provide the ability to mark your datasets according to how often you access the data: Continuous, Frequent, and Infrequent.

This page describes Sumo Logic's Data Tiers feature. For related information, see Data Tiers FAQs.

Modern enterprises collect and analyze vast amounts of data for a variety of use cases. Sumo Logic customers use ingested data to monitor operations, troubleshoot problems, to understand and better serve customers, to ensure security, and more. 

Some use cases require “high touch” data that you need to monitor and analyze continuously or frequently. For example, you need to constantly monitor production applications, troubleshoot issues, and understand your security posture. These use cases require continuous access to data like production web server and application logs; error and warning logs; and compliance and security assurance data.

Other use cases require much less frequent data analysis. Here, we’re talking about “low touch” data that can be very valuable when you want to mine your data for insights, provide periodic reports, or perform a root cause analysis. These use cases can require frequent or infrequent access to data like development, test, and pre-production logs; debug logs; CDN logs; and network logs.

Sumo Logic’s Data Tiers provide a comprehensive solution for all types of data that an organization has, low touch, high touch and everything in between, at an economical price. Data Tiers provide tier-based pricing based on your planned usage of the data you ingest. 

Types of Data Tiers 

Each Sumo Logic Data Tier supports a different use case and provides its own set of features and capabilities: 

  • The Continuous tier is for the data you use to monitor and troubleshoot production applications and to ensure the security of your applications. 
  • The Frequent tier is for data you need to frequently access to troubleshoot and investigate issues. For example, you might use the Frequent tier for development and test data that helps you investigate issues during development. Searching the Frequent tier is free: it's included in the data ingestion price.
  • The Infrequent tier is for data that is used to troubleshoot intermittent or hard-to-reproduce issues. For example, you might use the Infrequent Tier for debug logs, OS logs, thread dumps, and so on. The Infrequent Tier has a pay-per-search pricing model, and very low ingestion cost.   

Planning your use of Data Tiers 

All the data that is ingested into Sumo goes to the Continuous Tier, if no other tier has been specified. Only data that goes to a partition can go to the Frequent or Infrequent tiers. You configure the target tier for the data in a partition on the Partition page.

When planning your use of Data Tiers, it is important to remember the following guidelines:

  • The General Index cannot be changed, and it is always in the Continuous Tier.
  • The tier you assign your data to governs how you can search and analyze the data. The table below shows capabilities that are available in each tier. 

The amount of data you can ingest to the Frequent or Infrequent Tier is defined by your Sumo account plan. For more information, contact your Sumo Account Representative.

Feature support by tier

How you can search and use your ingested data varies by the Data Tier it resides in, as described in the following table. 

Feature support Continuous Tier Frequent Tier Infrequent Tier
Centralized, secure, multi-tenant cloud-native platform Check.png Check.png Check.png
Data replication across availability zones, data encryption 
 

Check.png

Check.png
Check.png
Interactive queries
(UI)
Check.png
Partitions can be specified, but are optional.
Check.png
Partitions must be specified.
Check.png
Partitions must be specified.
Support for Installed and Hosted Collectors Check.png Check.png Check.png
RBAC support Check.png Check.png Check.png
Support for search operators Check.png Check.png Check.png
Field Extraction Rules Check.png Check.png Check.png
Logs to Metrics Check.png Check.png Check.png
Data Forwarding Check.png Check.png Check.png
Live Tail Check.png Check.png Check.png
Dashboards Check.png x-sized.png x-sized.png
Scheduled Searches Check.png x-sized.png x-sized.png
Scheduled Views Check.png x-sized.png x-sized.png
API Queries Check.png x-sized.png Check.png

How to choose between Frequent and Infrequent 

Choosing between Frequent and Infrequent for a data set depends on how frequently you need to access the data. If you expect to search the data often, the Frequent Tier, with its predictable upfront pricing model, is appropriate. Data that you expect to access less often is an ideal candidate for the Infrequent Tier, which offers low ingest cost, and competitive on-demand search pricing.

For example, for a large development team with hundreds of developers, it is better to send development and test logs to the Frequent Tier if your developers are going to access it often during development. 

In contrast, debug or other verbose log sources that are only used to troubleshoot very specific issues that occur  infrequently, for example only a couple of times a week, are better off in the Infrequent tier to keep the cost of ownership low.  

Assigning data to a Data Tier

You assign data to a Data Tier at the partition level. When you create a partition, you define a routing expression and select the target tier for the data that matches the routing expression 

You define a routing expression using Sumo Logic's built-in metadata fields, such as _sourceCategory and custom metadata fields. You can also use keywords in a routing expression, but this practice is not recommended, as it is less maintainable. 

When defining a partition's routing expression, consider the following:

  • Sumo recommends that you do not use fields created by FERs in a partition routing expression. 
  • Unless you are using the Data Streams feature, currently in beta, we recommend you tag your Sources with a custom metadata field that indicates the tier to which you intend to route the data a source collects. For example, tier=infrequent. Then, you can use  that field in partition routing expressions, along with built-in Sumo metadata. Using a tier field to tag your sources is also useful because it makes it easy to reroute your data in the future by updating a source's tier field.

The following routing expression routes all data whose _sourceCategory begins with prod/* to the partition.

_sourceCategory=prod/* and tier=infrequent

For information about planning a partition, see Optimize Your Search with Partitions.

To assign data to a Data Tier

  1. In the Sumo left navigation bar, go to Manage Data > Logs, then select the Partitions tab.
  2. At the far right, click the plus sign (+). The Create Partition dialog appears.
    create-a-partition-2.png
  3. Partition Name. Enter a name for the partition.
  4. Routing Expression.
  5. Retention Period. Enter the number of days you wish to retain the data in the partition, or click Apply the retention period of the General Index.
  6. Data Tier. Click the radio button for the tier where you want the to partition to live.
  7. Data Forwarding. If you want to forward the data in the partition to a cloud environment, click Enable Data Forwarding and  specify the necessary information for the options that appear. For more information Data Forwarding.

Searching the Frequent and Infrequent Data Tiers 

When you search for data in the Frequent or Infrequent Tier, you must explicitly reference the partition. For example, to search for errors in a query you must reference the partition by name:

_index=my_freq_partition_name error

To search more than one partition:

(_index=my_freq_partition_name1 or _index=my_freq_partition_name2) error

Common error messages

This section describes the most common error messages for Data Tiers.

  • If you try to add a panel to a dashboard that uses data from the Frequent or Infrequent tiers, you receive the following error message, because you can only use data from the Continuous Tier in a dashboard:

    This query is not supported in Dashboards/Scheduled Searches because it is not in the Continuous Analytics tier. Please modify query and try again.

    create-panel.png
     
  • If you try to specify the scope of a Scheduled View or a Scheduled Search using a partition in the Frequent or Infrequent Data tiers, you receive this error message:

    This query is not supported in Dashboards/Scheduled Searches because it is not in the Continuous Analytics tier. Please modify query and try again.