A partition stores your data in an index separate from the rest of your account's data so you can optimize searches, manage variable retention, and specify certain data to forward to S3.
Partitions route your data to an index becoming a separate subset of data in your account. Creating smaller and separate subsets of data is central to search optimization. When you run a search against an index, results are returned more quickly and efficiently because the search runs against a smaller data set.
After routing messages to a partition, you can reference it in your search by using the field
_index with the partition's name. See Optimizing Search with Partitions for details.
Partitions ingest your messages in real time. They differ from scheduled views in that partitions don’t backfill with aggregate data. They begin building a non-aggregate index from the time the partition is created and index only the data moving forward. Scheduled views backfill with aggregate data, meaning that all data that extends back to the start date of the view query is added to the view.
To define partitions use simple search expressions, also called routing expressions. The routing expression is applied to all incoming messages as they enter the system. If the filter matches the message, it’s added to the partition.
- To create a partition you must be an Admin or you must have the Manage Partitions role capability.
- There is a limit of 50 partitions per account.
- Once they are created, partitions cannot be edited or deleted, and partition names cannot be reused. This is due to the fact that a partition may include log messages that are not stored anywhere else, and if the partition is deleted, the log messages will be lost. But if it is no longer needed, a partition may be decommissioned.