Partitions provide three primary functions:
About creating partitions
To create a partition in an index, you will create a routing expression, which is a kind of query. A partition routing expression can take anything in a regular search query up to the first pipe—in other words, the search constraints. Partitions must be named alphanumerically, with no special characters, with the exception of underscores ( _ ). The query can include wildcards, but it cannot include any parsing or search operators.
Create partitions for use cases that are not too general. The idea is to use partitions in an index to restrict your search for security and in order to improve search performance. If you create a partition for a very general use case, it would still work, you just wouldn’t benefit as much from increased performance.
For step-by-step instructions to create a partition, see Analytics Tiers. Partitions ingest your messages in real time, and differ from scheduled views in that partitions don't backfill with aggregate data. Partitions begin building a non-aggregate index from the time the partition is created and only index data moving forward (from the time of creation).
Best practices for optimum performance
When designing partitions, keep the following in mind:
- Avoid using queries that are subject to change. In order to benefit from using partitions, they should be used for long-term message organization.
- Make the query as specific as possible. Making the query specific reduces the amount of data in the partition, which increases search performance.
- Keep the query flexible. Use a flexible query, such as
sourceCategory=*Apache*, so that metadata can be adjusted without breaking the query.
- Group data together that is most often used together. For example, create partitions for categories such as web data, security data, or errors.
- Group data together that is used by teams. Partitions are an excellent way to organize messages by role and teams within your organization.
- Avoid including too much data in your Partition. Send between 2% and 20% of your data to a partition. Including 90% of the data in your index in a partition won’t improve search performance.
- Don’t create overlapping partitions. With multiple partitions, messages could be duplicated if you create routing expressions that overlap. For example, if you have the following partitions, messages for
_sourceCategory=prod/Apachewould be duplicated as they would be stored in both partitions.
Enhance search and retention
Analytics Tiers provide for the enhancement of search and retention capabilities. Analytics partitions enable economic flexibility by aligning your analytics to the value of your data. By using the Continuous, Frequent, or Infrequent tiers, you can appropriately segment your data by use case and analytics needs, thus enabling you to optimize your analytics investments.
- See the Analytics Tiers for information on creating Analytics Tiers, as well as the uses and capabilities of each of the Continuous, Frequent, and Infrequent tier types.
- See Run a Search Against a Partition and Optimize Your Search with Partitions for information on the various ways to run a search against a partition.
- See Manage Indexes with Variable Retention for information on data retention periods and how to modify them.