Skip to main content

Manage Indexes with Variable Retention

In managing your company's log data, it's crucial to recognize the diverse nature of data types, each potentially warranting distinct retention periods. For instance, sensitive information may necessitate a shorter lifespan for security purposes, while critical data like customer logins may require extended retention to support daily operations. Additionally, certain data serves a dual purpose, requiring availability for historical analysis and metric gathering despite not being accessed daily.

With Variable Retention, you can create Index Partitions and Scheduled Views to store your data as needed, and set different retention periods for those Partitions and Scheduled Views. This way, you can keep the data you need for as long as you need it, while other less important data can be retained for a shorter period, reducing your overall cost of operation.

You set the retention period when you first create the Partition or Scheduled View, and you can also edit the retention period at any time in the future. 

Retention is measured from the time that data was received, rather than the message time. 

The minimum retention period is 1 day, and the maximum retention period is 5,000 days.

On the Manage Data > Logs page, there is a tab for Partitions and a tab for Scheduled Views.

On the Partitions tab, the first entry is the Default Partition, which contains all ingested data that is not assigned to a partition or to views populated by Scheduled Searches. The default retention period of the Default Partition is 30 days.

Here you can see the total volume ingested in each Partition, including:

  • Index Name
  • Routing Expression
  • Storage Consumed—The total volume of uncompressed data ingested across the duration of the retention period.
  • Retention Period
  • Data Forwarding 

From the three-dot kebab menu on the right, you can select the following actions:

Edit the Retention Period

You may edit the retention period of Partitions and Scheduled Views. The minimum retention period is 1 day, and the maximum retention period is 5,000 days.

  1. Go to Manage Data > Logs.

  2. On the Partitions or Scheduled Views tab, hover over the item you would like to edit the retention period for.

  3. Click the three-dot kebab icon on the right of the desired row, and from the menu select Edit Retention Period. (You may also edit the retention period from the expanded information dialog. Next to Retention Period, click Edit.)

  4. In the Edit Retention Period dialog, enter the new retention period in days. Alternatively, click the Apply the retention period of the General Index check box.

    1. If the new retention period is longer than the previous period, click Save.

    edit retention

    1. If the new retention period is shorter than the previous period, a portion of your data will be deleted in 7 days or right away. Click Save and choose either:

      • Simulate this data deletion for a few days and then reduce it permanently by selecting Reduce retention period after 7 days. You will be billed for this data until the deletion is permanent, but it gives you a sense of how the deletion will impact you before it is final.
      note

      When you select Reduce Retention Period after 7 days, the retention period on your Partitions page will not change immediately. There will be a an icon indicating that a retention decrease is imminent in a number of days. When the counter expires, the data will be deleted, the warning icon will disappear, and the new retention policy will be in effect.

      The retention period change applies to all data (including data ingested before the change).

      If you want to change your mind at any time during that period and revert to your previous retention period, click Click here to undo the reduction in retention period. The retention will stay at previous retention level, and the job to reduce the retention is canceled.

      • Or, if you are sure you want to proceed and end billing for this data immediately, enter DELETE in the text box, then click Reduce retention period immediately. The data will be deleted immediately as well.

    DeleteNOW.png

  5. The retention period is updated and displayed in the Retention Period column. If you used the reduce retention after 7 days option, you can undo your data retention reduction until the date specified by clicking Click here to undo the reduction in retention period. Otherwise, the data will be deleted after 7 days.   

Legal
Privacy Statement
Terms of Use

Copyright © 2024 by Sumo Logic, Inc.