When collecting log data, the timestamp attached to messages is vital, both for the integrity of the data in your account, and for accurate query results. Because of the importance of timestamps, Sumo Logic indexes the timestamp of each message, making sure that data relevant to a query’s time range is returned properly in search results, which allows you to reconstruct a correct event timeline.
To keep pace with real time analytics, Sumo Logic creates indices in nearly real-time. However, when collecting data with incorrect timestamps (or if there is latency in the collection of data), Sumo Logic can over-generate indices in an attempt to properly handle the messages. These excess indices greatly degrade search performance, and this issue is referred to as index fragmentation. This can lead to the error message, "Your search contains messages that have been incorrectly parsed and cannot be displayed."
Sumo Logic has addressed index fragmentation by separating log messages into two categories:
- Messages with timestamps within plus or minus 24 hours from the present time.
- Messages with timestamps that fall outside this range (older than 24 hours from the present time, or in some cases, greater than 24 hours in the future).
During a typical search, only messages with timestamps within plus or minus 24 hours from the present time are queried. If a message’s timestamp and receipt time don’t match up, those messages may not be included in search results.
To search all data with any and all timestamps, select the Use Receipt Time check box. This option displays search results in reverse order of their receipt time, giving you the ability to view the difference in timestamp and receipt time to pinpoint Sources that may be generating incorrect timestamps.
Run a search by Receipt Time
To run a search by Receipt Time, select the Use Receipt Time check box:
- Enter your query in the search text box.
- Choose the Time Range for the query.
- Select Use Receipt Time.
- Review the search results for wide discrepancies between message timestamp and receipt time to pinpoint Sources with incorrect timestamps:
Resolving timestamp/receipt time issues
If you notice an issue between timestamps and receipt time values, you can double-check the Source’s settings. You can manually specify the parse format for the Source, and test the format to make sure it’s valid. Alternately, if you’re noticing that timestamps are parsing properly, check the timestamp conventions of your logs. Learn more in Timestamps, Time Zones, Time Ranges, and Date Formats.