In most scenarios, the message time and receipt time of a log message in Sumo Logic should be the same. However, network latency, random (not continuous) spikes in data volume, and service disruptions can cause delays, leading to a discrepancy between message time and receipt time. Large discrepancies can lead to incorrect events being displayed, and may even cause search performance issues. In some occasions, it can also prevent Dashboards from populating with data.
What is receipt time?
Receipt time is the timestamp when a log message hits the Sumo Logic receivers. See Use Receipt Time for details.
What is message time?
Message time represents the time of your log events, as parsed by our service. During data source setup, most customers choose to have our service automatically detect timestamps in their logs and parse them automatically by using the option below:
Time Zone Configuration
Large time discrepancies are typically related to the time zone settings selected during Collector setup. When incorrectly configured for a data source, the setting below can lead to message time and receipt time discrepancies, so pay careful attention to your selection here.
Selecting the first option assumes that a time zone is part of the message timestamp, and that it is in a supported, Java 6 Simple Date format, as shown below:
|Letter||Date or Time Component||Example|
|z||Time zone (General time zone)||Pacific Standard; PST; GMT-08:00|
|Z||Time zone (RFC 822 time zone)||-0800|
If no time zone is found in the message, or if the time zone is in an unsupported format, Sumo Logic uses the selected fallback option and applies the appropriate offset to the message time.
Possible Time Zone-Related Configuration Issues
Example 1 - Unsupported Time Zone format in File
Source Time Zone selection: UTC
Sample Message Timestamp: Sep 28 19:00:00 US/Pacific
In this scenario, “US/Pacific” is not in a valid time zone format, so instead of using GMT-08:00, Sumo Logic applies the UTC time zone. And because of this mismatch, when users run a search for the “last 15 minutes”, the events being retrieved are inaccurate.
Example 2 - Improperly configured Time Zone configuration
Source Time Zone selection: None Selected (”Select a Time Zone” appears in the UI)
Sample Message Timestamp: Sep 28 19:00:00
In this scenario, there is no time zone in the sample message at all. However, the server where the logs were sourced from is located in Los Angeles, so the expectation is that the time zone be GMT-08:00. Because no time zone option was actually selected for the Source, Sumo Logic applies the default UTC time zone to these messages. And similar to the first example, when users run a search for the “last 15 minutes”, the events being retrieved are inaccurate.
Finding Timestamp Deltas
The query below can be executed in your account to find the number of messages for each of your Sources where the receipt time and message time is more than 30 minutes apart. This should at least give you a good starting place for you to run additional analysis.
| _receiptTime as r
| _messageTime as t
| t - r as d
| toInt(d) as i
| abs(i) as i
| 30 as minutes
| minutes * 60 * 1000 as milliseconds
| where i > milliseconds
| formatDate(fromMillis(r),"MM/dd hh:mm") as r
| formatDate(fromMillis(t),"MM/dd hh:mm") as t
| count by _collector, _source
| order by _count
After identifying a list of (problematic) Sources in the previous search, you can then use the query below to view the time delay average, the max delay, and the min delay for each Source. Substitute <problem_child> with the Source(s) that you uncovered in the initial query above.
| _format as format
| formatDate(fromMillis(_receipttime),"MM/dd hh:mm") as r
| formatDate(fromMillis(_messagetime),"MM/dd hh:mm") as t
| abs(_receipttime - _messagetime) as delt| delt/1000/60 as delt
| min(delt), max(delt), avg(delt), stddev(delt), count(*) by _collector, _sourcename
If the average, max, and min delay values are very close in range, then the time difference is most likely the symptom of an incorrect time zone setting. You’ll want to go back and ensure that your Source configurations line up correctly with the log messages. Switch to the raw messages tab and the “format” field shows how the timestamp is parsed from the file.