Skip to main content
Sumo Logic

Lab 1 - Ad hoc investigation using real time central log management

This lab give you practical experience to use metadata and keywords to narrow your search scope and improve performance for a security context.

Sumologic is a centralized log management platform. Centralized log management is often undervalued as it can be used as a starting point for Security Information and Event Management, SIEM. With our Cloud Flex Credits and continuous intelligence platform, you can observe logs from across your system to do threat hunting and investigation. Click to see a security customer attestation video.

In this lab, you will learn the use of metadata and keywords to specify your query scope to improve performance and secure your system. Narrowing your searches is a best practice, since your monthly data ingested may be reduced. Also, a query with keywords often runs faster, as it may have less data to examine. You will also perform an adhoc investigation using LogReduce and Live Tail our real time data log management tools. 

Screen Shot 2020-08-05 at 4.43.38 PM.png

Lab Activity

Filtering specific incoming data

  1. First you will just get familiar with our incoming CloudTrail data using a simple scoping line that isolates specifically CloudTrail incoming data. Search for all messages for the last 15 min with  _sourceCategory=Labs/AWS/CloudTrail.


  1. With CloudTrail, you can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure. Now, to further improve the query performance and keep your ingesting data optimal, let's narrow your search down. Let's search for only incoming messages that contain the keyword root by adding the keyword root at the end.

_sourceCategory=Labs/AWS/CloudTrail and root

  1. To locate the keyword root in the log message results, you will need to expand userIdentity by clicking on the arrow expander to the right of userIdentityScreen Shot 2020-04-08 at 2.36.04 PM.png. You will now see root highlighted at this nested level. 

Screen Shot 2020-04-08 at 2.39.52 PM.png

  1. Keywords are case insensitive. To show this replace root with Root, or ROOT, and re-running the query.

_sourceCategory=Labs/AWS/CloudTrail and ROOT

  1. Using wildcards can be helpful. Let's search for messages across all your AWS data that contain the word "root".  To locate the keyword root in the log message results, you will need to expand userIdentity by clicking on the drop down. You will see root highlighted at this nested level. 

_sourceCategory=Labs/AWS/* and root

  1. There is another way to look at the nested JSON hierarchy in the log messages. In the middle right, click Expand JSON. You will see your nested levels all expanded for JSON display. You can toggle back using click Collapse JSON. Also try clicking View as Raw, if you don't care to see it displayed in JSON format.

 Screen Shot 2020-04-08 at 2.29.52 PM.png

  1. If you would like to look at high priority items across all your incoming data logs try this following query. Enclose the text string in double quotes "priority=1". You may leave the time to Last 15 min. You may optionally go into the Hidden Fields in the Field Browser and enable _sourceCategory metadata which will show you the value giving you the source of the data that had a priority=1 log message.

    _sourceCategory=* "priority=1"

Reducing incoming data logs by grouping

  1. When looking at all your incoming data logs, over longer amounts of time like Last 24 Hours, you may obtain vasts amounts of results. When this happens you may wish to apply a logReduce, as in the simple query below. First just run the query for the Last 24 Hours. Then rerun the query removing the comment // the second time. Notice in the second time running of the query, the signature tab the results are grouped into "like formats" and a count is also returned for each "like format". Compare the number of results between the first and second runs, also compare the number of result pages. You will notice that LogReduce significantly consolidated the same results down to just 1 or 2 pages. You can group "like formats from incoming logs for easier investigations.

_sourceCategory=labs/snort error

// | logreduce

Results with commented out logReduce. Notice results shown over thousands of pages.

Screen Shot 2020-09-08 at 2.50.30 PM.png

Results with logreduce. Notice the new Signature tab with "like format" counted results summarized on 1 or 2 pages.

Screen Shot 2020-09-08 at 2.59.23 PM.png

Observing data as ingested

  1. For security reasons you may want to quickly monitor incoming data logs as they are being ingested. Similar to Linux tail command, you can tail your incoming logs. Using the query below, run a Live Tail session. Click Live Tail. There currently is a limit of 10 concurrent Live Tail sessions per organization. For more help on troubleshooting Live Tail look here.

_sourceCategory=Labs/AWS/* and *identity

  1. Just as we expanded in the messages to look at userIdentity, perhaps you want to see all userIdentity's coming in with their associated userName's. We offer highlighting in Livetail. Click on the A button in the upper right and type userIdentity and press Enter, then type username and press Enter. Just like Keywords, metadata has no case sensitivity.

Screen Shot 2020-04-08 at 1.43.31 PM.png

  1. Click any where in the Live Tail window to observe the highlights clearly and the entry box will close. Maybe you want to pause the tail before the userIdentity disappears off the screen?  To do this, at the upper right, click Pause Screen Shot 2020-04-08 at 2.10.41 PM.png and then to resume click Jump to Bottom Screen Shot 2020-04-08 at 2.12.56 PM.png or Screen Shot 2020-04-08 at 2.13.27 PM.png.

  2. To disable the highlighted text, click the A button and click on the x.

  1. Often we may want to detect if a specific user is doing anything malicious. You can track a users ip address found in this query's CloudTrail logs to detect if they've been reported in GuardDuty. Amazon GuardDuty is a threat detection service that continuously monitors for malicious activity and unauthorized behavior to protect your AWS accounts and workloads. You can run up to 2 Live Tail session in a browser. To compare these results with another source of data using Live Tail, got to the upper top right of the Live Tail and click on the 3 vertical dots. Then select Split Screen. This allows you to run two Live Tails simultaneously. 

Screen Shot 2020-06-29 at 6.02.53 PM.png

  1. Let's say we observed IP address in our CloudTrail live tail on the left.  We want to go check our GuardDuty data to see if that particular IP address has been noted in the incoming logs.  In the right Live Tail at the magnifying glass type _sourceCategory=Labs/AWS/GuardDuty_V8 Notice that it appears in our GuardDuty logs and now we may want to further investigate.


Screen Shot 2020-04-08 at 2.54.48 PM.png

  1. To close the Live Tail you can either click on the 3 vertical dots and select Stop Live Tail, or just close out the window tab by clicking on the X  Screen Shot 2020-04-08 at 2.16.33 PM.png

Quiz (True or False?)

  1. _sourceCategory points to specific incoming data.

  2. Keywords are used to widen the search on incoming data.

  3. Keywords and metadata can use wildcards.

  4. I can only run one LIve Tail window in a browser.


Congratulations! You’ve completed these tasks:

  1. Signed into Sumo Logic.
  2. Pointed directly to specific incoming data, using source category
  3. Filtered incoming data logs to the ones we wanted, using keywords.
  4. Expanded JSON formatted files.
  5. Applied a split livetail, using some of it's features.