Skip to main content
Sumo Logic

Insight Generation Process

Learn how CSE correlates Signals by entity  to create Insights.

Understanding entities in CSE

The concept of an entity is central to the process CSE uses to correlate Signals and create Insights. So, what is an entity? In CSE, an entity is a unique IP address, hostname, username, or MAC address encountered in an incoming message.

Entities are identified during parsing process

CSE identifies entities during the parsing stage of the Record processing process. (Parsing is the process of creating a set of key-value pairs that represent the content of a message.) During message parsing, CSE detects each field that contains an IP address, hostname, username, or MAC address. For each entity type field encountered, CSE checks to see if the field value already exists in CSE. If it does not, CSE creates an entity for it in CSE.

For example, when CSE parses the following message it will identify two entities, a username (thedude) and an IP address (185.35.135.245). 

   "eventVersion":"1.05",
   "userIdentity":{ 
      "type":"IAMUser",
      "principalId":"TESTUFTUMGUBCINLVVPBH",
      "arn":"arn:aws:iam::357921232527:user/thedude",
      "accountId":"357921232527",
      "userName":"thedude"
   },
   "eventTime":"2019-09-10T00:00:35Z",
   "eventSource":"signin.amazonaws.com",
   "eventName":"ConsoleLogin",
   "awsRegion":"us-east-1",
   "sourceIPAddress":"185.35.135.245",
   "userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36",
   "requestParameters":null,
   "responseElements":{ 
      "ConsoleLogin":"Success"
   },
   "additionalEventData":{ 
      "LoginTo":"https://forums.aws.amazon.com/ann.js...sauthcode=true",
      "MobileVersion":"No",
      "MFAUsed":"Yes"
   },
   "eventID":"ea49159d-f294-47c3-8e5c-73f19436bc2e",
   "eventType":"AwsConsoleSignIn",
   "recipientAccountId":"357921232527"

Entities in messages are mapped to entity-type schema attributes

During the next step of the Record processing flow—log mapping—message fields are mapped to CSE schema attributes. During this process, each entity field from a message is mapped to one of the following CSE schema entity attributes:

  • device_hostname
  • device_ip
  • device_mac
  • user_username
  • srcDevice_hostname
  • srcDevice_ip
  • srcDevice_mac
  • dstDevice_hostname
  • dstDevice_ip
  • dstDevice_mac
  • fromUser_username
  • device_natIp
  • srcDevice_natIp
  • dstDevice_natIp

Which particular attribute an entity gets mapped to depends on the field mappings in the log mapper for the message source. Given the example message above, “thedude” might be mapped to user_username and "185.35.135.245" to srcDevice_ip

Viewing entities in CSE UI

You can view the entities that have been extracted from messages on the Entities page in the CSE UI.
  
entities-page.png

Note that the screenshot above shows an Activity Score for each entity. The following section explains what an Activity Score is and how it relates to the Insight creation process.

Understanding Entity Activity Scores

In a nutshell, an Activity Score is the cumulative severity of the Signals an entity has been a party to over a period of time, two weeks by default. 

More precisely, an entity’s Activity Score is the sum of the severities of the unique Signals associated with that entity during the previous two weeks. Typically, Signals generated by the same rule are not unique: a Signal takes its name from the rule that fired it, so unless a rule's name has a unique templated value in it, the Signals that the rule generates are not unique. 

Here are a couple practical examples:

  • If the RDP Brute Force Attempt rule fires 10 times, the Signals all have the same name, and are not unique. So, the severity of just one of the 10 Signals would be included in the entity’s Activity Score.
  • If the RDP Brute Force Attempt {{threat_name}} rule fires three times, where threat name is “bad”, “bad” and “worse”, two of the three Signals are unique:
    • RDP Brute Force Attempt bad
    • RDP Brute Force Attempt bad
    • RDP Brute Force Attempt worse

The severities of the RDP Brute Force Attempt bad and the 
RDP Brute Force Attempt worse Signals would be included in the entity’s Activity Score.

Note also that an entity’s Activity Score is reset when an Insight is generated for the entity.

Example of an Entity that has reached Activity Score threshold

In the screenshot below, the Details pane on the left shows that the Insight was created for the entity “192.168.1.1”, an IP address. The right side of the page shows the three Signals that contributed to the Insight. You can see each of the Signals relate to the IP address for which the Insight was created; in the Record underlying each of the Signals, is mapped to the srcDevice_ip schema attribute . 

The severity of each Signal is also shown. CSE generated an Insight for entity “192.168.1.1” because the cumulative severity of Signals fired for that entity within a two week period exceeds the threshold Activity Score.

insight.png

How rules fit into the picture

There are two aspects of a CSE rule that are worth pointing out.

The first is this: Every CSE rule is configured with one or more On Entity schema attributes. In the rules that fired the Signals shown in screenshot above, the configured On Entity attribute is srcDevice_ip. A rule author selects the desired attribute or attributes using the On Entity selector in the Then Create a Signal section in the rules UI. Ultimately, the On Entity attribute or attributes a rule author configures will determine what entity’s or entities’ Activity Score will be re-calculated when the rule fires Signals.

on-entity.png

To make this clearer, consider a rule whose On Entity attribute is srcDevice_ip. When the rule fires a Signal, CSE updates the Activity Score for the IP address mapped to the srcDevice_ip in the Signal. 

The other relevant fact is that severity of a Signal comes from the rule that fired it. When a rule author writes a rule, they assign it a severity. Every Signal that the rule files will have that severity.

About the Activity Score threshold

CSE generates an Insight for an entity based on the entity’s Activity Score. By default, CSE automatically sets and adjusts the Activity Score threshold value for Insight generation. 

If you prefer a static threshold or a different lookback period, contact Sumo Logic support for assistance. Note you can also take control Insight generation by creating custom Insights.

It’s good to keep in mind the relationship between rule severity and the volume of Insights that are generated. Specifically, creating many custom rules with high severities will result in more Insights. Rules with lower severities have a smaller effect on entity Activity Scores and hence the number of Insights you’ll have to deal with.