Skip to main content
Sumo Logic

Create a Field Extraction Rule

Field Extraction Rules tell Sumo Logic which fields to parse out automatically.

A Field Extraction Rule uses these components:

  • Rule Name. Describes the rule.
  • Scope. Specifies the subset of data to use when fields are parsed. For example, you might include the sourceCategory associated with the data you'd like to parse from, or perhaps the sourceHost. Think of the Scope as the first portion of an ad hoc search, before the first pipe ( | ). You'll use the Scope to run a search against the rule.
  • Parse Expression. Defines the fields you'd like to parse. Choose one or more fields to parse. Because fields are associated with the Rule Name, you can parse one particular field into as many rules as you'd like. For example, to parse a single field, the definition could look like this: 

    parse "message count = *," as msg_count.

    To parse multiple fields, the definition could look more like this

    parse "[hostId=*] [module=*] [localUserName=*] [logger=*] [thread=*]" as hostId, module, localUserName, logger, thread
     
  • Templates. Parse Templates are provided for common applications such as Apache, AWS, and Microsoft IIS logs. Instead of creating a parse expression, you can select a template from the list, preview it, and then click to apply it. The template will overwrite any existing parse expression.

Limitations

  • There is a limit of 50 Field Extraction Rules and 200 fields. This includes the default fields defined by Sumo Logic (about 16). The 200-field limit is per account, and deleting rules does not create more space.
  • Field Extraction Rules are limited to a maximum of 16k (16,384) characters.
  • Because fields are parsed at the time of data ingestion, Field Extraction Rules only apply to data moving forward. If you want to parse data ingested before the creation of your FER, you can either parse your data in your query, or create Scheduled Views to extract fields for your historical data.  

Best practices for designing Rules

Include the most accurate keywords to identify the subset of data from which you want to extract data. Lock down the scope as tightly as possible to make sure it's extracting just the data you want, nothing more. Using a broader scope means that Sumo Logic will inspect more data for the fields you'd like to parse, which may mean that fields are extracted when you don't actually need them.

Create multiple, specific rules. Instead of constructing complicated rules, create multiple rules with basic scope, then search on more than one (rules are additive). The OR and AND commands are supported, just as in any search. For example, you could use one rule to parse Apache log response codes, and then use another rule to parse response time. When used together, you can get all of the information you may need.

Don't extract fields you don't need. Extract the minimum number of fields that should all be present in logs. Every field you include in the scope shows up in every search, so including extra fields means you'll see more results than you may need. It's better to create more rules that extract the fields that are most commonly used. First, look at common data sources and see what's most frequently extracted. Then, think about what you most frequently parse from those sources, then create rules to automatically extract those fields.

Test the scope before creating the rule. Make sure that you can extract fields from all messages you need to be returned in search results. Test them by running a potential rule as a search.

Make sure all fields appear in the Scope you define. When Field Extraction is applied to data, all fields must be present to have any fields indexed; even if one field isn't found in a message, that message is dropped from the results. In other words, it's all or nothing. For multiple sets of fields that are somewhat independent, make two rules.

Re-use field names in multiple FERs if scope is distinct and separate and not matching same messages. To save space and allow for more FERs within your 200 field limit, you can re-use the field names as long as they are used in non-overlapping FERs. Don't reuse any field names in multiple FERs that target the same messages as this can cause your fields to be extracted incorrectly.

Avoid targeting the same message with multiple FERs. When more than one FER targets the same sourceCategory and messages with the same field name, one of the rules will NOT apply. The rule applied to the specific field name is randomly selected (the one that applies last usually wins assuming the parse statements work successfully).  Don't re-use any field names in multiple FERs that target the same messages.

Supported parsing and search operators

The following operators can be used as part of the Parse Expression in a Field Extraction rule:

  • parse regex
  • parse anchor
  • parse nodrop
  • csv
  • double
  • fields
  • json
  • keyvalue
  • num

 

Unsupported parsing options

The following parsing options are not supported in a Field Extraction Rule:

  • parse multi
  • parse regex multi
  • csv auto
  • json auto
  • keyvalue auto

Creating a new Field Extraction Rule

Field Extraction Rules are created and managed using the Field page in the Sumo Logic Web Application. Admins can create their own rules, and delete rules created by other admins.

To create a new Field Extraction Rule:

  1. Go to Manage > Field Extractions.
  2. Click Add.
  3. Enter text for Rule Name, then type the scope of the rule as well as the fields you'd like to parse.
  • Rule Name. Type a name that makes it easy to identify the rule.
  • Scope. Type the subset logs you'd like to parse. This could be a sourceCategory, sourceHost, or any other metadata that describes the data you want to extract from.
  • Parse Expression. Type the fields you'd like to parse, using the following syntax: parse "[field1=*] [field2=*] [field3=*] as field1, field2, field3
    Or select a Parse Template.
  • Templates. See Create a Field Extraction Rule Using a Template below for more information.

  1. Click Add.

Each template is designed for a specific log type, such as Microsoft IIS, Palo Alto Networks, Nginx, and so on.

To create a new Field Extraction Rule with a Template:

  1. Go to Manage > Field Extractions.
  2. Click Add.
  3. In the Create Field Extraction Rule dialog box, choose an option from the Templates menu to see the options.
  4. Choose a Template, then click Use Template.
  5. Enter the Rule Name and Scope.
  6. Click Add.