Skip to main content
Sumo Logic

Fields

Fields allow you to reference log data based on meaningful associations. They act as metadata tags that are assigned to your logs so you can search with them. Each field contains a key-value pair, where the field name is the key. Fields may be referred to as Log Metadata Fields.

In addition to defining fields through Field Extraction Rules, you can define fields on data sent to Sumo by manually defining them on Sources and Collectors, as well as dynamically through HTTP headers and tags from Amazon EC2.

The order of precedence for field assignment from highest to lowest is:

  1. Field Extraction Rule (FER)
  2. Amazon EC2 resource tags
  3. Amazon EC2 instance information
  4. HTTP Header
  5. Source
  6. Collector

So, if you have a field defined at the Collector or Source level, and you create a FER against the same source of data with the same field name, the FER will win the field assignment.

Any fields you want assigned to log data need to exist in a Fields table schema. Each account has its own Fields schema that is available to manage in the Sumo web interface. When a field is defined in the Fields schema it will be assigned to the appropriate log data as configured. If a field is sent to Sumo that does not exist in the Fields schema it is ignored, known as dropped.

Field management is important to ensure search performance is maintained and you continue to have meaningful fields assigned to your data. You can manage fields defined through any of these methods at any time, to include deleting unneeded fields, see manage fields for details.

Limitations
  • Fields created as log metadata and from Field Extraction Rules share the same quota of 200 fields. The Fields page shows how many fields your account is using out of the total available at the bottom of the table as Fields Capacity.
    fields capacity.png
  • It can take up to 10 minutes for fields to start being assigned to your data.
  • A Collector can have up to 10 fields.
  • A Source can have up to 10 fields.
  • An HTTP request is limited to 20 fields.
  • A field name (key) is limited to a maximum length of 255 characters.
  • A value is limited to a maximum length of 200 characters.
  • Fields cannot be used with Live Tail or in the scope of Field Extraction Rules.

Collector and Source fields

Fields can be assigned to a Collector and Source using the Fields input table in the Sumo user interface when creating or editing a Collector or Source.

  1. Navigate to Manage Data > Collection > Collection.
  2. Create or find and select the Collector or Source you want to assign fields to.
  3. Click the +Add Field link in the Fields section.
    • Define the fields you want to associate, each field needs a name (key) and value. 
      • green check circle.png A green circle with a check mark is shown when the field exists and is enabled in the Fields table schema.
      • orange exclamation point.png An orange triangle with an exclamation point is shown when the field doesn't exist, or is disabled, in the Fields table schema. In this case, an option to automatically add or enable the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo that does not exist in the Fields schema or is disabled it is ignored, known as dropped.
  4. Click Save.

edit collector fields name.png

In the above example, we have created a new field called cluster and set the value to k8s.dev. With this configuration, any logs sent to this Collector will now have this key-value pair associated with it.

With this association, you can search for cluster=k8s.dev to return your logs.

collector field search results.png

Using Collector API

Use the fields parameter with the Collector API to define fields on a Collector or Source.

Parameter Type Required? Description Access
fields JSON Object No JSON map of key-value fields (metadata) to apply to the Collector or Source. Modifiable

The following JSON is an example configuration of a Hosted Collector with the fields parameter:

{
  "collector":{
    "collectorType":"Hosted",
    "name":"My Hosted Collector",
    "description":"An example Hosted Collector",
    "category":"HTTP Collection",
    "fields": {
        "cluster":"k8s.dev"
    }
  }
}

Using Local Configuration

Installed Collectors can use JSON files to configure its Sources when using Local Configuration File Management. Use the fields parameter in your JSON configuration to define fields on a Source.

Parameter Type Required? Description Access
fields JSON Object No JSON map of key-value fields (metadata) to apply to the Collector or Source. Modifiable

The following JSON is an example configuration of a Local File Source with the fields parameter:

{
   "api.version":"v1",
   "sources":[{
    "name":"Test-Chef",
    "category":"Chef",
    "automaticDateParsing":true,
    "multilineProcessingEnabled":false,
    "useAutolineMatching":false,
    "forceTimeZone":false,
    "timeZone":"UTC",
    "filters":[],
    "cutoffTimestamp":1426057200000,
    "encoding":"UTF-8",
    "fields":{
      "node":"hornetq-livestream-9",
      "deployment":"sumologic",
      "cluster":"k8s.dev"
    },
    "pathExpression":"/home/ubuntu/chef*.log",
    "blacklist":[],
    "sourceType":"LocalFile"
  }]
}
​​​​​​

HTTP Source fields

When uploading log data with HTTP Sources you can pass fields with the X-Sumo-Fields HTTP header. Your fields need to be in a comma separated list of key-value pairs. For example, a cURL command posting data with custom fields would look like:

curl -v -X POST -H 'X-Sumo-Fields:environment=dev,cluster=k8s' -T /file.txt <HTTP endpoint>

See how to upload logs to an HTTP Source.

Any fields passed with your data need to exist in your Fields table schema defined in Sumo. Any fields not defined in Sumo that are passed through a header are dropped. See how to define fields in the manage fields section.

Tags from EC2

Create a Sumo Logic AWS Metadata Source to collect custom tags from EC2 instances running on AWS. An Installed Collector automatically pulls AWS instance identity documents from instances to get their instanceId, instanceType, and region.

Logs ingested by Installed Collectors on EC2 instances will be tagged as long as the tag, including instance information tags, exists in the organization's Fields table. See how to define fields in the manage fields section. EC2 resource tags take precedence over EC2 instance information. Only one AWS Metadata Source is required to collect tags from multiple hosts.

Tags are returned in your search results and can be referenced in queries. For information about assigning tags to EC2 instances, see Tagging Your Amazon EC2 Resources in AWS help. 

Using fields

Fields can be used in the following ways:

  • Log Search page. Use the key-value pair as a keyword search expression (before the first pipe, | ).
  • Role Based Access Control (RBAC). Fields can be used in role search filters to control access to data.
  • Partitions and Scheduled Views. Fields can be used in the scope of Partitions and Scheduled Views.

Manage fields

Fields in your account are manageable at Manage Data > Settings > Fields.

fields table aug 12th 2019.png

The Manage Data > Settings > Fields page displays the following information: 

  • Status shows a checkmark in a green circle check in green circle.png to indicate if the field is actively being applied or an exclamation mark in a red circle exclamation in red circle.png to indicate if the field is disabled and being dropped.
  • Field Name is the name of the field, known as the key in the key-value pair.
  • Data Type shows the data type of the field.
  • Field Extraction Rules shows the number of Field Extraction Rules that reference the field.
  • Role Based Access Control shows the number of Roles using a search filter that references the field.
  • Partitions shows the number of Partitions that reference the field.
  • Collectors shows the number of Collectors that reference the field. (Available when viewing custom fields.)
  • Sources shows the number of Sources that reference the field. (Available when viewing custom fields.)
  • Fields Capacity (bottom of table) shows how many fields your account is using, out of the total available for use.

On the Manage Data > Settings > Fields page you can:

  • Click + Add to add fields.
  • Search fields
    • The dropdown next to the add button lets you toggle between the following:
      • Existing - Built-in Fields. These are metadata fields created by Sumo Logic and cannot be modified.
      • Existing - Custom Fields. These fields were either created by FERs or users.
      • Dropped Fields. These fields are being dropped due to not existing in the fields table.
  • Disable fields
  • Delete fields 

For the fields listed, select a row to view its details. A details pane appears to the right of the table where you can disable and delete the field.

selected field details pane.png

Add field

Adding a field will define it in the Fields table schema allowing it to be assigned as metadata to your logs.

  1. Click the + Add button on the top right of the table. A panel named Add Field appears to the right of the fields table.
  2. Input a field name and click Save.

add field input.png

Disable field

Disabling a field will stop it from being assigned to new log data. Any searches still using the field will continue to work but will not have the field returned in its results once disabled. Data already collected is not affected, you can still search on a disabled field against data that was collected before it was disabled.

In the details pane of the field select the menu icon and select Disable.

disable field.png

Delete field

Select the delete icon delete icon.png at the right of the row on the Fields table or in the details pane of the field. To delete a field you need to remove any references to it from some features. If the field is used by any of the following

  • Field Extraction Rule
  • Role
  • Partition
  • Collector
  • Source

you will see the following prompt and you must remove the field reference before you can delete it.

For example, if the field is used by a Field Extraction Rule, you must first delete the Field Extraction Rule before you can delete the field.

field cannot delete.png

If the field is not used by those features you will see the following prompt.

delete field confirm.png

View dropped fields

Dropped fields are fields being sent to Sumo, but are being ignored since they are not defined in your Fields table schema. Use the dropdown option to the left of the + Add button to select and view dropped fields.

dropped fields table.png

Select a dropped field from the table to open a details pane. There is a convenient button provided to create the field if needed.

create field from dropped table.png