Skip to main content
Sumo Logic

Use JSON to Configure Sources

This topic provides general information on configuring sources by using JSON files. See the following topics for additional information:

Defining a source JSON file

When registering a Collector, you can define a source JSON file using the sources or syncSources parameter in your user.properties or sumo.conf configuration file. These parameters are used the first time a collector is set up.

Parameter Type Description
sources String Sets the JSON file describing sources to configure on registration. To make changes to collector sources after the Collector has been configured, you can use the Collector Management API or the Sumo web application.
syncSources String Sets the JSON file describing sources to configure on registration, which will be continuously monitored and synchronized with the Collector's configuration.

For more information on setting the syncSources parameter, see Local Configuration File Management

Using JSON to configure multiple sources

You can use JSON to configure multiple sources in either of the following ways:

  • Create a single JSON file with the configuration information for all the sources (sources.json).
  • Create individual JSON files, one for each source, and then combine them in a single folder. You then configure the source folder instead of the individual sources.

See Use Multiple JSON Files to Configure Sources for more information.

Types of sources

Each source can have its own unique fields in addition to the generic fields listed in the previous table. The next table lists the valid field types. The sections that follow list the unique parameters for each and the associated JSON examples.

Log sources for installed collectors

Source Type Type Value
Local File Source LocalFile
Remote File Source RemoteFileV2
Local Windows Event Log Source LocalWindowsEventLog
Remote Windows Event Log Source RemoteWindowsEventLog
Local Windows Performance Source LocalWindowsPerfMon
Remote Windows Performance Source RemoteWindowsPerfMon
Syslog Source Syslog
Script Source Script
Docker Log Source DockerLog
Docker Stats Source DockerStats

Metric sources for installed collectors

Field Type Type Value
Host Metrics Source SystemStats
Graphite Source Graphite

Log sources for hosted collectors

Field Type Type Value
HTTP Source HTTP
Cloud Syslog Source Cloudsyslog
Amazon S3 Source Polling
AWS Elastic Load Balancing Source Polling
AWS CloudFront Source Polling
AWS CloudTrail Source Polling
AWS S3 Audit Source Polling

Metrics sources for hosted collectors

Field Type Type Value
AWS CloudWatch Source Polling

Common parameters for all source types

The following parameters are used for all source types.

Parameter Type Required? Default Description Access
sourceType String Yes   Type the correct type of source (see specific source types below for more information). not modifiable
name String Yes   Type the name of the source. Example:"SourceName". modifiable
description String No null Type a description of the source. Example:"SourceDescription". modifiable
category String No null Type the description of the category of the source. Example:"auth_logs" or "mail". modifiable
hostName String No null Type the host name of the source. Example:"SourceHost". The hostname can be a maximum of 128 characters. modifiable
timeZone String No null Type the time zone you'd like the source to use in TZ database format. Example:"America/Los_Angeles". modifiable
automaticDateParsing Boolean No true Determines if timestamp information is parsed or not. Type true to enable automatic parsing of dates (the default setting); type false to disable. If disabled, no timestamp information is parsed at all. modifiable
multilineProcessingEnabled Boolean No true Type true to enable; type false to disable. The default setting is true. Consider setting to falseto avoid unnecessary processing if you are collecting single message per line files (for example, Linux system.log). If you're working with multi-line messages (for example, log4J or exception stack traces), keep this setting enabled. modifiable
useAutolineMatching Boolean No true Type true to enable if you'd like message boundaries to be inferred automatically; type false to prevent message boundaries from being automatically inferred (equivalent to the Infer Boundaries option in the UI). The default setting is true. modifiable
manualPrefixRegexp String No null When using useAutolineMatching=false, type a regular expression that matches the first line of the message to manually create the boundary. modifiable
forceTimeZone Boolean No false Type true to force the source to use a specific time zone, otherwise type false to use the time zone found in the logs. The default setting is false. modifiable
defaultDateFormat String No null (Deprecated) Type the default format for dates used in your logs. For more information about timestamp options, see Timestamps, Time Zones, Time Ranges, and Date Formats modifiable
defaultDateFormats Object array No null Define formats for the dates present in your log messages. You can also specify a locator regex to help us narrow down the search to a specific part of the log message. For more information about timestamp options, see Timestamps, Time Zones, Time Ranges, and Date Formats modifiable
filters String array No [ ] If you'd like to add a filter to the source, type the name of the filter (Exclude, Include, Mask, Hash, or Forward. Review the Rules and Limitations for filters and see Creating processing rules using JSON. modifiable
cutoffTimestamp Long No 0 (collects all data)
 
Only collect data more recent than this timestamp, specified as milliseconds since epoch (13 digit). You can use this site to convert to epoch time: http://www.epochconverter.com/ modifiable
cutoffRelativeTime String No   Can be specified instead of cutoffTimestamp to provide a relative offset with respect to the current time. Example: use "-1h", "-1d", or "-1w" to collect data that's less than one hour, one day, or one week old, respectively.  not modifiable

Non-configurable parameters

The following parameters are automatically configured by the Sumo Logic Service. Don't include them in the sources JSON file.

  • id
  • alive
  • status

Time zone format

In a JSON source configuration, a string for the timeZone setting does not follow the same format as the time zone setting shown in the Sumo Logic Web Application. The JSON timeZone property uses the underlying TZ database time zone format instead of (GMT+11:00) style values.

Example:

"timeZone": "America/Los_Angeles",

You can find a list of TZ environment variables here

Timestamp Example

The following is a Timestamp example in JSON with two default date formats, yyyy-MM-dd HH:mm:ss and yyMMdd HH:mm:s:

{
    "source": {
        "name": "test",
        "defaultDateFormats": [{
            "format": "yyyy-MM-dd HH:mm:ss",
            "locator": "time=(.*),"
        }, {
            "format": "yyMMdd HH:mm:ss"
        }]
    }
}

Creating processing rules using JSON

You can include processing (filtering) rules when using JSON to configure sources. A filter specifies rules about which messages are sent to Sumo Logic. There are four types of filters:

  • Exclude—Removes messages before ingestion to Sumo Logic. Think of Exclude as a "black list" filter. For more information, see Include and Exclude Rules.
  • Include—Sends only the data you explicitly define to Sumo Logic. Think of Include as a "white list" filter. For more information, see Include and Exclude Rules.
  • Hash—Replaces a message with a unique, randomly-generated code to protect sensitive or proprietary information, such as credit card numbers or user names. By hashing this type of data you can still track it, even though it's fully hidden. For more information, see Hash Rules.
  • Mask—Replaces an expression with a mask string that you can customize; especially useful for protecting passwords or other data you wouldn't normally track. For more information, see Mask Rules.
  • Forward—Sends matching log messages to a data forwarding destination. For more information, see Example: data forwarding rule below.
Parameter Type Required? Description Access
name String Yes A name for the rule. Modifiable
filterType   Yes The filter type. Must be one of the following: Exclude, Include, Hash, Mask, or Forward. Modifiable
regexp String Yes A regular expression used to define the filter. If filterType = Mask or Hash, this regular expression must have at least one matching group, specifying the regions to be replaced by a mask or hash.

For multi-line messages, add single line modifiers (?s) to the beginning and end of the expression to support matching your string regardless of where it occurs in the message. For example:
(?s).*secur.*(?s)

Syslog UDP messages may contain a trailing newline character, which will require the above regular expression to properly match your string.
Modifiable
mask String Yes when filterType = "Mask" The mask string used when covering the matching log text. Modifiable

Example:  exclude—filter by a specified string

The following is an example of a filter to exclude messages containing a specified keyword.

"filters": [{
 "filterType": "Exclude",
 "name": "Filter keyword",
 "regexp": ".*keyword.*"
}]

When excluding messages based on a string that contains special characters, for example *("test")*,you will need to double-escape the special characters so they're valid within the JSON.

Example:

Message content to filter:

*("test")*

Standard Regex (this is the syntax if you create the filter using the UI):

\*\("test"\)\*

Filter syntax in JSON:

\\*\\(\"test\"\\)\\*

Filter example in JSON with double-escaped special characters:

{
    "source": {
        "name": "test",
        "filters": [{
            "filterType": "Exclude",
            "name": "Filter keyword",
            "regexp": "\\*\\(\"test\"\\)\\*"
        }]
    }
}

Example: data forwarding rule

In the JSON below for a source, the filters array specifies a data forwarding rule.  Before you can configure a data forwarding rule in JSON, you must obtain the sinkId for the data forwarding data destination. For instructions, see Get sinkId for a data forwarding destination below.

{
"api.version": "v1",
    "sources": [{
        "sourceType": "Syslog",
        "name": "example",
        "port": 514,
        "protocol": "TCP",
        "encoding": "UTF-8",
        "category": "example",
        "useAutolineMatching": false,
        "multilineProcessingEnabled": false,
        "timeZone": "UTC",
        "automaticDateParsing": true,
        "forceTimeZone": false,
        "defaultDateFormat": "dd/MMM/yyyy HH:mm:ss",
        "filters": [{

              "filterType": "Forward",

              "name": "example", "regexp": "(?s).*(?s)", "sinkId": 22

        }]
     }]
}

Get sinkId for a data forwarding destination

To determine the sinkId for a data forwarding destination, you use the Sumo web app to create a test data forwarding rule. Sumo updates the JSON configuration for the source with the sinkId of the destination you select. Then you can view the JSON configuration for the source, make a note of the sinkId, and then delete the test processing rule.

These instruction assume you have already created a data forwarding destination.

  1. Follow the instructions in Configure processing rules for data forwarding to add a data forwarding rule to a source on an installed collector. As part of this process, you will select the data forwarding destination to which you want to forward data.
  2. To view the JSON configuration for the source you updated in the previous step:
    1. Select Manage Data > Collection > Collection (Manage > Collection in the classic UI). 
    2. Click the icon to the right of the source. The API usage information panel appears. Make a note of the sinkId in the filter section of the JSON.
  3. Click the icon to the right of the Source. Make a note of the sinkId in the filter section of the JSON.
  4. Click Done to close the API usage information panel.
  5. Now that you have determined the sinkId for the data forwarding destination, delete the test rule.
    1. Select Manage Data > Collection > Collection (Manage > Collection in the classic UI).
    2. Navigate to the source to which you added the test rule.
    3. In the Processing Rules section of the page, click the delete icon to the right of the test rule.

Now that you have the sinkId for the data forwarding destination, you can define the filter array in the JSON for your source, following the example in Example: Data Forwarding Rule above.