Skip to main content
Sumo Logic

Use JSON to Configure Sources

This topic provides general information on configuring Sources by using parameters specified in JSON files. See the following topics for additional information:

Defining a Source JSON file

When registering a Collector, you can define a Source JSON file using the sources or syncSources parameter within your user.properties or sumo.conf configuration file. These parameters are used the first time a Collector is set up.

Parameter Type Description
sources String Sets the JSON file describing sources to configure on registration. To make changes to Collector Sources after the Collector has been configured, you can use the Collector Management API or the Sumo Logic Web Application.
syncSources String Sets the JSON file describing sources to configure on registration, which will be continuously monitored and synchronized with the Collector's configuration.

For more information on setting the syncSources parameter, see Local Configuration File Management

Using JSON to configure multiple Sources

You can use JSON to configure multiple Sources in either of the following ways:

  • Create a single JSON file with the configuration information for all the Sources (sources.json).
  • Create individual JSON files, one for each Source, and then combine them in a single folder. You then configure the Source folder instead of the individual Sources.

See Use Multiple JSON Files to Configure Sources for more information.

Types of Sources

Each Source can have its own unique fields in addition to the generic fields listed in the previous table. The next table lists the valid field types. The sections that follow list the unique parameters for each and the associated JSON examples.

Log Sources for Installed Collectors

Source Type Type Value
Local File Source LocalFile
Remote File Source RemoteFileV2
Local Windows Event Log Source LocalWindowsEventLog
Remote Windows Event Log Source RemoteWindowsEventLog
Local Windows Performance Source LocalWindowsPerfMon
Remote Windows Performance Source RemoteWindowsPerfMon
Syslog Source Syslog
Script Source Script
Docker Log Source DockerLog
Docker Stats Source DockerStats

Metric Sources for Installed Collectors

Field Type Type Value
Host Metrics Source SystemStats
Graphite Source Graphite

Log Sources for Hosted Collectors

Field Type Type Value
HTTP Source HTTP
Cloud Syslog Source Cloudsyslog
Amazon S3 Source Polling
AWS Elastic Load Balancing Source Polling
AWS CloudFront Source Polling
AWS CloudTrail Source Polling
AWS S3 Audit Source Polling

Metrics Sources for Hosted Collectors

Field Type Type Value
AWS CloudWatch Source Polling

Common parameters for all Source types

The following parameters are used for all Source types.

Parameter Type Required? Default Description Access
sourceType String Yes   Type the correct type of Source (see specific Source types below for more information). not modifiable
name String Yes   Type the name of the Source. Example:"SourceName". modifiable
description String No null Type a description of the Source. Example:"SourceDescription". modifiable
category String No null Type the description of the category of the Source. Example:"auth_logs" or "mail". modifiable
hostName String No null Type the host name of the Source. Example:"SourceHost". The hostname can be a maximum of 128 characters. modifiable
timeZone String No null Type the time zone you'd like the Source to use in TZ database format. Example:"America/Los_Angeles". modifiable
automaticDateParsing Boolean No true Determines if timestamp information is parsed or not. Type true to enable automatic parsing of dates (the default setting); type false to disable. If disabled, no timestamp information is parsed at all. modifiable
multilineProcessingEnabled Boolean No true Type true to enable; type false to disable. The default setting is true. Consider setting to falseto avoid unnecessary processing if you are collecting single message per line files (for example, Linux system.log). If you're working with multi-line messages (for example, log4J or exception stack traces), keep this setting enabled. modifiable
useAutolineMatching Boolean No true Type true to enable if you'd like message boundaries to be inferred automatically; type false to prevent message boundaries from being automatically inferred (equivalent to the Infer Boundaries option in the UI). The default setting is true. modifiable
manualPrefixRegexp String No null When using useAutolineMatching=false, type a regular expression that matches the first line of the message to manually create the boundary. modifiable
forceTimeZone Boolean No false Type true to force the Source to use a specific time zone, otherwise type false to use the time zone found in the logs. The default setting is false. modifiable
defaultDateFormat String No null (Deprecated) Type the default format for dates used in your logs. For more information about timestamp options, see Timestamps, Time Zones, Time Ranges, and Date Formats modifiable
defaultDateFormats Object array No null

(New) Define formats for the dates present in your log messages. You can also specify a locator regex to help us narrow down the search to a specific part of the log message. For more information about timestamp options, see Timestamps, Time Zones, Time Ranges, and Date Formats

modifiable
filters String array No [ ] If you'd like to add a filter to the Source, type the name of the filter ("Exclude", "Include", "Mask", or "Hash"). Review the Rules and Limitations for filters. modifiable
cutoffTimestamp Long No 0 (collects all data)
 
Only collect data more recent than this timestamp, specified as milliseconds since epoch (13 digit). You can use this site to convert to epoch time: http://www.epochconverter.com/ modifiable
cutoffRelativeTime String No   Can be specified instead of cutoffTimestamp to provide a relative offset with respect to the current time. Example: use "-1h", "-1d", or "-1w" to collect data that's less than one hour, one day, or one week old, respectively.  not modifiable

Non-configurable parameters

The following parameters are automatically configured by the Sumo Logic Service. Don't include them in the Sources JSON file.

  • id
  • alive
  • status

Time zone format

In a JSON source configuration, a string for the timeZone setting does not follow the same format as the time zone setting shown in the Sumo Logic Web Application. The JSON timeZone property uses the underlying TZ database time zone format instead of (GMT+11:00) style values.

Example:

"timeZone": "America/Los_Angeles",

You can find a list of TZ environment variables here

Creating Processing Rules Using a JSON File

You can include processing (filtering) rules when using JSON to configure Sources. A filter specifies rules about which messages are sent to Sumo Logic. There are four types of filters:

  1. Exclude. Removes messages before ingestion to Sumo Logic. Think of Exclude as a "black list" filter.
  2. Include. Sends only the data you explicitly define to Sumo Logic. Think of Include as a "white list" filter.
  3. Hash. Replaces a message with a unique, randomly-generated code to protect sensitive or proprietary information, such as credit card numbers or user names. By hashing this type of data you can still track it, even though it's fully hidden.
  4. Mask. Replaces an expression with a mask string that you can customize; especially useful for protecting passwords or other data you wouldn't normally track.
Parameter Type Required? Description Access
name String Yes A name for the rule. Modifiable
filterType   Yes

The Filter type. Must be one of the following:
"Exclude"
"Include"
"Hash"
"Mask"

Modifiable
regexp String Yes

A regular expression used to define the filter. If filterType = "Mask" orFilterType = "Hash", this regular expression must have at least one matching group, specifying the regions to be replaced by a mask or hash.

For multi-line messages, add single line modifiers (?s) to the beginning and end of the expression to support matching your string regardless of where it occurs in the message. For example:
(?s).*secur.*(?s)

Syslog UDP messages may contain a trailing newline character, which will require the above regular expression to properly match your string.

Modifiable
mask String Yes when filterType = "Mask" The mask string used when covering the matching log text. Modifiable

The following is an example of a filter to exclude messages containing a specified keyword.

"filters": [{
 "filterType": "Exclude",
 "name": "Filter keyword",
 "regexp": ".*keyword.*"
}]

When excluding messages based on a string that contains special characters, for example *("test")*,you will need to double-escape the special characters so they're valid within the JSON.

Example:

Message content to filter:

*("test")*

Standard Regex (this is the syntax if you create the filter using the UI):

\*\("test"\)\*

Filter syntax in JSON:

\\*\\(\"test\"\\)\\*

Filter example in JSON with double-escaped special characters:

{
    "source": {
        "name": "test",
        "filters": [{
            "filterType": "Exclude",
            "name": "Filter keyword",
            "regexp": "\\*\\(\"test\"\\)\\*"
        }]
    }
}

Timestamp Example

The following is a Timestamp example in JSON with two default date formats, yyyy-MM-dd HH:mm:ss and yyMMdd HH:mm:s:

{
    "source": {
        "name": "test",
        "defaultDateFormats": [{
            "format": "yyyy-MM-dd HH:mm:ss",
            "locator": "time=(.*),"
        }, {
            "format": "yyMMdd HH:mm:ss"
        }]
    }
}