Skip to main content
Sumo Logic

Remote File Source

Log files collected using a Remote File Source must be encoded in UTF-8 or ASCII. If you are editing a Source, metadata changes are reflected going forward. Metadata for previously collected log data will not be retroactively changed.

Sumo Logic scans remote directories every 30 seconds. 

To configure a Remote File Source:

  1. In Sumo Logic select Manage Data > Collection > Collection.
  2. Find the name of the installed Collector here you'd like to add a Source. Click Add and then choose Add Source from the pop-up menu.
  3. Select Remote File for the Source type. 
  4. Set the following:
    • Name. Type the name you'd like to display for the new Source. Description is optional. Source name metadata is stored in a searchable field called _sourceName.
    • Host. Enter the hostname or the IP address of the remote machine (the hostname entered must be the system hostname or IP address and cannot be changed). The hostname is stored in a searchable field called _sourceHost. The hostname can be a maximum of 128 characters.
      If the source you are configuring is on the same installed collector as a Docker log source, you can construct the Source Host metadata field using Docker variables. For more information, see Configuring sourceCategory and sourceHost using variables below.
    • Port. If your SSH server is listening on a nonstandard port, type the port number.
    • Path Expression. Enter the absolute path expression to the file the Source should tail. Remote File Sources support wildcards in file paths. If the timestamp formats for the files are not identical, set up a separate Remote File Source for each file.
      For Windows collections using Open SSH and Cygwin, specify the File path starting with/cygdrive. For example, if the path is "C:\mandy test\6.log" enter "/cygdrive/c/mandy\ test/6.log" in the File field. Use "\" to escape any spaces if they are present in the file path.
    • Collection should begin. Choose or enter how far back you'd like to begin collecting historical logs. You can either:
      • Choose a predefined value from dropdown list, ranging from “Now” to “24 hours ago” to “All Time”, or
      • Enter a relative value. To enter a relative value, click the Collection should begin field and press the delete key on your keyboard to clear the field. Then, enter a relative time expression, for example “-1w”. You can define when you want collection to begin in terms of months (M), weeks (w), days (d), hours (h) and minutes (m).
    • Source Category. Type any string to tag the output collected from this Source with searchable metadata. For example, type firewall to tag all entries from this Source in a field called _sourceCategory.
      If the source you are configuring is on the same installed collector as a Docker log source, you can construct the Source Category metadata field using Docker variables. For more information, see Configuring sourceCategory and sourceHost using variables below.
  5. Choose the type of Credentials used for this Source:
    • Username and Password. Enter valid user credentials for the remote machine.
    • Local SSH Config. Enter the username and the absolute path, including file name, to the PEM SSH key file located on the Collector host. Enter a password if required.

  6. Set any of the following under Advanced.
    • Blacklist. Optional. Add any files to be excluded by including one or more path expressions separated by commas. Note that this field takes a maximum of 10240 characters.
    • Enable Timestamp Parsing. This option is selected by default. If it's deselected, no timestamp information is parsed at all.
      • Time Zone. There are two options for Time Zone. You can use the time zone present in your log files, and then choose an option in case time zone information is missing from a log message. Or, you can have Sumo Logic completely disregard any time zone information present in logs by forcing a time zone. It's very important to have the proper time zone set, no matter which option you choose. If the time zone of logs can't be determined, Sumo Logic assigns logs UTC; if the rest of your logs are from another time zone your search results will be affected.
      • Timestamp Format. By default, Sumo Logic will automatically detect the timestamp format of your logs. However, you can manually specify a timestamp format for a Source. See Timestamps, Time Zones, Time Ranges, and Date Formats for more information.
    • Enable Multiline Processing. See Collecting Multiline Logs for details on multiline processing and its options. This is enabled by default. Use this option if you're working with multiline messages (for example, log4J or exception stack traces). Deselect this option if you want to avoid unnecessary processing when collecting single-message-per-line files (for example, Linux system.log). Choose one of the following:
      • Infer Boundaries. Enable when you want Sumo Logic to automatically attempt to determine which lines belong to the same message.
        If you deselect the Infer Boundaries option, you will need to enter a regular expression in the Boundary Regex field to use for detecting the entire first line of multiline messages.
      • Boundary Regex. You can specify the boundary between messages using a regular expression. Enter a regular expression that matches the entire first line of every multiline message in your log files.
  7. Create any processing rules you'd like for the new Source.
  8. When you are finished configuring the Source click Save.

Configuring sourceCategory and sourceHost using variables

In Collector version 19.216-22 and later, if you have a Docker Logs Source on the same Installed Collector where you are configuring the new Source, you can define the Source Category and Source Host, if the Source supports that field, for the new Source using system environment variables defined on the Collector’s host. To do so, specify the environment variables to include the metadata field in this form:

{{sys.VAR_NAME}}

Where VAR_NAME is an environment variable name, for example:

{{sys.PATH}}

You can use multiple variables, for example:

{{sys.PATH}} - {{sys.YourEnvVar}}

You can incorporate text in the metadata expression, for example:

AnyTextYouWant {{sys.PATH}} - {{sys.YourEnvVar}}

If a user-defined variable doesn’t exist, that portion of the metadata field will be blank.