Skip to main content

Forward Data from an Installed Collector

You can set up one or more data forwarding destinations and configure an Installed Collector to send raw log data from specified Sources to those destinations. The Collector will send the raw data to external destinations at the same time it sends data to Sumo.

You can forward raw log data using the following protocols.

  • Syslog (TCP and UDP)—Send log data to a syslog server.
  • Generic REST API—Send log data to a web services endpoint.
  • Hitachi Data Systems HTTP REST API—Send log data to Hitachi Content Platform (HCP).

Follow the steps below to set up a Collector to forward raw log data to an external destination.

You can set up Installed Collector data forwarding when you first configure Sources or at a later time. If you apply rules at a later time, keep in mind that they are not applied retroactively.

note

Data forwarding processing rules are processed after all other processing rules.

Step 1: Configure data forwarding destination

You need the Manage Collectors role capability to create a data forwarding destination.

To set up a data forwarding destination:

  1. Choose Manage Data > Collection > Data Archiving.
  2. Click + Destination to add a new destination.
  3. Select one of these options for Destination Type
    • Hitachi
    • Generic REST
    • Syslog
  4. Enter a name to identify the destination.
  5. Follow the instructions for your destination type in Config settings for each destination type and then click Save to save the information and add the new destination to the list.

Config settings for each destination type

Follow the instructions for the destination type you chose.

  • URL. Enter a URL to access the destination.

  • Object ID (Optional). Enter a path name or other file format and include any of the following variables:

    • {day} - Replace with the day of the year in the yyyy-MM-dd format.
    • {hour} Replace with hour in day (0-23).
    • {minute} Replace with minute in hour.
    • {second} Replace with second in hour.
    • {uuid} Replace with a unique, randomly generated identifier (UUID)
  • Username and Password. Enter the credentials to access the destination. These are placed in a Basic Auth header in the HTTP request from the Collector. If you're sending to a Sumo Logic HTTP Source this header is simply ignored and your data is ingested. You must have administrator privileges for the Collector.

Step 2: Configure processing rules for data forwarding

In this procedure, you define one or more processing rules that define the raw log data from a Source that you want to send to the external destination. Data forwarding processing rules are processed after all other processing rules.

There are several methods you can use to configure processing rules: 

To configure processing rules for data forwarding using the web application

  1. Go to Manage Data > Collection > Collection.
  2. Search for the source that you want to configure, and click the Edit link for the source. The source must be associated with an Installed Collector.
  3. Scroll down to the Processing Rules section and click the arrow to expand the section.
  4. Click Add Rule.
  5. Enter a name to define the rule.
  6. In the Filter field, enter the regular expression that defines the messages you want to forward. The regular expression must be RE2 compliant. For example, the regular expression .*ERROR.* matches all messages that contain ERROR. For more information about creating processing rules, see Create a Processing Rule.
  7. Select Forward messages that match as the rule type. This option is visible only if you have defined at least one data forwarding destination, as described in the previous section. 
  8. Select the Destination from the dropdown menu. If a Syslog Destination Type is selected, an option to select Transparent Forwarding is provided. Syslog forwarding by default prepends a timestamp and hostname to messages to ensure they comply with RFC 3164. If your syslog messages already comply, you can enable Transparent Forwarding to disable the default prepending behavior.
    transparent syslog forwarding option.png
  9. Click Apply. The new rule is listed along with any other previously defined processing rules.
  10. Click Add Rule if you want to add another rule.
    ProcessingRule.png
  11. Click Save to save the rules you defined and start forwarding data that matches the rules.

Configuring the size of forwarded syslog messages

In accordance with RFC 3164, by default the Collector forwards syslog messages in 1024-byte segments, sending each segment as a separate message. To change the segment size, add the forwarding.syslog.maxMessageSize property to the Collector's collector.properties file (in the Collector's config directory) and restart the Collector. Specify the desired size in bytes. For example:

forwarding.syslog.maxMessageSize = 2048

Configure data forwarding queue size

In Collector version 19.216-22 and later, in-memory storage of an Installed Collector’s data forwarding queue is backed by disk storage. When the in-memory queue reaches a given size, the Collector extends the queue on disk.

Sumo allocates memory and disk storage for data to be forwarded to REST and TCP syslog destinations. By default, Sumo allocates:

  • 8MB of memory and 500MB of disk storage for each syslog destination.

    Note Data forwarding using UDP isn't queued.

  • 8MB of memory and 500MB of disk storage for each REST endpoint.  

You can add properties to the collector.properties file, in the Collector's /config directory, to specify how much memory and disk the data forwarding queue can consume. The limits you specify for a destination type will apply to each destination of that type.

After the memory and disk limits are reached, data will be dropped, so the limits should not be set too low

PropertyDescription
queue.rest.max.memory.mbSpecifies the amount of memory allocated to the data forwarding queue for each REST destination.

Default: 8MB
queue.rest.max.disk.mbSpecifies the amount of disk space allocated to the data forwarding queue for each REST destination.

Default: 500MB
queue.syslog.max.memory.mbSpecifies the amount of memory allocated to the data forwarding queue for each Syslog destination.

Default: 8MB
queue.syslog.max.disk.mbSpecifies the amount of disk space allocated to the data forwarding queue for each Syslog destination.

Default: 500MB
Legal
Privacy Statement
Terms of Use

Copyright © 2024 by Sumo Logic, Inc.