Skip to main content
Sumo Logic

Cloud Syslog Source

You can configure a cloud syslog source to allow a syslog client to send RFC 5424-compliant messages to Sumo Logic. A source-specific token, generated by Sumo Logic, is inserted into each message to identify the source. Transport-level security (TLS v1.1 or v1.2 over TCP) is required.

Sumo Logic manages an elastic scaling set of syslog servers, which scales up and down behind a set of AWS Elastic Load Balancers. The AWS ELB set can also scale up and down. For this reason, instead of IP address-based endpoints, Sumo Logic uses endpoint hostnames in this format:

syslog.collection.YOUR_DEPLOYMENT.sumologic.com

where the YOUR_DEPLOYMENT can correspond to us1us2au, or eu. The full hostname, with the correct YOUR_DEPLOYMENT, is displayed along with the Sumo Logic token.

First, use Sumo Logic to configure a cloud syslog source, as described below. This will generate a Sumo Logic token and the endpoint hostname. Set up TLS by downloading a cert to your server, as described next.

Sumo Logic supports syslog clients, including syslog-ng and rsyslog. Follow the instructions in the appropriate section to configure your server to send syslog data. If syslog data does not appear in Sumo Logic, refer to the Troubleshooting notes.

Configure a cloud syslog source

Cloud syslog configuration requires a token that is automatically generated when you configure a cloud syslog source. The token allows Sumo Logic to keep track of your log messages so they’re not confused with the log messages from any other customers. It’s tied to the source, but not to any specific user. 

Include the token as the Structured ID in every syslog message that is sent to Sumo Logic. The token is removed by Sumo Logic during ingestion and is not included with your syslog message in search results.

The token is deleted if you delete the source. To change a token, use the Regenerate Token option as described in the following procedure.

To configure a cloud syslog source:

  1. In Sumo Logic select Manage > Collection.
  2. On the Collection page, click Add Source next to a Hosted Collector.  See Set up a Hosted Collector for information on adding Hosted Collectors.
  3. Select Cloud Syslog.
  4. Enter a name to display for this source in Sumo Logic. Description is optional.
  5. (Optional) For Source Host and Source Category, enter any string to tag the output collected from this source. (Category metadata is stored in a searchable field called _sourceCategory.)
  6. Set any of the following under Advanced:
  • Enable Timestamp Parsing. This option is selected by default. If it's deselected, no timestamp information is parsed.
  • Time Zone. There are two options for Time Zone. You can use the time zone present in your log files, and then choose an option in case time zone information is missing from a log message. Or, you can have Sumo Logic completely disregard any time zone information present in logs by forcing a time zone. It's important to have the proper time zone set, no matter which option you choose. If the time zone of logs can't be determined, Sumo Logic assigns the UTC time zone; if the rest of your logs are from another time zone your search results will be affected.
  • Timestamp Format. By default, Sumo Logic will automatically detect the timestamp format of your logs. However, you can manually specify a timestamp format for a source. See Timestamps, Time Zones, and Time Ranges, and Date Formats.
  1. Create any Processing Rules you'd like for the new source.
  2. Click Save.
    cloudsyslog02.png

    The token information is displayed in a read-only dialog box.  
     
  3. Click Copy to copy the information for use in the syslog client. The information is copied in the following format:

    Token: 9HFxoa6+lXBmvSM9koPjGzvTaxXDQvJ4POE/WCURPAo+w4H7PmZm8H3mSEKxPl0Q@41123, Host: syslog.collection.YOUR_DEPLOYMENT.sumologic.com, TCP TLS Port: 6514

    The number 41123 associated with the token is the Sumo Logic Private Enterprise Number (PEN). (This number is already included in the token.) There are two options for including the token. You can include it in the structured data field or in the message body. 

    In the following example, the token is in the structured data field. 

    <165>1 2015-1-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [YOUR_TOKEN] msg

    In the following example, the token is in the message body.

    <165>1 2015-1-11T22:14:15.003Z mymachine.example.com evntslog - ID47 - YOUR_TOKEN msg
  4. After configuring the source, you can perform these token operations from the Collectors and Sources page:
  • Click Show Token to display the token for a cloud syslog source at any time. 
    showtoken.png
  • Click Regenerate Token if you need to generate a new token.
    regentoken.png

Set up TLS

Set up Transport Layer Security (TLS).

Download the GeoTrust certificate from https://www.geotrust.com/resources/r...Primary_CA.pem

Change to the appropriate directory. For syslog-ng:

$ cd /etc/syslog-ng/ca.d
$ wget -O geotrust_ca.crt https://www.geotrust.com/resources/root_certificates/certificates/GeoTrust_Primary_CA.pem
$ ln -s geotrust_ca.crt `openssl x509 -noout -hash -in geotrust_ca.crt`.0

For rsyslog:

$ cd /etc/rsyslog.d/keys/ca.d
$ wget -O geotrust_ca.crt https://www.geotrust.com/resources/root_certificates/certificates/GeoTrust_Primary_CA.pem

Configure a cloud syslog source for syslog-ng

If you are new to syslog-ng, follow this link to install syslog-ng

This section shows how to configure a syslog client using syslog-ng that will send the syslog message to be received by the Sumo Logic Cloud syslog service. You must specify a template, a destination, and a source. 

Define a template with the correct format for Sumo Logic. Messages must be in this format to be accepted, and the ordering of the $ fields must be as shown.

template t_sumo_syslog {
    template("<$PRI>1 $ISODATE $HOST $PROGRAM $PID $MSGID [E5kTyaEcth45/DU81M236oU4vM8j1ZaqTpWgjXB6lod7cFTeq09zzMn5ErmM0O/3@41123] $MSG\n"); template_escape(no);
};

Replace the sample token E5kTyaEcth45/DU81M236oU4vM8j1ZaqTpWgjXB6lod7cFTeq09zzMn5ErmM0O/3@41123 
with your token.

Define a destination to use the Sumo Logic endpoint. The following TCP destination option example specifies the endpoint (syslog.collection.YOUR_DEPLOYMENT.sumologic.com) and TCP TLS port 6514. It also includes the ca-dir for your CA certificate. Finally, it specifies that only trusted certificates will be accepted for connectivity to the remote endpoint.

destination d_sumo_tls {
   tcp("syslog.collection.YOUR_DEPLOYMENT.sumologic.com"
       port("6514")
       template(t_sumo_syslog)
       tls(
           ca-dir("/etc/syslog-ng/ca.d")
           peer_verify("required-trusted")
       )
   );
};

If you choose to use the syslog destination option, the following example applies. For reference, see syslog() destination options in the syslog-ng open source documentation. You must include the token in the message body instead of in the structured data field for the syslog destination.

destination d_sumo_tls {
   syslog("syslog.collection.YOUR_DEPLOYMENT.sumologic.com"
       port("6514")
       template(t_sumo_syslog)
       transport(tls)
       tls(
           ca-dir("/etc/syslog-ng/ca.d")
           peer_verify("required-trusted")
       )
   );
};

Specify which logs will be sent to the Sumo Logic destination. This example links specifies an existing syslog-ng source (s_sys), applies a syslog-ng filter (f_default), and specifies use of the Sumo Logic endpoint (d_sumo_tls).

log {
    source(s_sys);
    filter(f_default);
    destination(d_sumo_tls);
};

Configure a cloud syslog source for rsyslog

This section shows how to configure a syslog client using rsyslog that will send the syslog message to be received by the Sumo Logic Cloud syslog service. If you are new to rsyslog, follow the rsyslog documentation to install.

After rsyslog is installed, edit the configuration file to start sending logs to Sumo Logic. The configuration file is located at /etc/rsyslog.conf by default. 

For rsyslog v7 and earlier

# Setup disk assisted queues
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1     # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g       # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on     # save messages to disk on shutdown
$ActionQueueType LinkedList       # run asynchronously
$ActionResumeRetryCount -1        # infinite retries if host is down

# RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/geotrust_ca.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer syslog.collection.YOUR_DEPLOYMENT.sumologic.com

template(name="SumoFormat" type="string" string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [YOUR_TOKEN] %msg%\n")

*.* action(type="omfwd" protocol="tcp" target="syslog.collection.YOUR_DEPLOYMENT.sumologic.com" port="6514" template="SumoFormat")

In the template statement, be sure to replace YOUR_TOKEN with your actual token, and YOUR_DEPLOYMENT with us1us2au, or eu as appropriate. Properties in the string begin and end with '%'. All other text and white space is treated literally. For more information about rsyslog configuration, see the rsyslog template documentation or the rsyslog omfwd documentation.

For rsyslog v8 and later

# Setup disk assisted queues# Setup disk assisted queues
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1     # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g       # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on     # save messages to disk on shutdown
$ActionQueueType LinkedList       # run asynchronously
$ActionResumeRetryCount -1        # infinite retries if host is down

# RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/geotrust_ca.crt

template(name="SumoFormat" type="string" string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [YOUR_TOKEN] %msg%\n")

action(type="omfwd" 
        protocol="tcp" 
        target="syslog.collection.YOUR_DEPLOYMENT.sumologic.com" 
        port="6514" 
        template="SumoFormat" 
        StreamDriver="gtls" 
        StreamDriverMode="1" 
        StreamDriverAuthMode="x509/name" 
        StreamDriverPermittedPeers="syslog.collection.*.sumologic.com")

In the template statement, be sure to replace YOUR_TOKEN with your actual token, and YOUR_DEPLOYMENT with us1us2au, or eu as appropriate. Properties in the string begin and end with '%'. All other text and white space is treated literally. For more information about rsyslog configuration, see the rsyslog template documentation or the rsyslog omfwd documentation.

Message format

Syslog messages must be in RFC 5424-compliant form. Messages over 64KB in length are truncated.

This diagram shows the RFC 5424 format:

RFC 5424 Message Format

Known issues

TCP connections go through Amazon Elastic Load Balancer (ELB), which has an idle timeout set to 60 minutes. Connections are closed by ELB if they are idle for more than 60 minutes. It is possible to lose a few messages when the syslog client tries to re-establish the connection following an idle timeout.

Rsyslog provides the following configuration parameters to minimize data loss.

$ActionResumeRetryCount 1 # Reconnect but misses the first message after connection is idle for an hour.
$RebindInterval 1 #Establish a new connection every message. Change 1 to another number N so a new connection gets used every N messages.

Troubleshooting

If you followed the documentation for syslog-ng or rsyslog configurations but still do not see data in Sumo Logic, follow these steps to troubleshoot:

  1. Verify that your cloud syslog token is valid, contains the ‘@41123’ part, and is in the correct location of your message (in the Structured Data field or the message body).
  2. After each configuration change, restart your syslog client.

$ sudo service rsyslog restart
or
$ /etc/init.d/syslog-ng restart

  1. Verify that your syslog client is actually receiving data from your data sources.
  2. Verify that there no firewall rules are blocking the packets.
  3. Verify that your account is not throttled.
  4. Run tcpdump to check if packets are flowing in both directions:

$ sudo tcpdump host syslog.collection.YOUR_DEPLOYMENT.sumologic.com
    
The output should look something like this:

For TLS handshake:

21:53:08.005676 IP client-host.client-port > server-host: Flags [S], seq 471724626, win 26883, options [mss 8961,sackOK,TS val 523109896 ecr 0,nop,wscale 7], length 0

21:53:08.006321 IP server-host > client-host.client-port: Flags [S.], seq 3885139041, ack 471724627, win 28960, options [mss 1460,sackOK,TS val 112542097 ecr 523109896,nop,wscale 8], length 0

21:53:08.006349 IP client-host.client-port > server-host: Flags [.], ack 1, win 211, options [nop,nop,TS val 523109896 ecr 112542097], length 0
...

For sending data:

21:46:35.954790 IP client-host.client-port > server-host: Flags 
[P.], seq 471728183:471728572, ack 3885141774, win 256, options [nop,nop,TS val 523153927 ecr 112585055], length 389

21:46:35.955326 IP server-host > client-host.client-port: Flags [.], ack 389, win 181, options [nop,nop,TS val 112586129 ecr 523153927], length 0
...
  1. If the previous steps don't solve the problem, file a support case with information about the specific Collector and Source setup. You can also include the tcpdump if available.