Skip to main content
Sumo Logic

Collect Logs for Nginx

This procedure explains how to collect error logs and processed requests in NGINX Open Source and NGINX Plus.

Log Types

Nginx logs assume the NCSA extended/combined log file format for Access logs, and the default Nginx error log file format for error logs. For more details on the Nginx log file format, refer to:

http://nginx.org/en/docs/http/ngx_http_log_module.html

Configure Logging in Nginx

Before you can configure Sumo Logic to ingest logs, you must configure configure logging of errors and processed requests in NGINX Open Source and NGINX Plus. For instructions, refer to the following documentation:

https://www.nginx.com/resources/admin-guide/logging-and-monitoring/

Configure a Collector

Use one of the following Sumo Logic Collector options:

  1. To collect logs directly from the Nginx machine, configure an Installed Collector.
  2. If you are using a service like Fluentd, or you would like to upload your logs manually, configure a Hosted Collector

Configure a Source

For an Installed Collector

To collect logs directly from your Nginx machine, use an Installed Collector and a Local File Source. 

  1. Add a Local File Source.
  2. Configure the Local File Source fields as follows:
    1. Name. (Required)
    2. Description. (Optional)
    3. File Path (Required). Enter the path to your error.log or access.log. The files are typically located in /var/log/nginx/error.log. If you are using a customized path, check the nginx.conf file for this information. If you are using Passenger, you may have instructed Passenger to log to a specific log using the passenger_log_file option.
    4. Source Host. Sumo Logic uses the hostname assigned by the OS unless you enter a different host name.
    5. Source Category. Enter any string to tag the output collected from this Source, such as Nginx/Access or Apache/Access. (The Source Category metadata field is a fundamental building block to organize and label Sources. For details see Best Practices.)
  3. Configure the Advanced section:
    1. Enable Timestamp Parsing. Select Extract timestamp information from log file entries.
    2. Time Zone. For Access logs, use the time zone from the log file. For Error logs, make sure to select the correct time zone.
    3. Timestamp Format. The timestamp format is automatically detected.
    4. Encoding. Select UTF-8 (Default).
    5. Enable Multiline Processing. Select Detect messages spanning multiple lines and Infer Boundaries - Detect message boundaries automatically.
  4. Click Save.

For a Hosted Collector

If you are using a service like Fluentd, or you would like to upload your logs manually, use a Hosted Collector and an HTTP Source.

  1. Add an HTTP Source.
  2. Configure the HTTP Source fields as follows:
    1. Name. (Required)
    2. Description. (Optional)
    3. Source Host. Sumo Logic uses the hostname assigned by the OS unless you enter a different host name.
    4. Source Category. Enter any string to tag the output collected from this Source, such as Nginx/Access or Apache/Access. (The Source Category metadata field is a fundamental building block to organize and label Sources. For details see Best Practices.)
  3. Configure the Advanced section:
    1. Enable Timestamp Parsing. Select Extract timestamp information from log file entries.
    2. Time Zone. For Access logs, use the time zone from the log file. For Error logs, make sure to select the correct time zone.
    3. Timestamp Format. The timestamp format is automatically detected.
    4. Enable Multiline Processing. Select Detect messages spanning multiple lines and Infer Boundaries - Detect message boundaries automatically.
  4. Click Save.
  5. When the URL associated with the HTTP Source is displayed, copy the URL so you can add it to the service you are using, such as Fluentd.

Create Field Extraction Rules

Field Extraction Rules (FERs) tell Sumo Logic which fields to parse out automatically. For instructions, see Create a Field Extraction Rule

Nginx assumes the NCSA extended/combined log file format for Access logs and the default Nginx error log file format for error logs.

FER for Access Logs

Use the following Parse Expression:

parse regex "^(?<src_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
| parse regex "(?<method>[A-Z]+)\s(?<url>\S+)\sHTTP/[\d\.]+\"\s(?<status_code>\d+)\s(?<size>[\d-]+)\s\"(?<referrer>.*?)\"\s\"(?<user_agent>.+?)\".*"

FER for Error Logs

Use the following Parse Expression:

parse regex "\[(?<log_level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)$"

Query samples

Parsing expression for public/apache/access (Apache Access Parser)

parse regex "\"(?<method>[A-Z]+) (?<url>.+?) HTTP/[\d\.]+\" (?<status_code>\d+) (?<size>[\d-]+) \"(?<referrer>.*?)\" \"(?<user_agent>.+?)\".*" nodrop | parse regex "^(?<src_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" nodrop

Sample Log Messages

Access Log Example

50.1.1.1 - example [23/Sep/2016:19:00:00 +0000] "POST /api/is_individual HTTP/1.1" 200 58 "-" "python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-36-generic"

Error Log Example

2016/09/23 19:00:00 [error] 1600#1600: *61413 open() "/srv/core/client/dist/client/favicon.ico" failed (2: No such file or directory), client: 101.1.1.1, server: _, request: "GET /favicon.ico HTTP/1.1", host: "example.com", referrer: "https://abc.example.com/"