Skip to main content
Sumo Logic

Collect Logs and Metrics for Nginx (Legacy)

Learn how to collect logs and metrics for  Sumo Logic  App for Nginx (Legacy).

This page provides instructions for configuring log and metric collection for the Sumo Logic App for Nginx (Legacy).

Collection Process Overview

Configuring log and metric collection for the Nginx (Legacy) App includes the following tasks:

Collect Logs for Nginx

Non-Kubernetes

This section provides instructions for configuring log collection for the Sumo Logic App for Nginx. Follow the below instructions to set up the Log collection.

  1. Configure logging in Nginx 
  2. Configure a Collector
  3. Configure a Source
1. Configure logging in Nginx

Before you can configure Sumo Logic to ingest logs, you must configure the logging of errors and processed requests in NGINX Open Source and NGINX Plus. For instructions, refer to the following documentation:

https://www.nginx.com/resources/admin-guide/logging-and-monitoring/

2. Configure a Collector

Use one of the following Sumo Logic Collector options:

  1. To collect logs directly from the Nginx machine, configure an Installed Collector.
  2. If you are using a service like Fluentd, or you would like to upload your logs manually, Create a Hosted Collector
3. Configure a Source

For an Installed Collector

To collect logs directly from your Nginx machine, use an Installed Collector and a Local File Source. 

  1. Add a Local File Source.
  2. Configure the Local File Source fields as follows:
    • Name. (Required)
    • Description. (Optional)
    • File Path (Required). Enter the path to your error.log or access.log. The files are typically located in /var/log/nginx/error.log. If you are using a customized path, check the nginx.conf file for this information. If you are using Passenger, you may have instructed Passenger to log to a specific log using the passenger_log_file option.
    • Source Host. Sumo Logic uses the hostname assigned by the OS unless you enter a different hostname.
    • Source Category. Enter any string to tag the output collected from this Source, such as Nginx/Access or Nginx/Error. (The Source Category metadata field is a fundamental building block to organize and label Sources. For details see Best Practices.)
  3. Configure the Advanced section:
    • Enable Timestamp Parsing. Select Extract timestamp information from log file entries.
    • Time Zone. Automatically detect.
    • Timestamp Format. The timestamp format is automatically detected.
    • Encoding. Select UTF-8 (Default).
    • Enable Multiline Processing. 
      • Error logs. Select Detect messages spanning multiple lines and Infer Boundaries - Detect message boundaries automatically.
      • Access logs. These are single-line logs, uncheck Detect messages spanning multiple lines.
  4. Click Save.

For a Hosted Collector

If you are using a service like Fluentd, or you would like to upload your logs manually, use a Hosted Collector and an HTTP Source.

  1. Add an HTTP Source.
  2. Configure the HTTP Source fields as follows:
    • Name. (Required)
    • Description. (Optional)
    • Source Host. Sumo Logic uses the hostname assigned by the OS unless you enter a different hostname.
    • Source Category. Enter any string to tag the output collected from this Source, such as Nginx/Access or Nginx/Error. (The Source Category metadata field is a fundamental building block to organize and label Sources. For details see Best Practices.)
  3. Configure the Advanced section:
    • Enable Timestamp Parsing. Select Extract timestamp information from log file entries.
    • Time Zone. For Access logs, use the time zone from the log file. For Error logs, make sure to select the correct time zone.
    • Timestamp Format. The timestamp format is automatically detected.
    • Enable Multiline Processing. 
      • Error logs: Select Detect messages spanning multiple lines and Infer Boundaries - Detect message boundaries automatically.
      • Access logs: These are single-line logs, uncheck Detect messages spanning multiple lines.
  4. Click Save.
  5. When the URL associated with the HTTP Source is displayed, copy the URL so you can add it to the service you are using, such as Fluentd.

Sample Log Message

Access Log Example

50.1.1.1 - example [23/Sep/2016:19:00:00 +0000] "POST /api/is_individual HTTP/1.1" 200 58 "-" 
"python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-36-generic"

Error Log Example

2016/09/23 19:00:00 [error] 1600#1600: *61413 open() "/srv/core/client/dist/client/favicon.ico" 
failed (2: No such file or directory), client: 101.1.1.1, server: _, request: "GET /favicon.ico 
HTTP/1.1", host: "example.com", referrer: "https://abc.example.com/"

Create Field Extraction Rules

Field Extraction Rules (FERs) tell Sumo Logic which fields to parse out automatically. For instructions, see Create a Field Extraction Rule

Nginx assumes the NCSA extended/combined log file format for Access logs and the default Nginx error log file format for error logs.

Both the parse expressions can be used for logs collected from Nginx Server running on Local or container-based systems.

FER for Access Logs

Use the following Parse Expression:

| json field=_raw "log" as nginx_log_message nodrop
| if (isEmpty(nginx_log_message), _raw, nginx_log_message) as nginx_log_message
| parse regex field=nginx_log_message "(?<Client_Ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
| parse regex field=nginx_log_message "(?<Method>[A-Z]+)\s(?<URL>\S+)\sHTTP/[\d\.]+\
"\s(?<Status_Code>\d+)\s(?<Size>[\d-]+)\s\"(?<Referrer>.*?)\"\s\"(?<User_Agent>.+?)\".*"

FER for Error Logs

Use the following Parse Expression:

| json field=_raw "log" as nginx_log_message nodrop
| if (isEmpty(nginx_log_message), _raw, nginx_log_message) as nginx_log_message
| parse regex field=nginx_log_message "\s\[(?<Log_Level>\S+)\]\s\d+#\d+:\s(?:\*\d+\s|)(?<Message>[A-Za-z][^,]+)(?:,|$)"
| parse field=nginx_log_message "client: *, server: *, request: \"* * HTTP/1.1\", host: 
\"*\"" as Client_Ip, Server, Method, URL, Host nodrop

Query Samples

This sample Query is from the Requests by Clients panel of the Nginx (Legacy) - Overview dashboard.

_sourceCategory = Labs/Nginx/Logs
| json field=_raw "log" as nginx_log_message nodrop
| if (isEmpty(nginx_log_message), _raw, nginx_log_message) as nginx_log_message
| parse regex field=nginx_log_message "(?<Client_Ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
| parse regex field=nginx_log_message "(?<Method>[A-Z]+)\s(?<URL>\S+)\sHTTP/[\d\.]+\"\s(?<Status_Code>\d+)\s(?<Size>[\d-]+)\s\"(?<Referrer>.*?)\"\s\"(?<User_Agent>.+?)\".*"
| where _sourceHost matches "{{Server}}" and Client_Ip matches "{{Client_Ip}}" and Method matches "{{Method}}" and URL matches "{{URL}}" and Status_Code matches "{{Status_Code}}"
| count as count by Client_Ip
| sort count