Collect Logs and Metrics for Nginx Plus

Collection Process Overview

Sumo Logic supports a collection of logs and metrics data from Nginx Plus in both Kubernetes and non-Kubernetes environments. 

Please click on the appropriate links below based on the environment where your Nginx Plus servers are hosted.

• Collect Logs and Metrics for Kubernetes environments
• Collect Logs and Metrics for Non-Kubernetes environments

Sample Log Message 

50.1.1.1 - example [23/Sep/2016:19:00:00 +0000] "POST /api/is_individual HTTP/1.1" 200 58 "-" 
"python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-36-generic"

2016/09/23 19:00:00 [error] 1600#1600: *61413 open() "/srv/core/client/dist/client/favicon.ico" 
failed (2: No such file or directory), client: 101.1.1.1, server: _, request: "GET /favicon.ico 
HTTP/1.1", host: "example.com", referrer: "https://abc.example.com/"

{"timestamp":1620821977736,"log":"10.244.0.132 - - [12/May/2021:12:19:28 +0000] \"GET //demo-index.html HTTP/1.1\" 200 8777 \"-\" \"curl/7.68.0\"","stream":"stdout","time":"2021-05-12T12:19:28.975861476Z"}

{"timestamp":1620821977737,"log":"2021/05/12 12:19:36 [error] 7#7: *8192 upstream timed out (110: Connection timed out) while connecting to upstream, health check \"\" of peer 44.240.53.50:12345 in upstream \"stream_backend2\"","stream":"stderr","time":"2021-05-12T12:19:36.344706832Z"}

Create Field Extraction Rules

Field Extraction Rules (FERs) tell Sumo Logic which fields to parse out automatically. For instructions, see Create a Field Extraction Rule. 

Nginx assumes the NCSA extended/combined log file format for Access logs and the default Nginx error log file format for error logs.

Both the parse expressions can be used for logs collected from Nginx Plus Server running on Local or container-based systems.

FER for Access Logs

Use the following Parse Expression:

| json field=_raw "log" as nginx_log_message nodrop
| if (isEmpty(nginx_log_message), _raw, nginx_log_message) as nginx_log_message
| parse regex field=nginx_log_message "(?<Client_Ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
| parse regex field=nginx_log_message "(?<Method>[A-Z]+)\s(?<URL>\S+)\sHTTP/[\d\.]+\
"\s(?<Status_Code>\d+)\s(?<Size>[\d-]+)\s\"(?<Referrer>.*?)\"\s\"(?<User_Agent>.+?)\".*"

FER for Error Logs

Use the following Parse Expression:

| json field=_raw "log" as nginx_log_message nodrop
| if (isEmpty(nginx_log_message), _raw, nginx_log_message) as nginx_log_message
| parse regex field=nginx_log_message "\s\[(?<Log_Level>\S+)\]\s\d+#\d+:\s(?:\*\d+\s|)(?<Message>[A-Za-z][^,]+)(?:,|$)"
| parse field=nginx_log_message "client: *, server: *, request: \"* * HTTP/1.1\", host: 
\"*\"" as Client_Ip, Server, Method, URL, Host nodrop

Query Samples

This sample Query is from the Responses Over Time panel of the Nginx Plus - Overview dashboard.

_sourcecategory=Labs/Nginx/Logs 
| json auto maxdepth 1 nodrop
| if (isEmpty(log), _raw, log) as nginx_log_message
| parse regex field=nginx_log_message "(?<Client_Ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
| parse regex field=nginx_log_message "(?<Method>[A-Z]+)\s(?<URL>\S+)\sHTTP/[\d\.]+\"\s(?<Status_Code>\d+)\s(?<Size>[\d-]+)\s\"(?<Referrer>.*?)\"\s\"(?<User_Agent>.+?)\".*"
| where _sourceHost matches "{{Server}}" and Client_Ip matches "{{Client_Ip}}" and Method matches "{{Method}}" and URL matches "{{URL}}" and Status_Code matches "{{Status_Code}}"
| if(Status_Code matches "2*", 1, 0) as Successes 
| if(Status_Code matches "3*", 1, 0) as Redirects 
| if(status_code matches "4*", 1, 0) as Client_Errors 
| if(Status_Code matches "5*", 1, 0) as Server_Errors 
| timeslice by 5m
| sum(Successes) as Successes, sum(Client_Errors) as Client_Errors,  sum(Redirects) as Redirects, sum(Server_Errors) as Server_Errors by _timeslice
| sort by _timeslice asc