Collect logs and metrics for Docker EE
This page has instructions for collecting logs for the Sumo App for Docker EE. This app works in conjunction with the Docker ULM App, and the first step is to configure collection for Docker ULM App. Then, since Docker EE uses Kubernetes for orchestration, you install and deploy the Sumo Logic Fluentd plugin for collection of Docker EE specific components.
Step 1. Collect logs and metrics for the Docker ULM App
This section shows you how to configure log and metric collection for the Docker ULM App, which requires configuring two sources:
- Docker Logs. Collects stdout/stderr logs from processes that are running within Docker containers.
- Docker Stats. Collects metrics about Docker containers.
To configure log and metric collection for the Docker ULM App, do the following:
- Follow the instructions to Collect Logs and Metrics for Docker ULM.
- Follow the instructions to Install the Docker ULM App and dashboards.
Step 2. Deploy FluentD and FluentBit to collect logs
Follow the instructions in this section to create Sumo Logic fields and collector, then deploy Fluentd and FluentBit on Manager Node.
Troubleshooting
If you get the following error while executing kubectl, your kube config is not wired correctly to UCP’s kube:
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Do the following:
- Log in to the UCP Web UI (dashboard), navigate to your user account, and click My Profile.
- Then click Client Bundles > New Client Bundle > Generate Client Bundle.
-
Unpack the ucp-bundle-XYZ.zip file on the Manager Node/Machine.
unzip ucp-bundle-XYZ.zip
-
Run the following command.
source env.sh
Assuming you have a Mac/Unix environment, docker and kubectl will work on your cluster as your user ID, as shown in the following example where the user ID is admin.
Log types and examples
The Docker EE App uses the following log types.
Log Source Type |
Example Log Message |
---|---|
{"timestamp":1557369652796,"PRIORITY":"6","SYSLOG_FACILITY":"3","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_SLICE":"system.slice","_BOOT_ID":"27b76ce41e524d5786122a7601a30f01","_MACHINE_ID":"a211db66c10846958dace7213f56ad40","_HOSTNAME":"dhsumo1-ubuntu-0","_TRANSPORT":"stdout","_STREAM_ID":"1fc1aa8d32304654b4dedcfd82c595e1","SYSLOG_IDENTIFIER":"containerd","_PID":"3381","_UID":"0","_GID":"0","_COMM":"containerd","_EXE":"/usr/bin/containerd","_CMDLINE":"/usr/bin/containerd","_CAP_EFFECTIVE":"3fffffffff","_SYSTEMD_CGROUP":"/system.slice/containerd.service","_SYSTEMD_UNIT":"containerd.service","_SYSTEMD_INVOCATION_ID":"6b881bb9bcad433b9e2f01c229ec195d","MESSAGE":"time=\"2019-05-09T02:40:52.796014949Z\" level=info msg=\"shim reaped\" id=454288d0618791e40195dd7c44514fd7d47e33c35ee4d504a24cbe236afdcb22"} |
|
{"timestamp":1557369759861,"_TRANSPORT":"stdout","_STREAM_ID":"1acb0db4a82e4939bce686ec10fe7acb","PRIORITY":"6","SYSLOG_FACILITY":"3","SYSLOG_IDENTIFIER":"dockerd","_PID":"7665","_UID":"0","_GID":"0","_COMM":"dockerd","_EXE":"/usr/bin/dockerd","_CMDLINE":"/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2376","_CAP_EFFECTIVE":"3fffffffff","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_CGROUP":"/system.slice/docker.service","_SYSTEMD_UNIT":"docker.service","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_INVOCATION_ID":"2f449d0f7cc5447daf4949fb771486f4","_BOOT_ID":"834f3038ccd142e094f31476fa66505a","_MACHINE_ID":"a211db66c10846958dace7213f56ad40","_HOSTNAME":"dhsumo1-ubuntu-3","MESSAGE":"time=\"2019-05-09T02:42:39.861578501Z\" level=error msg=\"Handler for GET /v1.39/swarm/unlockkey returned error: This node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager.\""} |
|
127.0.0.1 - - [09/May/2019:01:29:17 +0000] "GET /health HTTP/1.1" 200 27 "-" "curl/7.61.1" |
|
{"auth.user.name":"b246c77d-4f32-4d7b-a05f-e9d0def0b739","go.version":"go1.11.5","http.request.contenttype":"application/vnd.docker.distribution.manifest.v2+json","http.request.host":"34.222.139.0","http.request.id":"d7c3c77a-c4de-4ef8-a1c5-c58192b70120","http.request.method":"PUT","http.request.remoteaddr":"71.204.129.194","http.request.uri":"/v2/admin/sampleapp/manifests/latest","http.request.useragent":"docker/0.0.0-20190424223053-cfe423a go/go1.11.5 git-commit/cfe423a kernel/4.15.0-47-generic os/linux arch/amd64 UpstreamClient(Docker-Client/0.0.0-20190424223053-cfe423a \\(linux\\))","level":"info","msg":"dispatching manifest put payload","payload":{"namespace":"admin","repository":"sampleapp","digest":"sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4","imageName":"admin/sampleapp@sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4","os":"linux","architecture":"amd64","author":"admin","pushedAt":"2019-05-07T01:58:24.198629261Z"},"time":"2019-05-07T01:58:24.198643304Z","vars.name":"admin/sampleapp","vars.reference":"latest"} |
|
Universal Control Plane (UCP) |
E0509 02:44:15.887502 1 goroutinemap.go:150] Operation for "provision-default/orcl-pv-claim[4e38340b-6866-11e9-95b6-0242ac11000a]" failed. No retries permitted until 2019-05-09 02:46:17.887477418 +0000 UTC m=+453713.589406681 (durationBeforeRetry 2m2s). Error: "AzureDisk - failed to get Azure Cloud Provider. GetCloudProvider returned <nil> instead" |
Query samples
This section provides query samples taken from the Docker EE App dashboards.
Containerd
_sourceCategory=kubernetes/system _sourceName=containerd !"level=debug" | json "MESSAGE", "_HOSTNAME", "_PID", "_SYSTEMD_INVOCATION_ID" as msg, hostname, pid, system_invocation_id | parse regex field=msg "time=\"(?<time>[\S]+)\" level=(?<level>[\S]+) msg=\"(?<message>.*)\""
Docker Trusted Registry (DTR)
_sourceCategory=docker _sourceName = dtr-registry-* (PUT) pushedAt "dispatching manifest put payload" | json "$['auth.user.name']", "$['http.request.host']", "$['http.request.method']", "$['http.request.remoteaddr']", "$['http.request.uri']", "$['http.request.useragent']", "level", "payload.repository", "payload.imageName", "payload.os", "payload.author", "payload.pushedAt" as user_id, host, method, remote_ip , uri, user_agent, level, repository, image_name, os, author, time_pushed_at | count by time_pushed_at, image_name, repository, author, user_id, host, method, remote_ip , uri, level, os
DTR Client Requests
_sourceCategory=docker _sourceName = dtr-nginx* | parse regex "^(?<src_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" nodrop | parse regex "(?<method>[A-Z]+)\s(?<url>\S+)\sHTTP/[\d\.]+\"\s(?<status_code>\d+)\s(?<size>[\d-]+)" nodrop | parse regex "(?<method>[A-Z]+)\s(?<url>\S+)\sHTTP/[\d\.]+\"\s(?<status_code>\d+)\s(?<size>[\d-]+)\s\"(?<referrer>.*?)\"\s\"(?<user_agent>.+?)\".*" nodrop
Dockerd
_sourceCategory=kubernetes/system _sourceName=docker | json "MESSAGE", "_HOSTNAME", "_PID", "_SYSTEMD_INVOCATION_ID" as msg, hostname, pid, system_invocation_id | parse regex field=msg "time=\"(?<time>[\S]+)\" level=(?<level>[\S]+) msg=\"(?<message>.*)\""
Universal Control Plane (UCP)
_sourceCategory=docker _sourceName= ucp-kube-controller-manager (fail* or error or except*) | parse "* * *" as misc,time,msg