Skip to main content
Sumo Logic

Collect logs and metrics for Docker EE

  1. Last updated
  2. Save as PDF
This page has instructions for collecting logs for the Sumo App for Docker EE. This app works in conjunction with the Docker ULM App.

This page has instructions for collecting logs for the Sumo App for Docker EE. This app works in conjunction with the Docker ULM App, and the first step is to configure collection for Docker ULM App. Then, since Docker EE uses Kubernetes for orchestration, you install and deploy the Sumo Logic Fluentd plugin for collection of Docker EE specific components.

Step 1. Collect logs and metrics for the Docker ULM App

This section shows you how to configure log and metric collection for the Docker ULM App, which requires configuring two sources:

  • Docker Logs. Collects stdout/stderr logs from processes that are running within Docker containers.
  • Docker Stats. Collects metrics about Docker containers.

To configure log and metric collection for the Docker ULM App, do the following:

  1. Follow the instructions to Collect Logs and Metrics for Docker ULM.
  1. Follow the instructions to Install the Docker ULM App and dashboards. 

Step 2. Deploy FluentD and FluentBit to collect logs

Follow the instructions in this section to create Sumo Logic fields and collector, then deploy Fluentd and FluentBit on Manager Node.

  1. Before you start

  2. Create Sumo Fields and a collector

  3. Deploy Fluentd

  4. Deploy FluentBit

If you get the following error while executing kubectl, your kube config is not wired correctly to UCP’s kube:

The connection to the server localhost:8080 was refused - did you specify the right host or port?

Do the following:

  1. Log in to the UCP Web UI (dashboard), navigate to your user account, and click My Profile.
  2. Then click Client Bundles > New Client Bundle > Generate Client Bundle.

DockerEE_ClientBundle.png

  1. Unpack the ucp-bundle-XYZ.zip file on the Manager Node/Machine.

unzip ucp-bundle-XYZ.zip

  1. Run the following command.

source env.sh

Assuming you have a Mac/Unix environment, docker and kubectl will work on your cluster as your user ID, as shown in the following example where the user ID is admin.

DockerEE_userid.png

Log types and examples

The Docker EE App uses the following log types.

Log Source Type

Example Log Message

Containerd

{"timestamp":1557369652796,"PRIORITY":"6","SYSLOG_FACILITY":"3","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_SLICE":"system.slice","_BOOT_ID":"27b76ce41e524d5786122a7601a30f01","_MACHINE_ID":"a211db66c10846958dace7213f56ad40","_HOSTNAME":"dhsumo1-ubuntu-0","_TRANSPORT":"stdout","_STREAM_ID":"1fc1aa8d32304654b4dedcfd82c595e1","SYSLOG_IDENTIFIER":"containerd","_PID":"3381","_UID":"0","_GID":"0","_COMM":"containerd","_EXE":"/usr/bin/containerd","_CMDLINE":"/usr/bin/containerd","_CAP_EFFECTIVE":"3fffffffff","_SYSTEMD_CGROUP":"/system.slice/containerd.service","_SYSTEMD_UNIT":"containerd.service","_SYSTEMD_INVOCATION_ID":"6b881bb9bcad433b9e2f01c229ec195d","MESSAGE":"time=\"2019-05-09T02:40:52.796014949Z\" level=info msg=\"shim reaped\" id=454288d0618791e40195dd7c44514fd7d47e33c35ee4d504a24cbe236afdcb22"}


 

Dockerd

{"timestamp":1557369759861,"_TRANSPORT":"stdout","_STREAM_ID":"1acb0db4a82e4939bce686ec10fe7acb","PRIORITY":"6","SYSLOG_FACILITY":"3","SYSLOG_IDENTIFIER":"dockerd","_PID":"7665","_UID":"0","_GID":"0","_COMM":"dockerd","_EXE":"/usr/bin/dockerd","_CMDLINE":"/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2376","_CAP_EFFECTIVE":"3fffffffff","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_CGROUP":"/system.slice/docker.service","_SYSTEMD_UNIT":"docker.service","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_INVOCATION_ID":"2f449d0f7cc5447daf4949fb771486f4","_BOOT_ID":"834f3038ccd142e094f31476fa66505a","_MACHINE_ID":"a211db66c10846958dace7213f56ad40","_HOSTNAME":"dhsumo1-ubuntu-3","MESSAGE":"time=\"2019-05-09T02:42:39.861578501Z\" level=error msg=\"Handler for GET /v1.39/swarm/unlockkey returned error: This node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager.\""}


 

Docker Trusted Registry (DTR) Client Requests

127.0.0.1 - - [09/May/2019:01:29:17 +0000] "GET /health HTTP/1.1" 200 27 "-" "curl/7.61.1"


 

Docker Trusted Registry (DTR)

{"auth.user.name":"b246c77d-4f32-4d7b-a05f-e9d0def0b739","go.version":"go1.11.5","http.request.contenttype":"application/vnd.docker.distribution.manifest.v2+json","http.request.host":"34.222.139.0","http.request.id":"d7c3c77a-c4de-4ef8-a1c5-c58192b70120","http.request.method":"PUT","http.request.remoteaddr":"71.204.129.194","http.request.uri":"/v2/admin/sampleapp/manifests/latest","http.request.useragent":"docker/0.0.0-20190424223053-cfe423a go/go1.11.5 git-commit/cfe423a kernel/4.15.0-47-generic os/linux arch/amd64 UpstreamClient(Docker-Client/0.0.0-20190424223053-cfe423a \\(linux\\))","level":"info","msg":"dispatching manifest put payload","payload":{"namespace":"admin","repository":"sampleapp","digest":"sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4","imageName":"admin/sampleapp@sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4","os":"linux","architecture":"amd64","author":"admin","pushedAt":"2019-05-07T01:58:24.198629261Z"},"time":"2019-05-07T01:58:24.198643304Z","vars.name":"admin/sampleapp","vars.reference":"latest"}


 
Universal Control Plane (UCP)

E0509 02:44:15.887502  1 goroutinemap.go:150] Operation for "provision-default/orcl-pv-claim[4e38340b-6866-11e9-95b6-0242ac11000a]" failed. No retries permitted until 2019-05-09 02:46:17.887477418 +0000 UTC m=+453713.589406681 (durationBeforeRetry 2m2s). Error: "AzureDisk -  failed to get Azure Cloud Provider. GetCloudProvider returned <nil> instead"

Query samples

This section provides query samples taken from the Docker EE App dashboards.

Containerd

_sourceCategory=kubernetes/system _sourceName=containerd !"level=debug"   
| json "MESSAGE", "_HOSTNAME", "_PID", "_SYSTEMD_INVOCATION_ID" as msg, hostname, pid, system_invocation_id
| parse regex field=msg "time=\"(?<time>[\S]+)\" level=(?<level>[\S]+) msg=\"(?<message>.*)\""

Docker Trusted Registry (DTR)

_sourceCategory=docker _sourceName = dtr-registry-* (PUT) pushedAt "dispatching manifest put payload"
| json  "$['auth.user.name']", "$['http.request.host']", "$['http.request.method']", "$['http.request.remoteaddr']", "$['http.request.uri']", "$['http.request.useragent']", "level", "payload.repository", "payload.imageName", "payload.os", "payload.author", "payload.pushedAt" as user_id, host, method, remote_ip , uri, user_agent, level, repository, image_name, os, author, time_pushed_at
| count by time_pushed_at, image_name, repository, author, user_id, host, method, remote_ip , uri, level, os

DTR Client Requests

_sourceCategory=docker _sourceName = dtr-nginx*
| parse regex "^(?<src_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" nodrop
| parse regex "(?<method>[A-Z]+)\s(?<url>\S+)\sHTTP/[\d\.]+\"\s(?<status_code>\d+)\s(?<size>[\d-]+)" nodrop
| parse regex "(?<method>[A-Z]+)\s(?<url>\S+)\sHTTP/[\d\.]+\"\s(?<status_code>\d+)\s(?<size>[\d-]+)\s\"(?<referrer>.*?)\"\s\"(?<user_agent>.+?)\".*" nodrop

Dockerd

_sourceCategory=kubernetes/system _sourceName=docker  
| json "MESSAGE", "_HOSTNAME", "_PID", "_SYSTEMD_INVOCATION_ID" as msg, hostname, pid, system_invocation_id
| parse regex field=msg "time=\"(?<time>[\S]+)\" level=(?<level>[\S]+) msg=\"(?<message>.*)\""

Universal Control Plane (UCP)

_sourceCategory=docker _sourceName= ucp-kube-controller-manager (fail* or error or except*)
| parse "* *       *" as misc,time,msg