Skip to main content
Sumo Logic

Collect logs and metrics for Docker EE

  1. Last updated
  2. Save as PDF
This page has instructions for collecting logs for the Sumo App for Docker EE. This app works in conjunction with the Docker ULM App.

This page has instructions for collecting logs for the Sumo App for Docker EE. This app works in conjunction with the Docker ULM App, and the first step is to configure collection for Docker ULM App. Then, since Docker EE uses Kubernetes for orchestration, you install and deploy the Sumo Logic Fluentd plugin for collection of Docker EE specific components.

Step 1. Collect logs and metrics for the Docker ULM App

This section shows you how to configure log and metric collection for the Docker ULM App, which requires configuring two sources:

  • Docker Logs. Collects stdout/stderr logs from processes that are running within Docker containers.
  • Docker Stats. Collects metrics about Docker containers.

To configure log and metric collection for the Docker ULM App, do the following:

  1. Follow the instructions to Collect Logs and Metrics for Docker ULM.
  1. Follow the instructions to Install the Docker ULM App and dashboards. 

Step 2. Create a hosted collector and Kubernetes secret

In order to collect logs from the Docker EE platform, you must create a hosted collector and Kubernetes secret, as described in this section. Then, install the Sumo Logic FluentD plugin, as described in Step 3.

To create a hosted collector and Kubernetes secret, do the following:

  1. Follow Step 1 in this document to create a hosted collector in Sumo Logic and make note of the URL.
  2. Follow Step 2 in this document to create a Kubernetes secret with the HTTP source URL, as shown in the following example.
kubectl create secret generic sumologic --from-literal=collector-url=https://collectors.sumologic.com/receiver/v1/http/ZaVnC4dhaVhtt4b6bub28B94zA1Ps_EXAMPLE-KUBERNETES-SECRET_m11OVhS8VwZWh5oJ52OAi089BeG0tq-aQ05gBU7bu6KO_zwrToeg==

Troubleshooting: If you get the following error while creating the Kubernetes secret, your kube config is not wired correctly to UCP’s kube:

The connection to the server localhost:8080 was refused - did you specify the right host or port?

Then do the following:

  1. Log in to the UCP Web UI (dashboard), navigate to your user account, and click My Profile.
  2. Then click Client Bundles > New Client Bundle > Generate Client Bundle.

DockerEE_Collection_ClientBundles_dialog.png

  1. Unpack the ucp-bundle-ankit.zip file on the Manager Node/Machine.
unzip ucp-bundle-ankit.zip
  1. Run the following command.
source env.sh

Assuming you have a Mac/Unix environment, docker and kubectl will work on your cluster as your user ID, as shown in the following example where the user ID is admin.

DockerEE_docker_user-ID_example.png

Step 3. Install the Fluentd plugin

This section shows you how to install the Fluentd plugin. This task assumes you have a working knowledge of github and are comfortable editing the fluentd.yaml file with an ASCII text editor.

  • If you use RBAC, the fluentd.yaml is located at: daemonset/nonrbac/fluentd.yaml
  • For non RBAC, the fluentd.yaml file is located: daemonset/nonrbac/fluentd.yaml

To install the Fluentd plugin, do the following:

  1. Clone the SumoLogic/fluentd-kubernetes-sumologic github repo.
  2. Open the fluentd.yaml file in an ASCII text editor.
  3. Add the FLUENTD_SOURCE environment variable and set to systemd as shown in the following example.

DockerEE_Collection_example-yaml-file.png

This is a link to a sample fluentd.yaml file with an RBAC cluster.

  1. Deploy fluentd to the manager node using the following command.
 kubectl create -f ./your/path/to/fluentd.yaml

Log types and examples

The Docker EE App uses the following log types.

Log Source Type

Example Log Message

Containerd

{"timestamp":1557369652796,"PRIORITY":"6","SYSLOG_FACILITY":"3","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_SLICE":"system.slice","_BOOT_ID":"27b76ce41e524d5786122a7601a30f01","_MACHINE_ID":"a211db66c10846958dace7213f56ad40","_HOSTNAME":"dhsumo1-ubuntu-0","_TRANSPORT":"stdout","_STREAM_ID":"1fc1aa8d32304654b4dedcfd82c595e1","SYSLOG_IDENTIFIER":"containerd","_PID":"3381","_UID":"0","_GID":"0","_COMM":"containerd","_EXE":"/usr/bin/containerd","_CMDLINE":"/usr/bin/containerd","_CAP_EFFECTIVE":"3fffffffff","_SYSTEMD_CGROUP":"/system.slice/containerd.service","_SYSTEMD_UNIT":"containerd.service","_SYSTEMD_INVOCATION_ID":"6b881bb9bcad433b9e2f01c229ec195d","MESSAGE":"time=\"2019-05-09T02:40:52.796014949Z\" level=info msg=\"shim reaped\" id=454288d0618791e40195dd7c44514fd7d47e33c35ee4d504a24cbe236afdcb22"}


 

Dockerd

{"timestamp":1557369759861,"_TRANSPORT":"stdout","_STREAM_ID":"1acb0db4a82e4939bce686ec10fe7acb","PRIORITY":"6","SYSLOG_FACILITY":"3","SYSLOG_IDENTIFIER":"dockerd","_PID":"7665","_UID":"0","_GID":"0","_COMM":"dockerd","_EXE":"/usr/bin/dockerd","_CMDLINE":"/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2376","_CAP_EFFECTIVE":"3fffffffff","_SELINUX_CONTEXT":"unconfined\n","_SYSTEMD_CGROUP":"/system.slice/docker.service","_SYSTEMD_UNIT":"docker.service","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_INVOCATION_ID":"2f449d0f7cc5447daf4949fb771486f4","_BOOT_ID":"834f3038ccd142e094f31476fa66505a","_MACHINE_ID":"a211db66c10846958dace7213f56ad40","_HOSTNAME":"dhsumo1-ubuntu-3","MESSAGE":"time=\"2019-05-09T02:42:39.861578501Z\" level=error msg=\"Handler for GET /v1.39/swarm/unlockkey returned error: This node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager.\""}


 

Docker Trusted Registry (DTR) Client Requests

127.0.0.1 - - [09/May/2019:01:29:17 +0000] "GET /health HTTP/1.1" 200 27 "-" "curl/7.61.1"


 

Docker Trusted Registry (DTR)

{"auth.user.name":"b246c77d-4f32-4d7b-a05f-e9d0def0b739","go.version":"go1.11.5","http.request.contenttype":"application/vnd.docker.distribution.manifest.v2+json","http.request.host":"34.222.139.0","http.request.id":"d7c3c77a-c4de-4ef8-a1c5-c58192b70120","http.request.method":"PUT","http.request.remoteaddr":"71.204.129.194","http.request.uri":"/v2/admin/sampleapp/manifests/latest","http.request.useragent":"docker/0.0.0-20190424223053-cfe423a go/go1.11.5 git-commit/cfe423a kernel/4.15.0-47-generic os/linux arch/amd64 UpstreamClient(Docker-Client/0.0.0-20190424223053-cfe423a \\(linux\\))","level":"info","msg":"dispatching manifest put payload","payload":{"namespace":"admin","repository":"sampleapp","digest":"sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4","imageName":"admin/sampleapp@sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4","os":"linux","architecture":"amd64","author":"admin","pushedAt":"2019-05-07T01:58:24.198629261Z"},"time":"2019-05-07T01:58:24.198643304Z","vars.name":"admin/sampleapp","vars.reference":"latest"}


 
Universal Control Plane (UCP)

E0509 02:44:15.887502  1 goroutinemap.go:150] Operation for "provision-default/orcl-pv-claim[4e38340b-6866-11e9-95b6-0242ac11000a]" failed. No retries permitted until 2019-05-09 02:46:17.887477418 +0000 UTC m=+453713.589406681 (durationBeforeRetry 2m2s). Error: "AzureDisk -  failed to get Azure Cloud Provider. GetCloudProvider returned <nil> instead"

Query samples

This section provides query samples taken from the Docker EE App dashboards.

Containerd

_sourceCategory=kubernetes/system _sourceName=containerd !"level=debug"   
| json "MESSAGE", "_HOSTNAME", "_PID", "_SYSTEMD_INVOCATION_ID" as msg, hostname, pid, system_invocation_id
| parse regex field=msg "time=\"(?<time>[\S]+)\" level=(?<level>[\S]+) msg=\"(?<message>.*)\""

Docker Trusted Registry (DTR)

_sourceCategory=docker _sourceName = dtr-registry-* (PUT) pushedAt "dispatching manifest put payload"
| json  "$['auth.user.name']", "$['http.request.host']", "$['http.request.method']", "$['http.request.remoteaddr']", "$['http.request.uri']", "$['http.request.useragent']", "level", "payload.repository", "payload.imageName", "payload.os", "payload.author", "payload.pushedAt" as user_id, host, method, remote_ip , uri, user_agent, level, repository, image_name, os, author, time_pushed_at
| count by time_pushed_at, image_name, repository, author, user_id, host, method, remote_ip , uri, level, os

DTR Client Requests

_sourceCategory=docker _sourceName = dtr-nginx*
| parse regex "^(?<src_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" nodrop
| parse regex "(?<method>[A-Z]+)\s(?<url>\S+)\sHTTP/[\d\.]+\"\s(?<status_code>\d+)\s(?<size>[\d-]+)" nodrop
| parse regex "(?<method>[A-Z]+)\s(?<url>\S+)\sHTTP/[\d\.]+\"\s(?<status_code>\d+)\s(?<size>[\d-]+)\s\"(?<referrer>.*?)\"\s\"(?<user_agent>.+?)\".*" nodrop

Dockerd

_sourceCategory=kubernetes/system _sourceName=docker  
| json "MESSAGE", "_HOSTNAME", "_PID", "_SYSTEMD_INVOCATION_ID" as msg, hostname, pid, system_invocation_id
| parse regex field=msg "time=\"(?<time>[\S]+)\" level=(?<level>[\S]+) msg=\"(?<message>.*)\""

Universal Control Plane (UCP)

_sourceCategory=docker _sourceName= ucp-kube-controller-manager (fail* or error or except*)
| parse "* *       *" as misc,time,msg