Skip to main content
Sumo Logic

Collect Logs and Stats from Docker

Sumo supports multiple methods of collecting data from Docker. This procedure describes how to collect data from Docker using an installed collector and Sumo’s Docker logs source and Docker stats source. 

With the method described in this topic you can collect Docker logs, stats, and events and view summaries of the data collected using the Sumo App for Docker. 

What Docker data Sumo collects and how

Sumo’s Docker logs source and Docker stats source use the Docker Engine API to gather the following data from Docker:

  • Docker container logs. Sumo’s Docker logs source collects container logs. For information about the API Sumo uses to collect logs, see the “Get Container Logs” topic in Docker API documentation
  • Docker engine events. Sumo’s Dockers log source collect Docker events. For information about Docker events, see the “Monitor Events” in Docker API documentation.
  • Docker container stats. Sumo’s Docker stats source collects stats. For information about Docker stats, see the ”Get Container Stats Based on Resource Usage” topic in Docker API documentation.

By default, you can monitor up to 40 Docker containers on a Docker host. If you want to monitor more than 40 containers on a given host you can configure a larger number in collector.properties. The procedures below explain how. We don’t support monitoring more than 100 containers on a Docker host.

Prerequisites 

Sumo's Docker sources support Docker API 1.21 or earlier. Docker API 1.22 or later is not supported.

The containers you’re going to monitor must use either the json-file or the journald driver. For more information, see Configure Logging Drivers in Docker help.

Process Overview

The sections below provide instructions for installing a collector on a Docker host, setting up both Sumo Docker sources (log source and stats source), and installing the Sumo app for Docker. With this configuration you can collect Docker logs, events, and stats, and visualize resource performance and event data in the dashboards provided by the app. 

Step 1: Create access keys

Follow the instructions in Access Keys to create a Sumo access ID and key. You’ll need to supply them when you set up a collector on your Docker host in the following step. 

Step 2: Install collector on each Docker host

Follow the instructions for the operating system of the host where you will install the collector: 

Step 3: Configure Docker log source

  1. In the Sumo web app, select Manage Data > Collection > Collection (Manage > Collection in the classic UI).
  2. Navigate to the collector you installed on the Docker host, and select Add > Add Source.
  3. Select Docker Logs.  The Docker Logs page appears.
  4. Configure the source fields:
    1. Name. (Required).
    2. Description. (Optional).
    3. URI. Enter the URI of the Docker daemon.
      • If your collector runs on the same host as the Docker containers it will monitor, enter the non-networked Unix socket:
        unix:///var/run/docker.sock
      • If your collector runs on a different machine than the Docker host, you can determine its URI from a Docker environment variable. Run the docker-machine command to find the Docker environment variables.
        $ docker-machine env machine-name

        For example:
      • $ docker-machine env default
        export DOCKER_TLS_VERIFY="1"
        export DOCKER_HOST="tcp://192.168.99.100:2376"
        export DOCKER_CERT_PATH="/Users/sumo/.docker/machine/machines/default"
        export DOCKER_MACHINE_NAME="default"
        # Run this command to configure your shell: 
        # eval "$(docker-machine env default)"
        

        Take the value of the DOCKER_HOST variable, change "tcp" to "https", and enter that value as the URI. For example,

        https://192.168.99.100:2376
    4. Cert Path. (Required for remote access only) Enter the path to the certificate files on the local machine where the collector runs. In the example above, the cert path is: /Users/sumo/.docker/machine/machines/default.
    5. Collect From and Container Filters. If you want to collect from all containers, click the All Containers radio button. If you want to collect from selected containers, click the Specified Container Filters radio button, and specify filter expressions in the Container Filters field. For information about how to define container filters, see More about defining container filters below.
      • By default, you can collect from up to 40 containers. To increase the limit, edit the collector.properties file (in the config subdirectory of the collector installation directory), and add the docker.maxPerContainerConnections property. The maximum supported value is 100.
    6. Source Host. Enter the hostname or IP address of the source host. If not specified, it’s assumed that the host is the machine where Docker is running. The hostname can be a maximum of 128 characters.
    7. Source Category. (Required) Enter the Sumo source category (such as prod/web/docker/logs). The source category metadata field is a fundamental building block to organize and label sources. For details see Best Practices.
  5. Configure the Advanced options.
    1. Enable Timestamp Parsing. This option is checked by default.
    2. Time Zone. Default is “Use time zone from log file”.
    3. Timestamp Format. Default is “Automatically detect the format”.
    4. Encoding. Default is “UTF-8”.
    5. Enable Multiline Processing. 
      • Detect messages spanning multiple lines. This option is checked by default.
      • Infer Boundaries. This option is checked by default.
      • Boundary Regex. If multiple processing is enabled, and Infer Boundaries is disabled, enter a regular expression for message boundaries.
  6. Configure processing rules. For more information, see Processing Rules.

Step 4: Configure Docker stats source

  1. In Sumo select Manage Data > Collection > Collection (Manage > Collection in the classic UI).
  2. Navigate to the collector you installed on the Docker host, and select Add > Add Source.
  3. Select Docker Stats. The Docker Stats page appears. docker stats source
  4. Configure the source fields:
    1. Name. (Required)
    2. Description. (Optional)
    3. URI. Enter the URI of the Docker daemon.
      • If your collector runs on the same host as the Docker containers it will monitor, enter the non-networked Unix socket:
        unix:///var/run/docker.sock
      • If your collector runs on a different machine than the Docker host, you can determine its URI from a Docker environment variable. Run the docker-machine command to find the Docker environment variables.
        $ docker-machine env machine-name

        For example:
      • $ docker-machine env default
        export DOCKER_TLS_VERIFY="1"
        export DOCKER_HOST="tcp://192.168.99.100:2376"
        export DOCKER_CERT_PATH="/Users/sumo/.docker/machine/machines/default"
        export DOCKER_MACHINE_NAME="default"
        # Run this command to configure your shell: 
        # eval "$(docker-machine env default)"
        

        Take the value of the DOCKER_HOST variable, change "tcp" to "https", and enter that value as the URI. For example,

        https://192.168.99.100:2376
    4. Cert Path. (Required for remote access only) Enter the path to the certificate files on the local machine where the collector runs. In the example above, the cert path is: /Users/sumo/.docker/machine/machines/default
    5. Collect From and Container Filters. If you want to collect from all containers, click the All Containers radio button. If you want to collect from selected containers, click the Specified Container Filters radio button, and specify filter expressions in the Container Filters field. For information about how to define container filters, see More about defining container filters below.
      • By default, you can collect from up to 40 containers. To increase the limit, edit the collector.properties file (in the config subdirectory of the collector installation directory), and add the docker.maxPerContainerConnections property. The maximum supported value is 100.
    6. Source Host. Enter the hostname or IP address of the source host. If not specified, it’s assumed that the host is the machine where Docker is running. The hostname can be a maximum of 128 characters.
    7. Source Category. (Required) Enter the Sumo source category (such as prod/web/docker/stats). The source category metadata field is a fundamental building block to organize and label sources. For details see Best Practices.
    8. Scan Interval. This option sets how often the source is scanned. Setting a shorter frequency increases message volume, and can cause your deployment to incur additional charges

Step 5: Install Sumo app for Docker

The Sumo App for Docker provides operational insight into your Docker containers. The app includes Dashboards that allow you to view your container performance statistics for CPU, memory, and the network. It also provides visibility into container events such as start, stop, and other important commands.  

For installation instructions, see Install the Docker App.

Step 6:  Run searches and use dashboards

At this point, Sumo should be receiving Docker data. For an example of logs collected from Docker, see Sample Docker log messages. For an example query, see  Sample query - containers created or started.  

For information about the dashboards provided by the Sumo App for Docker, see Docker App Dashboards.

More about defining container filters 

In the Container Filter field, you can enter a comma-separated list of one or more of the following types of filters:

  • A specific container name, for example, “my-container”
  • A wildcard filter, for example, “my-container-*”
  • An exclusion (blacklist) filter, which begins with an exclamation mark, for example, ”!master-container” or “!prod-*”

For example, this filter list:

prod-*, !prod-*-mysql, master-*-app-*, sumologic-collector

will cause the source to collect from all containers whose names start with “prod-”, except those that match “prod-*-mysql”. It will also collect from containers with names that match “master-*-app-*”, and from the “sumologic-collector” container.

If your filter list contains only exclusions, the source will collect all containers except from those that match your exclusion filters. For example:

!container123*, !prod-*

will cause the source to exclude containers whose names begin with “container123” and “prod-”.

Sample Docker messages 

This is an example of two Docker event logs:

{"status":"start", "id":"10adec58fa15202e06afef7b1b0b3b1464962a115ff56918444c3f22867d3f3b", "from":"hello-world", "time":1485975967}

{"status":"create", "id":"045599bc4d589264658f5f7f4efa3f1e3af9088ba1f7383a160cf344e1055d46", "from":"ubuntu", "time":1485966852}

This is an example of a Docker stats message:

{"read" : "2017-02-01T19:36:48.777487188Z", "network" : {"rx_bytes":87977,"rx_dropped":0,"rx_errors":0,"rx_packets":252,"tx_bytes":146194,"tx_dropped":0,"tx_errors":0,"tx_packets":302}, "cpu_stats" : {"cpu_usage":{"percpu_usage":[9469809313],"total_usage":9469809313,"usage_in_kernelmode":1050000000,"usage_in_usermode":8410000000},"system_cpu_usage":2496992710000000,"throttling_data":{"periods":0,"throttled_periods":0,"throttled_time":0}}, "blkio_stats" : {"io_merged_recursive":[],"io_queue_recursive":[],"io_service_bytes_recursive":[],"io_service_time_recursive":[],"io_serviced_recursive":[],"io_time_recursive":[],"io_wait_time_recursive":[],"sectors_recursive":[]}, "memory_stats" : {"limit":1033252864,"max_usage":202858496,"stats":{"active_anon":86831104,"active_file":13131776,"cache":24981504,"dirty":36864,"hierarchical_memory_limit":9223372036854771712,"inactive_anon":86786048,"inactive_file":11849728,"mapped_file":6430720,"pgfault":63351,"pgmajfault":146,"pgpgin":68526,"pgpgout":20040,"rss":173617152,"rss_huge":0,"total_active_anon":86831104,"total_active_file":13131776,"total_cache":24981504,"total_dirty":36864,"total_inactive_anon":86786048,"total_inactive_file":11849728,"total_mapped_file":6430720,"total_pgfault":63351,"total_pgmajfault":146,"total_pgpgin":68526,"total_pgpgout":20040,"total_rss":173617152,"total_rss_huge":0,"total_unevictable":0,"total_writeback":0,"unevictable":0,"writeback":0},"usage":201818112}}

Sample query - containers created or started

_sourceCategory=docker  ("\"status\":\"create\"" or "\"status\":\"start\"")  id from
| parse "\"status\":\"*\"" as status, "\"id\":\"*\"" as container_id, "\"from\":\"*\"" as image
| count_distinct(container_id)