Skip to main content
Sumo Logic

Collect Logs and Stats from Docker

This procedure describes how to collect events and statistics from Docker with Sumo Logic.

You can add the following types of Docker sources to an installed collector:

  • Docker Logs. Collects stdout/stderr logs from processes running in Docker containers.
  • Docker Stats. Collects metrics about Docker containers.

For more information about how Sumo Logic collects information from Docker, see Docker Sources or the Docker API documentation.

Log Types

The Sumo Logic App for Docker gathers log data, statistics, and events from the Docker Remote API on each host.

For more information about Docker events, see Monitor Docker Events

For information about Docker statistics based on resource usage, see Docker Container Stats.

Prerequisites and Requirements

The Sumo collector uses the Docker Remote API to collect Docker logs. The log driver configured on the Docker container must use either the json-file or journald option, as described in the Docker Logging Overview.

To collect events and statistics for the Sumo App for Docker, first create Access Keys, then download and run the Sumo Logic container, which includes the collector and the script source.

Create Access Keys 

Create Access Keys in Sumo to register your collector using the instructions in Access Keys.

Download and Run the Sumo Logic Container

The Sumo container, which includes a collector and a script source, is located in the public Sumo Logic Docker Hub. Download this container using the following procedure, then enter the access keys you created to register the collector.

To download and run the container

  1. On your Docker host, download the Sumo container using the following command:

docker pull sumologic/collector:latest

  1. Run the container with this command, substituting the AccessID and AccessKey that you created previously in Sumo:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock --name="sumologic-docker" sumologic/collector:latest <AccessID> <AccessKey>

The container creates a collector in your Sumo account named collector_container or collector_container-<ARandomString>.

It also creates two sources, a Docker Logs Source and a Docker Stats Source.

Once installed and configured, you can view the collector and its sources in Sumo on the Manage > Collection page.

Configure a Source

To configure a Docker Log or Docker Stats source:

  1. In Sumo select Manage Data > Collection > Collection (Manage > Collection in the classic UI).
  2. Locate the Collector for which you want to add a Docker Source, and select Add > Add Source.
  3. Select Docker Logs or Docker Stats.
  4. Configure the source fields:
    1. Name. (Required) 
    2. Description. (Optional)
    3. URI. Enter the URI of the Docker daemon.
      • Same host (typically applies for Linux hosts). If your collector runs on the same host as Docker containers, use the non-networked Unix socket:
        unix:///var/run/docker.sock
      • Remote access (typically applies for hosts on Mac or Windows where the Docker process runs in a VM and the collector runs outside of the Docker host). Run the docker-machine command to find the Docker environment variables.
        $ docker-machine env <machine-name>

      Example:

      $ docker-machine env default
      export DOCKER_TLS_VERIFY="1"
      export DOCKER_HOST="tcp://192.168.99.100:2376"
      export DOCKER_CERT_PATH="/Users/sumo/.docker/machine/machines/default"
      export DOCKER_MACHINE_NAME="default"
      # Run this command to configure your shell: 
      # eval "$(docker-machine env default)"

      ​​Change "tcp" to "https" in the DOCKER_HOST environment variable, for example,

      https://192.168.99.100:2376
      
    4. Cert Path. (Required for remote access only) Enter the path to the certificate files on the local machine where the collector runs. In the example above, the cert path is: /Users/sumo/.docker/machine/machines/default
    5. Event Logs. (For Docker Stats sources only) Choose this option to collect events from all containers, even if you do not collect logs from all containers.
    6. Collect From and Container Filters. If you want to collect from all containers, click the All Containers radio button. If you want to collect from selected containers, click the Specified Container Filters radio button, and specify filter expressions in the Container Filters field. For information about how to define container filters, see Container Filter Definitions below.
      • By default, you can collect from up to 40 containers.To increase the limit, edit the collector.properties file and change the value of docker.maxPerContainerConnections. The maximum supported value is 100.
    7. Source Host. Enter the hostname or IP address of the source host. If not specified, it’s assumed that the host is the machine where Docker is running. The hostname can be a maximum of 128 characters.
    8. Source Category. (Required) Enter the Sumo source category (such as prod/web/docker/stats). The source category metadata field is a fundamental building block to organize and label sources. For details see Best Practices.
    9. Scan Interval. (For Docker Stats sources only) This option sets how often the source is scanned. Setting a shorter frequency increases message volume, and can cause your deployment to incur additional charges.
  5. Configure the Advanced options (For Docker Log sources only):
    1. Enable Timestamp Parsing. True by default.
    2. Time Zone. By default, Use time zone from collector.
    3. Timestamp Format. Automatically detect the format by default.
    4. Encoding. UTF-8 by default.
    5. Enable Multiline Processing.
      • Detect Messages Spanning Multiple Lines. False
      • Infer Boundaries - Detect message boundaries automatically. False
      • Boundary Regex. None
  6. Click Save.

Container Filter Definitions

In the Container Filter field, you can enter a comma-separated list of one or more of the following types of filters:

  • A specific container name, for example, “my-container”
  • A wildcard filter, for example, “my-container-*”
  • An exclusion (blacklist) filter, which begins with an exclamation mark, for example, ”!master-container” or “!prod-*”

For example, this filter list:

prod-*, !prod-*-mysql, master-*-app-*, sumologic-collector

will cause the source to collect from all containers whose names start with “prod-”, except those that match “prod-*-mysql”. It will also collect from containers with names that match “master-*-app-*”, and from the “sumologic-collector” container.

If your filter list contains only exclusions, the source will collect all containers except from those that match your exclusion filters. For example:

!container123*, !prod-*

will cause the source to exclude containers whose names begin with “container123” and “prod-”

Sample Log Messages

{"status":"start", "id":"10adec58fa15202e06afef7b1b0b3b1464962a115ff56918444c3f22867d3f3b", "from":"hello-world", "time":1485975967}
{"status":"create", "id":"045599bc4d589264658f5f7f4efa3f1e3af9088ba1f7383a160cf344e1055d46", "from":"ubuntu", "time":1485966852}
{"read" : "2017-02-01T19:36:48.777487188Z", "network" : {"rx_bytes":87977,"rx_dropped":0,"rx_errors":0,"rx_packets":252,"tx_bytes":146194,"tx_dropped":0,"tx_errors":0,"tx_packets":302}, "cpu_stats" : {"cpu_usage":{"percpu_usage":[9469809313],"total_usage":9469809313,"usage_in_kernelmode":1050000000,"usage_in_usermode":8410000000},"system_cpu_usage":2496992710000000,"throttling_data":{"periods":0,"throttled_periods":0,"throttled_time":0}}, "blkio_stats" : {"io_merged_recursive":[],"io_queue_recursive":[],"io_service_bytes_recursive":[],"io_service_time_recursive":[],"io_serviced_recursive":[],"io_time_recursive":[],"io_wait_time_recursive":[],"sectors_recursive":[]}, "memory_stats" : {"limit":1033252864,"max_usage":202858496,"stats":{"active_anon":86831104,"active_file":13131776,"cache":24981504,"dirty":36864,"hierarchical_memory_limit":9223372036854771712,"inactive_anon":86786048,"inactive_file":11849728,"mapped_file":6430720,"pgfault":63351,"pgmajfault":146,"pgpgin":68526,"pgpgout":20040,"rss":173617152,"rss_huge":0,"total_active_anon":86831104,"total_active_file":13131776,"total_cache":24981504,"total_dirty":36864,"total_inactive_anon":86786048,"total_inactive_file":11849728,"total_mapped_file":6430720,"total_pgfault":63351,"total_pgmajfault":146,"total_pgpgin":68526,"total_pgpgout":20040,"total_rss":173617152,"total_rss_huge":0,"total_unevictable":0,"total_writeback":0,"unevictable":0,"writeback":0},"usage":201818112}}

Query Sample

Containers Created or Started

_sourceCategory=docker  ("\"status\":\"create\"" or "\"status\":\"start\"")  id from
| parse "\"status\":\"*\"" as status, "\"id\":\"*\"" as container_id, "\"from\":\"*\"" as image
| count_distinct(container_id)

Sumo Logic App

Now that you have set up collection for Docker, install the Sumo Logic App for Docker. The Sumo Logic App for Docker provides operational insight into your Docker containers. The App includes Dashboards that allow you to view your container performance statistics for CPU, memory, and the network. It also provides visibility into container events such as start, stop, and other important commands.