Skip to main content
Sumo Logic

Set up traces collection for other environments

Sumo Logic leverages OpenTelemetry standards for trace collection. To gather the spans generated by tracing clients, see Instrument your application with OpenTelemetry, the OpenTelemetry Collector instance needs to be run. It receives and buffers data, optionally attaches metadata tags, and pushes them to a Sumo Logic Hosted Collector endpoint, residing in the cloud.

Preparation

Depending on the architecture of the environment, OpenTelemetry Collector can be run in two types of deployment scenarios:

  • a gateway that retrieves all data, processes it and sends it to the backend (Sumo Logic Receiver)
  • an optional (though recommended) Agent, which can be run on each node or as a sidecar next to the service; it buffers and tags spans, sending them to the gateway

Due to the fact that vital metadata is available only with Installed Collectors, we recommend that instances of OpenTelemetry Collector should be run at least on each of the nodes as an agent. Doing so allows you to collect metadata tags locally and to buffer the messages, reducing the number of requests to the collector instance.

OpenTelemetry Deployment

Installation steps for OpenTelemetry Collector Gateway

Step 1 - generating tracing endpoint URL

Installation steps require you to add a Source to a Hosted Collector. Before creating the Source, identify the Hosted Collector you want to use or create a new Hosted Collector. For instructions, see Configure a Hosted Collector.

  1. In the Sumo Logic web app, select Manage Data > Collection > Collection.
  2. In the Collectors page, click Add Source next to a Hosted Collector.

  3. Select HTTP Traces from the available options.

    http traces icon bd.png

  4. Provide a name and click Save.

    HTTP traces set up bd.png

  5. An HTTP address is presented. Save it, you'll need to put it into your OpenTelemetry Collector configuration.

    http endpoint bd.png

Step 2 - prepare config file

config.yaml for OpenTelemetry Collector Gateway

Use the sample gateway configuration template available at GitHub and apply the following changes:

  • ENDPOINT_URL needs to be replaced with the value retrieved in Step 1, point 5.

Please save the file as config.yaml

receivers:
  zipkin:
    endpoint: 0.0.0.0:9411
  otlp:
    protocols:
      grpc: 
        endpoint: 0.0.0.0:55680
      http: 
        endpoint: 0.0.0.0:55681

  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_compact:
        endpoint: 0.0.0.0:6831
      thrift_http:
        endpoint: 0.0.0.0:14268
processors:
  memory_limiter:
    check_interval: 5s
    limit_mib: 1000
    spike_limit_mib: 500
  batch:
    send_batch_size: 256
    send_batch_max_size: 512
    timeout: 5s
  resourcedetection/aws:
    detectors: [ec2]
    timeout: 5s
    override: false
  resource/aws:
    attributes:
    - action: upsert
      key: cloud.namespace
      value: ec2
  resourcedetection/gcp:
    detectors: [gcp]
    timeout: 5s
    override: false
  resource/gcp:
    attributes:
    - action: upsert
      key: cloud.namespace
      value: gce
  queued_retry:
    num_workers: 16
    queue_size: 5000
    retry_on_failure: true
extensions:
  health_check: {}
exporters:
  zipkin:
    endpoint: ENDPOINT_URL
  logging:
    loglevel: debug
service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [jaeger, zipkin, otlp]
      # For AWS EC2 environment use this:
      processors: [memory_limiter, batch, resourcedetection/aws, resource/aws, queued_retry]
      # For GCP Compute Engine environment use this:
      processors: [memory_limiter, batch, resourcedetection/gcp, resource/gcp, queued_retry]
 # For other environments use this:
      processors: [memory_limiter, batch, queued_retry]
      # To enable verbose debugging, add “,logging” to the list of exporters
      exporters: [zipkin]

Step 3 - prepare the OpenTelemetry Collector binaries and run them

As spans are sent over TCP or UDP connections, the OpenTelemetry Collector service needs to have relevant ports exposed. The templates provided above support the following protocols:

  • Zipkin/HTTP on port 9411
  • Jaeger/gRPC on port 14250
  • Jaeger/HTTP on port 14268
  • Jaeger/Compact on port 6831 (UDP)
  • OTLP/gRPC on port 4317 (temporarily, also port 55680)
  • OTLP/HTTP on port 55681
Option I: Containerized environments

A published Docker image could be used: public.ecr.aws/sumologic/opentelemetry-collector:0.19.2-sumo 

For example:

docker run --rm -p 4317:4317 -p 55680:55680 -p 55681:55681 -p 9411:9411 \
  -p 6831:6831/udp -p 14250:14250 -p 14268:14268 \
  -v "${PWD}/config.yaml":/conf/config.yaml \
  public.ecr.aws/sumologic/opentelemetry-collector:0.19.2-sumo \

  --config /conf/config.yaml 

Option II: Non-containerized environments

For non-containerized environments, a binary can be downloaded from the project repository available under https://github.com/SumoLogic/opentelemetry-collector-contrib/releases/ and run on a given node with the appropriate config, such as:

otelcontribcol_linux_amd64 --config config.yaml

Step 4 - connecting clients to OpenTelemetry Collector Gateway (if Agent is not used)

During instrumentation of your application, typically you need to configure collector endpoint and protocol.

Tracing client libraries need to point spans accordingly to the protocol used. When a direct connection to OpenTelemetry Collector Gateway is used, the following settings can be used:

  • Jaeger GRPC: HOSTNAME:14250
  • Jaeger Thrift HTTP: HOSTNAME:14268
  • Jaeger Thrift Compact (UDP): HOSTNAME:6831
  • Zipkin: HOSTNAME:9411/api/v2/traces
  • OTLP (gRPC): HOSTNAME:4317
  • OTLP (HTTP): HOSTNAME:55681/v1/trace

Replace HOSTNAME with the host where the OpenTelemetry Collector/Agent is accessible. For example, this might be localhost or another address on the local node, depending on how the Agent was deployed.

Installation steps for OpenTelemetry Collector Agent

With OpenTelemetry Collector Gateway running, it is not possible to run the agents, to which clients connect locally and agents buffer, tag, and pass the data to the gateway. There are several ways to run the agent. It can be a sidecar, running together with the process in the same container. Or it could be another service deployed on each of the nodes. 

Step 1 - Prepare config file

Please use the template config file available at GitHub and make the following change (and others, as required):

  • HOSTNAME needs to be replaced with the address of the OpenTelemetry Collector Gateway installed in the previous section.

Please save the file as agent-config.yaml

Step 2 - prepare the OpenTelemetry Collector binaries and run them

OpenTelemetry Collector is using the same binary regardless if it's running in Gateway or Agent mode.

Option I: Containerized environments 

A published Docker image could be used: public.ecr.aws/sumologic/opentelemetry-collector:0.19.2-sumo 

For example:

docker run --rm -p 4317:4317 -p 55680:55680 -p 55681:55681 -p 9411:9411 \
  -p 6831:6831/udp -p 14250:14250 -p 14268:14268 \
  -v "${PWD}/config.yaml":/conf/config.yaml \
  public.ecr.aws/sumologic/opentelemetry-collector:0.19.2-sumo \

  --config /conf/agent-config.yaml 

Option II: Non-containerized environments 

For non-containerized environments, a binary can be downloaded from the project repository available under https://github.com/SumoLogic/opentelemetry-collector-contrib/releases/ and run on a given node with the appropriate config, such as:

otelcontribcol_linux_amd64 --config agent-config.yaml

Step 3 - connecting clients to OpenTelemetry Collector Agent

During instrumentation of your application, typically you need to configure collector endpoint and protocol.

Tracing client libraries need to point spans accordingly to the protocol used:

  • Jaeger GRPC: AGENT_HOSTNAME:14250
  • Jaeger Thrift HTTP: AGENT_HOSTNAME:14268
  • Jaeger Thrift Compact (UDP): AGENT_HOSTNAME:6831
  • Zipkin: AGENT_HOSTNAME:9411/api/v2/traces
  • OTLP (gRPC): AGENT_HOSTNAME:4317
  • OTLP (HTTP): AGENT_HOSTNAME:55681/v1/trace

Please fill AGENT_HOSTNAME with the actual address of the sidecar/instance of OpenTelemetry Collector Agent on a given node (run in the previous step). Frequently this will be localhost.