Skip to main content
Sumo Logic

Istio OpenTelemetry auto-instrumentation

Learn how to instrument OpenTelemetry Application code to collect Istio trace data generated from Envoy proxies.

OpenTelemetry Collector gives you the possibility to collect Istio trace data and forward it to Sumo Logic.

Istio is a popular service mesh technology using Envoy proxy technology that provides out-of-the-box tracing generation from their sidecars. Although its tracing capabilities cannot fully replace visibility that  you can obtain by collecting traces by instrumenting the application code directly, it can provide some additional details to delays introduced by sidecar proxies that will generate its own specific spans.

Configuration

To send data to Sumo Logic collector either in a Kubernetes or standalone setup, configure the OpenCensus Agent (recommended due to simpler setup) or Zipkin (if for any reason that’s your preference) exporter and send this data to the associated collector port. Enable one of the exporters setting meshConfig options in Envoy proxies.

OpenCensus Agent exporter

Set meshConfig.enableTracing=true and meshConfig.defaultConfig.tracing.openCensusAgent.zipkin.address= to the endpoint of the receiving collector:

  • For Kubernetes collectors:
    --set meshConfig.enableTracing=true
    --set meshConfig.defaultConfig.tracing.openCensusAgent.address=RELEASE_NAME-CHART_NAME-otelcol.NAMESPACE:55678 

  • For standalone collectors:
    --set meshConfig.enableTracing=true
    --set meshConfig.defaultConfig.tracing.openCensusAgent.address=HOSTNAME:55678

Zipkin exporter

Set meshConfig.enableTracing=true and meshConfig.defaultConfig.tracing.zipkin.address= to the URL of Sumo receiving collector:

  • For Kubernetes collectors:
    --set meshConfig.enableTracing=true
    --set meshConfig.defaultConfig.tracing.zipkin.address=RELEASE_NAME-CHART_NAME-otelcol.NAMESPACE:9411 

  • For standalone collectors:
    --set meshConfig.enableTracing=true
    --set meshConfig.defaultConfig.tracing.zipkin.address=HOSTNAME:9411

Reconfigure OpenTelemetry to use B3 - Zipkin exporter

Istio uses an older propagation standard for Zipkin’s exporter. Fortunately though there's an opt-in backward compatibility option to enable b3 format in OT which should be done to connect Istio spans to the rest of the trace.

Without this there will be a lack of context propagation between different services, after enabling span export on Istio.

You can reconfigure OpenTelemetry instrumentation to use b3 in addition to W3C context propagation.

The following is a Java example to set up b3 context propagation. Add the following parameter to JVM environment variables or command line:

OTEL_PROPAGATORS=b3,tracecontext,baggage 

For example:

JAVA_TOOL_OPTIONS="-javaagent:path/to/opentelemetry-javaagent.jar"
OTEL_TRACES_EXPORTER=otlp
OTEL_PROPAGATORS=b3,tracecontext,baggage
OTEL_EXPORTER_OTLP_ENDPOINT=http://OPENTELEMETRY_COLLECTOR_HOSTNAME:4317OTEL_SERVICE_NAME=SERVICE_NAME
OTEL_RESOURCE_ATTRIBUTES=application=YOUR_APP_NAME

Or

java -javaagent:path/to/opentelemetry-javaagent.jar \
    -Dotel.traces.exporter=otlp \
    -Dotel.exporter.otlp.endpoint=http://OPENTELEMETRY_COLLECTOR_HOSTNAME:4317 \
-Dotel.propagators=b3,tracecontext,baggage \
    -Dotel.service.name=SERVICE_NAME
-Dotel.resource.attributes=application=YOUR_APP_NAME \
    …

Distributed Tracing customization

See the Configure tracing using MeshConfig and Pod annotations guide for information to configure any advanced capabilities on span data generation from Istio, such as probabilistic sampling, extending the length of some fields, or adding custom metadata/annotations to spans.

Troubleshooting

  1. Validate if meshConfig options are properly configured. Execute: 
    kubectl get configmap istio –namespace istio-system –output "jsonpath={.data['mesh']}"

    Example output:
    istio1.png
    Check if correct values are set for enableTracing and defaultConfig.tracing.EXPORTER_NAME.address. If OpenCensus was configured, then Zipkin by default points to an Istio endpoint.

  2. Make sure the namespace is labeled. More details about sidecar injection can be found here. To check if namespace is labeled, execute:
    kubectl get namespace -L istio-injection

    istio2.png
    Make sure the namespace in which the application is running istio-injection label is set as enabled. If label value is disabled or not set then run:
    kubectl label namespace NAMESPACE_NAME istio-injection=enabled --overwrite

  3. Check if exporter configuration was used by the sidecar container. To see the logs of istio-proxy container and look for values provided in the meshConfig, execute:
    kubectl logs -n NAMESPACE POD_NAME istio-proxy

    Example output:
    istio3.png