Skip to main content
Sumo Logic

Traces

Traces provide transactional intelligence for distributed workflows, by combining telemetry from traces, logs, and metrics in the context of real time automatically tracked application topology.

Trace data is visualized through filtered trace lists and icicle charts allowing you to find and troubleshoot faulty transactions easily.

Trace Collection

Traces are collected with SumoLogic Kubernetes Collection or a standalone OpenTelemetry collector through an HTTP Traces Source.

See Getting Started with Transaction Tracing for details on how to set up your collection environment.

Traces

Use Traces to search and view traces.

To open go to + New > Traces.

traces menu option.png

A new Traces page opens:

trace page with 200 status.png

The Traces page displays all the traces in your account, up to the 1,000 limit, by default. You can view traces in a Table.

Table

Click on any row to open the Trace View. Traces are displayed in the following columns:

Column name Example value Description
  #A Indicates the query returning the trace.
Trace ID ffaf2f69ee8ad0c1 The unique identifier of the trace.
Root Service api The service that started the trace.
Started At 07/27/2020 09:01:04.533 When the trace started.
Duration 12.582 ms The amount of time the trace spans. 
Number of spans 35 A trace consists of spans. This number tells you how many spans are in the trace.
Duration Breakdown breakdown.png Each color indicates a service. The colors assigned to services are always the same on your account. You can change the color in the span summary tab after clicking on the individual span in trace view.

Hover over to view a percentage breakdown of how long each span covers in the trace.

breakdown hover.png
Number of errors 0 The number of errors in the trace.
Status 200 The HTTP status code of the trace.

A menu is available in this column when hovering on a row.
similar traces menu.png
The menu has an option to Show similar traces.

Trace query

A trace query allows you to search for traces representing transaction flows through your system using the following filters:

  • Root Service: name of the service that started the trace
  • Any Service: name of a service that took part in the trace
  • Duration: time in milliseconds of the trace
  • Number of spans
  • Number of errors

To write a query click on the Choose filters input line. You can select the desired filter type and value from the drop down menu or manually type them. Multiple filters are allowed in a query row, AND is implicit.

filters.png

You can add more queries by clicking the + icon on the right of the query row:

Add trace query.png

Each query is labeled with a letter, in the following screenshot the first query on the top row is labeled #A, the second query is labeled #B.

trace queries.png

Visibility

Use the eye icon to toggle the visibility of results from a query. When hidden, the traces returned from the query in the row are not displayed in your results.

hide and show query results.png

Time range 

Results are returned for the time window selected. See Time Range Expressions for details on defining a time range.

Refresh

The results are not automatically updated. If you want to refresh traces, click the refresh button on the top right corner of the page.

Refresh.png

Trace View

Trace View shows the time flow of a single trace by its spans. To open, click on a trace row in the Traces table. Each color represents a different service.

trace view main.png

Navigate
  • Use your mouse to drag and zoom in and out. To reset your view, click the Reset View button on the bottom left of the window.
    reset view.png
  • You can click the Critical path contribution by service bar at the top to filter out the services that are of less interest. The critical path is the sequence of span segments that contribute to the total trace duration.
    critical path on trace view.png
  • Use the Error Spans Only toggle to hide or show error spans and the Hide all services button to hide services.
    toggle and button hide.png
  • Hover over a span to view the parent span relationship.
    hover span.png
Details pane

Click on any span to open a side panel with details.

span details.png

The details pane provides the following tabs:

  • Summary
    • The details of the span are provided.
    • To drill down further into your data, the Logs section has links to run searches against related log data. Top links for span/trace IDs work if you have span and trace IDs injected into logs. Lower section links are available and work automatically if SumoLogic Kubernetes Collection is installed.
      Logs links.png
    • Service Color
    • The color of the Service can be changed by clicking the colored box and selecting a defined swatch or custom color.
      color option.png
  • Metadata
    • Lists all of the related service entities involved in the span. trace metadata entities.png
  • Pod
    • Shows metrics from the entities involved in the span.
      pod details.png
  • Error
    • An error span displays the type of error it is. Any span log fields will appear here.
      trace error details.png