Skip to main content
Sumo Logic

Collect logs and metrics for the Istio App

This page shows you how to configure log and metric collection for Istio, and provides sample metrics, log messages, and a query example.

This page provides instructions for collecting logs and metrics for the Sumo App for Istio. Logs are collected and forwarded to Sumo Logic with FluentBit and FluentD. Prometheus collects metrics data for Sumo Logic. Istio sample metrics and sample log messages are also provided, along with a query sample.

Log and Metric Types 

Istio components are built with a flexible logging framework that is leveraged by the Sumo Logic App for Istio. The Sumo Logic App for Istio utilizes logs from following Istio components:

  • Envoy - mediates all inbound and outbound traffic for all services in the service mesh.
  • Mixer - enforces access control and usage policies across the service mesh, and collects telemetry data from the Envoy proxy and other services.
  • Citadel - provides service discovery for the Envoy sidecars, traffic management capabilities for intelligent routing and resiliency.
  • Galley - enables strong service-to-service and end-user authentication with built-in identity and credential management.
  • Pilot - provides configuration validation, ingestion, processing and distribution.
  • Sidecar Injector - acts as a webhook, automatically adding a proxy into user-created pods.
  • Ingress and Egress gateway logs - exposes a service outside of the service mesh, and allows access to external HTTP and HTTPS services from applications inside the mesh respectively.

For details on Istio metric types, see the Sample Metrics section on this page. For details on Istio logs, see the Sample Log Messages section on this page. For more information on Istio logs, see the Istio documentation.

Collection overview

Logs are collected and forwarded to Sumo Logic with FluentBit and FluentD. Prometheus collects metrics data for Sumo Logic. To set up log and metric collection you'll complete the following tasks:

  1. Configure metric collection, which includes deploying FluentD and configuring Prometheus.
  2. Configure log collection, which includes deploying FluentBit.

Step 1. Configure metric collection

There are two options for setting up metric collection:

  1. Kubernetes collection is already set up
  2. Kubernetes collection has not been set up

Use the procedure that is appropriate for your current installation.

A. Kubernetes collection is already set up

Use this task if you have already set up  Kubernetes collection. Otherwise, go to task B below.

To collect metrics for Istio with  Kubernetes collection already set up, do one of the following:

  • If you did not install using the Sumo Logic Helm chart, do the following:
  1. Locate the prometheus overrides file you used.
  2. Add following scrape configs to the  additionalScrapeConfigs section of the prometheusSpec field. These configs scrape Istio endpoints for metrics.
  3. Add the following remoteWrite rules to send scraped Istio metrics to Sumo Logic. For an example of a prometheus-overrides.yaml with Istio scraped configs and rules, refer to this .yaml file.
  4. Upgrade the Prometheus operator to get the metrics into Prometheus with the following command.
helm upgrade prometheus-operator stable/prometheus-operator -f prometheus-overrides.yaml
  • If you did install using the Sumo Logic Helm chart, do the following:
  1. Update the helm chart values file by adding the following:
  • Add this additionalScrapeConfigs section to the prometheusSpec field of the values.yaml file. These configs scrape Istio endpoints for metrics.
  • Add these rules to the remoteWrite section of the values.yaml file. This sends scraped metrics to Sumo Logic.
  1. Upgrade the Sumo Logic helm chart with the following command.
helm upgrade collection-sumologic sumologic/sumologic -f values.yaml

B. Kubernetes collection has not been set up

Use this task if  Kubernetes collection. has not yet been set  up. This task includes deploying FluentD and configuring Prometheus.

To set up Kubernetes collection and metric collection for Istio, do the following:

  1. Deploy FluentD, as described in Step 1 of this document.
  2. Configure Prometheus, and download prometheus-overrides.yaml as described in Step 2 of this document.
  3. Open prometheus-overrides.yaml in editor and add the following:
  • Add  scrape configs to additionalScrapeConfigs section of prometheusSpec field.  These configs scrape Istio endpoints for metrics.
  • Add remoteWrite rules to send scraped Istio metrics to Sumo Logic. For an example of a prometheus-overrides.yaml with Istio scraped configs and rules, refer to this .yaml file.
  1. Install prometheus-operator using Helm, with the following command:
helm repo update \   && helm install stable/prometheus-operator --name prometheus-operator --namespace sumologic -f prometheus-overrides.yaml

Step 2. Configure log collection

To configure log collection: deploy FluentBit as described in Step 3 of this document.

Sample Logs and Metrics  

This section provides Istio sample metrics and sample log messages.

The Istio App utilizes logs from following Istio components.

  • Envoy
  • Mixer
  • Citadel
  • Galley
  • Pilot
  • Sidecar Injector
  • Ingress and Egress Gateway Logs

For more information on Istio logs and metrics, see this Istio document.

Sample Metrics

Citadel

citadel_secret_controller_csr_err_count

citadel_secret_controller_secret_deleted_cert_count

citadel_secret_controller_svc_acc_created_cert_count

citadel_secret_controller_svc_acc_deleted_cert_count

citadel_server_authentication_failure_count

citadel_server_citadel_root_cert_expiry_timestamp

citadel_server_csr_count

citadel_server_csr_parsing_err_count

citadel_server_id_extraction_err_count

citadel_server_success_cert_issuance_count

Envoy

galley_istio_authentication_meshpolicies

galley_istio_mesh_MeshConfig

galley_istio_networking_destinationrules

galley_istio_networking_gateways

galley_istio_networking_virtualservices

galley_istio_policy_attributemanifests

galley_istio_policy_handlers

galley_istio_policy_instances

galley_istio_policy_rules

galley_mcp_source_clients_total

galley_mcp_source_message_sizes_bytes_bucket

galley_mcp_source_message_sizes_bytes_count

galley_mcp_source_message_sizes_bytes_sum

galley_mcp_source_request_acks_total

galley_runtime_processor_events_processed_total

galley_runtime_processor_event_span_duration_milliseconds_bucket

galley_runtime_processor_event_span_duration_milliseconds_count

galley_runtime_processor_event_span_duration_milliseconds_sum

galley_runtime_processor_snapshots_published_total

galley_runtime_processor_snapshot_events_total_bucket

galley_runtime_processor_snapshot_events_total_count

galley_runtime_processor_snapshot_events_total_sum

galley_runtime_processor_snapshot_lifetime_duration_milliseconds_bucket

galley_runtime_processor_snapshot_lifetime_duration_milliseconds_count

galley_runtime_processor_snapshot_lifetime_duration_milliseconds_sum

galley_runtime_state_type_instances_total

galley_runtime_strategy_on_change_total

galley_runtime_strategy_timer_max_time_reached_total

galley_runtime_strategy_timer_quiesce_reached_total

galley_runtime_strategy_timer_resets_total

galley_source_kube_dynamic_converter_success_total

galley_source_kube_event_success_total

galley_validation_cert_key_updates

galley_validation_config_load

galley_validation_config_updates

galley_validation_passed

Galley

galley_istio_authentication_meshpolicies

galley_istio_mesh_MeshConfig

galley_istio_networking_destinationrules

galley_istio_networking_gateways

galley_istio_networking_virtualservices

galley_istio_policy_attributemanifests

galley_istio_policy_handlers

galley_istio_policy_instances

galley_istio_policy_rules

galley_mcp_source_clients_total

galley_mcp_source_message_sizes_bytes_bucket

galley_mcp_source_message_sizes_bytes_count

galley_mcp_source_message_sizes_bytes_sum

galley_mcp_source_request_acks_total

galley_runtime_processor_events_processed_total

galley_runtime_processor_event_span_duration_milliseconds_bucket

galley_runtime_processor_event_span_duration_milliseconds_count

galley_runtime_processor_event_span_duration_milliseconds_sum

galley_runtime_processor_snapshots_published_total

galley_runtime_processor_snapshot_events_total_bucket

galley_runtime_processor_snapshot_events_total_count

galley_runtime_processor_snapshot_events_total_sum

galley_runtime_processor_snapshot_lifetime_duration_milliseconds_bucket

galley_runtime_processor_snapshot_lifetime_duration_milliseconds_count

galley_runtime_processor_snapshot_lifetime_duration_milliseconds_sum

galley_runtime_state_type_instances_total

galley_runtime_strategy_on_change_total

galley_runtime_strategy_timer_max_time_reached_total

galley_runtime_strategy_timer_quiesce_reached_total

galley_runtime_strategy_timer_resets_total

galley_source_kube_dynamic_converter_success_total

galley_source_kube_event_success_total

galley_validation_cert_key_updates

galley_validation_config_load

galley_validation_config_updates

galley_validation_passed

Mixer

mixer_config_adapter_info_configs_total

mixer_config_adapter_info_config_errors_total

mixer_config_attributes_total

mixer_config_handler_configs_total

mixer_config_handler_validation_error_total

mixer_config_instance_configs_total

mixer_config_instance_config_errors_total

mixer_config_rule_configs_total

mixer_config_rule_config_errors_total

mixer_config_rule_config_match_error_total

mixer_config_template_configs_total

mixer_config_template_config_errors_total

mixer_config_unsatisfied_action_handler_total

mixer_dispatcher_destinations_per_request_bucket

mixer_dispatcher_destinations_per_request_count

mixer_dispatcher_destinations_per_request_sum

mixer_dispatcher_destinations_per_variety_total

mixer_dispatcher_instances_per_request_bucket

mixer_dispatcher_instances_per_request_count

mixer_dispatcher_instances_per_request_sum

mixer_handler_closed_handlers_total

mixer_handler_daemons_total

mixer_handler_handler_build_failures_total

mixer_handler_handler_close_failures_total

mixer_handler_new_handlers_total

mixer_handler_reused_handlers_total

mixer_loadshedding_requests_throttled

mixer_mcp_sink_reconnections

mixer_mcp_sink_recv_failures_total

mixer_mcp_sink_request_acks_total

mixer_runtime_dispatches_total

mixer_runtime_dispatch_duration_seconds_bucket

mixer_runtime_dispatch_duration_seconds_count

mixer_runtime_dispatch_duration_seconds_sum

Istio

istio_requests_total

istio_request_bytes_bucket

istio_request_bytes_count

istio_request_bytes_sum

istio_request_duration_seconds_bucket

istio_request_duration_seconds_count

istio_request_duration_seconds_sum

istio_response_bytes_bucket

istio_response_bytes_count

istio_response_bytes_sum

Sample Log Messages

Citadel {"timestamp":1567229498836,"log":"2019-08-31T05:31:38.836754Z\tinfo\tSecret kubernetes-dashboard/istio.kubernetes-dashboard is created successfully","stream":"stdout","time":"2019-08-31T05:31:38.836953218Z"}
Envoy {"timestamp":1567599304394,"log":"[2019-09-04 12:15:04.394][14][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 13, ","stream":"stderr","time":"2019-09-04T12:15:04.394322584Z"}
Galley {"timestamp":1566840793724,"log":"2019-08-26T17:33:13.679104Z\tinfo\tmcp\tSetSnapshot(): respond to watch 21 for istio/networking/v1alpha3/gateways @ version \"30\"","stream":"stdout","time":"2019-08-26T17:33:13.724575625Z"}
Mixer {"timestamp":1566752321320,"log":"2019-08-25T16:58:41.319910Z\terror\tistio.io/istio/pkg/kube/secretcontroller/secretcontroller.go:148: Failed to list *v1.Secret: Get https://10.56.0.1:443/api/v1/namespa...ourceVersion=0: net/http: TLS handshake timeout","stream":"stdout","time":"2019-08-25T16:58:41.320068921Z"}
Pilot {"timestamp":1567599723043,"log":"2019-09-04T12:22:03.043108Z\tinfo\tads\tRDS: PUSH for node:reviews-v3-6fbc779988-p99c7.default routes:18","stream":"stdout","time":"2019-09-04T12:22:03.043357438Z"}
Sidecar Injector

View as JSON

{"timestamp":1566752321306,"log":"2019-08-25T16:58:41.305874Z\terror\tistio.io/istio/pilot/cmd/sidecar-injector/main.go:173: Failed to list *v1beta1.MutatingWebhookConfiguration: Get https://10.56.0.1:443/apis/admission...ourceVersion=0: net/http: TLS handshake timeout","stream":"stdout","time":"2019-08-25T16:58:41.306191001Z"}

Ingress and Egress Gateway Logs {"timestamp":1567599258659,"log":"[2019-09-04T12:14:11.483Z] \"GET /productpage HTTP/1.1\" 200 - \"-\" \"-\" 0 4183 40 39 \"10.44.1.1\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36\" \"48408b99-bf18-9216-a369-dc471e889790\" \"35.238.76.93\" \"10.44.1.12:9080\" outbound|9080||productpage.default.svc.cluster.local - 10.44.1.4:80 10.44.1.1:60315 -","stream":"stdout","time":"2019-09-04T12:14:18.659156219Z"}

Query Sample

_collector="gke-istio-collector" 
| json field=_raw "time", "log.sourceApp" ,"log.destinationApp","log.level", "log.latency", 
"log.responseCode", "log.url", "log.sentBytes", "log.receivedBytes" as time, source, destination, 
level, latency, response, url, bytes_out, bytes_in