Collect logs and metrics for the Istio App

Log and Metric Types

Istio components are built with a flexible logging framework that is leveraged by the Sumo Logic App for Istio. The Sumo Logic App for Istio utilizes logs from following Istio components:

Envoy - mediates all inbound and outbound traffic for all services in the service mesh.
Mixer - enforces access control and usage policies across the service mesh, and collects telemetry data from the Envoy proxy and other services.
Citadel - provides service discovery for the Envoy sidecars, traffic management capabilities for intelligent routing and resiliency.
Galley - enables strong service-to-service and end-user authentication with built-in identity and credential management.
Pilot - provides configuration validation, ingestion, processing and distribution.
Sidecar Injector - acts as a webhook, automatically adding a proxy into user-created pods.
Ingress and Egress gateway logs - exposes a service outside of the service mesh, and allows access to external HTTP and HTTPS services from applications inside the mesh respectively.

For details on Istio metric types, see the Sample Metrics section on this page. For details on Istio logs, see the Sample Log Messages section on this page. For more information on Istio logs, see the Istio documentation.

Collection overview

Logs are collected and forwarded to Sumo Logic with FluentBit and FluentD. Prometheus collects metrics data for Sumo Logic. To set up log and metric collection you'll complete the following tasks:

Configure metric collection, which includes deploying FluentD and configuring Prometheus.
Configure log collection, which includes deploying FluentBit.

Step 1. Configure metric collection

There are two options for setting up metric collection:

Kubernetes collection is already set up
Kubernetes collection has not been set up

Use the procedure that is appropriate for your current installation.

A. Kubernetes collection is already set up

Use this task if you have already set up Kubernetes collection. Otherwise, go to task B below.

To collect metrics for Istio with Kubernetes collection already set up, do one of the following:

If you did not install using the Sumo Logic Helm chart, do the following:

Locate the prometheus overrides file you used.
Add following scrape configs to the additionalScrapeConfigs section of the prometheusSpec field. These configs scrape Istio endpoints for metrics.
Add the following remoteWrite rules to send scraped Istio metrics to Sumo Logic. For an example of a prometheus-overrides.yaml with Istio scraped configs and rules, refer to this .yaml file.
Upgrade the Prometheus operator to get the metrics into Prometheus with the following command.

helm upgrade prometheus-operator stable/prometheus-operator -f prometheus-overrides.yaml

If you did install using the Sumo Logic Helm chart, do the following:

Update the helm chart values file by adding the following:

Add this additionalScrapeConfigs section to the prometheusSpec field of the values.yaml file. These configs scrape Istio endpoints for metrics.
Add these rules to the remoteWrite section of the values.yaml file. This sends scraped metrics to Sumo Logic.

Upgrade the Sumo Logic helm chart with the following command.

helm upgrade collection-sumologic sumologic/sumologic -f values.yaml

B. Kubernetes collection has not been set up

Use this task if Kubernetes collection. has not yet been set up. This task includes deploying FluentD and configuring Prometheus.

To set up Kubernetes collection and metric collection for Istio, do the following:

Deploy FluentD, as described in Step 1 of this document.
Configure Prometheus, and download prometheus-overrides.yaml as described in Step 2 of this document.
Open prometheus-overrides.yaml in editor and add the following:

Add scrape configs to additionalScrapeConfigs section of prometheusSpec field. These configs scrape Istio endpoints for metrics.
Add remoteWrite rules to send scraped Istio metrics to Sumo Logic. For an example of a prometheus-overrides.yaml with Istio scraped configs and rules, refer to this .yaml file.

Install prometheus-operator using Helm, with the following command:

helm repo update \   && helm install stable/prometheus-operator --name prometheus-operator --namespace sumologic -f prometheus-overrides.yaml

Step 2. Configure log collection

To configure log collection: deploy FluentBit as described in Step 3 of this document.

Sample Logs and Metrics

This section provides Istio sample metrics and sample log messages.

The Istio App utilizes logs from following Istio components.

Envoy
Mixer
Citadel
Galley
Pilot
Sidecar Injector
Ingress and Egress Gateway Logs

For more information on Istio logs and metrics, see this Istio document.

Sample Metrics

Citadel	citadel_secret_controller_csr_err_count citadel_secret_controller_secret_deleted_cert_count citadel_secret_controller_svc_acc_created_cert_count citadel_secret_controller_svc_acc_deleted_cert_count citadel_server_authentication_failure_count citadel_server_citadel_root_cert_expiry_timestamp citadel_server_csr_count citadel_server_csr_parsing_err_count citadel_server_id_extraction_err_count citadel_server_success_cert_issuance_count
Envoy	galley_istio_authentication_meshpolicies galley_istio_mesh_MeshConfig galley_istio_networking_destinationrules galley_istio_networking_gateways galley_istio_networking_virtualservices galley_istio_policy_attributemanifests galley_istio_policy_handlers galley_istio_policy_instances galley_istio_policy_rules galley_mcp_source_clients_total galley_mcp_source_message_sizes_bytes_bucket galley_mcp_source_message_sizes_bytes_count galley_mcp_source_message_sizes_bytes_sum galley_mcp_source_request_acks_total galley_runtime_processor_events_processed_total galley_runtime_processor_event_span_duration_milliseconds_bucket galley_runtime_processor_event_span_duration_milliseconds_count galley_runtime_processor_event_span_duration_milliseconds_sum galley_runtime_processor_snapshots_published_total galley_runtime_processor_snapshot_events_total_bucket galley_runtime_processor_snapshot_events_total_count galley_runtime_processor_snapshot_events_total_sum galley_runtime_processor_snapshot_lifetime_duration_milliseconds_bucket galley_runtime_processor_snapshot_lifetime_duration_milliseconds_count galley_runtime_processor_snapshot_lifetime_duration_milliseconds_sum galley_runtime_state_type_instances_total galley_runtime_strategy_on_change_total galley_runtime_strategy_timer_max_time_reached_total galley_runtime_strategy_timer_quiesce_reached_total galley_runtime_strategy_timer_resets_total galley_source_kube_dynamic_converter_success_total galley_source_kube_event_success_total galley_validation_cert_key_updates galley_validation_config_load galley_validation_config_updates galley_validation_passed
Galley	galley_istio_authentication_meshpolicies galley_istio_mesh_MeshConfig galley_istio_networking_destinationrules galley_istio_networking_gateways galley_istio_networking_virtualservices galley_istio_policy_attributemanifests galley_istio_policy_handlers galley_istio_policy_instances galley_istio_policy_rules galley_mcp_source_clients_total galley_mcp_source_message_sizes_bytes_bucket galley_mcp_source_message_sizes_bytes_count galley_mcp_source_message_sizes_bytes_sum galley_mcp_source_request_acks_total galley_runtime_processor_events_processed_total galley_runtime_processor_event_span_duration_milliseconds_bucket galley_runtime_processor_event_span_duration_milliseconds_count galley_runtime_processor_event_span_duration_milliseconds_sum galley_runtime_processor_snapshots_published_total galley_runtime_processor_snapshot_events_total_bucket galley_runtime_processor_snapshot_events_total_count galley_runtime_processor_snapshot_events_total_sum galley_runtime_processor_snapshot_lifetime_duration_milliseconds_bucket galley_runtime_processor_snapshot_lifetime_duration_milliseconds_count galley_runtime_processor_snapshot_lifetime_duration_milliseconds_sum galley_runtime_state_type_instances_total galley_runtime_strategy_on_change_total galley_runtime_strategy_timer_max_time_reached_total galley_runtime_strategy_timer_quiesce_reached_total galley_runtime_strategy_timer_resets_total galley_source_kube_dynamic_converter_success_total galley_source_kube_event_success_total galley_validation_cert_key_updates galley_validation_config_load galley_validation_config_updates galley_validation_passed
Mixer	mixer_config_adapter_info_configs_total mixer_config_adapter_info_config_errors_total mixer_config_attributes_total mixer_config_handler_configs_total mixer_config_handler_validation_error_total mixer_config_instance_configs_total mixer_config_instance_config_errors_total mixer_config_rule_configs_total mixer_config_rule_config_errors_total mixer_config_rule_config_match_error_total mixer_config_template_configs_total mixer_config_template_config_errors_total mixer_config_unsatisfied_action_handler_total mixer_dispatcher_destinations_per_request_bucket mixer_dispatcher_destinations_per_request_count mixer_dispatcher_destinations_per_request_sum mixer_dispatcher_destinations_per_variety_total mixer_dispatcher_instances_per_request_bucket mixer_dispatcher_instances_per_request_count mixer_dispatcher_instances_per_request_sum mixer_handler_closed_handlers_total mixer_handler_daemons_total mixer_handler_handler_build_failures_total mixer_handler_handler_close_failures_total mixer_handler_new_handlers_total mixer_handler_reused_handlers_total mixer_loadshedding_requests_throttled mixer_mcp_sink_reconnections mixer_mcp_sink_recv_failures_total mixer_mcp_sink_request_acks_total mixer_runtime_dispatches_total mixer_runtime_dispatch_duration_seconds_bucket mixer_runtime_dispatch_duration_seconds_count mixer_runtime_dispatch_duration_seconds_sum
Istio	istio_requests_total istio_request_bytes_bucket istio_request_bytes_count istio_request_bytes_sum istio_request_duration_seconds_bucket istio_request_duration_seconds_count istio_request_duration_seconds_sum istio_response_bytes_bucket istio_response_bytes_count istio_response_bytes_sum

Sample Log Messages

Citadel	{"timestamp":1567229498836,"log":"2019-08-31T05:31:38.836754Z\tinfo\tSecret kubernetes-dashboard/istio.kubernetes-dashboard is created successfully","stream":"stdout","time":"2019-08-31T05:31:38.836953218Z"}
Envoy	{"timestamp":1567599304394,"log":"[2019-09-04 12:15:04.394][14][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 13, ","stream":"stderr","time":"2019-09-04T12:15:04.394322584Z"}
Galley	{"timestamp":1566840793724,"log":"2019-08-26T17:33:13.679104Z\tinfo\tmcp\tSetSnapshot(): respond to watch 21 for istio/networking/v1alpha3/gateways @ version \"30\"","stream":"stdout","time":"2019-08-26T17:33:13.724575625Z"}
Mixer	{"timestamp":1566752321320,"log":"2019-08-25T16:58:41.319910Z\terror\tistio.io/istio/pkg/kube/secretcontroller/secretcontroller.go:148: Failed to list *v1.Secret: Get https://10.56.0.1:443/api/v1/namespa...ourceVersion=0: net/http: TLS handshake timeout","stream":"stdout","time":"2019-08-25T16:58:41.320068921Z"}
Pilot	{"timestamp":1567599723043,"log":"2019-09-04T12:22:03.043108Z\tinfo\tads\tRDS: PUSH for node:reviews-v3-6fbc779988-p99c7.default routes:18","stream":"stdout","time":"2019-09-04T12:22:03.043357438Z"}
Sidecar Injector	View as JSON {"timestamp":1566752321306,"log":"2019-08-25T16:58:41.305874Z\terror\tistio.io/istio/pilot/cmd/sidecar-injector/main.go:173: Failed to list *v1beta1.MutatingWebhookConfiguration: Get https://10.56.0.1:443/apis/admission...ourceVersion=0: net/http: TLS handshake timeout","stream":"stdout","time":"2019-08-25T16:58:41.306191001Z"}
Ingress and Egress Gateway Logs	{"timestamp":1567599258659,"log":"[2019-09-04T12:14:11.483Z] \"GET /productpage HTTP/1.1\" 200 - \"-\" \"-\" 0 4183 40 39 \"10.44.1.1\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36\" \"48408b99-bf18-9216-a369-dc471e889790\" \"35.238.76.93\" \"10.44.1.12:9080\" outbound\|9080\|\|productpage.default.svc.cluster.local - 10.44.1.4:80 10.44.1.1:60315 -","stream":"stdout","time":"2019-09-04T12:14:18.659156219Z"}

Query Sample

_collector="gke-istio-collector" 
| json field=_raw "time", "log.sourceApp" ,"log.destinationApp","log.level", "log.latency", 
"log.responseCode", "log.url", "log.sentBytes", "log.receivedBytes" as time, source, destination, 
level, latency, response, url, bytes_out, bytes_in