Google Cloud Dataflow

Dataflow is a managed service for executing a wide variety of data processing patterns. For more details, refer to the GCP documentation

Log types

• Platform Logs
• Audit Logs

Setup

You can collect the logs for Sumo Logic's Google Cloud Dataflow integration by following the below steps.

Configure logs collection

• Collect Audit Logs using the Google Cloud Platform source. These Audit Logs can be accessed based on the permissions and roles. To enable logging for Google Dataflow, refer to Google documentation. For more detail on Dataflow operations being audited, refer to audited operations. While creating the sync in GCP, as part of the Choose logs to include in sink section, you can use the following query:

  (resource.type=audited_resource AND resource.labels.service=datapipelines.googleapis.com)

• Collect Platform Logs using the Google Cloud Platform source. Here are the log types collected as pipeline logs. By default, only log lines marked INFO and higher will be sent to Cloud Logging. While creating the sync in GCP, as part of the Choose logs to include in sink section, you can use the following query:

  (resource.type=dataflow_step )