Skip to main content
Sumo Logic

Collect Logs for Google Cloud Audit

This page describes the Sumo pipeline for ingesting logs from Google Cloud Platform (GCP) services, and provides instructions for configuring log collection for the Google Cloud Audit App.

This page describes the Sumo pipeline for ingesting logs from Google Cloud Platform (GCP) services, and provides instructions for collecting logs from Google Cloud Audit.

Collection process for GCP services

The key components in the collection process for GCP services are:  Google Stackdriver, Google Cloud Pub/Sub, and Sumo’s Google Cloud Platform (GCP) source running on a hosted collector. 

The integration works like this: Google Stackdriver collects logs from GCP services. Once you’ve configured the pipeline shown below, the logs collected by Stackdriver will be published to a Google Pub/Sub topic. A Sumo GCP source on a hosted collector subscribed to that topic ingests the logs into Sumo.


The configuration process is as follows. 

  1. Configure a GCP source on a hosted collector. You'll obtain the HTTP URL for the source, and then use Google Cloud Console to register the URL as a validated domain.  
  2. Create a topic in Google Pub/Sub and subscribe the GCP source URL to that topic.
  3. Create an export of GCP logs from Stackdriver. Exporting involves writing a filter that selects the log entries you want to export, and choosing a Pub/Sub as the destination. The filter and destination are held in an object called a sink. 

See the sections below for instructions.

Configure a Google Cloud Platform Source

The Google Cloud Platform (GCP) Source receives log data from Google Pub/Sub.

In this section, you add the Source URL as an allowed domain to your GCP account. This Source will be a Google Pub/Sub-only Source, which means that it will only be usable for log data formatted as data coming from Google Pub/Sub.

  1. In Sumo Logic select Manage Data > Collection > Collection
  2. Select an existing Hosted Collector upon which to add the Source. If you don't already have a Collector you'd like to use, create one, using the instructions on Configure a Hosted Collector.
  3. Click Add Source next to the Hosted Collector and click Google Cloud Platform.
  4. Enter a Name to display for the source in the Sumo web application. Description is optional.
  5. (Optional) For Source Host and Source Category, enter any string to tag the output collected from the source. Category metadata is stored in a searchable field called _sourceCategory, for example "gcp".
  6. Fields. Click the +Add Field link to add custom log metadata Fields.
    • Define the fields you want to associate, each field needs a name (key) and value. 
      • green check circle.png A green circle with a check mark is shown when the field exists and is enabled in the Fields table schema.
      • orange exclamation point.png An orange triangle with an exclamation point is shown when the field doesn't exist, or is disabled, in the Fields table schema. In this case, an option to automatically add or enable the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo that does not exist in the Fields schema or is disabled it is ignored, known as dropped.
  7. Log File Discovery. To set up the subscription you need to get an endpoint URL from Sumo to provide to Google. Click on Copy URL and use the provided URL as an allowed domain in GCP.
    log file discovery google platform.png Steps to add the Source's URL as an allowed domain in GCP:
    1. Open your Google Cloud Console.
    2. Select Products and services > APIs & Services > Credentials.
    3. Select Domain Verification > Add Domain.
    4. In the Configure webhook notifications for … dialog,  add the Source URL as valid domain and click Add Domain.
    5. Click Take Me There to verify ownership of the URL at Google’s webmaster central page. You are taken to the Google’s Webmaster Central interface to verify the URL.  
    6. Click Add Property in the Webmaster Central site and add the Source URL.
    7. In Google, shown in the following screenshot, you only need to complete step 1, Download and open the HTML verification file. Complete the next steps, 8-11, in Sumo Logic before clicking Verify. You will verify in step 12.
      verification from webmaster central.png
  8. Return to the Source configuration page in Sumo under Log File Discovery. 
    • For Verification File Name, enter the verification file name.
    • For Verification File Contents, copy and paste the full string from the verification file's body contents.
      verification file input in sumo.png
      Continue configuring the Source in Sumo.
  9. Advanced Options for Logs
    • Enable Timestamp Parsing. This option is selected by default. If it's deselected, no timestamp information is parsed at all.
    • Time Zone. There are two options for Time Zone. You can use the time zone present in your log files, and then choose an option in case time zone information is missing from a log message. Or, you can have Sumo Logic completely disregard any time zone information present in logs by forcing a time zone. It's very important to have the proper time zone set, no matter which option you choose. If the time zone of logs can't be determined, Sumo Logic assigns logs UTC; if the rest of your logs are from another time zone your search results will be affected.
    • Timestamp Format. By default, Sumo Logic will automatically detect the timestamp format of your logs. However, you can manually specify a timestamp format for a Source. See Timestamps, Time Zones, Time Ranges, and Date Formats for more information.
  10. Processing Rules for Logs. Configure desired filters—such as include, exclude, hash, or mask—as described in Create a Processing Rule. Processing rules are applied to log data, but not to metric data. Note that while the Sumo service will receive your data, data ingestion will be performed in accordance with the regular expressions you specify in processing rules.
  11. When you are finished configuring the Source click Save.
  12. Return to Google's Webmaster Central and click Verify. It should verify successfully.
  13. Finally, in your GCP console return to Products and services > APIs & Services > Domain verification and add the Sumo Logic endpoint URL that you just verified to have the endpoint show up as a verified domain in GCP.

Configure a Pub/Sub topic for GCP projects

In this step, you configure a Pub/Sub topic in GCP and add a subscription to the above source URL. Once you configure the Pub/Sub, you can export data from Stackdriver to the Pub/Sub. For example, you can export Google App Engine logs, as described on Collect Logs for Google App Engine.

  1. In GCP, select Pub/Sub in the left navigation pane.
  2. In the Pub/Sub pane, select Topics, then click Create Topic in the Topics pane.
  3. Name the topic and click Create.

  4. Select the new topic in the Topics pane, and select New subscription from the options menu.

  5. In the Create a subscription pane:
    1. Subscription Name. Enter a name for the subscription.
    2. Delivery Type. Choose “Push into an endpoint url”, and enter the upload URL for the Sumo HTTP source you created above.
    3. Click Create.

Create export of Cloud Audit logs from Stackdriver 

  1. Go to Logging in GCP.
  2. Go to Exports. Click Create Export.
  3. Add an advanced filter by clicking Convert to advanced filter under the dropdown.
  4. Create an advanced filter of logName="projects/{project_id}/logs/".
    For information about defining advanced filters, see Advanced Filters in GCP help.
    In the Edit Export window on the right:

    1. Set the Sink Name. For example, "gcp-all".
    2. Set Sink Service to “Cloud Pub/Sub”.
    3. Set Sink Destination to the newly created Pub/Sub topic. For example, "pub-sub-logs".
    4. Click Create Sink.

Sample Log Message

  "message": {
    "data": {
      "insertId": "55E06F0577741.AA05843.A90CA7B9",
      "logName": "projects/bmlabs-loggen/logs/",
      "operation": {
        "id": "operation-1510758777595-55e06f047a479-fd74bd40-dc6cfc9b",
        "last": true,
        "producer": ""
      "protoPayload": {
        "@type": "",
        "authenticationInfo": {
          "principalEmail": ""
        "methodName": "beta.compute.instanceTemplates.delete",
        "requestMetadata": {
          "callerIp": "",
          "callerSuppliedUserAgent": "cloud_workflow_service"
        "resourceName": "projects/bmlabs-loggen/global/instanceTemplates/dataflow-permissionlogs-johndoe-1-11150704-7cbb-harness",
        "serviceName": ""
      "receiveTimestamp": "2018-01-26T12:08:31.316UTC",
      "resource": {
        "labels": {
          "instance_template_id": "176548811930462611",
          "instance_template_name": "dataflow-permissionlogs-johndoe-1-11150704-7cbb-harness",
          "project_id": "bmlabs-loggen"
        "type": "gce_instance_template"
      "severity": "NOTICE",
      "timestamp": "2018-01-26T12:08:31.316UTC"
    "attributes": {
      "": "2018-01-26T12:08:31.316UTC"
    "message_id": "172054682231179",
    "messageId": "172054682231179",
    "publish_time": "2018-01-26T12:08:31.316UTC",
    "publishTime": "2018-01-26T12:08:31.316UTC"
  "subscription": "projects/bmlabs-loggen/subscriptions/sumo-test"

Query Sample

Recent firewall changes

_collector="HTTP Source for GCP Pub/Sub" logName methodName principalEmail request resource timestamp
| parse regex "\"logName\":\"(?<log_name>[^\"]+)\"" 
| where log_name matches "projects/*/logs/"
| json "" as data
| json field=data "resource.type" as type
| where type = "gce_firewall_rule"
| json field=data "timestamp", "resource.labels", "resource.labels.project_id", "protoPayload.authenticationInfo.principalEmail", "protoPayload.methodName", "protoPayload.request" as timestamp, labels, project, user, method, request
| json field=request "direction", "alloweds[*]", "denieds[*]" as direction, alloweds, denieds nodrop
| if(isNull(alloweds) OR alloweds="","deny","allow") as action
| parse "\"sourceRanges\":[*]" as ranges nodrop
| parse "\"destinationRanges\":[*]" as ranges
| parse regex field=alloweds "\"IPProtocol\":\"(?<protocol>[a-zA-Z\.]+)\"[,\"a-z:]*\[?(?<ports>[0-9-\",]+)?\]?" multi nodrop
| parse regex field=denieds "\"IPProtocol\":\"(?<protocol>[a-zA-Z\.]+)\"[,\"a-z:]*\[?(?<ports>[0-9-\",]+)?\]?" multi
| count as operations by timestamp, user, method, ranges, direction, action, protocol, ports
| fields timestamp, user, method, ranges, direction, action, protocol, ports
| sort by timestamp