If you are using Google Cloud Platform (GCP) services, all log data for these services is collected and exposed through the Google Cloud Stackdriver service. You can export in real time all of the data collected by Stackdriver to Google Cloud Pub/Sub. We use this Pub/Sub integration to push logs to our platform in real time.
To get started collecting data from Google Cloud Platform, you must:
- Configure an HTTP Source in Sumo Logic.
- Ensure that GCP logs data are exported to Stackdriver.
- Configure a Google Pub/Sub topic, and export GCP Stackdriver logs to Google Pub/Sub.
- Configure a Google Pub/Sub Push subscription to send data to your Sumo HTTP Source. As part of this process, you will need to use Google’s Webmaster Console to validate that you are the owner of the Sumo Logic API endpoint.
Before you can configure a Pub/Sub push subscription to send data to an external URL, you must validate that you own that URL. The instructions below help you validate that you ‘own’ the URL provided to you when you created the Sumo HTTP Source.
To collect logs from GCP:
- Create an HTTP Source.
- Add the HTTP source URL as an allowed domain to your GCP account. This source will be a Google Pub/Sub-only source, which means that it will only be usable for log data formatted as data coming from Google Pub/Sub.
- Add the HTTP source URL as an allowed domain to your GCP account.
- Open your Google Cloud Console.
- Select Products and services > API Manager > Credentials.
- Select Domain Verification > Add Domain.
- In the Configure webhook notifications for … dialog, add the HTTP source URL as valid domain and click Add Domain.
- Click Take Me There to verify ownership of the URL at Google’s webmaster central page. You are taken to the Google’s Webmaster Central interface to verify the URL.
- Click Add Property in the Webmaster Central site and add the HTTP Source URL.
- Update the HTTP Source configuration, using the Sumo Logic Collector Management API, with information from the “HTML verification file” that you just downloaded from Webmaster central. Add a thirdPartyRef field in the HTTP Source JSON file and insert the “HTML verification file”.
Review the Collector Management API.
Have an Access Key and ID. Save them because you will need them later.
Click on the information icon for the source to get the source API URL.
Copy the Source API URL field.
Using the Source API URL, make a GET request that looks like this:
curl -v -u 'accessid:accesskey' -X GET https://<yourendpoint>/api/v1/collectors/<collectorID>/sources/<sourceID> >source.json
Make note of the etag value in the response. You will need it.
Edit JSON file to insert the thirdPartyRef JSON block below, replacing the ’name’ field with the filename of the google HTML Verification File, and the ‘contents’ field with the full string from the body of the HTML Verification File.
Update the HTTP source with the new thirdPartyRef field and fill in the Google verification information
- Update the Source JSON file using the API with the following CURL command. You must add the etag value in the example below.
curl -u 'accessid:accesskey' -X PUT -H "Content-Type: application/json" -H "If-Match: \"<ETag>\"" -T <source>.json https://api.sumologic.com/api/v1/collectors/<collectorid>/sources/<sourceID>
- After the JSON file is uploaded, return to the Google, and Click on “Verify” on the page in point number 2. It should verify successfully.
- Create a Google Pub-Sub topic.
- Add HTTP source URL to the Google Pub/Sub topic.
- Go to the Pub/Sub topic.
- Create Subscription > Add the HTTP source URL and subscription name.
- Integration can be tested by going to the Pub/Sub topic > Publish Message > Type a message > Publish
The log message will be in JSON format with a field called “data” having your typed message as the value.
You’re done. To configure Stackdriver to export logs to Sumo, follow the Google Cloud Instructions.