An HTTP Source is an endpoint for receiving a file (or a batch of files) uploaded via a unique URL generated for the Source. The URL securely encodes the Collector and Source information. You can add as many HTTP Sources as you'd like to a single Hosted Collector. After that, you can see:
With an HTTP Source you can upload logs from data sources where you cannot install a Collector. For example, you can export data from a platform as a service (PaaS) or infrastructure as a service (IaaS) provider, allowing you to gain visibility from, say your billing system service provider, leveraging the same Sumo Logic tools your organization already uses. Please check with your IaaS or PaaS providers for information regarding using their APIs to forward log data into Sumo Logic's HTTP endpoint.
The generated URL is a long string of letters and numbers. You can generate a new URL at any time. For more information see Generating a new URL.
When you set up an HTTP Source, a unique URL is assigned to that Source. When you upload a file using that URL, it's associated with the Source, and metadata is tagged to the file.
To configure an HTTP Source:
- In the Sumo Logic Web Application, select Manage > Collection.
- In the Collectors page, click Add Source next to a Hosted Collector.
- Select HTTP.
- Enter a Name to display for this Source in the Sumo Logic Web Application. Description is optional.
- (Optional) For Source Host and Source Category, enter any string to tag the output collected from this Source. (Category metadata is stored in a searchable field called _sourceCategory.)
- Set any of the following under Advanced:
- Enable Timestamp Parsing. This option is selected by default. If it's deselected, no timestamp information is parsed at all.
- Time Zone. There are two options for Time Zone. You can use the time zone present in your log files, and then choose an option in case time zone information is missing from a log message. Or, you can have Sumo Logic completely disregard any time zone information present in logs by forcing a time zone. It's very important to have the proper time zone set, no matter which option you choose. If the time zone of logs can't be determined, Sumo Logic assigns logs UTC; if the rest of your logs are from another time zone your search results will be affected.
- Timestamp Format. By default, Sumo Logic will automatically detect the timestamp format of your logs. However, you can manually specify a timestamp format for a Source. See Timestamps, Time Zones, Time Ranges, and Date Formats for more information.
- Enable Multiline Processing. Use this option if you're working with multi-line messages (for example, log4J or exception stack traces). Deselect this option if you want to avoid unnecessary processing when collecting single-message-per-line files (for example, Linux system.log).
- Infer Boundaries. Enable when you want Sumo Logic to automatically attempt to determine which lines belong to the same message.
If you deselect the Infer Boundaries option, you will need to enter a regular expression in the Boundary Regex field to use for detecting the entire first line of multi-line messages.
- Boundary Regex. You can specify the boundary between messages using a regular expression. Enter a regular expression for the full first line of every multi-line message in your log files. For an example, see the boundary regex section in Configure a Local File Source.
- Enable One Message Per Request. Select this option if you'll be sending a single message with each HTTP request. For more information, see Multiline Options in HTTP Sources.
- Create any Processing Rules you'd like for the new Source to setup any include, exclude, hash or mask filters. Note that while the Sumo Logic service will receive your data, the ingestion of the same will be done in accordance with the regular expressions specified in the processing rules set up.
- When you are finished configuring the Source click Save.
- When the URL associated with the Source is displayed, copy the URL so you can use it to upload data.
- Choose the method you'll use to upload files to the Source.
Access a Source's URL
If you need to access the Source's URL again, just click Show URL.
Multiline options in HTTP Sources
The HTTP Source isn't designed to support large numbers of connections per Source. If possible, you should batch log messages locally and send batches on a single thread.
To increase throughput, batch multiple log messages in a single request to the HTTP Source. If any of those logs can contain multiline messages, like stack traces, activate Enable Multiline Processing.
For basic multiline processing select Infer Boundaries, but if this leads to malformed messages, you can instead specify a regular expression to determine the multiline boundary.
Also, in your HTTP Source configuration, make sure that the check box Enable One Message Per Request is deactivated. This option allows you to specify that all data sent within an individual HTTP request to HTTP Source endpoint should be considered to be one log message.
Sumo Logic expects that the entire content of an individual log message will be sent to Sumo Logic within the same HTTP request. Multiline processing rules are only applied within the bounds of the data sent within a single HTTP request. This means that a multiline log that is sent to Sumo Logic across multiple HTTP requests will not be detected as a single message. It will be broken into separate log messages. Sumo Logic does not currently have the ability to detect and thread together a distinct log message that has been sent via multiple HTTP requests.
For tools to help you batch messages, see https://github.com/SumoLogic/sumologic-net-appenders.