Skip to main content
Sumo Logic

Blacklisting Metrics Sources


In some cases, it's necessary for Sumo Logic to blacklist a metrics Source to limit the number of ingested time series. The following are typical cases where this can occur:

  • With Graphite metrics. Sometimes the names of the Graphite metrics contain strings that are dynamically generated: A common example is that a date or timestamp is inserted as part of the metric name. Another example, typical of dropwizard metrics, is that the names of threads are inserted into the metric names. With dynamically generated metric names, a new time series is created with each new name. When these cases happen, the total number of time series can be subjected to unbounded growth over time. And the time series are typically of little use, given their ephemeral nature.
  • With EC2 CloudWatch metrics. Amazon’s EC2 metric naming convention causes a new time series to be created for each EC2 instance.

If a metrics Source has been blacklisted, a message indicating that some metrics are missing is presented when you query the Source on the Metrics page in Sumo Logic.  A dialog box prompts you to re-enable the Source. Selecting the Re-enable option cancels the blacklist for the moment, but blacklisting will soon be reinstated if the underlying issue is not addressed.

To address the underlying blacklisting issue, do one of the following:

1. Modify the Source configuration or change the Source
If the Source is not a CloudWatch EC2 Source, modify the time series naming convention in the Source to exclude any dynamically inserted elements, such as date, timestamp, or thread names.

Unfortunately, this solution does not work for Amazon EC2 metrics, which are also subject to high latency and can increase the costs of your AWS account. Instead of using EC2 CloudWatch metrics, we recommend that you install a Sumo Logic Collector in your EC2 instance. A Sumo Logic Collector has the ability to send a suite of standard system metrics on the host it is running on, such as CPU, disk, and network metrics. The information is comparable to that provided by CloudWatch. See Host Metrics Source for Installed Collectors for more information. This is a much more cost-effective way to collect host metrics.

2. Clean up existing time series data to reduce the volume of data that was already ingested

Run a query to see what has been ingested for the Source. You can do this on the Metrics page or by using the Sumo Logic API.

For example, if the metrics Source is collect.graphite, enter the following query on the Metrics page to visualize the previously ingested data and verify that this is the Source you want to prevent from sending metrics.

_source=collect.graphite

Alternatively, you can use the following API query to show the first 100 times series from the Source. Substitute your own access ID/access key, Source name, and Sumo Logic endpoint (in bold). See Sumo Logic Endpoints for a list of endpoint URLs.

curl -u "accessid:accesskey" -X POST -H "Content-Type: application/json" -d '{"query":"_source=collect.graphite", "offset":0, "limit":100}' https://api.sumologic.com/api/v1/metrics/meta/catalog/query

b.  Use the following API query to delete the previously ingested time series for the Source (this query is supported only in the API).

curl -u "accessid:accesskey" -X POST -H "Content-Type: application/json" -d '{"delete":"_source=collect.graphite"}' https://api.sumologic.com/api/v1/metrics/meta/catalog/delete