Skip to main content
Sumo Logic

Blacklisted Metrics Sources

In some cases, Sumo Logic disables a metrics source to limit the number of ingested time series. This is referred to as blacklisting. Sumo blacklists a metric source has received too many unique time series.

For a Cloudwatch Metrics source, the limit is 500K time series. For other metric sources, the limit is 100K. In either case it is a  "lifetime limit", imposed based on the total count of unique time series received since the source was created; it is not a function of the timeframe over which the metrics were ingested.  

This page has information about the blacklisting process and how to resolve the problem.

Why Sumo blacklists a metrics source

Too many unique time series can occur with metrics whose names contain dynamically-generated strings, for example a timestamp. This is sometimes the case with Graphite metrics. Similarly, Dropwizard metric names sometimes contain names of threads.  With dynamically-generated metric names, a new time series is created with each new name. When this happend, the total number of time series can be subjected to unbounded growth over time. And the time series are typically of little use, given their ephemeral nature.

In the case of EC2 CloudWatch metrics, Amazon’s metric naming convention causes a new time series to be created for each EC2 instance.

When you run a metrics query that matches one or more blacklisted metrics sources, the following message is presented at the top metric query tab:

Some of your metrics Sources are sending too many unique time series and have been temporarily disabled. The data sent while Sources are disabled is not recoverable. Learn More.

missing-metrics-data.png

Click the Show Blacklisted Sources to see the source(s):

missing-metrics-data-blacklisted-sources.png

A dialog box prompts you to re-enable the Source. Selecting Re-enable Sources cancels the blacklist for the moment, but blacklisting will soon be reinstated if the underlying issue is not addressed.

Solve the underlying blacklisting issue

To address the underlying blacklisting issue, do one of the following:

Modify the source configuration

If the metric source is not a CloudWatch EC2 Source, modify the time series naming convention in the Source to exclude any dynamically inserted elements, such as date, timestamp, or thread names.

Replace EC2 CloudWatch metrics source with host metrics source

The recommendation described above (changing your time series naming convention) does not address the issue of too many unique time series from Amazon EC2 metrics. If you encounter this problem, we recommend that, instead of using Sumo's CloudWatch source for metrics, you install a Sumo Logic Collector in your EC2 instance, and configure a Sumo host metrics source on that collector.  The host metrics source send a suite of standard system metrics, such as CPU, disk, and network metrics. The information is comparable to that provided by CloudWatch. See Host Metrics Source for Installed Collectors for more information. This is a much more cost-effective way to collect host metrics.

Clean up existing time series data

It may be appropriate to delete previously ingested time series for a source that has been blacklisted.  

You can run a query to see what has been ingested for source, on the  Metrics page or by using the Sumo Logic API.

To query a source from Sumo UI

For example, if the metrics Source is collect.graphite, enter the following query on the Metrics page to visualize the previously ingested data and verify that this is the Source you want to prevent from sending metrics.

_source=collect.graphite

To query a source using Sumo API

You can use the following API query to show the first 100 times series from the Source. Substitute your own access ID/access key, Source name, and Sumo Logic endpoint (in bold). See Sumo Logic Endpoints for a list of endpoint URLs.

curl -u "accessid:accesskey" -X POST -H "Content-Type: application/json" -d '{"query":"_source=collect.graphite", "offset":0, "limit":100}' https://api.sumologic.com/api/v1/metrics/meta/catalog/query

To delete previously ingested time series for the source

You can use the following API query to delete previously ingested time series for the Source (this query is supported only in the API).

curl -u "accessid:accesskey" -X POST -H "Content-Type: application/json" -d '{"delete":"_source=collect.graphite"}' https://api.sumologic.com/api/v1/metrics/meta/catalog/delete