When you want to filter, you can use a mathematical expression in your query to combine aggregate functions, comparison and boolean operators, and numerical values to help limit your search to the data you need.
The following metrics operators can filter your time series:
topk - take the top X time series
bottomk - take the bottom X time series
filter - take a specific math function of a time series (max, min, avg, sum)
And with these filters you can reduce down your time series. For example:
metric=cpu | filter min > 20 and max < 50
dep=prod metric=cpu_system | topk (10, max /avg * 2)
This helps you focus on areas of interest in your metrics data, and remove the additional “noise” of less important data. For example, if you want to see just the cpu metrics with an average over the time range queried is greater than 95:
metric=cpu | filter avg > 95
Or if you want to take the top 5 max values for your CPU metric:
metric=CPU_User cluster=cqsplitter | topk (5, max)
If you want the top 10 times of the max of the time series divided by double the average to see spikes in CPU usage:
metric=cpu | topk (10, max/avg*2)
_sourceCategory=mysources | topk(5, max(max, min(min, max)))
The filters are in red, the math functions are in blue.