Collect metrics for JMX
This page explains how to collect metrics from JMX and ingest them into Sumo Logic to use with the predefined dashboards and searches in the JMX app.
Collection Process overview
You can collect JMX metrics using the Jolokia Plugin for telegraf. You can configure the metrics in the Jolokia Plugin configuration and these metrics are exposed on a URL.
These metrics are then forwarded to Sumo Logic HTTP Source. The collection starts at the current time.
Configuring metrics collection for JMX includes the following steps:
- Step1: Metric Collection in Non-Kubernetes Environment
- Step2: Metric Collection in Kubernetes Environment
Step 1: Metric Collection in Non-Kubernetes Environment
This section provides instructions for configuring metrics collection for the Sumo Logic App for JMX. Follow the below instructions to set up the metric collection.
- Configure metrics for JMX
- Configure a Hosted Collector
- Configure a Http Logs and Metrics Source
- Install Telegraf
- Configure Telegraf and Forward Metrics to Sumo Logic
1. Configure metrics for JMX
To configure JMX metrics using Jolokia please perform the below steps:
- Download the latest Jolokia JVM agent from Jolokia.
- You can attach the Jolokia JVM agent jar as a Java Agent to your application.
- Replace parameter agentContext with a value as per your environment or jmx instance.
java ... -javaagent:jolokia-jvm-1.6.2-agent.jar=port=8778,host=localhost,agentContext=/${agentContext} ...
Alternatively, you can also attach process to the Jolokia JVM agent.
# List available applications java -jar ./jolokia-jvm-1.6.2-agent.jar list # 156 ./jolokia-jvm-1.6.2-agent.jar list # 6 org.apache.catalina.startup.Bootstrap start # Attach to the application which should be monitored java -jar ./jolokia-jvm-1.6.2-agent.jar --agentContext /${agentContext} start 6 #Jolokia is already attached to PID 723#http://127.0.0.1:8778/${agentContext}/
More information can be found using a JVM agent.
Make a note of the URL that will be used in telegraf configuration.
2. Configure a Hosted Collector
To create a new Sumo Logic hosted collector, perform the steps in the Configure a Hosted Collector section of the Sumo Logic documentation.
3. Configure a HTTP Logs and Metrics Source
Create a new HTTP Logs and Metrics Source in the hosted collector created above by following these instructions.
- Make a note of HTTP Source URL.
4. Install Telegraf
Use the following steps to install Telegraf.
5. Configure and start Telegraf
Create a file called telegraf.conf and add the appropriate configuration. The following is a basic example:
[agent] interval = "60s" [[inputs.jolokia2_agent]] urls = ["<Jolokia URL from Step 1>"] [[inputs.jolokia2_agent.metric]] name = "java_lang_ClassLoading" mbean = "java.lang:type=ClassLoading" paths = ["LoadedClassCount", "TotalLoadedClassCount", "UnloadedClassCount"] [[inputs.jolokia2_agent.metric]] name = "java_lang_Compilation" mbean = "java.lang:type=Compilation" paths = ["TotalCompilationTime"] [[inputs.jolokia2_agent.metric]] name = "java_lang_GarbageCollector" mbean = "java.lang:name=*,type=GarbageCollector" paths = ["CollectionCount", "CollectionTime", "LastGcInfo"] tag_keys = ["name"] [[inputs.jolokia2_agent.metric]] name = "java_lang_MemoryPool" mbean = "java.lang:name=*,type=MemoryPool" paths = ["CollectionUsage", "CollectionUsageThresholdSupported", "PeakUsage", "Usage", "UsageThresholdSupported"] tag_keys = ["name"] [[inputs.jolokia2_agent.metric]] name = "java_lang_Memory" mbean = "java.lang:type=Memory" paths = ["HeapMemoryUsage", "NonHeapMemoryUsage", "ObjectPendingFinalizationCount"] [[inputs.jolokia2_agent.metric]] name = "java_lang_OperatingSystem" mbean = "java.lang:type=OperatingSystem" paths = ["AvailableProcessors", "CommittedVirtualMemorySize", "FreePhysicalMemorySize", "FreeSwapSpaceSize", "MaxFileDescriptorCount", "OpenFileDescriptorCount", "ProcessCpuLoad", "ProcessCpuTime", "SystemCpuLoad", "SystemLoadAverage", "TotalPhysicalMemorySize", "TotalSwapSpaceSize"] [[inputs.jolokia2_agent.metric]] name = "java_lang_Runtime" mbean = "java.lang:type=Runtime" paths = ["BootClassPathSupported", "StartTime", "Uptime"] [[inputs.jolokia2_agent.metric]] name = "java_lang_Threading" mbean = "java.lang:type=Threading" paths = ["CurrentThreadCpuTime", "CurrentThreadUserTime", "DaemonThreadCount", "ObjectMonitorUsageSupported", "PeakThreadCount", "SynchronizerUsageSupported", "ThreadContentionMonitoringEnabled", "ThreadContentionMonitoringSupported", "ThreadCount", "ThreadCpuTimeEnabled", "ThreadCpuTimeSupported", "TotalStartedThreadCount"] # Metrics which are unavailable for some of the jvm implementations ## Added in jdk14 [[inputs.jolokia2_agent.metric]] name = "java_lang_OperatingSystem" mbean = "java.lang:type=OperatingSystem" paths = ["FreeMemorySize", "TotalMemorySize"] ## not available for jdk8 [[inputs.jolokia2_agent.metric]] name = "java_lang_Runtime" mbean = "java.lang:type=Runtime" paths = ["Pid"] ## Added in jdk14 [[inputs.jolokia2_agent.metric]] name = "java_lang_Threading" mbean = "java.lang:type=Threading" paths = ["CurrentThreadAllocatedBytes"] ## Not available for adoptopenjdk-openj9 [[inputs.jolokia2_agent.metric]] name = "java_lang_Threading" mbean = "java.lang:type=Threading" paths = ["ThreadAllocatedMemoryEnabled", "ThreadAllocatedMemorySupported"] # The processor regex converts url like http://127.0.0.1:8778/${agentContext} to agentContext [[processors.regex]] [[processors.regex.tags]] key = "jolokia_agent_url" pattern = '.*?([a-zA-Z0-9-_]+)[\/]?$' replacement = "${1}" [[outputs.sumologic]] url = "<URL Created in Step 3>" data_format = "prometheus"
- interval. This is the frequency to send data to Sumo Logic, in this example, we will send the metrics every 60 seconds. Please refer to this doc for more properties that can be configured in the Telegraf agent globally.
- urls. The url to the Jolokia server. This can be a comma-separated list to connect to multiple Jolokia servers. Please refer to this doc for more information on configuring the Jolokia input plugin for Telegraf.
- url. This is the HTTP source URL created in step 3. Refer to this doc for more information on configuring the Sumo Logic Telegraf output plugin.
- data_format. The format to use when sending data to Sumo Logic. Please refer to this doc for more information on configuring the Sumo Logic Telegraf output plugin.
After you have finalized your telegraf.conf file, you can run the following command to start telegraf.
telegraf --config /path/to/telegraf.conf
Step 2: Metric Collection in Kubernetes Environment
The following steps assume you are collecting JMX metrics from a Kubernetes environment. In a Kubernetes environment, we use the Telegraf Operator, which is packaged with our Kubernetes collection. You can learn more about this here.
- Set up Kubernetes Collection with the Telegraf Operator.
- On your Pods, add the following annotations to configure Telegraf.
annotations: telegraf.influxdata.com/inputs: |+ [[inputs.jolokia2_agent]] urls = ["http://127.0.0.1:8778/jolokia"] [[inputs.jolokia2_agent.metric]] name = "java_lang_ClassLoading" mbean = "java.lang:type=ClassLoading" paths = ["LoadedClassCount", "TotalLoadedClassCount", "UnloadedClassCount"] [[inputs.jolokia2_agent.metric]] name = "java_lang_Compilation" mbean = "java.lang:type=Compilation" paths = ["TotalCompilationTime"] [[inputs.jolokia2_agent.metric]] name = "java_lang_GarbageCollector" mbean = "java.lang:name=*,type=GarbageCollector" paths = ["CollectionCount", "CollectionTime", "LastGcInfo"] tag_keys = ["name"] [[inputs.jolokia2_agent.metric]] name = "java_lang_MemoryPool" mbean = "java.lang:name=*,type=MemoryPool" paths = ["CollectionUsage", "CollectionUsageThresholdSupported", "PeakUsage", "Usage", "UsageThresholdSupported"] tag_keys = ["name"] [[inputs.jolokia2_agent.metric]] name = "java_lang_Memory" mbean = "java.lang:type=Memory" paths = ["HeapMemoryUsage", "NonHeapMemoryUsage", "ObjectPendingFinalizationCount"] [[inputs.jolokia2_agent.metric]] name = "java_lang_OperatingSystem" mbean = "java.lang:type=OperatingSystem" paths = ["AvailableProcessors", "CommittedVirtualMemorySize", "FreePhysicalMemorySize", "FreeSwapSpaceSize", "MaxFileDescriptorCount", "OpenFileDescriptorCount", "ProcessCpuLoad", "ProcessCpuTime", "SystemCpuLoad", "SystemLoadAverage", "TotalPhysicalMemorySize", "TotalSwapSpaceSize"] [[inputs.jolokia2_agent.metric]] name = "java_lang_Runtime" mbean = "java.lang:type=Runtime" paths = ["BootClassPathSupported", "StartTime", "Uptime"] [[inputs.jolokia2_agent.metric]] name = "java_lang_Threading" mbean = "java.lang:type=Threading" paths = ["CurrentThreadCpuTime", "CurrentThreadUserTime", "DaemonThreadCount", "ObjectMonitorUsageSupported", "PeakThreadCount", "SynchronizerUsageSupported", "ThreadContentionMonitoringEnabled", "ThreadContentionMonitoringSupported", "ThreadCount", "ThreadCpuTimeEnabled", "ThreadCpuTimeSupported", "TotalStartedThreadCount"] # Metrics which are unavailable for some of the jvm implementations ## Added in jdk14 [[inputs.jolokia2_agent.metric]] name = "java_lang_OperatingSystem" mbean = "java.lang:type=OperatingSystem" paths = ["FreeMemorySize", "TotalMemorySize"] ## not available for jdk8 [[inputs.jolokia2_agent.metric]] name = "java_lang_Runtime" mbean = "java.lang:type=Runtime" paths = ["Pid"] ## Added in jdk14 [[inputs.jolokia2_agent.metric]] name = "java_lang_Threading" mbean = "java.lang:type=Threading" paths = ["CurrentThreadAllocatedBytes"] ## Not available for adoptopenjdk-openj9 [[inputs.jolokia2_agent.metric]] name = "java_lang_Threading" mbean = "java.lang:type=Threading" paths = ["ThreadAllocatedMemoryEnabled", "ThreadAllocatedMemorySupported"] # The processor regex converts url like http://127.0.0.1:8778/jolokia to hostname [[processors.override]] [processors.override.tags] jolokia_agent_url = "$HOSTNAME" telegraf.influxdata.com/class: sumologic-prometheus telegraf.influxdata.com/limits-cpu: '750m' prometheus.io/scrape: "true" prometheus.io/port: "9273"
- telegraf.influxdata.com/inputs - This contains the required configuration for the Telegraf Jolokia Input plugin. Please refer to this doc for more information on configuring the Jolokia input plugin for Telegraf. Note: As telegraf will be run as a sidecar the host should always be localhost
- telegraf.influxdata.com/class: sumologic-prometheus - This instructs the Telegraf operator what output to use. This should not be changed.
- prometheus.io/scrape: "true" - This ensures our Prometheus will scrape the metrics.
- prometheus.io/port: "9273" - This tells Prometheus what ports to scrape on. This should not be changed.
Sample Metric Names
Metric Type | Sample |
---|---|
CPU | java_lang_OperatingSystem_ProcessCpuLoad |
GC | java_lang_GarbageCollector_LastGcInfo_duration |
Memory | java_lang_Memory_NonHeapMemoryUsage_committed |
Threads | java_lang_Threading_ThreadCount |
ClassLoader | java_lang_ClassLoading_LoadedClassCount |
Query sample
This query sample is from the CPU Load Vs Current Threads panel of JMX - Overview dashboard.
_sourceCategory=Labs/jmx/metrics metric=java_lang_OperatingSystem_TotalMemorySize jolokia_agent_url={{Server}} | eval (_value / 1024 / 1024 / 1024) | sum