Apache Tomcat - OpenTelemetry Collector

The Apache Tomcat app is a logs app that helps you get insight into visitor locations, traffic patterns, errors, resource utilization, garbage collection, web server operations, and access from known malicious sources.

Tomcat logs are sent to Sumo Logic through Opentelemetry filelog receiver.

Fields Created in Sumo Logic for Tomcat

The following are the Fields that will be created as part of the Tomcat App install, if not already present.

webengine.cluster.name. User configured. Set a value of the cluster where your Tomcat instance resides. This will be tagged along with the data sent to Sumo.
webengine.system. Has a fixed value of tomcat.
sumo.datasource. Has a fixed value of tomcat.

Prerequisites

The Sumo Logic app for Apache Tomcat uses three types of logs:

Tomcat Access logs. Log format description. Recommended pattern used is pattern="common".
Tomcat Catalina.out logs. Log format description
Tomcat Garbage Collection (GC) logs. Log format description

By default, Tomcat logs are stored in /usr/share/tomcat/logs/ The default directory for log files is listed in the /usr/share/tomcat/conf/logging.properties file.

For Linux systems with ACL Support, the otelcol install process should have created the ACL grants necessary for the otelcol system user to access default log locations. You can verify the active ACL grants using the getfacl command. Install the ACL in your Linux environment, if not installed.

The required ACL may not be supported for some rare cases, for example, Linux OS Distro, which is officially not supported by Sumo Logic. In this case, you can run the following command to explicitly grant the permissions.

sudo setfacl -R -m d:u:otelcol-sumo:r-x,d:g:otelcol-sumo:r-x,u:otelcol-sumo:r-x,g:otelcol-sumo:r-x <PATH_TO_LOG_FILE>

Run the above command for all the log files in the directory that need to be ingested, which are not residing in the default location.

If Linux ACL Support is not available, traditional Unix-styled user and group permission must be modified. It should be sufficient to add the otelcol system user to the specific group that has access to the log files.

For Windows systems, log files which are collected should be accessible by the SYSTEM group. Use the following set of PowerShell commands if the SYSTEM group does not have access.

$NewAcl = Get-Acl -Path "<PATH_TO_LOG_FILE>"
# Set properties
$identity = "NT AUTHORITY\SYSTEM"
$fileSystemRights = "ReadAndExecute"
$type = "Allow"
# Create new rule
$fileSystemAccessRuleArgumentList = $identity, $fileSystemRights, $type
$fileSystemAccessRule = New-Object -TypeName System.Security.AccessControl.FileSystemAccessRule -ArgumentList $fileSystemAccessRuleArgumentList
# Apply new rule
$NewAcl.SetAccessRule($fileSystemAccessRule)
Set-Acl -Path "<PATH_TO_LOG_FILE>" -AclObject $NewAcl

Collection configuration and app installation

As part of data collection setup and app installation, you can select the App from App Catalog and click on Install App. Follow the steps below.

Step 1: Set up Collector

note

If you want to use an existing OpenTelemetry Collector, you can skip this step by selecting the Use an existing Collector option.

To create a new Collector:

Select the Add a new Collector option.
Select the platform where you want to install the Sumo Logic OpenTelemetry Collector.

This will generate a command that you can execute in the machine environment you need to monitor. Once executed, it will install the Sumo Logic OpenTelemetry Collector.

Step 2: Configure integration

In this step, you will configure the yaml required for Tomcat Collection.

The path of the log file configured to capture tomcat logs is needed to be given here.

The files are typically located in /usr/share/tomcat/logs/*. If you're using a customized path, check the Tomcat.conf file for this information.

You can add any custom fields which you want to tag along with the data ingested in Sumo. Click on the Download YAML File button to get the yaml file.

Step 3: Send logs and metrics to Sumo Logic

Once you have downloaded the yaml file as described in the previous step, follow the below steps based on your platform.

Copy the yaml file to /etc/otelcol-sumo/conf.d/ folder in the Tomcat instance which needs to be monitored.
Restart the collector using:

sudo systemctl restart otelcol-sumo

Copy the yaml file to C:\ProgramData\Sumo Logic\OpenTelemetry Collector\config\conf.d folder in the machine which needs to be monitored.
Restart the collector using:

Restart-Service -Name OtelcolSumo

Copy the yaml file to /etc/otelcol-sumo/conf.d/ folder in the Tomcat instance which needs to be monitored.
Restart the otelcol-sumo process using the below command:

 otelcol-sumo --config /etc/otelcol-sumo/sumologic.yaml --config "glob:/etc/otelcol-sumo/conf.d/*.yaml"

Copy the yaml file into your Chef cookbook files directory files/<downloaded_yaml_file>.
Use a Chef file resource in a recipe to manage it. cookbook_file '/etc/otelcol-sumo/conf.d/<downloaded_yaml_file>' do mode 0644 notifies :restart, 'service[otelcol-sumo]', :delayed end
Add the recipe to your collector setup to start collecting the data. Every team typically has their established way of applying the Chef recipe. The resulting Chef recipe should look something like:
```
cookbook_file '/etc/otelcol-sumo/conf.d/<downloaded_yaml_file>' do
    mode 0644
    notifies :restart, 'service[otelcol-sumo]', :delayed
end
```

Place the file into your Ansible playbook files directory.

Run the Ansible playbook.

ansible-playbook -i inventory install_sumologic_otel_collector.yaml
-e '{"installation_token": "<YOUR_TOKEN>", "collector_tags": {<YOUR_TAGS>}, "src_config_path": "files/conf.d"}'

Place the file into your Puppet module files directory modules/install_otel_collector/files/<downloaded_yaml>.

Use a Puppet file resource to manage it.

file { '/etc/otelcol-sumo/conf.d/<downloaded_yaml_file>':
    ensure => present,
    source => 'puppet:///modules/install_otel_collector/<downloaded_yaml_file>',
    mode => '0644',
    notify => Service[otelcol-sumo],
  }

Apply the Puppet manifest. Every team typically has their established way of applying the Puppet manifest. The resulting Puppet manifest should look something like:

node 'default' {
    class { 'install_otel_collector'
      installation_token => '<YOUR_TOKEN>',
      collector_tags => { <YOUR_TAGS> },
    }

    service { 'otelcol-sumo':
      provider => 'systemd',
      ensure => running,
      enable => true,
      require => Class['install_otel_collector'],
    }

    file { '/etc/otelcol-sumo/conf.d/<downloaded_yaml_file>':
      ensure => present,
      source => 'puppet:///modules/install_otel_collector/<downloaded_yaml_file>',
      mode => '0644',
      notify => Service[otelcol-sumo],
    }
  }

After successfully executing the above command, Sumo Logic will start receiving data from your host machine.

Click Next. This will install the app (dashboards and monitors) to your Sumo Logic Org.

Dashboard panels will start to fill automatically. It's important to note that each panel fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but within 20 minutes, you'll see full graphs and maps.

Sample log messages

Dec 13, 2022 03:53:03 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 63394 ms
179.105.33.169 - - [13/Dec/2022:15:53:03 +0000] "PUT /aboutus/ HTTP/1.1" 404 76246453 "http://bing.com/Nutch-1.4" "-"

Sample queries

 %"sumo.datasource"=tomcat %"webengine.cluster.name"=*
| json "log" as _rawlog nodrop 
| if (isEmpty(_rawlog), _raw, _rawlog) as _raw 
| parse regex "(?<time>\w+\s+\d+,\s+\d+\s+\d+:\d+:\d+\s+\w+)\s+(?<component>[\w\S]+)\s+(?<method>[\w\S]+)\s+(?<loglevel>\w+):\s+(?<message>[\s\w]+)(?:$|\n(?<thrown>[\s\S]+)$)"
| where (loglevel != "INFO")
| timeslice 1h
| count by _timeslice, component
| transpose row _timeslice column component

Viewing Tomcat dashboards

Overview

The Apache Tomcat - Overview Dashboard provides a high-level view of information on visitor geographic locations, responses over time, and number of error codes and top urls causing error.

Use this dashboard to:

Analyze http request about status code.
Gain insights into originated traffic location by region. This can help you allocate computer resources to different regions according to their needs.
Gain insights into Client, Server Responses on Tomcat Server. This helps you identify errors in Tomcat Server.

Visitor Locations

The Apache Tomcat - Visitor Locations Dashboard provides a high-level view of Tomcat visitor geographic locations both worldwide and in the United States. Dashboard panels also show graphic trends for visits by country over time and visits by US region over time.

Worldwide. Uses a geo lookup operation to display worldwide visitor locations by IP address on a map of the world, which allows you to see a count of hits per location for the last 24 hours.
Visits by Country Over Time. Displays the number of visitors by country in a stacked column chart on a timeline for the last hour.
United States. Uses a geo lookup operation to display US visitor locations by IP address on a map of the world, which allows you to see a count of hits per location for the last 24 hours.
Visits by US State Over Time. Displays the number of US visitors by state in a stacked column chart on a timeline for the last hour.

Visitor Traffic Insight

The Apache Tomcat - Visitor Traffic Insight Dashboard provides detailed information on the top documents accessed, top referrers, top search terms from popular search engines, and the media types served.

Bytes Served. Displays bytes served in a single chart on a timeline for the last 60 minutes.

HTTP Methods. Shows the number of methods over time in a pie chart on a timeline for the last 60 minutes.

Top 5 url. Provides a list of the top 5 URLs being accessed by your visitors in a bar chart for the 60 minutes.

Media Types Served. Displays a list of file types being served in a pie chart for the 60 minutes.

Top 5 Referrers. Shows a list of the top 5 referring websites by URL in a bar chart for the 60 minutes.

Top 10 Search Terms from Popular Search Engines. Displays a list of the top 10 search terms and their count from search engines such as Google, Bing, and Yahoo in an aggregation table for the past hour.

Web Server Operations

The Apache Tomcat - Web Server Operations Dashboard provides a high-level view combined with detailed information on the top ten bots, geographic locations and data for clients with high error rates, server errors over time, and non 200 response code status codes. Dashboard panels also show information on server error logs, error log levels, error responses by server, and the top URIs responsible for 404 responses.

Non 200 Response Status Codes. Displays the number of non-200 response status codes in a bar chart for the past hour.

Client Locations - 4xx Errors. Uses a geo lookup operation to display the location of clients with 4xx errors by IP address on a map of the world, which allows you to see a count of hits per location for the last hour.

Server Errors Over Time. Provides information on the type and number of server errors in a column chart on a line chart for the past hour.

Error Responses by Server. Shows error responses and their distribution by server in a line chart for the past hour.

Top 5 Clients Cause 4xx Errors. Displays a list of the top 5 clients that have 4xx errors in a bar chart for the past hour.

Top 5 URIs Causing 404 Responses. Provides a list of the top 5 URIs with 404 response types in a pie chart for the past hour.

Logs Timeline Analysis

The Apache Tomcat - Logs Timeline Analysis dashboard provides a high-level view of the activity and health of Apache Tomcat servers on your network. Dashboard panels display visual graphs and detailed information on traffic volume and distribution, responses over time, as well as time comparisons for visitor locations and server hits.

Use this dashboard to:

To understand the traffic distribution across servers, provide insights for resource planning by analyzing data volume and bytes served.
Gain insights into originated traffic location by region. This can help you allocate compute resources to different regions according to their needs.

Outlier Analysis

The Apache Tomcat - Outlier Analysis dashboard provides a high-level view of Apache Tomcat server outlier metrics for bytes served, number of visitors, and server errors. You can select the time interval over which outliers are aggregated, then hover the cursor over the graph to display detailed information for that point in time.

Use this dashboard to:

Detect outliers in your infrastructure with Sumo Logic's machine learning algorithm.
To identify outliers in incoming traffic and the number of errors encountered by your servers.

Catalina Overview

The Apache Tomcat - Catalina dashboard provides information about events such as the startup and shutdown of the Apache Tomcat application server, the deployment of new applications, or the failure of one or more subsystems.

Log Levels. Displays log levels types (Info, Severe, and Warning) in a pie chart for the last 24 hours.

Non-INFO Errors. Shows the number and type of errors (Severe or Warning) in a stacked column chart on a timeline for the last 24 hours.

Component Errors. Provides information on errors by component in a pie chart for the last 24 hours.

Errors by Component. Displays Info level errors by component in a stacked column chart on a timeline for the last 24 hours.

Top 10 Recent Exceptions. Shows the top 10 most recent exceptions in an aggregation table with columns for time, log level, message, method, source file, and thrown for the last 24 hours.

Exceptions. Provides the number of exceptions in a column chart on a timeline for the last seven days.

Average Server Startup Time. Displays the average server startup time per second by day as a column chart on a timeline for the last seven days.

Server State Events Over Time. Shows server state events (shutdown or startup) in a stacked column chart on a timeline for the last seven days.

Garbage Collection

The Apache Tomcat - Garbage Collector dashboard provides information on the garbage collection of the Java Virtual Machine.

Top 10 Host - High GC Time. Displays the top 10 hosts with high garbage collection operation time as a bar chart for the last 12 hours.

Top 10 Hosts - Low Average JVM Up-Time. Shows the top 10 hosts by low average JVM up-time as a bar chart for the last 12 hours.

Total GC Operation Time. Provides the total garbage collection operation time by time slices of 15 minutes in a column chart on a timeline for the last 12 hours.

Total GC Operations. Displays the total number of times Full-GC and Minor-GC collection processes are executed in time slices of 15 minutes on a stacked column chart on a timeline for the past 12 hours.

Heap. Shows the total heap memory utilization just before garbage collection was executed vs. total heap memory utilization after garbage collection was executed, in a line chart on a timeline for the last 12 hours.

PS Young Gen. PS Young Gen also refers to "New Space," which is composed of Eden-Space and two Survivor-Spaces of identical size, usually called From and To. This panel shows Young Gen memory utilization just before garbage collection was executed vs. Young Gen memory utilization after garbage collection was executed. This part of the heap always gets garbage collected.

Par Old Gen. Par Old Gen is also referred to as "Tenured Space". This panel shows Old Gen memory utilization just before garbage collection was executed vs. Old Gen memory utilization after garbage collection was executed.

PS Perm Gen. PS Perm Gen is also referred to as "Permanent Space". This panel shows Perm Gen memory utilization just before garbage collection was executed vs. Perm Gen memory utilization after garbage collection was executed.

Threat intel

The Apache Tomcat - Threat Intel dashboard provides an at-a-glance view of threats to Apache Tomcat servers on your network. Dashboard panels display the threat count over a selected time period, geographic locations where threats occurred, source breakdown, actors responsible for threats, severity, and a correlation of IP addresses, method, and status code of threats.

Use this dashboard to:

To gain insights and understand threats in incoming traffic and discover potential IOCs. Incoming traffic requests are analyzed using the Sumo - Crowdstrikes threat feed.

Fields Created in Sumo Logic for Tomcat​

Prerequisites​

Collection configuration and app installation​

Step 1: Set up Collector​

Step 2: Configure integration​

Step 3: Send logs and metrics to Sumo Logic​

Sample log messages​

Sample queries​

Viewing Tomcat dashboards​

Overview​

Visitor Locations​

Visitor Traffic Insight​

Web Server Operations​

Logs Timeline Analysis​

Outlier Analysis​

Catalina Overview​

Garbage Collection​

Threat intel​

Fields Created in Sumo Logic for Tomcat

Prerequisites

Collection configuration and app installation

Step 1: Set up Collector

Step 2: Configure integration

Step 3: Send logs and metrics to Sumo Logic

Sample log messages

Sample queries

Viewing Tomcat dashboards

Overview

Visitor Locations

Visitor Traffic Insight

Web Server Operations

Logs Timeline Analysis

Outlier Analysis

Catalina Overview

Garbage Collection

Threat intel