Skip to main content

Cassandra - OpenTelemetry Collector

Thumbnail icon Thumbnail icon

The Cassandra - OpenTelemetry app is a log based app that helps you monitor the availability, performance, health, and resource utilization of your Cassandra clusters. Preconfigured dashboards provide insight into resource utilization, cache/Gossip/Memtable statistics and Error and warnings. Cassandra logs are sent to Sumo Logic through OpenTelemetry filelog receiver.

The app supports Logs from the open-source version of Cassandra. The App is tested on the 3.11.10 version of Cassandra.

Schematics

Fields creation in Sumo Logic for Cassandra

Following are the Fields which will be created as part of Cassandra App install if not already present:

  • db.cluster.name. User configured. Enter a name to identify this Cassandra cluster. This cluster name will be shown in the Sumo Logic dashboards.
  • db.system. Has fixed value of cassandra.
  • deployment.environment. User configured. Through this Cassandra cluster is identified by the environment where it resides. For example: dev, prod or qa.
  • sumo.datasource. Has fixed value of cassandra.

Prerequisites

Cassandra has three main logs: system.log, debug.log, and gc.log which hold general logging messages, debugging logging messages, and java garbage collection logs respectively.

These logs by default live in ${CASSANDRA_HOME}/logs, but most Linux distributions relocate logs to /var/log/cassandra. Operators can tune this location as well as what levels are logged using the provided logback.xml file. For more details on Cassandra logs, see this link.

For Linux systems with ACL Support, the otelcol install process should have created the ACL grants necessary for the otelcol system user to access default log locations. You can verify the active ACL grants using the getfacl command. Install the ACL in your Linux environment, if not installed.

The required ACL may not be supported for some rare cases, for example, Linux OS Distro, which is officially not supported by Sumo Logic. In this case, you can run the following command to explicitly grant the permissions.

sudo setfacl -R -m d:u:otelcol-sumo:r-x,d:g:otelcol-sumo:r-x,u:otelcol-sumo:r-x,g:otelcol-sumo:r-x <PATH_TO_LOG_FILE>

Run the above command for all the log files in the directory that need to be ingested, which are not residing in the default location.

If Linux ACL Support is not available, traditional Unix-styled user and group permission must be modified. It should be sufficient to add the otelcol system user to the specific group that has access to the log files.

Collection configuration and app installation

As part of data collection setup and app installation, you can select the App from App Catalog and click on Install App. Follow the steps below.

Step 1: Set up Collector

note

If you want to use an existing OpenTelemetry Collector, you can skip this step by selecting the Use an existing Collector option.

To create a new Collector:

  1. Select the Add a new Collector option.
  2. Select the platform where you want to install the Sumo Logic OpenTelemetry Collector.

This will generate a command that you can execute in the machine environment you need to monitor. Once executed, it will install the Sumo Logic OpenTelemetry Collector.

Collector

Step 2: Configure integration

In this step, you will configure the yaml required for Cassandra Collection. Path of the log file configured to capture Cassandra logs needs to be given here.

Below are the inputs required:

  • The path to system.log is required here. This file is typically located in /var/log/cassandra. If you're using a customized path, check the respective conf file for this information.

You can add any custom fields which you want to be tagged with the data ingested in Sumo. Click on the Download YAML File button to get the yaml file.

YAML

Step 3: Send logs to Sumo

Once you have downloaded the yaml file as described in the previous step, follow the below steps based on your platform.

  1. Copy the yaml to /etc/otelcol-sumo/conf.d/ folder for the Cassandra instance which needs to be monitored.
  2. Restart the collector using:
 sudo systemctl restart otelcol-sumo

After successfully executing the above command, Sumo Logic will start receiving data from your host machine.

Click Next. This will install the app (dashboards and monitors) to your Sumo Logic Org.

Dashboard panels will start to fill automatically. It's important to note that each panel fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but within 20 minutes, you'll see full graphs and maps.

Sample Log

  INFO [ScheduledTasks:1] 2023-01-08 09:18:47,347 StatusLogger.java:101 - system.schema_aggregates

Sample queries 

Following is the query from Cassandra App's overview Dashboard's Nodes Up Panel:

%"sumo.datasource"=cassandra %"deployment.environment"=* %"db.cluster.name"=*   "INFO" | json "log" as _rawlog nodrop 
| if (isEmpty(_rawlog), _raw, _rawlog) as _raw
| parse regex field=_raw "(?<level>[A-Z]*) *\[(?<thread_name>[^\]]*?)[:_-]?(?<thread_id>[0-9]*)\] (?<Date>.{10} .{12}) *(?<source_file>[^:]*):(?<source_line>[0-9]*) - (?<message>.*)"
| if (message matches "InetAddress * is now UP",1,0) as UP
| timeslice 1d
| sum(UP) as UP by _timeslice
| sort by _timeslice asc

Viewing Cassandra Dashboards

Overview

The Cassandra - Overview dashboard provides an at-a-glance view of Cassandra backend and frontend HTTP error codes percentage, visitor location, URLs, and clients causing errors.

Use this dashboard to:

  • Identify number of nodes which are up and down
  • Gain insights into Memory - Init, used, Max and committed
  • Gain insights into the error and warning logs by thread and Node activity
Collector

Cache Stats

The Cassandra - Cache Stats dashboard provides insight into the database cache status, schedule, and items.

Use this dashboard to:

  • Monitor Cache performance.
  • Identify Cache usage statistics.
Cache Stats

Errors and Warnings

The Cassandra - Errors and Warnings dashboard provides details of the database errors and warnings.

Use this dashboard to:

  • Review errors and warnings generated by the server.
  • Review the Threads errors and warning events.
Errors and Warnings

Gossip

The Cassandra - Gossip dashboard provides details about communication between various cassandra nodes.

Use this dashboard to:

  • Determine nodes with errors resulting in failures.
  • Review the node activity and pending tasks.
Gossip

Memtable

The Cassandra - Memtable dashboard provides insights into memtable statistics.

Use this dashboard to:

  • Review flush activity and memtable status.
Memtable

Resource Usage

The Cassandra - Resource Usage dashboard provides details of resource utilization across Cassandra clusters.

Use this dashboard to:

  • Identify resource utilization. This can help you to determine whether resources are over-allocated or under-allocated.
Resource Usage
Status
Legal
Privacy Statement
Terms of Use

Copyright © 2024 by Sumo Logic, Inc.