Skip to main content

Puppet - OpenTelemetry Collector

Thumbnail icon Thumbnail icon

Puppet is a software configuration management tool. Puppet can provision infrastructure and enforce desired configurations across new and existing servers. The Sumo Logic App for Puppet helps you monitor Puppet events. Puppet logs are sent to Sumo Logic through OpenTelemetry filelog receiver.

Schematics

Fields creation in Sumo Logic for Puppet​

Following are the tags which will be created as part of Puppet App install if not already present. 

  • sumo.datasource. Has a fixed value of Puppet.

Prerequisites​

This section provides instructions for configuring log collection for Puppet running on a non-Kubernetes environment for the Sumo Logic App for Puppet. F

We use OpenTelemetry collector to gather the following data from Puppet:

  • Puppet Server logs. For more information about the logs location, see Puppet Server Logs.
  • Puppet Server Access logs. For more information about the logs location, see Puppet Server Logs.
  • Puppet Reports. Puppet generates reports in YAML format. SumoLogic supports report format 10. This is the format of reports output by Puppet versions 5.5.3 and newer. It is backward compatible with report format 9 (in Puppet versions 5.5.0 to 5.5.2). For more information about the puppet reports, see Puppet Reports.

The default Puppet Server Access log file is: /var/log/puppetlabs/puppetserver/puppetserver-access.log. If your Puppet Server Access logs are located elsewhere, please note the path this will be used later while app installation.

Puppet reports are in YAML format. They must be converted into JSON format before Sumo ingests them. You can convert the YAML files using the Sumo-provided shell script below:

MaxFileSize=20
log_file_name=puppet_rpt_conversion.log
#Get size in bytes**
   file_size=`du -b puppet_rpt_conversion.log | tr -s '\t' ' ' | cut -d' ' -f1`
   file_size=$(($file_size/1048576))
   if [ $file_size -gt "$MaxFileSize" ];then  
       timestamp=`date +%s`
       mv puppet_rpt_conversion.log puppet_rpt_conversion.log.$timestamp
   fi
echo "YAML to JSON conversion started!!" >> "$log_file_name"
echo "Start Time: $(date)" >> "$log_file_name"
if [ ! -e last_run_date.tmp ]
then
date "+%Y-%m-%d" --date=1970-01-01 > last_run_date.tmp;
fi
find . -maxdepth 2 -type f -name "*.yaml" -newermt "$(cat last_run_date.tmp)" |
while read f;
do
if [ -s "$f" ]
then
sed 1d "$f" > tmpfile;
ruby -rjson -ryaml -e "resource_list = ''
json = JSON.generate({});data = JSON.parse(json);
obj = JSON.parse(YAML.load_file('tmpfile').to_json);
data['host'] = obj['host'];
data['time'] = obj['time'];
data['configuration_version'] = obj['configuration_version'];
data['transaction_uuid'] = obj['transaction_uuid'];
data['report_format'] = obj['report_format'];
data['puppet_version'] = obj['puppet_version'];
data['status'] = obj['status'];
data['transaction_completed'] = obj['transaction_completed'];
data['noop'] = obj['noop'];
data['noop_pending'] = obj['noop_pending'];
data['environment'] = obj['environment'];
data['metrics'] = obj['metrics'];
data['resource_list'] = resource_list;
i = 0;
obj['resource_statuses'].each{
 |j|
if i == 0 then
 tempdata = j[0];
else
 tempdata = ',' + j[0];
end
data['resource_list'] << tempdata;
i = i + 1;
};
puts data.to_json;
obj['resource_statuses'].each{
 |y|  logsres = y[1];
 logsres['transaction_uuid'] = obj['transaction_uuid'];
 logsres['environment'] = obj['environment'];
 logsres['host'] = obj['host'];
 puts logsres.to_json;
 };
obj['logs'].each {
 |x|logsjson = x;
 logsjson['host'] = obj['host'];
 logsjson['transaction_uuid'] = obj['transaction_uuid'];
 logsjson['environment'] = obj['environment'];
 puts logsjson.to_json
 };"
echo "$f conversion status:" $? >> "$log_file_name"
date "+%d-%b-%Y %H:%M:%S" > last_run_date.tmp
fi
done
echo "Conversion Completed. End Time: $(date)" >> "$log_file_name"

Copy this script to /opt/puppetlabs/server/data/puppetserver/reports with the name puppetReport.sh. If Puppet is installed in a different location, copy the script to the correct path.

Configure a cron job to trigger the python script using crontab. Frequency of this job can be set following the instructions from here. Here is the command which needs to be configured as part of cron to trigger the script:

<frequency_expression> /opt/puppetlabs/server/data/puppetserver/reports/puppetReport.sh

Please modify the location of the puppetReport.sh if required in the above command. The execution of the script above will generate a log file named puppet_rpt_conversion.log. The path to this log file needs to be provided during the app installation.

For Linux systems with ACL Support, the otelcol install process should have created the ACL grants necessary for the otelcol system user to access default log locations. You can verify the active ACL grants using the getfacl command. Install the ACL in your Linux environment, if not installed.

The required ACL may not be supported for some rare cases, for example, Linux OS Distro, which is officially not supported by Sumo Logic. In this case, you can run the following command to explicitly grant the permissions.

sudo setfacl -R -m d:u:otelcol-sumo:r-x,d:g:otelcol-sumo:r-x,u:otelcol-sumo:r-x,g:otelcol-sumo:r-x <PATH_TO_LOG_FILE>

Run the above command for all the log files in the directory that need to be ingested, which are not residing in the default location.

If Linux ACL Support is not available, traditional Unix-styled user and group permission must be modified. It should be sufficient to add the otelcol system user to the specific group that has access to the log files.

For Windows systems, log files which are collected should be accessible by the SYSTEM group. Use the following set of PowerShell commands if the SYSTEM group does not have access.

$NewAcl = Get-Acl -Path "<PATH_TO_LOG_FILE>"
# Set properties
$identity = "NT AUTHORITY\SYSTEM"
$fileSystemRights = "ReadAndExecute"
$type = "Allow"
# Create new rule
$fileSystemAccessRuleArgumentList = $identity, $fileSystemRights, $type
$fileSystemAccessRule = New-Object -TypeName System.Security.AccessControl.FileSystemAccessRule -ArgumentList $fileSystemAccessRuleArgumentList
# Apply new rule
$NewAcl.SetAccessRule($fileSystemAccessRule)
Set-Acl -Path "<PATH_TO_LOG_FILE>" -AclObject $NewAcl

Collection configuration and app installation​

As part of data collection setup and app installation, you can select the App from App Catalog and click on Install App. Follow the steps below.

Step 1: Set up Collector​

note

If you want to use an existing OpenTelemetry Collector, you can skip this step by selecting the Use an existing Collector option.

To create a new Collector:

  1. Select the Add a new Collector option.
  2. Select the platform where you want to install the Sumo Logic OpenTelemetry Collector.

This will generate a command that you can execute in the machine environment you need to monitor. Once executed, it will install the Sumo Logic OpenTelemetry Collector.

Collector

Step 2: Configure integration​

In this step, you will configure the yaml required for Puppet collection.

Path of the log file configured to capture puppet access logs and puppet report are needed to be given here. Refer to the Prerequisites section of this document.

Click on the Download YAML File button to get the yaml file.

YAML

Step 3: Send logs to Sumo Logic​

Once you have downloaded the yaml file as described in the previous step, follow the below steps based on your platform.

  1. Copy the yaml file to /etc/otelcol-sumo/conf.d/ folder in the Puppet instance that needs to be monitored.
  2. Place your Env file in the following directory:
/etc/otelcol-sumo/env/
  1. Restart the collector using:
  sudo systemctl restart otelcol-sumo

After successfully executing the above command, Sumo Logic will start receiving data from your host machine.

Click Next. This will install the app (dashboards and monitors) to your Sumo Logic Org.

Dashboard panels will start to fill automatically. It's important to note that each panel fills with data matching the time range query and received since the panel was created. Results won't immediately be available, but within 20 minutes, you'll see full graphs and maps.

Sample log messages​

This is a sample log message for non-Kubernetes environments.

5.35.225.115 - - [2023-01-16 06:49:18.751 +0000] "POST /puppet/v3/file_content/plugins/facter/sources_dir_exists_win.rb?environment=production& HTTP/1.1" 500 4484 "-" "Puppet/5.5.2 Ruby/2.4.4-p296 (x86_64-linux)" 5

Sample queries​

This sample query is from the Puppet - Overview dashboard > Node Requests Summary Over Time panel.

%"sumo.datasource"=puppet
| parse regex "^(?<src_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})" nodrop
| parse regex "(?<method>[A-Z]+)\s(?<url>\S+)\sHTTP/[\d\.]+\"\s(?<status_code>\d+)\s(?<size>[\d-]+)\s\"(?<referrer>.*?)\"\s\"(?<user_agent>.+?)\".*"
| if(status_code matches "2*", 1, 0) as successes 
| if(status_code matches "3*", 1, 0) as redirects
| if(status_code matches "4*", 1, 0) as client_errors 
| if(status_code matches "5*", 1, 0) as server_errors 
| timeslice by 1h
| sum(successes) as successes, sum(redirects) as redirects, sum(client_errors) as client_errors, sum(server_errors) as server_errors by _timeslice

Viewing Puppet Dashboards​

Overview​

Overview

Error Analysis: Puppet Server and Node Error Analysis​

Error Analysis

Node Puppet Runs Analysis: Puppet Node Runs Analysis​

Node Puppet Runs Analysis
Status
Legal
Privacy Statement
Terms of Use

Copyright © 2024 by Sumo Logic, Inc.