Skip to main content
Sumo Logic

Collect logs for the VMware App

Before you can install the VMware App, you'll need to configure log collection.

The logs collected from vCenter Servers enable you to use the Sumo Logic Application for VMware search, visualize, and analyze vCenter Server Events and Performance Data in real time to enable monitoring and detect important events within your virtual environment.

vCenter_graphic_700x192.png

Setting up a vMA Server to Collect Data

Before setting up a source to collect data, you'll need to install vMA through the vCenter Server (if it's not already installed) and then download and install a Collector

Step 1: Install vMA

vMA is an appliance (SUSE virtual machine) that includes vSphere CLI, and vSphere SDK for Perl. It allows administrators to run scripts or agents that interact with ESXi hosts and vCenter Server systems without having to authenticate each time.

To set up vMA:

  1. Download vMA from VMware, and follow the accompanying instructions to deploy the OVF. (Go to the vCenter Server through vSphere client, then choose File > Deploy OVF Template).
  1. Setup the authentication, timezone, and time for the vMA by following instructions in the VMware documentation.
  2. Take note to specify a user account in the next step that has adequate permissions on the vCenter servers. This user account will just need the Global.Service managers privilege to view only RESXTOP data. Refer this link for details.
  3. Run credstore_admin.pl on the vMA to add credentials for each vCenter server that generates performance and event data you'd like to collect. This script comes by default with the vSphere Perl SDK. On the vMA it's located under /usr/lib/vmware-vcli/apps/general. (See this VMware KB article for more information). For example, to add a user's account on a vCenter Server you can run:
/usr/lib/vmware-vcli/apps/general/credstore_admin.pl list
vi-admin@yourservername:/usr/lib/vmware-vcli/apps/general> ./credstore_admin.pl add --server 14.3.2.11 
--username "domainname\account name" --password 345345345
  1. Run the following command to verify that authentication is set up correctly and to see a list of network interfaces:
esxcli --server <vcenter host> --vihost <esxi host> network nic list

For example, if we run:

esxcli --server 192.168.23.242 --vihost 192.168.23.24 network nic 

we'll see a list of network interfaces for the ESXi host (192.168.23.24) managed by our vCenter Server (192.168.23.242).

Connect to <your vCenter Server> failed. Server SHA-1 thumbprint: <Your vCenter Thumbprint> (not trusted).

Run the following command to allow the connection:

/usr/lib/vmware-vcli/apps/<wbr/>general/credstore_admin.pl add -s <your vCenter Server>  --thumbprint <Thumbprint Above>

Step 2: Download and Install the Collector on vMA

  1. Download the appropriate Collector executable from this topic: Download a Collector from a Static URL.
  2. On the vMA machine, use wget or curl to download the file from the URL. The URL must be enclosed in double-quotes to work with wget. For example:
    sudo wget "https://collectors.sumologic.com/rest/download/linux/64" -O SumoCollector.sh
  3. From the download directory, run the installation file SumoCollector.sh as root. First, make sure root has executable privileges for the file by running: 
sudo chmod 740 SumoCollector.sh
  1. Run the install file on your server with root privileges: 
$ sudo ./SumoCollector.sh

The Collector runs as a service and starts automatically after installing or rebooting.
 

Collecting Event Messages

An event is an action that triggers an event message on a vCenter Server. Event messages are not logged, but are instead stored in the vCenter Server database. Sumo Logic for VMware retrieves these messages using the vSphere Perl SDK that comes with vMA (by default).

Step 1: Configure a Syslog Source for the Collector 

A Sumo Logic Syslog Source operates like a Syslog server listening on the designated port to receive Syslog messages.

  1. Go to Manage > Collection, and click Add Source.
  2. Select Syslog for the Source type.

    select_syslog_509x99.png
     
  3. Enter a Name to display for this Source. Source name metadata is stored in a searchable field called _sourceName.
  4. For Protocol choose TCP.
  5. Enter the Port number for the Source to listen to (for example, 1514, but choose the correct port for your Collector).
  1. For Source Category, we recommend using vcenter_log.
  2. Under Advanced, set the following options:
  • Select Extract timestamp information from log file entries.
  • Select Ignore time zone from log file and instead use and then choose UTC from the menu (as shown below).

    vCenter_Source_config_328x245.png
  1. Click Save.

Step 2: Configure Logs to be Collected

  1. On the vMA, create a directory to hold all Sumo Logic scripts, found under Sumo Logic Scripts for VMware. Name the directory /var/log/vmware or something similar.
  2. Download the Sumo Logic VMware scripts using the vSphere5.0.gz and vSphere5.5.gz links on this page, then put them in the directory you just created.
  3. We will assume the user account running the Sumo Logic vmware scripts  is the "vi-admin" user. This user account should not be a "root" user and should have full read write execute permissions to the directories where the vSphere5.0.gz and vSphere5.5.gz files are extracted. In the absence of adequate permissions, unexpected script errors will occur.
  4. Edit the cron_vcenter_events.sh script by changing the SCRIPT_PATH variable to reflect the absolute path where the script resides.
  1. Test running the query_vCenter.pl script (that queries the vCenter Server for events) as described in Troubleshooting and Manual Testing. Use the following example command:
    query_vCenter.pl -s [vcenterserver] -f output.txt
  2. Create a cron job to periodically run the cron_vcenter_events.sh script at the interval you'd like.
*/2 * * * * LD_LIBRARY_PATH=:/opt/vmware/vma/lib64:/opt/vmware/vma/lib /var/log/vmware/cron_vcenter_events.sh

Collect Performance Logs

Collecting performance logs involves using VMware tools and scripts running on vMA to extract performance statistics.

Step 1: Configure a Local File Source

Configure a Local File Source.

  1. Go to Manage > Collection, and click Add Source for your vCenter Server Collector.
  2. Select Local File for the Source type.
  3. Enter a Name to display for this Source. Source name metadata is stored in a searchable field called _sourceName.
  4. For File Path, enter /var/log/vmware/*.perf.out.
  5. For Source Category, enter esx_perf.
  6. Under Advanced, make sure that Timestamp Parsing is selected. Then for Time Zone choose the time zone of the vMA virtual machine.
    vmWare_LocalFileSource_389x675.png
  7. Click Save.

Step 2: Configure Performance Logs for Collection

Before collecting can begin, you'll need to invoke scripts to transform the performance data from the resxtop utility so it's delivered in a format that Sumo Logic can consume.

  1. On the vMA, create a directory to hold all scripts (for example, /var/log/vmware).
  2. Extract all files from the Zip bundle provided by Sumo Logic to the directory you just created.
  3. Edit the vcenter.info file so one vCenter Server and one username is on each line.
    For example:

vcenter01.company.com "domain_name\user_name"
vcenter02.company.com "domain_name\user_name"
vcenter03.company.com "domain_name\user_name"    
  1. Run /usr/lib/vmware-vcli/apps/general/credstore_admin.pl list to get a list of all the vCenter Servers you have already configured for authentication.

  2. Edit the following in the cron_vcenter_perf.sh script:
  • Change the SCRIPT_PATH variable to reflect the absolute path where the script resides.
  • Select the method you'd like to use to collect performance data. Then, uncomment the line that calls$SCRIPT_PATH/getserver_perf.pl. For more information, see Segmenting Collection.
  • Test the command used in the cron script before testing the cron command and enabling it  as described in Troubleshooting and Manual Testing.
  • $SCRIPT_PATH/getserver_perf.pl -type=vcenter -path=$SCRIPT_PATH -server_file=$SCRIPT_PATH/vcenter.info 
    -output_file=$SCRIPT_PATH/vcenter_all.perf.out
  1. Run the cron_vcenter_perf.sh script. After it finishes, verify that performance logs are being collected. You should see the esxi.perf.out file in the above $SCRIPT_PATH directory; the file should have at least 7000-8000 messages per ESXi server that is managed by a vCenter.

For example, in line 138, change:

`echo \'$pwd\' | /usr/bin/resxtop --server $host --username $user -b -n $iterations > $local_csv_file `;

to 

` echo \'$pwd\' | /usr/bin/resxtop --server $host -c esxtop50rc --username $user -b -n $iterations > $local_csv_file`;

Another way to reduce the amount of data is to reduce the frequency of running cron_vcenter_perf.sh (see below).

vi-admin@<vma-hostname>>~
/opt/SumoScripts/cron_vcenter_perf.sh
vi-admin's password:
domain_name\user_name  password:
  1. Create a cron job to periodically run the cron_vcenter_perf.sh script. For example to run the script every 15 minutes, it would look like:
     
*/15 * * * * LD_LIBRARY_PATH=:/opt/vmware/vma/lib64:/opt/vmware/vma/lib /var/log/vmware/cron_vcenter_perf.sh

Understand vCenter Scripts

In this section, we'll walk you through a few important tasks that involve the query_vCenter.pl script, or query_vCenter5.5.pl, depending on the exact version you are using. These scripts query the vCenter Server for events.

Troubleshooting and Manual Testing

Test the query_vCenter.pl script before setting up a CRON job.

To test the query_vCenter.pl script, go to the folder that holds all the scripts (for example, /var/log/vmware) and run: 

query_vCenter.pl -s [vcenterserver] -f output.txt


(Replace [vCenterServer] with the name of the target vCenter Server in your environment.)

In the standard output, you should see the query time range and the number of events collected. The events themselves are stored inside the output.txt file. If you're prompted to enter a username or password, it means that the credentials for the target vCenter Server are not set properly. By default, the first time query_vCenter is called, events from the past 24 hours are collected. If you want to collect events older than the past 24 hours, see Collect Historical Events.

Collect Historical Events

By default, the first time query_vCenter.pl is called, events from the past 24 hours are collected. Each time the script is called, it writes the timestamp of the last read event in a file named .timelog for the next call to pick up.

If you want to collect events older than the past 24 hours, before setting up the CRON job for cron_vcenter_events.sh, do the following on the VMA machine.

To collect historical events:

  1. Go to the vi-admin home directory, at /home/vi-admin.
  2. Set the SCRIPT_PATH environment variable to point to where all the Sumo Logic for VMware scripts reside. For example: 
vi-admin@vma1:~> export SCRIPT_PATH=/var/log/vmware
  1. Run the query_vCenter.pl script as follows:
./query_vCenter.pl --server <vcenter server> --target <syslog host>:<syslog port> --bT <time in UTC>
./query_vCenter.pl --server 192.168.23.242 --target vmahost:1514 --bT 2012-10-08T00:17:00.00Z

Once this command completes successfully, you can begin to pick up ongoing events by setting up the CRON job as described in step 2 of Collecting Event Messages.

Segmenting Performance Collection

Performance data collection for ESXi servers associated with a vCenter server works by sequentially getting data from each ESXi server. Having a large number of servers associated with a vCenter can cause delays. To avoid delays, you can parallelize collection by creating multiple segments from the list of ESXi servers associated with a vCenter server. The number of segments would depend on the amount of data you are collecting and how often you would like to collect performance data.

For example, let's say you have a 100 ESXi servers associated with a single vCenter server; it takes more than 2 minutes for the performance collection script to collect data from all ESXi servers. If you need to collect performance data snapshots every 2 minutes, you'd segment the collection into two or more instances of the cron_vcenter_perf.sh script to parallelize collection.

Or, if you have 100ESXi servers and five segments, then each segment would have 20 servers. However, if there's an odd number of servers, say 145 servers and 6 sgments, then 5 segments have 24 servers, and the last (6th) segment will have 20 servers.

The total number of segments is specified by the -segments option, and the segment number we want to run is specified by the segment_number option.

Before collecting from multiple segments, you can test how ESXi servers will be divided up into segments by using the -test option. For example, to test which servers get assigned to each segment assuming you have two segments, you would run the following command:

$SCRIPT_PATH/getserver_perf.pl -test -type=vcenter -path=$SCRIPT_PATH -server_file=$SCRIPT_PATH/vcenter.info 
-output_file=$SCRIPT_PATH/vcenter_segments.perf.out -segments=2 -segment_number=1 | /bin/logger

Then to get performance data jusy for segment one, you can run this command:

$SCRIPT_PATH/getserver_perf.pl -type=vcenter -path=$SCRIPT_PATH -server_file=$SCRIPT_PATH/vcenter.info 
-output_file=$SCRIPT_PATH/vcenter_segment-3.perf.out -segments=2 -segment_number=1 | /bin/logger

and so on.

Once you are satisfied with the time it takes for collecting data for a segment, create multiple scripts to collect data for each segment based on the cron_vcenter_perf.sh script. Schedule these scripts to run as a cron job according to the desired frequency.