Collect Logs for JFrog Xray
This page explains how to collect logs from JFrog Xray and ingest them into Sumo Logic for use with the JFrog Xray pre-defined dashboards and searches. To get the most of out this app, we recommend you also collect logs from Artifactory as well as Kubernetes.
Collection Overview
To configure log collection for the JFrog Xray app, you will perform the following tasks:
Step 1: Collect Jfrog Xray instance details
In this step you collect details for your JFrog Xray instance that you will use in the following tasks.
Collect the following details:
- HostName and Port for your JFrog Xray instance — such as, JFrog instance URL http://host-example:8000/web/#/login
- HostName = host-example
- Port = 8000
- Your Username and Password for your JFrog Xray instance
Step 2: Collect Artifactory logs
We recommend collecting data from JFrog Artifactory so as to investigate sources of vulnerable artifacts and who is using them. This is done by correlating Xray logs with Artifactory logs.
To do so, follow the instructions in Collect Logs for Artifactory.
Step 3: Collect Kubernetes logs
If you have set up a Docker repository in Aritfactory and are running containers in a Kubernetes cluster, we recommend collecting data from your Kubernetes cluster so as to understand all vulnerable containers running in production.
To perform this setup, follow the instructions in Collect Logs for Kubernetes.
Step 4: Add a Hosted Collector and HTTP Source
In this step you set up a hosted Sumo Logic collector and HTTP source to collect JFrog Xray logs.
Identify an existing Sumo Logic Hosted Collector you want to use, or create a new Hosted Collector as described in the following task.
To add a hosted collector and HTTP source
- Create a new Sumo Logic hosted collector by performing the steps in Configure a Hosted Collector.
- Create a new HTTP source on the hosted collector created above by following these instructions.
Step 5: Set up a collection method for JFrog Xray
This section covers the various ways in which to collect logs from JFrog Xray and send them to Sumo Logic. The logs are then shown in dashboards as part of the JFrog Xray App. You can configure a Sumo Logic collector for JFrog Xray in Amazon Web Services (AWS) using AWS Lambda service, or use a script on a Linux machine with a cron job. Choose the method that best suits your environment:
- Method 1 - AWS Lambda based collection via a Serverless Application Model (SAM) application
- Method 2 - Script based collection
Method 1- Sumo Logic JFrog Xray SAM application
In this collection method, you deploy the SAM application, which creates the necessary resources in your AWS account.
To deploy the Sumo Logic JFrog xray SAM Application, do the following:
- Go to https://serverlessrepo.aws.amazon.com/applications.
- Search for sumologic-jfrog-xray and make sure the checkbox Show apps that create custom IAM roles or resource policies is checked, and click the app link when it appears.
- When the page for the Sumo app appears, click Deploy.
- Go to the AWS Lambda > Functions > Application Settings panel, and enter parameters for the following fields:
- HTTPLogsEndpoint: Copy and paste the URL for the HTTP log source from Step 4.
- Hostname: Copy and paste the Hostname from Step 1.
- Port: Copy and paste the Port from Step 1.
- Username: Copy and paste the Username from Step 1.
- Password: Copy and paste the Password from Step 1.
- Click Deploy.
Optional - Configure multiple JFrog Xray instances
If you have multiple JFrog Xray instances from which you want to collect logs and send to Sumo Logic, perform the following task.
To configure collection for multiple JFrog Xray instances, do the following:
- Deploy the SAM application with configuration for a new project.
- After the deployment is complete, change the database name by adding environment variable (DBNAME) in AWS Lambda.
Method 2: Sumo Logic JFrog Xray Script based collection
This section provides instructions for configuring script based collection for the Sumo Logic JFrog Xray App.
Prerequisites
- You must have successfully added a Hosted Collector and HTTP source and copied details for JFrog Xray instance, as described in Step 4.
- You must be logged in to the user account with which you will install the collector. If you are not, use the following command to switch to that account:
sudo su <user_name> - A Linux machine compatible with either Python 3.7 or Python 2.7
Step 1. Configure the script on a Linux machine
This task shows you how to install the script on a Linux machine.
To deploy the script, do the following:
- If pip is not already installed, follow the instructions in the pip documentation to download and install pip.
- Log in to a Linux machine compatible with either Python 3.7 or Python 2.7.
- Do one of the following:
- For Python 2 - run the following command: pip install sumologic-jfrog-xray
- For Python 3 - run the following command: pip3 install sumologic-jfrog-xray
- Create a configuration file jfrogxraycollector.yaml in the home directory as shown below, and fill in the parameter <Variables> where indicated.
- Create a cron job to run the collector every 5 minutes, (use the crontab -e option), in one of the following ways:
- For Python 2 - add the following line in your crontab: */5 * * * * /usr/bin/python -m sumojfrogxray.main > /dev/null 2>&1
- For Python 3 - add the following line in your crontab: */5 * * * * /usr/bin/python3 -m sumojfrogxray.main > /dev/null 2>&1
Optional - Configure collection for multiple projects
If you have multiple projects from which you want to collect logs and send to Sumo Logic, perform the following task.
To configure collection for multiple projects, do the following:
- Configure the script on a Linux machine, then navigate to your configuration file.
- Change the DB_NAME in the jfrogxraycollector.yaml, the <DB NAME> variable in the following example.
Step 2. Advanced configuration
This section provides a list of variables for Jfrog Xray that you can define in the configuration file.
Variable | Usage |
---|---|
BACKFILL_DAYS in Collection Section | Number of days before the event collection will start. If the value is 1, then events are fetched from yesterday to today. |
LOG_FORMAT in Logging Section | Log format used by the python logging module to write logs in a file. |
ENABLE_LOGFILE in Logging Section | Set to TRUE to write all logs and errors to a log file. |
ENABLE_CONSOLE_LOG in Logging Section | Enables printing logs in a console. |
LOG_FILEPATH in Logging Section | Path of the log file used when ENABLE_LOGFILE is set to TRUE. |
NUM_WORKERS in Collection Section | Number of threads to spawn for API calls. |
MAX_RETRY in Collection Section | Number of retries to attempt in case of request failure. |
BACKOFF_FACTOR in Collection Section | A backoff factor to apply between attempts after the second try. If the backoff_factor is 0.1, then sleep() will sleep for [0.0s, 0.2s, 0.4s, ...] between retries. |
TIMEOUT in Collection Section | Request time out used by the requests library. |
HTTP_LOGS_ENDPOINT in SumoLogic section | HTTP source endpoint url created in Sumo Logic for ingesting Logs. |
Troubleshooting
This section shows you how to run the function manually and then verify that log messages are being sent from JFrog Xray.
To run the function manually, do the following:
- Do one of the following:
- For python, run this command: python -m sumojfrogxray.main
- For python3, run this command: python3 -m sumojfrogxray.main
- The script generates logs in /tmp/sumoapiclient.log by default. Check these logs to verify whether the script is getting triggered or not.
- If you get an error such as “unable to execute 'gcc': No such file or directory, error: command 'gcc' failed with exit status 1”, go to your AWS EC2 instance and run the following commands:
sudo yum -y install gcc sudo yum install python-devel
Sample Log Message
{
"created": "2019-09-03 22:01:19,804 +0530",
"top_severity": "High",
"watch_name": "Maven_watch",
"policy_name": "License_policy",
"issues": [
{
"severity": "medium",
"type": "License",
"provider": "JFrog",
"created": "2019-09-03 22:01:19,804 +0530",
"summary": "In Libgcrypt 1.8.4, the C implementation of AES is vulnerable to a flush-and-reload side-channel attack because physical addresses are available to other processes. (The C implementation is used on platforms where an assembly-language implementation is unavailable.)",
"description": "In Libgcrypt 1.8.4, the C implementation of AES is vulnerable to a flush-and-reload side-channel attack because physical addresses are available to other processes. (The C implementation is used on platforms where an assembly-language implementation is unavailable.)",
"impacted_artifacts": [
{
"name": "mina-core-2.0.0-RC1-javadoc.jar",
"display_name": "mina-core:2.0.0-RC1",
"path": "/milestone/org/apache/mina/mina-core/2.0.0-RC1/mina-core-2.0.0-RC1-javadoc.jar",
"pkg_type": "zip",
"sha256": "ca013ac5c09f9a9f6db8370c1b759a29fe997d64d6591e9a75b71748858f7da0",
"sha1": "4cc3661681baf84566f4e3f166127074548d4519",
"depth": 0,
"parent_sha": "ca013ac5c09f9a9f6db8370c1b759a29fe997d64d6591e9a75b71748858f7da0",
"infected_files": [
{
"name": "SQLAlchemy-1.3.8.tar.gz",
"path": "SQLAlchemy:1.3.8",
"sha256": "dd1ca0d765607415523d57b7464c0bb259412cff5d9a09c281d0acfbd4eed7e3",
"depth": 0,
"parent_sha": "35c102085707f703de2d9eaad8752d6fe1b8f02b5d2149f1d8357c9cc7fb7d0a",
"display_name": "/libs-milestone-local/org/springframework/spring/3.2.0.RC2/spring-framework-3.2.0.RC2-dist.zip",
"pkg_type": "spring"
}
]
}
],
"cve": "CVE-2019-12904"
}
]
}
Query Sample
The sample query is from Watches Invoked panel of the JFrog Xray - Overview dashboard.
_sourceCategory = Labs/jfrog/xray
| json "top_severity", "issues", "watch_name", "policy_name" as TopSeverity, Issues, WatchName, PolicyName nodrop
| where !(TopSeverity matches "Pending Scan")
| parse regex field=Issues "(?<Issue>\{.*?(?=,\{\"severity\"|\]$))" multi
| json field=Issue "impacted_artifacts", "severity", "summary", "cve", "provider", "created", "description", "type" as Artifacts, Severity, Summary, CVE, Provider, Created, Description, PolicyType nodrop
| parse regex field=Artifacts "(?<Artifact>\{.*?(?=,\{\"sha1\"|\]$))" multi
| json field=Artifact "infected_files", "sha1", "path", "depth", "sha256", "name", "parent_sha", "display_name", "pkg_type" as Files, ArtifactSha, ArtifactPath, ArtifactDepth, ArtifactSha256, ArtifactName, ArtifactParentSha, ArtifactDisplayName, ArtifactPkgType nodrop
| parse regex field=Files "(?<File>\{[^\}]+(?:\}\}|\}))" multi
| json field=File "path", "depth", "sha256", "name", "parent_sha", "display_name", "pkg_type" as ComponentPath, ComponentDepth, ComponentSha, ComponentName, ComponentParentSha, ComponentDisplayName, ComponentPkgType nodrop
| count_distinct(WatchName) as %"Number of Watches"