Collect Logs for Artifactory
This procedure documents how to collect logs from JFrog Artifactory into Sumo Logic.
Log Types
Sumo Logic reads logs in the directory /var/opt/jfrog/artifactory/logs
:
artifactory.log
access.log
request.log
traffic.*.log
For more information about Artifactory logs, see JFrog's Artifactory Log Files.
Configure a collector
Configure an Installed Collector.
Configure sources
In this step, you configure four local file sources, one for each log source listed in the table below. When you create a file source for a log type:
- Use the value from the File Path column below as the File Path for the source.
- The value you specify for the source's Source Category must end with the suffix shown below in the Source Category column. For example, you could set the Source Category for the Artifactory Server log source to be
foo/artifactory/console
, but notartifactory/console/foo
The following suffixes are required. For example, you could use _sourceCategory=<Foo>/artifactory/console, but the suffix artifactory/console must be used.
Log source | File Path | Source Category |
Artifactory Server | /var/opt/jfrog/artifactory/logs/artifactory.log | artifactory/console |
Access | /var/opt/jfrog/artifactory/logs/access.log | artifactory/access |
Request | /var/opt/jfrog/artifactory/logs/request.log | artifactory/request |
Traffic | /var/opt/jfrog/artifactory/logs/traffic.*.log | artifactory/traffic |
For complete instructions see Local File Source.
- Configure a Local File source.
- Configure the Source fields:
- Name. (Required) A name is required. Description is optional.
- Source Category. (Required)
- Configure the Advanced section:
- Enable Timestamp Parsing. True
- Time Zone. Logs are in UTC by default
- Timestamp Format. Auto Detect
- Encoding Type. UTF-8
- Multi-line Parsing. Detect Messages Spanning Multiple Lines, Infer Boundaries
- Click Save.
Field Extraction Rules
Here are Artifactory extraction rules that use different approaches.
Traffic
_sourceCategory=*artifactory*
| where _sourceCategory matches "*artifactory/traffic"
| parse regex "(?<year>\d{4})(?<month>\d{2})(?<day>\d{2})(?<hour>\d{2})(?<minute>\d{2})(?<second>\d{2})\|\d*\|(?<direction>[^|]*)\|\s*(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|[^|]*)\|(?<repo>[^:]*):(?<fullfilepath>[^|]*)\|(?<size>\d*)" nodrop
Access Logs
_sourceCategory=*artifactory*
| where _sourceCategory matches "*artifactory/access"
| parse "[*] *:* for */*" as what, repo, path, user, ip
Request Logs
_sourceCategory=*artifactory*
| where _sourceCategory matches "*artifactory/request"
| parse "*|*|*|*|*|*|*|*|*|*" as datetime, response_time, type, ip, user, method, path, protocol, status_code, size
Sample Log Messages
20170113185444|17|REQUEST|1.1.1.1|anonymous|GET|/cloudera-repos/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar|HTTP/1.1|200|8869
20170113185444|0|DOWNLOAD|1.1.1.1|cloudera-repos:org/apache/spark/spark-catalyst_2.11/2.0.1/spark-catalyst_2.11-2.0.1.jar.sha1|40
2017-01-13 18:54:12,121 [ACCEPTED DEPLOY] pypi-remote-cache:.pypi/test.html for billythekid/1.1.1.1.
Query Samples
Data Transfer Over Time
_sourceCategory=*artifactory*
| where _sourceCategory matches "*artifactory/traffic"
| parse regex
"(?<year>\d{4})(?<month>\d{2})(?<day>\d{2})(?<hour>\d{2})(?<minute>\d{2})(?<second>\d{2})\|\d*\|(?<direction>[^|]*)\|\s*(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|[^|]*)\|(?<repo>[^:]*):(?<fullfilepath>[^|]*)\|(?<size>\d*)" nodrop
| timeslice 1h
| sum(size) by _timeslice, direction
| _sum / (1024 * 1024 * 1024) as sizeinGB | sort by _sum
| fields -_sum
| transpose row _timeslice column direction
Requests by Status Code (Every 10 Minutes)
_sourcecategory=*artifactory*
| where _sourceCategory matches "*artifactory/request"
| parse "*|*|*|*|*|*|*|*|*|*" as datetime, response_time, type, ip, user, method, path, protocol, status_code, size
| timeslice 10m
| count _timeslice, status_code | sort by _count
| transpose row _timeslice column status_code
Unique Paths Accepted Deploys
_sourceCategory=*artifactory* "ACCEPTED DEPLOY" "-cache"
| where _sourceCategory matches "*artifactory/access"
| parse "[*] *:* for */*" as what, repo, path, user, ip
| parse regex field=ip "(?<ip>.*)\."
| where what = "ACCEPTED DEPLOY" and repo matches "*-cache"
| timeslice 10m
| count_distinct(path) as paths by _timeslice
| outlier paths