Skip to main content
Sumo Logic

Collect Squid Proxy Logs and Metrics for Non-Kubernetes environments

Sumo Logic uses the Telegraf operator for Squid Proxy metric collection and the Installed Collector for collecting Squid Proxy logs.

Sumo Logic uses the Telegraf operator for Squid Proxy metric collection and the Installed Collector for collecting Squid Proxy logs. The diagram below illustrates the components of the  Squid Proxy collection in a non-Kubernetes environment. Telegraf uses the SNMP input plugin to obtain Squid Proxy metrics and the Sumo Logic output plugin to send the metrics to Sumo Logic. Logs from Squid Proxy are collected by a Local File Source.

 

The process to set up collection for Squid Proxy data is done through the following steps:

  1. Configure Logs Collection
    1. Configure logging in Squid Proxy
    2. Configure Sumo Logic Installed Collector
    3. Configure a local file source
    4. Save
  2. Configure Metrics Collection
    1. Configure a Hosted Collector
    2. Configure an HTTP Logs and Metrics Source
    3. Enable SNMP agent on Squid Proxy
    4. Install Telegraf
    5. Configure and start Telegraf

Configure Logs Collection

Squid Proxy app supports the default access logs and cache logs format.

  1. Configure logging in Squid Proxy.

By default, the squid proxy will write the access log to the log directory that was configured during installation. For example, on Linux, the log directory would be /var/log/squid/access.log. If the access log is disabled then you must enable the access log following these instructions.

  1. Configure an Installed Collector. If you have not already done so, install and configure an installed collector for Windows by following the documentation.

  2. Configure a Collector

Use one of the following Sumo Logic Collector options:

  1. To collect logs directly from the Squid Proxy machine, configure an Installed Collector.

  2. If you are using a service like Fluentd, or you would like to upload your logs manually, Create a Hosted Collector.

  1. Configure a local file source

For an Installed Collector

To collect logs directly from your Squid Proxy machine, use an Installed Collector and a Local File Source.  

  1. Add a Local File Source.

  2. Configure the Local File Source fields as follows:

  • Name. (Required)
  • Description. (Optional)
  • File Path (Required). Enter the path to your access.log. The files are typically located in /var/log/squid/access.log. If you are using a customized path, check the squid.conf file for this information.
  • Source Host. Sumo Logic uses the hostname assigned by the OS unless you enter a different hostname.
  • Source Category. Enter any string to tag the output collected from this Source, such as SquidProxy/AccessLog. (The Source Category metadata field is a fundamental building block to organize and label Sources. For details see Best Practices.)
  • Fields. Set the following fields
    component = proxy
    proxy_system = squidproxy
    proxy_cluster = <Your_Squid_Proxy_Cluster_Name>. Enter Default if you do not have one.
    environment = <Your_Environment_Name> (for example, Dev, QA, or Prod)
    clipboard_e84130fb2e81a38833c84bc9a2cb76db0.png
  1. Configure the Advanced section:

  • Enable Timestamp Parsing. Select Extract timestamp information from log file entries.
  • Time Zone. Automatically detect.
  • Timestamp Format. The timestamp format is automatically detected.
  • Encoding. Select UTF-8 (Default).
  • Enable Multiline Processing
    • Error logs. Select Detect messages spanning multiple lines and Infer Boundaries - Detect message boundaries automatically.
    • Access logs. These are single-line logs, uncheck Detect messages spanning multiple lines.
  1. Click Save.

For a Hosted Collector

If you are using a service like Fluentd, or you would like to upload your logs manually, use a Hosted Collector and an HTTP Source.

  1. Add an HTTP Source.

  2. Configure the HTTP Source fields as follows:

  • Name. (Required)
  • Description. (Optional)
  • Source Host. Sumo Logic uses the hostname assigned by the OS unless you enter a different hostname.
  • Source Category. Enter any string to tag the output collected from this Source, such as SquidProxy/AccessLog. (The Source Category metadata field is a fundamental building block to organize and label Sources. For details see Best Practices.)
  1. Configure the Advanced section:

  • Enable Timestamp Parsing. Select Extract timestamp information from log file entries.
  • Time Zone. For Access logs, use the time zone from the log file. For Error logs, make sure to select the correct time zone.
  • Timestamp Format. The timestamp format is automatically detected.
  • Enable Multiline Processing
    • Error logs: Select Detect messages spanning multiple lines and Infer Boundaries - Detect message boundaries automatically.
    • Access logs: These are single-line logs, uncheck Detect messages spanning multiple lines.
  1. Click Save.

  2. When the URL associated with the HTTP Source is displayed, copy the URL so you can add it to the service you are using, such as Fluentd.

Configure Metrics Collection

Setup a Sumo Logic HTTP Source
  1. Configure a Hosted Collector for Metrics.
    To create a new Sumo Logic hosted collector, perform the steps in the Create a Hosted Collector documentation.

  2. Configure an HTTP Logs & Metrics source:

    1. On the created Hosted Collector on the Collection Management screen, select Add Source.

    2. Select HTTP Logs & Metrics.

      1. Name. (Required). Enter a name for the source.

      2. Description. (Optional).

      3. Source Category (Recommended). Be sure to follow the Best Practices for Source Categories. A recommended Source Category may be Prod/ProxyServer/SquidProxy/Metrics.

    3. Select Save.

    4. Take note of the URL provided once you click Save. You can retrieve it again by selecting the Show URL next to the source on the Collection Management screen.

Enable SNMP agent on Squid Proxy

By default, the SNMP agent will be disabled on squid proxy. You have to enable it. To enable the SNMP agent on squid, edit the configuration file of the squid proxy (squid.conf) and add the following section:

acl snmppublic snmp_community public
snmp_port 3401
snmp_access allow snmppublic localhost

Setup Telegraf
  1. Install Telegraf if you haven’t already. Use the following steps to install Telegraf.

  2. Configure and start Telegraf.
    As part of collecting metrics data from Telegraf, we will use the SNMP input plugin to get data from Telegraf and the Sumo Logic output plugin to send data to Sumo Logic.

    Create or modify telegraf.conf and copy and paste the text below: 

[[inputs.snmp]]]
  agents = ["udp://127.0.0.1:3401"]
  name = "squid"
  community = "public"
  [inputs.snmp.tags]
   proxy_cluster="<Squid Proxy_TO_BE_CHANGED>"
   component="proxy"
   environment="env_TO_BE_CHANGED"
   proxy_system="squidproxy"


  [[inputs.snmp.field]]
name = "uptime"
oid = "1.3.6.1.4.1.3495.1.1.3.0"
  [[inputs.snmp.field]]
      name = "cacheMemUsage"
      oid = "1.3.6.1.4.1.3495.1.3.1.3.0"
  [[inputs.snmp.field]]
      name = "cacheCpuUsage"
      oid = "1.3.6.1.4.1.3495.1.3.1.5.0"
  [[inputs.snmp.field]]
      name = "cacheClients"
      oid = "1.3.6.1.4.1.3495.1.3.2.1.15.0"
  [[inputs.snmp.field]]
      name = "cacheProtoClientHttpRequests"
      oid = "1.3.6.1.4.1.3495.1.3.2.1.1.0"
  [[inputs.snmp.field]]
  name = "cacheHttpHits"
      oid = "1.3.6.1.4.1.3495.1.3.2.1.2.0"
  [[inputs.snmp.field]]
   name = "cacheHttpErrors"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.3.0"
  [[inputs.snmp.field]]
   name = "uidcacheHttpInKb"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.4.0"


  [[inputs.snmp.field]]
   name = "cacheHttpOutKb"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.5.0"


  [[inputs.snmp.field]]
   name = "cacheServerInKb"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.12.0"
  [[inputs.snmp.field]]
   name = "cacheServerOutKb"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.13.0"
  [[inputs.snmp.field]]
   name = "cacheClients"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.15.0"
  [[inputs.snmp.field]]
   name = "cacheCpuTime"
   oid = "1.3.6.1.4.1.3495.1.3.1.4.0"
  [[inputs.snmp.field]]
   name = "cacheMemMaxSize"
   oid = "1.3.6.1.4.1.3495.1.2.5.1.0"
  [[inputs.snmp.field]]
   name = "cacheServerRequests"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.10.0"
  [[inputs.snmp.field]]
   name = "cacheHttpInKb"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.4.0"
  [[inputs.snmp.field]]
   name = "cacheHttpOutKb"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.5.0"
  [[inputs.snmp.field]]
   name = "cacheNumObjCount"
   oid = "1.3.6.1.4.1.3495.1.3.1.7.0"
  [[inputs.snmp.field]]
   name = "cacheHttpAllSvcTime1"
   oid = "1.3.6.1.4.1.3495.1.3.2.2.1.2.1"
  [[inputs.snmp.field]]
   name = "cacheDnsSvcTime1"
   oid = "1.3.6.1.4.1.3495.1.3.2.2.1.8.1"
  [[inputs.snmp.field]]
   name = "cacheHttpMissSvcTime60"
   oid = "1.3.6.1.4.1.3495.1.3.2.2.1.3.60"
  [[inputs.snmp.field]]
   name = " cacheHttpHitSvcTime60"
   oid = "1.3.6.1.4.1.3495.1.3.2.2.1.5.60"
  [[inputs.snmp.field]]
   name = "cacheIpEntries"
   oid = "1.3.6.1.4.1.3495.1.4.1.1.0"
  [[inputs.snmp.field]]
   name = "cacheIpMisses"
   oid = "1.3.6.1.4.1.3495.1.4.1.6.0"
  [[inputs.snmp.field]]
   name = "cacheVersionId"
   oid = "1.3.6.1.4.1.3495.1.2.3.0"
  [[inputs.snmp.field]]
   name = "cacheSysPageFaults"
   oid = "1.3.6.1.4.1.3495.1.3.1.1.0"
  [[inputs.snmp.field]]
   name = "cacheHttpErrors"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.3.0"
  [[inputs.snmp.field]]
   name = "cacheServerErrors"
   oid = "1.3.6.1.4.1.3495.1.3.2.1.11.0"
  [[inputs.snmp.field]]
   name = "cacheCpuUsage"
   oid = "1.3.6.1.4.1.3495.1.3.1.5.0"
  [[inputs.snmp.field]]
   name = "cacheCpuTime"
   oid = "1.3.6.1.4.1.3495.1.3.1.4.0"
  [[inputs.snmp.field]]
   name = "cacheSysVMsize"
   oid = "1.3.6.1.4.1.3495.1.1.1.0"
  [[inputs.snmp.field]]
   name = "cacheSysNumReads"
   oid = "1.3.6.1.4.1.3495.1.3.1.2.0"
  [[inputs.snmp.field]]
   name = "cacheCurrentUnusedFDescrCnt"
   oid = "1.3.6.1.4.1.3495.1.3.1.10.0"
  [[inputs.snmp.field]]
   name = "cacheCurrentFileDescrCnt"
   oid = "1.3.6.1.4.1.3495.1.3.1.12.0"
  [[inputs.snmp.field]]
   name = "cacheMaxResSize"
   oid = "1.3.6.1.4.1.3495.1.3.1.6.0"
  [[inputs.snmp.field]]
   name = "cacheCurrentResFileDescrCnt"
   oid = "1.3.6.1.4.1.3495.1.3.1.11.0"
  [[inputs.snmp.field]]
   name = "cacheIpRequests"
   oid = "1.3.6.1.4.1.3495.1.4.1.2.0"
  [[inputs.snmp.field]]
   name = "cacheIpHits"
   oid = "1.3.6.1.4.1.3495.1.4.1.3.0"
  [[inputs.snmp.field]]
   name = "cacheFqdnEntries"
   oid = "1.3.6.1.4.1.3495.1.4.2.1.0"
  [[inputs.snmp.field]]
   name = "cacheFqdnRequests"
   oid = "1.3.6.1.4.1.3495.1.4.2.2.0"
  [[inputs.snmp.field]]
   name = "cacheFqdnHits"
   oid = "1.3.6.1.4.1.3495.1.4.2.3.0"
  [[inputs.snmp.field]]
   name = "cacheFqdnMisses"
   oid = "1.3.6.1.4.1.3495.1.4.2.6.0"
  [[inputs.snmp.field]]
   name = "cacheDnsRequests"
   oid = "1.3.6.1.4.1.3495.1.4.3.1.0"
  [[inputs.snmp.field]]
   name = "cacheDnsReplies"
   oid = "1.3.6.1.4.1.3495.1.4.3.2.0"
  [[inputs.snmp.field]]
   name = "cacheDnsNumberServers"
   oid = "1.3.6.1.4.1.3495.1.4.3.3.0"
  [[inputs.snmp.field]]
   name = "version"
   oid = "1.3.6.1.4.1.3495.1.2.3.0"
   is_tag = true
  [[inputs.snmp.field]]
   name = "cacheHttpAllSvcTime5"
   oid = "1.3.6.1.4.1.3495.1.3.2.2.1.2.5"
  [[inputs.snmp.field]]
   name = "cacheHttpMissSvcTime5"
   oid = "1.3.6.1.4.1.3495.1.3.2.2.1.3.5"
  [[inputs.snmp.field]]
   name = "cacheHttpHitSvcTime5"
   oid = "1.3.6.1.4.1.3495.1.3.2.2.1.5.5"
  [[inputs.snmp.field]]
   name = "cacheDnsSvcTime5"
   oid = "1.3.6.1.4.1.3495.1.3.2.2.1.8.5"
[[outputs.sumologic]]
  url = "<URL_from_HTTP_Logs_and_Metrics_Source>"
  data_format = "prometheus"

Enter values for fields annotated with <VALUE_TO_BE_CHANGED> to the appropriate values. Do not include the brackets (<>) in your final configuration

  • In the tags section, which is [inputs.snmp.tags]:
    • environment - This is the deployment environment where the Squid Proxy server identified by the value of servers resides. For example; dev, prod, or QA. While this value is optional we highly recommend setting it. 
    • proxy_cluster - Enter a name to identify this Squid Proxy cluster. This cluster name will be shown in our dashboards. 
  • In the output plugins section, which is [[outputs.sumologic]]
    • URL - This is the HTTP source URL created previously. See this doc for more information on additional parameters for configuring the Sumo Logic Telegraf output plugin.

Here’s an explanation for additional values set by this Telegraf configuration.

  • data_format: “prometheus” - In the output [[outputs.sumologic]] plugins section. Metrics are sent in the Prometheus format to Sumo Logic.
  • component - “proxy” - In the input [[inputs.snmp]] plugins section. This value is used by Sumo Logic apps to identify application components.
  • proxy_system - “squidproxy” - In the input plugins sections. This value identifies the proxy system.

See this doc for all other parameters that can be configured in the Telegraf agent globally.

At this point, Telegraf should start collecting the Squid Proxy metrics and forward them to the Sumo Logic HTTP Source.