Skip to main content
Sumo Logic

Collect Logs for Fastly

Collect Logs for Sumo Logic Fastly App

To collect logs for the Sumo Logic Fastly App, perform the following steps.

  1. Add a Sumo Logic Hosted Collector to your Sumo Logic Org.

  2. Configure an HTTP Source for your Fastly request logs. Make sure to set the Source Category when configuring the HTTP source. For example, fastly. Make a note of the Source Category you assign to the source. You will provide this Source Category value when you install the Fastly app.

  3. Configure Fastly to send Sumo Logic the data in Log format version to 2 . (You can use the Fastly Documentation to determine which log file format you are using.)

    Fastly Create a Sumo Logic Endpoint

  4. After Step 5 in the Fastly documentation, you must set the log line format to "blank" for each configuration:
    FastlyConfigureLogtoBlank

  5. Then use this format string, which generates the necessary JSON output:

     {
     "service_id":"%{req.service_id}V",
     "service_version":"%{fastly_info.version}V",
     "time_start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
     "time_end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t",
     "time_elapsed":%{time.elapsed.usec}V,
     "client_ip":"%{req.http.Fastly-Client-IP}V",
     "request":"%{req.request}V",
     "protocol":"%{req.proto}V",
     "host":"%{req.http.Fastly-Orig-Host}V",
     "origin_host":"%{req.http.Host}V",
     "url":"%{cstr_escape(req.url)}V",
     "is_ipv6":%{if(req.is_ipv6, "true", "false")}V,
     "is_tls":%{if(req.is_ssl, "true", "false")}V,
     "tls_client_protocol":"%{cstr_escape(tls.client.protocol)}V",
     "tls_client_servername":"%{cstr_escape(tls.client.servername)}V",
     "tls_client_cipher":"%{cstr_escape(tls.client.cipher)}V",
     "tls_client_cipher_sha":"%{cstr_escape(tls.client.ciphers_sha )}V",
     "tls_client_tlsexts_sha":"%{cstr_escape(tls.client.tlsexts_sha)}V",
     "is_h2":%{if(fastly_info.is_h2, "true", "false")}V,
     "is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V,
     "h2_stream_id":"%{fastly_info.h2.stream_id}V",
     "request_referer":"%{cstr_escape(req.http.Referer)}V",
     "request_user_agent":"%{cstr_escape(req.http.User-Agent)}V",
     "request_accept_content":"%{cstr_escape(req.http.Accept)}V",
     "request_accept_language":"%{cstr_escape(req.http.Accept-Language)}V",
     "request_accept_encoding":"%{cstr_escape(req.http.Accept-Encoding)}V",
     "request_accept_charset":"%{cstr_escape(req.http.Accept-Charset)}V",
     "request_connection":"%{cstr_escape(req.http.Connection)}V",
     "request_dnt":"%{cstr_escape(req.http.DNT)}V",
     "request_forwarded":"%{cstr_escape(req.http.Forwarded)}V",
     "request_via":"%{cstr_escape(req.http.Via)}V",
     "request_cache_control":"%{cstr_escape(req.http.Cache-Control)}V",
     "request_x_requested_with":"%{cstr_escape(req.http.X-Requested-With)}V",
     "request_x_forwarded_for":"%{cstr_escape(req.http.X-Forwarded-For)}V",
     "status":"%{resp.status}V",
     "content_type":"%{cstr_escape(resp.http.Content-Type)}V",
     "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V",
     "is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V,
     "response_age":"%{cstr_escape(resp.http.Age)}V",
     "response_cache_control":"%{cstr_escape(resp.http.Cache-Control)}V",
     "response_expires":"%{cstr_escape(resp.http.Expires)}V",
     "response_last_modified":"%{cstr_escape(resp.http.Last-Modified)}V",
     "response_tsv":"%{cstr_escape(resp.http.TSV)}V",
     "geo_datacenter":"%{server.datacenter}V",
     "geo_city":"%{geoip.city}V",
     "geo_country_code":"%{geoip.country_code}V",
     "geo_continent_code":"%{geoip.continent_code}V",
     "geo_region":"%{geoip.region}V",
     "req_header_size":%{req.header_bytes_read}V,
     "req_body_size":%{req.body_bytes_read}V,
     "resp_header_size":%{resp.header_bytes_written}V,
     "resp_body_size":%{resp.body_bytes_written}V,
     "socket_cwnd":%{client.socket.cwnd}V,
     "socket_nexthop":"%{client.socket.nexthop}V",
     "socket_tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,
     "socket_tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,
     "socket_tcpi_rtt":%{client.socket.tcpi_rtt}V,
     "socket_tcpi_rttvar":%{client.socket.tcpi_rttvar}V,
     "socket_tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,
     "socket_tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,
     "socket_tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,
     "socket_tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,
     "socket_tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,
     "socket_ploss":%{client.socket.ploss}V
    }
    

Once you've done that hit save, deploy your version and your Sumo Logic account will start populating with data.

Collect WAF logs

Collect Request WAF logs

If you have WAF, add the following to the JSON object above, before the ending curly bracket (}):

"type":"request_logs",
"request_id":"%{req.http.x-request-id}V",
"waf_logged":"%{waf.logged}V",
"waf_block":"%{waf.blocked}V",
"waf_failures":"%{waf.failures}V",
"waf_rule_id":"%{waf.rule_id}V",
"waf_severity":"%{waf.severity}V",
"waf_passed":"%{waf.passed}V",
"waf_logdata":"%{cstr_escape(waf.logdata)}V",
"waf_executed":"%{waf.executed}V",
"waf_anomaly_score":"%{waf.anomaly_score}V",
"waf_sql_score":"%{waf.sql_injection_score}V",
"waf_rfi_score":"%{waf.rfi_score}V",
"waf_lfi_score":"%{waf.lfi_score}V",
"waf_xss_score":"%{waf.xss_score}V",
"waf_http_score":"%{waf.http_violation_score}V",
"waf_php_score":"%{waf.php_injection_score}V",
"waf_rce_score":"%{waf.rce_score}V",
"waf_session_fixation_score":"%{waf.session_fixation_score}V",
"waf_message":"%{cstr_escape(waf.message)}V"

 The x-request-id is based on creating or the existence of a header in your VCL with the following:

set req.http.x-request-id = digest.hash_sha256(now randomstr(64) req.http.host req.url req.http.Fastly-Client-IP server.identity);

Collect Debug WAF logs

  1. Configure another HTTP Source for debug logs and set it’s source category. Make a note of the Source Category you assign to the source. You will provide this Source Category value when you install the Fastly app.
  2. Create a second Fastly logging endpoint with the following string:
{"type":"debug_logs",
"service_id":"%{req.service_id}V",
"client_ip":"%{req.http.Fastly-Client-IP}V",
"request":"%{req.request}V",
"protocol":"%{req.proto}V",
"origin_host":"%{req.http.Host}V",
"url":"%{cstr_escape(req.url)}V",
"request_referer":"%{cstr_escape(req.http.Referer)}V", "request_user_agent":"%{cstr_escape(req.http.User-Agent)}V", "request_accept_content":"%{cstr_escape(req.http.Accept)}V",
"cache_status": "%{regsub(fastly_info.state, \"^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE|NONE)).*\", \"\\2\\3\")}V", "geo_datacenter":"%{server.datacenter}V",
"geo_city":"%{geoip.city}V",
"geo_country_code":"%{geoip.country_code}V",
"geo_continent_code":
"%{geoip.continent_code}V",
"geo_region":"%{geoip.region}V",
"request_id":"%{req.http.x-request-id}V",
"waf_logged":"%{waf.logged}V",
"waf_block":"%{waf.blocked}V",
"waf_failures":"%{waf.failures}V",
"waf_rule_id":"%{waf.rule_id}V",
"waf_severity":"%{waf.severity}V",
"waf_passed":"%{waf.passed}V",
"waf_logdata":"%{cstr_escape(waf.logdata)}V",
"waf_executed":"%{waf.executed}V",
"waf_anomaly_score":"%{waf.anomaly_score}V",
"waf_sql_score":"%{waf.sql_injection_score}V",
"waf_rfi_score":"%{waf.rfi_score}V",
"waf_lfi_score":"%{waf.lfi_score}V",
"waf_xss_score":"%{waf.xss_score}V",
"waf_http_score":"%{waf.http_violation_score}V",
"waf_php_score":"%{waf.php_injection_score}V",
"waf_rce_score":"%{waf.rce_score}V",
"waf_session_fixation_score":"%{waf.session_fixation_score}V",
"waf_message":"%{cstr_escape(waf.message)}V"
} 

Field Extraction Rule

This Field Extraction Rule (FER) is provided as an example to help you reduce your overall parsing time. Note that not all parse operators are supported in FERs. For more information, see Creating a Field Extraction Rule.

parse "\"reqMethod\":\"*\"" as method, "\"status\":\"*\"" as status, "\"fwdHost\":\"*\"" as origin| parse "\"bytes\":\"*\"" as bytes, "\"edgeIP\":\"*\"" as edgeip, "\"country\":\"*\"" as country, "\"cookie\":\"*\"" as cookie

Sample Queries

Top Error-causing URLs

_sourceCategory=fastly 50?
| parse "\"reqPath\":\"*\"" as path, "\"status\":\"*\"" as status
| urldecode(path) as path
| where status > 499
| where status < 600
| count as errors by path
| sort by errors

Cache Performance

_sourceCategory=fastly cacheStatus
| parse "\"cacheStatus\":\"*\"" as status
| where !(status="")
| if(status="0", "0 - Non cacheable", if(status="1" OR status="2", "1/2 - Cache Hit", if(status="3", "3 - Cache Miss", ""))) as cachestatus
| count by cachestatus

Top Denials by Host

_sourceCategory=fastly waf denyRules reqHost
| parse "\"denyRules\":\"*\"" as deny, "\"reqHost\":\"*\"" as host
| where deny != ""
| timeslice 1m
| count by host, _timeslice
| transpose row _timeslice column host