Collect Logs for Fastly

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

This page has instructions for setting up log collection for the Fastly app.

Step 1: Configure collector and source

In this step, you configure a collector and source to receive Fastly logs.

Add a Sumo Logic Hosted Collector to your Sumo Logic Org.
Configure an HTTP Source for your Fastly request logs. Make sure to set the Source Category when configuring the HTTP source. For example, fastly. Make a note of the Source Category you assign to the source. You will provide this Source Category value when you install the Fastly app.

Step 2. Configure endpoint in Fastly for CDN logs

In this step you add Sumo Logic as a logging endpoint for Fastly services, and configure it to receive CDN logs. The process is described in Adding Sumo Logic as a logging endpoint in Fastly help.

If you want to collect CDN and Request WAF logs, see Collect WAF Request logs below.

When you configure the Sumo Logic endpoint in Fastly:

Name. Enter a name for the connection. For example, “Prod Fastly”.
Log format. Use this format string, which generates the necessary JSON output:

{
 "service_id":"%{req.service_id}V",
 "service_version":"%{fastly_info.version}V",
 "time_start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t",
 "time_end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t",
 "time_elapsed":%{time.elapsed.usec}V,
 "client_ip":"%{req.http.Fastly-Client-IP}V",
 "request":"%{req.request}V",
 "protocol":"%{req.proto}V",
 "host":"%{req.http.Fastly-Orig-Host}V",
 "origin_host":"%{req.http.Host}V",
 "url":"%{cstr_escape(req.url)}V",
 "is_ipv6":%{if(req.is_ipv6, "true", "false")}V,
 "is_tls":%{if(req.is_ssl, "true", "false")}V,
 "tls_client_protocol":"%{cstr_escape(tls.client.protocol)}V",
 "tls_client_servername":"%{cstr_escape(tls.client.servername)}V",
 "tls_client_cipher":"%{cstr_escape(tls.client.cipher)}V",
 "tls_client_cipher_sha":"%{cstr_escape(tls.client.ciphers_sha )}V",
 "tls_client_tlsexts_sha":"%{cstr_escape(tls.client.tlsexts_sha)}V",
 "is_h2":%{if(fastly_info.is_h2, "true", "false")}V,
 "is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V,
 "h2_stream_id":"%{fastly_info.h2.stream_id}V",
 "request_referer":"%{cstr_escape(req.http.Referer)}V",
 "request_user_agent":"%{cstr_escape(req.http.User-Agent)}V",
 "request_accept_content":"%{cstr_escape(req.http.Accept)}V",
 "request_accept_language":"%{cstr_escape(req.http.Accept-Language)}V",
 "request_accept_encoding":"%{cstr_escape(req.http.Accept-Encoding)}V",
 "request_accept_charset":"%{cstr_escape(req.http.Accept-Charset)}V",
 "request_connection":"%{cstr_escape(req.http.Connection)}V",
 "request_dnt":"%{cstr_escape(req.http.DNT)}V",
 "request_forwarded":"%{cstr_escape(req.http.Forwarded)}V",
 "request_via":"%{cstr_escape(req.http.Via)}V",
 "request_cache_control":"%{cstr_escape(req.http.Cache-Control)}V",
 "request_x_requested_with":"%{cstr_escape(req.http.X-Requested-With)}V",
 "request_x_forwarded_for":"%{cstr_escape(req.http.X-Forwarded-For)}V",
 "status":"%{resp.status}V",
 "content_type":"%{cstr_escape(resp.http.Content-Type)}V",
 "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V",
 "is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V,
 "response_age":"%{cstr_escape(resp.http.Age)}V",
 "response_cache_control":"%{cstr_escape(resp.http.Cache-Control)}V",
 "response_expires":"%{cstr_escape(resp.http.Expires)}V",
 "response_last_modified":"%{cstr_escape(resp.http.Last-Modified)}V",
 "response_tsv":"%{cstr_escape(resp.http.TSV)}V",
 "geo_datacenter":"%{server.datacenter}V",
 "geo_city":"%{geoip.city}V",
 "geo_country_code":"%{geoip.country_code}V",
 "geo_continent_code":"%{geoip.continent_code}V",
 "geo_region":"%{geoip.region}V",
 "req_header_size":%{req.header_bytes_read}V,
 "req_body_size":%{req.body_bytes_read}V,
 "resp_header_size":%{resp.header_bytes_written}V,
 "resp_body_size":%{resp.body_bytes_written}V,
 "socket_cwnd":%{client.socket.cwnd}V,
 "socket_nexthop":"%{client.socket.nexthop}V",
 "socket_tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,
 "socket_tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,
 "socket_tcpi_rtt":%{client.socket.tcpi_rtt}V,
 "socket_tcpi_rttvar":%{client.socket.tcpi_rttvar}V,
 "socket_tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,
 "socket_tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,
 "socket_tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,
 "socket_tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,
 "socket_tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,
 "socket_ploss":%{client.socket.ploss}V
}

Collector URL. Enter the URL for the HTTP source you created in Step 1 above.
Click Advanced options.
By default the log line format is set to Classic. Change it to Blank.
Click the Create button to create the new logging endpoint.
Click the Activate button to deploy your configuration.

Step 3: Collect WAF Request logs

If you have Fastly's Web Application Firewall (WAF), perform these steps to update the configuration of the endpoint you created in Step 2 above. You are updating the endpoint to receive WAF Request logs as well as CDN logs.

Use the JSON object below into the Log format field instead of the one specified in Configure endpoint in Fastly for CDN logs.



{"service_id":"%{req.service_id}V","service_version":"%{fastly_info.version}V","time_start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t","time_end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t","time_elapsed":%{time.elapsed.usec}V,"client_ip":"%{req.http.Fastly-Client-IP}V","request":"%{req.request}V","protocol":"%{req.proto}V","host":"%{req.http.Fastly-Orig-Host}V","origin_host":"%{req.http.Host}V","url":"%{cstr_escape(req.url)}V","is_ipv6":%{if(req.is_ipv6,"true","false")}V,"is_tls":%{if(req.is_ssl,"true","false")}V,"tls_client_protocol":"%{cstr_escape(tls.client.protocol)}V","tls_client_servername":"%{cstr_escape(tls.client.servername)}V","tls_client_cipher":"%{cstr_escape(tls.client.cipher)}V","tls_client_cipher_sha":"%{cstr_escape(tls.client.ciphers_sha)}V","tls_client_tlsexts_sha":"%{cstr_escape(tls.client.tlsexts_sha)}V","is_h2":%{if(fastly_info.is_h2,"true","false")}V,"is_h2_push":%{if(fastly_info.h2.is_push,"true","false")}V,"h2_stream_id":"%{fastly_info.h2.stream_id}V","request_referer":"%{cstr_escape(req.http.Referer)}V","request_user_agent":"%{cstr_escape(req.http.User-Agent)}V","request_accept_content":"%{cstr_escape(req.http.Accept)}V","request_accept_language":"%{cstr_escape(req.http.Accept-Language)}V","request_accept_encoding":"%{cstr_escape(req.http.Accept-Encoding)}V","request_accept_charset":"%{cstr_escape(req.http.Accept-Charset)}V","request_connection":"%{cstr_escape(req.http.Connection)}V","request_dnt":"%{cstr_escape(req.http.DNT)}V","request_forwarded":"%{cstr_escape(req.http.Forwarded)}V","request_via":"%{cstr_escape(req.http.Via)}V","request_cache_control":"%{cstr_escape(req.http.Cache-Control)}V","request_x_requested_with":"%{cstr_escape(req.http.X-Requested-With)}V","request_x_forwarded_for":"%{cstr_escape(req.http.X-Forwarded-For)}V","status":"%{resp.status}V","content_type":"%{cstr_escape(resp.http.Content-Type)}V","cache_status":"%{regsub(fastly_info.state,"^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*","\\2\\3")}V","is_cacheable":%{if(fastly_info.state~"^(HIT|MISS)$","true","false")}V,"response_age":"%{cstr_escape(resp.http.Age)}V","response_cache_control":"%{cstr_escape(resp.http.Cache-Control)}V","response_expires":"%{cstr_escape(resp.http.Expires)}V","response_last_modified":"%{cstr_escape(resp.http.Last-Modified)}V","response_tsv":"%{cstr_escape(resp.http.TSV)}V","geo_datacenter":"%{server.datacenter}V","geo_city":"%{geoip.city}V","geo_country_code":"%{geoip.country_code}V","geo_continent_code":"%{geoip.continent_code}V","geo_region":"%{geoip.region}V","req_header_size":%{req.header_bytes_read}V,"req_body_size":%{req.body_bytes_read}V,"resp_header_size":%{resp.header_bytes_written}V,"resp_body_size":%{resp.body_bytes_written}V,"socket_cwnd":%{client.socket.cwnd}V,"socket_nexthop":"%{client.socket.nexthop}V","socket_tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,"socket_tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,"socket_tcpi_rtt":%{client.socket.tcpi_rtt}V,"socket_tcpi_rttvar":%{client.socket.tcpi_rttvar}V,"socket_tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,"socket_tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,"socket_tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,"socket_tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,"socket_tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,"socket_ploss":%{client.socket.ploss}V,"type":"request_logs","request_id":"%{req.http.x-request-id}V","waf_logged":"%{waf.logged}V","waf_block":"%{waf.blocked}V","waf_failures":"%{waf.failures}V","waf_rule_id":"%{waf.rule_id}V","waf_severity":"%{waf.severity}V","waf_passed":"%{waf.passed}V","waf_logdata":"%{cstr_escape(waf.logdata)}V","waf_executed":"%{waf.executed}V","waf_anomaly_score":"%{waf.anomaly_score}V","waf_sql_score":"%{waf.sql_injection_score}V","waf_rfi_score":"%{waf.rfi_score}V","waf_lfi_score":"%{waf.lfi_score}V","waf_xss_score":"%{waf.xss_score}V","waf_http_score":"%{waf.http_violation_score}V","waf_php_score":"%{waf.php_injection_score}V","waf_rce_score":"%{waf.rce_score}V","waf_session_fixation_score":"%{waf.session_fixation_score}V","waf_message":"%{cstr_escape(waf.message)}V"}

Point the logging endpoint to the waf_debug_log subroutine using curl, as described in waf_debug_log in Fastly help.
Create a request_id header to track a single request.

Step 4: Collect WAF Debug logs

If you have Fastly's Web Application Firewall (WAF), perform these steps to add a second logging endpoint in Fastly and configure it to send WAF request logs to Sumo Logic.

Configure another HTTP Source for debug logs and set its source category. For example, fastly/debug. Make a note of the Source Category you assign to the source. You will provide this Source Category value when you install the Fastly app.
Create another logging endpoint in Fastly following the instructions Step 2, but enter the JSON below in the Log format field.



{"type":"debug_logs",
"service_id":"%{req.service_id}V",
"client_ip":"%{req.http.Fastly-Client-IP}V",
"request":"%{req.request}V",
"protocol":"%{req.proto}V",
"origin_host":"%{req.http.Host}V",
"url":"%{cstr_escape(req.url)}V",
"request_referer":"%{cstr_escape(req.http.Referer)}V", "request_user_agent":"%{cstr_escape(req.http.User-Agent)}V", "request_accept_content":"%{cstr_escape(req.http.Accept)}V",
"cache_status": "%{regsub(fastly_info.state, \"^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE|NONE)).*\", \"\\2\\3\")}V", "geo_datacenter":"%{server.datacenter}V",
"geo_city":"%{geoip.city}V",
"geo_country_code":"%{geoip.country_code}V",
"geo_continent_code":
"%{geoip.continent_code}V",
"geo_region":"%{geoip.region}V",
"request_id":"%{req.http.x-request-id}V",
"waf_logged":"%{waf.logged}V",
"waf_block":"%{waf.blocked}V",
"waf_failures":"%{waf.failures}V",
"waf_rule_id":"%{waf.rule_id}V",
"waf_severity":"%{waf.severity}V",
"waf_passed":"%{waf.passed}V",
"waf_logdata":"%{cstr_escape(waf.logdata)}V",
"waf_executed":"%{waf.executed}V",
"waf_anomaly_score":"%{waf.anomaly_score}V",
"waf_sql_score":"%{waf.sql_injection_score}V",
"waf_rfi_score":"%{waf.rfi_score}V",
"waf_lfi_score":"%{waf.lfi_score}V",
"waf_xss_score":"%{waf.xss_score}V",
"waf_http_score":"%{waf.http_violation_score}V",
"waf_php_score":"%{waf.php_injection_score}V",
"waf_rce_score":"%{waf.rce_score}V",
"waf_session_fixation_score":"%{waf.session_fixation_score}V",
"waf_message":"%{cstr_escape(waf.message)}V"
}

Field Extraction Rule

This Field Extraction Rule (FER) is provided as an example to help you reduce your overall parsing time. Note that not all parse operators are supported in FERs. For more information, see Creating a Field Extraction Rule.

parse "\"reqMethod\":\"*\"" as method, "\"status\":\"*\"" as status, "\"fwdHost\":\"*\"" as origin| parse "\"bytes\":\"*\"" as bytes, "\"edgeIP\":\"*\"" as edgeip, "\"country\":\"*\"" as country, "\"cookie\":\"*\"" as cookie

Sample Queries

Top Error-causing URLs

Cache Performance

_sourceCategory=fastly cacheStatus | parse "\"cacheStatus\":\"*\"" as status | where !(status="") | if(status="0", "0 - Non cacheable", if(status="1" OR status="2", "1/2 - Cache Hit", if(status="3", "3 - Cache Miss", ""))) as cachestatus | count by cachestatus

Top Denials by Host