Collect Logs for Fastly
This page has instructions for setting up log collection for the Fastly app.
Step 1: Configure collector and source
In this step, you configure a collector and source to receive Fastly logs.
- Add a Sumo Logic Hosted Collector to your Sumo Logic Org.
- Configure an HTTP Source for your Fastly request logs. Make sure to set the Source Category when configuring the HTTP source. For example,
fastly
. Make a note of the Source Category you assign to the source. You will provide this Source Category value when you install the Fastly app.
Step 2. Configure endpoint in Fastly for CDN logs
In this step you add Sumo Logic as a logging endpoint for Fastly services, and configure it to receive CDN logs. The process is described in Adding Sumo Logic as a logging endpoint in Fastly help.
If you want to collect CDN and Request WAF logs, see Collect WAF Request logs below.
When you configure the Sumo Logic endpoint in Fastly:
- Name. Enter a name for the connection. For example, “Prod Fastly”.
- Log format. Use this format string, which generates the necessary JSON output:
{ "service_id":"%{req.service_id}V", "service_version":"%{fastly_info.version}V", "time_start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t", "time_end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t", "time_elapsed":%{time.elapsed.usec}V, "client_ip":"%{req.http.Fastly-Client-IP}V", "request":"%{req.request}V", "protocol":"%{req.proto}V", "host":"%{req.http.Fastly-Orig-Host}V", "origin_host":"%{req.http.Host}V", "url":"%{cstr_escape(req.url)}V", "is_ipv6":%{if(req.is_ipv6, "true", "false")}V, "is_tls":%{if(req.is_ssl, "true", "false")}V, "tls_client_protocol":"%{cstr_escape(tls.client.protocol)}V", "tls_client_servername":"%{cstr_escape(tls.client.servername)}V", "tls_client_cipher":"%{cstr_escape(tls.client.cipher)}V", "tls_client_cipher_sha":"%{cstr_escape(tls.client.ciphers_sha )}V", "tls_client_tlsexts_sha":"%{cstr_escape(tls.client.tlsexts_sha)}V", "is_h2":%{if(fastly_info.is_h2, "true", "false")}V, "is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V, "h2_stream_id":"%{fastly_info.h2.stream_id}V", "request_referer":"%{cstr_escape(req.http.Referer)}V", "request_user_agent":"%{cstr_escape(req.http.User-Agent)}V", "request_accept_content":"%{cstr_escape(req.http.Accept)}V", "request_accept_language":"%{cstr_escape(req.http.Accept-Language)}V", "request_accept_encoding":"%{cstr_escape(req.http.Accept-Encoding)}V", "request_accept_charset":"%{cstr_escape(req.http.Accept-Charset)}V", "request_connection":"%{cstr_escape(req.http.Connection)}V", "request_dnt":"%{cstr_escape(req.http.DNT)}V", "request_forwarded":"%{cstr_escape(req.http.Forwarded)}V", "request_via":"%{cstr_escape(req.http.Via)}V", "request_cache_control":"%{cstr_escape(req.http.Cache-Control)}V", "request_x_requested_with":"%{cstr_escape(req.http.X-Requested-With)}V", "request_x_forwarded_for":"%{cstr_escape(req.http.X-Forwarded-For)}V", "status":"%{resp.status}V", "content_type":"%{cstr_escape(resp.http.Content-Type)}V", "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V", "is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V, "response_age":"%{cstr_escape(resp.http.Age)}V", "response_cache_control":"%{cstr_escape(resp.http.Cache-Control)}V", "response_expires":"%{cstr_escape(resp.http.Expires)}V", "response_last_modified":"%{cstr_escape(resp.http.Last-Modified)}V", "response_tsv":"%{cstr_escape(resp.http.TSV)}V", "geo_datacenter":"%{server.datacenter}V", "geo_city":"%{geoip.city}V", "geo_country_code":"%{geoip.country_code}V", "geo_continent_code":"%{geoip.continent_code}V", "geo_region":"%{geoip.region}V", "req_header_size":%{req.header_bytes_read}V, "req_body_size":%{req.body_bytes_read}V, "resp_header_size":%{resp.header_bytes_written}V, "resp_body_size":%{resp.body_bytes_written}V, "socket_cwnd":%{client.socket.cwnd}V, "socket_nexthop":"%{client.socket.nexthop}V", "socket_tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V, "socket_tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V, "socket_tcpi_rtt":%{client.socket.tcpi_rtt}V, "socket_tcpi_rttvar":%{client.socket.tcpi_rttvar}V, "socket_tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V, "socket_tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V, "socket_tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V, "socket_tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V, "socket_tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V, "socket_ploss":%{client.socket.ploss}V }
- Collector URL. Enter the URL for the HTTP source you created in Step 1 above.
- Click Advanced options.
- By default the log line format is set to Classic. Change it to Blank.
- Click the Create button to create the new logging endpoint.
- Click the Activate button to deploy your configuration.
Step 3: Collect WAF Request logs
If you have Fastly's Web Application Firewall (WAF), perform these steps to update the configuration of the endpoint you created in Step 2 above. You are updating the endpoint to receive WAF Request logs as well as CDN logs.
- Use the JSON object below into the Log format field instead of the one specified in Configure endpoint in Fastly for CDN logs.
{"service_id":"%{req.service_id}V","service_version":"%{fastly_info.version}V","time_start":"%{begin:%Y-%m-%dT%H:%M:%S%Z}t","time_end":"%{end:%Y-%m-%dT%H:%M:%S%Z}t","time_elapsed":%{time.elapsed.usec}V,"client_ip":"%{req.http.Fastly-Client-IP}V","request":"%{req.request}V","protocol":"%{req.proto}V","host":"%{req.http.Fastly-Orig-Host}V","origin_host":"%{req.http.Host}V","url":"%{cstr_escape(req.url)}V","is_ipv6":%{if(req.is_ipv6,"true","false")}V,"is_tls":%{if(req.is_ssl,"true","false")}V,"tls_client_protocol":"%{cstr_escape(tls.client.protocol)}V","tls_client_servername":"%{cstr_escape(tls.client.servername)}V","tls_client_cipher":"%{cstr_escape(tls.client.cipher)}V","tls_client_cipher_sha":"%{cstr_escape(tls.client.ciphers_sha)}V","tls_client_tlsexts_sha":"%{cstr_escape(tls.client.tlsexts_sha)}V","is_h2":%{if(fastly_info.is_h2,"true","false")}V,"is_h2_push":%{if(fastly_info.h2.is_push,"true","false")}V,"h2_stream_id":"%{fastly_info.h2.stream_id}V","request_referer":"%{cstr_escape(req.http.Referer)}V","request_user_agent":"%{cstr_escape(req.http.User-Agent)}V","request_accept_content":"%{cstr_escape(req.http.Accept)}V","request_accept_language":"%{cstr_escape(req.http.Accept-Language)}V","request_accept_encoding":"%{cstr_escape(req.http.Accept-Encoding)}V","request_accept_charset":"%{cstr_escape(req.http.Accept-Charset)}V","request_connection":"%{cstr_escape(req.http.Connection)}V","request_dnt":"%{cstr_escape(req.http.DNT)}V","request_forwarded":"%{cstr_escape(req.http.Forwarded)}V","request_via":"%{cstr_escape(req.http.Via)}V","request_cache_control":"%{cstr_escape(req.http.Cache-Control)}V","request_x_requested_with":"%{cstr_escape(req.http.X-Requested-With)}V","request_x_forwarded_for":"%{cstr_escape(req.http.X-Forwarded-For)}V","status":"%{resp.status}V","content_type":"%{cstr_escape(resp.http.Content-Type)}V","cache_status":"%{regsub(fastly_info.state,"^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*","\\2\\3")}V","is_cacheable":%{if(fastly_info.state~"^(HIT|MISS)$","true","false")}V,"response_age":"%{cstr_escape(resp.http.Age)}V","response_cache_control":"%{cstr_escape(resp.http.Cache-Control)}V","response_expires":"%{cstr_escape(resp.http.Expires)}V","response_last_modified":"%{cstr_escape(resp.http.Last-Modified)}V","response_tsv":"%{cstr_escape(resp.http.TSV)}V","geo_datacenter":"%{server.datacenter}V","geo_city":"%{geoip.city}V","geo_country_code":"%{geoip.country_code}V","geo_continent_code":"%{geoip.continent_code}V","geo_region":"%{geoip.region}V","req_header_size":%{req.header_bytes_read}V,"req_body_size":%{req.body_bytes_read}V,"resp_header_size":%{resp.header_bytes_written}V,"resp_body_size":%{resp.body_bytes_written}V,"socket_cwnd":%{client.socket.cwnd}V,"socket_nexthop":"%{client.socket.nexthop}V","socket_tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,"socket_tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,"socket_tcpi_rtt":%{client.socket.tcpi_rtt}V,"socket_tcpi_rttvar":%{client.socket.tcpi_rttvar}V,"socket_tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,"socket_tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,"socket_tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,"socket_tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,"socket_tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,"socket_ploss":%{client.socket.ploss}V,"type":"request_logs","request_id":"%{req.http.x-request-id}V","waf_logged":"%{waf.logged}V","waf_block":"%{waf.blocked}V","waf_failures":"%{waf.failures}V","waf_rule_id":"%{waf.rule_id}V","waf_severity":"%{waf.severity}V","waf_passed":"%{waf.passed}V","waf_logdata":"%{cstr_escape(waf.logdata)}V","waf_executed":"%{waf.executed}V","waf_anomaly_score":"%{waf.anomaly_score}V","waf_sql_score":"%{waf.sql_injection_score}V","waf_rfi_score":"%{waf.rfi_score}V","waf_lfi_score":"%{waf.lfi_score}V","waf_xss_score":"%{waf.xss_score}V","waf_http_score":"%{waf.http_violation_score}V","waf_php_score":"%{waf.php_injection_score}V","waf_rce_score":"%{waf.rce_score}V","waf_session_fixation_score":"%{waf.session_fixation_score}V","waf_message":"%{cstr_escape(waf.message)}V"}
- Point the logging endpoint to the
waf_debug_log
subroutine using curl, as described in waf_debug_log in Fastly help. - Create a request_id header to track a single request.
Step 4: Collect WAF Debug logs
If you have Fastly's Web Application Firewall (WAF), perform these steps to add a second logging endpoint in Fastly and configure it to send WAF request logs to Sumo Logic.
- Configure another HTTP Source for debug logs and set its source category. For example, fastly/debug. Make a note of the Source Category you assign to the source. You will provide this Source Category value when you install the Fastly app.
- Create another logging endpoint in Fastly following the instructions Step 2, but enter the JSON below in the Log format field.
{"type":"debug_logs", "service_id":"%{req.service_id}V", "client_ip":"%{req.http.Fastly-Client-IP}V", "request":"%{req.request}V", "protocol":"%{req.proto}V", "origin_host":"%{req.http.Host}V", "url":"%{cstr_escape(req.url)}V", "request_referer":"%{cstr_escape(req.http.Referer)}V", "request_user_agent":"%{cstr_escape(req.http.User-Agent)}V", "request_accept_content":"%{cstr_escape(req.http.Accept)}V", "cache_status": "%{regsub(fastly_info.state, \"^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE|NONE)).*\", \"\\2\\3\")}V", "geo_datacenter":"%{server.datacenter}V", "geo_city":"%{geoip.city}V", "geo_country_code":"%{geoip.country_code}V", "geo_continent_code": "%{geoip.continent_code}V", "geo_region":"%{geoip.region}V", "request_id":"%{req.http.x-request-id}V", "waf_logged":"%{waf.logged}V", "waf_block":"%{waf.blocked}V", "waf_failures":"%{waf.failures}V", "waf_rule_id":"%{waf.rule_id}V", "waf_severity":"%{waf.severity}V", "waf_passed":"%{waf.passed}V", "waf_logdata":"%{cstr_escape(waf.logdata)}V", "waf_executed":"%{waf.executed}V", "waf_anomaly_score":"%{waf.anomaly_score}V", "waf_sql_score":"%{waf.sql_injection_score}V", "waf_rfi_score":"%{waf.rfi_score}V", "waf_lfi_score":"%{waf.lfi_score}V", "waf_xss_score":"%{waf.xss_score}V", "waf_http_score":"%{waf.http_violation_score}V", "waf_php_score":"%{waf.php_injection_score}V", "waf_rce_score":"%{waf.rce_score}V", "waf_session_fixation_score":"%{waf.session_fixation_score}V", "waf_message":"%{cstr_escape(waf.message)}V" }
Field Extraction Rule
This Field Extraction Rule (FER) is provided as an example to help you reduce your overall parsing time. Note that not all parse operators are supported in FERs. For more information, see Creating a Field Extraction Rule.
parse "\"reqMethod\":\"*\"" as method, "\"status\":\"*\"" as status, "\"fwdHost\":\"*\"" as origin| parse "\"bytes\":\"*\"" as bytes, "\"edgeIP\":\"*\"" as edgeip, "\"country\":\"*\"" as country, "\"cookie\":\"*\"" as cookie
Sample Queries
Top Error-causing URLs
_sourceCategory=fastly 50? | parse "\"reqPath\":\"*\"" as path, "\"status\":\"*\"" as status | urldecode(path) as path | where status > 499 | where status < 600 | count as errors by path | sort by errors
Cache Performance
_sourceCategory=fastly cacheStatus | parse "\"cacheStatus\":\"*\"" as status | where !(status="") | if(status="0", "0 - Non cacheable", if(status="1" OR status="2", "1/2 - Cache Hit", if(status="3", "3 - Cache Miss", ""))) as cachestatus | count by cachestatus
Top Denials by Host
_sourceCategory=fastly waf denyRules reqHost | parse "\"denyRules\":\"*\"" as deny, "\"reqHost\":\"*\"" as host | where deny != "" | timeslice 1m | count by host, _timeslice | transpose row _timeslice column host