Working with Fastly CDN Logs

Hydrolix supports native integration with Fastly’s real-time log streaming service via an HTTPS Endpoint. The combination of the two technologies offers a simple, yet powerful means of extending logging and data analytics capabilities without the associated expense.

The following walkthrough covers both the Fastly and Hydrolix configuration settings to enable the complete solution.

Similar documentation can also be found at the Fastly Integrations website:
Fastly Log Streaming Guide for Hydrolix

Setup Fastly Log Streaming: HTTPS

Prerequisites

When sending logs to an HTTPS endpoint, Fastly requires proof that you control the domain name specified in the endpoint URL field. This is accomplished by using an HTTP challenge on a well-known path. Fortunately, Hydrolix has a pre-configured path for quick validation.

Hydrolix pre-configured path for Fastly endpoint URL verification:

https://<hydrolix_instance_name>.hydrolix.live/.well-known/fastly/logging/challenge

By default, Hydrolix uses an asterisk (*) to allow any service to post to the HTTP endpoint.

Configure the HTTPS Logging Endpoint

In this example, we will create a new HTTPS logging endpoint that will be used to send real-time streaming logs to Hydrolix.

1. Login to Fastly

Login to manage.fastly.com and choose the appropriate service.

2. Create an HTTPS Logging Endpoint

Under the desired Fastly service configuration, select:

  • Logging
  • HTTPS Endpoint

Enter the following configuration options:

Name: <Name for the Hydrolix logging endpoint>

Log format: In order to better leverage the value provided by the Hydrolix platform from a data compression and query performance perspective, we will be using an extended version of the Fastly streaming log format that increases the logged fields from the default of 14 to 64 in our example below.

For reference, the 14-field default log format for a Fastly HTTPS Logging Endpoint can be found here.

Extended Fastly Log Format

Click arrow to expand

{
    "service_id": "%{req.service_id}V",
    "service_version": "%{fastly_info.version}V",
    "time_start": "%{begin:%Y-%m-%dT%H:%M:%S}t",
    "time_end": "%{end:%Y-%m-%dT%H:%M:%S}t",
    "time_elapsed":%{time.elapsed.usec}V,
    "client_ip": "%{req.http.Fastly-Client-IP}V",
    "request": "%{req.request}V",
    "protocol": "%{req.proto}V",
    "host": "%{req.http.Fastly-Orig-Host}V",
    "origin_host": "%{req.http.Host}V",
    "url": "%{cstr_escape(req.url)}V",
    "is_ipv6":%{if(req.is_ipv6, "true", "false")}V,
    "is_tls":%{if(req.is_ssl, "true", "false")}V,
    "tls_client_protocol": "%{cstr_escape(tls.client.protocol)}V",
    "tls_client_servername": "%{cstr_escape(tls.client.servername)}V",
    "tls_client_cipher": "%{cstr_escape(tls.client.cipher)}V",
    "tls_client_cipher_sha": "%{cstr_escape(tls.client.ciphers_sha )}V",
    "tls_client_tlsexts_sha": "%{cstr_escape(tls.client.tlsexts_sha)}V",
    "is_h2":%{if(fastly_info.is_h2, "true", "false")}V,
    "is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V,
    "h2_stream_id": "%{fastly_info.h2.stream_id}V",
    "request_referer": "%{cstr_escape(req.http.Referer)}V",
    "request_user_agent": "%{cstr_escape(req.http.User-Agent)}V",
    "request_accept_content": "%{cstr_escape(req.http.Accept)}V",
    "request_accept_language": "%{cstr_escape(req.http.Accept-Language)}V",
    "request_accept_encoding": "%{cstr_escape(req.http.Accept-Encoding)}V",
    "request_accept_charset": "%{cstr_escape(req.http.Accept-Charset)}V",
    "request_connection": "%{cstr_escape(req.http.Connection)}V",
    "request_dnt": "%{cstr_escape(req.http.DNT)}V",
    "request_forwarded": "%{cstr_escape(req.http.Forwarded)}V",
    "request_via": "%{cstr_escape(req.http.Via)}V",
    "request_cache_control": "%{cstr_escape(req.http.Cache-Control)}V",
    "request_x_requested_with": "%{cstr_escape(req.http.X-Requested-With)}V",
    "request_x_forwarded_for": "%{cstr_escape(req.http.X-Forwarded-For)}V",
    "status": "%{resp.status}V",
    "content_type": "%{cstr_escape(resp.http.Content-Type)}V",
    "cache_status": "%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V",
    "is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V,
    "response_age": "%{cstr_escape(resp.http.Age)}V",
    "response_cache_control": "%{cstr_escape(resp.http.Cache-Control)}V",
    "response_expires": "%{cstr_escape(resp.http.Expires)}V",
    "response_last_modified": "%{cstr_escape(resp.http.Last-Modified)}V",
    "response_tsv": "%{cstr_escape(resp.http.TSV)}V",
    "geo_datacenter": "%{server.datacenter}V",
    "geo_city": "%{geoip.city}V",
    "geo_country_code": "%{geoip.country_code}V",
    "geo_continent_code": "%{geoip.continent_code}V",
    "geo_region": "%{geoip.region}V",
    "req_header_size":%{req.header_bytes_read}V,
    "req_body_size":%{req.body_bytes_read}V,
    "resp_header_size":%{resp.header_bytes_written}V,
    "resp_body_size":%{resp.body_bytes_written}V,
    "socket_cwnd":%{client.socket.cwnd}V,
    "socket_nexthop": "%{client.socket.nexthop}V",
    "socket_tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,
    "socket_tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,
    "socket_tcpi_rtt":%{client.socket.tcpi_rtt}V,
    "socket_tcpi_rttvar":%{client.socket.tcpi_rttvar}V,
    "socket_tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,
    "socket_tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,
    "socket_tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,
    "socket_tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,
    "socket_tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,
    "socket_ploss":%{client.socket.ploss}V
}

URL: The streaming API endpoint for your Hydrolix instance:

https://<hydrolix-instance-name>.hydrolix.live/ingest/event

Maximum logs: 0
Maximum bytes: 0

EXPAND "Advanced options"

Content type: application/json
Custom header name: x-hdx-table
Custom header value: <hydrolix_project_name>.<hydrolix_table_name>
Method: POST
JSON log entry format: Newline delimited
Select a log line format: Blank
Placement: Format Version Default

Using your own certificate authority (CA)?
Leave all values empty or customize as required


Congratulations! Fastly has now been configured to send real-time streaming logs to the Hydrolix endpoint. On that note, we had better move on to the next section and ensure that the Hydrolix platform is properly configured to receive them.

Setup Hydrolix Streaming Intake

Welcome to the Hydrolix portion of this walkthrough. Please note that the following assumes that you have already created a project and an associated table. If you haven’t completed those steps yet, more information can be found in the Building Your Data Store section of the Hydrolix documentation website.

Create a Transform

Once you have a project and table created, the last remaining step to getting Fastly data into the Hydrolix platform is to create a transform. Think of a transform as a schema that helps tell the system what type of data to expect and ultimately how to deal with it. A more formal definition can be found here but for now we’ll save you some time and provide the full transform for the Fastly extended logs below:

Fastly Transform

Click arrow to expand

{
  "table": "<<table_uuid>>",
  "name": "<string>",
  "description": "<string>",
  "type": "json",
  "settings": {
    "is_default": true,
    "compression": "none",
    "format_details": {
    },
    "output_columns": [
      {
        "name": "cache_status",
        "datatype": {
          "type": "string",
          "index": true
        }
      },  
      {
        "name": "client_ip",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "content_type",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "geo_city",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "geo_continent_code",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "geo_country_code",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "geo_datacenter",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "geo_region",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "h2_stream_id",
        "datatype": {
          "type": "string"
        }
      },
      {
        "name": "host",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "is_cacheable",
        "datatype": {
          "type": "bool"
        }
      },
      {
        "name": "is_h2",
        "datatype": {
          "type": "bool"
        }
      },
      {
        "name": "is_h2_push",
        "datatype": {
          "type": "bool"
        }
      },
      {
        "name": "is_ipv6",
        "datatype": {
          "type": "bool"
        }
      },
      {
        "name": "is_tls",
        "datatype": {
          "type": "bool"
        }
      },
      {
        "name": "origin_host",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "protocol",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "req_body_size",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "req_header_size",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "request",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_accept_charset",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_accept_content",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_accept_encoding",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_accept_language",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_cache_control",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_connection",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_dnt",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_forwarded",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_referer",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_user_agent",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_via",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_x_forwarded_for",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "request_x_requested_with",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "resp_body_size",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "resp_header_size",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "response_age",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "response_cache_control",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "response_expires",
        "datatype": {
          "type": "string"
        }
      },
      {
        "name": "response_last_modified",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "response_tsv",
        "datatype": {
          "type": "string"
        }
      },
      {
        "name": "service_id",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "service_version",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "socket_cwnd",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "socket_nexthop",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "socket_ploss",
        "datatype": {
          "type": "double"
        }
      },
      {
        "name": "socket_tcpi_delta_retrans",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "socket_tcpi_last_data_sent",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "socket_tcpi_rcv_mss",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "socket_tcpi_rcv_rtt",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "socket_tcpi_rcv_space",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "socket_tcpi_rtt",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "socket_tcpi_rttvar",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "socket_tcpi_snd_mss",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "socket_tcpi_total_retrans",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "status",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "time_elapsed",
        "datatype": {
          "type": "uint32"
        }
      },
      {
        "name": "time_end",
        "datatype": {
          "type": "datetime",
          "format": "2006-01-02T15:04:05"
        }
      },
      {
        "name": "time_start",
        "datatype":{
          "type": "datetime",
          "primary": true,
          "format": "2006-01-02T15:04:05"
        }
      },
      {
        "name": "tls_client_cipher",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "tls_client_cipher_sha",
        "datatype": {
          "type": "string"
        }
      },
      {
        "name": "tls_client_protocol",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "tls_client_servername",
        "datatype": {
          "type": "string",
          "index": true
        }
      },
      {
        "name": "tls_client_tlsexts_sha",
        "datatype": {
          "type": "string"
        }
      },
      {
        "name": "url",
        "datatype": {
          "type": "string",
          "index": true
        }
      }
    ]
  }
}

Leveraging Views

At this point, Hydrolix is configured to accept the incoming Fastly log data.

Hydrolix supports the notion that a single data set can have many different query formats. The query data structure, or view, associated with a given data set not only allows for a customized representation of the queried data but also for a user’s access to the data to be restricted to a set of columns. Upon transform creation, Hydrolix automatically generates a default view that can be used to immediately query the data set - no additional configuration is required. However, users are encouraged to spend some time becoming familiar with the view concept and subsequent benefits that the feature can provide. More detailed information can be found here.

Querying Fastly Logs with Hydrolix

Now that both solutions are configured and Fastly real-time log data is streaming into Hydrolix, it’s time to make some queries! We’ve prepared a short tutorial to help get you started: Analyzing Transaction Logs