Work with Fastly CDN Logs
Hydrolix supports native integration with Fastly's real-time log streaming service via an HTTPS Endpoint. The combination of the two technologies offers a simple, yet powerful means of extending logging and data analytics capabilities without the associated expense.
The following walkthrough covers both the Fastly and Hydrolix configuration settings to enable the complete solution.
Similar documentation can also be found at the Fastly Integrations website:
Fastly Log Streaming Guide for Hydrolix
Setup Fastly Log Streaming: HTTPS
Prerequisites
When sending logs to an HTTPS endpoint, Fastly requires proof that you control the domain name specified in the endpoint URL field. This is accomplished by using an HTTP challenge on a well-known path. Fortunately, Hydrolix has a pre-configured path for quick validation.
Hydrolix pre-configured path for Fastly endpoint URL verification:
https://<hydrolix_instance_name>.hydrolix.live/.well-known/fastly/logging/challenge
By default, Hydrolix uses an asterisk (*) to allow any service to post to the HTTP endpoint.
Configure the HTTPS Logging Endpoint
In this example, we will create a new HTTPS logging endpoint that will be used to send real-time streaming logs to Hydrolix.
1. Login to Fastly
Login to manage.fastly.com and choose the appropriate service.
2. Create an HTTPS Logging Endpoint
Under the desired Fastly service configuration, select:
- Logging
- HTTPS Endpoint
Enter the following configuration options:
Name: <Name for the Hydrolix logging endpoint>
Log format: In order to better leverage the value provided by the Hydrolix platform from a data compression and query performance perspective, we will be using an extended version of the Fastly streaming log format that increases the logged fields from the default of 14 to 64 in our example below.
For reference, the 14-field default log format for a Fastly HTTPS Logging Endpoint can be found here.
Extended Fastly Log Format Click arrow to expand
{
"service_id": "%{req.service_id}V",
"service_version": "%{fastly_info.version}V",
"time_start": "%{begin:%Y-%m-%dT%H:%M:%S}t",
"time_end": "%{end:%Y-%m-%dT%H:%M:%S}t",
"time_elapsed":%{time.elapsed.usec}V,
"client_ip": "%{req.http.Fastly-Client-IP}V",
"request": "%{req.request}V",
"protocol": "%{req.proto}V",
"host": "%{req.http.Fastly-Orig-Host}V",
"origin_host": "%{req.http.Host}V",
"url": "%{cstr_escape(req.url)}V",
"is_ipv6":%{if(req.is_ipv6, "true", "false")}V,
"is_tls":%{if(req.is_ssl, "true", "false")}V,
"tls_client_protocol": "%{cstr_escape(tls.client.protocol)}V",
"tls_client_servername": "%{cstr_escape(tls.client.servername)}V",
"tls_client_cipher": "%{cstr_escape(tls.client.cipher)}V",
"tls_client_cipher_sha": "%{cstr_escape(tls.client.ciphers_sha )}V",
"tls_client_tlsexts_sha": "%{cstr_escape(tls.client.tlsexts_sha)}V",
"is_h2":%{if(fastly_info.is_h2, "true", "false")}V,
"is_h2_push":%{if(fastly_info.h2.is_push, "true", "false")}V,
"h2_stream_id": "%{fastly_info.h2.stream_id}V",
"request_referer": "%{cstr_escape(req.http.Referer)}V",
"request_user_agent": "%{cstr_escape(req.http.User-Agent)}V",
"request_accept_content": "%{cstr_escape(req.http.Accept)}V",
"request_accept_language": "%{cstr_escape(req.http.Accept-Language)}V",
"request_accept_encoding": "%{cstr_escape(req.http.Accept-Encoding)}V",
"request_accept_charset": "%{cstr_escape(req.http.Accept-Charset)}V",
"request_connection": "%{cstr_escape(req.http.Connection)}V",
"request_dnt": "%{cstr_escape(req.http.DNT)}V",
"request_forwarded": "%{cstr_escape(req.http.Forwarded)}V",
"request_via": "%{cstr_escape(req.http.Via)}V",
"request_cache_control": "%{cstr_escape(req.http.Cache-Control)}V",
"request_x_requested_with": "%{cstr_escape(req.http.X-Requested-With)}V",
"request_x_forwarded_for": "%{cstr_escape(req.http.X-Forwarded-For)}V",
"status": "%{resp.status}V",
"content_type": "%{cstr_escape(resp.http.Content-Type)}V",
"cache_status": "%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3")}V",
"is_cacheable":%{if(fastly_info.state ~"^(HIT|MISS)$", "true", "false")}V,
"response_age": "%{cstr_escape(resp.http.Age)}V",
"response_cache_control": "%{cstr_escape(resp.http.Cache-Control)}V",
"response_expires": "%{cstr_escape(resp.http.Expires)}V",
"response_last_modified": "%{cstr_escape(resp.http.Last-Modified)}V",
"response_tsv": "%{cstr_escape(resp.http.TSV)}V",
"geo_datacenter": "%{server.datacenter}V",
"geo_city": "%{client.geo.city}V",
"geo_country_code": "%{client.geo.country_code}V",
"geo_continent_code": "%{client.geo.continent_code}V",
"geo_region": "%{client.geo.region}V",
"req_header_size":%{req.header_bytes_read}V,
"req_body_size":%{req.body_bytes_read}V,
"resp_header_size":%{resp.header_bytes_written}V,
"resp_body_size":%{resp.body_bytes_written}V,
"socket_cwnd":%{client.socket.cwnd}V,
"socket_nexthop": "%{client.socket.nexthop}V",
"socket_tcpi_rcv_mss":%{client.socket.tcpi_rcv_mss}V,
"socket_tcpi_snd_mss":%{client.socket.tcpi_snd_mss}V,
"socket_tcpi_rtt":%{client.socket.tcpi_rtt}V,
"socket_tcpi_rttvar":%{client.socket.tcpi_rttvar}V,
"socket_tcpi_rcv_rtt":%{client.socket.tcpi_rcv_rtt}V,
"socket_tcpi_rcv_space":%{client.socket.tcpi_rcv_space}V,
"socket_tcpi_last_data_sent":%{client.socket.tcpi_last_data_sent}V,
"socket_tcpi_total_retrans":%{client.socket.tcpi_total_retrans}V,
"socket_tcpi_delta_retrans":%{client.socket.tcpi_delta_retrans}V,
"socket_ploss":%{client.socket.ploss}V
}
URL: The streaming API endpoint for your Hydrolix instance:
https://<hydrolix-instance-name>.hydrolix.live/ingest/event
Maximum logs: 0
Maximum bytes: 0
EXPAND "Advanced options"
Content type: application/json
Custom header name: x-hdx-table
Custom header value: <hydrolix_project_name>.<hydrolix_table_name>
Method: POST
JSON log entry format: Newline delimited
Select a log line format: Blank
Placement: Format Version Default
Using your own certificate authority (CA)?
Leave all values empty or customize as required
Congratulations! Fastly has now been configured to send real-time streaming logs to the Hydrolix endpoint. On that note, we had better move on to the next section and ensure that the Hydrolix platform is properly configured to receive them.
Setup Hydrolix Streaming Intake
Welcome to the Hydrolix portion of this walkthrough. Please note that the following assumes that you have already created a project and an associated table. If you haven't completed those steps yet, more information can be found in the Building Your Data Store section of the Hydrolix documentation website.
Create a Transform
Once you have a project and table created, the last remaining step to getting Fastly data into the Hydrolix platform is to create a transform. Think of a transform as a schema that helps tell the system what type of data to expect and ultimately how to deal with it. A more formal definition can be found here but for now we'll save you some time and provide the full transform for the Fastly extended logs below:
Fastly Transform Click arrow to expand
{
"table": "<<table_uuid>>",
"name": "<string>",
"description": "<string>",
"type": "json",
"settings": {
"is_default": true,
"compression": "none",
"format_details": {
},
"output_columns": [
{
"name": "cache_status",
"datatype": {
"type": "string"
}
},
{
"name": "client_ip",
"datatype": {
"type": "string"
}
},
{
"name": "content_type",
"datatype": {
"type": "string"
}
},
{
"name": "geo_city",
"datatype": {
"type": "string"
}
},
{
"name": "geo_continent_code",
"datatype": {
"type": "string"
}
},
{
"name": "geo_country_code",
"datatype": {
"type": "string"
}
},
{
"name": "geo_datacenter",
"datatype": {
"type": "string"
}
},
{
"name": "geo_region",
"datatype": {
"type": "string"
}
},
{
"name": "h2_stream_id",
"datatype": {
"type": "string"
}
},
{
"name": "host",
"datatype": {
"type": "string"
}
},
{
"name": "is_cacheable",
"datatype": {
"type": "bool"
}
},
{
"name": "is_h2",
"datatype": {
"type": "bool"
}
},
{
"name": "is_h2_push",
"datatype": {
"type": "bool"
}
},
{
"name": "is_ipv6",
"datatype": {
"type": "bool"
}
},
{
"name": "is_tls",
"datatype": {
"type": "bool"
}
},
{
"name": "origin_host",
"datatype": {
"type": "string"
}
},
{
"name": "protocol",
"datatype": {
"type": "string"
}
},
{
"name": "req_body_size",
"datatype": {
"type": "uint32"
}
},
{
"name": "req_header_size",
"datatype": {
"type": "uint32"
}
},
{
"name": "request",
"datatype": {
"type": "string"
}
},
{
"name": "request_accept_charset",
"datatype": {
"type": "string"
}
},
{
"name": "request_accept_content",
"datatype": {
"type": "string"
}
},
{
"name": "request_accept_encoding",
"datatype": {
"type": "string"
}
},
{
"name": "request_accept_language",
"datatype": {
"type": "string"
}
},
{
"name": "request_cache_control",
"datatype": {
"type": "string"
}
},
{
"name": "request_connection",
"datatype": {
"type": "string"
}
},
{
"name": "request_dnt",
"datatype": {
"type": "string"
}
},
{
"name": "request_forwarded",
"datatype": {
"type": "string"
}
},
{
"name": "request_referer",
"datatype": {
"type": "string"
}
},
{
"name": "request_user_agent",
"datatype": {
"type": "string"
}
},
{
"name": "request_via",
"datatype": {
"type": "string"
}
},
{
"name": "request_x_forwarded_for",
"datatype": {
"type": "string"
}
},
{
"name": "request_x_requested_with",
"datatype": {
"type": "string"
}
},
{
"name": "resp_body_size",
"datatype": {
"type": "uint32"
}
},
{
"name": "resp_header_size",
"datatype": {
"type": "uint32"
}
},
{
"name": "response_age",
"datatype": {
"type": "uint32"
}
},
{
"name": "response_cache_control",
"datatype": {
"type": "string"
}
},
{
"name": "response_expires",
"datatype": {
"type": "string"
}
},
{
"name": "response_last_modified",
"datatype": {
"type": "string"
}
},
{
"name": "response_tsv",
"datatype": {
"type": "string"
}
},
{
"name": "service_id",
"datatype": {
"type": "string"
}
},
{
"name": "service_version",
"datatype": {
"type": "string"
}
},
{
"name": "socket_cwnd",
"datatype": {
"type": "uint32"
}
},
{
"name": "socket_nexthop",
"datatype": {
"type": "string"
}
},
{
"name": "socket_ploss",
"datatype": {
"type": "double"
}
},
{
"name": "socket_tcpi_delta_retrans",
"datatype": {
"type": "uint32"
}
},
{
"name": "socket_tcpi_last_data_sent",
"datatype": {
"type": "uint32"
}
},
{
"name": "socket_tcpi_rcv_mss",
"datatype": {
"type": "uint32"
}
},
{
"name": "socket_tcpi_rcv_rtt",
"datatype": {
"type": "uint32"
}
},
{
"name": "socket_tcpi_rcv_space",
"datatype": {
"type": "uint32"
}
},
{
"name": "socket_tcpi_rtt",
"datatype": {
"type": "uint32"
}
},
{
"name": "socket_tcpi_rttvar",
"datatype": {
"type": "uint32"
}
},
{
"name": "socket_tcpi_snd_mss",
"datatype": {
"type": "uint32"
}
},
{
"name": "socket_tcpi_total_retrans",
"datatype": {
"type": "uint32"
}
},
{
"name": "status",
"datatype": {
"type": "string"
}
},
{
"name": "time_elapsed",
"datatype": {
"type": "uint32"
}
},
{
"name": "time_end",
"datatype": {
"type": "datetime",
"format": "2006-01-02T15:04:05"
}
},
{
"name": "time_start",
"datatype": {
"type": "datetime",
"primary": true,
"format": "2006-01-02T15:04:05"
}
},
{
"name": "tls_client_cipher",
"datatype": {
"type": "string"
}
},
{
"name": "tls_client_cipher_sha",
"datatype": {
"type": "string"
}
},
{
"name": "tls_client_protocol",
"datatype": {
"type": "string"
}
},
{
"name": "tls_client_servername",
"datatype": {
"type": "string"
}
},
{
"name": "tls_client_tlsexts_sha",
"datatype": {
"type": "string"
}
},
{
"name": "url",
"datatype": {
"type": "string"
}
}
]
}
}
Leveraging Views
At this point, Hydrolix is configured to accept the incoming Fastly log data.
Hydrolix supports the notion that a single data set can have many different query formats. The query data structure, or view, associated with a given data set not only allows for a customized representation of the queried data but also for a user’s access to the data to be restricted to a set of columns. Upon transform creation, Hydrolix automatically generates a default view that can be used to immediately query the data set - no additional configuration is required. However, users are encouraged to spend some time becoming familiar with the view concept and subsequent benefits that the feature can provide. More detailed information can be found here.
Querying Fastly Logs with Hydrolix
Now that both solutions are configured and Fastly real-time log data is streaming into Hydrolix, it's time to make some queries! We've prepared a short tutorial to help get you started: Analyzing Transaction Logs
Updated 29 days ago