Work with Fastly CDN Logs

Hydrolix supports native integration with the Fastly real-time log streaming service via an HTTPS Endpoint. The combination of the two technologies offers a simple, yet powerful means of extending logging and data analytics capabilities without the associated expense.

The following walkthrough covers both the Fastly and Hydrolix configuration settings to enable the complete solution.

Similar documentation can also be found at the Fastly Integrations website:
Fastly Log Streaming Guide for Hydrolix

Setup Fastly Log Streaming: HTTPS

Prerequisites

When sending logs to an HTTPS endpoint, Fastly requires proof that you control the domain name specified in the endpoint URL field. This is accomplished by using an HTTP challenge on a well-known path. Fortunately, Hydrolix has a pre-configured path for quick validation.

Hydrolix pre-configured path for Fastly endpoint URL verification:

https://<hydrolix_instance_name>.hydrolix.live/.well-known/fastly/logging/challenge

By default, Hydrolix uses an asterisk (*) to allow any service to post to the HTTP endpoint.

Configure the HTTPS Logging Endpoint

In this example, we will create a new HTTPS logging endpoint that will be used to send real-time streaming logs to Hydrolix.

1. Login to Fastly

2. Create an HTTPS Logging Endpoint

Under the desired Fastly service configuration, select:

Logging
HTTPS Endpoint

Enter the following configuration options:

Name: <Name for the Hydrolix logging endpoint>

Log format: Select and configure one of the below Fastly log formats. Use the extended Fastly log format to take advantage of the data compression and query performance benefits of the Hydrolix platform.

(recommended) rich 64-field extended Fastly log format
(original) 14-field default log format for a Fastly HTTPS logging endpoint

URL: The streaming API endpoint for your Hydrolix instance:

https://<hydrolix-instance-name>.hydrolix.live/ingest/event

Maximum logs: 0
Maximum bytes: 0

EXPAND "Advanced options"

Content type: application/json
Custom header name: x-hdx-table
Custom header value: <hydrolix_project_name>.<hydrolix_table_name>
Method: POST
JSON log entry format: Newline delimited
Select a log line format: Blank
Placement: Format Version Default

Using your own certificate authority (CA)?
Leave all values empty or customize as required

Congratulations! Fastly has now been configured to send real-time streaming logs to the Hydrolix endpoint. On that note, we had better move on to the next section and ensure that the Hydrolix platform is properly configured to receive them.

Setup Hydrolix Streaming Intake

The remaining configuration steps will take place in your Hydrolix cluster. Please note that the following assumes that you have already created a project and an associated table. If you haven't completed those steps yet, do so using the instructions found in Projects and Tables.

Create a Transform

Once you have a project and table created, the last remaining step to getting Fastly data into the Hydrolix platform is to create a transform. Think of a transform as a schema that helps tell the system what type of data to expect and ultimately how to deal with it. A more formal definition can be found here. The full contents of the transform can be viewed at Fastly transform.

Leveraging Views

At this point, Hydrolix is configured to accept the incoming Fastly log data.

Hydrolix supports the notion that a single data set can have many different query formats. The query data structure, or view, associated with a given data set not only allows for a customized representation of the queried data but also for a user’s access to the data to be restricted to a set of columns. Upon transform creation, Hydrolix automatically generates a default view that can be used to immediately query the data set - no additional configuration is required. However, users are encouraged to spend some time becoming familiar with the view concept and subsequent benefits that the feature can provide. More detailed information can be found here.

Querying Fastly Logs with Hydrolix

Now that both solutions are configured and Fastly real-time log data is streaming into Hydrolix, it's time to make some queries! We've prepared a short tutorial to help get you started: Analyzing Transaction Logs