Using Fastly with Hydrolix

Hydrolix provides a native integration for Fastly log storage and analysis. Fastly makes real-time log streaming available through Log streaming: HTTPS and Hydroilx provides the ability to ingest and query those logs in real-time.

These instructions will lead you through setting up Fastly Log Streaming with Hydrolix. It covers the basic log integration with some live Hydrolix examples.

Setup Fastly Log streaming: HTTPS

Prerequisites

When sending logs to a HTTPS endpoint, Fastly requires proof that you control the domain name specified in the URL field by using a HTTP challenge on a well-known path.

Hydrolix has already pre-configured Fastly well-known path requirements: https://{{your_instance}}.hydrolix.live/.well-known/fastly/logging/challenge

By default, Hydrolix uses an asterisk (*) to allow any service to post to the HTTP endpoint. *

Configuring Fastly HTTPS endpoint

For this example we will use the default log configuration provided by Fastly. When you enable logging in your Fastly account under: https://manage.fastly.com/

Create End-Point

Under your Fastly configuration select:

  • Logging
  • Create your first logging end point

Choose your logging endpoint:

  • Select HTTPS
  • Create End Point

Here are the configurations options:
Name: The name of your endpoint, for example: Hydrolix End-Point.
Log format: We are using the Fastly default for this example

{
  "timestamp":"%{begin:%Y-%m-%dT%H:%M:%S}t",
  "time_elapsed":%{time.elapsed.usec}V,
  "is_tls":%{if(req.is_ssl, "true", "false")}V,
  "client_ip":"%{req.http.Fastly-Client-IP}V",
  "geo_city":"%{client.geo.city}V",
  "geo_country_code":"%{client.geo.country_code}V",
  "request":"%{req.request}V",
  "host":"%{req.http.Fastly-Orig-Host}V",
  "url":"%{json.escape(req.url)}V",
  "request_referer":"%{json.escape(req.http.Referer)}V",
  "request_user_agent":"%{json.escape(req.http.User-Agent)}V",
  "request_accept_language":"%{json.escape(req.http.Accept-Language)}V",
  "request_accept_charset":"%{json.escape(req.http.Accept-Charset)}V",
  "cache_status":"%{regsub(fastly_info.state, "^(HIT-(SYNTH)|(HITPASS|HIT|MISS|PASS|ERROR|PIPE)).*", "\\2\\3") }V"
}

URL Your instance stream URL: https://{{your_instance}}.hydrolix.live/ingest/event
Maximum logs 0
Maximum bytes 0

EXPAND "Advanced options"

Content type application/json
Custom header name x-hdx-table
Custom header value Add your Hydrolix project.table names: hydrolix_project_name.hydrolix_table_name
Method POST
JSON log entry format Newline delimited
Select a log line format Blank
Placement Format Version Default

Using your own certificate authority (CA)?
Leave all values empty

Setup Hydrolix Streaming Intake

You have now configured Fastly to stream their logs to Hydrolix. You can refer to our getting started pages to create a project and a table https://docs.hydrolix.io/quickstart/data-mgmt

Once you have your project and table setup we can focus on setting-up the Fastly Log streaming:

  • Transform
  • View

Fastly Log streaming: Transform

Here is the transform example for Fastly default streaming logs. you need to have this transform setup as default: "is_default": true,

{
	"name": "fastly_transform",
	"description": "fastly https logs",
	"type": "json",
	"table": "{{table_uuid}}",
	"settings": {
		"is_default": true,
		"output_columns": [
			{
				"position": 0,
				"name": "timestamp",
				"type": "datetime",
				"treatment": "primary",
				"format": "2006-01-02T15:04:05"
			},
			{
				"position": 1,
				"name": "time_elapsed",
				"type": "uint64",
				"treatment": "tag"
			},
			{
				"position": 2,
				"name": "is_tls",
				"type": "bool",
				"treatment": "tag"
			},
			{
				"position": 3,
				"name": "client_ip",
				"type": "string",
				"treatment": "tag"
			},
			{
				"position": 4,
				"name": "geo_city",
				"type": "string",
				"treatment": "tag"
			},
			{
				"position": 5,
				"name": "geo_country_code",
				"type": "string",
				"treatment": "tag"
			},
			{
				"position": 6,
				"name": "request",
				"type": "string",
				"treatment": "tag"
			},
			{
				"position": 7,
				"name": "host",
				"type": "string",
				"treatment": "tag"
			},
			{
				"position": 8,
				"name": "url",
				"type": "string",
				"treatment": "tag"
			},
			{
				"position": 9,
				"name": "request_referer",
				"type": "string",
				"treatment": "tag"
			},
			{
				"position": 10,
				"name": "request_user_agent",
				"type": "string",
				"treatment": "tag"
			},
			{
				"position": 11,
				"name": "request_accept_language",
				"type": "string",
				"treatment": "tag"
			},
			{
				"position": 12,
				"name": "request_accept_charset",
				"type": "string",
				"treatment": "tag"
			},
			{
				"position": 13,
				"name": "cache_status",
				"type": "string",
				"treatment": "tag"
			}
		]
	}
}

Fastly Log streaming: View

Now that you have configured the Fastly Endpoint and your Hydrolix transform to ingest the data. You want to be able to query your logs. The following is a view that supports the Fastly default logs example, there is one difference with the transform:

The transform for the colunm "name": "is_tls", "type": "bool" becomes in the view "name": "is_tls", "type": "uint64"

{
	"name": "fastly_logs_view",
	"table": "{{table_uuid}}",
	"settings": {
		"is_default": true,
		"output_columns": [
			{
				"name": "timestamp",
				"type": "datetime",
				"treatment": "primary"
			},
			{
				"name": "time_elapsed",
				"type": "uint64",
				"treatment": "tag",
				"default": 0
			},
			{
				"name": "is_tls",
				"type": "uint64",
				"treatment": "tag"
			},
			{
				"name": "client_ip",
				"type": "string",
				"treatment": "tag"
			},
			{
				"name": "geo_city",
				"type": "string",
				"treatment": "tag"
			},
			{
				"name": "geo_country_code",
				"type": "string",
				"treatment": "tag"
			},
			{
				"name": "request",
				"type": "string",
				"treatment": "tag"
			},
			{
				"name": "host",
				"type": "string",
				"treatment": "tag"
			},
			{
				"name": "url",
				"type": "string",
				"treatment": "tag"
			},
			{
				"name": "request_referer",
				"type": "string",
				"treatment": "tag"
			},
			{
				"name": "request_user_agent",
				"type": "string",
				"treatment": "tag"
			},
			{
				"name": "request_accept_language",
				"type": "string",
				"treatment": "tag"
			},
			{
				"name": "request_accept_charset",
				"type": "string",
				"treatment": "tag"
			},
			{
				"name": "cache_status",
				"type": "string",
				"treatment": "tag"
			}
		]
	}
}

Querying your logs with Hydrolix

Now that you have configured the Fastly Endpoint and Hydrolix. You can query your logs. We have event produced a short tutorial here