CacheFly

CacheFly is a real-time content delivery network (CDN) that accelerates serving web content and services to end users. CacheFly can caches and deliver content directly to users rather than fetching it from the origin for every visitor.

CacheFly supports pushing logs to an HTTP endpoint, which Hydrolix supports. These logs contain structured fields such as request timestamps, client IPs, cache status, and response metrics, enabling visibility into CDN performance, reliability, and content delivery.

Follow the instructions on this page to stream CacheFly logs into Hydrolix.

Setup steps

There are some prerequisites and three major steps covered in this document. Following these steps will result in an integrated setup between CacheFly and a running Hydrolix cluster:

1. Before you Begin: Complete these prerequisites first
2. Configure a Hydrolix Transform: Prepare Hydrolix to receive and transform CacheFly log data
3. Configure CacheFly log shipping: Add Hydrolix as a log target in CacheFly to send HTTP logs to Hydrolix
4. Verification: Follow these steps to verify the integration is working correctly

Before you begin

You will need a CacheFly account and a running Hydrolix deployment. Follow the instructions for your preferred deployment method if you have yet to deploy Hydrolix. From your running Hydrolix cluster, you will need the following information:

Gather the following information:

Item	Description	Example value	How to obtain this information
Cluster ingest URL	This is the hostname and ingest endpoint of your Hydrolix cluster.	https://hostname.hydrolix.live/ingest/event	The value of hydrolix_url in your hydrolixcluster.yaml including the preceding https://.
Table name	The destination table in the Hydrolix cluster for CacheFly CDN logs. Specified in the format: project_name.table_name.	cachefly_project.cachefly_cdn_logs	See create a table if you need to create a Hydrolix table in which to store your CacheFly CDN logs.
Basic authentication credentials	The username and password to authenticate with the Hydrolix endpoint and grant streaming ingest access to the destination table.	Username: hydrolix Password: $TRAEFIK_PASSWORD	For a more thorough explanation of basic authentication as it pertains to a Hydrolix cluster, see Enable Basic Authentication.

Keep these values on hand for later steps within this guide.

Get started

There are two steps covered within this document. Following these steps will result in an integrated setup between CacheFly and a running Hydrolix cluster:

1. Create a Hydrolix transform
2. Configure CacheFly log shipping

Configure a Hydrolix transform

Register a Hydrolix transform. The transform defines a schema to map fields from the CacheFly logs onto the Hydrolix table.

Using the Hydrolix transform for CacheFly, see Publish Transform for instructions.

You have two options for the publishing the transform:

1. UI: Registering a transform through the UI requires the following information:

   • Project name
   • Table name
   • The contents of the output_columns property

2. API: Registering a transform through the API requires the following information:

   • Project ID
   • Table ID
   • The entirety of the CacheFly transform json

If the transform you have created is the only one for your table, it will become the default transform for the table. If you have multiple transforms configured for the destination table, and the transform registered for CacheFly CDN data isn't the default transform, you should specify the transform for this data as a query parameter in the endpoint URL.

Verify that the Hydrolix transform exists. You can do so using either method:

• UI: In the UI by navigating to Data in the left-hand nav, selecting the table from under the Tables tab, and locating the transform name under Table transforms.
• API: Verify the transform you created is returned as part of the response from the Get transforms endpoint.

Keep the transform name on hand for the next step.

Configure CacheFly log shipping

Follow these steps to configure CacheFly log shipping to send data to your Hydrolix cluster:

1. From the CacheFly portal, select the Configuration menu and then select Log Targets.

2. Select the New Target button, which opens the following form.

   

3. Fill out the form with these values:

4. Type: HTTP Server

5. Name: Hydrolix (or choose your own)

6. URL: Enter the following URL, replacing the {} fields:

   https://{hostname}.hydrolix.live/ingest/event?&transform={transform_name}
   

When to use the transform query parameter

If the CacheFly transform is the default transform for the Hydrolix table, you can omit the transform query parameter (?&transform={transform_name}).

• Method: Post
• Auth Type: Basic
• Username: your username, to authenticate with the Hydrolix endpoint
• Password: your password, to authenticate with the Hydrolix endpoint
• Format: json
• Compression: gzip

Verification

To verify that the integration is working correctly:

1. Wait a few minutes for data to start flowing from CacheFly to Hydrolix.
2. In the Hydrolix UI, navigate to the query interface. (https://hostname.hydrolix.live/queries/).

3. Run a simple query against your able. For example:

   SELECT
   timestamp,
     request_host,
     request_method,
     request_full_path,
     response_status_code,
     response_time_to_first_byte_ms
   FROM integrations.cachefly
   WHERE timestamp > NOW() - INTERVAL '1 hour'
   LIMIT 10
   

4. Verify that you see results similar to your data in CacheFly.

Going further

• For more information on querying your Hydrolix cluster, see ClickHouse SQL.
• Explore your CacheFly logs in Grafana dashboards

Troubleshoot

If you encounter issues with the integration:

Review the ingest logs in Hydrolix for any errors

• You can view and explore cluster logs using the cluster's System Health page.

Check ingest latency in table health

• Navigate to https://hostname.hydrolix.live/data/tables/. Select your project and table. This will take you to a page which displays some table health information.
• Check that the value for Ingest Latency indicates a time since you last sent data from CacheFly. Note that if the table is the destination for any other data streams, this number may be low due to data from another stream. If the latency indicates a time since the epoch (January 1st, 1970 at 00:00:00 UTC), then the table has received no data.

Ensure your transform correctly maps the fields you expect to see

• The Hydrolix transform provided acts a schema for storing HTTP requests from CacheFly log fields in Hydrolix. If you forward another data set to Hydrolix, see the Hydrolix documentation on transform output columns to modify the transform to correctly map the data you are sending.