Varnish Enterprise

Varnish Enterprise is the commercial, production-grade version of Varnish Cache, a high-performance HTTP caching and edge delivery engine. Ora Streaming is a fully-managed CDN, owned and operated by Varnish Software AB.

Logs collected from Varnish Enterprise and Ora Streaming contain structured fields such as request timestamps, client IPs, cache status, and response metrics, enabling visibility into performance, reliability, and content delivery for each underlying CDN.

Log delivery instructions for Varnish Enterprise will be provided in the near future by the Varnish Software team. Shipping logs from Varnish Enterprise installations requires admins to enable logging through the varnishncsa service on each host and log delivery using Vector.

For Ora Streaming customers, please contact your representative at Varnish Software to setup log delivery.

Setup steps

There are some prerequisites and three major steps covered in this document. Following these steps will result in an integrated setup between Varnish and a running Hydrolix cluster:

1. Before you Begin: Complete these prerequisites first
2. Configure a Hydrolix Transform: Prepare Hydrolix to receive and transform Varnish logs
3. Configure Varnish log shipping: Information needed by the Varnish Software team to set up log delivery.
4. Verification: Follow these steps to verify the integration is working correctly

Before you begin

You will need a running Hydrolix deployment. Follow the instructions for your preferred deployment method if you have yet to deploy Hydrolix. From your running Hydrolix cluster, you will need the following information:

Gather the following information:

Item	Description	Example value	How to obtain this information
Cluster ingest URL	This is the hostname and ingest endpoint of your Hydrolix cluster.	https://hostname.hydrolix.live/ingest/event	The value of hydrolix_url in your hydrolixcluster.yaml including the preceding https://.
Table name	The destination table in the Hydrolix cluster for Varnish logs. Specified in the format: project_name.table_name.	varnish.varnish_logs	See create a table if you need to create a Hydrolix table in which to store your Varnish logs.
Basic authentication credentials	The username and password to authenticate with the Hydrolix endpoint and grant streaming ingest access to the destination table.	Username: hydrolix Password: $TRAEFIK_PASSWORD	For a more thorough explanation of basic authentication as it pertains to a Hydrolix cluster, see Enable Basic Authentication.

Keep these values on hand for later steps within this guide.

Get started

There are two steps covered within this document.

1. Create a Hydrolix transform
2. Configure Varnish log streaming

Configure a Hydrolix transform

Register a Hydrolix transform. The transform defines a schema to map fields from the Varnish logs onto the Hydrolix table.

Using the Hydrolix transform for Varnish, see Publish Transform for instructions.

You have two options for the publishing the transform:

1. UI: Registering a transform through the UI requires the following information:

   • Project name
   • Table name
   • The contents of the output_columns property

2. API: Registering a transform through the API requires the following information:

   • Project ID
   • Table ID
   • The entirety of the Varnish transform json

If the transform you have created is the only one for your table, it will become the default transform for the table. If you have multiple transforms configured for the destination table, and the transform registered for Varnish isn't the default transform, you should specify the transform for this data as a query parameter in the endpoint URL.

Verify that the Hydrolix transform exists. You can do so using either method:

• UI: In the UI by navigating to Data in the left-hand nav, selecting the table from under the Tables tab, and locating the transform name under Table transforms.
• API: Verify the transform you created is returned as part of the response from the Get transforms endpoint.

Configure streaming logs

For Varnish Enterprise customers, log delivery instructions for Varnish Enterprise will be provided in the near future by the Varnish Software team.

For Ora Streaming customers, please contact your representative at Varnish Software to set up log delivery and have the Hydrolix cluster ingest URL, table name, and basic authentication credentials handy.

Verification

To verify that the integration is working correctly:

1. Wait a few minutes for data to start flowing from Varnish to Hydrolix.
2. In the Hydrolix UI, navigate to the query interface. (https://hostname.hydrolix.live/queries/).

3. Run a simple query against your able. For example:

   SELECT
     timestamp,
     request_host,
     request_method,
     request_full_path,
     response_status_code,
     response_time_to_first_byte_ms
   FROM integrations.varnish
   WHERE timestamp > NOW() - INTERVAL '1 hour'
   LIMIT 10
   

Going further

• For more information on querying your Hydrolix cluster, see ClickHouse SQL.
• Explore your Varnish logs in Grafana dashboards.

Troubleshoot

If you encounter issues with the integration:

Review the ingest logs in Hydrolix for any errors

• You can view and explore cluster logs using the cluster's System Health page.

Check ingest latency in table health

• Navigate to https://hostname.hydrolix.live/data/tables/. Select your project and table. This will take you to a page which displays some table health information.
• Check that the value for Ingest Latency indicates a time since you last sent data from Varnish. Note that if the table is the destination for any other data streams, this number may be low due to data from another stream. If the latency indicates a time since the epoch (January 1st, 1970 at 00:00:00 UTC), then the table has received no data.

Ensure your transform correctly maps the fields you expect to see

• The Hydrolix transform provided acts a schema for storing HTTP requests for Vanish log fields in Hydrolix. If you forward another data set to Hydrolix, see the Hydrolix documentation on transform output columns to modify the transform to correctly map the data you are sending.