Cloudflare Integration

Overview

Cloudflare is a cloud-based platform that provides CDN services, DDoS protection, internet security, and distributed DNS services to improve the performance and security of web applications. It acts as a reverse proxy between visitors and a website’s server, helping to mitigate attacks, reduce latency, and optimize content delivery.

Use Hydrolix with Cloudflare to route your HTTP logs directly to a Hydrolix cluster for fast queries and cost-effective long-term storage.

Setup Steps

There are some prerequisites and three major steps covered within this document. Following these steps will result in an integrated setup between Cloudflare and a running Hydrolix cluster:

1. Before you Begin: Complete these prerequisites first
2. Configure a Hydrolix Transform: Prepare Hydrolix to receive and transform Cloudflare log data
3. Configure Cloudflare Logpush: Set up a Logpush job in Cloudflare to send HTTP logs to Hydrolix
4. Verification: Follow these steps to verify the integration is working correctly

Before You Begin

The integration uses the Cloudflare Logpush service to send Cloudflare HTTP request log data directly to the Hydrolix HTTP Streaming API. The data flow is as follows:

Cloudflare Services → Cloudflare Logpush → Hydrolix HTTP Streaming API

You will need a Cloudflare account and a running Hydrolix deployment. Follow the instructions for your preferred deployment method if you have yet to deploy Hydrolix. From your running Hydrolix cluster, you will need the following information:

Keep these values on hand for later steps within this guide.

Getting Started

There are two required components to successfully integrate Cloudflare with a running Hydrolix cluster:

Configure a Hydrolix Transform

Register a Hydrolix transform. The transform defines a schema to map fields from the Cloudflare logs onto the Hydrolix table.

Using the Hydrolix transform for Cloudflare, see Publishing Your Transform to publish the transform.

You have the option of registering a transform through the UI which requires:

• Project name
• Table name
• The contents of the output_columns property from the transform

Alternatively, you can use the API which requires:

• Project ID
• Table ID
• The entirety of the Cloudflare transform json.

Configure Cloudflare Logpush

Follow these steps to configure Cloudflare Logpush to send data to your Hydrolix cluster:

1. From Account Home select the relevant domain in the Domains list.
2. Select Analytics & Logs to show options.
3. Select Logpush from the options.
4. Select the blue button on the right labeled Create a Logpush Job.
5. Select HTTP destination from the listed options.
6. Enter the following URL, replacing the {} fields:

https://{myhost}.hydrolix.live/ingest/event?header_content-type={format}&header_x-hdx-table={project.table}&header_Authorization=%20Bearer%20{bearer-token}

Examples for each field:

• myhost: my_cluster
• format: application/json or text/csv
• project.table: project_name.table_name
• bearer token: Your OAuth token from Before you Begin

7. Select HTTP requests from the listed options.
8. Fill in the Job name field (Note: Underscores aren't allowed).
9. Select the Select All option under Send the following fields....
10. Select Submit at the bottom of the page.

Verification

To verify that the integration is working correctly:

1. Wait a few minutes for data to start flowing from Cloudflare to Hydrolix. Recall that the Logpush service is sending Cloudflare HTTP request log data Hydrolix. Verify that Cloudflare is generating this data type if you don't see data in Hydrolix.
2. In the Hydrolix UI, navigate to the query interface. (https://{myhost}.hydrolix.live/queries/)
3. Run a simple query against your Cloudflare table. For example:

SELECT
  timestamp,
  request_host,
  request_method,
  request_full_path,
  response_status_code,
  response_time_to_first_byte_ms
FROM integrations.cloudflare
WHERE timestamp > NOW() - INTERVAL '1 hour'
LIMIT 10

4. Verify that you see results similar to your data in Cloudflare.

Going Further

• For more information on querying your Hydrolix cluster, see ClickHouse SQL.
• Explore your Cloudflare logs in Grafana dashboards
• Set up alerting based on Cloudflare events

Troubleshooting

If you encounter issues with the integration:

Verify your Cloudflare Logpush Job is enabled

• Retrieve the ID of the job you created. Use cURL to the jobs endpoint to obtain job status. Verify the result contains "enabled": true. If it contains "enabled": false, follow the instructions to enable the job.

Check the Logpush job status in Cloudflare for any errors

• You can use Logpush alerts and analytics to look for errors or to monitor job health.

Verify your Hydrolix bearer token is valid and hasn't expired

• If you receive a 401 Unauthorized response then it's possible the bearer token expired. You can retrieve a new bearer token using the Hydrolix login API endpoint.

Review the ingest logs in Hydrolix for any errors

• You can view and explore cluster logs using the cluster's System Health page.

Check ingest latency in table health

• Navigate to https://{myhost}.hydrolix.live/data/tables/. Select your project and table. This will take you to a page which displays some table health information.
• Check that the value for Ingest Latency indicates a time since you last sent data from Cloudflare. Note that if the table is the destination for any other data streams, this number may be low due to data from another stream. If the latency indicates a time since the epoch (January 1st, 1970 at 00:00:00 UTC), then the table has received no data.

Ensure your transform correctly maps the fields you expect to see

• The Hydrolix transform provided acts a schema for storing HTTP requests Cloudflare log fields in Hydrolix. If you forward another data set to Hydrolix, see the Hydrolix documentation on transform output columns to modify the transform to correctly map the data you are sending.