Cloudflare
Cloudflare is a reverse proxy that provides CDN, DDoS protection, security, and DNS services for websites. Cloudflare Logpush can send its HTTP request logs to a Hydrolix cluster for fast queries and low-cost long-term storage.
Setup steps⚓︎
Connect Cloudflare to a running Hydrolix cluster in these steps:
- Prerequisites: Gather the values the later steps need
- Configure a Hydrolix transform: Prepare Hydrolix to receive and transform Cloudflare log data
- Configure Cloudflare Logpush: Set up a Logpush job in Cloudflare to send HTTP logs to Hydrolix
- Verification: Confirm that data arrives
Prerequisites⚓︎
The integration uses the Cloudflare Logpush service to send Cloudflare HTTP request log data directly to the Hydrolix HTTP Streaming API:
This requires a Cloudflare account and a running Hydrolix cluster. To deploy one, see Deploy on Google GKE. Gather these values from the cluster:
| Item | Description | Example value | How to get it |
|---|---|---|---|
| Org ID | The Hydrolix organization that owns the table. | bc3f041b-ea52-45e1-b9d5-41819c465854 |
List orgs with the Config API. |
| Project name and ID | The project that contains the table. | Name: project_name ID: c2445da3-ec63-42be-9f12-f104f9656f4c |
See Create a project with the API. |
| Table name and ID | The table that stores Cloudflare data. | Name: table_name ID: 798dfcf4-7560-4b24-1337-29017619baa6 |
See Create a table using the API. |
| Authorization token | An OAuth bearer token for the HTTP Streaming API. | eyXrxkzoN2fRiiKpnV... |
See Acquire an authorization token. |
Keep these values for the later steps.
Get started⚓︎
The integration has two parts:
| Component | Description |
|---|---|
| Cloudflare Logpush | Collects Cloudflare logs and sends them to the Hydrolix cluster |
| Hydrolix transform | Maps Cloudflare log fields to Hydrolix table columns |
Configure a Hydrolix transform⚓︎
Publish the Hydrolix transform for Cloudflare to the table. See Publish a Transform.
Registering the transform in the UI requires:
- Project name
- Table name
- The
output_columnsproperty from the transform
Registering it with the API requires:
- Project ID
- Table ID
- The whole transform JSON
Configure Cloudflare Logpush⚓︎
- From Account Home, select the domain in the Domains list.
- Select Analytics & Logs, then Logpush.
- Select Create a Logpush Job.
- Select HTTP destination.
-
Enter this URL, replacing
hostname.hydrolix.livewith the cluster's hostname and each{}field:Logpush HTTP Destination format:application/jsonortext/csvproject.table: for example,project_name.table_namebearer-token: the token from Prerequisites
-
Select HTTP requests.
- Enter a Job name. Underscores aren't allowed.
- Under Send the following fields..., select Select All.
- Select Submit.
Verification⚓︎
- Wait a few minutes for data to arrive. Logpush sends Cloudflare HTTP request log data, so no data arrives until Cloudflare generates some.
- In the Hydrolix UI, open the query interface at
https://hostname.hydrolix.live/queries/. -
Query the Cloudflare table. For example:
-
Check that the results match the requests in Cloudflare.
Next steps⚓︎
- Query the data with SQL. See ClickHouse Considerations.
- Explore the logs in Grafana. See Grafana Automatic Installation.
- Set up alerts on Cloudflare events. See Configure Alerts in Grafana.
Troubleshoot⚓︎
Check these when data doesn't arrive in Hydrolix.
Check that the Logpush job is enabled⚓︎
Get the job's ID, then query the Logpush jobs endpoint with curl. The response should contain "enabled": true. If it contains "enabled": false, enable the job.
Check the Logpush job for errors⚓︎
Logpush alerts and analytics show job errors and health.
Verify your Hydrolix bearer token is valid and hasn't expired⚓︎
- If you receive a
401 Unauthorizedresponse, the authorization token may have expired. Retrieve a new one using the Hydrolix login API endpoint.
Review the ingest logs in Hydrolix for errors⚓︎
- View and explore cluster logs on the cluster's System Health page.
- You can also query cluster logs directly with Hydrologs.
Check ingest latency in table health⚓︎
Table health reports Ingest Latency for each table. A latency measured from the epoch (January 1st, 1970 at 00:00:00 UTC) means the table has never received data.
A recent latency doesn't prove your source is working. If the table also receives other streams, their data keeps the figure low on its own.
Check that the transform matches the dataset⚓︎
The transform on this page maps the Cloudflare HTTP requests dataset. To send another dataset, change the transform's output columns to match its fields.
Related information⚓︎
- Cloudflare and Hydrolix: Real-time Traffic Insights - How to use Hydrolix to analyze Cloudflare CDN logs for real-time traffic visibility