Skip to content

Cloudflare

Cloudflare is a reverse proxy that provides CDN, DDoS protection, security, and DNS services for websites. Cloudflare Logpush can send its HTTP request logs to a Hydrolix cluster for fast queries and low-cost long-term storage.

Setup steps⚓︎

Connect Cloudflare to a running Hydrolix cluster in these steps:

  1. Prerequisites: Gather the values the later steps need
  2. Configure a Hydrolix transform: Prepare Hydrolix to receive and transform Cloudflare log data
  3. Configure Cloudflare Logpush: Set up a Logpush job in Cloudflare to send HTTP logs to Hydrolix
  4. Verification: Confirm that data arrives

Prerequisites⚓︎

The integration uses the Cloudflare Logpush service to send Cloudflare HTTP request log data directly to the Hydrolix HTTP Streaming API:

Cloudflare Services → Cloudflare Logpush → Hydrolix HTTP Streaming API

This requires a Cloudflare account and a running Hydrolix cluster. To deploy one, see Deploy on Google GKE. Gather these values from the cluster:

Item Description Example value How to get it
Org ID The Hydrolix organization that owns the table. bc3f041b-ea52-45e1-b9d5-41819c465854 List orgs with the Config API.
Project name and ID The project that contains the table. Name: project_name
ID: c2445da3-ec63-42be-9f12-f104f9656f4c
See Create a project with the API.
Table name and ID The table that stores Cloudflare data. Name: table_name
ID: 798dfcf4-7560-4b24-1337-29017619baa6
See Create a table using the API.
Authorization token An OAuth bearer token for the HTTP Streaming API. eyXrxkzoN2fRiiKpnV... See Acquire an authorization token.

Keep these values for the later steps.

Get started⚓︎

The integration has two parts:

Component Description
Cloudflare Logpush Collects Cloudflare logs and sends them to the Hydrolix cluster
Hydrolix transform Maps Cloudflare log fields to Hydrolix table columns

Configure a Hydrolix transform⚓︎

Publish the Hydrolix transform for Cloudflare to the table. See Publish a Transform.

Registering the transform in the UI requires:

  • Project name
  • Table name
  • The output_columns property from the transform

Registering it with the API requires:

  • Project ID
  • Table ID
  • The whole transform JSON

Configure Cloudflare Logpush⚓︎

  1. From Account Home, select the domain in the Domains list.
  2. Select Analytics & Logs, then Logpush.
  3. Select Create a Logpush Job.
  4. Select HTTP destination.
  5. Enter this URL, replacing hostname.hydrolix.live with the cluster's hostname and each {} field:

    Logpush HTTP Destination
    https://hostname.hydrolix.live/ingest/event?header_content-type={format}&header_x-hdx-table={project.table}&header_Authorization=%20Bearer%20{bearer-token}
    
    • format: application/json or text/csv
    • project.table: for example, project_name.table_name
    • bearer-token: the token from Prerequisites
  6. Select HTTP requests.

  7. Enter a Job name. Underscores aren't allowed.
  8. Under Send the following fields..., select Select All.
  9. Select Submit.

Verification⚓︎

  1. Wait a few minutes for data to arrive. Logpush sends Cloudflare HTTP request log data, so no data arrives until Cloudflare generates some.
  2. In the Hydrolix UI, open the query interface at https://hostname.hydrolix.live/queries/.
  3. Query the Cloudflare table. For example:

    Query Recent Cloudflare Requests
    SELECT
      timestamp,
      request_host,
      request_method,
      request_full_path,
      response_status_code,
      response_time_to_first_byte_ms
    FROM integrations.cloudflare
    WHERE timestamp > NOW() - INTERVAL '1 hour'
    LIMIT 10
    
  4. Check that the results match the requests in Cloudflare.

Next steps⚓︎

Troubleshoot⚓︎

Check these when data doesn't arrive in Hydrolix.

Check that the Logpush job is enabled⚓︎

Get the job's ID, then query the Logpush jobs endpoint with curl. The response should contain "enabled": true. If it contains "enabled": false, enable the job.

Check the Logpush job for errors⚓︎

Logpush alerts and analytics show job errors and health.

Verify your Hydrolix bearer token is valid and hasn't expired⚓︎

  • If you receive a 401 Unauthorized response, the authorization token may have expired. Retrieve a new one using the Hydrolix login API endpoint.

Review the ingest logs in Hydrolix for errors⚓︎

  • View and explore cluster logs on the cluster's System Health page.
  • You can also query cluster logs directly with Hydrologs.

Check ingest latency in table health⚓︎

Table health reports Ingest Latency for each table. A latency measured from the epoch (January 1st, 1970 at 00:00:00 UTC) means the table has never received data.

A recent latency doesn't prove your source is working. If the table also receives other streams, their data keeps the figure low on its own.

Check that the transform matches the dataset⚓︎

The transform on this page maps the Cloudflare HTTP requests dataset. To send another dataset, change the transform's output columns to match its fields.