Skip to content

Data Sources

A data source sends data in from outside the cluster, with a token, a table name, and a transform mapping its fields to table columns. CDNs, log forwarders, and cloud services are all data sources. Most post to the streaming ingest API, but some use another method, such as Mux over Kinesis.

Integration failures fall into three cases sharing one symptom, an empty table: the sender never sent, Hydrolix refused the request, or the data didn't match the transform.

An integration's requests fail with an HTTP error⚓︎

Required access: User

Confirm: The status code and response body name the problem: 401: Unauthorized (the Authorization header), 403 with missing or bad auth token (a table access token), 403: Forbidden (the ingest_table permission), 400 with a message such as no table ‘x’ found or unhandled content-type ‘x’ (the table, transform, or format), or 404 with not found (the path). No response at all, such as a timeout, means the sender can't reach the cluster.

The status code shows which part of the request was refused.

Fix: correct the part the status code points to.

  • 401: the Authorization header is missing, malformed, or not valid, such as an expired token. Get a new token. An integration that runs unattended should use a service account token. See Manage Service Accounts: Service account tokens.
  • 403 with missing or bad auth token: the table requires a table access token, in addition to the Authorization header. Send one from its token list in the X-Hdx-Token header or the token query string parameter. See Table Access Tokens.
  • 403 with 403: Forbidden: the cluster checks ingest permissions, and the token's account doesn't have the ingest_table permission for the table. Give the account a role with that permission at the table, project, or organization scope. A token that's already in use keeps its old permissions for up to an hour. To use the new role right away, create a new token. See HTTP Stream API: Ingest authorization.
  • 400: the response body names the problem. See Ingest returns HTTP 400.
  • 404: send requests to https://hostname.hydrolix.live/ingest/event.
  • No response: check the hostname, and that the sender can reach the cluster over HTTPS.

An integration sends data but nothing arrives⚓︎

Required access: User

Confirm: The sender reports no error but the table stays empty. Table health reports an Ingest Latency measured from the epoch, meaning the table has never received data. A recent latency doesn't prove the source is working, since other streams into the same table keep the figure low on their own.

The token, the table name, the transform name, and the sending configuration can each be wrong without the sender reporting a failure. Six integrations share one set of checks; the rest carry their own.

Fix: find your integration under Data Sources and work through its troubleshooting section.

Back to all symptoms