Table Access Tokens

Table access tokens limit which applications are permitted to stream data into a table. The feature is off by default.

Network-layer controls and stream authorization checks are enforced before the HTTP Stream API checks this optional application-layer access control.

Don't confuse table access tokens with OAuth bearer tokens

See authorization tokens issued to user and service accounts.

Table settings for access tokens⚓︎

The stream field of a table's table settings contains the configurable fields:

Field	Default	Description
`token_auth_enabled`	`false`	Boolean. If true, enforce token-based access control for the table.
`token_list`	`[]`	An array of strings. These are valid tokens.

When token_auth_enabled is true, the ingestion system checks requests for a valid token.

A token is any arbitrary string. The token list is an array to allow for easy token rotations and concurrent use of multiple tokens supporting different client applications.

Streaming requests with a token matching any in the token_list are allowed. For all other requests, the ingestion API returns an HTTP 403.

Token list values

A token_list can contain any number of tokens. Any Unicode string is a valid token and there are no restrictions on a token's length.

An empty token_list ([]) or null is also permitted. An empty token list is equivalent to token_auth_enabled: false.

Require table access tokens⚓︎

Use these steps to require table access tokens for a table.

Create one or more tokens using any tool, mechanism, or naming pattern. These are arbitrary strings.
Reconfigure any active, existing streams to send table access tokens with streaming requests.
Set token_list to the array of tokens.
Set token_auth_enabled to true in the stream settings of the table to enforce checking.
Verify by sending HTTP requests with valid tokens. Expect receipt of HTTP 200 responses.
Verify by sending an HTTP request with an invalid token. Expect receipt of an HTTP 403 response.
Confirm any existing applications continue to stream data successfully.

Table access token changes apply immediately

Requiring table access tokens can cause data loss for active streams not already configured to present tokens.

Configure the sending client applications before requiring table access tokens in the Hydrolix ingestion system.

Example table settings fragment⚓︎

Table settings fragment with feature enabled and two table access tokens

"settings": {
  ...
  "stream": {
    "token_auth_enabled": true,
    "token_list": ["token1", "token2"]
  }
  ...
}

Ingestion metrics⚓︎

Valid requests using any acceptable token are counted in metric http_source_request_count emitted by the intake-head application.

Requests failing to present an acceptable token are counted in metric http_source_request_error_count{status_code="403"}. This shows rejected requests.

Send table access tokens with streaming requests⚓︎

When a table requires access tokens, every streaming request must include a valid token. Clients can send the token in one of two ways:

Method	How to send it	Notes
HTTP header (recommended)	`X-Hdx-Token: token1`	Keeps the token out of request URLs and access logs.
Query parameter	Append `token` to the ingest URL: `https://hostname.hydrolix.live/ingest/event?token=token1`	URL-encode the token, or generate tokens without special characters.

Protect tokens in transit

Connect to the HTTP Stream API over TLS, and prefer the X-Hdx-Token header over the query parameter, so tokens can't be sniffed.