Skip to content

Ingest Integration Guide

Overview⚓︎

This guide is for engineers building or configuring clients that send data to Hydrolix through HTTP streaming ingest. It highlights important endpoint features, authentication requirements, and operational guidance needed to build a reliable, production-quality integration.

Hydrolix streaming ingest accepts data through an HTTP POST endpoint secured with JWT bearer-token authentication. A robust integration involves:

  • token lifecycle management
  • retry logic with backpressure propagation
  • correct header configuration
  • timestamp normalization

This guide covers each of these areas along with deployment considerations like intake pool routing and IP filtering.

Hydrolix ingest API specification⚓︎

A robust Hydrolix data source integration requires a thorough understanding of the HTTP streaming ingest API. Send all streaming ingest through the default HTTP POST endpoint with authentication and routing controlled by headers.

Endpoint⚓︎

Property Example
URL https://hostname.hydrolix.live/ingest/event
Method POST
Port 443 (HTTPS, default)
Content-Type application/json
Compression gzip (strongly recommended; see Compression)

Required headers⚓︎

Header Description Required
Authorization Bearer token (JWT). See Authentication for lifecycle. Yes
x-hdx-table Target table in format project.table. Can also be specified using the table query parameter. Conditional
x-hdx-transform Transform name for schema mapping. Omit to use table default. Can also be specified using the transform query parameter. Conditional
Content-Type application/json Yes
Content-Encoding gzip if payload is compressed. If compressed

Firewall advice⚓︎

Integrating systems must allow x-hdx-* and Authorization headers through any intermediate proxies or WAFs. If x-hdx- headers are stripped, Hydrolix will respond with a 4xx.

HTTP response codes⚓︎

Code Meaning Integrator action
200 All messages accepted Continue normally.
429 Rate limited: Hydrolix is under backpressure MUST retry with exponential backoff for up to 5 minutes. Don't drop the batch. Respect Retry-After header if present.
207 Partial success: some messages rejected Indicates transform mismatch or schema error. Inspect response body for per-message status. This is almost always a misconfigured transform.
400 Bad request: malformed payload or missing headers Don't retry, but fix the request. Check x-hdx-* headers and payload format.
401 / 403 Auth failure: token expired or invalid Re-authenticate and obtain a fresh bearer token. See Authentication.
5xx Server-side error Retry with backoff. Escalate if persistent.

Retry strategy (429 handling)⚓︎

When Hydrolix returns 429, the ingest pipeline is under load. The correct behavior is:

  1. Buffer the failed batch locally (don't drop it)
  2. Wait for the duration specified in the Retry-After header, or default to 1 second
  3. Retry with exponential backoff: 1s, 2s, 4s, 8s, 16s,...
  4. Continue retrying for at least 5 minutes before considering the batch failed
  5. If retries are exhausted, write to a dead-letter queue or local spill file. Never silently drop data.

Authentication⚓︎

For full details, see Authentication and authorization.

Property Detail
Auth method Service account token or bearer token, using Authorization: Bearer <token> HTTP header
Token source Using the UI or one of the /config/v1/service_accounts/{uuid}/tokens/ or /config/v1/login API endpoints
Default TTL 365 days for a service account token, 24 hours for a bearer token.
TTL discovery JWT-decode the token to read the exp claim for exact expiry
Refresh strategy Re-authenticate before expiry (for example, at 80% of TTL). Don't wait for a 401.

Token lifecycle note: Integrators should JWT-decode the auth token on receipt to extract the exp (expiry) claim. This gives the exact TTL. Implement proactive token refresh well before expiry. Waiting until Hydrolix returns a 401 response may require request retries and delay data.

Example: Get a token⚓︎

A Hydrolix administrator generates a service account token and provides it to the integrator.

Generate a Service Account Token (Admin Step)
1
2
3
4
5
curl -s -X POST "https://${HDX_HOSTNAME}/config/v1/service_accounts/${HDX_SA_UUID}/tokens/" \
  -H "Authorization: Bearer ${HDX_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"expiry": "2027-03-31T00:00:00Z"}' \
  | jq -r '.token'

The response contains the token. Use it directly as Authorization: Bearer <token> in ingest requests. Default TTL is 365 days if expiry is omitted.

If you're using a user account instead of a service account, POST credentials to the login endpoint.

Acquire a Bearer Token
1
2
3
4
curl -s -X POST "https://${HDX_HOSTNAME}/config/v1/login" \
  -H "Content-Type: application/json" \
  -d '{"username":"'${HDX_USER}'","password":"'${HDX_PASSWORD}'"}' \
  | jq -r '.auth_token.access_token'

This token expires after 24 hours. For long-running integrations, service account tokens are strongly preferred.

Either token type is a JWT that can be decoded to read the exp field using any standard JWT library.

Payload best practices⚓︎

Payload size⚓︎

Target ~1 MB uncompressed per HTTP request. Performance degrades noticeably as payloads get significantly larger: intake processing time increases and memory pressure on intake heads rises. Staying around the 1 MB range gives the best throughput-per-request without hitting diminishing returns.

Parameter Recommendation
Target payload size ~1 MB uncompressed
Batching strategy Batch by time (for example, flush every 5-10s) AND by size (flush at ~1 MB), whichever comes first

Content-Type⚓︎

JSON (application/json) is the recommended content type for streaming ingest. CSV/TSV are also supported.

Compression⚓︎

gzip is strongly recommended. Always send payloads with Content-Encoding: gzip. Log data often compresses to 10-20% of its original size, which dramatically reduces network transfer time and improves ingest throughput. While not strictly required, gzip should be considered a default for any production integration.

Sort data by time⚓︎

Pre-sorting events by timestamp within each batch is a best practice that improves Hydrolix intake performance. The Hydrolix storage engine is optimized for time-ordered data. Sorted batches compress better and merge more efficiently downstream. This isn't a hard requirement, but it's a low-effort optimization that pays off at scale.

Timestamp handling⚓︎

Timestamp handling is the #1 source of silent data quality issues

Every log source has its own timestamp convention. Not normalizing, or incorrectly normalizing, timestamp data from disparate sources doesn't cause ingest failures; it causes wrong query results, which is worse.

Primary timestamp requirement⚓︎

Hydrolix requires a single, consistent primary timestamp for every record in a given table. The transform defines which field is the primary timestamp and what format it's in. All records in a batch must use the same timestamp field and format.

If a record has no timestamp, Hydrolix will automatically use the receive time (the time the record was ingested) as the primary timestamp. You don't need to handle missing timestamps on the integrator side. Hydrolix takes care of it.

Multi-source timestamp divergence⚓︎

When a pipeline aggregates logs from multiple sources into a single Hydrolix table, timestamp formats and field names diverge. For example:

Source Timestamp field Format example
Google Cloud Logging timestamp 2026-03-11T14:30:00.000Z (RFC 3339)
AWS CloudWatch @timestamp 1710168600000 (epoch ms)
Syslog (RFC 5424) header timestamp Mar 11 14:30:00 (no year, no TZ)
Kubernetes time 2026-03-11T14:30:00.123456789Z (nano precision)
Custom apps Varies Anything from epoch seconds to locale-formatted strings

What integrators must do⚓︎

  1. Normalize to a single field name, for example @timestamp or timestamp, before shipping to Hydrolix
  2. Normalize to a single format. ISO 8601 / RFC 3339 with timezone (2026-03-11T14:30:00.000Z) is the safest default.
  3. Preserve the original timestamp in a secondary field (for example, _original_timestamp) for debugging

Google Cloud note: Google Cloud Logging emits different timestamp formats depending on the originating service: Compute Engine, GKE, Cloud Functions, and others may all use slightly different field names and precisions. Integrators must normalize these before they hit Hydrolix, not rely on the Hydrolix transform to handle every variant.

HTTP timeout configuration⚓︎

Setting Value Rationale
Hydrolix default timeout 30 seconds Server-side timeout for ingest HTTP requests
Client timeout 29 seconds Cancel client-side before server-side timeout to maintain control of retry logic
On timeout Retry the batch Treat as transient failure: the batch may or may not have been accepted

Why 29s, not 30s? If the client timeout equals the server timeout, there's a race condition: the server may have accepted the batch but the client already gave up. By setting the client timeout one second shorter, the client always cancels first, retains control, and can retry cleanly.

Backpressure propagation⚓︎

This is required for any production integration. When Hydrolix returns 429, the integrating system should retry with exponential backoff and pause or slow upstream collection to prevent a cascading failure.

Failure without backpressure⚓︎

Without backpressure propagation, a 429 from Hydrolix triggers a dangerous sequence:

  1. Hydrolix returns 429: intake is overloaded
  2. The integration buffers the failed batch and retries
  3. Meanwhile, new data keeps arriving from upstream sources at full speed
  4. The integration's internal buffer grows unchecked
  5. Memory usage climbs until the collector/agent OOMs and crashes
  6. On restart, there's now a data gap and a burst of buffered data to re-process, making the original backpressure worse

This is the most common failure mode seen in production integrations that handle retries correctly, but don't propogate backpressure.

How integrators should handle backpressure⚓︎

  • On HTTP 429: Signal upstream sources to slow down or pause. The specifics depend on the pipeline architecture: this might mean pausing file reads, reducing poll frequency, sending TCP backpressure to syslog sources, or throttling API collection intervals.
  • Set buffer limits: Never buffer unboundedly. Configure a maximum buffer size for both memory and disk. When the buffer fills, the integration should either drop the oldest data (with logging/alerting) or refuse new data from upstream. Both are better than running out of memory.
  • On recovery: When Hydrolix starts accepting data again, returning HTTP 200 responses, gradually ramp collection back up rather than releasing the full buffer at once. A burst of buffered data can immediately re-trigger the HTTP 429 cycle.
  • Alert on sustained backpressure: If the integration is throttling upstream for more than a few minutes, something is wrong at a higher level. Alert on this condition so operators can investigate. This could be due to a Hydrolix scaling issue, a misconfigured intake pool, an unexpected data volume spike, or other conditions.

Retries and backpressure work together⚓︎

Retry logic without backpressure propagation is incomplete. They're two halves of the same mechanism: retries handle the already-in-flight data, backpressure handles the not-yet-sent data. You need both.

Intake pools: workload isolation⚓︎

Hydrolix supports intake pools, which allow you to route ingest traffic to a dedicated, isolated set of intake infrastructure. This is a powerful mechanism for multi-tenant deployments and workload isolation.

Traffic isolation⚓︎

Each intake pool is a separate set of intake heads with its own scaling, resource allocation, and backpressure characteristics. During normal operation, traffic routed to one intake pool is isolated from another. Under sustained backpressure, however, intake spill may redistribute work across pools.

Use cases⚓︎

  • Per-tenant isolation: Each customer or tenant gets their own intake pool. This guarantees that one tenant's ingest spike doesn't degrade another tenant's ingest latency or cause HTTP 429s for others.
  • Priority tiers: Route high-priority, low-latency data (like security alerts) to a dedicated intake pool, while bulk/batch data goes through a shared pool.
  • Blast radius containment: If a misconfigured integration sends malformed data at high volume, it only impacts its own intake pool.

How to route⚓︎

The target intake pool is specified at the infrastructure/routing level (not through HTTP headers). Work with Hydrolix engineering to configure intake pool routing for your deployment. The integrating system needs to point at the correct hostname or endpoint for the desired intake pool.

Talk to us: If intake pool isolation is relevant to your deployment, let us know. We'll work with you to configure the right topology.

Deduplication⚓︎

Hydrolix doesn't perform deduplication. There is no ingest-time or storage-time deduplication built into the platform. If the same record is ingested twice (for example, due to a retry after a timeout), both copies will be stored and both will appear in query results.

For most log analytics use cases, occasional duplicates are acceptable and don't materially impact analysis. If your use case requires strict deduplication:

  • The integrating system is responsible for deduplication logic (for example, idempotency keys, exactly-once delivery guarantees on the sender side).
  • If you can attach a unique batch ID or message ID to records, it enables post-hoc deduplication using query-time filtering.
  • Discuss with Hydrolix engineering if you need guidance on deduplication strategies for your specific use case.

IP filtering⚓︎

Hydrolix supports IP allowlisting, but it's scoped to the cluster level, not to an ingest endpoint. Enabling an IP allowlist restricts access to the entire cluster: ingest, query, config API, UI, everything.

Implications for integrators⚓︎

  • If the integrating system runs on dynamic IPs (autoscaling groups, Kubernetes pods with rotating egress, serverless), IP allowlisting may be impractical without stable egress IPs.
  • Service account auth (JWT bearer tokens) is the recommended primary access control for ingest. Use that instead of, or in addition to, IP filtering.
  • If IP filtering is a hard requirement for your deployment (for example, compliance mandates), let us know. We can allocate engineering resources to work through the configuration with you.

Additional operational considerations⚓︎

Multi-table routing⚓︎

Integrators may route different log types to different Hydrolix tables. Each table has its own transforms. The x-hdx-table and x-hdx-transform headers must be set per HTTP POST, not globally. If an integration batches mixed log types into a single HTTP request targeting one table, records that don't match the transform will fail with HTTP 207 errors.

Health check endpoint⚓︎

Before routing traffic to a Hydrolix cluster, verify the cluster is reachable. Use a lightweight health check (for example, GET /config/v1/health or equivalent) rather than discovering unavailability on the first ingest POST.

Connection pooling and keep-alive⚓︎

Use HTTP keep-alive and connection pooling to the Hydrolix endpoint. TLS handshake overhead on every request is significant at high ingest rates. A pool of 4-8 persistent connections per collector is a reasonable starting point.

Integration checklist⚓︎

This page covers the items on this list, ordered by importance. Each item links to its explanation.

Item Priority
Retry on 429 with exponential backoff (up to 5 min) Required
Backpressure propagation: throttle upstream on 429 Required
Handle 207: log per-message errors, alert on elevated rate Required
JWT token lifecycle: decode exp, proactive refresh Required
Client timeout at 29s (server = 30s) Required
Normalize all timestamps to single field and format Required
x-hdx-table and x-hdx-transform set per batch Required
gzip compression on all payloads Strongly recommended
~1 MB uncompressed payload target Best practice
Pre-sort batch by timestamp Best practice
Health check before routing traffic Recommended
HTTP keep-alive / connection pooling Recommended
Intake pool configuration (if multi-tenant) Situational
IP filtering coordination (cluster-level, if required) Situational
Dedup strategy (integrator-managed) Situational

Learn more⚓︎