Load Data

Data reaches a table by streaming, batch, or manual ingest, and each path ends with an indexer writing partitions. For the systems that send the data, see Data Sources.

Failures are either back pressure, where senders exceed cluster capacity and requests are rejected, or data accepted but absent from the table because it doesn't match the transform.

Ingested data doesn't appear in the table

Required access: Cluster

Confirm: Intake head logs carry Error level messages. Common causes: the primary column's datetime format, a string sent to a UINT column, or a transform file type that doesn't match the payload.

Fix: Stream Debugging: Ingested Data Doesn't Appear in Table.

Ingest returns HTTP 400

Required access: User

Confirm: The response body names the problem, such as no destination table specified, unknown transform, unhandled content-type, or unknown compression algorithm.

Intake head rejected the request before reading any rows. The response body says which part of the request was wrong.

Fix: Correct the part of the request the error names.

• Path: send requests to https://hostname.hydrolix.live/ingest/event.
• Table: name the destination as project.table, in the table query string parameter or the X-Hdx-Table header.
• Transform: if the request names one, in the transform query string parameter or the X-Hdx-Transform header, it must belong to that table.
• Content type: set Content-Type to a format intake head accepts, such as application/json or text/csv.
• Compression: intake head decompresses the body using the Content-Encoding header. Set it to the compression the sender used, such as gzip, or leave it unset for an uncompressed body.

See the Streaming Ingest API.

Ingest returns HTTP 429

Required access: Cluster

Confirm: sum by (service) (rate(http_source_request_count{status_code="429"}[30m])) returns a non-zero rate for {service="intake-head"}.

Intake head rejected the request instead of queuing it. Senders should back off or drop the batch, depending on their retry policy.

Intake head returns a 429 for one of three reasons, and only when the corresponding limit is set. All three are off by default:

• Too many requests in progress: more requests are in progress than intake_head_max_outstanding_requests allows.
• Pipeline too slow to accept: the pipeline that builds partitions didn't accept the request within intake_head_accept_data_timeout.
• Rate limit reached: the project, table, or transform passed its Rate Limiting setting. The response body says rate exceeded.

Navigate to https://hostname.hydrolix.live/prometheus and run:

sum by (service) (rate(http_source_request_count{status_code="429"}[30m]))

Fix if the rate limit was reached: raise the rate limit or slow the sender.

Find the bottleneck: for the other two causes, check each intake head pool's p99 bucket maintenance time, in seconds:

max by (pool) (hdx_sink_bucket_maint_duration_ns{service="intake-head", quantile="0.99"}) / 1e9

Intake head sorts rows into buckets by time. Once a second, it closes old or full buckets and hands them off to become partitions. This metric is how long that step takes. The hand-off waits until partition creation can accept more work. A high p99 means partition creation is behind, not parsing. The p99 shows the slowest runs, which are the ones that hold up requests.

Fix if parsing is the bottleneck (p99 of a few seconds or less): scale up intake head. Intake spill doesn't help here.

spec:
  scale:
    intake-head:
      replicas: 6

If incoming volume varies a lot, autoscale intake head instead of setting a fixed replica count. A fixed count can be too many at low load and too few at peak demand. See Prevent.

Side effects of adding replicas: each intake head replica keeps its own time buckets, and a bucket closes when it reaches its row limit or its time limit. With more replicas, the same data is split more ways. Buckets fill more slowly, and more of them reach their time limit before they're full. That means more, smaller partitions, which adds uploads and catalog writes, and gives merge more work.

Fix if partition creation is the bottleneck (p99 of more than a few seconds): this is often slow object storage. See Configure intake spill.

Side effects of intake spill: it keeps data flowing, but spilled data goes to object storage and comes back later. That adds storage operations, and the oldest spilled data reaches the table last.

Prevent: Autoscale intake head on http_source_outstanding_reqs. See HDX Autoscaler with Prometheus. If partition creation slows down from time to time, keep intake spill enabled. Make sure senders back off on a 429, since immediate retries prolong a burst. See Backpressure propagation.

Ingest returns HTTP 503

Required access: Cluster

Confirm: sum by (service) (rate(http_source_request_count{status_code="503"}[30m])) returns a non-zero rate for {service="intake-head"}.

A 503 usually means no intake head replica was available to take the request.

Navigate to https://hostname.hydrolix.live/prometheus and run:

sum by (service) (rate(http_source_request_count{status_code="503"}[30m]))

Then check how many intake head replicas are available:

kubectl get deployment/intake-head -o wide

Fix: Scale intake head in the cluster spec:

spec:
  scale:
    intake-head:
      replicas: 2

See Scale Profiles.

Ingest clients get a connection timeout

Required access: Cluster

Confirm: kubectl get deployment/traefik -o wide shows fewer than one available replica, or the requesting IP is absent from the cluster spec allowlist.

Nothing accepted the connection. Either no Traefik replica is running, or the load balancer drops traffic from the client's IP address.

Check that Traefik has at least one available replica:

kubectl get deployment/traefik -o wide

If Traefik is running, check that the cluster spec's ip_allowlist includes the client's address. The load balancer accepts only the CIDR ranges listed there, and a new cluster allows no outside clients.

kubectl get hydrolixcluster -o jsonpath='{.items[*].spec.ip_allowlist}{"\n"}'

Fix: If no Traefik replica is available, scale Traefik in the cluster spec:

spec:
  scale:
    traefik:
      replicas: 2

If the client's address is missing, add its CIDR range to ip_allowlist. See Configure IP Access.

If clients get 403 Forbidden instead of a timeout, the Traefik IP allowlist may be blocking them. See Configure the Traefik IP Allowlist.

Intake pods are OOM-killed

Required access: Cluster

Confirm: kubectl get events --field-selector reason=OOMKilling names intake-head, kinesis-peer, kafka-peer, or akamai-siem-peer pods.

An intake container tried to use more memory than its limit allows, and Kubernetes stopped it. The affected pods can be intake-head, kinesis-peer, kafka-peer, or akamai-siem-peer.

Fix: Configure the OOMKill detector and data splitter. See OOMKill Detector and Data Splitter.

Back to all symptoms