Load Data

Data reaches a table by streaming, batch, or manual ingest, and each path ends with an indexer writing partitions. For the systems that send the data, see Data Sources.

Failures are either back pressure, where senders exceed cluster capacity and requests are rejected, or data accepted but absent from the table because it doesn't match the transform.

Ingested data doesn't appear in the table

Confirm: Intake head logs carry Error level messages. Common causes: the primary column's datetime format, a string sent to a UINT column, or a transform file type that doesn't match the payload.

Fix: Stream Debugging: Ingested Data Doesn't Appear in Table.

Ingest returns HTTP 429

Confirm: sum by (service) (rate(http_source_request_count{status_code="429"}[30m])) returns a non-zero rate for {service="intake-head"}.

Intake head rejects requests when senders exceed what the cluster can accept. The sender receives a 429 and, depending on its retry policy, either backs off or drops the batch.

Navigate to https://{hostname}/prometheus and run:

sum by (service) (rate(http_source_request_count{status_code="429"}[30m]))

Fix. Scale intake head in the cluster spec:

spec:
  scale:
    intake-head:
      replicas: 6

Then enable intake spill so bursts are buffered rather than rejected.

Prevent. Size intake head for peak rather than average throughput, and confirm senders implement backoff. A sender that retries immediately on 429 turns a brief burst into a sustained one.

Ingest returns HTTP 503

Confirm: sum by (service) (rate(http_source_request_count{status_code="503"}[30m])) returns a non-zero rate for {service="intake-head"}.

Fix: Stream Debugging: 503 Service Temporarily Unavailable.

Clients time out reaching the ingest endpoint

Confirm: kubectl get deployment/traefik -o wide shows fewer than one available replica, or the requesting IP is absent from the cluster spec allowlist.

Fix: Stream Debugging: Client Connection Timeout.

Intake pods are OOM-killed

Confirm: kubectl get events --field-selector reason=OOMKilling names intake-head, kinesis-peer, kafka-peer, or akamai-siem-peer pods.

Fix: Stream Debugging: Out of Memory (OOMKill) Errors in Turbine.

Back to all symptoms