Stream Debugging

Configure components

Traefik

Minimum instances: 2

To check the number of instances currently running, run the following command:

kubectl get deployment/traefik -o wide

To change the number of instances, adjust the following section of the Hydrolix cluster configuration:

hydrolixcluster.yaml

...
traefik:
   replicas: {replica_count}
...

Intake head

Minimum instances: 1

To check the number of instances currently running, run the following command:

kubectl get deployment/intake-head -o wide

To change the number of instances, adjust the following section of the Hydrolix cluster configuration:

hydrolixcluster.yaml

...
intake-head:
   replicas: {replica_count}
...

Something not working?

Start at Troubleshooting Symptoms and Fixes. Find the symptom, confirm it with the signal listed there, and follow the link to the procedure.

Issue: Client connection timeout

Required access: Cluster

Confirm: kubectl get deployment/traefik -o wide shows fewer than one available replica, or the requesting IP is absent from the cluster spec allowlist.

Nothing accepted the connection. Either no Traefik replica is running, or the load balancer drops traffic from the client's IP address.

Check that Traefik has at least one available replica:

kubectl get deployment/traefik -o wide

If Traefik is running, check that the cluster spec's ip_allowlist includes the client's address. The load balancer accepts only the CIDR ranges listed there, and a new cluster allows no outside clients.

kubectl get hydrolixcluster -o jsonpath='{.items[*].spec.ip_allowlist}{"\n"}'

Fix: If no Traefik replica is available, scale Traefik in the cluster spec:

spec:
  scale:
    traefik:
      replicas: 2

If the client's address is missing, add its CIDR range to ip_allowlist. See Configure IP Access.

If clients get 403 Forbidden instead of a timeout, the Traefik IP allowlist may be blocking them. See Configure the Traefik IP Allowlist.

Issue: HTTP 429 errors from intake head

Required access: Cluster

Confirm: sum by (service) (rate(http_source_request_count{status_code="429"}[30m])) returns a non-zero rate for {service="intake-head"}.

Intake head rejected the request instead of queuing it. Senders should back off or drop the batch, depending on their retry policy.

Intake head returns a 429 for one of three reasons, and only when the corresponding limit is set. All three are off by default:

• Too many requests in progress: more requests are in progress than intake_head_max_outstanding_requests allows.
• Pipeline too slow to accept: the pipeline that builds partitions didn't accept the request within intake_head_accept_data_timeout.
• Rate limit reached: the project, table, or transform passed its Rate Limiting setting. The response body says rate exceeded.

Navigate to https://hostname.hydrolix.live/prometheus and run:

sum by (service) (rate(http_source_request_count{status_code="429"}[30m]))

Fix if the rate limit was reached: raise the rate limit or slow the sender.

Find the bottleneck: for the other two causes, check each intake head pool's p99 bucket maintenance time, in seconds:

max by (pool) (hdx_sink_bucket_maint_duration_ns{service="intake-head", quantile="0.99"}) / 1e9

Intake head sorts rows into buckets by time. Once a second, it closes old or full buckets and hands them off to become partitions. This metric is how long that step takes. The hand-off waits until partition creation can accept more work. A high p99 means partition creation is behind, not parsing. The p99 shows the slowest runs, which are the ones that hold up requests.

Fix if parsing is the bottleneck (p99 of a few seconds or less): scale up intake head. Intake spill doesn't help here.

spec:
  scale:
    intake-head:
      replicas: 6

If incoming volume varies a lot, autoscale intake head instead of setting a fixed replica count. A fixed count can be too many at low load and too few at peak demand. See Prevent.

Side effects of adding replicas: each intake head replica keeps its own time buckets, and a bucket closes when it reaches its row limit or its time limit. With more replicas, the same data is split more ways. Buckets fill more slowly, and more of them reach their time limit before they're full. That means more, smaller partitions, which adds uploads and catalog writes, and gives merge more work.

Fix if partition creation is the bottleneck (p99 of more than a few seconds): this is often slow object storage. See Configure intake spill.

Side effects of intake spill: it keeps data flowing, but spilled data goes to object storage and comes back later. That adds storage operations, and the oldest spilled data reaches the table last.

Prevent: Autoscale intake head on http_source_outstanding_reqs. See HDX Autoscaler with Prometheus. If partition creation slows down from time to time, keep intake spill enabled. Make sure senders back off on a 429, since immediate retries prolong a burst. See Backpressure propagation.

Issue: 503 Service Temporarily Unavailable

Required access: Cluster

Confirm: sum by (service) (rate(http_source_request_count{status_code="503"}[30m])) returns a non-zero rate for {service="intake-head"}.

A 503 usually means no intake head replica was available to take the request.

Navigate to https://hostname.hydrolix.live/prometheus and run:

sum by (service) (rate(http_source_request_count{status_code="503"}[30m]))

Then check how many intake head replicas are available:

kubectl get deployment/intake-head -o wide

Fix: Scale intake head in the cluster spec:

spec:
  scale:
    intake-head:
      replicas: 2

See Scale Profiles.

Issue: Intake head 4XX

Required access: User

Confirm: The response body names the problem, such as no destination table specified, unknown transform, unhandled content-type, or unknown compression algorithm.

Intake head rejected the request before reading any rows. The response body says which part of the request was wrong.

Fix: Correct the part of the request the error names.

• Path: send requests to https://hostname.hydrolix.live/ingest/event.
• Table: name the destination as project.table, in the table query string parameter or the X-Hdx-Table header.
• Transform: if the request names one, in the transform query string parameter or the X-Hdx-Transform header, it must belong to that table.
• Content type: set Content-Type to a format intake head accepts, such as application/json or text/csv.
• Compression: intake head decompresses the body using the Content-Encoding header. Set it to the compression the sender used, such as gzip, or leave it unset for an uncompressed body.

See the Streaming Ingest API.

Issue: Ingested Data Doesn't Appear in Table

Check incoming payloads match transform

Review intake head logs, looking for messages of level Error.

Check:

• Datetime format for Primary. This is often the cause of rejection of rows.
• Strings are trying to be stored as a UINT.
• Transform File type CSV/JSON

Fix incoming payloads match transform

Review transform and edit accordingly.

• Data Source Configuration
• Timestamp Data Types

Check data stream type matches

Review intake head logs.

Check:

• Datetime format for Primary. This is often the cause of rejection of rows.
• Strings are trying to be stored as a UINT.
• Transform File type CSV/JSON
• Transform SQL

Fix data stream type matches

Review transform and edit accordingly.

• Data Source Configuration
• Timestamp Data Types

Issue: Intake pods are OOM-killed

Required access: Cluster

Confirm: kubectl get events --field-selector reason=OOMKilling names intake-head, kinesis-peer, kafka-peer, or akamai-siem-peer pods.

An intake container tried to use more memory than its limit allows, and Kubernetes stopped it. The affected pods can be intake-head, kinesis-peer, kafka-peer, or akamai-siem-peer.

Fix: Configure the OOMKill detector and data splitter. See OOMKill Detector and Data Splitter.

Streaming service components

HTTP Streaming ingest uses the intake head component to process incoming data, with the Traefik load balancer directing traffic to intake head.

Legacy stream-head and stream-peer were removed

The diagram shows the legacy stream-head and stream-peer components, which were removed in Hydrolix version 6.1.5. Intake head does the work of both.

Accessible through the path: https://hostname.hydrolix.live/ingest/event

Component	Used to
Traefik - Application Load-balancer	Routes requests to appropriate end-points. Requests to the path /ingest are routed to intake head components. API and UI requests are routed using their own paths.
Intake head	Checks that messages conform to basic syntax and message structure rules. When a message passes these checks, it sends the message to a listing queue. When a message fails these checks, it returns an HTTP 400 response to the client. It applies the Hydrolix transform and outputs indexed database partitions to the Hydrolix Database bucket. It also reports created partition metadata to the Catalog.
Catalog	Stores information on the basic storage structure and partitions of the data in cloud storage, for example GCS and S3. Includes a persistent volume in Kubernetes.
Cloud Storage Bucket	Storage bucket, for example GCS or S3, containing the stateful data required to run the system, including configuration files (/config/), database (/db/), and a copy of the system logs (/logs/).
UI	User interface / Portal. Is built upon the Turbine-API.
Turbine-API	REST based API for configuration of the data system. Includes API end-points for creation, deletion, editing of tables and their transforms (schemas).
Keycloak	Provides authorization and RBAC for access to the Turbine-API and the Portal. Stores metadata and user information with the CatalogDB instance.