OpenTelemetry Collector

The cluster-monitoring OpenTelemetry Collector is available in Hydrolix version 6.0 and later.

A Hydrolix cluster can monitor itself by running an operator-managed OpenTelemetry Collector. The Collector receives the OTLP logs, metrics, and traces that the cluster's own services produce. It writes them to Hydrolix tables in the same cluster or a different one, or sends them to another OpenTelemetry-compatible destination. The hydrolix_otel tunables enable and configure the Collector.

How it works

When hydrolix_otel.enabled is true, the Hydrolix operator deploys a Collector named hdx-otel-collector as its own Kubernetes Deployment and Service. The operator generates a default Collector configuration that:

• Receives OTLP over gRPC on port 5317 and HTTP on port 5318.
• Batches the data and adds the deployment.id and service.version resource attributes, which the exporters copy into a serviceTags map on every record.
• Exports each signal type to the endpoint set in hydrolix_otel.exporter_endpoint, using the table and transform configured for that signal. The exporters authenticate with a bearer token.

%%{init: {'theme':'base','themeVariables':{'background':'transparent','fontSize':'18px','edgeLabelBackground':'transparent'},'flowchart':{'padding':20,'nodeSpacing':80,'rankSpacing':55,'curve':'basis'}}}%%
graph TD
    recv["Receivers<br/>accept OTLP on gRPC 5317 / HTTP 5318"] --> proc["Processors<br/>batch and stamp resource attributes"] --> exp["Exporters<br/>send each signal to its table"]
    exp --> logs[(otel_logs)]
    exp --> metrics[(otel_metrics)]
    exp --> traces[(traces)]

    classDef box fill:none,stroke:#003D66,stroke-width:3px,color:#003D66;
    classDef accent fill:none,stroke:#00A99D,stroke-width:3px,color:#035F60;
    class recv,proc,exp box;
    class logs,metrics,traces accent;

%%{init: {'theme':'base','themeVariables':{'background':'transparent','fontSize':'18px','edgeLabelBackground':'transparent'},'flowchart':{'padding':20,'nodeSpacing':80,'rankSpacing':55,'curve':'basis'}}}%%
graph TD
    recv["Receivers<br/>accept OTLP on gRPC 5317 / HTTP 5318"] --> proc["Processors<br/>batch and stamp resource attributes"] --> exp["Exporters<br/>send each signal to its table"]
    exp --> logs[(otel_logs)]
    exp --> metrics[(otel_metrics)]
    exp --> traces[(traces)]

    classDef box fill:none,stroke:#88C9F2,stroke-width:3px,color:#FAFBFC;
    classDef accent fill:none,stroke:#6EF6D9,stroke-width:3px,color:#FAFBFC;
    class recv,proc,exp box;
    class logs,metrics,traces accent;

The default exporter endpoint is the cluster's own stream head, http://stream-head:8089/ingest/event. Monitoring data lands in the same cluster that produces it.

Cluster services send their telemetry to the Collector, and supporting trace Collectors on individual pods forward to it over OTLP HTTP on port 5318. The cluster's Traefik ingress also exposes the Collector's gRPC port at the cluster edge, behind unified authentication. External sources with a valid bearer token can send OTLP data to the Collector.

%%{init: {'theme':'base','themeVariables':{'background':'transparent','fontSize':'18px','edgeLabelBackground':'transparent'},'flowchart':{'padding':30,'nodeSpacing':150,'rankSpacing':120,'curve':'basis'}}}%%
graph LR
    external[External source<br/>with bearer token] -.->|OTLP/gRPC| traefik

    subgraph hdx[Hydrolix]
        services[Cluster services] -->|OTLP| collector[hdx-otel-collector]
        traefik[Traefik ingress<br/>unified auth] -.-> collector
    end

    classDef box fill:none,stroke:#003D66,stroke-width:3px,color:#003D66;
    classDef accent fill:none,stroke:#00A99D,stroke-width:3px,color:#035F60;
    classDef ext fill:none,stroke:#8B9FAD,stroke-width:2px,stroke-dasharray:5 4,color:#4A5F73;
    classDef group fill:none,stroke:#003D66,stroke-width:2px,color:#003D66;
    class services box;
    class collector accent;
    class external,traefik ext;
    class hdx group;

%%{init: {'theme':'base','themeVariables':{'background':'transparent','fontSize':'18px','edgeLabelBackground':'transparent'},'flowchart':{'padding':30,'nodeSpacing':150,'rankSpacing':120,'curve':'basis'}}}%%
graph LR
    external[External source<br/>with bearer token] -.->|OTLP/gRPC| traefik

    subgraph hdx[Hydrolix]
        services[Cluster services] -->|OTLP| collector[hdx-otel-collector]
        traefik[Traefik ingress<br/>unified auth] -.-> collector
    end

    classDef box fill:none,stroke:#88C9F2,stroke-width:3px,color:#FAFBFC;
    classDef accent fill:none,stroke:#6EF6D9,stroke-width:3px,color:#FAFBFC;
    classDef ext fill:none,stroke:#8B9FAD,stroke-width:2px,stroke-dasharray:5 4,color:#B8C3CB;
    classDef group fill:none,stroke:#88C9F2,stroke-width:2px,color:#FAFBFC;
    class services box;
    class collector accent;
    class external,traefik ext;
    class hdx group;

Prerequisites

The Collector doesn't create its destination tables or transforms. Before enabling it, create the following in the destination cluster:

1. A table for each signal type the Collector sends: logs, metrics, and traces.
2. A transform for each table that maps the OpenTelemetry fields to columns. Hydrolix publishes example transforms for the logs, metrics, and traces signals that match the Collector's exporter output. Use them as a starting point.

You also need a service account bearer token that's valid on the destination cluster. The service account needs a role with the ingest_table permission scoped to each destination table.

Enable the Collector

Add the hydrolix_otel block to the cluster spec in hydrolixcluster.yaml. Name the table and transform for each signal type:

spec:
  hydrolix_otel:
    enabled: true
    traces:
      hdx_table: hydro.traces
      hdx_transform: tf_otel_traces
    metrics:
      hdx_table: hydro.otel_metrics
      hdx_transform: tf_otel_metrics
    logs:
      hdx_table: hydro.otel_logs
      hdx_transform: tf_otel_logs

The hydrolix_otel tunable accepts these settings:

Tunable	Default	Description
enabled	false	Whether to deploy the Collector
configmap_name	hdx-otel-cfg	Name of the ConfigMap holding the Collector configuration, set for a custom configuration
exporter_endpoint	http://stream-head:8089/ingest/event	Destination for the monitoring data, defaulting to the cluster's own stream head
grpc_port	5317	OTLP gRPC receiver port
http_port	5318	OTLP HTTP receiver port
traces, metrics, logs	empty	Per-signal destination, each with an hdx_table and an hdx_transform

For defaults and an example, see the tunables reference.

Configure the bearer token

The Collector authenticates to the destination with a bearer token. It reads the token from the OTEL_EXPORTER_OTLP_BEARER_TOKEN key of the cluster's general secret. The operator builds the general secret from the cluster's curated secret, so add the key to curated and the operator merges it in.

The curated secret usually already holds keys that other features use. Add the key to it rather than recreate the secret, which would drop those keys. This kubectl patch adds the one key and leaves the rest in place:

kubectl -n ${HDX_KUBERNETES_NAMESPACE} patch secret curated --type merge \
  -p '{"stringData":{"OTEL_EXPORTER_OTLP_BEARER_TOKEN":"<bearer-token>"}}'

Using stringData lets you supply the token as plain text; Kubernetes encodes it. For other ways to create, display, and edit the curated secret, see Curated secrets.

The token must be valid on the cluster named in exporter_endpoint and authorized to ingest into the destination tables.

Query the monitoring data

The Collector flattens each OpenTelemetry record into the columns the destination table defines. The service.name resource attribute lands in the serviceName column. The remaining resource attributes, including deployment.id and service.version, land in the serviceTags map, where each row carries its own copy. Record-level attributes land in the tags map.

Query a map column by key. This example reads recent logs from hydro.otel_logs:

SELECT timestamp, serviceName, serviceTags['deployment.id'] AS deployment, severity_text, body
FROM hydro.otel_logs
ORDER BY timestamp DESC
LIMIT 100

Send monitoring data to another Hydrolix cluster

To send monitoring data somewhere other than the cluster's own stream head, set exporter_endpoint to the ingest URL of the destination cluster. This pattern suits a central observability cluster that collects monitoring data from several production clusters.

spec:
  hydrolix_otel:
    enabled: true
    exporter_endpoint: https://observability.example.com/ingest/event
    traces:
      hdx_table: hydro.traces
      hdx_transform: tf_otel_traces
    metrics:
      hdx_table: hydro.otel_metrics
      hdx_transform: tf_otel_metrics
    logs:
      hdx_table: hydro.otel_logs
      hdx_transform: tf_otel_logs

The tables, transforms, and bearer token must all exist on the destination cluster.

Use a custom Collector configuration

For anything beyond the generated defaults, supply your own Collector configuration. Create a ConfigMap with the full Collector configuration under a config.yaml key, then point configmap_name at it:

spec:
  hydrolix_otel:
    enabled: true
    configmap_name: my-otel-config

A custom ConfigMap replaces the generated configuration entirely. It must define the receivers, processors, exporters, and pipelines itself. To use the generated configuration as a starting point, read it from the ConfigMap the operator creates:

kubectl get configmap hdx-otel-cfg -n <namespace> -o yaml

Each signal pipeline in the generated configuration uses a hydrolix exporter, which writes to a Hydrolix table through the hdx_table, hdx_transform, and hdx_bearer_token settings. Keep a hydrolix exporter on any pipeline that should continue writing to Hydrolix.

A custom configuration can use any Collector components the Hydrolix Collector image ships with. Common uses include:

• Exporting to a non-Hydrolix destination, such as Prometheus, alongside or instead of the Hydrolix exporters.
• Adding receivers for other sources, such as Kubernetes events or log files.
• Filtering or sampling data before export.

This example adds a Prometheus remote-write exporter to the metrics pipeline alongside the Hydrolix exporter:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-otel-config
data:
  config.yaml: |
    # ... receivers, processors, and hydrolix exporters ...
    exporters:
      prometheusremotewrite:
        endpoint: http://prometheus:9090/api/v1/write
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [resource, batch]
          exporters: [hydrolix/metrics, prometheusremotewrite]

Send Traefik telemetry to the Collector

Traefik metrics and tracing aren't included in hydrolix_otel.enabled. Each requires its own flag in the cluster spec, and both depend on the Collector being enabled:

spec:
  hydrolix_otel:
    enabled: true
  enable_traefik_otel_tracing: true
  enable_traefik_otel_metrics: true

Relationship to the legacy otel_endpoint tunable

Before version 6.0, the otel_endpoint tunable pointed per-pod trace Collectors at an external OTLP HTTP server. With hydrolix_otel.enabled set to true, those Collectors send to the in-cluster hdx-otel-collector instead, and otel_endpoint is no longer needed. If both are set, otel_endpoint takes precedence as the trace destination.