Skip to content

Overview

Native OpenTelemetry ingest is available in Hydrolix version 6.3 and later.

OpenTelemetry is an open standard for collecting observability data - logs, metrics, and traces - from applications and infrastructure. Storing this data in Hydrolix enables large-scale analysis alongside other analytics data.

Hydrolix accepts OpenTelemetry data natively over gRPC and HTTP, routing each signal to a user-defined table for analysis.

Enable OpenTelemetry ingest⚓︎

Native OpenTelemetry ingest is off by default. A self-managed cluster administrator enables it in the HydrolixCluster spec. On a Hydrolix-managed deployment, contact Hydrolix to enable it.

To enable native ingest on a self-managed cluster, set intake_head_df_enabled to true and scale on the intake-head-df service, which needs at least one replica. Setting the flag doesn't scale the service. The Hydrolix operator rejects the configuration if the flag is set while the service is scaled to zero.

Enable Native OpenTelemetry Ingest
1
2
3
4
5
spec:
  intake_head_df_enabled: true
  scale:
    intake-head-df:
      replicas: 1

Size the service for the expected ingest volume. See Scale by Component to set replicas and resources per service.

After the operator applies the change, confirm an intake-head-df pod is running and the cluster answers on port 4317 for gRPC, or on /v1/traces for HTTP.

Signal types⚓︎

Hydrolix ingests OpenTelemetry logs, writing them to a user-defined table. It also accepts OpenTelemetry metrics and traces, each written to its own table.

Routing⚓︎

Hydrolix routes data using request headers, carried as gRPC metadata or as HTTP headers. Because schemas are user-defined, routing requires explicit table and transform headers on each request.

Each table has a name in project.table format, and one or more transforms map incoming fields to columns. Create the target table and its transform first, then start sending. Reference transforms gives a starting schema for each signal type.

Two headers control routing:

Header Required Description
x-hdx-table Yes Target table in project.table format
x-hdx-transform No Transform to apply. Uses the table's default transform if omitted.

Set these headers in the OpenTelemetry Collector's exporter configuration or, for SDK-direct ingest, in the OTLP exporter's metadata or headers.

Ingest modes⚓︎

Data can reach Hydrolix in two ways.

%%{init: {
  'theme': 'base',
  'themeVariables': {
    'background': 'transparent',
    'fontSize': '18px',
    'edgeLabelBackground': 'transparent',
    'lineColor': '#003D66',
    'primaryColor': '#F4F6F8',
    'primaryBorderColor': '#003D66',
    'primaryTextColor': '#424D57'
  },
  'flowchart': {
    'padding': 30,
    'nodeSpacing': 120,
    'rankSpacing': 100,
    'curve': 'basis'
  }
}}%%
graph LR
    subgraph direct["SDK-direct"]
        App1["Application"]
    end
    subgraph mediated["Collector-mediated"]
        App2["Application"]
        App3["Application"]
        Coll["OpenTelemetry Collector"]
    end
    EP["Hydrolix<br/>OTLP endpoint"]

    App1 -->|"OTLP/gRPC"| EP
    App2 --> Coll -->|"OTLP/gRPC"| EP
    App3 --> Coll

    style App1 fill:none,stroke:#003D66,stroke-width:2px
    style App2 fill:none,stroke:#003D66,stroke-width:2px
    style App3 fill:none,stroke:#003D66,stroke-width:2px
    style Coll fill:none,stroke:#003D66,stroke-width:2px
    style EP fill:#00A99D,stroke:#035F60,stroke-width:2px,color:#FFFFFF
    style direct fill:none,stroke:#003D66,stroke-width:1px
    style mediated fill:none,stroke:#003D66,stroke-width:1px
%%{init: {
  'theme': 'base',
  'themeVariables': {
    'background': 'transparent',
    'fontSize': '18px',
    'edgeLabelBackground': 'transparent',
    'lineColor': '#88C9F2',
    'primaryColor': 'transparent',
    'primaryBorderColor': '#88C9F2',
    'primaryTextColor': '#FAFBFC'
  },
  'flowchart': {
    'padding': 30,
    'nodeSpacing': 120,
    'rankSpacing': 100,
    'curve': 'basis'
  }
}}%%
graph LR
    subgraph direct["SDK-direct"]
        App1["Application"]
    end
    subgraph mediated["Collector-mediated"]
        App2["Application"]
        App3["Application"]
        Coll["OpenTelemetry Collector"]
    end
    EP["Hydrolix<br/>OTLP endpoint"]

    App1 -->|"OTLP/gRPC"| EP
    App2 --> Coll -->|"OTLP/gRPC"| EP
    App3 --> Coll

    style App1 fill:none,stroke:#88C9F2,stroke-width:2px
    style App2 fill:none,stroke:#88C9F2,stroke-width:2px
    style App3 fill:none,stroke:#88C9F2,stroke-width:2px
    style Coll fill:none,stroke:#88C9F2,stroke-width:2px
    style EP fill:#00A99D,stroke:#008C82,stroke-width:2px,color:#FFFFFF
    style direct fill:none,stroke:#88C9F2,stroke-width:1px
    style mediated fill:none,stroke:#88C9F2,stroke-width:1px

Collector-mediated ingest uses an OpenTelemetry Collector between the application and Hydrolix. The Collector batches, retries, and buffers for many applications at once, and gives them a single egress path. It can also filter and enrich telemetry before it reaches Hydrolix. See the Collector setup guide for configuration instructions.

SDK-direct ingest sends data from a language SDK directly to a Hydrolix OpenTelemetry endpoint. The SDKs batch and retry on their own, so this path suits high-volume senders as well as simple ones, and it needs less setup than a Collector. See the SDK setup guide for language-specific examples.

Protocol⚓︎

Hydrolix accepts OTLP over two transports. Both carry the same routing headers and the same bearer token.

Transport Endpoint Encoding
OTLP/gRPC Port 4317 protobuf
OTLP/HTTP /v1/traces, /v1/metrics, /v1/logs protobuf or JSON

The HTTP routes answer on the cluster's standard HTTPS port, so a sender needs no additional port opened outbound. Port 4318 serves the same routes for senders that expect the OpenTelemetry default port.

The HTTP routes also return CORS headers and answer the preflight OPTIONS request, so a browser application can send telemetry to a cluster directly.

Authentication⚓︎

Both endpoints use TLS and require a bearer token, sent as an Authorization header. A service account token works well for a long-running sender. See the Collector setup guide for configuration.

A table can also require a table access token, which is a separate, optional control on which senders may write to it.