Skip to content

Collector Setup Guide

Native OpenTelemetry ingest is available in Hydrolix version 6.3 and later.

An OpenTelemetry Collector sits between applications and Hydrolix, receiving telemetry from one or more sources and forwarding it to the Hydrolix gRPC endpoint on port 4317. This guide covers configuring a Collector to route OpenTelemetry logs, metrics, and traces to Hydrolix tables.

For an overview of how Hydrolix ingests OpenTelemetry data, see OpenTelemetry ingest.

To send telemetry from an application without a Collector, see the SDK setup guide.

When to use a Collector⚓︎

Using a Collector is recommended for production deployments with benefits at scale:

  • Batching and retries buffer data and smooth out bursts, so a momentary network problem doesn't drop telemetry.
  • Filtering and redaction drop or mask fields before they leave the network, which helps keep sensitive data out of storage.
  • Fan-out and aggregation let many applications share one egress path to Hydrolix instead of each holding its own connection.

Prerequisites⚓︎

Before configuring the Collector, create the tables and transforms that receive the data. Each target table and its transform must exist before the Collector sends data.

Set up the following in Hydrolix:

  1. Native OpenTelemetry ingest enabled on the target cluster. A Collector can't reach the endpoint until it is. See Enable OpenTelemetry ingest.
  2. A table for each signal type you plan to send, named in project.table format. See Projects and tables.
  3. A transform for each table that maps the OpenTelemetry fields to columns. See Reference transforms for the standard logs, metrics, and traces schemas.

Configure the OTLP exporter⚓︎

Hydrolix accepts OTLP over gRPC and over HTTP. This guide configures the Collector's otlp exporter to point at the cluster endpoint on port 4317. Add routing headers to identify the target table and transform. To send over HTTP instead, use the otlphttp exporter with the endpoints listed under Protocol.

Collector Exporter for Hydrolix
1
2
3
4
5
6
7
exporters:
  otlp/hydrolix:
    endpoint: hostname.hydrolix.live:4317
    headers:
      Authorization: "Bearer ${env:HDX_TOKEN}"
      x-hdx-table: "my_project.logs"
      x-hdx-transform: "otel_logs"

Routing headers⚓︎

Hydrolix schemas are user-defined. Each request must carry metadata headers that identify the target table and transform.

Header Required Description
x-hdx-table Yes Target table in project.table format
x-hdx-transform No Transform to apply. Uses the table's default transform if omitted.

Route signal types to separate tables⚓︎

Write each signal type to its own table:

  1. Define one otlp exporter per target table. Give each exporter its own x-hdx-table header.
  2. Add each exporter to the pipeline for its signal type.

This example sends logs, metrics, and traces to separate tables:

Route Logs, Metrics, and Traces to Separate Tables
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 10s
    send_batch_size: 8192

exporters:
  otlp/hydrolix_logs:
    endpoint: hostname.hydrolix.live:4317
    headers:
      Authorization: "Bearer ${env:HDX_TOKEN}"
      x-hdx-table: "my_project.logs"
      x-hdx-transform: "otel_logs"
  otlp/hydrolix_metrics:
    endpoint: hostname.hydrolix.live:4317
    headers:
      Authorization: "Bearer ${env:HDX_TOKEN}"
      x-hdx-table: "my_project.metrics"
      x-hdx-transform: "otel_metrics"
  otlp/hydrolix_traces:
    endpoint: hostname.hydrolix.live:4317
    headers:
      Authorization: "Bearer ${env:HDX_TOKEN}"
      x-hdx-table: "my_project.traces"
      x-hdx-transform: "otel_traces"

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/hydrolix_logs]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/hydrolix_metrics]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/hydrolix_traces]

Test the connection⚓︎

telemetrygen sends synthetic telemetry to verify that data reaches a table before wiring up real applications. This command sends 400 log records (100 from each of 4 workers). The --rate 0 flag removes the default one-record-per-second limit so they send right away:

Send Test Logs with telemetrygen
1
2
3
4
5
telemetrygen logs \
  --otlp-endpoint hostname.hydrolix.live:4317 \
  --otlp-header 'Authorization="Bearer ${HDX_TOKEN}"' \
  --otlp-header 'x-hdx-table="my_project.logs"' \
  --logs 100 --workers 4 --rate 0

The endpoint uses TLS, so telemetrygen connects securely without extra flags. Add --otlp-insecure only when the target cluster has TLS disabled, such as a development cluster.

When the command finishes, query the target table to confirm all 400 rows arrived:

Confirm Test Data Arrived
1
SELECT count() FROM my_project.logs

A count of 400 confirms that the Collector reached the endpoint and the transform accepted the data.

Authentication⚓︎

The gRPC endpoint requires a bearer token, which can be a service account token. Add an Authorization header to each exporter:

Bearer Token Authentication
1
2
3
4
exporters:
  otlp/hydrolix:
    headers:
      Authorization: "Bearer ${env:HDX_TOKEN}"

Limitations⚓︎

Hydrolix doesn't create tables or transforms. Both must exist before the Collector sends data.