Skip to content

SDK Setup Guide

Native OpenTelemetry ingest is available in Hydrolix version 6.3 and later.

An OpenTelemetry SDK exports telemetry directly from an application to the Hydrolix gRPC endpoint on port 4317, without a Collector in between.

This guide covers configuring the Python, Java, Go, and Node.js SDKs to send OpenTelemetry logs, metrics, and traces to Hydrolix tables. Each example uses the same endpoint, routing headers, and TLS through the SDK's OTLP gRPC exporter. Hydrolix also accepts OTLP over HTTP. To send over HTTP instead, use the SDK's OTLP/HTTP exporter with the endpoints listed under Protocol.

For an overview of how Hydrolix ingests OpenTelemetry data, see OpenTelemetry ingest.

To forward telemetry from many applications through a shared pipeline, see the Collector setup guide.

When to send from an SDK⚓︎

Exporting directly from an application is the quickest path to get telemetry into Hydrolix. It works well for a single application, local development, and early testing.

For production deployments with many applications, a Collector adds batching, retries, filtering, and a shared egress path. See the Collector setup guide.

Prerequisites⚓︎

Set up the following before configuring an SDK:

  1. Enable native ingest. Native OpenTelemetry ingest is off by default. See Enable OpenTelemetry ingest. A Hydrolix-managed deployment has this handled for you.
  2. Find your endpoint. The endpoint is the hostname you use to reach Hydrolix, on port 4317. The examples use hostname.hydrolix.live:4317 as a placeholder.
  3. Create a service account token. The endpoint requires a bearer token. Create a service account token and make it available to the examples as the HDX_TOKEN environment variable.
  4. Create a table and transform for each signal. Each target table and its transform must exist before the SDK sends data. Name each table in project.table format; see Projects and tables. A transform maps the OpenTelemetry fields to columns; see Reference transforms for the standard logs, metrics, and traces schemas.

Install the SDK⚓︎

Install the OpenTelemetry SDK and the OTLP gRPC exporter for each signal type.

1
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc

Import the OpenTelemetry BOM to align artifact versions, then declare the API, SDK, and OTLP exporter without individual versions.

Maven:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-bom</artifactId>
      <version>1.63.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk</artifactId>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
  </dependency>
</dependencies>

Gradle:

1
2
3
4
5
6
dependencies {
  implementation platform("io.opentelemetry:opentelemetry-bom:1.63.0")
  implementation "io.opentelemetry:opentelemetry-api"
  implementation "io.opentelemetry:opentelemetry-sdk"
  implementation "io.opentelemetry:opentelemetry-exporter-otlp"
}

The opentelemetry-exporter-otlp artifact contains the gRPC span, metric, and log-record exporters.

1
2
3
4
5
go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go get go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc
go get go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
npm install @opentelemetry/api \
  @opentelemetry/sdk-trace \
  @opentelemetry/sdk-metrics \
  @opentelemetry/sdk-logs \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions \
  @opentelemetry/exporter-trace-otlp-grpc \
  @opentelemetry/exporter-metrics-otlp-grpc \
  @opentelemetry/exporter-logs-otlp-grpc \
  @grpc/grpc-js

The logs signal is the least mature part of the OpenTelemetry SDKs. The Go and Node.js logs packages are still pre-1.0 and their APIs can change between releases, so pin the versions you test against.

Configure the exporter⚓︎

Every exporter points at the same gRPC endpoint on port 4317 and carries routing headers that identify the target table and transform. Define the endpoint, token, and resource once. The resource identifies the service that emits the telemetry.

Some examples end with a flush or shutdown call, such as Python's force_flush or Go's Shutdown, to send buffered telemetry before a short program exits. A long-running application instead relies on the batch processor's periodic export.

1
2
3
4
5
6
import os
from opentelemetry.sdk.resources import Resource

HDX_ENDPOINT = "hostname.hydrolix.live:4317"
HDX_TOKEN = os.environ["HDX_TOKEN"]
resource = Resource.create({"service.name": "my-service"})

1
2
3
4
5
6
7
8
import io.opentelemetry.sdk.resources.Resource;

String endpoint = "https://hostname.hydrolix.live:4317";
String token = System.getenv("HDX_TOKEN");
Resource resource =
    Resource.getDefault().toBuilder()
        .put("service.name", "my-service")
        .build();

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import (
    "context"
    "os"

    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

const endpoint = "hostname.hydrolix.live:4317"

ctx := context.Background()
token := os.Getenv("HDX_TOKEN")
res, err := resource.New(ctx,
    resource.WithAttributes(semconv.ServiceName("my-service")),
)

The gRPC exporters carry routing headers as gRPC metadata, so build the metadata for each signal with @grpc/grpc-js.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
const grpc = require('@grpc/grpc-js');
const { resourceFromAttributes } = require('@opentelemetry/resources');
const { ATTR_SERVICE_NAME } = require('@opentelemetry/semantic-conventions');

const endpoint = 'hostname.hydrolix.live:4317';
const token = process.env.HDX_TOKEN;
const resource = resourceFromAttributes({ [ATTR_SERVICE_NAME]: 'my-service' });

function routing(table, transform) {
  const metadata = new grpc.Metadata();
  metadata.set('authorization', `Bearer ${token}`);
  metadata.set('x-hdx-table', table);
  metadata.set('x-hdx-transform', transform);
  return metadata;
}

The service.name resource attribute maps to the service_name column in the reference transforms. Write the header keys in lowercase, such as authorization, to match the gRPC metadata format.

The endpoint requires TLS. Java connects over the https scheme; Python, Go, and Node.js use a bare host:port and connect over TLS by default.

Routing headers⚓︎

Hydrolix schemas are user-defined. Each request must carry a metadata header to identify the target table, and can carry a header specifying a non-default transform.

Header Required Description
x-hdx-table Yes Target table in project.table format
x-hdx-transform No Transform to apply. Uses the table's default transform if omitted.

These headers travel as gRPC metadata. Most SDKs set them through a headers or addHeader option on the exporter. The OpenTelemetry JavaScript gRPC exporter ignores the headers option and reads a metadata object built with @grpc/grpc-js instead.

Logs⚓︎

Send log records through a logger provider with an OTLP log exporter.

Log records require a timestamp

The native endpoint silently drops log records whose time_unix_nano is unset. Logging bridges set it for you, including Log4j, Logback, and the Python LoggingHandler in these examples. If you emit records directly, set the timestamp explicitly: .setTimestamp(Instant.now()) in Java, timestamp= on a Python LogRecord, record.SetTimestamp(time.Now()) in Go, or the timestamp field on the record passed to logger.emit() in Node.js.

The OpenTelemetry logs API for Python is in the opentelemetry.sdk._logs module.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import logging
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

provider = LoggerProvider(resource=resource)
provider.add_log_record_processor(
    BatchLogRecordProcessor(
        OTLPLogExporter(
            endpoint=HDX_ENDPOINT,
            headers=(
                ("authorization", f"Bearer {HDX_TOKEN}"),
                ("x-hdx-table", "my_project.logs"),
                ("x-hdx-transform", "otel_logs"),
            ),
        )
    )
)

handler = LoggingHandler(logger_provider=provider)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)
logging.getLogger("my-service").info("request handled", extra={"route": "/api"})

provider.force_flush()

In Java, applications usually emit logs through a Log4j or Logback bridge rather than calling the logs API directly. The SdkLoggerProvider is the export pipeline behind that bridge.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import io.opentelemetry.exporter.otlp.logs.OtlpGrpcLogRecordExporter;
import io.opentelemetry.sdk.logs.SdkLoggerProvider;
import io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor;

OtlpGrpcLogRecordExporter logExporter =
    OtlpGrpcLogRecordExporter.builder()
        .setEndpoint(endpoint)
        .addHeader("authorization", "Bearer " + token)
        .addHeader("x-hdx-table", "my_project.logs")
        .addHeader("x-hdx-transform", "otel_logs")
        .build();

SdkLoggerProvider loggerProvider =
    SdkLoggerProvider.builder()
        .setResource(resource)
        .addLogRecordProcessor(BatchLogRecordProcessor.builder(logExporter).build())
        .build();

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
import (
    "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
    sdklog "go.opentelemetry.io/otel/sdk/log"
)

logExporter, err := otlploggrpc.New(ctx,
    otlploggrpc.WithEndpoint(endpoint),
    otlploggrpc.WithHeaders(map[string]string{
        "authorization":   "Bearer " + token,
        "x-hdx-table":     "my_project.logs",
        "x-hdx-transform": "otel_logs",
    }),
)

loggerProvider := sdklog.NewLoggerProvider(
    sdklog.WithResource(res),
    sdklog.WithProcessor(sdklog.NewBatchProcessor(logExporter)),
)
defer loggerProvider.Shutdown(ctx)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
const { LoggerProvider, BatchLogRecordProcessor } = require('@opentelemetry/sdk-logs');
const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');

const logExporter = new OTLPLogExporter({
  url: endpoint,
  metadata: routing('my_project.logs', 'otel_logs'),
});

const loggerProvider = new LoggerProvider({
  resource,
  processors: [new BatchLogRecordProcessor({ exporter: logExporter })],
});

Log attributes map to the log_attributes column and resource attributes map to the resource_attributes column in the reference logs transform.

Metrics⚓︎

Record measurements through a meter provider with a periodic reader that pushes to the OTLP metric exporter.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(
        endpoint=HDX_ENDPOINT,
        headers=(
            ("authorization", f"Bearer {HDX_TOKEN}"),
            ("x-hdx-table", "my_project.metrics"),
            ("x-hdx-transform", "otel_metrics"),
        ),
    )
)
provider = MeterProvider(resource=resource, metric_readers=[reader])
meter = provider.get_meter("my-service")

requests = meter.create_counter("requests", unit="1")
requests.add(1, {"route": "/api"})

provider.force_flush()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.metrics.LongCounter;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter;
import io.opentelemetry.sdk.metrics.SdkMeterProvider;
import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader;

OtlpGrpcMetricExporter metricExporter =
    OtlpGrpcMetricExporter.builder()
        .setEndpoint(endpoint)
        .addHeader("authorization", "Bearer " + token)
        .addHeader("x-hdx-table", "my_project.metrics")
        .addHeader("x-hdx-transform", "otel_metrics")
        .build();

SdkMeterProvider meterProvider =
    SdkMeterProvider.builder()
        .setResource(resource)
        .registerMetricReader(PeriodicMetricReader.builder(metricExporter).build())
        .build();

Meter meter = meterProvider.get("my-service");
LongCounter requests = meter.counterBuilder("requests").setUnit("1").build();
requests.add(1, Attributes.builder().put("route", "/api").build());

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import (
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
    "go.opentelemetry.io/otel/metric"
    sdkmetric "go.opentelemetry.io/otel/sdk/metric"
)

metricExporter, err := otlpmetricgrpc.New(ctx,
    otlpmetricgrpc.WithEndpoint(endpoint),
    otlpmetricgrpc.WithHeaders(map[string]string{
        "authorization":   "Bearer " + token,
        "x-hdx-table":     "my_project.metrics",
        "x-hdx-transform": "otel_metrics",
    }),
)

meterProvider := sdkmetric.NewMeterProvider(
    sdkmetric.WithResource(res),
    sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter)),
)
defer meterProvider.Shutdown(ctx)

meter := meterProvider.Meter("my-service")
requests, err := meter.Int64Counter("requests", metric.WithUnit("1"))
requests.Add(ctx, 1, metric.WithAttributes(attribute.String("route", "/api")))

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
const { MeterProvider, PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-grpc');

const metricExporter = new OTLPMetricExporter({
  url: endpoint,
  metadata: routing('my_project.metrics', 'otel_metrics'),
});

const meterProvider = new MeterProvider({
  resource,
  readers: [new PeriodicExportingMetricReader({ exporter: metricExporter })],
});

const meter = meterProvider.getMeter('my-service');
const requests = meter.createCounter('requests', { unit: '1' });
requests.add(1, { route: '/api' });

The requests counter arrives as an OTLP sum metric. Sum is the metric type, not part of the name, so the metric is still called requests.

Different metric types carry their values in different fields, and the reference metrics transform maps those fields to table columns. For example, a sum's total maps to the value column, and a histogram's per-bucket counts map to the bucket_counts column.

Traces⚓︎

Create spans through a tracer provider with a batch processor that pushes to the OTLP span exporter.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint=HDX_ENDPOINT,
            headers=(
                ("authorization", f"Bearer {HDX_TOKEN}"),
                ("x-hdx-table", "my_project.traces"),
                ("x-hdx-transform", "otel_traces"),
            ),
        )
    )
)
tracer = provider.get_tracer("my-service")

with tracer.start_as_current_span("GET /api/users") as span:
    span.set_attribute("http.method", "GET")

provider.force_flush()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;

OtlpGrpcSpanExporter spanExporter =
    OtlpGrpcSpanExporter.builder()
        .setEndpoint(endpoint)
        .addHeader("authorization", "Bearer " + token)
        .addHeader("x-hdx-table", "my_project.traces")
        .addHeader("x-hdx-transform", "otel_traces")
        .build();

SdkTracerProvider tracerProvider =
    SdkTracerProvider.builder()
        .setResource(resource)
        .addSpanProcessor(BatchSpanProcessor.builder(spanExporter).build())
        .build();

Tracer tracer = tracerProvider.get("my-service");
Span span = tracer.spanBuilder("GET /api/users").startSpan();
try (Scope scope = span.makeCurrent()) {
    span.setAttribute("http.method", "GET");
} finally {
    span.end();
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import (
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

spanExporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint(endpoint),
    otlptracegrpc.WithHeaders(map[string]string{
        "authorization":   "Bearer " + token,
        "x-hdx-table":     "my_project.traces",
        "x-hdx-transform": "otel_traces",
    }),
)

tracerProvider := sdktrace.NewTracerProvider(
    sdktrace.WithResource(res),
    sdktrace.WithBatcher(spanExporter),
)
defer tracerProvider.Shutdown(ctx)

tracer := tracerProvider.Tracer("my-service")
_, span := tracer.Start(ctx, "GET /api/users")
span.SetAttributes(attribute.String("http.method", "GET"))
span.End()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
const { TracerProvider, BatchSpanProcessor } = require('@opentelemetry/sdk-trace');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const spanExporter = new OTLPTraceExporter({
  url: endpoint,
  metadata: routing('my_project.traces', 'otel_traces'),
});

const tracerProvider = new TracerProvider({
  resource,
  spanProcessors: [new BatchSpanProcessor({ exporter: spanExporter })],
});

const tracer = tracerProvider.getTracer('my-service');
const span = tracer.startSpan('GET /api/users');
span.setAttribute('http.method', 'GET');
span.end();

The reference traces transform derives duration_ms from each span's start and end times.

Configure with environment variables⚓︎

The OpenTelemetry SDKs read exporter settings from OTEL_EXPORTER_OTLP_* environment variables. Set these variables to configure the endpoint and headers outside application code, then create the exporter with no arguments.

Configure the Exporter with Environment Variables
1
2
3
export OTEL_EXPORTER_OTLP_ENDPOINT="https://hostname.hydrolix.live:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ${HDX_TOKEN},x-hdx-table=my_project.logs,x-hdx-transform=otel_logs"

OTEL_EXPORTER_OTLP_HEADERS applies to every signal. To send different signals to different tables, set the signal-specific variables instead:

  • OTEL_EXPORTER_OTLP_LOGS_HEADERS
  • OTEL_EXPORTER_OTLP_METRICS_HEADERS
  • OTEL_EXPORTER_OTLP_TRACES_HEADERS

Support for these variables differs by SDK:

  • Python and Go read them directly through the OTLP gRPC exporter.
  • Java reads them only when the application adds the opentelemetry-sdk-extension-autoconfigure module.
  • Node.js honors the endpoint and TLS variables, but the gRPC exporter doesn't reliably apply OTEL_EXPORTER_OTLP_HEADERS. Set routing headers with the metadata option instead.

Verify data arrived⚓︎

After running an example, query the target table to confirm the records arrived. Run the query through any Hydrolix query interface, such as the Hydrolix UI or the HTTP query API.

Confirm Data Arrived
1
SELECT count() FROM my_project.logs

A non-zero count confirms the SDK reached the endpoint and the transform accepted the data.

Authentication⚓︎

Both endpoints require a bearer token, which can be a service account token. Pass the token in the authorization header on each exporter, or through the OTEL_EXPORTER_OTLP_HEADERS environment variable.

Limitations⚓︎

Hydrolix doesn't create tables or transforms. Both must exist before the SDK sends data.