Vector Integration

Vector is a high-performance observability data pipeline that can collect, transform, and route logs into Hydrolix.

Use this page to receive logs from a Vector application. See Logging Configuration for a description of the same software running inside a Hydrolix cluster.

For Vector documentation, see vector.dev.

Prerequisites

A running Hydrolix deployment is required. See deployment instructions for cloud setup steps.

Collect the following values from the active Hydrolix cluster:

Item	Description	Example value	How to obtain
Org ID	Unique identifier for the Hydrolix organization	`bc3f041b-ea52-45e1-b9d5-41819c465854`	Use the Org API
Project name and ID	Logical namespace for the destination table	`hdx_project_for_vector` / `c2445da3-ec63-42be-9f12-f104f9656f4c`	Create a project
Table name and ID	Destination for incoming Vector data	`hdx_table_for_vector` / `798dfcf4-7560-4b24-1337-29017619baa6`	Create a table
Username/password	Optional credentials for basic authentication In version 5.4 and later, the Bearer token method or use of a service account is preferred	`[email protected]` / `MyPassword`	Create a user
Bearer token	Preferred authentication method for Streaming API In version 5.4 and later, these tokens are revocable, and may have custom TTL settings	`eyXrxkzoN2fRiiKpnV...`	Generate a token

Required components

Integration requires three main components:

Component	Purpose
Data source	Log data input. Example: Vector’s demo logs.
Vector instance	Pipeline for collecting and routing data
Hydrolix cluster	Destination for ingested log data

Vector configuration includes:

Source: Collects log data.
Transform (optional): Restructures log data. This is not the same as Hydrolix transforms.
Sink: Defines the destination. For Hydrolix, this is the streaming API endpoint.

Install Vector

Install Vector using the official installation script.

Verify installation:

vector --version

Expected output:

vector 0.40.1 <architecture details>

Configure the Vector sink

Create a config file named vector-hdx.yaml:

sources:
  generate_demo_logs:
    type: "demo_logs"
    format: "syslog"
    count: "100"

sinks:
  hydrolix:
    type: http
    inputs:
      - generate_demo_logs
    uri: https://{myhost}.hydrolix.live/ingest/event
    encoding:
      codec: json
    compression: gzip
    headers:
      X-HDX-Table: <project_name>.<table_name>
      X-HDX-Transform: <transform_id>
      Authorization: Bearer <token>

Replace the placeholder values with the actual project, table, transform ID, and token.

This configuration inserts data into your running Hydrolix cluster using the HTTP Streaming API. The headers provided correspond to those required by the Streaming API endpoint (https://{myhost}.hydrolix.live/ingest/event) which specify the table and project into which it should insert the log data routed through Vector.

Create a Hydrolix transform

Use the Transform API to define a schema for incoming log data:

{
  "name": "hydrolix_transform_for_vector_data",
  "description": "Transform sample log data from Vector",
  "type": "json",
  "settings": {
    "is_default": true,
    "compression": "none",
    "output_columns": [
      { "name": "host", "datatype": { "type": "string", "index": true }},
      { "name": "message", "datatype": { "type": "string", "index": true }},
      { "name": "source_type", "datatype": { "type": "string", "index": true }},
      {
        "name": "timestamp",
        "datatype": {
          "type": "datetime",
          "format": "2006-01-02T15:04:05.999999Z",
          "resolution": "ms",
          "primary": true
        }
      }
    ]
  }
}

To submit this request using curl:

curl --request POST \
  --url https://{myhost}.hydrolix.live/config/v1/orgs/$ORG_ID/projects/$PROJECT_ID/tables/$TABLE_ID/transforms/ \
  --header 'accept: application/json' \
  --header "authorization: Bearer $TOKEN" \
  --header 'content-type: application/json' \
  --data @transform.json

The returned UUID becomes the value for X-HDX-Transform.

Run Vector

Run Vector with this configuration:

vector --config vector-hdx.yaml

For verbose logging:

vector -v --config vector-hdx.yaml

Expected logs

Example output from a healthy pipeline:

INFO vector::app: Log level is enabled. level="debug"
DEBUG vector::topology::builder: Building new source. component=generate_demo_logs
DEBUG vector::topology::builder: Building new sink. component=hydrolix
INFO vector: Vector has started. version="0.40.1" arch="x86_64"
...
DEBUG http_client: Sending HTTP request. uri=https://{myhost}.hydrolix.live/ingest/event method=POST
DEBUG http_client: HTTP response. status=200 OK

Add custom sources

Additional sources can be added to the sources and referenced in the inputs for the Hydrolix sink:

sources:
  custom-source-1:
    type: "<type>"
    format: "<format>"
  custom-source-2:
    type: "<type>"
    format: "<format>"

sinks:
  hydrolix:
    type: http
    inputs:
      - generate_demo_logs
      - custom-source-1
      - custom-source-2
    uri: https://{myhost}.hydrolix.live/ingest/event
    encoding:
      codec: json
    compression: gzip
    headers:
      X-HDX-Table: <project_name>.<table_name>
      X-HDX-Transform: <transform_id>
      Authorization: Bearer <token>

Each source must align with the transform schema used by the sink. For sources with different formats, create additional transforms and configure the headers accordingly.