Skip to content

Vector

Vector is a high-performance observability data pipeline that can collect, transform, and route logs into Hydrolix.

Use this page to receive logs from a Vector application. See Logging Configuration for a description of the same software running inside a Hydrolix cluster.

For Vector documentation, see vector.dev.

Prerequisites⚓︎

A running Hydrolix deployment is required. See deployment instructions for cloud setup steps.

Collect the following values from the active Hydrolix cluster:

Item Description Example value How to obtain
Org ID Unique identifier for the Hydrolix organization bc3f041b-ea52-45e1-b9d5-41819c465854 Use the Org API
Project name and ID Logical namespace for the destination table hdx_project_for_vector / c2445da3-ec63-42be-9f12-f104f9656f4c Create a project
Table name and ID Destination for incoming Vector data hdx_table_for_vector / 798dfcf4-7560-4b24-1337-29017619baa6 Create a table
Username/password Optional credentials for basic authentication

In version 5.4 and later, the Bearer token method or use of a service account is preferred		 myemail@address.com / MyPassword Create a user
Bearer token Preferred authentication method for Streaming API

In version 5.4 and later, these tokens are revocable, and may have custom TTL settings		 eyXrxkzoN2fRiiKpnV... Generate a token

Required components⚓︎

Integration requires three main components:

Component Purpose
Data source Log data input. Example: Vector’s demo logs.
Vector instance Pipeline for collecting and routing data
Hydrolix cluster Destination for ingested log data

Vector configuration includes:

  • Source: Collects log data.
  • Transform (optional): Restructures log data. This is not the same as Hydrolix transforms.
  • Sink: Defines the destination. For Hydrolix, this is the streaming API endpoint.

Install Vector⚓︎

Install Vector using the official installation script.

Verify installation:

1
vector --version

1
vector 0.40.1 <architecture details>

Configure the Vector sink⚓︎

Create a config file named vector-hdx.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
sources:
  generate_demo_logs:
    type: "demo_logs"
    format: "syslog"
    count: "100"

sinks:
  hydrolix:
    type: http
    inputs:
      - generate_demo_logs
    uri: https://{myhost}.hydrolix.live/ingest/event
    encoding:
      codec: json
    compression: gzip
    headers:
      X-HDX-Table: <project_name>.<table_name>
      X-HDX-Transform: <transform_id>
      Authorization: Bearer <token>

Replace the placeholder values with the actual project, table, transform ID, and token.

This configuration inserts data into your running Hydrolix cluster using the HTTP Streaming API. The headers provided correspond to those required by the Streaming API endpoint (https://{myhost}.hydrolix.live/ingest/event) which specify the table and project into which it should insert the log data routed through Vector.

Create a Hydrolix transform⚓︎

Use the Transform API to define a schema for incoming log data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
  "name": "hydrolix_transform_for_vector_data",
  "description": "Transform sample log data from Vector",
  "type": "json",
  "settings": {
    "is_default": true,
    "compression": "none",
    "output_columns": [
      { "name": "host", "datatype": { "type": "string", "index": true }},
      { "name": "message", "datatype": { "type": "string", "index": true }},
      { "name": "source_type", "datatype": { "type": "string", "index": true }},
      {
        "name": "timestamp",
        "datatype": {
          "type": "datetime",
          "format": "2006-01-02T15:04:05.999999Z",
          "resolution": "ms",
          "primary": true
        }
      }
    ]
  }
}

To submit this request using curl:

1
2
3
4
5
6
curl --request POST \
  --url https://{myhost}.hydrolix.live/config/v1/orgs/$ORG_ID/projects/$PROJECT_ID/tables/$TABLE_ID/transforms/ \
  --header 'accept: application/json' \
  --header "authorization: Bearer $TOKEN" \
  --header 'content-type: application/json' \
  --data @transform.json

The returned UUID becomes the value for X-HDX-Transform.

Run Vector⚓︎

Run Vector with this configuration:

1
vector --config vector-hdx.yaml

1
vector -v --config vector-hdx.yaml

Expected logs⚓︎

Example output from a healthy pipeline:

1
2
3
4
5
6
7
INFO vector::app: Log level is enabled. level="debug"
DEBUG vector::topology::builder: Building new source. component=generate_demo_logs
DEBUG vector::topology::builder: Building new sink. component=hydrolix
INFO vector: Vector has started. version="0.40.1" arch="x86_64"
...
DEBUG http_client: Sending HTTP request. uri=https://{myhost}.hydrolix.live/ingest/event method=POST
DEBUG http_client: HTTP response. status=200 OK

Add custom sources⚓︎

Additional sources can be added to the sources and referenced in the inputs for the Hydrolix sink:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
sources:
  custom-source-1:
    type: "<type>"
    format: "<format>"
  custom-source-2:
    type: "<type>"
    format: "<format>"

sinks:
  hydrolix:
    type: http
    inputs:
      - generate_demo_logs
      - custom-source-1
      - custom-source-2
    uri: https://{myhost}.hydrolix.live/ingest/event
    encoding:
      codec: json
    compression: gzip
    headers:
      X-HDX-Table: <project_name>.<table_name>
      X-HDX-Transform: <transform_id>
      Authorization: Bearer <token>

Each source must align with the transform schema used by the sink. For sources with different formats, create additional transforms and configure the headers accordingly.