Journald

Vector can help you send logs into Hydrolix with HTTP output and JSON formatting.

In this example we'll:

1. Install vector on an Ubuntu server.
2. Get the logs from Journald.
3. Ingest the logs into Hydrolix.

Install vector on Ubuntu

Follow the vector installation guide. Run the following commands to install the apt package:

curl -1sLf   'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' | sudo -E bash
sudo apt-get install vector

This also launches Vector as a service on your machine.

Set Hydrolix table and transform for Journald

Hydrolix requires the table and the schema to ingest your Journald logs. Use the following example to create the project and table. To use this example you can refer to this article on how to setup VSCode.

Global variable to replace with your own needs

@host = $customer
@projectname = sample
@tablename = journald
@transformname = journald_transform
@username = "YYYYYYY"
@password = "XXXXXXXX"

Don't modify these values

@base_url = https://{{host}}.hydrolix.live/config/v1/
@post_endpoint = https://{{host}}.hydrolix.live/ingest/event
@sql_endpoint = https://{{host}}.hydrolix.live/query/

Login authentication get access token and UUID Org variable

## @name login
POST {{base_url}}login
Content-Type: application/json

{
    "username": {{username}},
    "password": {{password}}
}

Store, parse the login response body to store the access token and organization id

@access_token = {{login.response.body.auth_token.access_token}}
@org_id = {{login.response.body.orgs[0].uuid}}

Create a new project using the variable {{projectname}}

## @name new_project
POST  {{base_url}}orgs/{{org_id}}/projects/
Authorization: Bearer {{access_token}}
Content-Type: application/json

{
    "name": "{{projectname}}",
    "org": "{{org_id}}"
}

Store, parse project ID from response

@projectid = {{new_project.response.body.uuid}}

Create a new table named {{tablename}} in the {{projectname}}

## @name new_table
POST  {{base_url}}orgs/{{org_id}}/projects/{{projectid}}/tables/
Authorization: Bearer {{access_token}}
Content-Type: application/json

{
    "name": "{{tablename}}",
    "project": "{{projectid}}",
    "description": "Journald Logs from Vector",
    "settings": {
        "merge": {
            "enabled": true
        }
    }
}

Store, parse table ID from response

@tableid = {{new_table.response.body.uuid}}

Creates a a transform for the json format and upload to our table

## @name new_transform
POST {{base_url}}orgs/{{org_id}}/projects/{{projectid}}/tables/{{tableid}}/transforms/
Authorization: Bearer {{access_token}}
Content-Type: application/json

{
    "name": "{{transformname}}",
    "description": "Transform JournalD logs from Vector",
    "table": "{{tableid}}",
    "type": "json",
    "settings": {
        "is_default": true,
        "compression": "gzip",
        "format_details": {
            "flattening": {
                "active": false
            }
        },
        "output_columns": [
            {
                "name": "PRIORITY",
                "datatype": {
                    "type": "uint8",
                    "index": true
                }
            },
            {
                "name": "SYSLOG_FACILITY",
                "datatype": {
                    "type": "uint8",
                    "index": true
                }
            },
            {
                "name": "SYSLOG_IDENTIFIER",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "SYSLOG_TIMESTAMP",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_AUDIT_LOGINUID",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_AUDIT_SESSION",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_BOOT_ID",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_CAP_EFFECTIVE",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_CMDLINE",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_COMM",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_EXE",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_GID",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_MACHINE_ID",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_PID",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_SELINUX_CONTEXT",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_SOURCE_REALTIME_TIMESTAMP",
                "datatype": {
                    "type": "epoch",
                    "index": true,
                    "format": "us",
                    "resolution": "ms"
                }
            },
            {
                "name": "_SYSTEMD_CGROUP",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_INVOCATION_ID",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_OWNER_UID",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_SESSION",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_SLICE",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_UNIT",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_SYSTEMD_USER_SLICE",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_TRANSPORT",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "_UID",
                "datatype": {
                    "type": "uint64",
                    "index": true
                }
            },
            {
                "name": "__MONOTONIC_TIMESTAMP",
                "datatype": {
                    "type": "uint64",
                    "index": true

                }
            },
            {
                "name": "__REALTIME_TIMESTAMP",
                "datatype": {
                    "type": "epoch",
                    "index": true,
                    "format": "us",
                    "resolution": "ms"
                }
            },
            {
                "name": "host",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "message",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "source_type",
                "datatype": {
                    "type": "string",
                    "index": true
                }
            },
            {
                "name": "timestamp",
                "datatype": {
                    "type": "datetime",
                    "format": "2006-01-02T15:04:05.999999Z",
                    "resolution": "ms",
                    "primary": true
                }
            }
        ]
    }
}

Configure Vector

Now you can set up Vector. Edit the configuration found in /etc/vector/vector.toml:

[sources.journal_source]
type = "journald"

[sinks.hydrolix]
type = "http"
inputs = ["journal_source"]
uri = "https://$hostname.hydrolix.live/ingest/event"
encoding = "json"
compression = "gzip"

[sinks.hydrolix.headers]
X-HDX-Table = "$project.$table"
X-HDX-Transform = "$transform"

Replace the following in the above example configuration to connect to your cluster:

• hostname
• project
• table
• transform name