Journald Integration
Vector can help you send logs into Hydrolix with HTTP output and JSON formatting.
In this example we'll:
- Install vector on an Ubuntu server.
- Get the logs from Journald.
- Ingest the logs into Hydrolix.
Install vector on Ubuntu
Follow the vector installation guide. Run the following commands to install the apt package:
curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' | sudo -E bash
sudo apt-get install vector
This also launches Vector as a service on your machine.
Setting Hydrolix table and transform for Journald
Hydrolix requires the table and the schema to ingest your Journald logs. Use the following example to create the project and table.
To use this example you can refer to this article on how to setup VSCode.
### Global variable to replace with your own needs
@host = $customer
@projectname = sample
@tablename = journald
@transformname = journald_transform
@username = "YYYYYYY"
@password = "XXXXXXXX"
#### DO NOT MODIFY BELOW ####
@base_url = https://{{host}}.hydrolix.live/config/v1/
@post_endpoint = https://{{host}}.hydrolix.live/ingest/event
@sql_endpoint = https://{{host}}.hydrolix.live/query/
### Login authentication get access token and UUID Org variable
# @name login
POST {{base_url}}login
Content-Type: application/json
{
"username": {{username}},
"password": {{password}}
}
### Store, parse the login response body to store the access token and organization id
@access_token = {{login.response.body.auth_token.access_token}}
@org_id = {{login.response.body.orgs[0].uuid}}
### Create a new project using the variable {{projectname}}
# @name new_project
POST {{base_url}}orgs/{{org_id}}/projects/
Authorization: Bearer {{access_token}}
Content-Type: application/json
{
"name": "{{projectname}}",
"org": "{{org_id}}"
}
### Store, parse project ID from response
@projectid = {{new_project.response.body.uuid}}
### Create a new table named {{tablename}} in the {{projectname}}
# @name new_table
POST {{base_url}}orgs/{{org_id}}/projects/{{projectid}}/tables/
Authorization: Bearer {{access_token}}
Content-Type: application/json
{
"name": "{{tablename}}",
"project": "{{projectid}}",
"description": "Journald Logs from Vector",
"settings": {
"merge": {
"enabled": true
}
}
}
### Store, parse table ID from response
@tableid = {{new_table.response.body.uuid}}
#### Creates a a transform for the json format and upload to our table
# @name new_transform
POST {{base_url}}orgs/{{org_id}}/projects/{{projectid}}/tables/{{tableid}}/transforms/
Authorization: Bearer {{access_token}}
Content-Type: application/json
{
"name": "{{transformname}}",
"description": "Transform JournalD logs from Vector",
"table": "{{tableid}}",
"type": "json",
"settings": {
"is_default": true,
"compression": "gzip",
"format_details": {
"flattening": {
"active": false
}
},
"output_columns": [
{
"name": "PRIORITY",
"datatype": {
"type": "uint8",
"index": true
}
},
{
"name": "SYSLOG_FACILITY",
"datatype": {
"type": "uint8",
"index": true
}
},
{
"name": "SYSLOG_IDENTIFIER",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "SYSLOG_TIMESTAMP",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_AUDIT_LOGINUID",
"datatype": {
"type": "uint64",
"index": true
}
},
{
"name": "_AUDIT_SESSION",
"datatype": {
"type": "uint64",
"index": true
}
},
{
"name": "_BOOT_ID",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_CAP_EFFECTIVE",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_CMDLINE",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_COMM",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_EXE",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_GID",
"datatype": {
"type": "uint64",
"index": true
}
},
{
"name": "_MACHINE_ID",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_PID",
"datatype": {
"type": "uint64",
"index": true
}
},
{
"name": "_SELINUX_CONTEXT",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_SOURCE_REALTIME_TIMESTAMP",
"datatype": {
"type": "epoch",
"index": true,
"format": "us",
"resolution": "ms"
}
},
{
"name": "_SYSTEMD_CGROUP",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_SYSTEMD_INVOCATION_ID",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_SYSTEMD_OWNER_UID",
"datatype": {
"type": "uint64",
"index": true
}
},
{
"name": "_SYSTEMD_SESSION",
"datatype": {
"type": "uint64",
"index": true
}
},
{
"name": "_SYSTEMD_SLICE",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_SYSTEMD_UNIT",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_SYSTEMD_USER_SLICE",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_TRANSPORT",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "_UID",
"datatype": {
"type": "uint64",
"index": true
}
},
{
"name": "__MONOTONIC_TIMESTAMP",
"datatype": {
"type": "uint64",
"index": true
}
},
{
"name": "__REALTIME_TIMESTAMP",
"datatype": {
"type": "epoch",
"index": true,
"format": "us",
"resolution": "ms"
}
},
{
"name": "host",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "message",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "source_type",
"datatype": {
"type": "string",
"index": true
}
},
{
"name": "timestamp",
"datatype": {
"type": "datetime",
"format": "2006-01-02T15:04:05.999999Z",
"resolution": "ms",
"primary": true
}
}
]
}
}
Configuring Vector
Now you can set up Vector. Edit the configuration found in /etc/vector/vector.toml:
[sources.journal_source]
type = "journald"
[sinks.hydrolix]
type = "http"
inputs = ["journal_source"]
uri = "https://$hostname.hydrolix.live/ingest/event"
encoding = "json"
compression = "gzip"
[sinks.hydrolix.headers]
X-HDX-Table = "$project.$table"
X-HDX-Transform = "$transform"
Replace the following in the above example configuration to connect to your cluster:
- hostname
- project
- table
- transform name.
Updated 5 months ago