Skip to content

Logstash

Logstash collects, transforms, and ships events. Its http output plugin can send them to the Hydrolix HTTP Stream API. The examples on this page use Elastic Common Schema (ECS) fields, and the same steps work for any JSON events.

Prerequisites⚓︎

Create a transform for ECS data⚓︎

Logstash sends ECS fields as nested JSON objects, such as {"log": {"level": "info"}}. A transform reads a nested field with from_json_pointers and a JSON Pointer path, such as /log/level.

This transform maps the ECS fields most events carry. is_default: true makes it the table's default transform. To publish it to the table, see Publish a Transform.

ECS Transform
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
  "name": "ecs_json",
  "type": "json",
  "settings": {
    "is_default": true,
    "compression": "none",
    "output_columns": [
      { "name": "timestamp", "datatype": { "type": "datetime", "primary": true, "format": "2006-01-02T15:04:05.999Z", "resolution": "ms", "source": { "from_input_field": "@timestamp" } } },
      { "name": "message", "datatype": { "type": "string", "source": { "from_input_field": "message" } } },
      { "name": "log_level", "datatype": { "type": "string", "source": { "from_json_pointers": ["/log/level"] } } },
      { "name": "host_name", "datatype": { "type": "string", "source": { "from_json_pointers": ["/host/name"] } } },
      { "name": "service_name", "datatype": { "type": "string", "source": { "from_json_pointers": ["/service/name"] } } },
      { "name": "event_dataset", "datatype": { "type": "string", "source": { "from_json_pointers": ["/event/dataset"] } } },
      { "name": "ecs_version", "datatype": { "type": "string", "source": { "from_json_pointers": ["/ecs/version"] } } }
    ]
  }
}

To store more fields, add an output column for each. Find field names and types in the ECS field reference.

Configure the Logstash http output⚓︎

Add an http output to the pipeline. Replace my_project.ecs_logs and ecs_json with the table and transform names.

Logstash Pipeline Output
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
output {
  http {
    url         => "https://${HDX_HOSTNAME}/ingest/event"
    http_method => "post"
    format      => "json_batch"
    headers     => {
      "Authorization"   => "Bearer ${HDX_TOKEN}"
      "X-Hdx-Table"     => "my_project.ecs_logs"
      "X-Hdx-Transform" => "ecs_json"
    }
  }
}
  • format => "json_batch" sends each batch of events as one JSON array in one request.
  • ${HDX_HOSTNAME} and ${HDX_TOKEN} come from environment variables or the Logstash keystore. Keep the token in the keystore, not in the pipeline file.

For every setting, see the http output plugin reference.

Logstash drops batches that Hydrolix rejects

By default, the http output retries only 429, 500, 502, 503, and 504 responses. A batch rejected with any other status, such as 401 for a bad token, is dropped. Logstash logs Encountered non-2xx HTTP code, with the status and the response body.

Verify⚓︎

Query the table after a minute or two. Right after the table or transform is created, requests can fail with 400 no table ‘x’ found or unknown transform ‘x’ for a short time. For causes and fixes, see Ingest returns HTTP 400.

Count Recent ECS Events
1
2
3
4
SELECT log_level, service_name, count() AS events
FROM my_project.ecs_logs
WHERE timestamp > NOW() - INTERVAL 1 HOUR
GROUP BY log_level, service_name

Send Beats data⚓︎

Beats agents, such as Filebeat, can reach Hydrolix two ways:

  • Through Logstash: send Beats output to the Logstash beats input, then use the http output on this page. To configure Filebeat, see Configure the Logstash output.
  • Through Kafka: send Beats output to a Kafka topic, and ingest the topic with a Hydrolix Kafka source. To set up the source, see Kafka.

Troubleshoot⚓︎

Something not working?

Start at Troubleshooting Symptoms and Fixes. Find the symptom, confirm it with the signal listed there, and follow the link to the procedure.

Requests fail with an HTTP error⚓︎

Required access: User

Confirm: The status code and response body name the problem: 401: Unauthorized (the Authorization header), 403 with missing or bad auth token (a table access token), 400 with a message such as no table ‘x’ found or unhandled content-type ‘x’ (the table, transform, or format), or 404 with not found (the path). No response at all, such as a timeout, means the sender can't reach the cluster.

The status code shows which part of the request was refused.

Fix: correct the part the status code points to.

  • 401: the Authorization header is missing, malformed, or not valid, such as an expired token. Get a new token. An integration that runs unattended should use a service account token. See Manage Service Accounts: Service account tokens.
  • 403: the table requires a table access token, in addition to the Authorization header. Send one from its token list in the X-Hdx-Token header or the token query string parameter. See Table Access Tokens.
  • 400: the response body names the problem. See Ingest returns HTTP 400.
  • 404: send requests to https://hostname.hydrolix.live/ingest/event.
  • No response: check the hostname, and that the sender can reach the cluster over HTTPS.