Cribl Integration
Overview
Cribl is a platform for telemetry data management for IT and security teams.
Use Cribl with Hydrolix as storage to enable high-speed, cost-efficient ingestion and querying of massive telemetry datasets and optimize long-term analytics.
Before you begin
Gather the following information prior to integrating Cribl with Hydrolix.
| Item | Description | Example value | How to obtain this information |
|---|---|---|---|
| Cluster hostname | This is the hostname of your Hydrolix cluster. | {hostname}.hydrolix.live | The value of hydrolix_url in your hydrolixcluster.yaml without the preceding https://. |
| Hydrolix Table | The data routing destination to your Cribl instance. Name the table in the format project_name.table_name. | cribl_project.cribl_table | Follow these instructions to create a table. |
Register a Hydrolix Transform
Register a Hydrolix transform. The transform determines how your Cribl data will be mapped onto your Hydrolix table.
See Publishing Your Transform to create and publish a transform. Use this Cribl transform.
You have the option of registering a transform through the UI using the project name, table name, and output_columns property. Alternatively, you can use the API which requires the project ID, table ID, and the entirety of the Cribl transform json.
The default transform combines all incoming fields into a single field called _raw except for the timestamp which is in a field called _time. To improve the structure of various data types stored in Hydrolix from Cribl, reference the Hydrolix Write Transforms documentation.
Configure Cribl
Configure a pipeline and webhook destination in Cribl pointing to your cluster, then route your source data through the pipeline to the destination.
- Create a
hydrolixpipeline in Cribl using the cribl-hydrolix-pipeline.json file. - Create a webhook destination in Cribl with the following settings in Configure > General Settings:
| Setting | Value |
|---|---|
| Output ID | hydrolix |
| Webhook URL | https://{hostname}.hydrolix.live/ingest/event?table={my_project.my_table}&token=ctoken |
| Method | POST |
| Format | JSON Array |
| Backpressure behavior | Block |
And these settings in Configure > Authentication
| Setting | Value |
|---|---|
| Authentication Type | Basic |
| Username | {your_hydrolix_username} |
| Password | {your_hydrolix_password} |
- Connect your source data to the
hydrolixpipeline. - Test that data is flowing by querying the destination table in Cribl. For example, you can try running:
select _time, _raw from my_project.my_table limit 10
Visit Write Your First Query for more information on querying with Cribl.
Dashboards
Hydrolix offers a starter Grafana dashboard to show data flow and high-level statistics. Import this dashboard JSON into your Grafana instance:
Updated 8 days ago