Cribl

Overview

Cribl is a platform for telemetry data management for IT and security teams.

Use Cribl with Hydrolix as storage to enable high-speed, cost-efficient ingestion and querying of massive telemetry datasets and optimize long-term analytics.

Before you begin

Gather the following information prior to integrating Cribl with Hydrolix.

Item	Description	Example value	How to obtain this information
Cluster hostname	This is the hostname of your Hydrolix cluster.	{hostname}.hydrolix.live	The value of hydrolix_url in your hydrolixcluster.yaml without the preceding https://.
Hydrolix Table	The data routing destination to your Cribl instance. Name the table in the format project_name.table_name.	cribl_project.cribl_table	Follow these instructions to create a table.

Register a Hydrolix Transform

Register a Hydrolix transform. The transform determines how your Cribl data will be mapped onto your Hydrolix table.

See Publishing Your Transform to create and publish a transform. Use this Cribl transform.

You have the option of registering a transform through the UI using the project name, table name, and output_columns property. Alternatively, you can use the API which requires the project ID, table ID, and the entirety of the Cribl transform json.

The default transform combines all incoming fields into a single field called _raw except for the timestamp which is in a field called _time. To improve the structure of various data types stored in Hydrolix from Cribl, reference the Hydrolix Write Transforms documentation.

Configure Cribl

Configure a pipeline and webhook destination in Cribl pointing to your cluster, then route your source data through the pipeline to the destination.

1. Create a hydrolix pipeline in Cribl using the cribl-hydrolix-pipeline.json file.
2. Create a webhook destination in Cribl with the following settings in Configure > General Settings:

Setting	Value
Output ID	hydrolix
Webhook URL	https://{hostname}.hydrolix.live/ingest/event?table={my_project.my_table}&token=ctoken
Method	POST
Format	JSON Array
Backpressure behavior	Block

And these settings in Configure > Authentication

Setting	Value
Authentication Type	Basic
Username	{your_hydrolix_username}
Password	{your_hydrolix_password}

1. Connect your source data to the hydrolix pipeline.

2. Test that data is flowing by querying the destination table in Cribl. For example, you can try running:

   select _time, _raw from my_project.my_table limit 10
   

Visit Write Your First Query for more information on querying with Cribl.

Dashboards

Hydrolix offers a starter Grafana dashboard to show data flow and high-level statistics. Import this dashboard JSON into your Grafana instance:

cribl-default-dashboard.json