Kibana Manual Installation

Learn how to set up Kibana, Kibana Gateway, and Elasticsearch manually. Alternatively, Hydrolix provides these services pre-configured. See Kibana Automatic Installation for more details.

What is Kibana Gateway?

Kibana Gateway is a lightweight database proxy that enables interoperability between a front end such as Kibana and multiple back end datastores such as Hydrolix and Elasticsearch. It provides this interoperability by acting as a SQL translation layer that decouples your dashboarding and ingestion layers from your datastore. Key use cases for Kibana Gateway include:

• Translation layer for queries during a datastore platform migration, removing the need to alter application code or refactor SQL queries to maintain datastore compatibility
• Using Kibana with Hydrolix instead of Elasticsearch
• Simultaneous querying of multiple back end datastores

Container and pod names

Hydrolix acquired the database proxy software from Quesma. The Kubernetes pod, container, and tunable are named quesma since the cluster integration predated the software acquisition in 2025 September. The product is called Kibana Gateway.

Installation guide for existing environments

If you have an existing system into which you want to incorporate Kibana Gateway and Hydrolix, there are various scenarios and their instructions described in the Kibana Gateway installation guide.

Installation guide from scratch

This tutorial covers standing up a Kibana front end, a Kibana Gateway query and ingest routing and translation layer, and an Elasticsearch back end datastore. This setup integrates with your existing Hydrolix cluster and can simulate a multi-datastore architecture. Ingest is routed to Elasticsearch, while queries are routed to both Hydrolix and Elasticsearch.

Before you begin

Ensure you have the following:

• A running Hydrolix deployment. Follow the instructions for your preferred cloud vendor if you have not deployed Hydrolix yet.
• Git. Install the git CLI or GitHub Desktop.
• Docker v20.10 or higher. Install Docker Desktop.

Gather the following information from your running Hydrolix cluster:

Item	Description	Example values
Hydrolix URL + ClickHouse port	The URL of your Hydrolix cluster appended with the ClickHouse Native protocol SSL/TLS port.	https://{myhostname}.hydrolix.live:9440
Project name	The project name in Hydrolix corresponds to the database name configured for Kibana Gateway. It is the logical namespace for your tables. For instructions on creating a project, see Creating a project with the API.	Name: project_name
Table name(s)	The names of the tables you want to expose within the Kibana interface through Kibana Gateway configuration. For instructions on creating a table, see Create a table with the API.	Name:table_name0, table_name1
Credentials	The username and password for your Hydrolix cluster account.	username: user@domain.com password: correcthorsebatterystaple

Configure containers with Docker Compose

These steps are similar to the Kibana Gateway quick start demo setup for deploying Kibana and Elasticsearch. Use the Docker Compose file from those directions but omit the ClickHouse server and replace it with your running Hydrolix cluster.

The following also incorporates the Hydrolix-specific instructions to configure Kibana Gateway to route queries to your Hydrolix cluster rather than a ClickHouse server.

Ingest currently not supported

Kibana Gateway currently supports routing queries to Hydrolix. Ingest to Hydrolix using Kibana Gateway is not supported.

1. Clone the Kibana Gateway repository:

   git clone https://github.com/QuesmaOrg/quesma.git
   

2. Navigate to the working directory containing the Docker Compose file:

   cd quesma/examples/kibana-sample-data
   

3. Edit docker-compose.yml in this directory and remove or comment out the clickhouse container:

   services:
   {...}
     #clickhouse:
     #  user: 'default', no password
     #  image: clickhouse/clickhouse-server:23.12.2.59-alpine
     #  ports:
     #    - "8123:8123"
     #    - "9000:9000"
     #  healthcheck:
     #    test: wget --no-verbose --tries=1 --spider http://clickhouse:8123/ping || exit 1
     #    interval: 1s
     #    timeout: 1s
     #    start_period: 1m
   

   Save your changes.

With these steps completed, executing this Docker Compose file locally deploys:

• A Kibana Gateway container (not yet configured to communicate with your Hydrolix cluster)
• An Elasticsearch datastore and query engine
• A Kibana data visualization interface
• Three sample data sets that load data into Elasticsearch using two containers (log-generator and kibana-sidecar)

Configure Kibana Gateway

Configure Kibana Gateway to route queries to your running Hydrolix cluster and remove any references to the ClickHouse cluster. For more information about configuration, see the Configuration Primer.

1. Navigate to the directory containing the Kibana Gateway configuration:

   cd quesma/config
   

2. Edit local-dev.yaml and add the following to backendConnectors:

     - name: my-hydrolix-instance
       type: hydrolix
       config:
         url: https://{hydrolix_host}:9440
         user: {username}
         password: {password}
         database: {hydrolix_project_name}
   

   Replace {hydrolix_host}, {username}, {password}, and {hydrolix_project_name} with your Hydrolix cluster information. This provides connection information that enables Kibana Gateway to communicate with your running Hydrolix cluster. The name you specify for the Hydrolix backend connector is for referencing it within this configuration file only. It does not have to be the name of your running Hydrolix cluster.

   Determining your database name

   The project name in Hydrolix corresponds to the database name for Kibana Gateway.

3. Remove or comment out ClickHouse from backendConnectors:

   backendConnectors:
   {...}
     #- name: my-clickhouse-data-source
     #  type: clickhouse-os
     #  config:
     #    url: "clickhouse://clickhouse:9000"
     #    adminUrl: "http://localhost:8123/play"
   

4. In the pipelines section, replace mentions of my-clickhouse-data-source with my-hydrolix-instance. This configuration change ensures that Kibana Gateway routes queries to both the Hydrolix and Elasticsearch back ends. For consistency, you can also rename the pipelines:

   • my-pipeline-elasticsearch-query-clickhouse → my-pipeline-elasticsearch-query-hydrolix
   • my-pipeline-elasticsearch-ingest-to-clickhouse → my-pipeline-elasticsearch-ingest-to-elastic

5. Replace all other instances of my-clickhouse-data-source with my-minimal-elasticsearch. There should be six altogether configured for the following indices:

   • kibana_sample_data_ecommerce
   • kibana_sample_data_flights
   • logs-generic-default

   These indices are specified for the following processors:

   • my-query-processor
   • my-ingest-processor

   This configuration tells Kibana Gateway to ingest data into and query those indices solely within Elasticsearch.

6. Add your Hydrolix table(s) to processors.my-query-processor.config.indexes with target my-hydrolix-instance. For example, to use tables table_name0 and table_name1:

   processors:
     - name: my-query-processor
       type: quesma-v1-processor-query
       config:
         indexes:
           table_name0:
             target: [ my-hydrolix-instance ]
           table_name1:
             target: [ my-hydrolix-instance ]
   

   This tells Kibana Gateway to query the data stored in those tables in your Hydrolix cluster.

7. Save your changes.

Deploy Kibana, Elasticsearch, and Kibana Gateway

Deploy your containers locally:

docker compose -f docker-compose.yml up

Verify functionality

Kibana

Access the Kibana UI at localhost:5601:

Kibana Gateway

Access the Kibana Gateway admin panel at localhost:9999:

The admin panel does not require authentication by default and can be disabled in docker-compose.yaml by commenting out or removing port forwarding for port 9999:

services:
  quesma:
    ports:
      #- "9999:9999"
      - "9200:8080"

View your tables, including Hydrolix tables, at localhost:9999/schemas.

Kibana Gateway container logs to stderr

If you encounter problems with your Kibana Gateway container and want to search its logs, combine both stderr and stdout to the stdout stream with 2>&1:

docker logs {quesma_container_id} 2>&1 | grep "ERROR"

Otherwise, grep does not function on the logs from the Kibana Gateway container.

Create a Data View in Kibana for your Hydrolix data

To view your Hydrolix tables in Kibana, create Data Views for tables (indexes). Follow these directions to create one.