Skip to content

Design and Configuration

Kibana Gateway uses a pipeline-based architecture. Each pipeline connects frontend connectors, processors, and backend connectors to route queries between Kibana and your data stores.

In a Hydrolix cluster, the operator deploys and configures Kibana Gateway for you. To install the integration and choose which Hydrolix projects and tables Kibana can query, see Set up Kibana.

Kibana Gateway is query-only

Kibana Gateway exposes existing data for querying. It doesn't ingest data into Hydrolix. See Known Limitations.

Design: pipelines⚓︎

Kibana Gateway is a set of pipelines for processing incoming requests.

A pipeline consists of:

  • Frontend connectors: Receive incoming requests and send responses. They define the API that Kibana Gateway exposes, for example, the Elasticsearch REST API.
  • Processors: Handle incoming data, for example, translating an Elasticsearch Query DSL into SQL.
  • Backend connectors: Send results from the processor to a target server using a specified protocol or API, for example, a Hydrolix cluster.

This diagram shows an example Kibana Gateway query pipeline that retrieves data from both Elasticsearch and Hydrolix.

---
config:
  themeVariables:
    fontSize: 25px
    subgraphPadding: 20
---
flowchart LR
  subgraph "Kibana Gateway"
    direction TB
    subgraph " Query Pipeline "
        direction LR
        subgraph " Frontend Connector "
            direction LR
            i1[Incoming traffic, for example Kibana]
        end
        subgraph query-procs[" Processors "]
            kgw-v1-processor-query 
        end
        subgraph " Backend Connectors "
            elasticConn[Elasticsearch backend connector]
            hdxConn[Hydrolix backend connector]
        end
    end
  end
  Queries["Incoming queries"] --> i1 --> 
  kgw-v1-processor-query --> elasticConn --> Elasticsearch[(Elasticsearch)]
  kgw-v1-processor-query --> hdxConn --> Hydrolix[(Hydrolix)]

Note

Each index routes to a single backend. A query resolves to one backend and can't span Elasticsearch and Hydrolix together. See Known Limitations.

Connectors⚓︎

Kibana Gateway uses two kinds of connectors:

  • Frontend connector: The elasticsearch-fe-query connector exposes the Elasticsearch query API to Kibana.
  • Backend connectors: The hydrolix connector serves queries against your Hydrolix tables. The elasticsearch connector connects to the in-cluster Elasticsearch, which stores Kibana metadata and is the default target for any index that isn't mapped to Hydrolix.

Kibana Gateway only uses ClickHouse native

Kibana Gateway connects to Hydrolix using the ClickHouse native protocol (clickhouse://). It doesn't support HTTP.

Processors⚓︎

Kibana Gateway uses two processor types:

  • kgw-v1-processor-query translates Elasticsearch queries into SQL and routes them to a backend connector.
  • kgw-v1-processor-noop passes traffic through without modification, for indexes that Elasticsearch serves directly.

Index and table mapping⚓︎

Kibana Gateway discovers the tables in the configured Hydrolix projects and exposes each one as an Elasticsearch index that Kibana can query. Any index that isn't mapped to Hydrolix falls back to Elasticsearch.

To configure which projects and tables are exposed, override a table name, or hide a table, see Query additional projects and tables.

Tables with the same name in multiple projects

When tables in different projects share a name, Kibana Gateway can't expose them through discovery alone. Map each one to a unique index name, as described in Why differentiate the table name and index name.

Feature flags⚓︎

The operator sets these feature flags on the Kibana Gateway deployment:

  • enableMultiTableQuery (default false): Allows a single query to span multiple Hydrolix tables. See Known Limitations.
  • multiTableMaxFanout (default 16): Limits the number of Hydrolix tables one multi-table query fans out to.
  • defaultStringColumnType (keyword or text, default text): Sets the default Elasticsearch type Kibana Gateway maps string columns to.

Legacy Quesma branding

These posts predate the Kibana Gateway rename and use the former Quesma name.