Reverse Proxy Configuration

Dynamic routing configuration in the Hydrolix cluster spec config results in updates to Traefik's configuration. This configuration will be updated with any new dynamic routing paths using headers or query parameters.

Dynamic routing uses Traefik rules and priority to determine which ingest pool should handle an incoming request. Note that dynamic ingestion rules are only executed on requests that arrive at the default pool (/ingest/event). Incoming requests to any non-default pool (/pool/{pool_name}/ingest/event) are handled exclusively by that pool.

Traefik matchers

The Hydrolix operator dynamically reconfigures the traefik matchers according to instructions in the cluster spec file.

The following matchers are used:

• PathPrefix - explicit routing to a specific pool, always present for each defined pool
• Header - exact string match on HTTP header and value
• Query - exact string match on query parameter and value
• HeaderRegexp - regular expression match on a value in a specific HTTP header
• QueryRegexp - regular expression match on a value in a specific query parameter

Verify Traefik config updates

If you haven't already, install K9s.

If an existing pool called intake-head-private-pool is updated with the following routing configuration:

Hydrolix spec config: Configuration beforeHydrolix spec config: Configuration after

spec:
  pools:
    intake-head-private-pool:
      name: intake-head-private-pool
      service: intake-head

spec:
  pools:
    intake-head-private-pool:
      name: intake-head-private-pool
      routing:
        headers:
          x-hdx-header: my_project.my_table
        query_params:
          intake-pool: private
      service: intake-head

You can confirm the Traefik configuration has been updated using the following steps.

1. Start k9s from a shell: k9s.
2. Open up the pods selector by entering: :pods.
3. Select the Traefik pod and shell into the traefik container using the command s.

4. Run the following command:

   watch -n 1 grep -B8 -A2 'PathPrefix\(\`/pool/intake-head-p' /etc/traefik/dynamic_conf.yaml
   

After a few minutes of latency at most, you will observe the following changes:

Traefik pool config before routing changesTraefik pool config after routing changes

http:
  routers: 
    slash-pool/intake-head-private-pool-router:
      rule: PathPrefix(`/pool/intake-head-private-pool`)
      service: intake-head-private-pool

http:
  routers: 
    slash-pool/intake-head-private-pool-router:
      rule: PathPrefix(`/pool/intake-head-private-pool`) || (PathPrefix(`/ingest`) && Header(`x-hdx-table`, `my_project.my_table`)) && Query(`intake-pool`, `private`)
      service: intake-head-private-pool

Rules and priority

Rules and priority within a Hydrolix cluster therefore respect the following descending order of precedence:

1. Explicit pool endpoint, PathPrefix(/pool/pool-name): Which endpoint the request arrives at, if the endpoint is a non-default ingest endpoint. For example, requests arriving at https://hostname.hydrolix.live/pool/{pool_name}/ingest/event will be handled by the ingest pool called {pool_name} regardless of the headers or query parameters included.
2. Default ingestion endpoint and query parameters, PathPrefix(/ingest) and Query Parameters (Query(key, value)): For example, a request sent to https://hostname.hydrolix.live/ingest/event?table=my_table&transform=my_transform and the header x-hdx-myheader: secondary_poolwith the following cluster configuration:

=== "Configuration example: Pools demonstrating query parameter routing for one pool and header routing for another"

     ```yaml
        spec:
          pools:
            custom-ingest-pool:
              routing:
                query_params:
                  table: my_table
                  transform: my_transform
              name: custom-ingest-pool
              service: intake-head
            secondary-pool:
              routing:
                headers:
                  x-hdx-myheader: secondary_pool
              name: secondary-pool
              service: intake-head
     ```

would be handled by custom-ingest-pool rather than secondary-pool.

1. Default ingestion endpoint and HTTP headers, PathPrefix(/ingest) and HTTP Headers (Header(key, value)): For example, a request sent to https://hostname.hydrolix.live/ingest/event with headers X-Hdx-Table: my_table and X-Hdx-Transform: my_transform with the following cluster configuration:

=== "Configuration example: Pool demonstrating header routing"

    ```yaml
       spec:
         pools:
           custom-ingest-pool:
             routing:
               headers:
                 x-hdx-table: my_table
                 x-hdx-transform: my_transform
             name: custom-ingest-pool
             service: intake-head
    ```

would be handled by the custom-ingest-pool ingest pool.

Overlapping rules

Multiple rules can match an incoming request's header and query parameter configuration.

In this case, Traefik determines which ingest pool will handle the request using a rules length priority calculation.

For example, given the following configuration:

Configuration example: Pools demonstrating rules length priority

spec:
  pools:
    long-rule-pool:
      routing:
        query_params:
          table: my_table
          transform: my_transform
      name: long-rule-pool
      service: intake-head
    short-rule-pool:
      routing:
        query_params:
          table: my_table
      name: short-rule-pool
      service: intake-head

This generates the Traefik rules:

rule: PathPrefix(/pool/long-rule-pool) || PathPrefix(/ingest) && Query(table, my_table) && Query(transform, my_transform)
rule: PathPrefix(/pool/short-rule-pool) || PathPrefix(/ingest) && Query(table, my_table)

A request coming sent to https://hostname.hydrolix.live/ingest/event?table=my_table&transform=my_transform matches both rules. The longer rule has priority, so the long-rule-pool processes the incoming request.