Skip to content

Dynamic Ingest Routing

Configure your Hydrolix cluster to send traffic to different ingest pools. The dynamic ingest routing feature selects a non-default intake-head pool based on HTTP headers or query parameters on incoming requests without requiring endpoint changes in the client configuration. Use dynamic routing in clusters with multiple resource pools and when sending data to the HTTP Stream API.

Dynamic routing configuration in the hdx spec config alters the configuration for Traefik, the cluster's reverse proxy, which is in front of ingest. See Reverse Proxy Configuration.

The Hydrolix cluster routes traffic to the appropriate intake-head pool using rules and priority and rules length priority for tie-breaking.

If no headers or query parameters match, the default intake-head pool will process the incoming request.

Dynamic routing using headers⚓︎

The reverse proxy can inspect a standard or configurable set of HTTP headers and match header values exactly or by regular expression to select a non-default intake pool.

Default headers⚓︎

Configure dynamic routing using the following standard headers:

Update your Kubernetes config for an intake-head pool with the following routing block:

1
2
3
4
5
6
7
8
9
spec:
  pools:
    routing_demo:
      routing:
        headers:
          x-hdx-table: my_project.my_table
          x-hdx-transform: my_transform
      name: routing_demo
      service: intake-head

With the above configuration, incoming requests matching both headers X-Hdx-Table: my_project.my_table and X-Hdx-Transform: my_transform will be routed through the intake-head pool called routing_demo. Requests matching only one of the two specified headers are sent through the default intake-head pool.

For more information on creating and updating service pools, see the Resource Pools page.

Custom headers⚓︎

Set the traefik_service_allowed_headers tunable to specify custom headers for controlling intake-head routing.

When customizing the tunable traefik_service_allowed_headers list, you must specify the full list of headers used in any routing rules. When the list is empty, the headers X-Hdx-Transform, X-Hdx-Table, and X-Hdx-Project are used, as demonstrated above. If setting a custom header with this tunable, include all the standard headers already in use.

Custom header keys should match the entries passed in the pool annotations. Values must match an existing pool name.

The following Hydrolix spec configures only HTTP header x-hdx-intake-pool for dynamic routing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
spec:
  pools:
    intake-head-private-pool:
      routing:
        headers:
          x-hdx-intake-pool: intake-head-private-pool
      name: intake-head-private-pool
      service: intake-head
      cpu: "8"
      memory: 8Gi
      replicas: "5"

  traefik_service_allowed_headers: ['x-hdx-intake-pool']

Incoming traffic sent to the default ingest endpoint /ingest/event and containing the header key/value pair x-hdx-intake-pool: intake-head-private-pool will be routed through the intake-head-private-pool ingest pool.

All other incoming requests use the default ingest pool.

Regex matching on headers⚓︎

Regular expression matching on headers is supported.

1
2
3
4
5
6
7
8
spec:
  pools:
    routing_demo:
      routing:
        headers:
          x-hdx-table: regex|company[.]cdn.*
      name: routing_demo
      service: intake-head

Prefix the value with regex| to signal the value is a regular expression.

Any incoming request with header X-Hdx-Table matching the regular expression company[.]cdn.* will be sent through the routing_demo intake pool.

Dynamic routing using query parameters⚓︎

Configure dynamic routing using the following standard parameters:

The reverse proxy can inspect a standard or configurable set of query parameters and match values exactly or by regular expression to select a non-default intake pool.

Default query parameters⚓︎

Use query parameters to specify table and transform. The dynamic ingest routing feature allows the cluster administrator to select an alternate intake-head pool without updating the endpoint in the client configuration.

For example, the following Hydrolix configuration spec:

1
2
3
4
5
6
7
8
9
spec:
  pools:
    intake-head-private-pool:
      routing:
        query_params:
          table: my_project.my_table
          transform: my_transform
      name: intake-head-private-pool
      service: intake-head

allows a user to send data to intake-head-private-pool with a request using the specified query parameters:

1
2
3
POST /ingest/event?table=my_project.my_table&transform=my_transform HTTP/1.1
Host: hostname.hydrolix.live
Content-Type: application/json

Custom query parameters⚓︎

Set the traefik_service_allowed_query_params tunable to specify custom query parameters for controlling intake-head routing.

Custom parameter keys should match the entries passed in the pool annotations. Values must match an existing pool name.

The following Hydrolix spec configures only query parameter intake-pool for dynamic routing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
spec:
  pools:
    intake-head-private-pool:
      routing:
        query_params:
          intake-pool: intake-head-private-pool
      name: intake-head-private-pool
      service: intake-head
      cpu: "8"
      memory: 8Gi
      replicas: "5"

  traefik_service_allowed_query_params: ['intake-pool']

Then incoming traffic sent to the default ingest endpoint with the specified query parameter:

https://hostname.hydrolix.live/ingest/event?intake_pool=intake-head-private-pool

will be routed through the intake-head-private-pool ingest pool.

When customizing the tunable traefik_service_allowed_query_params you must specify the full list of query parameters. When the list is empty, the query parameters table and transform are used, as demonstrated above. If enabling a custom query parameters with this tunable, include all the standard query parameters already in use.

Regex matching on query parameters⚓︎

Regular expression matching on query parameters is supported.

1
2
3
4
5
6
7
8
spec:
  pools:
    intake-head-private-pool:
      routing:
        query_params:
          table: regex|company[.]cdn.*
      name: intake-head-private-pool
      service: intake-head

Prefix the value with regex| to signal the value is a regular expression.

Any incoming request with query parameter table matching the regular expression company[.]cdn.* will be sent through the intake-head-private-pool intake pool.