Hydrolix has the capability to create pools of services for specific workloads. Each pool has its own discrete resources so that a workload will not be affected by other processes or users querying, ingesting, or merging data.

For example, a more heavily-resourced query worker pool can be created for a subset of priority users or a resource-intensive dashboard, with a separate, default pool set for general use. Similarly, you can configure a specific pool of Kafka peers to be used solely for ingesting server logs and a different Kafka pool for ingesting performance logs. This capability allows independent access and scaling of workload components without the need to over-provision a single deployment.

With this functionality comes the ability to generate discrete ingest workers pools with their own endpoints for receiving streaming data over HTTP, all of which you can scale independently. You can read more about sending data to independent ingest pools in our Using Ingest Pools section.

Additionally, Role-Based Access Control (RBAC) allows you to scope access to particular functionality to your users by assigning specific roles.

The following System Components support pooling for their respective Service type(s):

• Intake (intake-head)
• Kafka (kafka-peer)
• Kinesis (kinesis-peer and kinesis-kcl-peer)
• Siem (akamai-siem-peer)
• Merge (merge-peer)
• Query (query-peer)
• Stream (stream-peerand stream-head) ( DEPRECATED, please use intake-head)

Some services, such as Keycloak, create their own resource pools which can be updated but not created or deleted. You can update your Keycloak pool (see below under Update Pools) in order to scale it, but you can’t add or remove additional pools for the Keycloak service.

🚧

The Hydrolix API and Kubernetes config contain more pools-related options

Resource pool creation and management is limited in the Hydrolix UI. Use the Hydrolix Config Api or Kubernetes for more advanced pool scaling and management.

List Pools

List Pools in the UI

A list of active pools can be seen on the Scale page.

List Pools with the API

To list pools via the API, use the List Pools endpoint.

Create Pools (Manual)

Any of the pool types listed above can be created manually. However, some of these pool types will also be created automatically when a table source is specified. Directions for automated pool creation are described below under Create Pools (Automatic)

Create Pools in the UI

You can create, delete, and edit pools anywhere in the UI by clicking "Add new" -> "Resource pool" then filling in the indicated fields. Service and name are required. All other attributes are optional and have default values.

For more information on these fields and to give you a better idea of reasonable values for them, jump to the section on scaling resource pools.

Create Pools with the API

To create a pool via the API use the Create Pool endpoint.

{
  "settings": {
    "k8s_deployment": {
      "service": "intake-head",
      "storage": "2Gi",
      "cpu": "1",
      "replicas": "1",
      "memory": "2Gi"
    }
  },
  "name": "intake-pool",
  "description": "A brief but helpful description"
}

Create Pools via hydrolixcluster.yaml

You can define resource pools directly in your hydrolixcluster.yaml file under the spec section. Creating a pool with the same specs as the one above would look like

spec:
  pools:
  - name: "intake-pool"
    service: "intake-head"
    cpu: 1
    memory: 2Gi
    replicas: 1
    storage: 2Gi
    annotations:
      description: "A brief but helpful description"

Use kubectl apply -f hydrolixcluster.yaml within the directory containing your Hydrolix config file to apply this change.

Create Pools (Automatic)

Pools can also be created automatically by configuring one of the following table sources:

Kafka

Kinesis

Akamai (SIEM)

Whenever a new table source is configured, a pool is created with that source. Once the source is created, its underlying pool can’t be updated using the source endpoints. To update the pool, use the Update Pool endpoint.

Update Pools (Scaling)

Pools cannot be renamed nor can their type be changed. However, you can scale a pool which has been previously created, including pools created for a user-configured source or those automatically created for a default service such as Keycloak. The following are scale-related settings and what they control:

* target_cpu_utilization_percentage cannot be set via the API. You must set this within your hydrolixcluster.yaml file or directly in Kubernetes.

All scale-related settings affect the container of the same name as the specified service. So, if you have a pool defined like the following

pools:
  - name: merge-peer-ii  
    replicas: 2-4
    service: merge-peer  
    target_cpu_utilization_percentage: 30

That means you want the target CPU utilization for the merge-peer container running within pods in the merge-peer-ii pool to be 30%. If you want to modify the target CPU of any other container running within the merge-peer service you need to use a scale profile.

Update Pools in the UI

To update a pool in the UI, go to the Scale page, select the menu icon on the pool to be updated, update any of the available fields that aren't grayed out, and select “Publish update” when your changes are complete.

Update Pools with the API

To edit a pool via the API, use the Update Pool endpoint.

Update Pools via hydrolixcluster.yaml

🚧

How a Pool Is Created Impacts How It Can Be Updated via the YAML Config

Pools created manually should be updated in the pools section. Pools created automatically (via a configured Source or for default services) should be updated in the scale section. If a setting is configured for a service type (such as query-peer) in the scale section, this overwrites the value for any manually-created pools of the same type that didn't have that value explicitly set within the pools section. So in the following example:

pools:
- name: "intake-pool-1"
  service: "intake-head"
  replicas: 2
- name: "intake-pool-2"
  service: "intake-head"
  cpu: 1
scale:
  intake-head:
    replicas: 3
    cpu: 3

The resulting live Kubernetes config would be:

   pools:
      cpu: 3
      name: intake-pool-1
      replicas: 2
      service: intake-head
    - cpu: 1
      name: intake-pool-2
      replicas: 3
      service: intake-head
    scale:
      intake-head:
        cpu: 3
        replicas:  3

The pools attribute takes a list of dictionaries as an argument. The nested attributes name and service are required. All others are optional, including the scale-related keys listed above. The following would scale the intake-head container within the intake-pool resource pool (created above):

pools:
- name: "intake-pool"
  service: "intake-head"
  cpu: 3
  memory: 3Gi
  replicas: 3
  storage: 4Gi
  annotations:
    description: "A brief but helpful description"

Remember to apply your updates with kubectl apply -f hydrolixcluster.yaml within the directory containing your Hydrolix config file.

The scale attribute takes a dictionary where the keys are the names of services as defined by the operator (intake-head, query-peer, etc.) or the special key profile, used to configure scale profiles. If the key is a service, the value is a dictionary that respects any of the scale-related keys listed above.

The following would scale the automatically generated pool for the intake-api service:

scale:
  intake-api:
    replicas: 3
    cpu: "500m"
    memory: "2Gi"
    storage: "4Gi"

Scale profiles can also be used with pools. You can read more about configuring scale profiles here.

Limitations

• Pool naming conventions:
  • 64 character max
  • characters must be lowercase, digits, or hyphens
  • You cannot use reserved names or overloaded names. For example, you can’t make an additional query-peer pool with the name query-peer, another keycloak, or turbine-api, etc.
• Existing pools cannot be renamed. Modifying the pool name within your hydrolixcluster.yaml and deploying this change will create a new pool to replace the old one. To change your pool name through the API, you will need to create a new pool, then delete your original pool.
• Pool type cannot be changed. If you created a pool for the service type intake-head, then this cannot be converted to a pool for merge-peer.
• As mentioned above, some services such as Keycloak have their own pools which can be updated, but not created or deleted. This means if you remove any configuration related to one of these pools, it will not delete the pool but instead return it to its default settings.

Delete Pools

Delete Pools in the UI

Pools can be deleted on the Scale page.

Delete Pools with the API

To delete a pool via the API, use the Delete Pool endpoint.

You can only delete pools which you created directly. You cannot delete pools that were created automatically for services such as Keycloak. If a pool was created automatically by registering a table source, once the source is deleted, the pool will also be deleted.

Using Pools

Ingest

You can use your ingest pools based on which endpoint you send your streaming data to.

Every ingest pool gets its own /pool/poolname URL path. For example, a custom pool called custom-pool-stream-head is accessible at

https://<your-hdx-host-url>/pool/custom-pool-stream-head

and events for this pool can be sent via HTTP POST to

https://<your-hdx-host-url>/pool/custom-pool-stream-head/ingest/event

Additionally, the default stream-head and intake-head pools are accessible at the default streaming ingest endpoint at

https://<your-hdx-host-url>/ingest/event

Ingest to this endpoint is round robined among the default stream-head and intake-head pools.

You cannot specify a custom pool as the default pool for ingest. Instead, scale the default intake-head or stream-head and stream-peer pools directly via the Update Pool endpoint. If you're not sure which ingest component to scale, see this section of the streaming ingest page for more information on the streaming ingest architectures.

For more information on intake via the Hydrolix Streaming API, See more information about intake on our HTTP Stream API page.

Merge

Merge pools are set at configuration time.

To use custom merge pools, use the Update Merge Pools Config API endpoint to set the small, medium, and large merge pools. To read more about how each of these pools is used, see this section of our Merge documentation.

Query

Specifying which query pool to use can occur at configuration time or at query execution time. After creating a custom query-peer pool, this pool can be configured as the default pool at the org, project, or table level using the hdx_query_pool_name query option within the body parameter settings.default_query_options using the Config API:

Without further configuration, queries are executed against the default query-peer pool. To execute against a non-default query pool, you can append SETTINGS hdx_query_pool_name to your query like below:

SELECT count(*)
FROM table 
WHERE column1='value'
SETTINGS hdx_query_pool_name = 'query-peer-pool-high-capacity'

This query option can also be appended when using the Query API.

It is possible to configure different pools for a related org, project, or table, or within an executed query. In the event of a conflict, the order of settings precedence from highest priority to lowest priority is:

Sources

The remaining service pools are set at configuration time (Create or Update) using the Config API. This includes:

• Kafka (kafka-peer) - Create, Update
• Kinesis (kinesis-peer, kinesis-kcl-peer) - Create, Update
• Akamai (akamai-siem-peer) - Create, Update

For these services which are created from user-configured sources, you can specify a custom pool name at source configuration time using the pool_name body parameter.

User Permissions (RBAC)

The following RBAC roles grant access to their indicated pools-related functionality. These permissions are globally scoped, meaning they cannot be limited to a particular org or table.

Even if a user doesn’t have any merge pools-related permissions, they can still change merge pool settings on tables if they have permission to create/update tables.

🚧

A Note on Permissions

If a user has permission to create or update sources, then they will be able to indirectly make and delete pools when they create and delete sources as that automatically creates or deletes the underlying resource pool. However, they will not be able to update those pools as that requires using the pools endpoints.