Resource Pools

Use pools

You can create pools of services for specific workloads. Each pool has its own discrete resources, so a workload will not be affected by other processes or users querying, ingesting, or merging data.

For example, you can create a heavily resourced query worker pool for a subset of priority users or a resource-intensive dashboard, with a separate default pool set for general use.

Use pools to ingest data

You can configure a specific pool of Kafka peers to be used solely for ingesting server logs and a different Kafka pool for ingesting performance logs. This provides independent access and scaling of workload components without the need to over-provision a single deployment.

Generate discrete ingest workers pools with their own endpoints for receiving streaming data over HTTP, which you can scale independently. Read more about sending data to independent ingest pools in our Ingest data to pools section.

RBAC

Role-Based Access Control (RBAC) allows you to scope access to particular functionality to your users by assigning specific roles.

The following System Components support pooling for their respective Service type(s):

• Intake (intake-head)
• Kafka (kafka-peer)
• Kinesis (kinesis-peer and kinesis-kcl-peer)
• Siem (akamai-siem-peer)
• Merge (merge-peer)
• Query (query-peer)
• Stream (intake-head)

⚠️

stream-peer and stream-head are deprecated

Use intake-head instead.

Keycloak resource pools

Some services, such as Keycloak, create their own resource pools which can be updated but not created or deleted. You can update your Keycloak pool in order to scale it, but you can’t add or remove additional pools for the Keycloak service.

See Scale pools for more about scaling with Keycloak.

Hydrolix API and Kubernetes

📘

The Hydrolix API Config and Kubernetes

Resource pool creation and management is limited in the Hydrolix UI. Use the Hydrolix Config API or Kubernetes for more advanced pool scaling and management.

List pools

You can list pools using the web user interface (UI) or API.

List pools in the UI

To see a list of pools in the UI, select Scale from the left nav.

List pools with the API

To list pools with the API, use the List Pools endpoint.

Create pools

Any of the pool types listed can be created manually. However, some pool types are created automatically when a table source is specified. See more about automatic pool creation in Create pools (automatic).

Pools as a list

You can create pools as lists. A list is helpful when the order of its items are important. Lists don't require strict identifiers.

Pools as a dictionary

Create pools as a dictionary to enforce unique pool names. A pool as a dictionary doesn't allow duplicates, and can be easier to read and document.

Create pools in the UI

Create, delete, and edit pools anywhere in the UI.

To create a new pool in the UI:

From the top nav, click Add new > Resource pool, then fill in the indicated fields.
Service and Name are required. All other attributes are optional and have default values.

For more information on these fields and to give you a better idea of reasonable values for them, see Scale settings.

Create pools with the API

To create a pool with the API use the Create Pool endpoint.

In this example, we create a pool with the intake-head service and some basic storage and CPU requirements.

{
  "settings": {
    "k8s_deployment": {
      "service": "intake-head",
      "storage": "2Gi",
      "cpu": "1",
      "replicas": "1",
      "memory": "2Gi"
    }
  },
  "name": "intake-pool",
  "description": "A brief but helpful description"
}

Create pools with hydrolixcluster.yaml

Define resource pools directly in your hydrolixcluster.yaml file under the spec section.

You can create a pool as a list , or as a dictionary .

spec:
  pools:
  - name: "intake-pool"
    service: "intake-head"
    cpu: 1
    memory: 2Gi
    replicas: 1
    storage: 2Gi
    annotations:
      description: "A brief but helpful description"

spec:
  pools:
    new-pool1:
      cpu: 1
      memory: 1Gi
      replicas: 5
      service: intake-head
    new-pool2:
      cpu: 1
      memory: 1Gi
      replicas: 6
      service: query-peer
      storage: 1Gi

Run kubectl apply -f hydrolixcluster.yaml in the directory containing your Hydrolix config file to apply this change.

Create pools (automatic)

Pools can also be created automatically by configuring one of the following table sources:

• Kafka
• Kinesis
• Akamai (SIEM)

You can only make changes to automatic pools with the API. See Update an automatic pool.

Scale pools

Pools can't be renamed, nor can their type be changed. You can scale an existing pool, including pools created for a user-configured source, or those automatically created for a default service such as Keycloak.

See Update pools to update and scale pools.

Scale settings

The following are scale-related settings and what they control.

* target_cpu_utilization_percentage can't be set with the API. You must set this in your hydrolixcluster.yaml file or directly in Kubernetes.

All scale-related settings affect the container of the same name as the specified service.

In this example, we set the target CPU utilization for the merge-peer container running in pods in the merge-peer-ii pool to be 30%.

pools:
  - name: merge-peer-ii  
    replicas: 2-4
    service: merge-peer  
    target_cpu_utilization_percentage: 30

To modify the target CPU of any other container running in the merge-peer service, you need to use a scale profile.

Update pools

Most pools can be updated to increase or decrease their scale.

Update pools in the UI

To update a pool in the UI:

1. Go to the Scale page.
2. Select the menu icon of the pool to update.
3. Update the necessary fields.
4. Click Publish update when complete.

Update pools with the API

To edit a pool with the API, use the Update pool endpoint.

Update an automatic pool

When you configure a new table source, a pool is created automatically for that source. Once the source is created, its base pool can’t be updated using the source endpoints.

Instead, use the Update pool endpoint to update an automatic pool.

Update pools with hydrolixcluster.yaml

The way a pool is created impacts how it can be updated with the YAML config.

Pools created manually should be updated in the pools section. Pools created automatically with a configured source or for default services should be updated in the scale section.

If a setting is configured for a service type, such as query-peer in the scale section, this overwrites the value for any manually created pools of the same type without that value explicitly set in the pools section.

In this example, we add two intake pools with the necessary additional CPU and storage information

pools:
- name: "intake-pool-1"
  service: "intake-head"
  replicas: 2
- name: "intake-pool-2"
  service: "intake-head"
  cpu: 1
scale:
  intake-head:
    replicas: 3
    cpu: 3

   pools:
    - cpu: 3
      name: intake-pool-1
      replicas: 2
      service: intake-head
    - cpu: 1
      name: intake-pool-2
      replicas: 3
      service: intake-head
    scale:
      intake-head:
        cpu: 3
        replicas:  3

The pools attribute takes a list of dictionaries as an argument.

The nested attributes name and service are required. All others are optional, including the scale-related keys in Scale settings.

Scale intake-head example

This example scales the intake-head container in the intake-pool resource pool.

pools:
- name: "intake-pool"
  service: "intake-head"
  cpu: 3
  memory: 3Gi
  replicas: 3
  storage: 4Gi
  annotations:
    description: "A brief but helpful description"

Remember to apply your updates with kubectl apply -f hydrolixcluster.yaml in the directory containing your Hydrolix config file.

The scale attribute takes a dictionary where the keys are the names of services as defined by the operator (intake-head, query-peer, etc.) or the special key profile, used to configure scale profiles.

If the key is a service, the value is a dictionary that respects any of the scale-related keys in Scale settings.

Scale automatic pool for intake-api example

This example scales the automatically generated pool for the intake-api service.

scale:
  intake-api:
    replicas: 3
    cpu: "500m"
    memory: "2Gi"
    storage: "4Gi"

Scale profiles can also be used with pools. See Scale Profiles for more information.

Limitations

• Pool naming conventions:
  • 64 character max
  • Characters must be lowercase, digits, or hyphens
  • You cannot use reserved names or overloaded names. For example, you can’t make an additional query-peer pool with the name query-peer, another keycloak, or turbine-api.
• Existing pools can't be renamed.
  Modifying the pool name in your hydrolixcluster.yaml and deploying this change will create a new pool to replace the old one.
  To change a pool name with the API, create a new pool then delete the original pool.
• The pool type can't be changed.
  For example, if you created a pool for the service type intake-head, this can't be converted to a pool for merge-peer.
• Some services, such as Keycloak, have their own pools which can be updated, but not created or deleted.
  If you remove any configuration related to one of these pools, it will not delete the pool but instead return it to its default settings.

Delete pools

You can only delete pools which you created directly. You can't delete pools that were created automatically for services.

If a pool was created automatically by registering a table source, once the source is deleted, the pool will also be deleted.

Delete pools in the UI

To delete a pool in the UI:

1. Select Scale from the left nav.
2. Find the pool to delete and click the three-dot icon.
3. Click Delete.

Delete pools with the API

To delete a pool with the API, use the Delete Pool endpoint.

Ingest data to pools

You can use your ingest pools based on which endpoint you send your streaming data to.

Every ingest pool gets its own /pool/poolname URL path. For example, a custom pool called custom-pool-stream-head is accessible using HTTPS.

https://<your-hdx-host-url>/pool/custom-pool-stream-head

Events for this pool can be sent using HTTP POST.

https://<your-hdx-host-url>/pool/custom-pool-stream-head/ingest/event

The default intake-head pools are accessible at the default streaming ingest endpoint.

https://<your-hdx-host-url>/ingest/event

Data ingested to this endpoint is distributed among the default pools.

Ingest to custom pools

You cannot specify a custom pool as the default pool for ingest. Instead, scale the default intake-head pools directly using the Update Pool endpoint.

If you're not sure which ingest component to scale, see HTTP streaming for more information on the streaming ingest architectures.

See more information about intake on our HTTP Stream API page.

Merge pools

Merge pools are set during configuration.

To use custom merge pools, use the Update Merge Pools Config API endpoint to set the small, medium, and large merge pools. To read more about how each of these pools is used, see Merge pools.

Query pools

You can specify which query pool to use during configuration or at query execution.

When you create a custom query-peer pool, this pool can be configured as the default at the org, project, or table level.

Use the hdx_query_pool_name query option in the body parameter settings.default_query_options with the Config API.

Without further configuration, queries are executed against the default query-peer pool. To execute against a non-default query pool, you can append SETTINGS hdx_query_pool_name to your query.

SELECT count(*)
FROM table 
WHERE column1='value'
SETTINGS hdx_query_pool_name = 'query-peer-pool-high-capacity'

This query option can also be appended when using the Query API.

It is possible to configure different pools for a related org, project, or table, or in an executed query. In the event of a conflict, the order of settings precedence from highest priority to lowest priority is:

Sources

The remaining service pools are set at configuration time using the Config API.

• Kafka (kafka-peer) - Create, Update
• Kinesis (kinesis-peer, kinesis-kcl-peer) - Create, Update
• Akamai (akamai-siem-peer) - Create, Update

For services created from user-configured sources, you can specify a custom pool name at source configuration using the pool_name body parameter.

User permissions (RBAC)

🚧

A Note on Permissions

If a user has permission to create or update sources, they will be able to indirectly make and delete pools for sources that automatically create or delete the underlying resource pool.

They will not be able to update those pools.

The following RBAC roles grant access to their indicated pools-related functionality. These permissions are globally scoped, meaning they cannot be limited to a particular org or table.

Even if a user doesn’t have any merge pools-related permissions, they can still change merge pool settings on tables if they have permission to create/update tables.