Scale your Cluster
How to scale Hydrolix with Kubernetes
Basic Scaling
Scaling a component's resources or replica count is done within the scale:
section of your hydrolixcluster.yaml
file. Hydrolix has both stateful and stateless components, and the scaling of these components need to take this into account.
Rather than require you to scale each of the many components by hand, we provide pre-made Scale Profiles that work for various throughput levels. For instance, scale_profile: prod
provides components scaled for a typical 1-4 TB/daily workload.
You can always override a component's scale profile setting by providing your own setting in the scale
section of the hydrolixcluster.yaml
file.
Stateful Components
These components have a data_storage
scale key:
Service | Description |
---|---|
postgres | Core |
redpanda | Ingest |
prometheus | Reporting and Control |
Stateless Components
These components all have cpu
, memory
& storage
scale keys along with replicas
:
Service | Description | Service | Description |
---|---|---|---|
alter-peer | Alter Jobs | query-head | Query |
batch-head | Ingest | query-peer | Query |
batch-peer | Ingest | reaper | Data Lifecycle |
decay | Data Lifecycle | stream-head | Ingest |
intake-api | Ingest | stream-peer | Ingest |
kafka-peer | Ingest | turbine-api | Query |
keycloak | Core | traefik | Core |
merge-head | Merge | version | Core |
merge-peer | Merge | zookeeper | Core |
operator | Kubernetes Operator |
To edit the file, you can use this command
kubectl edit hydrolixcluster --namespace="$HDX_KUBERNETES_NAMESPACE"
Stateful Persistent Volume changes
You can only increase persistent volume storage, decreasing the PVC size will not work.
Scale Settings
The following settings can be used to set the resource values of a pod
Value | Description | Example |
---|---|---|
cpu | Amount of CPU to use for the pod/container | cpu: 2 cpu: 2.5 |
memory | The amount of RAM to use for the pod/container | memory: 500Gi |
storage | The amount of ephemeral storage to be used: note this type of storage is not stateful | storage: 10Gi |
data_storage | For those pods that use PVC's, to scale the PVC you can use the data_storage key | data_storage: 1TB |
Specifying Pods and Their Settings
To modify a pod's individual settings in your hydrolixcluster.yaml, the pod component name can be given and the setting you wish to apply can be added.
For example, this modifies the stream-head
pods to have 2 CPUs and 10 Gibibytes of RAM allocated:
scale:
stream-head:
cpu: 2
memory: 10Gi
Some components have multiple containers within a pod. Typically these containers will be seen as <service>-peer
and a turbine
container. For example, stream peers have a container called stream-peer
and a container called turbine
. Turbine is the "Indexer" component that executes some transformation and indexes the content.
Any setting you use with the default name (e.g. stream-peer
) will only define the stream-peer component. It will not edit the turbine
component.
Specify the turbine component in your hydrolixcluster.yaml file using the name <component>-indexer
. For example, for stream
, use stream-indexer
. A list of the different pods can be found here.
Scale Profiles
Specify a scale_profile:
key in your hydrolixcluster.yaml file with a value of prod or mega.
-
prod - a fully resilient production deployment (1-4 TB/day)
-
mega - a fully resilient large scale production deployment (10-50 TB/day)
An example of this setting in the hydrolixcluster.yaml file is shown below.
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
name: hdxcli-xxxyyyy
namespace: hdxcli-xxxyyyy
spec:
admin_email: [email protected]
kubernetes_namespace: hdxcli-xxxyyyy
kubernets_profile: gcp
env:
EMAIL_PASSWORD:
hydrolix_url: https://host.hydrolix.net
db_bucket_region: us-central1
scale_profile: prod <--- For the Prod Profile
Once you kubectl apply -f
this change the system will be automatically scaled.
You can view details about each of these scale profiles.
Scale Overriding
You will most likely need to override some of the components scale settings. You may need more instances of a type of component (horizontally scaling), or you may need more resources (vertical scaling). Both needs are served by the scale:
section of your hydrolixcluster.yaml file.
Lets say you needed to scale the batch-peer component, as your task requires five instances and more memory that the scale profile provided. The prod
scale profile provides two Gi memory and one replica. Edit the hydrolixcluster.yaml to add this override:
.....
spec:
.....
scale_profile: prod
scale:
batch-peer:
memory: 5G
replicas: 5
.....
Once ready, the configuration is applied as follows:
kubectl apply -f hydrolixcluster.yaml
Services with PVC Storage
Some of the services Hydrolix uses need to maintain state in order to provide high availability and redundancy. The
postgres
andredpanda
services use PVC storage, so when setting storage overrides you should specify storage using thedata_storage
key.PVC changes are also significant changes, so we recommend talking to us before making them.
Don't forget to apply your changes
kubectl apply -f hydrolixcluster.yaml && kubectl -n $HDX_KUBERNETES_NAMESPACE rollout restart deployment operator
Scaling to Zero
If you want to turn everything in your cluster off, you can do that by adding the following into the top level spec:
scale_off : true
.....
kubernets_profile: gcp
hydrolix_url: https://host.hydrolix.net
env:
EMAIL_PASSWORD:
db_bucket_region: us-central1
scale_off : true <--- Turn everything off
This will turn all stateless components off that can be turned off. It will take a couple of minutes for all those components to shut down. This should be used with care.
Configuring Query Peer Pools
Hydrolix has the capability to create pools of services for specific workloads. Each pool has a ring-fenced capacity so that a workload will not be affected by other processes or users querying data or ingesting data.
For example, a query-pool can be created for a subset of users that is accessible only by them, with another pool being used by everyone else. Similarly you can configure a specific pool of Kafka servers which be used solely for ingesting server logs, and another Kafka pool used for performance logs.
This capability allows independent access and scaling of workload components without the need to overprovision a single architecture. In addition to query server components, Role Based Performance Control gives users access to different levels of performance based on their role within a company.
To create a pool, specify the pool name in the hydrolixcluster.yaml
configuration file:
apiVersion: v1
items:
- apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
pools:
- name: demo-pool
replicas: 1
service: query-peer
cpu: "1.0"
Within the pools:
configuration, specify the scale similarly to the scale:
configuration above.
Updated 2 months ago