Scaling your Cluster
How to scale Hydrolix with Kubernetes
Basic Scaling
Altering the Scale of your cluster is most easily achieved through Kubernetes using the edit
command.
Hydrolix has both Stateful and Stateless components and editing of these components need to take this into account.
Stateful components
Service | Description |
---|---|
postgres | The Core |
redpanda | Ingest |
prometheus | Reporting and Control |
Stateless Components
Service | Description | Service | Description |
---|---|---|---|
alter-peer | Alter | query-head | Query |
batch-head | Ingest | query-peer | Query |
batch-peer | Ingest | reaper | Age |
decay | Age | stream-head | Ingest |
intake-api | Ingest | stream-peer | Ingest |
kafka-peer | Ingest | turbine-api | Query |
keycloak | The Core | traefik | The Core |
merge-head | Merge | version | The Core |
merge-peer | Merge | zookeeper | The Core |
operator | Kubernetes Operator |
kubectl edit hydrolixcluster --namespace="$CLIENT_ID"
Stateful Persistent Volume changes
You can only increase persistent volume storage, decreasing the PVC size will not work
Advanced and Pre-defined scaling options
Hydrolix provides the ability through Kubernetes to define your own scale as well as for the user to use some pre-defined scaling options.
To either use these options or define your own configuration, you can use the hkt
tool to give you a basic template for services.
For example :
hkt hydrolix-cluster --env EMAIL_PASSWORD=$EMAIL_PASSWORD > hydrolixcluster.yaml
The output will look similar to the following:
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
name: hdxcli-xxxyyyy
namespace: hdxcli-xxxyyyy
spec:
admin_email: [email protected]
client_id: hdxcli-xxxyyyy
cloud: gcp
domain: hydrolix.net
env:
EMAIL_PASSWORD: XXXXXXYYYYYYYYZZZZZZZZ
host: hydrolix
owner: admin
region: us-central1
scale_profile: prod
The file can then be edited with your favourite text editor, adding and changing the components you require. For example altering the number of batch and alter-peers to 5 each.
.....
spec:
.....
owner: admin
region: us-central1
scale:
alter-peer:
replicas: 5
batch-peer:
replicas: 5
.....
Once ready the configuration is applied as follows:
.....
cloud: gcp
domain: hydrolix.net
env:
EMAIL_PASSWORD:
host: hydrolix
owner: admin
region: us-central1
scale_off : true <--- To Turn everything off.
kubectl apply -f hydrolixcluster.yaml
Services with PVC Storage.
Some of the Services Hydrolix uses need to maintain state in order to provide a good level of High Availability and redundancy. The
postgres
,redpanda
services use PVC storage and so when setting storage manually you should specify storage using thedata_storage
key.Note PVC changes are also significant changes, and we recommend talking to us before making them.
Don't forget to apply your changes
kubectl apply -f hydrolixcluster.yaml
Predefined Scale
Hydrolix provides a short-cut to specify pre-defined scaling options. The scale for each setting is specified in the latter half of this page and can be found here - Predefined Scaling Options.
To specify the options they are placed in the top-level spec of your hydrolixcluster.yaml using the scale-profile
key. The options to available are:
An example is shown below.
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
name: hdxcli-xxxyyyy
namespace: hdxcli-xxxyyyy
spec:
admin_email: [email protected]
client_id: hdxcli-xxxyyyy
cloud: gcp
domain: hydrolix.net
env:
EMAIL_PASSWORD:
host: hydrolix
owner: admin
region: us-central1
scale_profile: prod <--- For the Prod Profile
Turn off everything
If you want to turn everything off you can do that by adding the following into the top level spec:
scale_off : true
This will turn all stateless components off that can be turned off. This should be used with care.
Predefined Scaling Option Descriptions
Default/Minimal
A basic scale profile, focused around a minimal trial deployment.
Service | Description | Replicas | CPU | Memory | Storage | Data Storage |
---|---|---|---|---|---|---|
alter-head | Alter | 0 | 1 | 1Gi | 5Gi | - |
alter-peer | Alter | 0 | 1 | 1Gi | 5Gi | - |
autoingest | Auto Batch | 0 | 500m | 512Mi | 1Gi | - |
batch-head | Ingest | 1 | 500m | 1Gi | 5Gi | - |
batch-peer | Ingest | 1-12 | 2 | 8Gi | 10Gi | - |
decay | Age | 1 | 500m | 1Gi | 5Gi | - |
indexer | Sidecar Indexer component for ingest peers | 0* | 1 | 8Gi | 10Gi* | - |
init-cluster | Initialisation Job for cluster | 1 | 500m | 1Gi | 5Gi | - |
init-turbine-api | Initialisation Job for cluster | 1 | 500m | 1Gi | 5Gi | - |
intake-api | Ingest | 1 | 500m | 1Gi | 5Gi | - |
kafka-peer | Ingest | 0 | 2 | 8Gi | 10Gi | - |
keycloak | The Core | 1 | 1 | 2Gi | 5Gi | - |
kinesis-peer | Kinesis service placeholder | 0 | 2 | 8Gi | 10Gi | - |
load-sample-project | Initialisation Job for cluster | 1 | 500m | 1Gi | 1Gi | - |
merge-head | Merge | 1 | 500m | 1Gi | 5Gi | - |
merge-peer | Merge | 1-12 | 1 | 8Gi | 10Gi | - |
operator | Kubernetes Operator | 1 | 500m | 1Gi | 5Gi | - |
postgres | The Core | 1 | 4 | 16Gi | 5Gi | 100Gi |
prometheus | Reporting and Control ) | 1 | 1 | 1Gi | 5Gi | 50Gi |
promwaltz | SideCar discovery Service for Prometheus | 0* | 500m | 512Mi | 1Gi* | - |
query-head | Query | 1 | 6 | 48Gi | 50Gi | - |
query-peer | Query | 3-12 | 15 | 60Gi | 50Gi | - |
rabbitmq | Ingest | 1 | 1 | 4Gi | 5Gi | 10Gi |
reaper | Age | 1 | 2 | 1Gi | 5Gi | - |
redpanda | Ingest | 2 | 2 | 6Gi | 5Gi | 1Ti |
stream-head | Ingest | 1-12 | 4 | 10Gi | 10Gi | - |
stream-peer | Ingest | 1-12 | 2 | 10Gi | 10Gi | - |
traefik | The Core | 2 | 1 | 1Gi | 5Gi | - |
turbine-api | Query | 1 | 500m | 1Gi | 5Gi | - |
ui | Reporting and Control | 1 | 500m | 1Gi | 5Gi | - |
version | The Core | 1 | 500m | 1Gi | 1Gi | - |
zookeeper | The Core | 1 | 1 | 1Gi | 5Gi | 5Gi |
*Replicas and Data Storage do not apply to these components and will show as default setting.
Prod
A suggested profile when looking at a Production account.
Service | Description | Replicas | CPU | Memory | Storage | Data Storage |
---|---|---|---|---|---|---|
alter-head | Alter | 0 | 1 | 1Gi | 5Gi | - |
alter-peer | Alter | 0 | 2 | 1Gi | 5Gi | - |
autoingest | Auto Batch | 0 | 500m | 512Mi | 1Gi | - |
batch-head | Ingest | 1 | 500m | 1Gi | 5Gi | - |
batch-peer | Ingest | 1-12 | 2 | 8Gi | 10Gi | - |
decay | Age | 1 | 500m | 1Gi | 5Gi | - |
indexer | Sidecar Indexer component for ingest peers | 0* | 1 | 8Gi | 10Gi* | - |
init-cluster | Initialisation Job for cluster | 1 | 500m | 1Gi | 5Gi | - |
init-turbine-api | Initialisation Job for cluster | 1 | 500m | 1Gi | 5Gi | - |
intake-api | Ingest | 2 | 500m | 1Gi | 5Gi | - |
kafka-peer | Ingest | 0 | 2 | 8Gi | 10Gi | - |
keycloak | The Core | 1 | 1 | 2Gi | 5Gi | - |
kinesis-peer | Kinesis service placeholder | 0 | 2 | 8Gi | 10Gi | - |
load-sample-project | Initialisation Job for cluster | 1 | 500m | 1Gi | 1Gi | - |
merge-head` | Merge | 1 | 500m | 1Gi | 5Gi | - |
merge-peer | Merge | 2-12 | 1 | 8Gi | 10Gi | - |
operator | Kubernetes Operator | 1 | 500m | 1Gi | 5Gi | - |
postgres | The Core | 1 | 6 | 16Gi | 5Gi | 100Gi |
prometheus | Reporting and Control ) | 1 | 1 | 1Gi | 1Gi | 50Gi |
promwaltz | SideCar discovery Service for Prometheus | 0* | 500m | 512Mi | 1Gi* | - |
query-head | Query | 2 | 6 | 48Gi | 50Gi | - |
query-peer | Query | 3-12 | 15 | 60Gi | 50Gi | - |
rabbitmq | Ingest | 3 | 1 | 4Gi | 5Gi | 10Gi |
reaper | Age | 1 | 2 | 1Gi | 5Gi | - |
redpanda | Ingest | 2 | 2 | 8Gi | 5Gi | 1Ti |
stream-head | Ingest | 2-12 | 4 | 8Gi | 10Gi | - |
stream-peer | Ingest | 1-12 | 2 | 8Gi | 10Gi | - |
traefik | The Core | 2 | 1 | 1Gi | 5Gi | - |
turbine-api | Query | 2 | 500m | 1Gi | 5Gi | - |
ui | Reporting and Control | 1 | 500m | 1Gi | 5Gi | - |
version | The Core | 1 | 500m | 1Gi | 1Gi | - |
zookeeper | The Core | 3 | 1 | 1Gi | 5Gi | 5 |
*Replicas and Data Storage do not apply to these components and will show as default setting.
mega
A suggested profile when looking at Production streaming traffic.
Service | Description | Replicas | CPU | Memory | Storage | Data Storage |
---|---|---|---|---|---|---|
alter-head | Alter | 0 | 1 | 1Gi | 5Gi | - |
alter-peer | Alter | 0 | 2 | 1Gi | 5Gi | - |
autoingest | Auto Batch | 0 | 500m | 512Mi | 1Gi | - |
batch-head | Ingest | 1 | 500m | 1Gi | 5Gi | - |
batch-peer | Ingest | 1-12 | 2 | 10Gi | 10Gi | - |
decay | Age | 1 | 500m | 1Gi | 5Gi | - |
indexer | Sidecar Indexer component for ingest peers | 0* | 1 | 8Gi | 10Gi* | - |
init-cluster | Initialisation Job for cluster | 1 | 500m | 1Gi | 5Gi | - |
init-turbine-api | Initialisation Job for cluster | 1 | 500m | 1Gi | 5Gi | - |
intake-api | Ingest | 2 | 500m | 1Gi | 5Gi | - |
kafka-peer | Ingest | 0 | 2 | 8Gi | 10Gi | - |
keycloak | The Core | 1 | 1 | 2Gi | 5Gi | - |
kinesis-peer | Kinesis service placeholder | 0 | 2 | 8Gi | 10Gi | - |
load-sample-project | Initialisation Job for cluster | 1 | 500m | 1Gi | 1Gi | - |
merge-head | Merge | 1 | 500m | 1Gi | 5Gi | - |
merge-peer | Merge | 15-45 | 1 | 8Gi | 10Gi | - |
operator | Kubernetes Operator | 1 | 500m | 1Gi | 5Gi | - |
postgres | The Core | 1 | 8 | 16Gi | 5Gi | 100Gi |
prometheus | Reporting and Control ) | 1 | 1 | 1Gi | 1Gi | 50Gi |
promwaltz | SideCar discovery Service for Prometheus | 0* | 500m | 512Mi | 1Gi* | - |
query-head | Query | 2 | 15 | 55Gi | 50Gi | - |
query-peer | Query | 3-20 | 15 | 55Gi | 50Gi | - |
rabbitmq | Ingest | 3 | 1 | 4Gi | 5Gi | 10Gi |
reaper | Age | 1 | 2 | 1Gi | 5Gi | - |
redpanda | Ingest | 5 | 2 | 8Gi | 5Gi | 1Ti |
stream-head | Ingest | 4-12 | 4 | 8Gi | 10Gi | - |
stream-peer | Ingest | 15-45 | 2 | 8Gi | 10Gi | - |
traefik | The Core | 3 | 1 | 1Gi | 5Gi | - |
turbine-api | Query | 2 | 500m | 1Gi | 5Gi | - |
ui | Reporting and Control | 1 | 500m | 1Gi | 5Gi | - |
version | The Core | 1 | 500m | 1Gi | 1Gi | - |
zookeeper | The Core | 3 | 1 | 1Gi | 5Gi | 5Gi |
*Replicas and Data Storage do not apply to these components and will show as default setting.
Dev
A suggested profile for a Development account. Note has a very low resource specification.
Service | Description | Replicas | CPU | Memory | Storage | Data Storage |
---|---|---|---|---|---|---|
alter-head | Alter | 0 | 500m | 1Gi | 1Gi | - |
alter-peer | Alter | 0 | 500m | 1Gi | 1Gi | - |
autoingest | Auto Batch | 0 | 500m | 512Mi | 1Gi | - |
batch-head | Ingest | 1 | 500m | 1Gi | 1Gi | - |
batch-peer | Ingest | 1 | 1 | 4Gi | 1Gi | - |
decay | Age | 1 | 500m | 512Mi | 1Gi | - |
indexer | Sidecar Indexer component for ingest peers | 0* | 500m | 1Gi | 1Gi* | - |
init-cluster | Initialisation Job for cluster | 1 | 500m | 1Gi | 1Gi | - |
init-turbine-api | Initialisation Job for cluster | 1 | 500m | 1Gi | 1Gi | - |
intake-api | Ingest | 1 | 500m | 1Gi | 1Gi | - |
kafka-peer | Ingest | 0 | 500m | 1Gi | 1Gi | - |
keycloak | The Core | 1 | 1 | 1Gi | 1Gi | - |
kinesis-peer | Kinesis service placeholder | 0 | 500m | 1Gi | 1Gi | - |
load-sample-project | Initialisation Job for cluster | 1 | 500m | 1Gi | 1Gi | - |
merge-head | Merge | 1 | 500m | 1Gi | 1Gi | - |
merge-peer | Merge | 1 | 500m | 1Gi | 1Gi | - |
operator | Kubernetes Operator | 1 | 500m | 1Gi | 1Gi | - |
postgres | The Core | 1 | 1 | 1Gi | 1Gi | 5Gi |
prometheus | Reporting and Control ) | 1 | 1 | 1Gi | 1Gi | 5Gi |
promwaltz | SideCar discovery Service for Prometheus | 0* | 500m | 512Mi | 1Gi* | - |
query-head | Query | 1 | 1 | 1Gi | 1Gi | - |
query-peer | Query | 1 | 1 | 1Gi | 1Gi | - |
rabbitmq | Ingest | 1 | 500m | 1Gi | 1Gi | 5Gi |
reaper | Age | 1 | 500m | 512Mi | 1Gi | - |
redpanda | Ingest | 1 | 1 | 1Gi | 1Gi | 5Gi |
stream-head | Ingest | 1 | 500m | 1Gi | 1Gi | - |
stream-peer | Ingest | 1 | 500m | 1Gi | 1Gi | - |
traefik | The Core | 1 | 500m | 1Gi | 1Gi | - |
turbine-api | Query | 1 | 1 | 1Gi | 1Gi | - |
ui | Reporting and Control | 1 | 500m | 1Gi | 1Gi | - |
version | The Core | 1 | 500m | 1Gi | 1Gi | - |
zookeeper | The Core | 1 | 500m | 1Gi | 1Gi | 1Gi |
*Replicas and Data Storage do not apply to these components and will show as default setting.
Configure Pools
Hydrolix has the capability to create pools of services for specific workloads. Each Pool has a ring-fenced capacity so that a workload will not be affected by other processes or users querying data or ingesting data.
For example a query-pool can be created for a subset of users that is accessible only by them, with another pool being used by everyone else or a specific pool of kafka servers can be used for the ingesting of server logs, but another Kafka pool used for performance logs.
By having this capability it allows independent access and scaling of workload components without the need to overprovision a single architecture. In addition with query server components, Role Based Performance Control is also possible - so users can have access to different levels of performance based on their role within a company.
The components that support pooling are:
- query-peer
- merge-peer
- kafka-peer
- batch-peer
- stream-peer
To create a pool you need to specify the pool name in the hydrolixcluster.yaml
configuration file:
apiVersion: v1
items:
- apiVersion: hydrolix.io/v1
kind: HydrolixCluster
spec:
pools:
- name: demo-pool
replicas: 1
service: query-peer
cpu: "1.0"
In the pool configuration you can specify the scale similarly to the scale configuration
Updated about 1 month ago