Scale Profiles
Support common deployment scales using scale profiles
Overview
Deploying a development cluster for testing will differ in scale and resiliency requirements from a production cluster. Use a scale profile to easily tune a deployment to its use case.
Using a Scale Profile
To use a scale profile, add the key scale-profile and a value to the Hydrolix configuration spec file, for example:
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
name: hdxcli-xxxyyyy
namespace: hdxcli-xxxyyyy
spec:
scale_profile: prod
Apply the changes to automatically scale the system:
kubectl apply -f
If you don't define a scale profile in your Hydrolix configuration spec file, the cluster defaults to eval.
Available Profiles
Here's the list of profiles to choose from, from small to large:
-
dev- a deployment good for small tests on small clusters -
eval- a deployment suitable for trying out new features -
prod- a fully resilient production deployment (1-4 TB/day) -
mega- a fully resilient large scale production deployment (10-50 TB/day)
Profile Tuning
Profiles are a starting point, and can be overridden per service in your Hydrolix configuration spec.
Node Size Recommendations
16 vCPU Nodes
Note: For
evalandprod-scale deployments, we recommend using 16 vCPU nodes:
- EKS c5n.4xlarge
- GKE n2-standard-16
- LKE Dedicated 32 GB
32 vCPU Nodes
Note: For
mega-scale production deployments, we recommend using 32 vCPU nodes:
- EKS c5n.9xlarge
- GKE n2-standard-32
- LKE Dedicated 64 GB
Examining Scale Profiles
List the resources currently defined by each scale profile for your particular version of Hydrolix by querying the HKW service. Make sure to include the Hydrolix version number and your Kubernetes provider. The allowed Kubernetes provider values are lke, gke, eks, and generic.
For example, to find the prod scale profile when running Hydrolix v4.14.2 on Linode's LKE service:
% curl 'https://hkw.hydrolix.live/v4.14.2/scale-defaults?profile=prod&kubernetes=lke'
service pool replicas cpu memory storage data_storage
-------------------- ------ ---------- ----- -------- --------- --------------
acme 1 0.25 256Mi 256Mi
akamai-siem-indexer 0 2 2Gi 5Gi
akamai-siem-peer 0 2 2Gi 5Gi
alter-head 0 0.25 256Mi 256Mi
alter-indexer 0 2 8Gi 5Gi
alter-peer 0 0.25 256Mi 5Gi
autoingest 0 0.25 256Mi 256Mi
batch-head 1 0.5 512Mi 256Mi
batch-indexer 0 2 4Gi 5Gi
batch-peer 1 2 4Gi 5Gi
decay 1 0.25 256Mi 256Mi
envoy 1 2 512Mi 256Mi
hdx-traefik-auth 0 0.25 256Mi 256Mi
init-cluster 1 0.5 512Mi 512Mi
init-turbine-api 1 0.5 512Mi 512Mi
intake-api 2 0.25 256Mi 256Mi
intake-head 0 4 4Gi 5Gi
intake-indexer 0 2 4Gi 5Gi
job-purge 1 0.25 256Mi 256Mi
kafka-indexer 0 2 2Gi 5Gi
kafka-peer 0 2 2Gi 5Gi
keycloak 1 4 4Gi 1Gi
kinesis-coordinator 0 0.25 256Mi 256Mi
kinesis-indexer 0 2 2Gi 5Gi
kinesis-peer 0 2 2Gi 5Gi
load-sample-project 1 0.5 512Mi 512Mi
log-vacuum 1 0.25 512Mi 256Mi
merge 4 0.25 512Mi 5Gi
merge I 4 0.25 512Mi 5Gi
merge II 4 0.25 512Mi 5Gi
merge III 4 0.25 512Mi 5Gi
merge-cleanup 1 0.25 256Mi 256Mi
merge-head 1 1 1Gi 512Mi
merge-indexer 0 2 4Gi 5Gi
merge-indexer I 0 2 4Gi 5Gi
merge-indexer II 0 2 6Gi 5Gi
merge-indexer III 0 2 12Gi 5Gi
merge-peer 1-4 0.25 512Mi 5Gi
merge-peer I 1-4 0.25 512Mi 5Gi
merge-peer II 1-4 0.25 512Mi 5Gi
merge-peer III 1-4 0.25 512Mi 5Gi
monitor-ingest 1 0.05 64Mi 64Mi
operator 1 0.25 512Mi 256Mi
otel 0 0.5 512Mi 512Mi
partition-vacuum 1 0.5 2Gi 512Mi
postgres 1 4 8Gi 1Gi 100Gi
prometheus 1 2 2Gi 1Gi 50Gi
promwaltz 0 0.25 256Mi 256Mi
prune-locks 1 0.25 256Mi 256Mi
pushgateway 1 1 512Mi 512Mi
query-head 1 14 28Gi 50Gi
query-peer 3 14 28Gi 50Gi
rabbitmq 3 1 512Mi 512Mi 5Gi
reaper 1 0.5 512Mi 512Mi
redpanda 3 4 8Gi 1Gi 1Ti
refresh-job-statuses 1 0.25 256Mi 256Mi
rejects-vacuum 1 0.25 256Mi 256Mi
stale-job-monitor 1 0.25 256Mi 256Mi
stream-head 2 4 4Gi 5Gi
stream-indexer 0 2 4Gi 5Gi
stream-peer 2-4 3 4Gi 5Gi
summary-indexer 0 2 2Gi 5Gi
summary-peer 2-4 3 4Gi 5Gi
task-monitor 1 0.25 256Mi 256Mi
tooling 0 1 1Gi 16Gi
traefik 2 2 512Mi 256Mi
traefik-cfg 0 1 1Gi 256Mi
turbine-api 2 1 1Gi 512Mi
ui 1 0.25 256Mi 256Mi
usagemeter 1 0.25 256Mi 256Mi
validator 1 0.5 512Mi 512Mi
validator-indexer 0 0.5 1536Mi 512Mi
vector 0 0.5 512Mi 512Mi
version 1 0.25 256Mi 256Mi
zookeeper 3 0.5 512Mi 512Mi 512Mi
Another example for Hydrolix version 4.14.2 on Amazon EKS, using the mega scale:
% curl 'https://hkw.hydrolix.live/v4.14.2/scale-defaults?profile=mega&kubernetes=eks'
service pool replicas cpu memory storage data_storage
-------------------- ------ ---------- ----- -------- --------- --------------
acme 1 0.25 256Mi 256Mi
akamai-siem-indexer 0 2 2Gi 5Gi
akamai-siem-peer 0 2 2Gi 5Gi
alter-head 0 0.25 256Mi 256Mi
alter-indexer 0 2 8Gi 5Gi
alter-peer 0 0.25 256Mi 5Gi
autoingest 0 0.25 256Mi 256Mi
batch-head 1 0.5 512Mi 256Mi
batch-indexer 0 2 4Gi 5Gi
batch-peer 1 2 4Gi 5Gi
decay 1 0.25 256Mi 256Mi
envoy 1 4 512Mi 256Mi
hdx-traefik-auth 0 0.25 256Mi 256Mi
init-cluster 1 0.5 512Mi 512Mi
init-turbine-api 1 0.5 512Mi 512Mi
intake-api 2 0.25 256Mi 256Mi
intake-head 0 6 6Gi 10Gi
intake-indexer 0 2 5Gi 5Gi
job-purge 1 0.25 256Mi 256Mi
kafka-indexer 0 2 2Gi 5Gi
kafka-peer 0 2 2Gi 5Gi
keycloak 1 4 4Gi 1Gi
kinesis-coordinator 0 0.25 256Mi 256Mi
kinesis-indexer 0 2 2Gi 5Gi
kinesis-peer 0 2 2Gi 5Gi
load-sample-project 1 0.5 512Mi 512Mi
log-vacuum 1 0.25 512Mi 256Mi
merge 8 0.25 512Mi 5Gi
merge I 8 0.25 512Mi 5Gi
merge II 8 0.25 512Mi 5Gi
merge III 8 0.25 512Mi 5Gi
merge-cleanup 1 0.25 256Mi 256Mi
merge-head 1 1 2Gi 512Mi
merge-indexer 0 2 4Gi 5Gi
merge-indexer I 0 2 4Gi 5Gi
merge-indexer II 0 2 6Gi 5Gi
merge-indexer III 0 2 12Gi 5Gi
merge-peer 2-8 0.25 512Mi 5Gi
merge-peer I 2-8 0.25 512Mi 5Gi
merge-peer II 2-8 0.25 512Mi 5Gi
merge-peer III 2-8 0.25 512Mi 5Gi
monitor-ingest 1 0.05 64Mi 64Mi
operator 1 0.25 512Mi 256Mi
otel 0 0.5 512Mi 512Mi
partition-vacuum 1 0.5 2Gi 512Mi
postgres 1 6 16Gi 1Gi 100Gi
prometheus 1 4 12Gi 1Gi 50Gi
promwaltz 0 0.25 256Mi 256Mi
prune-locks 1 0.25 256Mi 256Mi
pushgateway 1 1 512Mi 512Mi
query-head 1 32 85Gi 50Gi
query-peer 10 32 85Gi 50Gi
rabbitmq 3 1 512Mi 512Mi 5Gi
reaper 2 0.5 1Gi 512Mi
redpanda 6 12 8Gi 1Gi 1Ti
refresh-job-statuses 1 0.25 256Mi 256Mi
rejects-vacuum 1 0.25 256Mi 256Mi
stale-job-monitor 1 0.25 256Mi 256Mi
stream-head 3-8 6 6Gi 10Gi
stream-indexer 0 2 5Gi 5Gi
stream-peer 4-16 3 4Gi 5Gi
summary-indexer 0 2 2Gi 5Gi
summary-peer 4-16 3 4Gi 5Gi
task-monitor 1 0.25 256Mi 256Mi
tooling 0 1 1Gi 16Gi
traefik 3 4 512Mi 256Mi
traefik-cfg 0 1 1Gi 256Mi
turbine-api 2 1 1Gi 512Mi
ui 1 0.25 256Mi 256Mi
usagemeter 1 0.25 256Mi 256Mi
validator 1 0.5 512Mi 512Mi
validator-indexer 0 0.5 1536Mi 512Mi
vector 0 0.5 512Mi 512Mi
version 1 0.25 256Mi 256Mi
zookeeper 3 0.5 512Mi 512Mi 512Mi
Override a scale profile
To override component scale settings, add more instances for components, or increase resources, use the scale: section of your Hydrolix configuration spec file.
This example scales the batch-peer component, with five instances and more memory than the scale profile provides.
Edit the Hydrolix configuration spec to add this override:
.....
spec:
.....
scale_profile: prod
scale:
batch-peer:
memory: 5G
replicas: 5
.....
Apply your changes:
kubectl apply -f hydrolixcluster.yaml && kubectl -n $HDX_KUBERNETES_NAMESPACE rollout restart deployment operator
Services with PVC Storage
Some of the services Hydrolix uses need to maintain state in order to provide high availability and redundancy. The
postgresservice uses PVC storage. Specify storage using thedata_storagekey.PVC changes can have significant impact. Contact Hydrolix Support if you have any questions.
Updated 6 days ago