Scale Profiles
Deploying a development cluster for testing is not the same as deploying a production cluster. They will differ on resource scale and resiliency requirements. While each deployment needs to be turned to the use case at hand, we have simplified the process by supplying a few different scale profiles listed below. To use a scale profile, you simply add the tag scale-profile and a value to the hydrolix cluster.yaml file, for example:
hkt hydrolix-cluster --scale-profile dev > hydrolixcluster.yaml
If you don't define one the cluster defaults to eval
Available profiles to choose from:
-
prod - a fully resilient production deployment (1-4 TB/day)
-
mega - a fully resilient large scale production deployment (10-50 TB/day)
16 vCPU Nodes
Note: For eval and prod-scale deployments we recommend using 16 vCPU nodes:
- EKS c5n.4xlarge
- GKE n2-standard-16
- LKE Dedicated 32 GB
32 vCPU Nodes
Note: For bench and mega-scale production deployments we recommend using 32 vCPU nodes:
- EKS c5n.9xlarge
- GKE n2-standard-32
- LKE Dedicated 64 GB
Profile Tuning
Profiles are a starting point, and can be overridden per service in your cluster configuration (
hydrolixcluster.yaml).
You can check the resources defined by each profile using the hkt scale-defaults command and pass in the profile via the flag -p
hkt scale-defaults --profile prod
service pool replicas cpu memory storage data_storage
------------------- ------ ---------- ----- -------- --------- --------------
acme 1 0.25 256Mi 256Mi
akamai-siem-indexer 0 2 2Gi 5Gi
akamai-siem-peer 0 2 2Gi 5Gi
alter-head 0 0.25 256Mi 256Mi
alter-indexer 0 2 8Gi 5Gi
alter-peer 0 0.25 256Mi 5Gi
autoingest 0 0.25 256Mi 256Mi
batch-head 1 0.5 512Mi 256Mi
batch-indexer 0 2 4Gi 5Gi
batch-peer 1 2 4Gi 5Gi
decay 1 0.25 256Mi 256Mi
hdx-traefik-auth 0 0.25 256Mi 256Mi
init-cluster 1 0.5 512Mi 512Mi
init-turbine-api 1 0.5 512Mi 512Mi
intake-api 2 0.25 256Mi 256Mi
intake-head 0 4 4Gi 5Gi
intake-indexer 0 2 4Gi 5Gi
job-purge 1 0.25 256Mi 256Mi
kafka-indexer 0 2 2Gi 5Gi
kafka-peer 0 2 2Gi 5Gi
keycloak 1 4 4Gi 1Gi
kinesis-coordinator 0 0.25 256Mi 256Mi
kinesis-indexer 0 2 2Gi 5Gi
kinesis-peer 0 2 2Gi 5Gi
load-sample-project 1 0.5 512Mi 512Mi
log-vacuum 1 0.25 512Mi 256Mi
merge 4 0.25 512Mi 5Gi
merge I 4 0.25 512Mi 5Gi
merge II 4 0.25 512Mi 5Gi
merge III 4 0.25 512Mi 5Gi
merge-cleanup 1 0.25 256Mi 256Mi
merge-head 1 1 1Gi 512Mi
merge-indexer 0 2 4Gi 5Gi
merge-indexer I 0 2 4Gi 5Gi
merge-indexer II 0 2 6Gi 5Gi
merge-indexer III 0 2 12Gi 5Gi
merge-peer 1-4 0.25 512Mi 5Gi
merge-peer I 1-4 0.25 512Mi 5Gi
merge-peer II 1-4 0.25 512Mi 5Gi
merge-peer III 1-4 0.25 512Mi 5Gi
monitor-ingest 1 0.05 64Mi 64Mi
operator 1 0.25 512Mi 256Mi
otel 0 0.5 512Mi 512Mi
partition-vacuum 1 0.5 2Gi 512Mi
postgres 1 4 16Gi 1Gi 100Gi
prometheus 1 2 2Gi 1Gi 50Gi
promwaltz 0 0.25 256Mi 256Mi
prune-locks 1 0.25 256Mi 256Mi
pushgateway 1 1 512Mi 512Mi
query-head 1 14 56Gi 50Gi
query-peer 3 14 56Gi 50Gi
rabbitmq 3 1 512Mi 512Mi 5Gi
reaper 1 0.5 512Mi 512Mi
redpanda 3 4 8Gi 1Gi 1Ti
rejects-vacuum 1 0.25 256Mi 256Mi
stale-job-monitor 1 0.25 256Mi 256Mi
stream-head 2 4 4Gi 5Gi
stream-indexer 0 2 4Gi 5Gi
stream-peer 2-4 3 4Gi 5Gi
summary-indexer 0 2 2Gi 5Gi
summary-peer 2-4 3 4Gi 5Gi
task-monitor 1 0.25 256Mi 256Mi
tooling 0 1 1Gi 16Gi
traefik 2 2 512Mi 256Mi
traefik-cfg 0 1 1Gi 256Mi
turbine-api 2 1 1Gi 512Mi
ui 1 0.25 256Mi 256Mi
usagemeter 1 0.25 256Mi 256Mi
validator 1 0.5 512Mi 512Mi
validator-indexer 0 0.5 1536Mi 512Mi
vector 0 0.5 512Mi 512Mi
version 1 0.25 256Mi 256Mi
zookeeper 3 0.5 512Mi 512Mi 512Mi
hkt scale-defaults --profile mega
service pool replicas cpu memory storage data_storage
------------------- ------ ---------- ----- -------- --------- --------------
acme 1 0.25 256Mi 256Mi
akamai-siem-indexer 0 2 2Gi 5Gi
akamai-siem-peer 0 2 2Gi 5Gi
alter-head 0 0.25 256Mi 256Mi
alter-indexer 0 2 8Gi 5Gi
alter-peer 0 0.25 256Mi 5Gi
autoingest 0 0.25 256Mi 256Mi
batch-head 1 0.5 512Mi 256Mi
batch-indexer 0 2 4Gi 5Gi
batch-peer 1 2 4Gi 5Gi
decay 1 0.25 256Mi 256Mi
hdx-traefik-auth 0 0.25 256Mi 256Mi
init-cluster 1 0.5 512Mi 512Mi
init-turbine-api 1 0.5 512Mi 512Mi
intake-api 2 0.25 256Mi 256Mi
intake-head 0 6 6Gi 10Gi
intake-indexer 0 2 5Gi 5Gi
job-purge 1 0.25 256Mi 256Mi
kafka-indexer 0 2 2Gi 5Gi
kafka-peer 0 2 2Gi 5Gi
keycloak 1 4 4Gi 1Gi
kinesis-coordinator 0 0.25 256Mi 256Mi
kinesis-indexer 0 2 2Gi 5Gi
kinesis-peer 0 2 2Gi 5Gi
load-sample-project 1 0.5 512Mi 512Mi
log-vacuum 1 0.25 512Mi 256Mi
merge 8 0.25 512Mi 5Gi
merge I 8 0.25 512Mi 5Gi
merge II 8 0.25 512Mi 5Gi
merge III 8 0.25 512Mi 5Gi
merge-cleanup 1 0.25 256Mi 256Mi
merge-head 1 1 2Gi 512Mi
merge-indexer 0 2 4Gi 5Gi
merge-indexer I 0 2 4Gi 5Gi
merge-indexer II 0 2 6Gi 5Gi
merge-indexer III 0 2 12Gi 5Gi
merge-peer 2-8 0.25 512Mi 5Gi
merge-peer I 2-8 0.25 512Mi 5Gi
merge-peer II 2-8 0.25 512Mi 5Gi
merge-peer III 2-8 0.25 512Mi 5Gi
monitor-ingest 1 0.05 64Mi 64Mi
operator 1 0.25 512Mi 256Mi
otel 0 0.5 512Mi 512Mi
partition-vacuum 1 0.5 2Gi 512Mi
postgres 1 6 24Gi 1Gi 100Gi
prometheus 1 4 12Gi 1Gi 50Gi
promwaltz 0 0.25 256Mi 256Mi
prune-locks 1 0.25 256Mi 256Mi
pushgateway 1 1 512Mi 512Mi
query-head 1 28 112Gi 50Gi
query-peer 10 28 112Gi 50Gi
rabbitmq 3 1 512Mi 512Mi 5Gi
reaper 2 0.5 1Gi 512Mi
redpanda 6 12 8Gi 1Gi 1Ti
rejects-vacuum 1 0.25 256Mi 256Mi
stale-job-monitor 1 0.25 256Mi 256Mi
stream-head 3-8 6 6Gi 10Gi
stream-indexer 0 2 5Gi 5Gi
stream-peer 4-16 3 4Gi 5Gi
summary-indexer 0 2 2Gi 5Gi
summary-peer 4-16 3 4Gi 5Gi
task-monitor 1 0.25 256Mi 256Mi
tooling 0 1 1Gi 16Gi
traefik 3 4 512Mi 256Mi
traefik-cfg 0 1 1Gi 256Mi
turbine-api 2 1 1Gi 512Mi
ui 1 0.25 256Mi 256Mi
usagemeter 1 0.25 256Mi 256Mi
validator 1 0.5 512Mi 512Mi
validator-indexer 0 0.5 1536Mi 512Mi
vector 0 0.5 512Mi 512Mi
version 1 0.25 256Mi 256Mi
zookeeper 3 0.5 512Mi 512Mi 512Mi
Updated 7 days ago