Scale Profiles
Deploying a development cluster for testing is not the same as deploying a production cluster. They will differ on resource scale and resiliency requirements. While each deployment needs to be turned to the use case at hand, we have simplified the process by supplying a few different scale profiles listed below. To use a scale profile, you simply add the tag scale-profile and a value to the hydrolix cluster.yaml file, for example:
hkt hydrolix-cluster --scale-profile dev > hydrolixcluster.yaml
If you don't define one the cluster defaults to eval
Available profiles to choose from:
-
prod - a fully resilient production deployment (1-4 TB/day)
-
mega - a fully resilient large scale production deployment (10-50 TB/day)
16 vCPU Nodes
Note: For eval and prod-scale deployments we recommend using 16 vCPU nodes:
- EKS c5n.4xlarge
- GKE n2-standard-16
- LKE Dedicated 32 GB
32 vCPU Nodes
Note: For bench and mega-scale production deployments we recommend using 32 vCPU nodes:
- EKS c5n.9xlarge
- GKE n2-standard-32
- LKE Dedicated 64 GB
Profile Tuning
Profiles are a starting point, and can be overridden per service in your cluster configuration (
hydrolixcluster.yaml).
You can check the resources defined by each profile using the hkt scale-defaults command and pass in the profile via the flag -p
hkt scale-defaults -p prod
service cloud prof repl cpu memory store pvc
------------------- ----- ---- --- --- ----- ----- ---
alter-head - - 0 250m 256Mi 256Mi -
alter-indexer - - 0 2 8Gi 5Gi -
alter-peer - - 0 250m 256Mi 5Gi -
autoingest - - 0 250m 256Mi 256Mi -
batch-head - - 1 500m 512Mi 256Mi -
batch-indexer - - 0 1 2Gi 5Gi -
batch-peer - - 1 2 2Gi 5Gi -
decay - - 1 250m 256Mi 256Mi -
init-cluster - - 1 500m 512Mi 512Mi -
init-turbine-api - - 1 500m 512Mi 512Mi -
intake-api - - 2 250m 256Mi 256Mi -
kafka-indexer - - 0 2 2Gi 5Gi -
kafka-peer - - 0 2 2Gi 5Gi -
keycloak - - 1 1 2Gi 1Gi -
kinesis-indexer - - 0 2 2Gi 5Gi -
kinesis-peer - - 0 2 2Gi 5Gi -
load-sample-project - - 1 500m 512Mi 512Mi -
merge-head - - 1 1 1Gi 512Mi -
merge-indexer - I 0 2 3Gi 5Gi -
merge-indexer - II 0 2 6Gi 5Gi -
merge-indexer - III 0 2 12Gi 5Gi -
merge-peer - I 1-2 500m 256Mi 5Gi -
merge-peer - II 1-2 500m 256Mi 5Gi -
merge-peer - III 1-2 500m 256Mi 5Gi -
operator - - 1 250m 256Mi 256Mi -
otel - - 0 500m 512Mi 512Mi -
postgres eks - 1 4 12Gi 1Gi 100Gi
postgres generic - 1 4 16Gi 1Gi 100Gi
postgres gke - 1 4 16Gi 1Gi 100Gi
postgres lke - 1 4 8Gi 1Gi 100Gi
prometheus - - 1 2 2Gi 1Gi 50Gi
promwaltz - - 0 250m 256Mi 256Mi -
query-head eks - 1 14 36Gi 50Gi -
query-head generic - 1 14 56Gi 50Gi -
query-head gke - 1 14 56Gi 50Gi -
query-head lke - 1 14 28Gi 50Gi -
query-peer eks - 3 14 36Gi 50Gi -
query-peer generic - 3 14 56Gi 50Gi -
query-peer gke - 3 14 56Gi 50Gi -
query-peer lke - 3 14 28Gi 50Gi -
rabbitmq - - 3 1 512Mi 512Mi 5Gi
reaper - - 1 500m 512Mi 512Mi -
redpanda - - 3 4 8Gi 1Gi 1Ti
stream-head - - 2 2 4Gi 5Gi -
stream-indexer - - 0 2 2Gi 5Gi -
stream-peer - - 1-4 2 2Gi 5Gi -
summary-indexer - - 0 2 2Gi 5Gi -
summary-peer - - 0 2 2Gi 5Gi -
traefik - - 2 2 512Mi 256Mi -
traefik-cfg - - 0 250m 256Mi 256Mi -
turbine-api - - 2 1 1Gi 512Mi -
ui - - 1 250m 256Mi 256Mi -
vacuum - - 1 500m 512Mi 512Mi -
validator - - 1 250m 256Mi 256Mi -
validator-indexer - - 0 250m 256Mi 256Mi -
vector - - 0 250m 512Mi 512Mi -
version - - 1 250m 256Mi 256Mi -
zookeeper - - 3 500m 512Mi 512Mi 512Mi
hkt scale-defaults -p mega
service cloud prof repl cpu memory store pvc
------------------- ----- ---- --- --- ----- ----- ---
alter-head - - 0 250m 256Mi 256Mi -
alter-indexer - - 0 2 8Gi 5Gi -
alter-peer - - 0 250m 256Mi 5Gi -
autoingest - - 0 250m 256Mi 256Mi -
batch-head - - 1 500m 512Mi 256Mi -
batch-indexer - - 0 1 2Gi 5Gi -
batch-peer - - 1 2 2Gi 5Gi -
decay - - 1 250m 256Mi 256Mi -
init-cluster - - 1 500m 512Mi 512Mi -
init-turbine-api - - 1 500m 512Mi 512Mi -
intake-api - - 2 250m 256Mi 256Mi -
kafka-indexer - - 0 2 2Gi 5Gi -
kafka-peer - - 0 2 2Gi 5Gi -
keycloak - - 1 1 2Gi 1Gi -
kinesis-indexer - - 0 2 2Gi 5Gi -
kinesis-peer - - 0 2 2Gi 5Gi -
load-sample-project - - 1 500m 512Mi 512Mi -
merge-head - - 1 1 1Gi 512Mi -
merge-indexer - I 0 2 3Gi 5Gi -
merge-indexer - II 0 2 6Gi 5Gi -
merge-indexer - III 0 2 12Gi 5Gi -
merge-peer - I 4-8 500m 256Mi 5Gi -
merge-peer - II 2-4 500m 256Mi 5Gi -
merge-peer - III 2-4 500m 256Mi 5Gi -
operator - - 1 250m 256Mi 256Mi -
otel - - 0 500m 512Mi 512Mi -
postgres eks - 1 6 16Gi 1Gi 100Gi
postgres generic - 1 6 24Gi 1Gi 100Gi
postgres gke - 1 6 24Gi 1Gi 100Gi
postgres lke - 1 6 12Gi 1Gi 100Gi
prometheus - - 1 4 8Gi 1Gi 50Gi
promwaltz - - 0 250m 256Mi 256Mi -
query-head eks - 1 32 85Gi 50Gi -
query-head generic - 1 28 112Gi 50Gi -
query-head gke - 1 28 112Gi 50Gi -
query-head lke - 1 28 56Gi 50Gi -
query-peer eks - 10 32 85Gi 50Gi -
query-peer generic - 10 28 112Gi 50Gi -
query-peer gke - 10 28 112Gi 50Gi -
query-peer lke - 10 28 56Gi 50Gi -
rabbitmq - - 3 1 512Mi 512Mi 5Gi
reaper - - 2 500m 512Mi 512Mi -
redpanda - - 5 4 8Gi 1Gi 1Ti
stream-head - - 3-8 4 8Gi 10Gi -
stream-indexer - - 0 2 2Gi 5Gi -
stream-peer - - 4-16 2 2Gi 5Gi -
summary-indexer - - 0 2 2Gi 5Gi -
summary-peer - - 0 2 2Gi 5Gi -
traefik - - 3 4 512Mi 256Mi -
traefik-cfg - - 0 250m 256Mi 256Mi -
turbine-api - - 2 1 1Gi 512Mi -
ui - - 1 250m 256Mi 256Mi -
vacuum - - 1 500m 512Mi 512Mi -
validator - - 1 250m 256Mi 256Mi -
validator-indexer - - 0 250m 256Mi 256Mi -
vector - - 0 250m 512Mi 512Mi -
version - - 1 250m 256Mi 256Mi -
zookeeper - - 3 500m 512Mi 512Mi 512Mi
Updated about 2 months ago