Scale Profiles

Deploying a development cluster for testing is not the same as deploying a production cluster. They will differ on resource scale and resiliency requirements. While each deployment needs to be turned to the use case at hand, we have simplified the process by supplying a few different scale profiles listed below. To use a scale profile, you simply add the tag scale-profile and a value to the hydrolix cluster.yaml file, for example:

hkt hydrolix-cluster --scale-profile dev > hydrolixcluster.yaml

If you don't define one the cluster defaults to eval

Available profiles to choose from:

  • prod - a fully resilient production deployment (1-4 TB/day)

  • mega - a fully resilient large scale production deployment (10-50 TB/day)

📘

16 vCPU Nodes

Note: For eval and prod-scale deployments we recommend using 16 vCPU nodes:

  • EKS c5n.4xlarge
  • GKE n2-standard-16
  • LKE Dedicated 32 GB

📘

32 vCPU Nodes

Note: For bench and mega-scale production deployments we recommend using 32 vCPU nodes:

  • EKS c5n.9xlarge
  • GKE n2-standard-32
  • LKE Dedicated 64 GB

👍

Profile Tuning

Profiles are a starting point, and can be overridden per service in your cluster configuration (hydrolixcluster.yaml).

You can check the resources defined by each profile using the hkt scale-defaults command and pass in the profile via the flag -p

hkt scale-defaults -p prod

service              cloud    prof repl  cpu   memory store  pvc
-------------------  -----    ---- ---   ---   -----  -----  ---
alter-head           -        -    0     250m  256Mi  256Mi  -
alter-indexer        -        -    0     2     8Gi    5Gi    -
alter-peer           -        -    0     250m  256Mi  5Gi    -
autoingest           -        -    0     250m  256Mi  256Mi  -
batch-head           -        -    1     500m  512Mi  256Mi  -
batch-indexer        -        -    0     1     2Gi    5Gi    -
batch-peer           -        -    1     2     2Gi    5Gi    -
decay                -        -    1     250m  256Mi  256Mi  -
init-cluster         -        -    1     500m  512Mi  512Mi  -
init-turbine-api     -        -    1     500m  512Mi  512Mi  -
intake-api           -        -    2     250m  256Mi  256Mi  -
kafka-indexer        -        -    0     2     2Gi    5Gi    -
kafka-peer           -        -    0     2     2Gi    5Gi    -
keycloak             -        -    1     1     2Gi    1Gi    -
kinesis-indexer      -        -    0     2     2Gi    5Gi    -
kinesis-peer         -        -    0     2     2Gi    5Gi    -
load-sample-project  -        -    1     500m  512Mi  512Mi  -
merge-head           -        -    1     1     1Gi    512Mi  -
merge-indexer        -        I    0     2     3Gi    5Gi    -
merge-indexer        -        II   0     2     6Gi    5Gi    -
merge-indexer        -        III  0     2     12Gi   5Gi    -
merge-peer           -        I    1-2   500m  256Mi  5Gi    -
merge-peer           -        II   1-2   500m  256Mi  5Gi    -
merge-peer           -        III  1-2   500m  256Mi  5Gi    -
operator             -        -    1     250m  256Mi  256Mi  -
otel                 -        -    0     500m  512Mi  512Mi  -
postgres             eks      -    1     4     12Gi   1Gi    100Gi
postgres             generic  -    1     4     16Gi   1Gi    100Gi
postgres             gke      -    1     4     16Gi   1Gi    100Gi
postgres             lke      -    1     4     8Gi    1Gi    100Gi
prometheus           -        -    1     2     2Gi    1Gi    50Gi
promwaltz            -        -    0     250m  256Mi  256Mi  -
query-head           eks      -    1     14    36Gi   50Gi   -
query-head           generic  -    1     14    56Gi   50Gi   -
query-head           gke      -    1     14    56Gi   50Gi   -
query-head           lke      -    1     14    28Gi   50Gi   -
query-peer           eks      -    3     14    36Gi   50Gi   -
query-peer           generic  -    3     14    56Gi   50Gi   -
query-peer           gke      -    3     14    56Gi   50Gi   -
query-peer           lke      -    3     14    28Gi   50Gi   -
rabbitmq             -        -    3     1     512Mi  512Mi  5Gi
reaper               -        -    1     500m  512Mi  512Mi  -
redpanda             -        -    3     4     8Gi    1Gi    1Ti
stream-head          -        -    2     2     4Gi    5Gi    -
stream-indexer       -        -    0     2     2Gi    5Gi    -
stream-peer          -        -    1-4   2     2Gi    5Gi    -
summary-indexer      -        -    0     2     2Gi    5Gi    -
summary-peer         -        -    0     2     2Gi    5Gi    -
traefik              -        -    2     2     512Mi  256Mi  -
traefik-cfg          -        -    0     250m  256Mi  256Mi  -
turbine-api          -        -    2     1     1Gi    512Mi  -
ui                   -        -    1     250m  256Mi  256Mi  -
vacuum               -        -    1     500m  512Mi  512Mi  -
validator            -        -    1     250m  256Mi  256Mi  -
validator-indexer    -        -    0     250m  256Mi  256Mi  -
vector               -        -    0     250m  512Mi  512Mi  -
version              -        -    1     250m  256Mi  256Mi  -
zookeeper            -        -    3     500m  512Mi  512Mi  512Mi
hkt scale-defaults -p mega

service              cloud    prof repl  cpu   memory store  pvc
-------------------  -----    ---- ---   ---   -----  -----  ---
alter-head           -        -    0     250m  256Mi  256Mi  -
alter-indexer        -        -    0     2     8Gi    5Gi    -
alter-peer           -        -    0     250m  256Mi  5Gi    -
autoingest           -        -    0     250m  256Mi  256Mi  -
batch-head           -        -    1     500m  512Mi  256Mi  -
batch-indexer        -        -    0     1     2Gi    5Gi    -
batch-peer           -        -    1     2     2Gi    5Gi    -
decay                -        -    1     250m  256Mi  256Mi  -
init-cluster         -        -    1     500m  512Mi  512Mi  -
init-turbine-api     -        -    1     500m  512Mi  512Mi  -
intake-api           -        -    2     250m  256Mi  256Mi  -
kafka-indexer        -        -    0     2     2Gi    5Gi    -
kafka-peer           -        -    0     2     2Gi    5Gi    -
keycloak             -        -    1     1     2Gi    1Gi    -
kinesis-indexer      -        -    0     2     2Gi    5Gi    -
kinesis-peer         -        -    0     2     2Gi    5Gi    -
load-sample-project  -        -    1     500m  512Mi  512Mi  -
merge-head           -        -    1     1     1Gi    512Mi  -
merge-indexer        -        I    0     2     3Gi    5Gi    -
merge-indexer        -        II   0     2     6Gi    5Gi    -
merge-indexer        -        III  0     2     12Gi   5Gi    -
merge-peer           -        I    4-8   500m  256Mi  5Gi    -
merge-peer           -        II   2-4   500m  256Mi  5Gi    -
merge-peer           -        III  2-4   500m  256Mi  5Gi    -
operator             -        -    1     250m  256Mi  256Mi  -
otel                 -        -    0     500m  512Mi  512Mi  -
postgres             eks      -    1     6     16Gi   1Gi    100Gi
postgres             generic  -    1     6     24Gi   1Gi    100Gi
postgres             gke      -    1     6     24Gi   1Gi    100Gi
postgres             lke      -    1     6     12Gi   1Gi    100Gi
prometheus           -        -    1     4     8Gi    1Gi    50Gi
promwaltz            -        -    0     250m  256Mi  256Mi  -
query-head           eks      -    1     32    85Gi   50Gi   -
query-head           generic  -    1     28    112Gi  50Gi   -
query-head           gke      -    1     28    112Gi  50Gi   -
query-head           lke      -    1     28    56Gi   50Gi   -
query-peer           eks      -    10    32    85Gi   50Gi   -
query-peer           generic  -    10    28    112Gi  50Gi   -
query-peer           gke      -    10    28    112Gi  50Gi   -
query-peer           lke      -    10    28    56Gi   50Gi   -
rabbitmq             -        -    3     1     512Mi  512Mi  5Gi
reaper               -        -    2     500m  512Mi  512Mi  -
redpanda             -        -    5     4     8Gi    1Gi    1Ti
stream-head          -        -    3-8   4     8Gi    10Gi   -
stream-indexer       -        -    0     2     2Gi    5Gi   -
stream-peer          -        -    4-16  2     2Gi    5Gi   -
summary-indexer      -        -    0     2     2Gi    5Gi   -
summary-peer         -        -    0     2     2Gi    5Gi   -
traefik              -        -    3     4     512Mi  256Mi  -
traefik-cfg          -        -    0     250m  256Mi  256Mi  -
turbine-api          -        -    2     1     1Gi    512Mi  -
ui                   -        -    1     250m  256Mi  256Mi  -
vacuum               -        -    1     500m  512Mi  512Mi  -
validator            -        -    1     250m  256Mi  256Mi  -
validator-indexer    -        -    0     250m  256Mi  256Mi  -
vector               -        -    0     250m  512Mi  512Mi  -
version              -        -    1     250m  256Mi  256Mi  -
zookeeper            -        -    3     500m  512Mi  512Mi  512Mi