Scale Profiles

Deploying a development cluster for testing is not the same as deploying a production cluster. They will differ on resource scale and resiliency requirements. While each deployment needs to be turned to the use case at hand, we have simplified the process by supplying a few different scale profiles listed below. To use a scale profile, you simply add the tag scale-profile and a value to the hydrolix cluster.yaml file, for example:

hkt hydrolix-cluster --scale-profile dev > hydrolixcluster.yaml

If you don't define one the cluster defaults to eval

Available profiles to choose from:

  • prod - a fully resilient production deployment (1-4 TB/day)

  • mega - a fully resilient large scale production deployment (10-50 TB/day)

πŸ“˜

16 vCPU Nodes

Note: For eval and prod-scale deployments we recommend using 16 vCPU nodes:

  • EKS c5n.4xlarge
  • GKE n2-standard-16
  • LKE Dedicated 32 GB

πŸ“˜

32 vCPU Nodes

Note: For bench and mega-scale production deployments we recommend using 32 vCPU nodes:

  • EKS c5n.9xlarge
  • GKE n2-standard-32
  • LKE Dedicated 64 GB

πŸ‘

Profile Tuning

Profiles are a starting point, and can be overridden per service in your cluster configuration (hydrolixcluster.yaml).

You can check the resources defined by each profile using the hkt scale-defaults command and pass in the profile via the flag -p

hkt scale-defaults -p prod
service              pool    replicas      cpu  memory    storage    data_storage
-------------------  ------  ----------  -----  --------  ---------  --------------
acme                         1            0.25  256Mi     256Mi
akamai-siem-indexer          0            2     2Gi       5Gi
akamai-siem-peer             0            2     2Gi       5Gi
alter-head                   0            0.25  256Mi     256Mi
alter-indexer                0            2     8Gi       5Gi
alter-peer                   0            0.25  256Mi     5Gi
autoingest                   0            0.25  256Mi     256Mi
batch-head                   1            0.5   512Mi     256Mi
batch-indexer                0            2     4Gi       5Gi
batch-peer                   1            2     4Gi       5Gi
decay                        1            0.25  256Mi     256Mi
init-cluster                 1            0.5   512Mi     512Mi
init-turbine-api             1            0.5   512Mi     512Mi
intake-api                   2            0.25  256Mi     256Mi
job-purge                    1            0.25  256Mi     256Mi
kafka-indexer                0            2     2Gi       5Gi
kafka-peer                   0            2     2Gi       5Gi
keycloak                     1            4     4Gi       1Gi
kinesis-coordinator          0            0.25  256Mi     256Mi
kinesis-indexer              0            2     2Gi       5Gi
kinesis-peer                 0            2     2Gi       5Gi
load-sample-project          1            0.5   512Mi     512Mi
log-vacuum                   1            0.25  512Mi     256Mi
merge                        4            0.25  512Mi     5Gi
merge                I       4            0.25  512Mi     5Gi
merge                II      4            0.25  512Mi     5Gi
merge                III     4            0.25  512Mi     5Gi
merge-cleanup                1            0.25  256Mi     256Mi
merge-controller             0            1     1Gi       512Mi
merge-head                   1            1     1Gi       512Mi
merge-indexer                0            2     4Gi       5Gi
merge-indexer        I       0            2     4Gi       5Gi
merge-indexer        II      0            2     6Gi       5Gi
merge-indexer        III     0            2     12Gi      5Gi
merge-peer                   1-4          0.25  512Mi     5Gi
merge-peer           I       1-4          0.25  512Mi     5Gi
merge-peer           II      1-4          0.25  512Mi     5Gi
merge-peer           III     1-4          0.25  512Mi     5Gi
monitor-ingest               1            0.05  64Mi      64Mi
operator                     1            0.25  256Mi     256Mi
otel                         0            0.5   512Mi     512Mi
partition-vacuum             1            0.5   2Gi       512Mi
postgres                     1            4     16Gi      1Gi        100Gi
prometheus                   1            2     2Gi       1Gi        50Gi
promwaltz                    0            0.25  256Mi     256Mi
prune-locks                  1            0.25  256Mi     256Mi
pushgateway                  1            1     512Mi     512Mi
query-head                   1           14     56Gi      50Gi
query-peer                   3           14     56Gi      50Gi
rabbitmq                     3            1     512Mi     512Mi      5Gi
reaper                       1            0.5   512Mi     512Mi
redpanda                     3            4     8Gi       1Gi        1Ti
rejects-vacuum               1            0.25  256Mi     256Mi
stale-job-monitor            1            0.25  256Mi     256Mi
stream-head                  2            4     4Gi       5Gi
stream-indexer               0            2     4Gi       5Gi
stream-peer                  2-4          3     4Gi       5Gi
summary-indexer              0            2     2Gi       5Gi
summary-peer                 2-4          3     4Gi       5Gi
task-monitor                 1            0.25  256Mi     256Mi
tooling                      0            1     1Gi       16Gi
traefik                      2            2     512Mi     256Mi
traefik-cfg                  0            0.25  256Mi     256Mi
turbine-api                  2            1     1Gi       512Mi
ui                           1            0.25  256Mi     256Mi
usagemeter                   1            0.25  256Mi     256Mi
validator                    1            0.25  256Mi     256Mi
validator-indexer            0            0.25  750Mi     256Mi
vector                       0            0.25  512Mi     512Mi
version                      1            0.25  256Mi     256Mi
zookeeper                    3            0.5   512Mi     512Mi      512Mi
 hkt scale-defaults -p mega
service              pool    replicas      cpu  memory    storage    data_storage
-------------------  ------  ----------  -----  --------  ---------  --------------
acme                         1            0.25  256Mi     256Mi
akamai-siem-indexer          0            2     2Gi       5Gi
akamai-siem-peer             0            2     2Gi       5Gi
alter-head                   0            0.25  256Mi     256Mi
alter-indexer                0            2     8Gi       5Gi
alter-peer                   0            0.25  256Mi     5Gi
autoingest                   0            0.25  256Mi     256Mi
batch-head                   1            0.5   512Mi     256Mi
batch-indexer                0            2     4Gi       5Gi
batch-peer                   1            2     4Gi       5Gi
decay                        1            0.25  256Mi     256Mi
init-cluster                 1            0.5   512Mi     512Mi
init-turbine-api             1            0.5   512Mi     512Mi
intake-api                   2            0.25  256Mi     256Mi
job-purge                    1            0.25  256Mi     256Mi
kafka-indexer                0            2     2Gi       5Gi
kafka-peer                   0            2     2Gi       5Gi
keycloak                     1            4     4Gi       1Gi
kinesis-coordinator          0            0.25  256Mi     256Mi
kinesis-indexer              0            2     2Gi       5Gi
kinesis-peer                 0            2     2Gi       5Gi
load-sample-project          1            0.5   512Mi     512Mi
log-vacuum                   1            0.25  512Mi     256Mi
merge                        8            0.25  512Mi     5Gi
merge                I       8            0.25  512Mi     5Gi
merge                II      8            0.25  512Mi     5Gi
merge                III     8            0.25  512Mi     5Gi
merge-cleanup                1            0.25  256Mi     256Mi
merge-controller             0            1     2Gi       512Mi
merge-head                   1            1     2Gi       512Mi
merge-indexer                0            2     4Gi       5Gi
merge-indexer        I       0            2     4Gi       5Gi
merge-indexer        II      0            2     6Gi       5Gi
merge-indexer        III     0            2     12Gi      5Gi
merge-peer                   2-8          0.25  512Mi     5Gi
merge-peer           I       2-8          0.25  512Mi     5Gi
merge-peer           II      2-8          0.25  512Mi     5Gi
merge-peer           III     2-8          0.25  512Mi     5Gi
monitor-ingest               1            0.05  64Mi      64Mi
operator                     1            0.25  256Mi     256Mi
otel                         0            0.5   512Mi     512Mi
partition-vacuum             1            0.5   2Gi       512Mi
postgres                     1            6     24Gi      1Gi        100Gi
prometheus                   1            4     12Gi      1Gi        50Gi
promwaltz                    0            0.25  256Mi     256Mi
prune-locks                  1            0.25  256Mi     256Mi
pushgateway                  1            1     512Mi     512Mi
query-head                   1           28     112Gi     50Gi
query-peer                   10          28     112Gi     50Gi
rabbitmq                     3            1     512Mi     512Mi      5Gi
reaper                       2            0.5   1Gi       512Mi
redpanda                     5            4     8Gi       1Gi        1Ti
rejects-vacuum               1            0.25  256Mi     256Mi
stale-job-monitor            1            0.25  256Mi     256Mi
stream-head                  3-8          6     6Gi       10Gi
stream-indexer               0            2     5Gi       5Gi
stream-peer                  4-16         3     4Gi       5Gi
summary-indexer              0            2     2Gi       5Gi
summary-peer                 4-16         3     4Gi       5Gi
task-monitor                 1            0.25  256Mi     256Mi
tooling                      0            1     1Gi       16Gi
traefik                      3            4     512Mi     256Mi
traefik-cfg                  0            0.25  256Mi     256Mi
turbine-api                  2            1     1Gi       512Mi
ui                           1            0.25  256Mi     256Mi
usagemeter                   1            0.25  256Mi     256Mi
validator                    1            0.25  256Mi     256Mi
validator-indexer            0            0.25  750Mi     256Mi
vector                       0            0.5   512Mi     512Mi
version                      1            0.25  256Mi     256Mi
zookeeper                    3            0.5   512Mi     512Mi      512Mi