Scale Profiles

Overview

Deploying a development cluster for testing is not the same as deploying a production cluster. They will differ on resource scale and resiliency requirements. While each deployment needs to be turned to the use case at hand, we have simplified the process by supplying a few different scale profiles listed below.

Using a Scale Profile

To use a scale profile, add the tag scale-profile and a value to the hydrolixcluster.yaml file, for example:

  scale_profile: dev

If you don't define a scale profile in your hydrolixcluster.yaml file, the cluster defaults to eval.

Available Profiles

Here's the list of profiles to choose from, from small to large:

  • dev - a deployment good for small tests on small clusters

  • eval - a deployment suitable for trying out new features

  • prod - a fully resilient production deployment (1-4 TB/day)

  • mega - a fully resilient large scale production deployment (10-50 TB/day)

πŸ‘

Profile Tuning

Profiles are a starting point, and can be overridden per service in your cluster configuration (hydrolixcluster.yaml).

Node Size Reccomendations

πŸ“˜

16 vCPU Nodes

Note: For eval and prod-scale deployments, we recommend using 16 vCPU nodes:

  • EKS c5n.4xlarge
  • GKE n2-standard-16
  • LKE Dedicated 32 GB

πŸ“˜

32 vCPU Nodes

Note: For mega-scale production deployments, we recommend using 32 vCPU nodes:

  • EKS c5n.9xlarge
  • GKE n2-standard-32
  • LKE Dedicated 64 GB

Examining Scale Profiles

List the resources currently defined by each scale profile for your particular version of Hydrolix by querying the HKW service. Make sure to include the Hydrolix version number and your Kubernetes provider. The allowed Kubernetes provider values are lke, gke, eks, and generic.

For example, to find the prod scale profile when running Hydrolix v4.14.2 on Linode's LKE service:

% curl 'https://hkw.hydrolix.live/v4.14.2/scale-defaults?profile=prod&kubernetes=lke'
service               pool    replicas      cpu  memory    storage    data_storage
--------------------  ------  ----------  -----  --------  ---------  --------------
acme                          1            0.25  256Mi     256Mi
akamai-siem-indexer           0            2     2Gi       5Gi
akamai-siem-peer              0            2     2Gi       5Gi
alter-head                    0            0.25  256Mi     256Mi
alter-indexer                 0            2     8Gi       5Gi
alter-peer                    0            0.25  256Mi     5Gi
autoingest                    0            0.25  256Mi     256Mi
batch-head                    1            0.5   512Mi     256Mi
batch-indexer                 0            2     4Gi       5Gi
batch-peer                    1            2     4Gi       5Gi
decay                         1            0.25  256Mi     256Mi
envoy                         1            2     512Mi     256Mi
hdx-traefik-auth              0            0.25  256Mi     256Mi
init-cluster                  1            0.5   512Mi     512Mi
init-turbine-api              1            0.5   512Mi     512Mi
intake-api                    2            0.25  256Mi     256Mi
intake-head                   0            4     4Gi       5Gi
intake-indexer                0            2     4Gi       5Gi
job-purge                     1            0.25  256Mi     256Mi
kafka-indexer                 0            2     2Gi       5Gi
kafka-peer                    0            2     2Gi       5Gi
keycloak                      1            4     4Gi       1Gi
kinesis-coordinator           0            0.25  256Mi     256Mi
kinesis-indexer               0            2     2Gi       5Gi
kinesis-peer                  0            2     2Gi       5Gi
load-sample-project           1            0.5   512Mi     512Mi
log-vacuum                    1            0.25  512Mi     256Mi
merge                         4            0.25  512Mi     5Gi
merge                 I       4            0.25  512Mi     5Gi
merge                 II      4            0.25  512Mi     5Gi
merge                 III     4            0.25  512Mi     5Gi
merge-cleanup                 1            0.25  256Mi     256Mi
merge-head                    1            1     1Gi       512Mi
merge-indexer                 0            2     4Gi       5Gi
merge-indexer         I       0            2     4Gi       5Gi
merge-indexer         II      0            2     6Gi       5Gi
merge-indexer         III     0            2     12Gi      5Gi
merge-peer                    1-4          0.25  512Mi     5Gi
merge-peer            I       1-4          0.25  512Mi     5Gi
merge-peer            II      1-4          0.25  512Mi     5Gi
merge-peer            III     1-4          0.25  512Mi     5Gi
monitor-ingest                1            0.05  64Mi      64Mi
operator                      1            0.25  512Mi     256Mi
otel                          0            0.5   512Mi     512Mi
partition-vacuum              1            0.5   2Gi       512Mi
postgres                      1            4     8Gi       1Gi        100Gi
prometheus                    1            2     2Gi       1Gi        50Gi
promwaltz                     0            0.25  256Mi     256Mi
prune-locks                   1            0.25  256Mi     256Mi
pushgateway                   1            1     512Mi     512Mi
query-head                    1           14     28Gi      50Gi
query-peer                    3           14     28Gi      50Gi
rabbitmq                      3            1     512Mi     512Mi      5Gi
reaper                        1            0.5   512Mi     512Mi
redpanda                      3            4     8Gi       1Gi        1Ti
refresh-job-statuses          1            0.25  256Mi     256Mi
rejects-vacuum                1            0.25  256Mi     256Mi
stale-job-monitor             1            0.25  256Mi     256Mi
stream-head                   2            4     4Gi       5Gi
stream-indexer                0            2     4Gi       5Gi
stream-peer                   2-4          3     4Gi       5Gi
summary-indexer               0            2     2Gi       5Gi
summary-peer                  2-4          3     4Gi       5Gi
task-monitor                  1            0.25  256Mi     256Mi
tooling                       0            1     1Gi       16Gi
traefik                       2            2     512Mi     256Mi
traefik-cfg                   0            1     1Gi       256Mi
turbine-api                   2            1     1Gi       512Mi
ui                            1            0.25  256Mi     256Mi
usagemeter                    1            0.25  256Mi     256Mi
validator                     1            0.5   512Mi     512Mi
validator-indexer             0            0.5   1536Mi    512Mi
vector                        0            0.5   512Mi     512Mi
version                       1            0.25  256Mi     256Mi
zookeeper                     3            0.5   512Mi     512Mi      512Mi

Another example for Hydrolix version 4.14.2 on Amazon EKS, using the mega scale:

% curl 'https://hkw.hydrolix.live/v4.14.2/scale-defaults?profile=mega&kubernetes=eks'
service               pool    replicas      cpu  memory    storage    data_storage
--------------------  ------  ----------  -----  --------  ---------  --------------
acme                          1            0.25  256Mi     256Mi
akamai-siem-indexer           0            2     2Gi       5Gi
akamai-siem-peer              0            2     2Gi       5Gi
alter-head                    0            0.25  256Mi     256Mi
alter-indexer                 0            2     8Gi       5Gi
alter-peer                    0            0.25  256Mi     5Gi
autoingest                    0            0.25  256Mi     256Mi
batch-head                    1            0.5   512Mi     256Mi
batch-indexer                 0            2     4Gi       5Gi
batch-peer                    1            2     4Gi       5Gi
decay                         1            0.25  256Mi     256Mi
envoy                         1            4     512Mi     256Mi
hdx-traefik-auth              0            0.25  256Mi     256Mi
init-cluster                  1            0.5   512Mi     512Mi
init-turbine-api              1            0.5   512Mi     512Mi
intake-api                    2            0.25  256Mi     256Mi
intake-head                   0            6     6Gi       10Gi
intake-indexer                0            2     5Gi       5Gi
job-purge                     1            0.25  256Mi     256Mi
kafka-indexer                 0            2     2Gi       5Gi
kafka-peer                    0            2     2Gi       5Gi
keycloak                      1            4     4Gi       1Gi
kinesis-coordinator           0            0.25  256Mi     256Mi
kinesis-indexer               0            2     2Gi       5Gi
kinesis-peer                  0            2     2Gi       5Gi
load-sample-project           1            0.5   512Mi     512Mi
log-vacuum                    1            0.25  512Mi     256Mi
merge                         8            0.25  512Mi     5Gi
merge                 I       8            0.25  512Mi     5Gi
merge                 II      8            0.25  512Mi     5Gi
merge                 III     8            0.25  512Mi     5Gi
merge-cleanup                 1            0.25  256Mi     256Mi
merge-head                    1            1     2Gi       512Mi
merge-indexer                 0            2     4Gi       5Gi
merge-indexer         I       0            2     4Gi       5Gi
merge-indexer         II      0            2     6Gi       5Gi
merge-indexer         III     0            2     12Gi      5Gi
merge-peer                    2-8          0.25  512Mi     5Gi
merge-peer            I       2-8          0.25  512Mi     5Gi
merge-peer            II      2-8          0.25  512Mi     5Gi
merge-peer            III     2-8          0.25  512Mi     5Gi
monitor-ingest                1            0.05  64Mi      64Mi
operator                      1            0.25  512Mi     256Mi
otel                          0            0.5   512Mi     512Mi
partition-vacuum              1            0.5   2Gi       512Mi
postgres                      1            6     16Gi      1Gi        100Gi
prometheus                    1            4     12Gi      1Gi        50Gi
promwaltz                     0            0.25  256Mi     256Mi
prune-locks                   1            0.25  256Mi     256Mi
pushgateway                   1            1     512Mi     512Mi
query-head                    1           32     85Gi      50Gi
query-peer                    10          32     85Gi      50Gi
rabbitmq                      3            1     512Mi     512Mi      5Gi
reaper                        2            0.5   1Gi       512Mi
redpanda                      6           12     8Gi       1Gi        1Ti
refresh-job-statuses          1            0.25  256Mi     256Mi
rejects-vacuum                1            0.25  256Mi     256Mi
stale-job-monitor             1            0.25  256Mi     256Mi
stream-head                   3-8          6     6Gi       10Gi
stream-indexer                0            2     5Gi       5Gi
stream-peer                   4-16         3     4Gi       5Gi
summary-indexer               0            2     2Gi       5Gi
summary-peer                  4-16         3     4Gi       5Gi
task-monitor                  1            0.25  256Mi     256Mi
tooling                       0            1     1Gi       16Gi
traefik                       3            4     512Mi     256Mi
traefik-cfg                   0            1     1Gi       256Mi
turbine-api                   2            1     1Gi       512Mi
ui                            1            0.25  256Mi     256Mi
usagemeter                    1            0.25  256Mi     256Mi
validator                     1            0.5   512Mi     512Mi
validator-indexer             0            0.5   1536Mi    512Mi
vector                        0            0.5   512Mi     512Mi
version                       1            0.25  256Mi     256Mi
zookeeper                     3            0.5   512Mi     512Mi      512Mi