Skip to content

Scale Profiles

Overview⚓︎

Scale profiles provide predefined resource and replica settings for all cluster components. They give you a consistent baseline without tuning each service individually.

When to use a scale profile⚓︎

  • Create a new cluster: Start with a profile that matches your expected ingest and query load.
  • Standardize resources: Apply the same profile across environments for consistency.
  • Simplify scaling: Use a single profile instead of setting replicas for every service.

How scale profiles work⚓︎

Each profile defines default CPU, memory, and replica values for services in a cluster. You apply the profile once, and the operator propagates those settings across the cluster.

Scale profile changes apply cluster-wide.

Profiles are named for typical use cases. For example:

  • prod: tuned for steady ingest of 1–4 TB per day with balanced query load.
  • dev: lighter settings for development or test environments.

Set a profile⚓︎

Add the profile to your hydrolixcluster.yaml file:

Set Scale Profile
1
2
spec:
  scale_profile: prod

The operator applies the prod settings to all cluster services.

Default profile⚓︎

If no scale profile is set, the cluster defaults to eval.

Available profiles⚓︎

Hydrolix includes predefined profiles for common use cases:

  • dev: light settings for small clusters and testing.
  • eval: evaluation settings for trying out features.
  • prod: production-ready profile for 1–4 TB/day ingest with balanced query load.
  • mega: large-scale production profile for 10–50 TB/day ingest.

Profiles provide a baseline and can be customized or overridden. See the default settings for each component for a profile using:

Scale Defaults Endpoint
1
curl 'https://www.hydrolix.io/operator/${VERSION}/scale-defaults?profile=${PROFILE}&kubernetes=${PLATFORM}'

For example:

Example Request to the Scale Defaults Endpoint
1
curl 'https://www.hydrolix.io/operator/v6.2.4/scale-defaults?profile=prod&kubernetes=gke'

The response lists settings for all services:

Example response from Scale Defaults Endpoint 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
service                  pool    replicas       cpu  memory    storage    data_storage
-----------------------  ------  ----------  ------  --------  ---------  --------------
acme                             1            0.128  64Mi      64Mi
ad                               1            2      2Gi       256Mi
akamai-siem-indexer              0            2      2Gi       5Gi
akamai-siem-peer                 0            2      2Gi       5Gi
alter-head                       0            0.25   256Mi     256Mi
alter-indexer                    0            2      8Gi       5Gi
alter-peer                       0            0.25   256Mi     5Gi
anomaly-rca                      1            0.75   1Gi       256Mi
ariadne-core                     1            0.5    2Gi       256Mi
autoingest                       0            0.25   256Mi     256Mi
batch-controller                 0            1      1Gi       512Mi
batch-head                       1            0.5    512Mi     256Mi
batch-indexer                    0            2      4Gi       5Gi
batch-peer                       1            2      4Gi       5Gi
catalog-proxy                    1            2      1Gi       512Mi
cert-expiry-check                1            0.5    1Gi       512Mi
check-bucket-access              1            0.5    1Gi       512Mi
clickprox                        2            0.5    512Mi     512Mi
crl-downloader                   1            0.128  128Mi     70Mi
decay                            1            0.25   256Mi     256Mi
descheduler                      1            0.25   256Mi     256Mi
elasticsearch                    1            1      2Gi       2Gi        5Gi
grafana                          2            4      8Gi       512Mi
hdx-ariadne-janus                1            2      4Gi       256Mi
hdx-ariadne-janus-proxy          1            0.5    256Mi     256Mi
hdx-binpack-scheduler            2            0.25   256Mi     256Mi
hdx-gate                         1            0.25   526Mi     526Mi
hdx-node                         0            0.25   256Mi     256Mi
hdx-otel-collector               1            0.5    512Mi     8Gi
hdx-pg-monitor                   1            0.25   256Mi     256Mi
hdx-pod-metrics                  1            0.025  5Mi       5Mi
hdx-query-discovery              1            0.1    64Mi      128Mi
hdx-scaler                       0            1      1Gi       256Mi
hdx-scaler-go                    0            0.5    512Mi     256Mi
hdx-traefik-auth                 0            0.25   256Mi     256Mi
hdx-vpa-metrics                  0            1      1Gi       256Mi
http-head                        0            4      2Gi       4Gi
http-proxy                       2            0.5    512Mi     9Gi
hydroglyph                       1            4      16Gi      40Gi
hydroglyph-mirror                1            2      4Gi       1Gi
hydroglyph-onboard               1            0.5    1Gi       1Gi
init-cluster                     1            0.5    512Mi     512Mi
init-turbine-api                 1            0.5    1Gi       512Mi
intake-api                       2            0.25   256Mi     256Mi
intake-head                      2            4      4Gi       5Gi
intake-head-df                   0            0.5    512Mi     512Mi
intake-indexer                   0            2      4Gi       5Gi
intake-peer                      0            0.25   526Mi     512Mi
intake-router                    0            2      1Gi       512Mi
janus-model-cost-reload          1            0.025  32Mi      32Mi
job-purge                        1            0.25   256Mi     256Mi
kafka-indexer                    0            2      2Gi       5Gi
kafka-peer                       0            2      2Gi       5Gi
keycloak                         1            4      4Gi       1Gi
kibana                           1            1      1Gi       512Mi
kibana-gateway                   1            1      1Gi       1Gi
kinesis-coordinator              0            0.25   256Mi     256Mi
kinesis-indexer                  0            2      2Gi       5Gi
kinesis-kcl-consumer             0            2      2Gi       256Mi
kinesis-peer                     0            2      2Gi       5Gi
load-sample-project              1            0.5    1Gi       512Mi
mcp-hydrolix                     1            1      4Gi       1Gi
merge                            4            0.25   512Mi     5Gi
merge                    I       4            0.25   512Mi     5Gi
merge                    II      4            0.25   512Mi     5Gi
merge                    III     4            0.25   512Mi     5Gi
merge-cleanup                    1            0.25   256Mi     256Mi
merge-controller                 1            1      1Gi       512Mi
merge-indexer                    0            2      4Gi       5Gi
merge-indexer            I       0            2      4Gi       5Gi
merge-indexer            II      0            2      6Gi       5Gi
merge-indexer            III     0            2      12Gi      5Gi
merge-peer                       1-4          0.25   512Mi     5Gi
merge-peer               I       1-4          0.25   512Mi     5Gi
merge-peer               II      1-4          0.25   512Mi     5Gi
merge-peer               III     1-4          0.25   512Mi     5Gi
monitor-ingest                   1            0.05   64Mi      64Mi
ollama                           1            2      24Gi      2Gi
ollama-exporter                  1            0.025  64Mi      64Mi
operator                         1            0.25   512Mi     256Mi
otel                             0            0.5    512Mi     512Mi
partition-cleaner                0            2      2Gi       5Gi
periodic-service                 0            1      1Gi       512Mi
pgbouncer                        1            2      512Mi     512Mi
pgbouncer-exporter               1            0.25   128Mi     128Mi
postgres                         1            4      16Gi      1Gi        100Gi
presidio-analyzer                1            4      8Gi       1Gi
presidio-anonymizer              1            1      1Gi       1Gi
prometheus                       1            2      2Gi       1Gi        50Gi
promwaltz                        0            0.25   256Mi     256Mi
prune-locks                      1            0.25   256Mi     256Mi
pushgateway                      1            1      512Mi     512Mi
query-head                       1           14      56Gi      50Gi
query-head-api                   1            1      2Gi       4Gi
query-peer                       3           14      56Gi      50Gi
queue-peer                       0            1      1Gi       512Mi
rabbitmq                         3            1      512Mi     512Mi      5Gi
reaper                           1            0.5    512Mi     512Mi
reloader                         1            1      512Mi     256Mi
renderer                         2            6      8Gi       512Mi
semantix-agent                   1            0.5    1Gi       256Mi
semantix-llm                     1            0.5    2Gi       512Mi
semantix-peer                    0            0.5    1Gi       256Mi
silence-linode                   0            0.25   256Mi     256Mi
spill-controller                 0            1      1Gi       256Mi
stale-job-monitor                1            0.25   256Mi     256Mi
summary-indexer                  0            2      2Gi       5Gi
summary-peer                     0            3      4Gi       5Gi
superset                         1            2      2Gi       512Mi
superset-init-db                 1            0.5    512Mi     512Mi
syslog-config-copy               1            0.06   64Mi      64Mi
syslog-ingest                    1            0.25   512Mi     512Mi
syslog-ingest-watcher            1            0.25   128Mi     256Mi
task-monitor                     1            0.25   256Mi     256Mi
thanos                           0            4      8Gi       1Gi
tooling                          0            1      1Gi       16Gi
traefik                          2            2      512Mi     256Mi
traefik-cfg                      0            1      1Gi       256Mi
tulugaq                          1            1      1Gi       512Mi
turbine-api                      2            1      2Gi       512Mi
turbine-api-worker               2            1      1Gi       512Mi
ui                               1            0.25   256Mi     256Mi
usagemeter                       1            0.25   256Mi     256Mi
validator                        1            0.5    512Mi     512Mi
validator-indexer                0            0.5    1536Mi    512Mi
vector                           0            0.5    512Mi     512Mi
version                          1            0.125  64Mi      64Mi
wait-dep                         1            0.025  50Mi      50Mi
wait-for-db-access               1            0.5    1Gi       512Mi
zookeeper                        3            0.5    512Mi     512Mi      512Mi

In Hydrolix version 5.11 and later, hdx-scaler-go also appears in the service list.

Node size recommendations⚓︎

16 vCPU nodes

For eval and prod deployments, use 16 vCPU nodes:

  • EKS: c5n.4xlarge
  • GKE: n2-standard-16
  • LKE: Dedicated 32 GB

32 vCPU nodes

For mega deployments, use 32 vCPU nodes:

  • EKS: c5n.9xlarge
  • GKE: n2-standard-32
  • LKE: Dedicated 64 GB