Scale Profiles
Overview
Deploying a development cluster for testing is not the same as deploying a production cluster. They will differ on resource scale and resiliency requirements. While each deployment needs to be turned to the use case at hand, we have simplified the process by supplying a few different scale profiles listed below.
Using a Scale Profile
To use a scale profile, add the tag scale-profile and a value to the hydrolixcluster.yaml file, for example:
scale_profile: dev
If you don't define a scale profile in your hydrolixcluster.yaml file, the cluster defaults to eval.
Available Profiles
Here's the list of profiles to choose from, from small to large:
-
dev- a deployment good for small tests on small clusters -
eval- a deployment suitable for trying out new features -
prod- a fully resilient production deployment (1-4 TB/day) -
mega- a fully resilient large scale production deployment (10-50 TB/day)
Profile Tuning
Profiles are a starting point, and can be overridden per service in your cluster configuration (
hydrolixcluster.yaml).
Node Size Reccomendations
16 vCPU Nodes
Note: For
evalandprod-scale deployments, we recommend using 16 vCPU nodes:
- EKS c5n.4xlarge
- GKE n2-standard-16
- LKE Dedicated 32 GB
32 vCPU Nodes
Note: For
mega-scale production deployments, we recommend using 32 vCPU nodes:
- EKS c5n.9xlarge
- GKE n2-standard-32
- LKE Dedicated 64 GB
Examining Scale Profiles
List the resources currently defined by each scale profile for your particular version of Hydrolix by querying the HKW service. Make sure to include the Hydrolix version number and your Kubernetes provider. The allowed Kubernetes provider values are lke, gke, eks, and generic.
For example, to find the prod scale profile when running Hydrolix v4.14.2 on Linode's LKE service:
% curl 'https://hkw.hydrolix.live/v4.14.2/scale-defaults?profile=prod&kubernetes=lke'
service pool replicas cpu memory storage data_storage
-------------------- ------ ---------- ----- -------- --------- --------------
acme 1 0.25 256Mi 256Mi
akamai-siem-indexer 0 2 2Gi 5Gi
akamai-siem-peer 0 2 2Gi 5Gi
alter-head 0 0.25 256Mi 256Mi
alter-indexer 0 2 8Gi 5Gi
alter-peer 0 0.25 256Mi 5Gi
autoingest 0 0.25 256Mi 256Mi
batch-head 1 0.5 512Mi 256Mi
batch-indexer 0 2 4Gi 5Gi
batch-peer 1 2 4Gi 5Gi
decay 1 0.25 256Mi 256Mi
envoy 1 2 512Mi 256Mi
hdx-traefik-auth 0 0.25 256Mi 256Mi
init-cluster 1 0.5 512Mi 512Mi
init-turbine-api 1 0.5 512Mi 512Mi
intake-api 2 0.25 256Mi 256Mi
intake-head 0 4 4Gi 5Gi
intake-indexer 0 2 4Gi 5Gi
job-purge 1 0.25 256Mi 256Mi
kafka-indexer 0 2 2Gi 5Gi
kafka-peer 0 2 2Gi 5Gi
keycloak 1 4 4Gi 1Gi
kinesis-coordinator 0 0.25 256Mi 256Mi
kinesis-indexer 0 2 2Gi 5Gi
kinesis-peer 0 2 2Gi 5Gi
load-sample-project 1 0.5 512Mi 512Mi
log-vacuum 1 0.25 512Mi 256Mi
merge 4 0.25 512Mi 5Gi
merge I 4 0.25 512Mi 5Gi
merge II 4 0.25 512Mi 5Gi
merge III 4 0.25 512Mi 5Gi
merge-cleanup 1 0.25 256Mi 256Mi
merge-head 1 1 1Gi 512Mi
merge-indexer 0 2 4Gi 5Gi
merge-indexer I 0 2 4Gi 5Gi
merge-indexer II 0 2 6Gi 5Gi
merge-indexer III 0 2 12Gi 5Gi
merge-peer 1-4 0.25 512Mi 5Gi
merge-peer I 1-4 0.25 512Mi 5Gi
merge-peer II 1-4 0.25 512Mi 5Gi
merge-peer III 1-4 0.25 512Mi 5Gi
monitor-ingest 1 0.05 64Mi 64Mi
operator 1 0.25 512Mi 256Mi
otel 0 0.5 512Mi 512Mi
partition-vacuum 1 0.5 2Gi 512Mi
postgres 1 4 8Gi 1Gi 100Gi
prometheus 1 2 2Gi 1Gi 50Gi
promwaltz 0 0.25 256Mi 256Mi
prune-locks 1 0.25 256Mi 256Mi
pushgateway 1 1 512Mi 512Mi
query-head 1 14 28Gi 50Gi
query-peer 3 14 28Gi 50Gi
rabbitmq 3 1 512Mi 512Mi 5Gi
reaper 1 0.5 512Mi 512Mi
redpanda 3 4 8Gi 1Gi 1Ti
refresh-job-statuses 1 0.25 256Mi 256Mi
rejects-vacuum 1 0.25 256Mi 256Mi
stale-job-monitor 1 0.25 256Mi 256Mi
stream-head 2 4 4Gi 5Gi
stream-indexer 0 2 4Gi 5Gi
stream-peer 2-4 3 4Gi 5Gi
summary-indexer 0 2 2Gi 5Gi
summary-peer 2-4 3 4Gi 5Gi
task-monitor 1 0.25 256Mi 256Mi
tooling 0 1 1Gi 16Gi
traefik 2 2 512Mi 256Mi
traefik-cfg 0 1 1Gi 256Mi
turbine-api 2 1 1Gi 512Mi
ui 1 0.25 256Mi 256Mi
usagemeter 1 0.25 256Mi 256Mi
validator 1 0.5 512Mi 512Mi
validator-indexer 0 0.5 1536Mi 512Mi
vector 0 0.5 512Mi 512Mi
version 1 0.25 256Mi 256Mi
zookeeper 3 0.5 512Mi 512Mi 512Mi
Another example for Hydrolix version 4.14.2 on Amazon EKS, using the mega scale:
% curl 'https://hkw.hydrolix.live/v4.14.2/scale-defaults?profile=mega&kubernetes=eks'
service pool replicas cpu memory storage data_storage
-------------------- ------ ---------- ----- -------- --------- --------------
acme 1 0.25 256Mi 256Mi
akamai-siem-indexer 0 2 2Gi 5Gi
akamai-siem-peer 0 2 2Gi 5Gi
alter-head 0 0.25 256Mi 256Mi
alter-indexer 0 2 8Gi 5Gi
alter-peer 0 0.25 256Mi 5Gi
autoingest 0 0.25 256Mi 256Mi
batch-head 1 0.5 512Mi 256Mi
batch-indexer 0 2 4Gi 5Gi
batch-peer 1 2 4Gi 5Gi
decay 1 0.25 256Mi 256Mi
envoy 1 4 512Mi 256Mi
hdx-traefik-auth 0 0.25 256Mi 256Mi
init-cluster 1 0.5 512Mi 512Mi
init-turbine-api 1 0.5 512Mi 512Mi
intake-api 2 0.25 256Mi 256Mi
intake-head 0 6 6Gi 10Gi
intake-indexer 0 2 5Gi 5Gi
job-purge 1 0.25 256Mi 256Mi
kafka-indexer 0 2 2Gi 5Gi
kafka-peer 0 2 2Gi 5Gi
keycloak 1 4 4Gi 1Gi
kinesis-coordinator 0 0.25 256Mi 256Mi
kinesis-indexer 0 2 2Gi 5Gi
kinesis-peer 0 2 2Gi 5Gi
load-sample-project 1 0.5 512Mi 512Mi
log-vacuum 1 0.25 512Mi 256Mi
merge 8 0.25 512Mi 5Gi
merge I 8 0.25 512Mi 5Gi
merge II 8 0.25 512Mi 5Gi
merge III 8 0.25 512Mi 5Gi
merge-cleanup 1 0.25 256Mi 256Mi
merge-head 1 1 2Gi 512Mi
merge-indexer 0 2 4Gi 5Gi
merge-indexer I 0 2 4Gi 5Gi
merge-indexer II 0 2 6Gi 5Gi
merge-indexer III 0 2 12Gi 5Gi
merge-peer 2-8 0.25 512Mi 5Gi
merge-peer I 2-8 0.25 512Mi 5Gi
merge-peer II 2-8 0.25 512Mi 5Gi
merge-peer III 2-8 0.25 512Mi 5Gi
monitor-ingest 1 0.05 64Mi 64Mi
operator 1 0.25 512Mi 256Mi
otel 0 0.5 512Mi 512Mi
partition-vacuum 1 0.5 2Gi 512Mi
postgres 1 6 16Gi 1Gi 100Gi
prometheus 1 4 12Gi 1Gi 50Gi
promwaltz 0 0.25 256Mi 256Mi
prune-locks 1 0.25 256Mi 256Mi
pushgateway 1 1 512Mi 512Mi
query-head 1 32 85Gi 50Gi
query-peer 10 32 85Gi 50Gi
rabbitmq 3 1 512Mi 512Mi 5Gi
reaper 2 0.5 1Gi 512Mi
redpanda 6 12 8Gi 1Gi 1Ti
refresh-job-statuses 1 0.25 256Mi 256Mi
rejects-vacuum 1 0.25 256Mi 256Mi
stale-job-monitor 1 0.25 256Mi 256Mi
stream-head 3-8 6 6Gi 10Gi
stream-indexer 0 2 5Gi 5Gi
stream-peer 4-16 3 4Gi 5Gi
summary-indexer 0 2 2Gi 5Gi
summary-peer 4-16 3 4Gi 5Gi
task-monitor 1 0.25 256Mi 256Mi
tooling 0 1 1Gi 16Gi
traefik 3 4 512Mi 256Mi
traefik-cfg 0 1 1Gi 256Mi
turbine-api 2 1 1Gi 512Mi
ui 1 0.25 256Mi 256Mi
usagemeter 1 0.25 256Mi 256Mi
validator 1 0.5 512Mi 512Mi
validator-indexer 0 0.5 1536Mi 512Mi
vector 0 0.5 512Mi 512Mi
version 1 0.25 256Mi 256Mi
zookeeper 3 0.5 512Mi 512Mi 512Mi
Updated 2 months ago