Custom Scale Profiles

Custom scale profiles allow you to override the predefine service scale in the context of pools. While it is simple to override a primary pods scale (replicas, memory, cpu, storage), this provides a mechanism to override a secondary container, like merge-indexer or stream-indexer that may be used in multiple pools.

As an example, imagine that our second generation merge-peer-ii pod is being evicted due to memory (OOM) on a prod scale cluster due to it's secondary indexer container. If we check the scale profile we can see that each generation has it own profile I, II, III.

$hkt scale-defaults -p prod

service              cloud    prof repl  cpu   memory store  pvc
-------------------  -----    ---- ---   ---   -----  -----  ---
..
merge-indexer        -        I    0     2     3Gi    5Gi    -
merge-indexer        -        II   0     2     6Gi    5Gi    -
merge-indexer        -        III  0     2     12Gi   5Gi    -

These three services are actually referencing the secondary container for the merge-peer service. If you attempt to override one using the scale: section of hydrolixcluster.yaml, it would actually override all three since the name alone is not unique. Not what we want.

To override a single one, we create a custom scale profile and select a different memory amount. The syntax looks like this:

scale:
  profile:
   merge-indexer-2:
    merge-indexer:
     memory: 8Gi

Here I've defined a new profile call merge-indexer-2 but you can give it any name and added the increase memory for the merge-indexer container. All the other values will be inherited from the base profile as normal.

Now we need to create a pool that is based on this scale profile. We do that by overriding the scale_profile inside the pool section of the yaml file with the same name you defined above.

pools:
 - name: merge-peer-ii
   scale_profile: merge-indexer-2
   service: merge-peer

Once saved, the old merge-peer-ii pod(s) will be terminated and another with more memory will be started.

Changing Defaults (Merge)

When your cluster is created there will be some default sizing put in place for certain components.

To change the merge-peers for they have default profiles built in. The profiles are named I, II, III.

This means to change these default profiles to a different value you can reference it in the scale section. E.g. to change profile III Indexer to 15Gi of memory you would put:

scale:
  profile:
   III:
    merge-indexer:
     memory: 15Gi