Custom Scale Profiles

Custom scale profiles allow you to override the predefined service scale in the context of pools. While it is simple to override a primary pod's scale (i.e. replicas, memory, cpu, storage), this provides a mechanism to override a secondary container, like merge-indexer or stream-indexer that may be used in multiple pools.

As an example, imagine that our second generation merge-peer-ii pod is being evicted due to memory (OOM) on a prod scale cluster due to its secondary indexer container. If we check the scale profile, we can see that each generation has its own profile I, II, III.

$hkt scale-defaults -p prod

service              cloud    prof repl  cpu   memory store  pvc
-------------------  -----    ---- ---   ---   -----  -----  ---
..
merge-indexer        -        I    0     2     3Gi    5Gi    -
merge-indexer        -        II   0     2     6Gi    5Gi    -
merge-indexer        -        III  0     2     12Gi   5Gi    -

These three services are actually referencing the secondary container for the merge-peer service. If you attempt to override one using the scale: section of hydrolixcluster.yaml, it would actually override all three since the name alone is not unique. This is not what we want.

To override a single one, we create a custom scale profile and select a different memory amount. The syntax looks like this:

scale:
  profile:
   merge-indexer-2:
    merge-indexer:
     memory: 8Gi

Here we've defined a new profile called merge-indexer-2, but we can give it any name and add the increased memory for the merge-indexer container. All the other values will be inherited from the base profile as normal.

Now we need to create a pool that is based on this scale profile. We do that by overriding the scale_profile inside the pool section of the YAML file with the same name we defined above:

pools:
 - name: merge-peer-ii
   scale_profile: merge-indexer-2
   service: merge-peer

Once saved, the old merge-peer-ii pod(s) will be terminated and others with more memory will be started.

Changing Defaults (Merge)

When your cluster is created, certain components have default sizes. Some components have predefined profiles built in. Use these profile names and avoid the extra complexity of creating pools.

The profiles are named I, II, and III.

To change these default profiles to a different value, reference it in the scale section. For example, to change the profile III merge indexer to 15Gi of memory:

scale:
  profile:
   III:
    merge-indexer:
     memory: 15Gi