Skip to content

Advanced Merge Controller Configuration

The merge-controller uses efficiency as the primary signal to determine the pace of merging available partitions. Under unusual data ingestion circumstances, such as high traffic events or backfilling, you can override the core functionality of merge-controller with a Kubernetes ConfigMap. Memory usage can also be influenced, applying globally to the cluster, projects, or tables. Diagnostic logging and debugging information is also available for the application. This page shows how to inject configuration data into application pods.

merge-controller tuning carries risk

Tuning the configuration file for merge-controller can render it inoperable. An incorrect configuration won't result in invalid partitions or corrupt data, but it can bring the merging of partitions to a halt.

Format⚓︎

The merge-controller override configuration is a YAML file that's loaded and parsed during process bootstrapping. A fully defined configuration contains the following specs:

  • constructor
  • coefficient_provider

  • merge_target_overrides

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
    constructor:
  fetch_limit: 5000
  cooldown:
    strategy: Exponential
    efficiency_floor: 0.0
    efficiency_ceil: 1.0
    min_cooldown_sec: 0
    max_cooldown_sec: 60
    factor: 2.0
coefficient_provider:
  strategy: EWMA
  alpha: 0.7
merge_target_overrides:
  per_project:
    3c0bb3f1-564b-41e2-890b-cb476a1d5e8c: #PROJECT ID
      per_table:
        2322bd6f-e416-4803-aa65-39a5a2e53fcd: #TABLE ID
          additions:
            new_table_specific:
              pool: merge-iii
              memory: 4 GB
              bounds:
                start: 90d
                end: 91d
              memory_coefficient: 1.0
          patches:
            medium:
              memory: 1 GB
              bounds:
                start: 0ms
                end: 10m
      additions:
        new_project_specific:
          pool: merge-iii
          memory: 4 GB
          bounds:
            start: 90d
            end: 91d
          memory_coefficient: 1.0
      patches:
        medium:
          memory: 1 GB
          bounds:
            start: 0ms
            end: 10m
  additions:
    new_global:
      pool: merge-iii
      memory: 4 GB
      bounds:
        start: 90d
        end: 91d
      memory_coefficient: 1.0

Partition stream constructor⚓︎

Partitions are retrieved from the catalog as a stream of entries defined by a merge target range. Once the stream ends, it's reconstructed and the process repeats. Continuously re-issuing the catalog query is expensive and wasteful, so constructor employs an adaptive cooldown with configurable strategies. There are two available cooldown strategies:

  • static: original strategy used by merge-head
  • exponential: constructor calculates cooldown duration according to the efficiency of the stream

Efficiency⚓︎

The merge-controller tracks all partitions active in the merge system, whether that's actively being merged by a merge-peer, or awaiting candidate construction in an internal bucket.

When partitions are sourced from the catalog, they're de-duplicated against the set of currently active partitions. Each stream tracks the number of new and duplicate partitions retrieved from the catalog, and at the end of the stream produces an efficiency value reflecting how "useful" the stream was. For example, a stream that contained only partitions currently unknown to merge-controller (no duplicates) has an efficiency of 1.0. Conversely, a stream existing entirely of duplicates has an efficiency of 0.0.

This efficiency calculation is used to determine the cooldown, or how long we should delay before recreating the stream to locate partitions. The more efficient a stream, the more often we'll query. The more duplicates in the stream, the less often we'll query.

1
2
3
4
5
6
7
8
9
constructor:
  fetch_limit: 5000
  cooldown:
    strategy: Exponential
    efficiency_floor: 0.2
    efficiency_ceil: 0.8
    min_cooldown_sec: 0
    max_cooldown_sec: 60
    factor: 2.0

Set the number of rows to be requested per batch.

1
  fetch_limit: 5000

Select the Exponential cooldown strategy. Acceptable values are Static and Exponential.

1
2
  cooldown:
    strategy: Exponential

Set the endpoints for efficiency normalization. For example, an observed efficiency below the efficiency_floor of 0.2 would instead be capped to 0.2 and an observed efficiency above efficiency_ceil will instead be capped to 0.8.

  • Valid values are between 0.0 and 1.0 inclusive.

  • efficiency_floor must be less than efficiency_ceil.

    1
2
        efficiency_floor: 0.2
    efficiency_ceil: 0.8

Set the minimum and maximum cooldown in seconds. For example, given an efficiency of 0.2 (the lowest possible in this config), the cooldown would be 60 seconds, while an efficiency of 0.8 (the highest possible in this config), the cooldown would be 0 seconds.

  • min_cooldown_sec must be less than max_cooldown_spec

    1
2
        min_cooldown_sec: 0
    max_cooldown_sec: 60

Set the scale factor for calculating cooldown relative to observed efficiency.

  • A value of 1.0 is effectively linear

  • factor must be >= 0

    1
        factor: 2.0

Memory coefficient calculation⚓︎

The memory coefficient is a multiplier applied to partitions during bin packing calculations. It accounts for inaccuracies in the estimated memory required for merging. merge-controller can observe the accuracy of the coefficient for any given merge operation. This allows for more intelligent adjustment of the memory coefficient as merge operations are performed.

There are currently two available strategies for managing the memory coefficient:

  • Static defines a static value pulled from the existing merge settings on the table configuration

  • EWMA observes the difference between estimated memory requirements and actual memory requirements as reported by turbine. That difference represents the memory coefficient. Each value is recorded using an Exponentially-Weighted Moving Average. The current value is used for any future merge operations, and is iteratively adjusted

    1
2
3
    coefficient_provider:
  strategy: EWMA
  alpha: 0.7

Select the EWMA coefficient calculation strategy. Acceptable values are Static and EWMA.

1
  strategy: EWMA

Set the alpha value for Exponentially-Weighted Moving Average calculation. The value must be positive and comprised between 0.0 and 1.0. A larger alpha places more importance on recent observations.

1
  alpha: 0.7

Merge target additions and overrides⚓︎

Merge targets define a set of parameters for locating and combining partitions to be merged. Canonically, merge-head supports three hard-coded targets: small, medium, and large. Each of them consists of:

  • time bounds relative to now
  • a memory target
  • a pool on which the merge should be executed.

It's important to note the existing targets' bounds don't overlap, but instead abut.

With merge-controller you can override various parameters of the existing merge targets, as well as introduce additional targets. Overrides and additions can be applied globally, per project, or per table.

  • Do not to define overlapping targets, or they'll fight one another for work
  • A wider target range results in more partitions to consider, which in turn requires more memory

The most specific definition takes precedence. For example, if the large target is overridden to span 100 days instead of 90 at the global level, but a specific table overrides it to 50 days, the table override will be used, not the global override.

Below is a merge_target_override definition that has additions and overrides at the global, project, and table levels:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
merge_target_overrides:
  per_project:
    {project_ID}:
      per_table:
        {table_ID}:
          additions:
            new_table_specific:
              pool: merge-iii
              memory: 4 GB
              bounds:
                start: 90d
                end: 91d
              memory_coefficient: 1.0
          patches:
            medium:
              memory: 1 GB
              bounds:
                start: 0ms
                end: 10m
      additions:
        new_project_specific:
          pool: merge-iii
          memory: 4 GB
          bounds:
            start: 90d
            end: 91d
          memory_coefficient: 1.0
      patches:
        medium:
          memory: 1 GB
          bounds:
            start: 0ms
            end: 10m
  additions:
    new_global:
      pool: merge-iii
      memory: 4 GB
      bounds:
        start: 90d
        end: 91d
      memory_coefficient: 1.0

Add a new target called new_table_specific that only applies to one table.

1
2
3
4
5
6
7
merge_target_overrides:
  per_project:
    3c0bb3f1-564b-41e2-890b-cb476a1d5e8c:
      per_table:
        2322bd6f-e416-4803-aa65-39a5a2e53fcd:
          additions:
            new_table_specific:

Direct any candidates from new_table_specific merge target to specific-pool merge-peer pool.

1
2
            new_table_specific:
              pool: special-pool

Build candidates for new_table_specific target with a memory requirement of up to 4 GB.

1
2
            new_table_specific:
              memory: 4 GB

Target partitions containing data with a primary between 90 days ago, and 91 days ago from now.

1
2
3
4
            new_table_specific:
              bounds:
                start: 90d
                end: 91d

Set the memory coefficient for the new_table_specific target to 1.0.

1
2
            new_table_specific:
              memory_coefficient: 1.0

Application⚓︎

The advanced configuration is deployed to merge-controller with a ConfigMap. On deploy, merge-controller will map the contents of the ConfigMap named merge-controller into the container.

It is expected that ConfigMap contains a single file named config.yaml whose contents will be mounted inside the merge-controller container at /etc/merge/config.yaml.

The merge-controller handles both missing and malformed configurations by ignoring them and falling back to defaults. Errors reading or processing the configuration are logged at debug level, and can be enabled with a targeted logging config by adding the following entry to the HydrolixCluster config file:

1
2
  log_level:
    merge-controller: merge::config=debug

To apply the config defined at the top of this document, save its contents to config.yaml, then issue the following command:

1
kubectl create configmap merge-controller --from-file=config.yaml

This will result in a ConfigMap that looks similar to this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
apiVersion: v1
data:
  config.yaml: |
    constructor:
      fetch_limit: 100
    coefficient_provider:
      ...
    merge_target_overrides:
      ...
kind: ConfigMap
metadata:
  name: merge-controller
  namespace: some_namespace