Skip to content

Hierarchy Adoption

This behavior was introduced in Hydrolix version 6.4.

This page describes how a cluster adopts and uses the Customer hierarchy. It also describes Config API behavior changes and services depending on the newly-published multi-org configuration files.

Behavior changes⚓︎

  • Project to customer assignment is immutable. A project can no longer be reassigned to another customer.
  • New model enforcement changes in the Config API affect project and table creation and administration.
  • A new customer RBAC check requires that an account is assigned to a Customer in addition to existing scope and permission checking.
  • The Config API publishes single-org, and new, multi-org formats of cluster configuration files.
  • See Switch consumers between formats to reconfigure running services to use old configs, in case of problems.
  • The background relationship establishment jobs stop, making all corrective re-assignment a manual operation. See relationship audit before upgrade.
  • Workflows for table, storage, and credential creation must account for customer assignment. See Example changed workflows.

Upgrade⚓︎

Preparation⚓︎

Before upgrading to v6.4, Hydrolix recommends performing a relationship audit.

The audit can help to discover unexpected configuration before the new parallel hierarchy goes into service after the upgrade.

During upgrade⚓︎

To upgrade, use the Hydrolix v6.4 | Upgrade instructions.

The init-turbine-api task may take longer than usual during the upgrade cycle. It must complete without error.

After upgrade⚓︎

  • The Project to Customer assignment is immutable.
  • The Config API continues to publishes the single-org config formats.
  • The Config API publishes new, multi-org configs, by default.
  • The Config API provides a full suite of v2 endpoints. These are now recommended.
  • All services read and use the new, multi-org configs published into object storage and the Config API v2 endpoints, by default.

Cluster operation⚓︎

All services in the cluster read the new multi-org formats from the primary storage path. Tunable settings can configure services to prefer the older single-org formats.

If you encounter problems that you suspect are related to the new multi-org formats, you can switch consumers between formats or even completely suppress multi-org publishing.

Config API publishing⚓︎

The Config API publishes internal configuration files necessary for applications in the cluster to provide services. These files describe objects like organizations, projects, tables, transforms, dictionaries, functions, and storage definitions.

Control format publication to the primary storage path by adjusting the tunable turbine_api_feature_publish_multi_org_configs. The tunable defaults to True:

Config type True False Storage path Notes
main configs yes no /config/v3/ File per customer, multi-org, new in v6.4
main configs yes (always) yes (always) /config/v2/ File per cluster, single-org

Suppress multi-org publishing⚓︎

Don't forget to switch consumers when suppressing multi-org publishing

If you disable multi-org publishing, use the instructions to switch any consumers to the single-org format.

Consumers of configuration files, such as intake and query services, may operate with stale configuration until they are switched to using the single-org configuration files.

  • Turn off multi-org publishing by setting tunable turbine_api_feature_publish_multi_org_configs=false.
  • Restore multi-org publishing by setting tunable turbine_api_feature_publish_multi_org_configs=true.

Switch consumers between formats⚓︎

Use the configurables shown in this section to switch services or groups of services away from the multi-org formats if something goes wrong.

Services inside the cluster read their configuration files directly from the primary object storage path.

Several tunables enable groups of services to roll back to the earlier format without downgrading the cluster software release.

Switch intake services⚓︎

Intake services respect the tunable intake_use_multi_org_config.

  • Set to True, the default, so intake services read the new v3, multi-org format from /config/v3/.
  • Set to False for intake services to use the older configuration format from /config/v2/.

Switch query and indexer services⚓︎

The query system and indexers both respect the tunable turbine_core_enable_multi_org.

  • Set to True, the default, so query and indexer services read the new multi-org format from /config/v3/.
  • Set to False for query and indexer services to read the older configuration format from /config/v2/.

Switch Kibana Gateway⚓︎

The Kibana Gateway depends on the Config API version specified in tunable kibana_gateway_config.config_api_version.

  • Set to v2, the default, to use the Config API v2 endpoints.
  • Set to v1 to configure Kibana Gateway to prefer the v1 endpoints.

Spark connector⚓︎

The Hydrolix Connector for Apache Spark runs outside the cluster and can't infer cluster configuration from the outside.

  • Upgrade the cluster to v6.4.
  • Restart Spark and use the software as before.

If, after upgrade, a cluster turns off multi-org publishing, Spark connector clients may not work correctly. To restore proper operation

  • Set Spark catalog flags multi_org=false and api_version=1. These aren't the default settings.
  • Restart Spark and use the software as before.

Config API changes⚓︎

Model enforcement changes⚓︎

These model enforcement changes are necessary to support the new hierarchy.

Linked endpoints below are all v2 endpoints. Unless otherwise indicated, the changes apply to both v1 and v2 endpoints.

  • Project to customer assignment is immutable. The PUT and PATCH /projects/:id endpoints return an HTTP 400 if the customer parameter in the request body differs from the existing object.
  • Customer ID is always required on projects now. POST /projects requires a customer ID. See also Example changed workflows.
  • Deployment ID moves from project to customer. Accounts with new permission deployment_id_customer can use the PUT /customers/:id/deployment_id to change deployment IDs. It completely replaces PUT /projects/:id/deployment_id. Projects continue to return the deployment ID as a read-only field derived from the customer. Accounts which could previously modify deployment IDs have the new permission.
  • Credentials and storages can't be detached from a customer if in use. The POST /customers/:id/remove_credential and POST /customers/:id/remove_storage endpoints return HTTP 400 if the credential or storage is still in use by any tables belonging to the customer.
  • Storages must be assigned to a customer before use. Tables can't use a storage object until it's assigned to the table's customer. Use POST /customers/:id/add_storage.
  • Credentials must be assigned to a customer before use. Tables, sources, and jobs can't use a credential until it's assigned to the customer. Use POST /customers/:id/add_credential. Credentials must belong to every customer using the storage to be used with the storage.

Customer RBAC check⚓︎

The Config API adds a new check on customer assignment for the requesting account, before allowing an operation.

No accounts are exempt from this check, not even those with the super_admin role. To perform management operations on objects like projects, tables, transforms, columns, even an account with super_admin role must be assigned to the customer. When creating a new customer object, the Config API also assigns accounts with global scope or super_admin role to the newly created customer.

See instructions for how to manually assign users.

The additional RBAC check remains until Customers become Orgs

The Customer object is a secondary hierarchy. See Organizations.

In a future release, Customer objects become Orgs and the check remains implicitly as a part of the permissions system.

v2 endpoints⚓︎

The Config API publishes a complete set of v2 endpoints. New endpoints and features will only be added to the v2 namespace.

For an explanation of the shorter v2 endpoints and changes to calling conventions, see Config API v2 Endpoints.

The v1 endpoints are deprecated and remain supported in v6.4.

Config API v1 endpoints are deprecated

All v1 endpoints are deprecated. They remain available for client transition.

Background job behavior⚓︎

The background relationship establishment jobs stop running as a part of the upgrade to Hydrolix v6.4.

While Hydrolix v6.4 runs, a single, new background job updates the query options setting for each Customer whenever the original, single Org query options change.

Example changed workflows⚓︎

New storage and credential objects must be assigned to a customer before use.

Similarly, new projects must be assigned to a customer starting in v6.4. This changes the table creation workflow.

This section includes examples of the changed workflow when creating tables, credentials, or storage locations.

Create table - UI⚓︎

Use the following steps to create a new customer, project, and table in the UI.

  1. In the top right corner of the screen, click the + Add new button.
  2. Select the Customer option in the right sidebar.
  3. Fill in the fields. You must provide a valid customer name, at a minimum.
  4. Click Create.
  5. In the top right corner of the screen, click the + Add new button.
  6. Select the Table option in the right sidebar.
  7. Switch the customer selector in the top navigation bar to the newly created customer.
  8. From the Select project drop down, select + Add new, which displays the Project name field.
  9. Fill in the Project name and Table name fields. Optionally, add a Description.
  10. Click Create table.

Create table - API⚓︎

Use the following steps to create a new customer, project, and table with the API.

  1. Create a customer, if not using an existing customer. Accounts with global scope or super_admin role are assigned to the new customer.
  2. Save the customer ID for use in subsequent calls.
  3. Create a project using the customer and organization IDs.
    • The organization ID, org, is required to create a project until retirement of the original hierarchy. See Organizations.
    • The customer ID, customer, is required for project creation starting in v6.4.
  4. (optional) Create storage and Create credential to support the new customer's tables.
  5. Add storages to Customer using the customer ID.
  6. Add credentials to Customer using the customer ID.
  7. Create table using the project ID.

Create credential - API⚓︎

Use the following steps to create a new credential using the API. The Hydrolix UI lacks support for this workflow in v6.4.

  1. (optional) Use Get credential types to list supported types and required fields.
  2. Create a credential with the specified credential type.
  3. Collect the customer IDs for use in customer assignment.
  4. Add the credential to customers using the customer IDs from the prior step.
  5. The new credential is now available to table sources, autoingest settings, batch jobs, or storage definitions for any newly-assigned customer.

For detailed examples of assignment invocations, see Assign credentials.

Create storage - API⚓︎

Use the following steps to create a new storage location definition using the API. The Hydrolix UI lacks support for this workflow in v6.4.

  1. Create storage.
  2. Collect the customer IDs for use in customer assignment.
  3. Add the storage to customers using the customer IDs from the prior step.
  4. The storage location is now available to any newly-assigned customer.

Credentials linked to a storage object are also assigned

If the storage object requires a credential, both the storage object and the linked credential are assigned to the customer.

Once assigned, the storage ID can be used as the default storage for a table or with the spread list or column-value mapping features.

For detailed examples of assignment invocations, see Assign storages.