Skip to content

Establish Relationships

This behavior was introduced in Hydrolix version 6.2.

A cluster running Hydrolix v6.2 and v6.3 builds new relationships in the parallel Customer hierarchy created in an earlier transition stage.

In a future release, all services in the cluster begin to depend on the parallel hierarchy and the relationships built by these background jobs.

This page describes the automation, audit, and manual assignment tooling.

Relationship audit⚓︎

Every cluster is configured differently and contains different tenants. While the background jobs construct the necessary relationships, each cluster may have special cases, such as a tenant with multiple projects or an account with access to multiple customer's projects.

An audit identifies where the linking rules and background jobs haven't constructed all of the desired relationships from customers to users, credentials, and storage locations.

Only an administrator with familiarity with the tenants and configuration can determine if the future Orgs for the cluster are correctly modeled. The Customer objects are the future Orgs for this cluster. See also Organizations.

Audit recommendations⚓︎

Hydrolix recommends administrators perform the following audit while a cluster is running v6.3 and before upgrading to v6.4:

  • Audit user and service account permissions and scopes for accuracy and intent. Do users have correct permissions? The background jobs update customer assignments based on account permissions.
  • Ensure every project has an assigned customer. See Find orphaned projects. Orphaned projects are ignored in a future release, meaning that services won't work.
  • Ensure projects are assigned correctly if a customer holds multiple projects. See Assign project to customer.
  • Review user and service accounts. Are all accounts assigned to the correct customer? Are there old accounts which should be deleted? See Find unassigned objects.
  • Review storage and credentials. Are the assignments correct? Are there vestigial objects that should be deleted? See Find unassigned objects.
  • If certain accounts should have access to multiple tenants (customers), are they also assigned to those customers? See Find objects assigned to multiple customers.
  • Are there storage or credential objects used across multiple customers? Is that expected? See Find objects assigned to multiple customers.

Audit endpoints⚓︎

Use auditing filters on endpoints to examine the relationships of objects with the new Customer hierarchy.

One filter is available for projects

  • customer_id_is_none=True returns projects lacking a customer; correct before upgrading beyond v6.3

Two filters are available on users and service accounts, credentials, storages:

  • unassigned=True returns objects lacking a customer; correct missing assignments before upgrading beyond v6.3
  • assigned_multiple=True returns objects assigned to more than one customer

Find orphaned projects⚓︎

Use v1 /config/v1/orgs/:org_id/projects/?customer_id_is_none=True.

1
2
3
4
5
6
7
8
curl \
  --get \
  --url "https://hostname.hydrolix.live/config/v1/orgs/ae5e3698-b13a-4f8f-ab82-ad2fa391a1a8/projects/" \ # (1)!
  --data "customer_id_is_none=True" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"
  1. Replace the example with the original cluster Org UUID.

Unassigned users won't work in a future release. Delete the user account or assign it to the correct customer.

Find unassigned objects⚓︎

Use v1 /config/v1/users/?unassigned=True.

1
2
3
4
5
6
7
8
curl \
  --get \
  --url "https://hostname.hydrolix.live/config/v1/users/" \
  --data "unassigned=True" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"

Unassigned users won't work in a future release. Delete the user account or assign it to the correct customer.

Use v1 /config/v1/service_accounts/?unassigned=True.

1
2
3
4
5
6
7
8
curl \
  --get \
  --url "https://hostname.hydrolix.live/config/v1/service_accounts/" \
  --data "unassigned=True" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"

Unassigned service accounts won't work in a future release. Delete the user account or assign it to the correct customer.

Use v2 /config/v2/credentials/?unassigned=True.

1
2
3
4
5
6
7
8
curl \
  --get \
  --url "https://hostname.hydrolix.live/config/v2/credentials/" \
  --data "unassigned=True" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"

Unassigned credentials won't work in a future release. Delete the credential or assign it to the correct customer.

Use v2 /config/v2/storages/?unassigned=True.

1
2
3
4
5
6
7
8
curl \
  --get \
  --url "https://hostname.hydrolix.live/config/v2/storages/" \
  --data "unassigned=True" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"

Unassigned storages won't work in a future release. Delete the storages or assign it to the correct customer.

Find objects assigned to multiple customers⚓︎

Use v1 /config/v1/users/?assigned_multiple=True.

1
2
3
4
5
6
7
8
curl \
  --get \
  --url "https://hostname.hydrolix.live/config/v1/users/" \
  --data "assigned_multiple=True" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"

User accounts with more than one Customer assignment might have super_admin role or have a reason to see or manage resources for all mentioned customers.

Use v1 /config/v1/service_accounts/?assigned_multiple=True.

1
2
3
4
5
6
7
8
curl \
  --get \
  --url "https://hostname.hydrolix.live/config/v1/service_accounts/" \
  --data "assigned_multiple=True" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"

Service accounts with more than one Customer assignment can see or manage resources for all mentioned customers.

Use v2 /config/v2/credentials/?assigned_multiple=True.

1
2
3
4
5
6
7
8
curl \
  --get \
  --url "https://hostname.hydrolix.live/config/v2/credentials/" \
  --data "assigned_multiple=True" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"

Credentials with more than one Customer assignment are used in a storage, table source, or job for different customers.

Use v2 /config/v2/storage/?assigned_multiple=True.

1
2
3
4
5
6
7
8
curl \
  --get \
  --url "https://hostname.hydrolix.live/config/v2/storages/" \
  --data "assigned_multiple=True" \
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"

Storages with more than one Customer assignment are available to different customers.

Primary storage will be assigned to multiple if used outside the hydro project

The required hdx_primary storage is always assigned to the system Customer containing the hydro project.

It may be present in the output, if tables in other projects use the primary storage. This is not a problem.

Manual assignment⚓︎

Administrators can use Config API endpoints to specify manual assignment. These are explicit and take precedence over derived assignments.

Assign projects⚓︎

With the Customer UUID, use the v1 Patch project endpoint to change the Customer associated with a project.

A project must be assigned to a customer to remain active in a future release. There's no way to remove a project's customer assignment. Instead, remove the project.

1
2
3
4
5
6
7
8
9
curl \
  --request PATCH \
  --url https://hostname.hydrolix.live/config/v1/orgs/${HDX_ORGID}/projects/${HDX_PROJECTID}/ \ # (1)!
  --data '{ "customer": "a209969c-5fed-42c7-97a1-9358e57377ef" }' \ # (2)!
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json'
  1. Replace the original Org UUID and desired Project UUID.
  2. Replace the example with the correct Customer UUID in your cluster.

In a future release

  • Customer reassignment capability will be removed, preventing changes to a project's customer.
  • Projects lacking a customer are ignored. No services in the cluster will consider the project.

Correct assignment is important preparation for a future release, when the customer hierarchy is adopted for running services.

Assign users⚓︎

With the Customer and User UUIDs, use the v1 endpoints to change assignments

Add a user or service account to a customer endpoint to create a new association.

1
2
3
4
5
6
7
8
curl \
  --request POST \
  --url "https://hostname.hydrolix.live/config/v1/customers/ee25654c-1e1b-47a2-a867-961341ab848c/add_user/" \ # (1)!
  --data '{"users":[{"uuid":"09a9934f-6036-4c48-bd8f-117292e21f5b"}]}' \ # (2)!
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"
  1. Replace the example with the Customer UUID.
  2. Replace the example with the User UUID. Assigning multiple Users to the same Customer is valid in a single call.
Expected Output
1
{"success":true,"message":"1 accounts were added to customer customername"}

Remove user or service account from Customer endpoint to remove an existing association.

1
2
3
4
5
6
7
8
curl \
  --request POST \
  --url "https://hostname.hydrolix.live/config/v1/customers/ee25654c-1e1b-47a2-a867-961341ab848c/remove_user/" \ # (1)!
  --data '{"users":[{"uuid":"09a9934f-6036-4c48-bd8f-117292e21f5b"}]}' \ # (2)!
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"
  1. Replace the example with the Customer UUID.
  2. Replace the example with the User UUID. Removing multiple Users from the same Customer is valid in a single call.
Expected Output
1
{"success":true,"message":"1 accounts were removed from customer customername"}

Assign storages⚓︎

With the Customer and Storage UUIDs, use the v1 endpoints to change assignments

Add storages to Customer endpoint to create a new association.

1
2
3
4
5
6
7
8
curl \
  --request POST \
  --url "https://hostname.hydrolix.live/config/v1/customers/ee25654c-1e1b-47a2-a867-961341ab848c/add_storage/" \ # (1)!
  --data '{"storages":[{"uuid":"a2b8a015-4e4e-468d-8479-0b5605166bea"}]}' \ # (2)!
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"
  1. Replace the example with the Customer UUID.
  2. Replace the example with the Storage UUID. Assigning multiple Storages to the same Customer is valid in a single call.
Expected Output
1
{"success":true,"message":"1 storages were added to customer customername"}

Remove storages from Customer endpoint to remove an existing association.

1
2
3
4
5
6
7
8
curl \
  --request POST \
  --url "https://hostname.hydrolix.live/config/v1/customers/ee25654c-1e1b-47a2-a867-961341ab848c/remove_storage/" \ # (1)!
  --data '{"storages":[{"uuid":"09a9934f-6036-4c48-bd8f-117292e21f5b"}]}' \ # (2)!
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"
  1. Replace the example with the Customer UUID.
  2. Replace the example with the Storage UUID. Removing multiple Storages from the same Customer is valid in a single call.
Expected Output
1
{"success":true,"message":"1 storages were removed from customer customername"}

Assign credentials⚓︎

With the Customer and Credential UUIDs, use the v1 endpoints to change assignments

Add credentials to Customer endpoint to create a new association.

1
2
3
4
5
6
7
8
curl \
  --request POST \
  --url "https://hostname.hydrolix.live/config/v1/customers/ee25654c-1e1b-47a2-a867-961341ab848c/add_credential/" \ # (1)!
  --data '{"credentials":[{"uuid":"0d285aea-98af-43fd-b9f4-f76d09fc77b4"}]}' \ # (2)!
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"
  1. Replace the example with the Customer UUID.
  2. Replace the example with the Credential UUID. Assigning multiple Credentials to the same Customer is valid in a single call.
Expected Output
1
{"success":true,"message":"1 credentials were added to customer customername"}

Remove credentials from Customer endpoint to remove an existing association.

1
2
3
4
5
6
7
8
curl \
  --request POST \
  --url "https://hostname.hydrolix.live/config/v1/customers/ee25654c-1e1b-47a2-a867-961341ab848c/remove_credential/" \ # (1)!
  --data '{"credentials":[{"uuid":"09a9934f-6036-4c48-bd8f-117292e21f5b"}]}' \ # (2)!
  --fail-with-body \
  --silent \
  --header "Authorization: Bearer ${HDX_TOKEN}" \
  --header "Content-Type: application/json"
  1. Replace the example with the Customer UUID.
  2. Replace the example with the Credential UUID. Removing multiple Credentials from the same Customer is valid in a single call.
Expected Output
1
{"success":true,"message":"1 credentials were removed from customer customername"}

Upgrade and downgrade notes⚓︎

Downgrading from v6.2 to an earlier release removes relationships. Earlier releases don't support relationships to customer objects.

Reset with different tunable⚓︎

Reconfigure while the cluster runs, without a downgrade.

Manual tunable reset instructions⚓︎

Use these instructions to change the turbine_api_feature_migrate_projects_to_orgs tunable while v6.2 or v6.3 is running without downgrading.

Switch tunable to create multiple customers⚓︎

Use these instructions if you want many Customer objects and you forgot to set tunable turbine_api_feature_migrate_projects_to_orgs to true.

  1. Confirm that you are running v6.2 or v6.3.
  2. Find the default customer ID.
  3. Delete the customer object using DELETE /config/v1/customers/{id}.
  4. Set tunable turbine_api_feature_migrate_projects_to_orgs to true.
  5. Wait for init-turbine-api job to finish.
  6. Connect to the Kubernetes cluster using k9s and select the namespace holding the Hydrolix cluster.
  7. Select the turbine-api pod and turbine-api container.
  8. Invoke a shell by pressing s.

  9. Execute this command, which will produce STDERR to terminal. It must exit cleanly.

    Create Customers From Projects
    1
    ./manage.py create_customers_from_projects

  10. Wait for customers to be created. This may take minutes on a large cluster.

  11. Execute this command, which will produce STDERR to terminal. It must exit cleanly.

    Re-run the v6.2 Release Task
    1
    ./manage.py release_6_2

  12. Exit the k9s utility by pressing Ctrl+c.

Object ownership and usage⚓︎

Storage and credential objects can be used by multiple customers. These objects are assigned to multiple customers.

Only one Customer can own a storage or credential object.

1
2
3
4
5
6
7
      "customers": [
        {
          "uuid": "ee25654c-1e1b-47a2-a867-961341ab848c",
          "name": "customer-name"
        }
      ],
      "owned_by_customer": "ee25654c-1e1b-47a2-a867-961341ab848c",

This configuration is common, for single- and multi-tenant clusters.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
      "customers": [
        {
          "uuid": "ee25654c-1e1b-47a2-a867-961341ab848c",
          "name": "customer-name"
        },
        {
          "uuid": "4ad54085-ff9a-4015-8b7f-f5e37025cdd6",
          "name": "another-customer"
        }
      ],
      "owned_by_customer": "ee25654c-1e1b-47a2-a867-961341ab848c",

Projects assigned to two different customers could use the same object storage and credentials, particularly in multi-tenant clusters. See also Find objects assigned to multiple customers

Rules for linking⚓︎

These rules describe how the background jobs determine assignment.

Use explicit manual assignment to assign additional relationships.

Users and service accounts⚓︎

The background job

  • links user and service accounts to the Customers holding projects and tables to which the account has permissions.
  • sets the default_customer if the account has only a single Customer relationship.
  • links every account to the Customer which owns the project named sample_project.
  • skips human-initiated invitations and manual customer assignments. These contain explicit intent and are never modified.
  • prevents any account from assignment to the system Customer, which owns the hydro project.

The background job re-evaluates these rules on every permissions or role change and modifies relationships accordingly.

This allows administrators to make role and scope changes and review the changes using the audit endpoints in an iterative loop.

Storage and credentials⚓︎

The background job

  • links the hdx_primary storage and its corresponding global credential to the Customer which owns the hydro project
  • links each storage object to each Customer whose tables depend on the storage location
  • links each credential object to each Customer whose storage, table, or table source requires the credential
  • sets owned_by_customer for each storage and credential, if there's only a single Customer relationship.

Background job behavior⚓︎

While Hydrolix v6.2 and v6.3 run, background jobs continuously construct relationships between invite, storage, and credential objects and the Customer containing each project. They also set defaults to prepare each Customer for future promotion to an Org object.

These continuously-running API worker jobs perform the work:

  • create_storage_relation_to_customers - link Customers to storage objects for their projects
  • create_credential_relation_to_customers - link Customers to credentials for their projects
  • create_invite_relation_to_customers - link Customers to their invited users
  • sync_user_relation_to_customers - link Customers to users with permissions to their projects
  • assign_storage_owners - for storage objects with only one Customer, mark it as owned by that Customer
  • assign_credential_owners - for credential objects with only one Customer, mark it as owned by that Customer
  • sync_user_default_customers - select a default customer for each User
  • create_default_customer_projects - Customer management job started in v6.1 continues to run on single-tenant clusters (tunable false)
  • create_customers_from_projects - Customer management job started in v6.1 continues to run on multi-tenant clusters (tunable true)