Skip to content

Google Pub/Sub

Overview⚓︎

Use Google Cloud Pub/Sub push subscriptions to stream data into Hydrolix. Pub/Sub sends message payloads to the Hydrolix HTTP Stream API, which runs your transform and writes the data to the specified table.

This integration works by configuring a Pub/Sub push subscription to send messages to the Hydrolix /ingest/event endpoint. The push subscription delivers the message payload to Hydrolix, where your transform processes it.

Performance limitations

Google Pub/Sub push subscriptions send one message per HTTP request, which creates significant overhead and can result in poor ingestion performance. For high-throughput workloads, consider alternative streaming methods such as Kafka or Kinesis that support batching multiple messages per request.

Architecture diagram showing data flow from Google Pub/Sub to Hydrolix HTTP Stream API

Before you begin⚓︎

Verify you have:

  • A Google Cloud project with Pub/Sub API enabled
  • A Pub/Sub topic with data to send to Hydrolix
  • Permission to create service accounts and push subscriptions in Google Cloud
  • The Hydrolix cluster endpoint URL
  • Permission to create credentials in Hydrolix
  • A Hydrolix project and table to receive the data. See Projects and Tables
  • A transform to process the incoming data. See Write Transforms

Setup steps⚓︎

  1. Create a Pub/Sub topic or use an existing topic.
  2. Create a Google Cloud service account with appropriate permissions.
  3. Download the service account JSON key file.
  4. Import the service account credentials into Hydrolix.
  5. Create a Hydrolix table and transform to receive the data.
  6. Create a push subscription that sends messages to the Hydrolix HTTP Stream API endpoint.
  7. Test the integration by publishing messages to the topic.

Create a service account⚓︎

Create a Google Cloud service account for Pub/Sub authentication.

  1. In the Google Cloud Console, go to IAM & Admin > Service Accounts.
  2. Select Create service account.
  3. Enter a Service account name. For example, hydrolix-pubsub-push.
  4. Optionally, enter a Service account description.
  5. Select Create and continue.
  6. Skip granting roles or assign minimal permissions as needed.
  7. Select Continue, then Done.

Download the service account key⚓︎

  1. In the Service Accounts list, select the service account you created.
  2. Open the Keys tab.
  3. Select Add key > Create new key.
  4. Select JSON.
  5. Select Create. The JSON key file downloads to your computer.

Protect the service account key

Store the JSON key file securely. Don't commit it to version control.

Import the service account into Hydrolix⚓︎

Import the Google Cloud service account credentials into Hydrolix so the platform can authenticate incoming Pub/Sub push requests.

  1. In the Hydrolix UI, go to Admin > Credentials.
  2. Select Add credential.
  3. For Name, enter a descriptive name. For example, pubsub-service-account.
  4. For Type, select GCP Service Account Keys.
  5. For Cloud, select GCP.
  6. Upload the JSON key file downloaded from Google Cloud.
  7. Select Save.

Configure a Hydrolix table and transform⚓︎

Define a table in your Hydrolix cluster to receive the data from Google Pub/Sub, then define a transform to process the data.

Create the destination table⚓︎

Create a table using the Hydrolix UI by selecting the Table option under the Add new menu on the upper right-hand corner of the screen. You can also use the Hydrolix Configuration API to create the table. See Create a Table via API.

Create the transform⚓︎

Google Pub/Sub push subscriptions deliver messages to Hydrolix, but only the message payload is processed and stored. Metadata like message IDs, timestamps, and attributes aren't stored in Hydrolix. This is similar to the intake stages, where the payload passes through the transform pipeline.

Design the transform to process the actual data payload that the application publishes to the Pub/Sub topic. The transform should match the structure of the message data, not the Pub/Sub envelope format.

For detailed information on creating transforms, see Write Transforms. To see the structure of the incoming data, consider using a catch-all transform.

Create a Pub/Sub push subscription⚓︎

Create a push subscription in Google Cloud to send messages to Hydrolix.

  1. In the Google Cloud Console, go to Pub/Sub > Subscriptions.
  2. Select Create subscription.
  3. Enter a Subscription ID. For example, hydrolix-push-subscription.
  4. Select the Pub/Sub topic to send to Hydrolix.
  5. For Delivery type, select Push.

  6. Enter the Endpoint URL for the Hydrolix HTTP Stream API:

    1
    https://<hostname>/ingest/event?table=<project>.<table>&transform=<transform>

    For example:

    1
    https://hydrolix.example.com/ingest/event?table=production.events&transform=pubsub_transform

  7. Configure authentication:

    • Select Enable authentication
    • For Authentication method, select OIDC token
    • Select the service account created earlier. For example, hydrolix-pubsub-push.
    • For Audience, enter the Hydrolix cluster URL. For example, https://hydrolix.example.com.

    Service account authentication

    Google Pub/Sub sends push requests with a JWT Bearer token in the Authorization header. Hydrolix validates this token using the service account credentials you imported. The service account email in the JWT must match a credential imported into Hydrolix.

  8. Configure message retention, acknowledgement deadline, and retry policy as needed:

    • Message retention: Set how long Pub/Sub retains unacknowledged messages. See Subscription message retention.
    • Acknowledgement deadline: Set the deadline to prevent message redelivery during processing delays.
    • Retry policy: Configure exponential backoff parameters. See Retry and error handling.
  9. Select Create.

For more information, see Create push subscriptions and Authenticate push requests.

Test the integration⚓︎

Publish a test message to verify the integration works.

  1. In the Google Cloud Console, go to Pub/Sub > Topics.
  2. Select the topic.
  3. Open the Messages tab.
  4. Select Publish message.
  5. Enter a test payload in JSON format that matches the transform schema.
  6. Select Publish.

Query the Hydrolix table to verify the message arrived.

Additional information⚓︎

Scaling⚓︎

Google Pub/Sub push subscriptions scale automatically for high throughput.

  • Endpoint capacity: Verify the Hydrolix cluster can handle the message volume.
  • Subscription parallelism: Pub/Sub sends messages to multiple endpoints simultaneously.
  • Message ordering: Standard push subscriptions don't guarantee message order. For ordered delivery, use ordering keys. See Message ordering.
  • Acknowledgement deadlines: Set acknowledgement deadlines to prevent message redelivery during processing delays.

Delivery guarantees

Standard push subscriptions don't guarantee message order. If ordered delivery is needed, configure ordering keys when creating the Pub/Sub topic. This ensures messages with the same ordering key are delivered in the order they were published.

For information about Pub/Sub quotas and limits, see Google Pub/Sub quotas and limits.

Retry and error handling⚓︎

When Hydrolix fails to process a message, the subscription's retry policy controls what happens:

  • Pub/Sub retries messages based on the subscription's retry policy
  • Send failed messages to a dead-letter topic for analysis. See Dead-letter topics.
  • Configure exponential backoff parameters to control retry behavior

Security⚓︎

  • Service account authentication: Configure OIDC token authentication in the push subscription settings. Google sends a JWT Bearer token that Hydrolix validates using the imported service account credentials. This ensures only authorized Google Cloud projects can send data to the cluster.
  • Network security: Use VPC Service Controls or Private Service Connect to restrict network access to the Hydrolix endpoints.
  • Credential management: Protect service account key files. Don't commit them to version control.
  • Principle of least privilege: Grant the service account only the minimum permissions needed.

Troubleshooting⚓︎

Verify service account configuration⚓︎

If messages aren't arriving in Hydrolix:

  1. Check the service account email: Verify the service account email configured in the push subscription matches the service account imported into Hydrolix.
  2. Verify credentials in Hydrolix: In the Hydrolix UI, go to Admin > Credentials and confirm the GCP service account credential exists and is properly configured.
  3. Check subscription status: In the Google Cloud Console, go to Pub/Sub > Subscriptions and check for delivery errors or failed push attempts.
  4. Review Hydrolix logs: Check the Hydrolix intake logs for authentication errors or validation failures.

Test message delivery⚓︎

  1. In the Google Cloud Console, go to Pub/Sub > Topics.
  2. Select the topic.
  3. Select Publish message to send a test message.
  4. Check the subscription's Metrics tab to see if messages were delivered successfully.
  5. Query the Hydrolix table to verify the data arrived.