Any services that leverage the core data that is sitting in object storage all act as separate and distinct processes.
Turbine makes object storage access "fast" by:
- Leveraging the massive file serving and redundancy capabilities of object storage.
- Using the Turbine Index which allows Turbine Query Service to quickly determine which files in object storage are needed and which specific bytes to retrieve.
- Data is not decompressed until it reaches the calling service.
A service can serve many users, or could be dedicated to a single user. Performance can be dialed up and down per service instance.
Turbine Ingest can be scaled up to accommodate large volumes or faster access to data as needed. Ingest services can be used for different types of streams of incoming data, meaning ingest can be prioritized to process quickly for critical data and more slowly for less critical data.
The compute allocated for Turbine Query can be dialed up or down as well. The user decides how much they want to pay for the performance and use appropriate resources. Query services can be set up and dedicated to different query profiles or pools.
The query service contains one or more stateless QueryHeads. Once a query arrives at any QueryHead, the node will parse the query and then consult with the Catalog Service to determine the hdx file parts that need to be queried. This is usually determined based on the namespace the query defines and the time range specified. Based on the data volume we need to query, we can spread the work to one or more stateless QueryPeers to execute a portion of the query.
QueryPeers will fetch from object storage the exact bytes that are needed to answer the query. It does a partial aggregation and returns back the partial results to the QueryHead that received the user query. Once the QueryHead receives the partial aggregate results from all of the QueryPeers, it will do the final aggregation and return back the result to the user.
Given the stateless nature of QueryPeers, resizing the cluster doesn't have consequences / recovery.
Updated about 1 month ago