Query API Options

For HTTP POST requests, the following can be added to the body of the request to alter how a query is executed or how a response is formulated at run-time.

need example

For HTTP GET requests, the following can be appended to the end of a query URL to alter how a query is executed or how a response is formulated at run-time.

For example:

https://demo.hydrolix.live/query/?query=<query>&query.output_format=JSON'

Output Formatting

By default, Hydrolix Query API returns results as formatted JSON. Many other formats are supported; for more information visit https://clickhouse.tech/docs/en/interfaces/formats/

Tunable Values default
query.output_format too many to list JSON

Rate Limiting

The following are “rate” limiters for how the query components execute a query and retrieve partitions from distributed storage. It is strongly suggested that should a user wish to change these away from the defaults they talk to Hydrolix support first.

query.max_peers

By default, query processing is distributed across all available query peers (to maximize massively parallel processing). If fewer peers are desired, this flag will instruct the query head to use a subset of available peers.

storage.max_streams

By default, each query peer will run one process (aka, a “stream”) per CPU core, and this is the recommended configuration. In the unlikely situation where a different number of processes is desired, that number may be specified here.

storage.max_concurrent_partitions

Hydrolix query processing generally requires each query process to extract data from many HDX partitions. This flag sets a limit on the number of partitions which may be “open” at the same time. Note that this setting is applied per process, so, for example, a 4 core query peer running 4 processes, each opening 25 partitions, will result in 100 open partitions per peer. Decreasing this setting may slow down query performance. Depending on the size of each partition, there’s some risk that opening more than 25 will result in excessive memory pressure on the peer. Tread carefully.

storage.max_concurrent_iops

Depending on the number of columns required and predicates applied to a particular query, the processing of each HDX partition may generate many independent HTTP range requests from the query peer to cloud storage to extract data. This flag sets a limit on the number of in-flight downloads which may be “open” at the same time. Please note that this setting is applied per partition, so, if there are 100 open partitions (as in the example above), and each partition is allowed to have 2 concurrent iops, a single box may have 200 simultaneous HTTP requests awaiting data. Tread carefully.

Query Option Min Value Max Value Default Value
query.max_peers 1 total # available query peers null (all peers)
storage.max_streams 1 2x # available CPU cores null (1 per core)
storage.max_concurrent_partitions 1 n/a 25
storage.max_concurrent_iops 1 n/a 1