Observability
Intake pool monitoring posts a timestamp to the hydro.monitor table once a second per ingest pool. The gap between timestamps indicates whether a pool is keeping up. The service is disabled by default.
A monitoring failure resembles an ingest failure. Rule it out first: an empty hydro.monitor usually means the service was never enabled.
Intake latency spikes without an obvious cause⚓︎
Confirm: kubectl top pods -n <namespace> | grep intake shows pods at their CPU or memory limits.
Fix: Intake Pool Monitoring: Unexpected latency spikes.
No rows arrive in hydro.monitor⚓︎
Confirm: kubectl get hdx -n <namespace> -o yaml | grep monitor_ingest shows the service disabled, or kubectl get pods -n <namespace> | grep monitor-ingest returns no pod.
Fix: Intake Pool Monitoring: No data in hydro.monitor table.
Heartbeats stop arriving from a pool⚓︎
Confirm: The pool's most recent hydro.monitor timestamp is older than the threshold for its submission interval.
Fix: Intake Pool Monitoring: Identify missing heartbeats.
A pool is missing from monitoring results⚓︎
Confirm: The pool's service type isn't monitored, or the pool is on the exemption list.