PromQL patterns
A phone panel is one number, one small chart or one short list. These are the queries that survive that constraint. Copy them and swap the job label.
Error rate
Section titled “Error rate”The single most useful stat on a pager, as a percentage with the thresholds that page you.
- type: stat title: Error rate query: 100 * sum(rate(http_requests_total{job="gateway-edge",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="gateway-edge"}[5m])) unit: percent thresholds: direction: above steps: - { at: 1, level: warning } - { at: 5, level: critical }Latency quantiles, three on one chart
Section titled “Latency quantiles, three on one chart”p50 next to p95 and p99 tells you whether it is everyone or the tail. Each quantile is its own histogram_quantile, and label_replace gives the lines names; the or joins them into one result.
- type: line title: Latency query: >- label_replace(histogram_quantile(0.5, sum by (le) (rate(http_request_duration_seconds_bucket{job="gateway-edge"}[5m]))) * 1000, "quantile", "p50", "", "") or label_replace(histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="gateway-edge"}[5m]))) * 1000, "quantile", "p95", "", "") or label_replace(histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="gateway-edge"}[5m]))) * 1000, "quantile", "p99", "", "") unit: milliseconds series: label: "{{quantile}}"The pod that is actually on fire
Section titled “The pod that is actually on fire”A list with topk, five rows and a name per row.
- type: list title: CPU by pod query: topk(5, sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))) unit: cores label: "{{namespace}}/{{pod}}" limit: 5Saturation against the limit, not the box
Section titled “Saturation against the limit, not the box”Absolute memory means nothing on a phone. A percentage of the limit does, and thresholds can be put on it.
- type: list title: Memory against limit query: 100 * max by (namespace, pod) (container_memory_working_set_bytes{container!=""} / on (namespace, pod, container) kube_pod_container_resource_limits{resource="memory"}) unit: percent label: "{{namespace}}/{{pod}}" limit: 5 thresholds: direction: above steps: - { at: 80, level: warning } - { at: 95, level: critical }Who is busiest, stacked
Section titled “Who is busiest, stacked”Part-to-whole is the one case for stack, with top keeping the chart readable.
- type: line title: CPU by namespace query: sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=""}[5m])) unit: cores stack: true series: label: "{{namespace}}" top: 5 rank_by: mean