Skip to content

PromQL patterns

A phone panel is one number, one small chart or one short list. These are the queries that survive that constraint. Copy them and swap the job label.

The single most useful stat on a pager, as a percentage with the thresholds that page you.

stat
- type: stat
title: Error rate
query: 100 * sum(rate(http_requests_total{job="gateway-edge",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="gateway-edge"}[5m]))
unit: percent
thresholds:
direction: above
steps:
- { at: 1, level: warning }
- { at: 5, level: critical }

p50 next to p95 and p99 tells you whether it is everyone or the tail. Each quantile is its own histogram_quantile, and label_replace gives the lines names; the or joins them into one result.

line
- type: line
title: Latency
query: >-
label_replace(histogram_quantile(0.5, sum by (le) (rate(http_request_duration_seconds_bucket{job="gateway-edge"}[5m]))) * 1000, "quantile", "p50", "", "")
or label_replace(histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="gateway-edge"}[5m]))) * 1000, "quantile", "p95", "", "")
or label_replace(histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="gateway-edge"}[5m]))) * 1000, "quantile", "p99", "", "")
unit: milliseconds
series:
label: "{{quantile}}"

A list with topk, five rows and a name per row.

list
- type: list
title: CPU by pod
query: topk(5, sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{container!=""}[5m])))
unit: cores
label: "{{namespace}}/{{pod}}"
limit: 5

Absolute memory means nothing on a phone. A percentage of the limit does, and thresholds can be put on it.

list
- type: list
title: Memory against limit
query: 100 * max by (namespace, pod) (container_memory_working_set_bytes{container!=""} / on (namespace, pod, container) kube_pod_container_resource_limits{resource="memory"})
unit: percent
label: "{{namespace}}/{{pod}}"
limit: 5
thresholds:
direction: above
steps:
- { at: 80, level: warning }
- { at: 95, level: critical }

Part-to-whole is the one case for stack, with top keeping the chart readable.

line
- type: line
title: CPU by namespace
query: sum by (namespace) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
unit: cores
stack: true
series:
label: "{{namespace}}"
top: 5
rank_by: mean