Quick promql tip: promql _hates_ duplicate data in joins and will throw an absolute fit if you ever have them.
Unfortunately sometimes you end up with overlapping data points, because, e.g., kube-state-metrics is rolling out and there are temporarily two copies running, or because Prometheus also keeps stale data for up to five minutes after a datapoint is no longer seen.
The way I've worked around this in the past is to throw my query inside `max(metric) by(labels I care about)`, but I also learned/realized today that you could instead do `topk(1, metric)`, which is like `max`, except that it keeps all the labels on that datapoint.
Obviously you should think carefully about whether this is correct or not for your use case. If your metric is binary (only ever 0 or 1) the max is fine. If you use topk and you depend on the labels output, be aware that it's just going to pick one arbitrarily.
It would be best if you are able to clean up/select out the duplicate metrics, but sometimes that's.... not possible or more annoying than it's worth.
"Fear is the "Thief of Dreams," says 
Dann halt nicht.
Heinrich Friedrich Füger
