Anycast DNS: what actually breaks in practice
Anycast looks like free latency until you measure the tail. Two nodes announcing the same prefix will attract traffic according to BGP path selection, not according to RTT, and the mismatch shows up in p99 rather than in the median.
The usual failure mode is a flapping announcement on a single upstream. When the path changes, established TCP sessions to the old node are reset, and resolvers fall back to a different server after one or two seconds. Users see this as intermittent slowdowns, not as an outage, which makes it hard to notice in aggregate metrics.
What helps: keep the prefix announced from at least two independent ASNs, set a conservative keepalive on the resolver side, and watch per-node query rates rather than totals. A sudden drop of 30% on one node with a matching rise on another is the signal.