DNS is Simple. DNS is Hard. - Wespiser

It feels like configuration. A lookup. Some project metadata you change, and then it’s changed. But that’s not what actually happens. When your application makes a DNS request, it doesn’t go straight to the authoritative server. It goes to a recursive resolver that is run by your ISP, your company, or a public provider like 8.8.8.8. … There is no global view of DNS state. There is no control plane. There is no way to ask, “what does the system believe right now?” When you change DNS, you are not updating configuration. **You are initiating a convergence process across a distributed system you don’t control, can’t observe, and can’t roll back.** … By 12:26 AM PDT, the team had narrowed the event to DNS resolution issues for the regional DynamoDB endpoint. The underlying problem: a race condition in DynamoDB’s DNS management system. In simple terms: the database servers were still there, the network mostly still existed, but the naming layer that told systems how to reach DynamoDB had broken. The failure wasn’t just a race condition. It was a race condition in a system where **partial state is globally visible—and cached**. Multiple automation paths were updating DNS without coordination. When those updates collided, DNS didn’t fail cleanly. It propagated inconsistent state outward. Once that happened, everything depending on DynamoDB couldn’t reliably find it.