Where the model broke

Part 6 of 7. Part 5 explained probes and evaluation stages.

The design was attacked with concrete scenarios rather than reviewed for elegance. Five rounds so far and forty-three scenarios, each one a workload the design had not been written against. Two are worth reading because they killed mechanisms that looked sound.

A healthy cluster has no healthy members🔗

The original execution model deployed in waves. Activate everything in wave zero, check each instance is healthy, then move to wave one. Clean, ordered, and verifiable at every step.

Now deploy a three-node etcd cluster. A member's health check asks whether it has reached quorum with its peers. Its peers are in the same wave. The first member can never pass its check, because the machines that would let it pass are waiting for it to pass.

Weakening the check to "the process is running" is not a fix, because it gives up the only thing the gate was for. The mechanism was wrong: per-instance gates assume a service becomes individually healthy, and a cluster member never does. Mesh VPNs, Consul, Patroni, and Ceph all behave this way. The gate has to close over a set of activations and ask whether the set is jointly valid.

Do not orchestrate what you do not own🔗

Your DNS zone lives at a registrar. Your cloud account has resources in it. The obvious move is to manage both from the same deployment, so one command brings up everything.

Consider what that command has to do. It reads the registrar's current records, computes a difference against your declaration, and writes back. If someone edited a record by hand, the write either destroys their change or fails. If the API is down mid-deploy, the zone is left half converged, and the deployer has no way to know which half. Read-modify-write against state you share with other people cannot be made safe by trying harder.

Record what must be true and verify it instead:

externals.dns-delegation = {
  provider = "registrar";
  credential.owner = { tier = "cold"; };
  setup = "ceremony: delegate clan.example to ns1/ns2, by hand, once";
  attest = { probe = "ns-delegation-correct"; interval = "24h"; };
};

You delegate the zone once, by hand, with the registrar credential in the cold tier, meaning an offline key that no deploy ever loads. Every deploy afterward runs the ns-delegation-correct probe and refuses to proceed if the delegation stopped holding. The deployer never writes to the registrar, so it can never corrupt it.

The general rule is that setup happens once with a human present, and after that the system only attests. Two runtime shapes survive that rule because neither can destroy someone else's work: appending to an artifact store, such as a CI push to the binary cache, and renewing something that reissues on a schedule, such as an ACME certificate. Rewriting existing state at deploy time is refused.

Records inside the zone are a separate question, and the delegation is what makes it easy. Once the name servers are yours, the zone is a service you run like any other. Its records become ordinary settings, and none of it is external state anymore. What gets you out of read-modify-write is not a safer diff algorithm. It is owning one more thing.

The standing rule🔗

Both failures share a shape. A mechanism was designed against one scenario, held up under that scenario, and broke on a class of workload nobody had written down. Neither was caught by reasoning about the design in isolation.

Every new mechanism gets checked against two lists before adoption: the plane table from part 4, and the counterexamples already on record. A mechanism that cannot state which class its values belong to, or that contradicts a counterexample, is not ready.

What a dead machine costs🔗

A machine dies at three in the morning. Kubernetes reschedules the pods it was running and nobody has to be woken up. Here nothing moves. The machine stops answering, it degrades to pending, and every service placed on it stays placed there until a person computes a new plan.

That is a decision rather than an oversight, and it is the same decision as part 4. Placement lives in the persisted allocation table so that replanning an unrelated change does not reshuffle the fleet. A service whose machine changed is a service whose key changed, so a planner free to move things would manufacture restarts out of nothing. Stable placement and automatic failover are one knob held at opposite ends, and we picked the end that keeps the diff honest.

The bill arrives with the replan. A new generation goes through the build farm before anything activates, so recovering from a lost machine costs what a routine settings change costs. A plan that carried a pre-authorized standby placement would let the orchestrator promote a candidate without evaluating Nix, which is the one property part 2 refuses to give up. Nobody has written that down as a mechanism yet, so it is a direction and not an answer. The next section reaches the same complaint from the direction of a stolen device.

What is still open🔗

The first four rounds attacked mechanisms one at a time, and each one ended with the design changed. The fifth attacks the interactions between mechanisms that were adopted separately, and it has no answer yet. Nine scenarios are on that list and none of them is settled.

Three contradict things earlier posts in this series stated plainly. Revoking a guest's access is a new generation, and a generation is gated on a build farm, so the defense against a stolen device rides the same pipeline as a routine settings change. The reload-only artifact from part 4 is written to a stable path before the generation is promoted, so a machine that hard-crashes and boots the previous closure reads the new file. And a debounced restart from the dynamic plane is deliberately uncoordinated, which is right for a game server and wrong for one member of a three-node cluster whose peer is already down for a reboot.

None of that is a reason to stop. It is the fifth round doing its job, and the rule above is what says the design is not finished until those have answers. One more sits at the end of part 4, which is whether a consumer should be allowed to state the plane it expects. The other question that used to sit beside it, an edge that reads every placement of a provider, has its field now: the binary cache case narrowed the ask to one named provider at its own placements, and reach is what a consumer writes to say so. What is still open there is the unbounded form, every provider of an interface anywhere in the deployment, which no owner computes.

Next: the gaps invented scenarios miss.