The gaps invented scenarios miss

Part 7 of 7. Part 6 collected where the model broke.

Every round of review so far worked the same way. We wrote down a workload the design had not been built against, ran it through the mechanisms on paper, and kept whatever survived. Post 6 counted the result: five rounds, forty-three scenarios, two mechanisms killed outright.

This round changed the method. Instead of inventing workloads we read four repositories that people deploy from today: the clan platform itself, two production fleets of eight machines each, and a benchmark harness that installs twelve different mesh VPNs one after another. Five read-only passes over roughly a hundred cited defects, deduplicated into 22 problem classes.

Ten of those classes the design handles. Seven have a mechanism that stops short of the whole problem, and all seven residuals were already written down somewhere in our own notes. Five have no mechanism at all.

The five are boring. Not one of them needed a clever adversary, an unusual topology, or a race. That is the finding worth a post, because it says something about the previous six rounds rather than about the fleets.

Inventing a scenario selects for failures that are interesting to reason about. A three-node etcd cluster where no member can pass its own health check is a good puzzle, which is why post 6 opens with it. A user id that is 989 on one machine and 997 on the next is not a puzzle at all. It took a production host down for longer.

The gap sitting under our own worked example🔗

Post 4 sells a service move as an edit to one line:

-instances.pg-shared.placement.every.main = { machines = [ "alpha" ]; };
+instances.pg-shared.placement.every.main = { machines = [ "beta" ]; };

One of the fleets we read did that move. Here is what it cost, quoted from the tool they had to write afterward:

gi-infra/services/restic_restore.py, trimmedrestic stores ownership numerically and 0.19 keeps no user/group names, while
NixOS allocates system uids dynamically, so they differ per install (gi-04's
gitea moved 989 -> 997 during the 2026 migration, leaving /var/lib/gitea
unreadable and the unit dead at CHDIR).

NixOS hands out system user ids in the order services ask for them, so the same service gets a different number on a different host. Move the files and the owner recorded in them no longer exists. The service starts, cannot read its own data directory, and dies.

Repairing that took an id-to-name table written into every snapshot, a restore script that translates each file to the current machine's ids, and a NixOS VM test to keep it honest. Roughly 500 lines. It also left a permanent cliff, because snapshots taken before the fix carry no table and still restore numerically.

Our model has none of this. A piece of state is a name, a list of folders, and a backend type, which is the shape clan already uses and the shape we kept. Ownership is part of what identifies a state directory, and we never wrote it down. Search the design notes for uid and there is nothing to find.

What makes this the worst of the five is where it sits. The argument for making a service the unit of deployment is that moving one becomes a line in a file. The uid is the reason it is a line in a file and then three days of recovery.

Four more, each one dull🔗

A port range we costed and then forgot. Round one already found this: allocating a port at plan time is only sound if the deployment is the complete authority over ports, and on a general-purpose host it is not. Linux hands out roughly 32768 through 60999 to outbound connections, so a listening socket in that window can find its port already taken. The fix was written down in the same paragraph that found the problem, which is to allocate outside the range and reserve anything inside it. Nobody adopted it. Two clan services ship defaults in that window today, dyndns at 54805 and wireguard at 51820, and the resulting failure appears at activation, intermittently, only on a host with enough outbound traffic to be interesting.

Externals that depend on other externals. Post 6 argues that you should never orchestrate what you do not own, and that an external gets declared once, set up with a human present, and checked by a probe from then on. That holds up. What the schema cannot say is that one external depends on another. One fleet runs two separate infrastructure-as-code state directories with no edge between them, and a certificate validation token that the first one issues appears in the second as a pasted literal:

clan-infra/machines/web01/terraform-configuration.nix, trimmedcache_acme_challenge = {
  name = "_acme-challenge.cache";
  type = "CNAME";
  records = [ { value = "61s5zcfes5290tjs5r.fastly-validations.com."; } ];
};

The certificate in question fronts the binary cache every machine in that fleet pulls from. It renews when a person reads one state's output and retypes it into the other's source, and not otherwise. Nothing fails loudly, so the symptom is an expired certificate. A CDN in front of a bucket is two externals with a dependency between them, and both fleets we read have exactly that shape.

An external that mints a credential. We modeled the credential an external consumes, gave it a trust tier, and said nothing about the credential an external produces. An object store hands you an application key when you create it, and hands you a different one when it gets recreated. With no field for that, here is where the value ends up:

clan-infra/modules/web01/niks3.nix, trimmedclan.core.vars.generators.niks3-s3 = {
  script = ''
    echo "niks3-s3 credentials are populated by terraform (cache-new), not generated" >&2
    exit 1
  '';
};

A generator whose entire body is exit 1, standing in as a mailbox for a value that arrives from the other direction, written by a provisioner that edits the repository from inside an apply. Their CI runs a command that calls that generator, which then exits 1.

Sharing a secret in order to share a value. The only channel clan offers for one machine to read a value another machine generated is a flag, share = true, that produces one copy the whole fleet sees. We recorded one failure against it, which is that four binary caches sharing a generator end up with one identity behind four names. The failure that actually happens is the reverse:

gi-infra/services/saml-cert.nix, trimmed# Shared self-signed signing keypair for the Zulip SAML provider. Authentik
# (gi-04) signs assertions with it, Zulip (gi-05) trusts it as the IdP x509cert.
{
  share = true;
  files.cert = { };
  files.key = { };

One machine signs, the other verifies. The verifier needs the certificate and gets the private key, because sharing is the only way to express "read what that machine generated". The same fleet does it again with a mailbox password, and the comment there says the quiet part: the plaintext deploys to a machine that never reads it, because clan requires a shared generator's definitions to match everywhere.

Typed capabilities fix this, and the binary-cache example in our notes already shows the shape with a public key that a consumer may read and a private key that never enters the plan. What we never wrote is the rule that turns the old habit into a refusal. A cross-machine edge must not require the private half to travel.

The measurement that changed how I read post 3🔗

The clan platform has two ways for one service to get a value from another, and both were built deliberately. The first is typed: a service declares what it publishes, and the platform checks that declaration against a registry of known interfaces. The second reads a file out of another machine's directory by building the path as a string.

Count which one people use.

The producing half is enforced, and nine services publish through it. The consuming half exists as an option a module can set, nothing validates it, and two services bother. The untyped path checks nothing and carries 34 reads.

Nobody chose that distribution. It came out of which declarations had a check behind them, in one repository, written by people who all agreed the typed edge was the better idea. Types get used in proportion to what enforces them.

Why this matters for the slot

Post 3 says two things about a slot: the provider's exports are checked at the node that produced them, and the consumer's read is checked against the interface it declared. The second half reads like completeness for its own sake. It is the difference between 34 and 2.

The scope that exists because a daemon has no key🔗

Everything above came from the same kind of evidence: a fleet that broke, or a tool somebody had to write around a gap. The platform's own source yields a different kind, and it gets a section because it is the only thing this round found that reversed a decision we had already taken.

A clan service module has two places to put its output. perInstance runs once per instance per machine. perMachine runs once per machine. They differ by one line in what each is handed. perInstance is given a name (service-module.nix:629):

v: instanceName: machineName:

perMachine is given the set (:814):

inherit (machineScope) instances;

Something has to need the set, and here is what does. Put a laptop on two ZeroTier networks, the company mesh and the family mesh. That is two instances of one module with one machine in both. The machine needs two memberships and exactly one daemon, because a host has one node identity, one control socket and one UDP port, so the module has to compute a list that spans instances (zerotier/default.nix:479):

joinNetworks = allNetworkIds;

The author left a note next to the fold that produces it, at :353: it cannot compute in peer.perInstance. That is correct, and it is the whole reason the scope is there.

Run the same deployment through our model and it produces two daemons, because a service is evaluated once per placement and both networks are placed on that laptop. My first answer was to give units a cardinality field, so an author could declare one process per machine fed by every instance on it. That answer was wrong, and the reason it was wrong is more useful than the answer.

The daemon and the network are two different things. One host runs one daemon and joins many networks, the daemon holds the identity, and the identity should outlive every network on the machine. Upstream has them in one module because a clan service is one module, so the daemon has no name of its own and cannot be placed, wired, or counted. Give it its own instance and the problem disappears without a new field: two networks reaching for a local control socket on one laptop produce one plan entry, because zt:default@laptop is one key however many edges asked for it. The deduplication perMachine performs is the shape of a key we already had.

The decomposition also settles where the three cardinalities live. Upstream encodes them by choosing which variables to interpolate into a generator name, which is the same untyped-string habit the section above measures:

clanServices/zerotier/default.nix, three generator nameszerotier-identity-${machine.name}            one per machine
zerotier-network-${instanceName}             one per network
zerotier-ip-${machine.name}-${instanceName}  one per machine per network

Split into a node service and a network service, each of the three sits on the service whose placements already have that shape, and no new granularity is needed anywhere.

Then the interesting part. The network identifier is derived from the controller's private node identity, and the derivation runs at plan time, so whatever runs it needs the secret. The identifier must be one value per network, so its generator is fleet-shared. And the tool refuses a shared generator that depends on a machine-specific one:

clan_lib/vars/generator.py, trimmedif self.share and not dep_generator.share:
    msg = (
        f"Shared generators must not depend on machine specific generators. "

The controller's identity generator has to be fleet-shared as well, and its name is uniform across machines, so this is not one host's key. Every machine's private node identity sits in the namespace every machine reads. The address generator makes the same move a third time.

That is the failure this post already named, sharing a secret in order to share a value, with one difference that matters. There the author chose to share, because sharing was the only channel that worked. Here a rule inside the tool leaves no other branch, so no amount of care at the call site avoids it. The repair is not a new mechanism either: move the derivation onto the machine that already holds the secret, let the daemon mint the identifier, and publish the result as a probed fact. The secret stays capped at its machine and the public value crosses on an ordinary edge. It costs one extra round on first deploy, which is a diagnostic row rather than a hidden property.

What we still owe after all that is one field, and it is not a new ask. The daemon has to know which networks to join, and its author cannot name them. What the author can name is the far end of every edge into one of its capabilities. Our own notes already wanted exactly that from the other side: a database that has to say which machines may reach its listening socket wants the machines at the far end of its inbound edges. One mechanism, two projections, and the planner holds the set in both cases because it resolved those edges in order to place the consumers.

This is not a twenty-third problem class. It is the one case in this round where reading the tool rather than a fleet changed the design. The direction it changed in is the part worth recording: the mechanism I was about to add turned out to be a decomposition that the platform's own shape had stopped anybody from making.

What the partial cases taught🔗

Seven classes have a mechanism that does not reach the whole problem, and the residual was on record for all seven. What the corpus changed is the estimate of how often each residual gets hit.

The clearest case is the collector, meaning a service that needs a fact about every other service around it. Post 5 inverts the firewall so that no module imports the world: a service states the port it wants open and the firewall reads a merged table. The general form of that inversion, where any service collects typed fragments from everything on its machine, is sketched in our notes and exists in no tree. We treated it as one case, the firewall. The corpus has four, and the firewall is the cheapest of them.

The backup row already lost data in one of these fleets. A service that never declares its state directory is not backed up and nothing says so, which is how a Matrix server's federation signing key stayed outside the backup set until somebody noticed and added it by hand. The comment they left gives the stakes: lose that key and every federated peer tombstones every device.

That row is also the one that changed the mechanism rather than just using it. Holding a database still long enough to copy it is a verb, and a verb the collector owns closes the set of stateful software at the collector author's time, so the subject has to hand over something the collector only starts and waits for. Two services that write to each other then have to be held still together, which neither of them can arrange alone, so a third service owns the group and asks its own members who writes what. That third service is a collector whose answer is another collector's input, and the general form stops being "everything answers, then the collector reads": the machine has to apply the answers in dependency order, and a collector that is also a member can be pulled into a set that stops the unit it was waiting for. The firewall shows none of that, which is exactly why the cheapest of the four was the wrong one to generalise from. Our notes carry the corrected rules and no tree carries the mechanism they correct.

The second case worth naming is a decision we took on purpose. Only the platform may add a new machine-scoped resource, so a foreign module cannot introduce one. Our own binary-cache example ran into it and reported that the resource it needed, the list of trusted binary caches, does not exist. Five separate implementations write that resource today across the four repositories, which makes the most contended machine resource in the corpus one a third-party module cannot touch without a platform release.

What the corpus deletes🔗

Reading other people's fleets also produces evidence in the other direction, and it is the strongest kind, because it is a cost somebody is paying right now.

One fleet carries a 329-line tool whose only job is to reconstruct a plan before applying one. It builds the new closure locally, copies the running closure back over SSH, and diffs store paths. It gives up and reports no diff at all when the operator is not a trusted Nix user. That tool is post 2.

The benchmark harness picks which host runs a mesh control plane by taking the first entry of a Python list. Reordering the list moves the control plane, which quietly invalidates the comparison the repository exists to produce. That is the placement block from post 1, with a stable allocation.

Both fleets have a machine marked with a flag that means "skip this one during a fleet-wide update", and in one of them the comment carries a date and the observation that the host stopped answering on three ports. An operator edited a repository to record a runtime fact, because there was nowhere else to put it. That is post 5.

The rule gains a clause🔗

Post 6 ended with a standing rule: a new mechanism gets checked against the plane table and against the counterexamples already on record, and a mechanism that contradicts either is not ready. That rule holds, and this round found the assumption hiding inside it. Checking against the recorded counterexamples only works if the recorded counterexamples are representative of what fleets do.

Forty-three scenarios that we wrote produced no numeric user id, no ephemeral port range, and no credential minted by somebody else's API. Four repositories that we did not write produced all three in an afternoon, along with two more.

The rule gains one more clause. Before treating the counterexample list as complete, read a fleet you did not design for.

A caveat on this round, in the same spirit. We meant to read five repositories. The fifth was unreadable in the environment the audit ran in, so it was reported as unread rather than quietly dropped, and it is still owed. Whatever it holds is not in the numbers above.