Change one value, move one service

Part 4 of 7. Part 3 covered asking first and receiving later.

You edit one line. settings.server.slots = 32 becomes settings.server.slots = 64, and the game server that reads it runs on one machine out of forty. One process should restart. Nothing else should rebuild, restart, or notice that you touched anything.

You get that if every entry in the plan carries a key. The key is a hash of everything that affects that entry: the store path, the resolved environment it was handed, the keys of everything it depends on, and the machine unit it runs on.

"mygame-eu:server@alpha": { "key": "sha256-9d41f7a0", "closure": "/nix/store/9k2h..." }

Two plans are then two sets of keys, and the work is the difference between them:

deactivate = previous - both
activate   = new - both
both       = untouched, not rebuilt, not restarted

Postgres kept its key, so it lands in both and nothing touches it. The game server's key changed, so it shows up in deactivate and again in activate. One process restarts, and the thing doing the restarting never learned that slots was the value you edited.

Moving a service is the same mechanism🔗

Six months later the database outgrows alpha. pg-shared listens on TCP, so you edit placement and nothing else:

-instances.pg-shared.placement.every.main = { machines = [ "alpha" ]; };
+instances.pg-shared.placement.every.main = { machines = [ "beta" ]; };

Three things follow without you asking for them. Postgres is now keyed on beta, so it deactivates on the old host and activates on the new one. The game server's DB_DSN becomes the address of beta, and that address is part of its key, so the game server restarts with the new value. Everything else on both machines keeps the key it had and is left alone.

That works because this database is shared and reached over the network, and the reason the planner lets it move is that the provider said so. A postgres configured to listen on TCP tags its own dsn routable, and an edge to a routable export is an edge the planner may split across machines. The same module configured for a unix socket tags that dsn machine-local, and part 3's round 5 then refuses to separate the two services, because a socket is not something you can move away from its reader.

So the two cases are one module and one field. Nothing about the consumer changes between them: the game server declares uses.db and reads results.db.dsn either way, and never learns which kind of database it got.

The cascade is not a second mechanism bolted on. A dependency's key is an input to the dependent's key, so invalidation travels along the edges by itself. No separate invalidation pass exists to forget about.

Which direction the mistakes run matters more than getting it perfect. Leave a value out of the key and you get a process running on stale input while the diff reports no work at all. Put in a value that did not need to be there and you get a restart nobody needed. The second mistake costs seconds, so hash generously.

What the consumer declares🔗

Every value a service consumes travels on one of four planes, and the plane decides what a change to it costs. Neither end of the edge picks the plane. The consumer declares a slot and a tolerance, and that is all it declares:

{ mygame, postgresqlDatabase, meshPeer }:
{ settings, ... }:
{
  uses.db   = { interface = postgresqlDatabase; };
  uses.mesh = { interface = meshPeer; };

  reactions.mesh.address = {
    reaction = "restart";
    onMissing = "keep-last";
    maxStaleness = "5m";
  };

  impl = { results, alloc, ... }: {
    units.gameserver.env = {
      DB_DSN    = results.db.dsn;
      PEER_ADDR = results.mesh.address;
      PORT_GAME = toString alloc.ports.game;
    };
  };
}

uses.mesh is part 3's slot, so it names an interface and the deployment says which mesh fills it. Nothing here says how the address travels or when it is known.

A lifecycle belongs to one export rather than to the whole provider, so the tolerance names one too. reactions mirrors results path for path: what you read at results.mesh.address takes its reaction at reactions.mesh.address, and both are checked against the interface, so reactions.mesh.addres is a plan error rather than configuration that quietly does nothing.

reaction has no default and the consumer has to state it, because what staleness costs is different per value. A peer's address going stale means connections fail, so the reaction is a restart. A membership list going stale means the lobby lists a player who already left, so a reload is enough. A rotated credential keeps working until the old one is revoked, so a restart inside the overlap window is right.

maxStaleness is the other half of the same judgment, and it says how long this consumer can run on a value that may be wrong before the staleness costs more than the restart. The reaction and the staleness budget are both properties of the consumer, and the consumer is the only party that knows them.

There is no source field and no debounce field. Which mesh publishes the address is the deployment's business, and the rate at which a host reacts belongs to the machine.

The planner picks the plane🔗

Two facts decide it. What the provider tagged the export, which part 3 covers, and where impl put the value.

A value in the closure and a value in env are the two common cases, and a deploy-time fact such as "this host can load the wireguard module" behaves like the second, because a fact that changes produces a new plan. The table is not documentation about the deployer. It decides whether the deployer is correct: give a value the live reaction when a restart was needed and the process holds a dead connection open instead.

A probed value in env comes out of the fact snapshot the plan was computed against. If the fact was never published, the entry becomes a typed hole instead: the plan commits with a placeholder, and the orchestrator resolves it at the step that needs it, which is the mechanism part 5 describes.

Three of the twelve cells are refused, and each refusal used to be either a silent mistake or something the schema had no way to say. Two further checks fall out of the same derivation. The planner knows when it landed on the dynamic plane, so it can see that a consumer never said what to do, and it knows when it did not, so it can see a reaction declared for a value that only ever changes by generation:

x mesh.address (dynamic) is interpolated into the closure of mygame-eu:server@alpha
  resolution: put the value in env, or have the provider publish it as a fact
x mesh.address (dynamic) appears in requirements.mesh of mygame-eu:server
  resolution: placement cannot wait for a running service, use a probed fact
x mesh.address lands in the dynamic plane and declares no reaction
  resolution: state reaction = restart | reload | live
x reactions.db.dsn declared, derived plane is env
  resolution: an env value changes by generation, the diff already restarts it

Those are diagnostic rows in the plan, not thrown errors, which is the property part 3 needs for a plan to come out even when the input is wrong.

Values that no key can cover🔗

A peer's mesh address is not knowable when you plan, and it keeps changing while the service runs. Nobody commits anything when it changes, so no key can cover it, and comparing two plans will never reveal it. Both plans are identical.

The plan still carries it, as a contract rather than a value. The contract sits in its own map, beside the resolved environment and outside the key:

"dynamic": {
  "PEER_ADDR": {
    "from": "vpn-core:node/provides/peer/address",
    "transport": "mesh:vpn-core",
    "reaction": "restart",
    "onMissing": "keep-last",
    "maxStaleness": "5m"
  }
}

env is hashed into the key and dynamic is not, which is what makes the two plans identical when the address changes. The contract names the export the value comes from, the transport that carries it, and what the agent does when a new one arrives.

Values also have to be resolved together. Two meshes renumbering in one event is one restart, not two, which means the agent gathers every dynamic value in a window before it reacts.

Why a watcher that restarts is not enough🔗

A laptop moves from one network to another and its mesh address changes six times in two minutes. A watcher that restarts on every change restarts the service six times and drops every connected player six times, and for the first ninety seconds the stale address was the less harmful of the two options. The agent collapses every change inside a window into one restart, and a burst past the limit trips a circuit breaker.

The breaker needs somewhere to put the unit. Healthy is wrong and failed is wrong. Going back to the old address is no rollback either, because the old address is not an earlier generation. It is a fact that is now wrong. A unit therefore has a third state, running-stale, which it leaves only when a fresh resolution succeeds.

The window is not the module's to set. An author writing window = "30s" cannot know how many other units on that host react to the same mesh event, and the operator who does know has no way to override a number compiled into a service module. The module states the staleness it can live with and the machine states the rates:

machines.alpha.reactions = {
  debounce = { window = "30s"; burst = 3; };
  restartBudget = { perUnit = 5; window = "1h"; };
};

Rate policy sits beside the key rather than inside it. The numbers change how the agent reacts and not what the unit is, so tuning a window must not restart the services the window governs.

One thing that looks like it belongs here does not. A three-node cluster needs a floor on how many members stay up, and that floor is a property of the set, not of any value in it or any machine under it. Part 6 collects the failure: a mesh renumber restarts one member while another is already down for a reboot, and both actions were individually correct. A floor declared on the activation set is the open answer, and it is not a field on a value.

The key stops depending on discipline🔗

The key at the top of this post hashes the store path, the resolved environment, the dependencies' keys, and the machine, and it leaves one question open: which environment entries count. A dynamic value has to stay out, or no two plans would ever be identical and the diff would report work on every deploy. An author who has to remember that is an author who will one day forget it.

The planner writes resolved values into env and contracts into dynamic, so nobody has to remember:

key = hash(closure, dependency keys, machine key, env)

The failure worth worrying about needs an env entry that should have been a dynamic one. No author can write that entry, because no author writes either map.

One service, four values🔗

The game server consumes four values, and one declaration shape sends them down four planes.

The plan entry carries all four, and only one of the four maps is hashed:

"mygame-eu:server@alpha": {
  "key": "sha256-9d41f7a0",
  "closure": "/nix/store/9k2h...-mygame-2.3.1",
  "env": { "DB_DSN": "postgresql://mygame_eu@10.0.0.4:5432/eu", "PORT_GAME": "27015" },
  "configData": { "/etc/mygame/maps.json": "sha256-4b7e1f08" },
  "dynamic": {
    "PEER_ADDR": {
      "from": "vpn-core:node/provides/peer/address",
      "transport": "mesh:vpn-core",
      "reaction": "restart",
      "onMissing": "keep-last",
      "maxStaleness": "5m"
    }
  },
  "requirements": [
    { "subject": "kernel.module.wireguard", "predicate": "exists", "severity": "require" }
  ]
}

Move the database to another machine and DB_DSN changes, so the key changes and the game server restarts once without rebuilding. Publish a new map file and only the configData hash moves, so the unit reloads. Renumber the mesh and nothing in the plan changes at all, because the two generations are byte-identical and the agent on alpha reacts on its own. Boot a kernel without wireguard and the machine publishes a fact that fails the predicate, which is the only one of the four that produces a new plan and can move the service to a different machine.

Those four costs were always different. What changed is that the author writes one kind of declaration for all four, and the plan says which cost applies to which value.

How many providers an edge means🔗

An edge names one provider's capability, and some consumers need every placement of it. Four binary caches, one per machine, each with its own signing key, and every machine on the fleet has to end up trusting all four. No plane in the table above covers that read, because the question is not which plane a value travels on but how many values there are.

Bundling both halves into one module shrinks the question to something bounded. The cache role and the substituter role are siblings in one authored unit, so the edge between them is a binding, and what the consumer is asking for is one named sibling at each of its placements. That set is the placement set the planner computes anyway.

The field that says so sits on the consumer's slot and has three values. one is the default and is written by omission. local is the co-located placement. all is always a set keyed by machine, including when the provider happens to be placed once.

uses.caches = {
  interface = nixBinaryCache;
  reach = "all";
  requires.count = { predicate = "in-bounds"; bounds = ">= 1"; severity = "want"; };
};

A number would be the wrong shape there, and that is the argument the field is built on. Write placements = 1 and a client written against one server breaks the day an operator sets count = 2, and the party who then edits impl is not the party who changed the number. Named shapes keep the type of the answer out of the deployment's hands: under all the consumer writes results.caches.<machine>.url and maps, always, and under one it writes results.caches.url and never has to check.

What the planner counts is the placements this consumer can read, and that is what makes the field cheap enough to almost never write. An export the provider tagged machine-local is readable only from the provider's own machine, and a routable one is readable from anywhere. A database on six machines publishing a machine-local socket therefore presents exactly one readable placement to each of its consumers, and every edge of that kind stays a plain uses.db with no field. A provider with no units at all, or an external, counts as one placement-free capability that everybody can read. local earns its place only when a provider's exports are all routable and a consumer specifically wants the one beside it.

Reach is authored and cardinality is checked, which is why they are two fields rather than one. A predicate never changes the shape of the answer, so requires.count can only pass or produce a row, and it has to run after member disposition: four caches can pass placement and three then fail their export checks, so a count satisfied when machines were picked is violated by the time impl runs.

The defect this fixes was already in the corpus. A consumer reading results.server.address got one value only because the operator wrote count = 1, and nothing produced a row when that number changed. Now the slot says one by saying nothing, the planner counts two readable placements, and the plan is refused with a row naming the consumer's slot and the count that produced the second one. What stays open is the unbounded form, every provider of an interface anywhere in the deployment, which is a query the asking half cannot run and no owner computes.

One thing this does not answer🔗

Whether a consumer should be allowed to state the plane it expects. Deriving the plane means a provider can retag an export from static to dynamic and change every consumer's restart behaviour with no consumer edit. The plan diff reports the retag and names the affected consumers, but nobody has to acknowledge it. The alternative is a consumer-side expect = "dynamic" that fails on mismatch, which catches the retag at every consumer and adds a field that has to be kept in sync. The diff route is the current choice, because the plan is already the artifact you read before applying. That choice rests on an assumption worth writing down: it holds only while diffs stay small enough that one new diagnostic row gets noticed.

Next: the facts only the machine can answer.