The facts only the machine can answer

Part 5 of 7. Part 4 priced what a change costs.

A mesh service needs the wireguard kernel module. Can alpha load it? Reading the source tree will not tell you. The answer sits on the machine, and the way to get it is to run a program there.

Start with what the service says, which is a predicate and nothing else:

requirements.kernel.module.wireguard = {
  predicate = "exists";
  severity = "require";
};

Nothing in that declaration mentions probes, machines, or execution. severity sets the price of a missing answer: require filters placement and produces an error row, want only makes the machine less preferred and leaves a warning.

Three sources can answer the predicate, and they have a fixed order. Tables generated from the kernel sources answer the stock case without asking anyone. A fact published by the machine beats the table, because vendor and patched kernels deviate from the universal. If neither knows, the planner queues a probe for that machine, and that machine alone goes to pending while the rest of the fleet proceeds.

What a probe is🔗

A probe is a program plus a declaration of how to run it:

{ name = "wireguard-loadable";
  predicate = ...;      # exits pass or fail, plus a typed payload
  class = "probed";
  sandbox = "readonly";
  context = "machine";
  timeout = 5;
  outputSchema = ...; }

Every probe a machine needs builds into one closure. The machine runs that closure and publishes a signed report, and planning reads reports rather than running probes.

A narrow version of this ships in clan today. clan machines update-hardware-config runs nixos-facter over SSH and writes machines/<name>/facter.json, which every later evaluation reads back. Three things change.

The machine refreshes its own report on boot, on kernel change, and on an interval, so a stale report is no longer something an operator has to notice. A report holds predicate results and never raw measurements: hugepages >= 512: PASS stays stable across reboots, while 1024 flaps between them and would restart services for nothing. The hardware bundle facter produces is one probe group among others, and a service can ship probes of its own into the same report.

Probes are nodes, so the stage count is not fixed🔗

Probes, services, and machine units live in one graph. A probe can depend on a service the same way a service depends on a probe, which is what makes "evaluate twice" the wrong model.

The wave-2 probe cannot run earlier, because it runs inside the service's own sandbox as the service's own user, and that sandbox does not exist until the service does. Sorting the graph gives the number of waves, so it comes from the edges rather than from a pipeline someone fixed at two. Probes that sit at the same frontier merge into one execution per machine. A probe that needs its own result is a cycle, and the planner reports it instead of spinning.

Generated secrets produce stages with no probe involved at all. WireGuard peer keys are the standard case: nobody authors them, and each machine needs every other machine's public key.

Four stages, and the count came from the content. Adding a fifth machine regenerates nothing that already exists, because every key belongs to one machine and the fact set is a union.

The authoring surface for that is one submodule and one argument. A service declares what has to be generated, and the answers arrive the way allocated ports do:

modules/nix-cache/cache.nix, inside implgenerates.harmonia = {
  share = false;
  files.ca-priv = { secret = true; owner = "harmonia"; };
  files.ca-pub  = { secret = false; };
  runtimeInputs = [ coreutils nix ];
  script = ''
    nix-store --generate-binary-cache-key "harmonia-${placement.machine}" \
      "$out"/ca-priv "$out"/ca-pub
  '';
};

share = false is what makes four caches four identities, because the generator then runs once per placement. share = true hands one keypair to all four, and nothing catches it: the plan is well formed, the fleet works, and the operator has one identity behind four names.

What comes back has two kinds, and the difference is the whole of the staged pass. vars.harmonia.ca-priv.path is deterministic from machine, generator and file name, so it is knowable on the first pass and the file itself is delivered by the agent, which keeps the private key out of the plan. vars.harmonia.ca-pub.value is content, and it does not exist until the generator has run. Asking for .value on a file marked secret is a rejection row rather than a fallback.

The lifecycle bound is what admits the second one. nix-binary-cache.publicKey is capped at at-most-probed, which permits a value that only appears on a later pass and still refuses one that exists only while a process is up.

Nothing reads config.clan.core.vars.generators, because there is no ambient namespace to read it from. The clan module doing this today builds a path into another machine's vars directory by hand and turns the ungenerated case into "", so a cache nobody can verify renders exactly like a cache that is fine. Both of those are rows here.

Where an answer lands decides what it costs🔗

A gitea CI token exists only after gitea runs, so nothing at plan time can know it. Gitea tags that export dynamic, its consumer states what to do while the value is missing, and the agent injects it after activation. That path is the last row of the table in part 4, and it needs no evaluation and no replanning.

Some answers have to appear in the plan itself. The generation then commits with a typed hole plus an order in which holes get resolved. The orchestrator reaches that step, runs the probe, writes the result into the ledger, and substitutes the value into every entry that referenced the hole. Checks over a hole are limited to type and bounds, and value-level checks happen at resolution, which is why holes stay rare. Substitution is textual, so the rule from part 2 holds and nothing below the plan evaluates Nix.

An answer that feeds a build flag lands in the closure, where no substitution can reach. Evaluating Nix again mid-deployment is refused, because a plan that rewrites itself cannot be the thing the register recorded. The planner says so instead:

x probe wireguard-loadable feeds the closure of mesh:node@alpha
  resolution: split into two generations, or move the value into env

Terraform users know the shape of this from -target, where the two-step is real but implicit. Ours is explicit and appears before anything runs.

The machine listens as well as answers🔗

Facts are what a machine tells the services placed on it. Contributions run the other way, and they are why a machine has an entry of its own in the plan. A machine does not disappear when it stops being the unit of deployment. It stops owning the services placed on it and becomes something those services contribute to.

A NixOS configuration is one large fixpoint. Any module may read any other module's settings, which is powerful, and it also means the interfaces between services are implicit. A service reaches into another service's option paths, and that wiring is visible only if you already know where to look. Ask which service is responsible for a value on a machine and the honest answer is to go read modules.

Here a service never writes into another service's settings. It declares what it requires, in a typed namespace, and the declaration is the interface:

machine.firewall.input = [
  { allow = { proto = "udp"; port = alloc.ports.game; from = "mesh:vpn-core"; }; }
];

The game server has stated what it needs on the port the planner allocated to it. It has not configured a firewall, and it does not know which firewall the host runs. The service that owns the firewall reads the merged set for its machine and renders its own configuration from that. Note which way the dependency now points: a NixOS firewall module has to know about every service that might want a port open, while this one only has to read a table.

Every resource carries a merge class, and the class decides what a second contributor means. Firewall rules compose in a deterministic order. A sysctl has to agree across its contributors. Reserved memory pages sum and are then checked against what the machine has. A resolver has exactly one owner and everyone else contributes routes through it. Only the last of those is exclusive. Every contribution is typed and carries its contributor, so when a class does reject a combination the planner can name the parties and suggest a way out:

✗ machine host-alpha: sysctl vm.overcommit_memory   [equal-or-conflict]
    redis-cache wants 1, kv (etcd) wants 0
    resolution: align the values, mkDefault in one contributor, or place them on separate machines

The diagnostic carries the resolution line, so nobody has to work it out. Errors shaped like that are easy to automate, and an agent editing infrastructure gets a correction it can apply.

Planning stays a local join🔗

One rule keeps all of this cheap: the planner reads the register and nothing else. It never opens an SSH connection, so "probe now, wait, then place" cannot be expressed. Planning is therefore fast enough to run on every keystroke in a user interface, it works offline, and it never stalls on a machine that is behind NAT or switched off.

What planning does is a join of requirements against published facts and generated tables. Everything expensive already happened on the machines, in parallel, before anyone asked for a plan.

Next: where the model broke.