Services are the new atoms
Part 1 of 7. You wrote an application, and it passes its tests on your laptop. That says nothing about the Mac, the ARM board in a drawer, or the older Ubuntu box someone else will run it on. To know it works there you have to build and test it there, on every push, which is a CI system.
Hosted runners give you x86 Linux and a macOS image. The hosts the application actually has to survive are the ones in your cupboard and your drawer, and one of them is a shared box where you have a user account and no root. So the CI has to run on hardware you own, and putting a build agent and a test job on each of those hosts is a deployment problem.
The obvious answer is a container. You build an image, Kubernetes runs it as a Pod on whichever node it picks, and you never name the host. That is the shape the CI needs. The image is also the unit of deployment here, the smallest thing the tool knows how to install.
The unit decides which machines can take part, and neither the Mac nor the borrowed box can take a Pod. We arrived at the same wall from the other side. We build Clan, a deployment tool for NixOS machines, and its unit is the whole machine: you describe a host, and deploying means rebuilding that host to match the description. So a host qualifies only if you are free to wipe it, and the Mac and the borrowed box are outside a Clan fleet too.
The hosts are not the whole system either. The builds have to land somewhere every machine can pull them from, which is a binary cache, and whatever your users download comes off a CDN. You run neither. The cache is a bucket in somebody's cloud, the CDN is rented by definition, and both need credentials, a DNS record, and a name that outlives any host you own. Clan models none of this, because it assumes everything is self-hosted, so the cache ends up configured by hand beside the deployment that cannot work without it.
This series describes the design we are building next, and its unit is one service: a single process, every dependency it needs pinned down to the C library, and an entry that tells the host's own supervisor how to start it. Those dependencies live in a Nix store, a directory of immutable packages each named after a hash of its inputs, and that directory can sit under your home directory. A host qualifies if it can hold the store and keep a process running. It does not have to run Linux, it does not have to be yours alone, and you do not need root on it. One deployment then spans a NixOS server, a Mac, an ARM board, and a shared Ubuntu box. The cache and the CDN are in the same model, as external resources the deployer verifies and never writes to, which part 6 explains.
Code examples below sketch the authoring surface we intend. None of it is a working API yet.
Why not use Kubernetes?
Kubernetes made the service the unit years before we tried to, and a Deployment names no host. Two things keep us off it.
The first is the cluster you have to run before you can run anything on it. The control plane, the network plugin, the storage driver, ingress, and certificates are separate projects on separate release schedules, on top of etcd backups and a few Kubernetes releases a year, so keeping the cluster alive becomes somebody's job. The cluster it gives you is Linux-only. The unit is a container, so a node runs Linux or Windows Server under an agent that has root, which is why the two hosts above cannot join it. Nothing outside the cluster is modeled either, so your registrar and your cloud account end up driven by some other tool.
The second is that declarative is not reproducible. A pod spec names an image that some pipeline built, the cluster holds no recipe for building it again, and the live state sits in mutable etcd. A Nix closure, a package together with everything it needs transitively, names every input down to the C library and builds to the same bytes twice. Clan already has that property and we are not giving it up to get the service as a unit.
What we learned building Clan
Very little here is a new idea. Disnix was deploying individual services across a network of machines in 2008, with the services, the machines, and the mapping between them written down as three separate models. Writing a custom application has got much cheaper since then. Getting one onto hardware you own has not. Most of the design that follows comes from running Clan and watching where it did not reach.
Clan is good at the part that usually stops people. It brings a fleet up from nothing, and it moves a machine from a cloud provider to the hardware in your flat and back, because no part of the configuration is tied to the provider. It also made highly interconnected P2P applications deployable, which is where DVT, a way to self-host Ethereum nodes, came from. A community formed around it and started publishing infrastructure code that used to stay in private repositories.
Three assumptions in that model did not survive real fleets. Two are already above: every target is a NixOS machine you can rebuild, and everything is self-hosted, so an external dependency has nowhere to live. The third is that sharing a service means joining the infrastructure that runs it, so handing somebody a game client meant handing them the fleet.
Each assumption turns into something concrete you cannot do, and five of those cases run through the rest of this series.
Take the shared Ubuntu box. A clan target is a host you rebuild with nixos-rebuild switch, so a
machine where you have a user account and no root cannot be a target at all. The game client is
either installed by hand there or it does not run. With one service as the unit, the closure lands
in a Nix store under $HOME and a user systemd unit starts it, and the box is an ordinary member of
the deployment.
Now move the database from alpha to beta. Clan's unit of change is the machine closure, so the
move rebuilds both hosts and every other service on them comes along for the ride. You cannot read
what will happen before it happens, you cannot build today and deploy on Thursday, and a rollback is
another rebuild. Part 2 puts one flat keyed file between evaluation
and execution. The same move becomes two entries whose keys changed, a diff you can read first, and
a rollback that is the previous file.
The third case fails the most quietly. A clan module that needs another machine's cache signing key builds the path to it by hand:
clanModules/nix-cache-new, trimmedcaPubPath = name: "${config.clan.core.settings.directory}/vars/per-machine/${name}/harmonia/ca-pub/value";
trusted-public-key = if builtins.pathExists (caPubPath name) then builtins.readFile (caPubPath name) else ""; Nothing checks that read, because there is no interface behind it. Rename a generator and the path
misses. Deploy before the key exists and the fallback is "", so a cache nobody can verify renders
into nix.conf exactly like a cache that is fine. Here the same value arrives as an argument, and
the interface that declares it says what kind of value it may be:
the same key, under this designpublicKey = vars.harmonia.ca-pub.value; On the first pass the generator has not run, and the plan carries a row that says so instead of an empty trust anchor. Part 3 is where the typed edge comes from.
The binary cache those keys belong to is the fourth case. It is a bucket in somebody's cloud, so clan has nowhere to put it, and it ends up configured by hand beside the deployment that cannot work without it. Part 6 gives it a home: an external is declared once, set up with a human present, and checked by a probe on every deploy. The deployer reads it and never writes to it, so it cannot corrupt something you share with other people.
The last case is handing the game client to a friend. Today that means adding them to the infrastructure that runs it, with the visibility that comes with membership. Here a guest is a principal holding a grant, the grant is an instance like any other, and deleting it retracts the account, the ingress rule and the network ACL in dependency order.
Each assumption bought something, and the first bought the most. Requiring NixOS everywhere is what lets a service depend on a specific sysctl and then verify it is set, which a container cannot do because it only ever sees its own walled-off environment. Part 5 keeps that check and drops the requirement.
What a deployment says instead
universe-deploy makes the service the unit. An instance is one module, that module owns the
services, and the deployment says which machines each service runs on.
instances.mygame = {
module = mygame.services.default;
settings = {
server.slots = 32;
client.logLevel = "info";
};
placement.every = {
server = { machines = [ "alpha" ]; };
client = { tags = [ "desktop" ]; };
};
};A deployment writes six things and no more: module, settings, members, placement, wire and exposes. Three of them are above. members says which of the module's services this deployment
wants, so members.db.enable = false deletes one and every plan entry it would have produced, and the
instance then names something else to fill the hole. wire fills what a module could not fill from
inside itself, including a hole a cut just opened, and exposes is how an instance opts into being
wired to by another. The last three are the subject of part 3.
That the game has a server and a client is the module's business, and so is anything else it needs to bring along. Which of them you keep is yours.
Settings are keyed by the service that reads them, the same way placement is, so one instance is described through one shape of namespace. A module with a single service keys that one too, because the namespace belongs to the service and nothing forwards a value into it.
Placement names machines two ways and both are membership. machines is a list of names. tags is
every machine carrying any of the tags you list, which is clan's inventory model unchanged, so adding
a laptop to desktop deploys the client to it and removing the tag undeploys it. Nothing is chosen
and nothing is a number. Holding both services under one module is what lets them share values
without either of them naming the other, which is part 3.
The two ways of naming a machine fail differently, and that is the point of having both. A tag is a
set that stays open to machines added later, so a member that cannot host the service drops with a
row and the rest of the tag is placed. The game server declares the platforms it builds for, so one
aarch64 laptop in desktop costs you a row rather than the placement, and nobody has to invent an amd64 tag to keep it out. A machine you named by hand that fails a requirement is an error instead,
because getting beta when you asked for alpha is worse than getting a refused plan.
Letting the planner choose is a different block
Sometimes you do not care which machine, only how many. That is a different question, so it is a different block:
placement.pick.server = {
tags = [ "always-on" ];
count = 1;
strategy = "stable";
};Now the tag is a candidate set rather than the membership, and the planner picks one machine out of it
and leaves the rest alone. Candidates that fail a requirement are silently not candidates, which is
what having alternatives means. What it picked is persisted next to the port allocations, so an
unrelated replan leaves the server where it is, and strategy = "stable" is the deployment saying so
out loud. Retire that machine and the next plan picks another candidate.
A service can appear in both blocks. every.server.machines = [ "alpha" ] beside pick.server = { tags = [ "always-on" ]; count = 1; } pins alpha and lets the planner find one
more, because each block places from its own set and neither constrains the other.
Two blocks rather than one field with a default is a deliberate trade. Everything that only applies to a chosen placement lives in one of them: the persisted allocation row, the ranking of candidates, and the question of whether a healthy service may move on the next replan. Reading a deployment, you can see which services delegated that decision without checking a number on each one. The price is that the pinned-plus-one case is a sum across two lines instead of one.
Clan declared, per service, which machine it should run on. That is still the common case here, and tags is how it stays true of a fleet that grows: the operator states the membership and the service
states its own requirements. What is new is that an operator who does not care which machine can say
that instead, in a block that also says what the planner is allowed to do later.
Why the new atom is a service
The unit has to be expressible on the target, and that is the whole constraint. A game client on macOS is a launchd job and a store path. A metrics agent on an unmanaged Ubuntu box is a user systemd unit under your home directory. Both need a Nix store on the host to receive the closure, and that is the real floor. Neither needs the host to be a NixOS machine, because neither claims the whole host.
A whole-machine closure claims the host by definition, so a tool built on it stays on the fleet it was designed for.
A machine does not disappear when it stops being the unit. It becomes something the services placed on it contribute to, and it carries a key of its own in the plan for that reason. Part 5 is where that side of it lives.