Nix Config Architecture

Maintinance Guidelines

Documentation: https://tj-coding.github.io/nixos-config/

New here? Read Bag of Materials for what exists, and the playbooks for how to change it.

Directory Structure

📂 nixos-config/
├─ 📂 hosts/      Contains configuration of physical machines
├─ 📂 templates/  Groups of modules and functions reused by different machines
├─ 📂 modules/    Groups of moudles for use by different templates
├─ 📂 functions/  Smallest component, who serves a specific function
├─ 📂 apps/       Custom Nix Packages and Docker Files who serves a function

Choosing Folder

Dependency Direction

hosts
  ↓
templates
  ↓
modules
  ↓
functions

Bag of Materials

An inventory of what this flake is made of, and where each piece lives.

Machines

Both machines set networking.hostName = "nixos", so the flake attribute name identifies a host — not the hostname. That name is also what nixos-secrets/secrets/<name>/ is keyed by, which is why it matters.

Templates

A template is the composition layer: it imports modules and sets the service-specific values for one class of machine.

Modules

Functions

Apps

Playbooks

• Adding a New Machine
• Handling Secrets
• Adding a New Docker Compose Project
• University (NAIST) SSH Access

Playbooks

Step-by-step procedures for the work this repository needs most often. Each one is written to be followed on a real machine, in order.

A new machine needs two independent credentials before it can use any secret: a GitHub deploy key for the private nixos-secrets flake input, and the SOPS age key. Handling Secrets covers both, and nix run .#enroll sets them up.

It also needs flakes and the nix-command CLI enabled — without them nothing in this repository can be evaluated. That is covered in Adding a New Machine, and bootstrap-auth (which enroll calls) checks for it up front.

For an inventory of what exists in the repository, see Bag of Materials.

Adding a New Machine

1. Install NixOS

Install a normal NixOS installation on the machine, following the NixOS manual. (If the machine is not applying this repository’s configuration yet, the bootstrap example in flakes/bootstrap/flake.nix is a minimal system for the first install.)

At this point the installer environment has neither flakes nor the unified nix CLI enabled — they are experimental features, off until something turns them on. That is what the next step is for.

2. Enable flakes and the nix-command CLI

This repository is a flake, so every command in this playbook needs nix-command plus flakes. Enable them before enrolling, or bootstrap-auth will stop with a message pointing back here.

Option A — declarative (preferred, and what this repo does)

This is the durable fix: the features are declared in the configuration, so they survive reboots and are not per-user state.

• Fleet-wide, for hosts that import modules/common.nix (which every server template does), it is already declared:

  nix.settings.experimental-features = [ "nix-command" "flakes" ];
  

• For a host that does not import modules/common.nix yet, add that line to hosts/<host-name>/configuration.nix.

See the manual for the option and the full list of features: https://nixos.org/manual/nixos/stable/#opt-nix.settings.experimental-features

Because the features are not on yet, this first rebuild has to pass them on the command line:

sudo nixos-rebuild switch \
  --flake .#<host-name> \
  --show-trace \
  --extra-experimental-features 'nix-command flakes'

Option B — imperative (temporary; no config of this repo needed)

Use this when the machine is brand new, or when you deliberately do not want to touch its configuration — for instance during the very first bootstrap of flakes/bootstrap/flake.nix, which is what nixos-install would use.

• For the current shell only:

  export NIX_CONFIG="$(printf 'experimental-features = nix-command flakes\n')\n$NIX_CONFIG"
  

• Or persist for the current user:

  mkdir -p ~/.config/nix
  printf 'experimental-features = nix-command flakes\n' >> ~/.config/nix/nix.conf
  

To confirm the features are active:

nix config show experimental-features
# fetch-tree flakes nix-command

Two things that trip people up here:

• Nix reads its configuration once, at start-up. After enabling the features durably, a shell that was already running still lacks them — open a new shell, and remember that nixos-rebuild must be started after the rebuild that enabled the features in order to see them.
• Any bare nix invocation made before the features are enabled needs --extra-experimental-features 'nix-command flakes'; that is why the clone step below spells it out.

3. Clone the repository

1. Install git

   nix shell nixpkgs#git --extra-experimental-features 'nix-command flakes'
   

2. Then clone the repository:

   git clone https://github.com/TJ-coding/nixos-config.git ~/nixos-config
   cd ~/nixos-config
   

4. Enroll the machine

Run the repository’s enrollment command on the new machine:

cd ~/nixos-config
nix run ".?dir=flakes/bootstrap#enroll"

Pass the host name explicitly unless the machine’s hostname happens to match its flake attribute and host directory:

nix run ".?dir=flakes/bootstrap#enroll" -- highperformancecomputing

Prefer the bootstrap flake for this first run: it depends on nixpkgs alone, so no missing private input can affect it, and it carries nixosConfigurations.bootstrap as well.

To be precise about why: flake inputs are lazy, so nix run .#enroll does evaluate without a deploy key — evaluating packages never touches the private secrets input. What fails before the key exists is anything that evaluates a host configuration, which is what the rebuild after enrollment does:

$ nix eval .#nixosConfigurations.highperformancecomputing.config.system.build.toplevel.drvPath
error: Failed to fetch git repository 'ssh://git@github.com/TJ-coding/nixos-secrets.git'

Once the credentials are in place the two entry points are equivalent.

The ?dir= form is what makes this work: flakes/bootstrap is a flake inside this repository, and pointing Nix at the repository root keeps the helper scripts in apps/ reachable from it.

This generates hosts/<hostname>/hardware-configuration.nix and then runs bootstrap-auth, which sets up the GitHub deploy key that makes the private secrets flake input fetchable, the SOPS age key — plus a user-readable copy at ~/.config/sops/age/keys.txt, so that sops can decrypt and edit secrets from that host — a checkout of the secrets repository at ~/nixos-secrets, and NetBird membership, registering the peer with the shared setup key from nixos-secrets so it is not subject to the account’s Peer Session Expiration. NetBird is enrolled last, once the age key exists to decrypt that setup key. Should that first enrolment ever fall back to interactive SSO, the deployed configuration clears the peer’s expiration flag at the next rebuild (functions/netbird-session-expiration.nix).

The credentials step is the one that is easy to get half-right — a deploy key that exists but does not match the host’s key looks fine and fails later with Permission denied (publickey). bootstrap-auth verifies each step and prints a PASS/FAIL summary. See Handling Secrets for what it is doing and how to do it by hand.

If the host will not use any secrets, it can be built without step 3’s age key; it simply cannot decrypt anything.

5. Configure the host

1. Open the configuration

   nano ~/nixos-config/hosts/<host-name>/configuration.nix
   

2. Add a template to the configuration file

   imports =
   [ # Include the results of the hardware scan.
     ./hardware-configuration.nix
     ../../templates/artifacts.nix
   ];
   

6. Add the Configuration as a Flake

1. Open the flake file

   nano ~/nixos-config/flake.nix
   

2. Add a new nixosConfigurations entry

   nixosConfigurations.<host-name> = nixpkgs.lib.nixosSystem {
     system = "x86_64-linux";
     specialArgs = {
       kohaku-hub = kohaku-hub;
       rustfs = rustfs;
       secrets = secrets;
     };
     modules = [
         ./hosts/<host-name>/configuration.nix
         ./hosts/<host-name>/hardware-configuration.nix
         sops-nix.nixosModules.sops
     ];
   };
   

3. Replace <host-name> with the actual host name

specialArgs only needs the inputs that the host’s templates actually use. Passing secrets to a host that imports no secrets-using module is harmless but pointless — and, as Handling Secrets explains, it is also why a missing deploy key can stay hidden until the first secret is added.

If the host consumes secrets, its encrypted files belong in nixos-secrets/secrets/<host-name>/, matching the flake attribute name.

7. Apply the configuration

1. Check the configuration

   nix flake check --show-trace
   

   If nix reports experimental Nix feature 'nix-command' is disabled, step 2 was skipped or the current shell predates it — see 2. Enable flakes and the nix-command CLI.
2. Apply the configuration

   sudo nixos-rebuild switch \
   --flake .#<host-name> \
   --show-trace
   

3. Confirm secrets landed, if the host uses them

   sudo ls /run/secrets
   

Handling Secrets

Secrets do not live in this repository. They live in a separate, private repository, TJ-coding/nixos-secrets, encrypted at rest with SOPS using age. sops-nix decrypts them during system activation and exposes the plaintext under /run/secrets/..., where systemd units read them.

The rule that follows from this: the machine needs two independent credentials before it can use a secret at all.

1. a way to read the encrypted secrets repository (a GitHub SSH key), and
2. the age private key that decrypts them.

Missing either one produces a different error, and neither is obvious from the Nix code. This playbook is mostly about those two credentials.

1. The pieces

The age key is the same on every host (infrastructure.age.key) — it is a single shared secret, so anything that host can decrypt, every host can decrypt. That is a deliberate simplicity trade-off for a small fleet. If that ever stops being acceptable, see §7.

The repository layout is keyed by the flake attribute name, not by networking.hostName (two of the machines are both called nixos, so the hostname cannot identify a host):

nixos-secrets/
├── .sops.yaml                       # who may decrypt what
└── secrets/
    ├── shared/                      # read by more than one host
    │   ├── university-ssh-key.yaml  # -> functions/ssh-uni.nix
    │   └── netbird.yaml             # -> functions/netbird*.nix (setup_key, api_token)
    └── artifacts/                   # nixosConfigurations.artifacts
        ├── kohaku-hub.env           # dotenv: whole file is one service env
        └── rustfs.yaml              # YAML: one key per sops.secrets entry

2. Why a fresh machine cannot read secrets

Three things have to be true, and none of them is set up by installing NixOS:

0. The nix-command and flakes experimental features must be enabled. Nothing in this repository can be evaluated without them, since it is a flake input fetched through the unified nix CLI. On a fresh install they are off by default, and the first thing that fails is any nix subcommand:

$ nix flake check
warning: 'flake-check' is a deprecated alias for 'check'
error: experimental Nix feature 'nix-command' is disabled; add '--extra-experimental-features nix-command' to enable it

This is a prerequisite for the two credentials below, not a third credential. See Adding a New Machine, or the option reference at https://nixos.org/manual/nixos/stable/#opt-nix.settings.experimental-features. modules/common.nix declares the features for every host that imports it, and bootstrap-auth refuses to run without them, so a host that has been through enroll already has this covered.

a. The private flake input must be fetchable. flake.nix pulls the secrets repository as an input:

secrets = {
  url = "git+ssh://git@github.com/TJ-coding/nixos-secrets.git";
  flake = false;
};

A git+ssh:// URL is fetched by git over SSH, so it needs an SSH key that GitHub accepts for that repository. gh auth setup-git is not enough: it configures an HTTPS credential helper, and SSH URLs never consult it. Without a suitable key the failure is:

error: Failed to fetch git repository 'ssh://git@github.com/TJ-coding/nixos-secrets.git'
git@github.com: Permission denied (publickey).

Note that this only bites when the input is actually used. A host whose template imports no secrets-using module never forces the input, so the flake evaluates happily and the missing credential stays invisible until the first service that needs a secret is added.

b. The age key must be installed — twice, for two different readers. functions/sops.nix declares sops.age.keyFile = "/var/lib/sops-nix/infrastructure.age.key", which is what sops-nix reads at boot; if that file is absent, sops-nix silently has nothing to decrypt and /run/secrets/ stays empty.

That copy is root-only (mode 600), which is right for a service and wrong for a human. This is the trap: on a host that decrypts secrets perfectly well at boot, editing one as your own user fails, because the only identity on the machine is unreadable to you:

Failed to get the data key required to decrypt the SOPS file.
  age18tqkdukn4cfs0es7zknm9eacpagmfqrfaaclcdzhs46cylrqg92smff6qm: FAILED
    - | failed to create reader for decrypting sops data key with age: identity
      | did not match any of the recipients ... failed to open
      | SOPS_AGE_KEY_FILE file: open
      | /var/lib/sops-nix/infrastructure.age.key: permission denied.
Recovery failed because no master key was able to decrypt the file.

The fix is a second copy at sops’ own default location, ~/.config/sops/age/keys.txt, which bootstrap-auth installs and which needs no SOPS_AGE_KEY_FILE:

sudo install -m 600 -o "$(id -un)" -g "$(id -gn)" \
  /var/lib/sops-nix/infrastructure.age.key ~/.config/sops/age/keys.txt

It hands over nothing sudo did not already: the same user can sudo cat the root copy. See §9 for custody.

3. Bootstrapping a new host

Run this on the freshly installed machine, from a checkout of this repository:

cd ~/nixos-config
nix run ".?dir=flakes/bootstrap#enroll"

It is the bootstrap flake and not nix run .#enroll because it depends on nixpkgs alone, so no missing private input can affect it — and because it also carries nixosConfigurations.bootstrap for the bare-metal install.

Strictly, nix run .#enroll from the main flake does work here too: flake inputs are lazy, so evaluating packages never touches secrets. What does fail before the deploy key exists is anything that evaluates a host configuration — see §2a:

$ nix eval .#nixosConfigurations.highperformancecomputing.config.system.build.toplevel.drvPath
error: Failed to fetch git repository 'ssh://git@github.com/TJ-coding/nixos-secrets.git'

Once the credentials are in place the main flake is fully usable and nix run .#enroll behaves identically.

enroll writes hosts/<flake-host>/hardware-configuration.nix and then runs bootstrap-auth, which does the following, idempotently:

1. ensures an SSH key exists, offers to register it as a read-only deploy key on nixos-secrets, and verifies the result with git ls-remote;
2. installs the age key for sops-nix (paste an existing one, or generate a new one) and a user-readable copy at ~/.config/sops/age/keys.txt, so that sops can decrypt — and therefore edit — secrets on that host (§2b);
3. checks the key’s recipient against nixos-secrets/.sops.yaml, and leaves a checkout behind at ~/nixos-secrets (which is also what step 4 reads);
4. brings the machine onto NetBird, registering it with the setup key from secrets/shared/netbird.yaml, which it decrypts using the age key from step 2;
5. prints a PASS/FAIL summary and exits non-zero if the repository is still unreachable.

NetBird runs last because it is the one step that consumes a secret this script has to decrypt itself, and it can only do that once step 2 has installed the age key. Enrolling through an interactive SSO login instead — which is what this script used to do first — silently opts the peer into the account’s Peer Session Expiration (24h by default), so the machine drops off the mesh daily until somebody runs netbird up on it by hand. Peers registered with a setup key are exempt from session expiration.

That also makes NetBird the only skippable step: with no key to hand, the host is simply not on the mesh yet, and the deployed configuration registers it at the first rebuild (see functions/netbird.nix). NETBIRD_SETUP_KEY_FILE=/path/to/key overrides the decryption, for the case where the secrets repository cannot be read yet.

Doing it by hand

# 1. GitHub: a key GitHub accepts for the private repository
ssh-keygen -t ed25519 -C "$(hostname)" -f ~/.ssh/id_ed25519      # if needed
gh auth login                                                    # as the repo owner
gh repo deploy-key add ~/.ssh/id_ed25519.pub \
  --repo TJ-coding/nixos-secrets --title "$(hostname)"

# verify
ssh -T git@github.com                       # "Hi TJ-coding/nixos-secrets! ..."
git ls-remote git@github.com:TJ-coding/nixos-secrets.git HEAD

# 2. The age key (copy it from a host that already has it)
ssh <existing-host> sudo cat /var/lib/sops-nix/infrastructure.age.key

sudo install -d -m 700 /var/lib/sops-nix
sudo tee /var/lib/sops-nix/infrastructure.age.key >/dev/null   # paste, then Ctrl-D
sudo chmod 600 /var/lib/sops-nix/infrastructure.age.key
sudo chown root:root /var/lib/sops-nix/infrastructure.age.key

# ...and a copy your own user can read, or `sops` cannot edit anything here (§2b)
sudo install -m 600 -o "$(id -un)" -g "$(id -gn)" \
  /var/lib/sops-nix/infrastructure.age.key ~/.config/sops/age/keys.txt

# ...and the checkout secrets are edited from (NetBird reads it in step 3 too)
git clone git@github.com:TJ-coding/nixos-secrets.git ~/nixos-secrets

# 3. NetBird, with the shared setup key, so the peer never expires
sudo SOPS_AGE_KEY_FILE=/var/lib/sops-nix/infrastructure.age.key \
  sops --decrypt --extract '["setup_key"]' \
  ~/nixos-secrets/secrets/shared/netbird.yaml > /tmp/netbird-setup-key
chmod 600 /tmp/netbird-setup-key
sudo netbird up --setup-key-file /tmp/netbird-setup-key
rm -f /tmp/netbird-setup-key

# 4. Confirm the key can actually decrypt something
cd ~/nixos-config
nix flake archive                    # forces every input, including secrets
sudo nixos-rebuild switch --flake .#<host>
sudo ls /run/secrets                 # populated only for hosts that use secrets

Only hosts that define sops.secrets.* need the age key. A pure compute host can be built without one; it just cannot decrypt anything.

4. Adding or changing a secret

git clone git@github.com:TJ-coding/nixos-secrets.git ~/nixos-secrets

# dotenv file: the whole file is exported to one service
sops ~/nixos-secrets/secrets/artifacts/kohaku-hub.env

# YAML file: one key per sops.secrets entry
sops ~/nixos-secrets/secrets/artifacts/rustfs.yaml

# shared YAML: the NetBird setup key and the API token that clears peer session
# expiration (functions/netbird.nix, functions/netbird-session-expiration.nix)
sops ~/nixos-secrets/secrets/shared/netbird.yaml

git -C ~/nixos-secrets commit -am "Update <service> secrets"
git -C ~/nixos-secrets push

# pin the new revision in this repository
cd ~/nixos-config
nix flake update secrets
sudo nixos-rebuild switch --flake .#artifacts

sops picks the age recipient from .sops.yaml automatically, so you never paste one while editing. It still needs an identity to decrypt with — the user-readable age key from §2b — and that is what bootstrap-auth sets up (§3).

Three more assumptions this block makes:

• The checkout exists. bootstrap-auth leaves one at ~/nixos-secrets (§3); otherwise the git clone above is the first step.
• You can push. The deploy key a bootstrapped host gets is read-only by design, so git push from that host is refused:

  ERROR: The key you are authenticating with has been marked as read only.
  

  Push from the machine where you normally edit secrets, or run gh auth login on the host and push over HTTPS.
• Commits work. git commit needs an author identity, and a freshly bootstrapped host often has none — it fails with “Author identity unknown”, which reads like a repository problem. Set one in the checkout before the first commit (bootstrap-auth reports this at enrolment):

  git -C ~/nixos-secrets config user.name  '<your name>'
  git -C ~/nixos-secrets config user.email '<your email>'
  

5. Wiring a secret into a module

dotenv — one file, all variables, useful for EnvironmentFile:

sops.secrets."kohaku-hub-env" = {
  sopsFile = "${secrets}/secrets/artifacts/kohaku-hub.env";
  format = "dotenv";
};

systemd.services.kohaku-hub.serviceConfig.EnvironmentFile =
  config.sops.secrets."kohaku-hub-env".path;

YAML — select individual keys, useful for services that take a file path:

sops.secrets."rustfs-access-key" = {
  sopsFile = "${secrets}/secrets/artifacts/rustfs.yaml";
  key = "access_key";
  restartUnits = [ "rustfs.service" ];
};

A file, not an environment variable — a private key has to be readable by one local user and nobody else, so it needs owner/mode (functions/ssh-uni.nix):

sops.secrets."university-ssh-key" = {
  sopsFile = "${secrets}/secrets/shared/university-ssh-key.yaml";
  key = "private_key";
  owner = "tj-coding";
  mode = "0400";
};

The variable secrets is the flake input, passed down through specialArgs:

specialArgs = { inherit kohaku-hub rustfs secrets; };

Always reference config.sops.secrets.<name>.path; never read the file at evaluation time, because at evaluation time the decrypted file does not exist.

A credential for one systemd unit — the NetBird login oneshot does not put its key in an environment variable at all. The module passes setupKeyFile to systemd as LoadCredential, so the key is exposed to netbird up as $CREDENTIALS_DIRECTORY/setup-key and never appears in the unit’s environment or in /proc/*/environ (functions/netbird.nix):

sops.secrets."netbird-setup-key" = {
  sopsFile = "${secrets}/secrets/shared/netbird.yaml";
  key = "setup_key";
};

services.netbird.clients.default.login = {
  enable = true;
  setupKeyFile = config.sops.secrets."netbird-setup-key".path;
  systemdDependencies = [ "sops-install-secrets.service" ];
};

The systemdDependencies entry is the load-bearing part: without it the login unit is free to start before sops-nix has written the file, and it reads a path that does not exist yet.

The same pattern carries the NetBird personal access token that functions/netbird-session-expiration.nix uses to clear the account’s 24h Peer Session Expiration on this host’s own peer (LoadCredential = "api-token:${…}", read by the script as $CREDENTIALS_DIRECTORY/api-token). Create the token once in the NetBird dashboard — user menu → Personal access tokens → Create — under a user who may see and update these peers (an admin, or whoever enrolled them), then add it to secrets/shared/netbird.yaml as api_token and pin the new revision, exactly as in §4.

That key is a prerequisite, not an optional nicety. sops-install-secrets aborts the whole activation when a declared sops.secrets.*.key is absent from the file (the key 'api_token' cannot be found), so a rebuild that lands before the key exists stops with a decryption error rather than silently leaving the peer on the daily-expiry path.

6. Verifying that it actually works

# Is the private input fetchable at all? (forces every input)
nix flake archive

# Which key does this host have, and is it the authorised one?
sudo age-keygen -y /var/lib/sops-nix/infrastructure.age.key
grep -oE 'age1[0-9a-z]{58}' ~/nixos-secrets/.sops.yaml

# Can this key decrypt a file?
cd ~/nixos-secrets && sops -d secrets/artifacts/rustfs.yaml >/dev/null && echo ok

# What did sops-nix actually produce, and when?
sudo ls -la /run/secrets
sudo journalctl -u sops-nix -b --no-pager

7. Rotating the age key

The recipient lives in exactly one place, nixos-secrets/.sops.yaml:

creation_rules:
  - path_regex: .*
    age: age1...

To add a host’s newly generated key, add its recipient to that list and re-encrypt every file — SOPS wraps the data key once per recipient, so the data itself does not change and everyone keeps reading the same plaintext:

sops updatekeys secrets/artifacts/rustfs.yaml
sops updatekeys secrets/artifacts/kohaku-hub.env

git commit -am "Add <host> to age recipients" && git push
cd ~/nixos-config && nix flake update secrets

To remove a host, delete its recipient, sops updatekeys every file, and rotate anything that host could have read. Removing a recipient does not undo access it already had.

Deploy keys are revoked independently, which is the point of --allow-read-only:

gh repo deploy-key list --repo TJ-coding/nixos-secrets
gh repo deploy-key delete <id> --repo TJ-coding/nixos-secrets

8. Troubleshooting

9. Rules of custody

• Never commit the age private key, a decrypted file, or a token into either repository. .gitignore is not a safety net.
• Keep an offline copy of the age key somewhere you would keep a password manager. Losing it means every secret has to be re-encrypted from scratch.
• Prefer per-machine read-only deploy keys over personal SSH keys: they are scoped to one repository and can be revoked without touching your GitHub account.
• Installing the key must be done as root with mode 600: sudo install -m 600 -o root -g root <key> /var/lib/sops-nix/infrastructure.age.key.
• The user-readable copy at ~/.config/sops/age/keys.txt (§2b) is that same shared key in a second place. It is what makes sops work on the host; keep it mode 600, and never copy it into either repository.
• Rotate a secret by editing it with sops, not by rewriting the file by hand; writing plaintext and re-encrypting later leaks it into shell history and backups.

10. Known rough edges

• A bootstrapped host can read nixos-secrets but not push to it: the deploy key bootstrap-auth registers is read-only by design. Editing a secret from a host is therefore a two-machine dance unless you add a write-enabled deploy key or authenticate gh there.
• secrets/<flake-host>/ is named after the flake attribute, while both machines set networking.hostName = "nixos". Auto-deriving secret paths from the hostname is therefore not possible today; templates set secrets-path/rustfs-secrets-path explicitly. Renaming a host means renaming its directory in the secrets repository.
• apps/kohaku-hub and functions/rustfs.nix have defaults like secrets/kohaku-hub.env that do not match the layout above. They are always overridden by templates/artifacts.nix; do not rely on the defaults.
• secrets/artifacts/kohaku-hub.env currently contains LAKEFS_AUTH_ENCRYPT_SECRET_KEY twice. In a dotenv file the last definition wins, so the first one is dead. Worth cleaning up in the secrets repository.

Adding a New Docker Compose Project

This playbook captures the pattern used by this repository for adding a new containerized service managed by Docker Compose on NixOS.

The key idea is simple: keep the service declarative, keep state under well-known host paths, keep secrets SOPS-managed, and keep networking explicit.

1. Add the compose file and the service module

Create a service directory under apps/ and add:

• a compose file such as apps/my-service/docker-compose.yml
• a Nix module such as apps/my-service/my-service.nix

The service module should usually:

• define an option set under services.<name>
• declare the app-specific settings such as enable, base-url, secrets-path, etc.
• create a sops.secrets.* definition for environment variables
• generate the systemd unit that runs docker compose ... up --build
• point to the compose file under /etc using environment.etc if needed

A good pattern looks like this:

{ config, lib, pkgs, ... }:

let
  cfg = config.services.my-service;
in {
  options.services.my-service = {
    enable = lib.mkEnableOption "My Service";
    secrets-path = lib.mkOption {
      type = lib.types.str;
      default = "secrets/my-service.env";
    };
  };

  config = lib.mkIf cfg.enable {
    sops.secrets."my-service-env" = {
      sopsFile = "${secrets}/${cfg.secrets-path}";
      format = "dotenv";
    };

    systemd.services.my-service = {
      wantedBy = [ "multi-user.target" ];
      after = [ "docker.service" ];
      requires = [ "docker.service" ];
      serviceConfig = {
        EnvironmentFile = config.sops.secrets."my-service-env".path;
        ExecStart = "${pkgs.docker}/bin/docker compose -f /etc/my-service-compose.yml up --build";
        ExecStop = "${pkgs.docker}/bin/docker compose -f /etc/my-service-compose.yml down";
      };
    };
  };
}

Then import the module from the host template:

imports = [
  ../apps/my-service/my-service.nix
];

2. Keep the app source and runtime state in stable locations

For projects like KohakuHub, the app source should be kept in an explicit immutable source path rather than being modified in place.

The repo uses a stable state directory such as:

/var/lib/kohaku-hub
/var/lib/kohaku-hub/data
/var/lib/kohaku-hub/data/hub-meta

and creates those directories declaratively with systemd.tmpfiles.rules:

systemd.tmpfiles.rules = [
  "d /var/lib/kohaku-hub 0755 root root -"
  "d /var/lib/kohaku-hub/data 0755 root root -"
  "d /var/lib/kohaku-hub/data/hub-meta 0755 root root -"
];

This matters because Docker bind mounts should target stable directories under /var/lib or other known host paths; these are easier to back up, inspect, and reconstruct.

Overlay strategy for source immutability

When an application needs an immutable source tree but still needs a writable layer, use an overlay mount pattern. This repo previously used the pattern:

fileSystems."/var/lib/kohaku-hub/merged" = {
  device = "overlay";
  fsType = "overlay";
  options = [
    "lowerdir=${kohaku-hub}"
    "upperdir=/var/lib/kohaku-hub/upper"
    "workdir=/var/lib/kohaku-hub/work"
  ];
};

This gives you:

• a read-only lower layer from the upstream source tree
• a writable upper layer under /var/lib/kohaku-hub/upper
• a merged view mounted at /var/lib/kohaku-hub/merged

This is useful for reproducibility and making container build contexts deterministic. In many cases, if the upstream project is already static and buildable, it is simpler to use the upstream source directly instead of a custom overlay layer. The repo currently prefers the simpler and more robust approach of mounting the upstream source directly when no writable overlay is needed.

3. Handle volume paths carefully

Volume paths are one of the most important parts of a Docker Compose project.

The practical rule is:

• do not put mutable state under random ephemeral directories
• bind persistently to /var/lib/...
• create the directories explicitly with systemd.tmpfiles.rules
• mount them as named volumes or host bind mounts in the compose file

Example from this repo:

volumes:
  - /var/lib/kohaku-hub/data/hub-meta/lakefs-data:/var/lakefs/data
  - /var/lib/kohaku-hub/data/hub-meta/postgres-data:/var/lib/postgresql/data
  - /var/lib/kohaku-hub/data/hub-meta/valkey-data:/data

This gives each dependency its own durable state and keeps data ownership consistent across rebuilds.

Avoid:

• mounting directly into project directories that are not persistent
• relying on auto-created directories without declarative rules
• mixing mutable state with immutable source trees

4. Handle networking explicitly

For containerized services, networking needs to be predictable.

The repo uses a named default network:

networks:
  default:
    name: hub-net

and services then reach each other by service name, for example:

hub-api:
  environment:
    KOHAKU_HUB_DATABASE_URL: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/kohakuhub
    KOHAKU_HUB_LAKEFS_ENDPOINT: http://lakefs:28000
    KOHAKU_HUB_CACHE_URL: redis://valkey:6379/0

This makes internal communication container-local and stable.

For external access, expose only the ports you need:

ports:
  - "28080:80"
  - "48888:48888"

and make sure the host firewall allows those ports:

networking.firewall.allowedTCPPorts = [ 22 28080 ];

If the service must reach a host service such as the local MinIO or local S3 endpoint, use extra_hosts:

extra_hosts:
  - "host.docker.internal:host-gateway"

This is useful when the containerized app needs to talk to a process running on the host machine.

5. Handle secrets with SOPS and dotenv files

The credentials a host needs before any of this can decrypt anything (GitHub deploy key, SOPS age key) are covered in Handling Secrets.

nixosConfigurations.artifacts = nixpkgs.lib.nixosSystem {
    system = "x86_64-linux";
    specialArgs = {
    kohaku-hub = kohaku-hub;
    rustfs = rustfs;
    secrets = secrets;
    };
    modules = [
        ./hosts/artifacts/configuration.nix
        ./hosts/artifacts/hardware-configuration.nix
        sops-nix.nixosModules.sops
    ];
};

Secrets should not be embedded in the compose file or stored as plain values in the repo.

This repo uses a SOPS-managed .env file, for example:

sops.secrets."kohaku-hub-env" = {
  sopsFile = "${secrets}/${cfg.secrets-path}";
  format = "dotenv";
};

Then the service injects it with:

EnvironmentFile = config.sops.secrets."kohaku-hub-env".path;

and the compose file reads the values as environment variables:

environment:
  KOHAKU_HUB_SESSION_SECRET: ${KOHAKU_HUB_SESSION_SECRET}
  KOHAKU_HUB_ADMIN_SECRET_TOKEN: ${KOHAKU_HUB_ADMIN_SECRET_TOKEN}
  KOHAKU_HUB_DATABASE_KEY: ${KOHAKU_HUB_DATABASE_KEY}

This keeps secrets out of the repository while preserving a declarative runtime shape.

Recommended secret layout

For each app:

• keep a single secret file under the external secrets repo
• store all environment variables for that service in a dotenv file
• point sopsFile to that file
• only expose the minimum set required to the app container

6. Template for a new Compose-backed app

Use this checklist when adding a new project:

1. Add the project under apps/<project>/
2. Create docker-compose.yml
3. Add a Nix module that exposes services.<project> options
4. Create stable data directories under /var/lib/<project>
5. Add the systemd.tmpfiles.rules entries
6. Add the service to the host template imports
7. Expose required ports in the firewall
8. Define network names and dependencies explicitly
9. Keep all secrets in SOPS-managed dotenv files
10. Rebuild the host with sudo nixos-rebuild switch --flake .#artifacts

7. Operational example from this repo

The live pattern here is:

• build an immutable source or Nix-built frontend artifact
• mount it read-only into a container
• keep mutable state under /var/lib/kohaku-hub/data/...
• let the app talk over the default Docker network
• load environment values from SOPS-managed secrets instead of ad hoc shell exports

That is the core pattern to follow for new Compose projects in this repository.

University (NAIST) SSH Access

The HPC host reaches the university filesystem over SSH using functions/ssh-uni.nix. Everything needed is in git: the client configuration, the pinned gateway host key, and the private key (encrypted in TJ-coding/nixos-secrets).

How it is put together

The key is never copied into ~/.ssh. SSH reads it straight out of /run/secrets, which sops-nix writes at activation time, so a rebuilt machine gets the access back without anyone remembering which file to scp.

services.university-ssh = {
  enable = true;
  remoteUser = "tomoyuki-j";       # the university account
  keyOwner = "tj-coding";          # the local account that may read the key
  computeHosts = [ "pine11" "pine12" "pine13" ];
};

Using it

ssh naist                 # gateway, sh.naist.jp
ssh pine11                # lab node, via ProxyJump through the gateway
rsync -av foo/ pine11:~/  # works unattended, no passphrase prompt

About the key

The key is a dedicated ed25519 key, uni-ssh-hpc, with no passphrase.

That is deliberate. The key that was originally copied to the artifact host was the workstation’s personal id_rsa, which is passphrase-protected; without an agent holding the passphrase it can never authenticate, so unattended use (rsync, systemd units, Slurm jobs) fails with Permission denied (publickey). A per-host key with no passphrase is what makes the access actually reproducible. Anything that can read /run/secrets/university-ssh-key can use the key, which is why it is 0400 and owned by a single user.

Adding a new host

1. Generate a key on the new machine:

   ssh-keygen -t ed25519 -N '' -C "uni-ssh-<hostname>" -f /tmp/uni_ssh
   

2. Authorise the public key on both university filesystems. This trips people up: sh.naist.jp serves ~ from local ZFS, while pine11-pine13 serve ~ from the lab NFS export, so the same path is two different files.

   key="$(cat /tmp/uni_ssh.pub)"
   for host in sh.naist.jp pine11; do
     ssh "$host" "printf '%s\n' '$key' >> ~/.ssh/authorized_keys"
   done
   

3. Check it authenticates with only that key before storing it:

   ssh -i /tmp/uni_ssh -o IdentitiesOnly=yes -o BatchMode=yes naist hostname
   

4. Encrypt it into the secrets repository:

   cd ~/nixos-secrets
   cp /tmp/uni_ssh /tmp/key-material
   { echo 'private_key: |'; sed 's/^/  /' /tmp/key-material; } \
     > secrets/shared/university-ssh-key.yaml
   sops --encrypt --in-place secrets/shared/university-ssh-key.yaml
   shred -u /tmp/key-material
   git add -A && git commit -m "Add university SSH key for <hostname>" && git push
   

5. Point the host at it and rebuild:

   imports = [ ../../functions/ssh-uni.nix ];
   services.university-ssh.enable = true;
   

   cd ~/nixos-config
   nix flake update secrets
   sudo nixos-rebuild switch --flake .#<hostname>
   

6. Verify the decrypted key is the right one:

   sudo ssh-keygen -lf /run/secrets/university-ssh-key
   

Removing access

Delete the key’s line from ~/.ssh/authorized_keys on both university filesystems and remove the entry from secrets/shared/university-ssh-key.yaml.