Codifying a Homelab with Ansible — Fail-Closed Defense-in-Depth Infrastructure

Turning my whole home server into code

To run an AI-agent platform on my home server (homelab), I hand-built the entire stack: container runtime, network isolation, encrypted DNS, kernel hardening, and audit logging. It worked, but one thing kept nagging at me. If I ever reinstalled the OS, I wasn’t confident I could rebuild the same thing. Exactly what I had changed and where, and which security settings I might have forgotten to apply — in the end, it all lived only in my head.

So I encoded the build procedure itself in Ansible. It ended up as 9 roles and roughly 3,900 lines. It doesn’t just turn the steps into code — it goes as far as stopping the build when a setting is dangerous. If you try to proceed with an empty allowlist or an old Podman, an assert fails before the playbook even starts running.

This article walks through excerpts of that setup — a defense-in-depth homelab built on rootless Podman + Quadlet. Real hostnames, IPs, usernames, tokens, and component names are either redacted or generalized, and the code is shown as excerpts that convey the key points. Wherever proper nouns are involved, I generalized variable and file names; where things are generic and not sensitive, I kept them close to the real thing.

The setup in numbers

Since I can’t share what’s actually running, I’ll describe it by the size of the code instead. The figures below are measured from this Ansible setup.

Overall architecture

┌─────────────┐   ansible-playbook    ┌──────────────────────────────┐
│Command host │ ───── over SSH ─────▶ │   Production host (target)   │
│  (operator) │   apply --tags …      │                              │
└─────────────┘                       │  ┌────────────────────────┐  │
                                      │  │ bootstrap: OS/pkg/UFW  │  │
                                      │  │            /DoT/helpers│  │
                                      │  ├────────────────────────┤  │
                                      │  │ podman 5.x rootless    │  │
                                      │  │  + cgroup v2 + Quadlet │  │
                                      │  ├──────────┬─────────────┤  │
   internal-net ◀─────────────────────┼──│ policy / │ notification│  │
   (inter-service)                    │  │ LLM svc  │ / relay     │  │
   External API ◀─────────────────────┼──│ gateway  │             │  │
                                      │  ├──────────┴─────────────┤  │
                                      │  │ audit: logs+signed hb  │  │
                                      │  └────────────────────────┘  │
                                      └──────────────────────────────┘

The responsibilities are split across nine roles (names generalized).

Stop the build on a dangerous setting

The part I cared about most in this setup is stopping the playbook on the spot the moment a value fails to meet the preconditions. There are 13 fail-closed assert checks, and any attempt to proceed with malformed input fails right there.

Take the authorization daemon’s “allowed provider list,” for example. If it’s empty, contains duplicates, or includes unexpected characters, it can lead to strange behavior downstream. So I validate its shape before anything runs. The following is a generic pattern close to the real one, with only the variable names generalized and the logic left intact.

- name: Fail when provider allowlist is unsafe
  ansible.builtin.assert:
    that:
      - allow_providers is sequence
      - allow_providers is not string
      - allow_providers | length > 0
      - allow_providers | reject('match', '^[A-Za-z0-9][A-Za-z0-9_.-]*$') | list | length == 0
      - allow_providers | unique | list | length == allow_providers | length
    fail_msg: "allow_providers must be a non-empty list of unique provider names."

It must be a sequence, non-empty, with each element restricted to the expected character set, and free of duplicates. If even one of these five conditions is missed, it stops. Version dependencies are gated on the same principle.

- name: Fail if Podman version below 5
  ansible.builtin.assert:
    that:
      - podman_ver.stdout is search('podman version 5')
    fail_msg: "Podman 5+ required. Got: {{ podman_ver.stdout }}"

If you make precondition checks fail-open (letting them pass anyway even when they fail), you end up moving ahead without noticing the mishap. My default is that validation stops when it fails.

Place containers declaratively with Quadlet

Instead of running docker run by hand, I distribute Podman’s Quadlet units (.container / .network) from templates and let systemd take over. Networks are separated by purpose, expressing the defense-in-depth boundaries as files. The distribution task itself looks like this (pseudo-code with generalized file names).

# Pseudo-code (file names generalized)
- name: Deploy internal network unit
  ansible.builtin.template:
    src: internal-net.network.j2
    dest: "/home/{{ svc_user }}/.config/containers/systemd/internal-net.network"
    owner: "{{ svc_user }}"
    group: "{{ svc_user }}"
    mode: '0644'

What matters is the content of the network unit being distributed: it declares in the file that there is no external egress. With Podman Quadlet, a single line — Internal=true — is enough (only the network name and label are generalized; the key points are unchanged).

[Unit]
Description=Internal service network (no external egress)

[Network]
NetworkName=internal-net
Driver=bridge
Internal=true          # no egress to the outside
Subnet=10.89.0.0/24
Label=zone=internal

Containers placed on an Internal=true network can only talk to one another. Only the components that genuinely need outbound access are attached to a separate egress network. Restricting where things can communicate at the level of the network definition lets you stop an accident — like an application-side bug that inadvertently reaches outside — one layer lower down.

Make the code traceable back to the design docs

At the top of each role, I place a one-line comment pointing to the corresponding design spec (the path is a generalized example).

---
# spec: docs/infra/<NN>-<topic>.md §X.Y

Under docs/infra/ there are numbered design docs (01–06), mapped to the code. It’s unglamorous, but being able to trace “why is this setting here” from the code when you look back six months later is a big deal. Even for settings you wrote yourself, you forget the reasons more often than you’d expect.

What changed after moving away from manual work

• With apply --tags <stage>, you can build everything stage by stage, from the OS base to the container platform to auditing. Secrets are entered from the console, and each stage is then validated.
• Because the configuration is code, even after reinstalling the OS I can rebuild the same thing with the same procedure.
• Secrets are stored separately by purpose: the authorization daemon’s secrets in a root-owned file, the service user’s CLI authentication managed separately, and the keys generated for the internal proxy in a root-owned environment file. The inventory holds only the minimum, such as connection details.
• Before a run, a preflight script and the assert statements at the top of the playbook mechanically check the allowed tags and preconditions.

By moving from manual SSH to “infrastructure you can validate as code,” the person-dependence was taken out of the build. Forgotten settings get caught for me by the 13 assert checks instead.

If you want to copy this, start here

You don’t need to copy the whole Ansible setup as-is. What works is the habit of “putting a fail-closed assert at the top of each role.” Proceeding anyway on an empty list, an old version, or unexpected input is where accidents start, so stop each of those, one line at a time. You can add this to your existing Ansible or Terraform bit by bit.

For help with turning existing infrastructure into IaC, building out fail-closed validation, or securing a rootless container platform, feel free to reach out via Services.