szinn / k8s-homelab

_... managed with Flux, Renovate and GitHub_ 🤖                               --- Overview This is my mono repo for my home infrastructure. It's based loosely on the template at onedr0p/cluster-template as well as many of the exemplar repos, searchable via https://kubesearch.dev/. It follows the concept of Infrastructure as Code and by using tools such Flux, Renovate, go-task and shell scripts, creates a reproducible, mostly self-managing implementation. My original implementation was running on the Ryzen using custom shell scripts and 35+ docker containers managed by hand. Any upgrades, system resets, etc, all had to be manually resolved. It mostly ran just fine. Applying the principle "If it ain't broke, it isn't complicated enough" led me to add machines, memory, functionality to achieve a much more automated, self-managing cluster. Plus I have learned a lot! At the bottom of this page, is the bringup process that I follow for this cluster. I recommend reading top-to-bottom to understand the cluster structure which will help understand what's needed for the bringup. --- Hardware My HomeLab consists of a bunch of machines and Ubiquity networking. | Device | Count | OS Disk Size | Data Disk Size | RAM | Operating System | | -------------------------------------------------- | ----- | ------------ | --------------------- | ---- | ------------------------- | | Ryzen 3900 12c24t NAS server | 1 | 1TB | 1TB NVME, 6x16Tb SATA | 64GB | TrueNAS Scale - Ragnar | | Raspberry Pi 4B | 1 | | | | Artemis - AdGuardHome DNS | | Raspberry Pi 5 | 1 | | | | Apollo - AdGuardHome DNS | | TESmart 16-port HDMI Switch | 1 | | | | | | PiKVM | 1 | | | | | | Intel NUC11PAHi7 (worker nodes) | 3 | 500GB SSD | 1TB NVMe | 64GB | Talos | | Beelink MiniPC, Celeron J4125 (controlplane nodes) | 3 | 256GB SSD | | 8GB | Talos | | Synology 1019+ (NFS server) | 1 | | 5x12TB SATA | | | | UniFi UDM SE | 1 | | | | | | USW-Pro-24-PoE | 1 | | | | | | USW-Aggregation | 1 | | | | | | USW-Enterprise-8-PoE | 2 | | | | | | USW-Flex XG | 1 | | | | Office Hub | | USW-Flex | 1 | | | | Office Hub | | U7 Pro Wall | 2 | | | | Access points | | U6 Mesh | 1 | | | | Offic access point | | USP-PDU Pro | 2 | | | | | | Intel NUC11TNHi7 | 1 | 1Tb | | 64GB | Proxmox | | Intel NUC13 Pro | 1 | 1Tb | | 32GB | Hera Fedora | | UVC G4 Doorbell | 1 | | | | Front Door Camera | | UVC G4 Pro | 1 | | | | Basement Camera | The Proxmox Intel NUC runs a 3-node Talos staging cluster where I can try out various patterns before deploying in the main cluster. The Intel NUC13 (Hera) is a spare NUC that I'm currently using as a Fedora platform with a graphical UI. Artemis runs services that need to be outside the cluster: • DNS (AdGuard Home) • gatus to track machine and non-cluster services Apollo serves as a DNS server as well. Both Artemis and Apollo are served up as DNS servers to the appropriate VLANs. Kubernetes The cluster is based on Talos with 3 control-plane nodes running on the Beelink MiniPCs and 3 worker nodes running on the Intel NUCs. Core Components • kubernetes/ingress-nginx: Manages reverse-proxy access to Kubernetes services. • rook/rook: Distributed block storage for persistent storage. • jetstack/cert-manager: Creates SSL certificates for services in my Kubernetes cluster. • kubernetes-sigs/external-dns: Automatically manages DNS records from my cluster in a cloud DNS provider. GitOps Flux watches my main cluster folder (see Directories below) and makes the changes to my cluster based on the YAML manifests. Renovate watches my **entire** repository looking for dependency updates, when they are found a PR is automatically created. When PRs are merged Flux applies the changes to my cluster. Charts and images are tagged in the various YAML files to enable Renovate to watch and update them as needed. Network Configuration See diagram of the backbone. The external network is connected to the UDM SE with a Wireguard port being the only exposed access. This allows me to connect into the network when I'm traveling. Inbound services are managed with cloudflared. The main cluster and IPs are on the 10.11.x.x subnet on VLAN HOMELAB. The stagint cluster and IPs are on the 10.12.x.x subnet on VLAN STAGING. External machines (Synology, etc) are SERVERS VLAN subnet. IoT devices are on an isolated IoT VLAN. They cannot reach the other VLANs directly but will answer when spoken to. DNS is managed by CoreDNS in the cluster which then forwards unresolved requests to DNS running on the Titan server. Titan will forward accepted addresses onto the UDM-SE for resolution. The external DNS is managed via Cloudflare. External names are managed by external-dns on the cluster and, since my home IP can be changed at any time, DDNS is maintained by the oznu/cloudflare-ddns docker image. Certificates are managed through CloudFlare as well using cert-manager and the DNS01 challenge protocol. Repository Structure The repository supports multiple clusters -- in particular, I have a "main" cluster which runs on the above hardware. I previously had a staging cluster that could use the same source, but have since moved that to a separate repo. Adding something new to the cluster usually requires a lot of trial and error initially. When I am trying something out, I will work in a staging environment as much as possible and then move to the main cluster. If additional iterations are required, I will usually try and do amended commits rather than a chain of commits with comments such as "Trying again" or "Maybe this will work", etc. The repository directories are: • **.github**: GitHub support files and renovate configuration. • **.taskfiles**: Auxiliary files used for the task command-line tool. • **kubernetes**: The clusters themselves. • **main**: The main cluster • **apps**: The applications to load. • **bootstrap**: The initial code loaded o…