Kube-System and Cluster Infrastructure
Relevant source files
- bootstrap/helmfile.d/01-apps.yaml
- bootstrap/helmfile/apps.yaml
- kubernetes/apps/actions-runner-system/kustomization.yaml
- kubernetes/apps/actions-runner-system/namespace.yaml
- kubernetes/apps/cert-manager/kustomization.yaml
- kubernetes/apps/external-secrets/external-secrets/app/ocirepository.yaml
- kubernetes/apps/kube-system/kustomization.yaml
- kubernetes/apps/kube-system/spegel/app/helmrelease.yaml
- kubernetes/apps/kube-system/spegel/app/kustomization.yaml
- kubernetes/apps/kube-system/spegel/ks.yaml
- kubernetes/apps/media/opencloud/app/ocirepository.yaml
- kubernetes/apps/network/kustomization.yaml
- kubernetes/components/common/repos/app-template/ocirepository.yaml
The kube-system namespace serves as the operational bedrock of the home-ops cluster. It contains the essential daemonsets, controllers, and plugins that interface with the underlying Talos Linux nodes and provide core services like DNS, networking, and hardware acceleration to all other namespaces.
System Architecture Overview
The infrastructure layer is managed through a layered approach:
- Bootstrap: Initial CRDs and core controllers (Cilium, CoreDNS, Spegel) are deployed via
helmfileto establish the network and image registry connectivity bootstrap/helmfile.d/01-apps.yaml1-142 - Kustomization: The
kube-systemapps are structured as Kustomize resources, inheriting common components like SOPS secret decryption kubernetes/apps/kube-system/kustomization.yaml1-17 - Hardware & Features: Node-level capabilities are exposed to the cluster via specialized plugins for Intel GPUs and general hardware labeling.
Component Interaction Map
This diagram illustrates how core system components interact to provide the foundation for application workloads.
“Kube-System Core Infrastructure”
Sources:kubernetes/apps/kube-system/kustomization.yaml1-17kubernetes/apps/kube-system/spegel/app/helmrelease.yaml26-28bootstrap/helmfile.d/01-apps.yaml44-57
Node Features and Hardware Plugins
The cluster utilizes specialized plugins to bridge the gap between physical hardware and Kubernetes orchestration. This ensures that workloads requiring specific capabilities (like hardware-accelerated transcoding) are scheduled on compatible nodes.
- Node Feature Discovery (NFD): Scans nodes for hardware features (CPU instructions, USB devices, PCI cards) and applies labels used for pod affinity kubernetes/apps/kube-system/kustomization.yaml15
- Intel Device Plugin: Manages access to Intel iGPUs, enabling QuickSync hardware transcoding for services like Plex and Jellyfin kubernetes/apps/kube-system/kustomization.yaml13
- Spegel: A stateless peer-to-peer container image advertisement and proxy mechanism. It allows nodes to share container images directly from their local
containerdcaches, reducing external bandwidth and speeding up pod startup times kubernetes/apps/kube-system/spegel/app/helmrelease.yaml21-28 - Reloader: Watches for changes in
ConfigMapsandSecretsand performs rolling upgrades on associatedDeploymentsorStatefulSetskubernetes/apps/kube-system/kustomization.yaml16
For details, see Node Features and Hardware Plugins.
Sources:kubernetes/apps/kube-system/kustomization.yaml12-17kubernetes/apps/kube-system/spegel/app/helmrelease.yaml1-29
System Upgrades and Bootstrap
The lifecycle of the cluster—from initial provisioning to continuous updates—is handled by a dedicated set of tools and controllers.
Bootstrap Process
The cluster bootstrap follows a strict dependency chain defined in helmfile. This ensures that networking (Cilium) and DNS (CoreDNS) are functional before higher-level operators like Flux or External Secrets are introduced bootstrap/helmfile.d/01-apps.yaml44-124
“Bootstrap Dependency Chain”
Sources:bootstrap/helmfile.d/01-apps.yaml15-124
Automated Upgrades
System maintenance is managed via the system-upgrade namespace (governed by TUPPR), which coordinates rolling upgrades of Talos Linux and Kubernetes versions across the node pool. This process utilizes a versions.env file to maintain a single source of truth for the desired cluster state.
For details, see System Upgrades and Bootstrap.
Sources:bootstrap/helmfile.d/01-apps.yaml1-142bootstrap/helmfile/apps.yaml1-139
Core Networking and Storage Services
While kube-system handles low-level infrastructure, it also hosts the controllers for cluster-wide networking and shared storage access.
| Component | Role | File Reference |
|---|---|---|
| Cilium | eBPF-based CNI for networking and security policies. | kubernetes/apps/kube-system/cilium/ks.yaml |
| CoreDNS | Cluster-internal DNS and service discovery. | kubernetes/apps/kube-system/coredns/ks.yaml |
| CSI Driver NFS | Allows pods to mount shared storage from the Proxmox/ZFS host. | kubernetes/apps/kube-system/csi-driver-nfs/ks.yaml |
| Metrics Server | Provides resource usage data for HPA and kubectl top. | kubernetes/apps/kube-system/metrics-server/ks.yaml |
| Descheduler | Evicts pods to balance node resource utilization. | kubernetes/apps/kube-system/descheduler/ks.yaml |
Sources:kubernetes/apps/kube-system/kustomization.yaml9-14bootstrap/helmfile.d/01-apps.yaml15-50