Cloudjur

Ansible-and-Host-Configuration

3 min read

Ansible and Host Configuration

Relevant source files

This section details the Ansible-based automation used to configure the underlying Proxmox hypervisor and its storage subsystems. While the Kubernetes cluster is managed via Talos Linux and Flux CD, the physical host configuration—including ZFS pool management, file sharing (NFS/SMB), and host-level observability—is handled through Ansible playbooks.

Storage and Fileserver Configuration

The primary storage engine for the infrastructure is a ZFS pool named /tank. Ansible is used to ensure the directory structure, permissions, and network sharing protocols (NFS and SMB) are consistently applied across the Proxmox host.

User and Permission Model

The configuration standardizes on a specific service user and group for all network-attached storage (NAS) operations:

The playbook ensures this user exists with a password hashed via sha512 for local access (e.g., via Cockpit) infrastructure/ansible/playbooks/fileserver.yaml96-103

NFS and SMB Implementation

The host serves files to the Kubernetes cluster (via NFS) and local network devices (via SMB).

  1. NFS Exports: Configured via the exports.j2 template infrastructure/ansible/playbooks/fileserver.yaml177 It defines access for specific subnets, such as the Kubernetes node network (10.10.30.0/24), using all_squash to map all requests to the nas user (UID 2000) infrastructure/ansible/playbooks/fileserver.yaml13-23
  2. SMB Shares: Managed via Samba, providing access to shares like Workstation, Apps, and Storageinfrastructure/ansible/playbooks/fileserver.yaml36-85 The Workstation share is specifically optimized for macOS Time Machine using vfs_objects: fruit and fruit:time_machine: "yes"infrastructure/ansible/playbooks/fileserver.yaml42-43

Directory Hierarchy

The playbook idempotently creates the entire directory structure required by various cluster applications under /tank/Apps and /tank/Storageinfrastructure/ansible/playbooks/fileserver.yaml130-174

Fileserver Configuration Flow

[Flowchart Diagram]

Sources:infrastructure/ansible/playbooks/fileserver.yaml1-216

Host Observability

To monitor the health of the Proxmox host and its ZFS pools, Ansible deploys node_exporter and other monitoring agents directly to the host.

Node Exporter Deployment

The monitoring.yaml playbook handles the installation and lifecycle of the Prometheus node_exporter. This allows the central kube-prometheus-stack within the Kubernetes cluster to scrape host-level metrics (CPU, Memory, Disk I/O) by targeting the Proxmox host IP.

ZFS and SMART Monitoring

Specific exporters are configured to provide visibility into the storage layer:

  • ZFS Health: Metrics regarding pool status, scrub completion, and dataset usage.
  • SMART Metrics: Deployed to monitor physical drive longevity and hardware errors.

These metrics are integrated into the cluster-wide Prometheus instance via ScrapeConfig resources kubernetes/apps/observability/kube-prometheus-stack/app/scrapeconfig.yaml

Observability Data Flow

[Flowchart Diagram]

Sources:infrastructure/ansible/playbooks/monitoring.yamlkubernetes/apps/observability/kube-prometheus-stack/app/scrapeconfig.yaml

Technical Implementation Details

Secrets Management in Ansible

Ansible leverages the community.sops collection to decrypt secrets at runtime infrastructure/ansible/requirements.yaml9-10

Ansible Environment

The Ansible environment is managed with specific versions of collections and Python dependencies to ensure reproducibility:

DependencyVersionPurpose
ansible13.6.0Core engine infrastructure/ansible/requirements.txt1
ansible.posix2.1.0ACL and Mount management infrastructure/ansible/requirements.yaml3-4
community.general12.6.0Proxmox and storage modules infrastructure/ansible/requirements.yaml7-8
netaddr1.3.0IP address manipulation infrastructure/ansible/requirements.txt6

Sources:infrastructure/ansible/requirements.txt1-9infrastructure/ansible/requirements.yaml1-19infrastructure/ansible/playbooks/fileserver.yaml34-35

Fileserver Directory Structure

The following table describes the primary ZFS datasets managed by Ansible:

PathPurposeNFS ClientsSMB Access
/tank/AppsApplication data (Configs, DBs)K8s Nodes (RW)Read-Only infrastructure/ansible/playbooks/fileserver.yaml12-70
/tank/StorageBulk media and documentsK8s Nodes (RW)Read-Write infrastructure/ansible/playbooks/fileserver.yaml16-75
/tank/BackupBackups and Time MachineK8s Nodes (RW)Read-Only (except TM) infrastructure/ansible/playbooks/fileserver.yaml20-80
/tank/SharedTemporary file exchangePublic (RW)Guest Access (RW) infrastructure/ansible/playbooks/fileserver.yaml25-85

Sources:infrastructure/ansible/playbooks/fileserver.yaml10-85

Graph View

Backlinks