VolSync and Volume Backup

Relevant source files

• .taskfiles/VolSync/Taskfile.yaml
• .taskfiles/VolSync/resources/list-snapshots.yaml.j2
• .taskfiles/VolSync/resources/replicationdestination-r2.yaml.j2
• .taskfiles/VolSync/resources/replicationdestination.yaml.j2
• .taskfiles/VolSync/resources/unlock-r2.yaml.j2
• .taskfiles/VolSync/resources/unlock.yaml.j2
• docs/ai-context/SCHEDULES.md
• kubernetes/apps/ai/toolhive/config/virtualmcpserver.yaml
• kubernetes/apps/default/atuin/app/helmrelease.yaml
• kubernetes/apps/default/homepage/app/helmrelease.yaml
• kubernetes/apps/default/n8n/app/helmrelease.yaml
• kubernetes/apps/downloads/profilarr/app/helmrelease.yaml
• kubernetes/apps/downloads/webhook/app/helmrelease.yaml
• kubernetes/apps/flux-system/flux-instance/app/httproute.yaml
• kubernetes/apps/media/maintainerr/app/helmrelease.yaml
• kubernetes/apps/media/watchstate/app/helmrelease.yaml
• kubernetes/apps/observability/kromgo/app/helmrelease.yaml
• kubernetes/apps/security/authentik/app/httproute.yaml
• kubernetes/apps/storage/democratic-csi/app/helmrelease.yaml
• kubernetes/apps/storage/garage/webui/helmrelease.yaml
• kubernetes/apps/storage/staticgarage/webui/helmrelease.yaml
• kubernetes/apps/volsync-system/kopia/app/helmrelease.yaml
• kubernetes/apps/volsync-system/kustomization.yaml
• kubernetes/apps/volsync-system/snapshot-controller/app/helmrelease.yaml
• kubernetes/apps/volsync-system/snapshot-controller/ks.yaml
• kubernetes/apps/volsync-system/volsync/app/helmrelease.yaml
• kubernetes/apps/volsync-system/volsync/app/kustomization.yaml
• kubernetes/apps/volsync-system/volsync/app/mutatingadmissionpolicy.yaml
• kubernetes/apps/volsync-system/volsync/app/prometheusrule.yaml
• kubernetes/apps/volsync-system/volsync/ks.yaml
• kubernetes/apps/volsync-system/volsync/maintenance/externalsecret.yaml
• kubernetes/apps/volsync-system/volsync/maintenance/kopiamaintenance.yaml
• kubernetes/apps/volsync-system/volsync/maintenance/kustomization.yaml
• kubernetes/apps/volsync-system/volsync/maintenance/tbd.yaml
• kubernetes/components/volsync-no-r2/kustomization.yaml
• kubernetes/components/volsync-only-r2/kustomization.yaml
• kubernetes/components/volsync-only-r2/replicationdestination.yaml
• kubernetes/components/volsync/kopia.yaml
• kubernetes/components/volsync/kustomization.yaml
• kubernetes/components/volsync/pvc.yaml
• kubernetes/components/volsync/r2.yaml
• scripts/httproute-csv.sh

The volume backup architecture in this repository provides a multi-tier data protection strategy. It leverages the VolSync operator to manage asynchronous replication of persistent volumes, utilizing Kopia for incremental, deduplicated backups to local NFS storage and off-site replication to Cloudflare R2. The system is built upon a foundation of local-path storage with snapshot capabilities provided by democratic-csi and the snapshot-controller.

Storage Foundation and Snapshots

Data persistence for most applications is handled by the local-hostpath StorageClass, which provides high-performance local storage while maintaining the ability to take volume snapshots.

• democratic-csi: Implements the CSI driver org.democratic-csi.local-hostpathkubernetes/apps/storage/democratic-csi/app/helmrelease.yaml35 It manages local directories on the Talos nodes at /var/lib/csi-local-hostpathkubernetes/apps/storage/democratic-csi/app/helmrelease.yaml77
• StorageClass: The local-hostpath class is configured as the cluster default kubernetes/apps/storage/democratic-csi/app/helmrelease.yaml40-41
• Snapshot Controller: The snapshot-controller is deployed to manage VolumeSnapshot resources kubernetes/apps/volsync-system/snapshot-controller/app/helmrelease.yaml19-20 It enables the VolumeSnapshotClass used by VolSync to create point-in-time copies of data before backup kubernetes/apps/storage/democratic-csi/app/helmrelease.yaml45-46

Data Flow: Application to Snapshot

Sources: kubernetes/apps/storage/democratic-csi/app/helmrelease.yaml34-52kubernetes/apps/volsync-system/snapshot-controller/app/helmrelease.yaml1-53

VolSync Implementation

The VolSync operator is the core orchestration engine for backups. It is deployed using a fork (ghcr.io/perfectra1n/volsync) that supports specific enhancements for the home-ops environment kubernetes/apps/volsync-system/volsync/app/helmrelease.yaml26-29

Kustomize Components

To simplify the addition of backups to applications, a set of reusable Kustomize components is provided in kubernetes/components/volsync/kubernetes/components/volsync/kustomization.yaml1-10

1. kopia.yaml: Configures the ReplicationSource for local backups.

• Schedule: Backups are triggered twice daily at 04:00 and 16:00 kubernetes/components/volsync/kopia.yaml72
• Method: Uses the Snapshot copy method to ensure data consistency kubernetes/components/volsync/kopia.yaml81
• Destination: Backs up to an NFS mount at smb.cloudjur.com:/tank/Backup/Volsynckubernetes/components/volsync/kopia.yaml85-87
• Retention: Keeps 24 hourly and 7 daily snapshots kubernetes/components/volsync/kopia.yaml94-96

2. r2.yaml: (Implicitly referenced) Handles off-site replication to Cloudflare R2 object storage for disaster recovery.
3. pvc.yaml: Manages the underlying persistent volume claims required by the VolSync movers.

Secret Management

VolSync requires repository passwords to encrypt Kopia data. These are managed via ExternalSecrets that pull kopia_password from Bitwarden kubernetes/components/volsync/kopia.yaml2-24

Sources: kubernetes/components/volsync/kopia.yaml1-99kubernetes/apps/volsync-system/volsync/app/helmrelease.yaml15-43

Backup Architecture

The backup process bridges the local Kubernetes environment with external storage targets.

Code-to-System Mapping

System Component	Code Entity / CRD	Configuration File
Backup Source	ReplicationSource	kubernetes/components/volsync/kopia.yaml65-68
Restore Target	ReplicationDestination	kubernetes/components/volsync/kopia.yaml28-30
Storage Target	moverVolumes (NFS)	kubernetes/components/volsync/kopia.yaml82-87
Backup Engine	kopia	kubernetes/apps/volsync-system/volsync/app/helmrelease.yaml30
Secret Template	ExternalSecret	kubernetes/components/volsync/kopia.yaml3-24

Multi-Tier Data Flow

Sources: kubernetes/components/volsync/kopia.yaml26-99kubernetes/apps/volsync-system/kopia/app/helmrelease.yaml88-128

Restore Operations

Restoration is handled through the ReplicationDestination resource, which defines how to pull data from the Kopia repository back into a PVC.

• Bootstrap Pattern: A ReplicationDestination named ${APP}-bootstrap is defined with a manual trigger restore-oncekubernetes/components/volsync/kopia.yaml30-35
• Storage Mapping: It restores data to a PVC using the local-hostpath StorageClass kubernetes/components/volsync/kopia.yaml61

Taskfile Commands

Operational tasks for VolSync are automated via Taskfiles located in .taskfiles/VolSync/. These commands allow for manual intervention and disaster recovery:

• List Snapshots: Uses .taskfiles/VolSync/resources/list-snapshots.yaml.j2 to query available recovery points in the Kopia repository.
• Unlock Repository: Uses .taskfiles/VolSync/resources/unlock.yaml.j2 to clear stale locks from the backup repository.
• Manual Restore: Deploys the ReplicationDestination CRD using templates like .taskfiles/VolSync/resources/replicationdestination.yaml.j2 to trigger a data pull.

System Maintenance and Security

• MutatingAdmissionPolicy: A policy is applied in the volsync-system namespace to ensure backup pods (movers) adhere to security standards kubernetes/apps/volsync-system/volsync/app/mutatingadmissionpolicy.yaml
• Kopia Maintenance: Scheduled maintenance jobs (e.g., KopiaMaintenance) are used to perform garbage collection and data integrity checks on the Kopia repositories kubernetes/apps/volsync-system/volsync/maintenance/kopiamaintenance.yaml
• Pod Security: The VolSync operator and its movers run with non-root security contexts (UID/GID 2000) kubernetes/apps/volsync-system/volsync/app/helmrelease.yaml39-42kubernetes/components/volsync/kopia.yaml88-91

Sources: kubernetes/apps/volsync-system/volsync/app/helmrelease.yaml39-42kubernetes/components/volsync/kopia.yaml54-57kubernetes/apps/volsync-system/volsync/maintenance/kopiamaintenance.yaml