Secrets Management

Relevant source files

The home-ops repository utilizes a multi-layered secrets management pipeline to ensure sensitive data is encrypted at rest in Git while being dynamically injected into the Kubernetes cluster. This architecture combines SOPS with Age for static encryption and the External Secrets Operator (ESO) with a Bitwarden SDK Server for dynamic secret synchronization.

Secrets Pipeline Overview

The lifecycle of a secret in this cluster follows a distinct path from creation to consumption by an application.

Encryption at Rest: High-level bootstrap secrets are encrypted using sops and age before being committed to the repository.
External Storage: Most application secrets (API tokens, database passwords) are stored in Bitwarden.
Synchronization: The external-secrets operator polls the Bitwarden SDK Server.
Injection: ExternalSecret resources define how to map Bitwarden items to Kubernetes Secret objects.
Consumption: Applications reference these standard Kubernetes Secret objects via envFrom or secretRef.

Data Flow Diagram: Secret Synchronization

The following diagram illustrates the flow from the Bitwarden vault to a running Pod.

[Flowchart Diagram]

Sources:kubernetes/apps/external-secrets/bitwarden-sdk-server/app/clustersecretstore.yaml1-10 kubernetes/apps/flux-system/flux-instance/app/externalsecret.yaml1-21 kubernetes/apps/default/paperless/app/helmrelease.yaml59-61

Git-Stored Secrets (SOPS & Age)

For secrets required during the initial bootstrap of the cluster (such as the Bitwarden Access Token itself or Cloudflare API keys used by external-dns), SOPS (Secrets Operations) is used.

Encryption Provider: age is the primary encryption backend.
Automation: The community.sops Ansible collection is included in the infrastructure requirements to handle encrypted files during host provisioning infrastructure/ansible/requirements.yaml9-10
Flux Integration: Flux’s kustomize-controller is configured to decrypt SOPS-encrypted files in-cluster using a private Age key stored in a Kubernetes secret.

Sources:infrastructure/ansible/requirements.yaml9-10 kubernetes/apps/flux-system/flux-instance/app/helmrelease.yaml27-30

External Secrets Operator (ESO)

The external-secrets operator is the core component for dynamic secret management. It is organized within the external-secrets namespace kubernetes/apps/external-secrets/kustomization.yaml1-12

Bitwarden SDK Server Backend

Instead of the standard Bitwarden CLI, this implementation uses a Bitwarden SDK Server. This server provides a more stable and performant API interface for the operator to communicate with the Bitwarden vault.

ClusterSecretStore: A cluster-scoped resource named bitwarden-fields acts as the bridge between the cluster and the SDK server kubernetes/apps/external-secrets/bitwarden-sdk-server/app/clustersecretstore.yaml1-10
Authentication: The SDK server authenticates to Bitwarden using an Access Token (managed via a SOPS-encrypted secret during bootstrap).

Secure Communication (cert-manager)

Communication between the External Secrets Operator and the Bitwarden SDK Server is secured via TLS.

Issuer: A local Issuer is defined for the SDK server kubernetes/apps/external-secrets/bitwarden-sdk-server/app/issuer.yaml1-10
Certificate: A Certificate resource ensures the SDK server has a valid TLS certificate, managed by cert-managerkubernetes/apps/external-secrets/bitwarden-sdk-server/app/certificate.yaml1-10

Sources:kubernetes/apps/external-secrets/kustomization.yaml4-11 kubernetes/apps/cert-manager/cert-manager/app/helmrelease.yaml1-24

Mapping Bitwarden to Kubernetes

The ExternalSecret resource maps specific fields from a Bitwarden item (identified by its UUID) to keys within a Kubernetes Secret.

Implementation Example: GitHub Webhook Token

The flux-instance uses an ExternalSecret to retrieve its GitHub webhook token from Bitwarden.

Code Entity	Value / Reference
Resource Name	`github-webhook-token`
Store Reference	`bitwarden-fields` (ClusterSecretStore)
Bitwarden UUID	`b67e4583-a622-4cf2-a172-58ee67c703da`
Property	`github-webhook-token`
Target Secret	`github-webhook-token-secret`

# Simplified representation of mapping
spec:
  data:
    - secretKey: FLUX_GITHUB_WEBHOOK_TOKEN
      remoteRef:
        key: b67e4583-a622-4cf2-a172-58ee67c703da
        property: github-webhook-token

Sources:kubernetes/apps/flux-system/flux-instance/app/externalsecret.yaml1-21

Application Consumption

Once the ExternalSecret is reconciled into a standard Kubernetes Secret, applications consume it using standard envFrom or secretRef patterns. For example, the paperless application loads its environment variables from paperless-secretkubernetes/apps/default/paperless/app/helmrelease.yaml59-61

To ensure pods restart when secrets are updated in Bitwarden, the cluster utilizes reloader. Applications are annotated with reloader.stakater.com/auto: "true" to trigger a rolling update upon secret change kubernetes/apps/default/paperless/app/helmrelease.yaml15-16 kubernetes/apps/network/external-dns/app/helmrelease.yaml45-46

Sources:kubernetes/apps/default/paperless/app/helmrelease.yaml59-61 kubernetes/apps/kube-system/reloader/app/helmrelease.yaml1-41

Entity Relationship: Secret Management Codebase

This diagram bridges the natural language concepts to the specific code entities and file locations.

[Class Diagram]

Sources:kubernetes/apps/flux-system/flux-instance/app/externalsecret.yaml1-21 kubernetes/apps/external-secrets/bitwarden-sdk-server/app/clustersecretstore.yaml1-10 kubernetes/apps/kube-system/reloader/app/helmrelease.yaml32-34

Cloudjur

Explorer

Secrets-Management