Cloudjur

Flux-Operator-and-Instance

3 min read

Flux Operator and Instance

Relevant source files

The Flux ecosystem in this cluster is managed via the flux-operator, which provides a declarative way to manage Flux CD instances themselves. This setup moves away from traditional flux bootstrap methods in favor of a Kubernetes-native lifecycle where the FluxInstance CRD defines the desired state of the GitOps controllers.

Flux Operator Deployment

The flux-operator is deployed as a HelmRelease from an OCIRepository source kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml3-10 It is responsible for managing the lifecycle of the Flux controllers (Source, Kustomize, Helm, and Notification).

The operator includes a web interface accessible via flux-operator.cloudjur.com, which is exposed through an HTTPRoute connected to the envoy-internal gateway kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml29-41 Authentication for this dashboard is configured as Anonymous with flux-admin group permissions for internal cluster access kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml23-28

Sources:

Flux Instance and Performance Tuning

The flux-instance``HelmRelease defines the actual Flux CD installation. It specifies the distribution version (currently 2.8.7) and the components to be installed kubernetes/apps/flux-system/flux-instance/app/helmrelease.yaml21-31

To handle the scale of the home-ops repository, significant performance tuning is applied via Kustomize patches within the FluxInstance resource.

Controller Optimization Table

ControllerTuning ParameterValuePurpose
Global--concurrent10Increases simultaneous reconciliation tasks kubernetes/apps/flux-system/flux-instance/app/helmrelease.yaml47
Global--requeue-dependency5sFaster retries when dependencies are not met kubernetes/apps/flux-system/flux-instance/app/helmrelease.yaml50
GlobalMemory Limit6GiPrevents OOM kills during large builds kubernetes/apps/flux-system/flux-instance/app/helmrelease.yaml68
Kustomize--concurrent20Specialized high concurrency for Kustomize builds kubernetes/apps/flux-system/flux-instance/app/helmrelease.yaml77
KustomizeemptyDir (Memory)/tmpEnables in-memory builds to reduce IO wait kubernetes/apps/flux-system/flux-instance/app/helmrelease.yaml81-83
Source--helm-cache-max-size10Caches Helm repositories in memory kubernetes/apps/flux-system/flux-instance/app/helmrelease.yaml92
HelmOOMWatchEnabledActive memory monitoring to prevent kernel OOM kills kubernetes/apps/flux-system/flux-instance/app/helmrelease.yaml107
Helmmemory-threshold95%Threshold for OOMWatch to trigger protective actions kubernetes/apps/flux-system/flux-instance/app/helmrelease.yaml110

Sources:

Secret Lifecycle and Bitwarden Integration

Secrets required by Flux and the Tofu Controller are managed via ExternalSecrets that interface with a Bitwarden-backed ClusterSecretStore.

Secret Data Flow

The following diagram illustrates how a secret (e.g., the GitHub Webhook Token) moves from Bitwarden into the Flux environment.

Flux Secret Resolution Diagram

For Terraform operations, a ClusterExternalSecret is used to propagate the terraform-backend-secret (containing Cloudflare R2 credentials) to any namespace labeled with terraform-shared-credentials: "true"kubernetes/apps/flux-system/tofu-controller/app/externalsecret.yaml1-11

Sources:

Webhooks and Observability

GitHub Webhook Receiver

To ensure near-instantaneous reconciliation upon a git push, a Receiver resource is configured kubernetes/apps/flux-instance/app/receiver.yaml It uses the github-webhook-token-secret to validate incoming requests from GitHub.

Health and Alerts

The Flux instance is monitored via PrometheusRuleskubernetes/apps/flux-instance/app/prometheusrule.yaml Key metrics include:

  • Reconciliation Failures: Alerts if Kustomization or HelmRelease resources stay in a non-ready state.
  • Controller Health: Monitoring the memory usage relative to the tuned limits and OOMWatch events.

Schema Management

The kubernetes-schemas application serves as a central repository for CRD schemas used by the cluster kubernetes/apps/flux-system/kubernetes-schemas/app/helmrelease.yaml5-13 It includes a Grafana dashboard for monitoring the schema server’s performance kubernetes/apps/flux-system/kubernetes-schemas/app/helmrelease.yaml20-26

Sources:

Code Entity Mapping

The following diagram maps the high-level GitOps concepts to the specific Kubernetes resources and files that implement them in the flux-system namespace.

Flux Architecture Mapping

Sources:

Graph View

Backlinks