Cloudjur

Garage-S3-Object-Storage

3 min read

Garage S3 Object Storage

Relevant source files

Garage is a lightweight, distributed S3-compatible storage service designed for self-hosting. In this repository, Garage provides the primary object storage backend for internal cluster services (like database backups and WAL archiving) and serves static web content.

Deployment Architecture

The infrastructure utilizes two distinct Garage instances:

  1. Primary Garage (garage): Handles high-performance and critical data for services like postgresql (CloudNativePG) and dragonflyinfrastructure/terraform/s3/garage/main.tf7-9
  2. Static Garage (staticgarage): Dedicated to serving static websites, such as the cluster documentation wiki and schemas infrastructure/terraform/s3/staticgarage/main.tf1-6

Both instances are deployed using the bjw-s/app-template Helm chart kubernetes/apps/storage/garage/app/helmrelease.yaml10 and utilize LoadBalancer services with fixed IP addresses assigned via Cilium L2 announcements kubernetes/apps/storage/garage/app/helmrelease.yaml43-45

Component Relationship Diagram

This diagram illustrates how the Kubernetes resources, configuration files, and Terraform modules interact to provide the S3 service.

[Flowchart Diagram]

Sources:kubernetes/apps/storage/garage/app/helmrelease.yaml108-129kubernetes/apps/storage/garage/app/externalsecret.yaml1-40infrastructure/terraform/s3/garage/modules/garage/main.tf10-20

Configuration (garage.toml)

Garage is configured via a garage.toml file, which is managed as a template inside an ExternalSecret. This allows the injection of sensitive tokens (RPC secrets, Admin tokens) directly into the configuration file at runtime.

Key Configuration Parameters

Sources:kubernetes/apps/storage/garage/app/resources/garage.toml1-27kubernetes/apps/storage/staticgarage/app/externalsecret.yaml8-15

Terraform Provisioning

Buckets and access keys are managed as Infrastructure-as-Code using the schwitzd/garage Terraform provider infrastructure/terraform/s3/garage/providers.tf20-23

Resource Management Flow

The modules/garage module encapsulates the creation of a bucket, its primary access key, and the association between them.

[Flowchart Diagram]

Managed Buckets

Sources:infrastructure/terraform/s3/garage/modules/garage/main.tf10-35infrastructure/terraform/s3/garage/main.tf1-23infrastructure/terraform/s3/staticgarage/main.tf1-27

Service Consumption

Services interact with Garage through the internal Envoy Gateway ingress or direct LoadBalancer IPs.

Data Flow: CloudNativePG WAL Archiving

CloudNativePG (CNPG) uses the postgresql bucket for Write-Ahead Log (WAL) archiving and cluster backups.

  1. Request: CNPG instance sends S3 API calls to s3.cloudjur.comkubernetes/apps/storage/garage/app/helmrelease.yaml59
  2. Ingress: envoy-internal routes the traffic to the garage service on port 3900kubernetes/apps/storage/garage/app/helmrelease.yaml61-66
  3. Storage: Garage writes the data to the persistent NFS volume mounted at /datakubernetes/apps/storage/garage/app/helmrelease.yaml121-123

Static Web Hosting

The staticgarage instance is configured to serve web content directly from buckets.

Sources:kubernetes/apps/storage/garage/app/helmrelease.yaml53-107kubernetes/apps/storage/staticgarage/app/helmrelease.yaml53-101infrastructure/terraform/s3/garage/modules/garage/main.tf10-16

Persistence and Storage Tiering

Garage data is not stored on local NVMe but is offloaded to the bulk storage pool (/tank) via NFS to ensure high capacity for object storage.

Mount PathSource (NFS)Purpose
/datasmb.cloudjur.com:/tank/Apps/Garage/dataBlock data storage kubernetes/apps/storage/garage/app/helmrelease.yaml121-123
/metasmb.cloudjur.com:/tank/Apps/Garage/metaMetadata and LMDB database kubernetes/apps/storage/garage/app/helmrelease.yaml127-129

Sources:kubernetes/apps/storage/garage/app/helmrelease.yaml117-130kubernetes/apps/storage/staticgarage/app/helmrelease.yaml110-123

Graph View

Backlinks