Envoy Gateway and Ingress

Relevant source files

• .taskfiles/Flux/Taskfile.yaml
• .taskfiles/bootstrap/Taskfile.yaml
• kubernetes/apps/cert-manager/cert-manager/app/grafanadashboard.yaml
• kubernetes/apps/database/dragonfly-operator/app/grafanadashboard.yaml
• kubernetes/apps/games/minecraft/app/grafanadashboard.yaml
• kubernetes/apps/home-automation/frigate/app/grafanadashboard.yaml
• kubernetes/apps/media/syncthing/app/grafanadashboard.yaml
• kubernetes/apps/network/cloudflare-tunnel/app/grafanadashboard.yaml
• kubernetes/apps/network/envoy-gateway/app/envoy.yaml
• kubernetes/apps/network/envoy-gateway/app/grafanadashboard.yaml
• kubernetes/apps/network/envoy-gateway/app/helmrelease.yaml
• kubernetes/apps/network/envoy-gateway/app/kustomization.yaml
• kubernetes/apps/network/envoy-gateway/app/scaledobject.yaml
• kubernetes/apps/network/external-dns/app/grafanadashboard.yaml
• kubernetes/apps/observability/exporters/blackbox-exporter/app/grafanadashboard.yaml
• kubernetes/apps/observability/exporters/opnsense-exporter/app/grafanadashboard.yaml
• kubernetes/apps/storage/garage/app/grafanadashboard.yaml
• scripts/bootstrap-cluster.sh
• scripts/render-machine-config.sh

The cluster networking ingress layer is powered by Envoy Gateway, implementing the Kubernetes Gateway API. This architecture replaces traditional Ingress controllers with a more expressive, role-oriented API that decouples infrastructure configuration from application routing. The setup utilizes a dual-gateway pattern to isolate internal-only traffic from services exposed to the public internet.

Dual-Gateway Architecture

The deployment consists of two distinct Gateway instances, both managed by a single GatewayClass named envoykubernetes/apps/network/envoy-gateway/app/envoy.yaml34-43 This separation allows for distinct security postures, IP assignments, and traffic policies.

1. envoy-external: Handles traffic originating from outside the cluster. It is assigned a specific LoadBalancer IP (10.10.30.26) via Cilium L2 announcements kubernetes/apps/network/envoy-gateway/app/envoy.yaml61
2. envoy-internal: Handles traffic for internal services. It is assigned IP 10.10.30.25 kubernetes/apps/network/envoy-gateway/app/envoy.yaml109 and supports advanced features like TLS Passthrough for specific hostnames kubernetes/apps/network/envoy-gateway/app/envoy.yaml156-161

Gateway Configuration and Code Mapping

The following diagram illustrates how the Gateway API resources map to the underlying Envoy proxy infrastructure.

Gateway Infrastructure Mapping

[Flowchart Diagram]

Sources: kubernetes/apps/network/envoy-gateway/app/envoy.yaml2-109

Traffic Policies and Security

Envoy Gateway is configured via several Custom Resource Definitions (CRDs) that provide fine-grained control over connection behavior and security.

BackendTrafficPolicy

The BackendTrafficPolicy resource defines how Envoy interacts with upstream services kubernetes/apps/network/envoy-gateway/app/envoy.yaml201-225

• Compression: Supports Zstd, Brotli, and Gzip to reduce bandwidth usage kubernetes/apps/network/envoy-gateway/app/envoy.yaml206-212
• Retries: Configured for 2 retries on connection resets kubernetes/apps/network/envoy-gateway/app/envoy.yaml213-217
• Timeouts: HTTP request timeout is set to 0s (disabled) to support long-lived connections like WebSockets or SSE kubernetes/apps/network/envoy-gateway/app/envoy.yaml224

ClientTrafficPolicy

The ClientTrafficPolicy resource manages downstream client connections kubernetes/apps/network/envoy-gateway/app/envoy.yaml226-248

• TLS Security: Enforces a minimum TLS version of 1.2 and supports h2 (HTTP/2) and http/1.1 ALPN protocols kubernetes/apps/network/envoy-gateway/app/envoy.yaml243-247
• Client IP Detection: Uses x-forwarded-for headers and trusts the local cluster CIDR 10.69.0.0/16 for accurate IP attribution kubernetes/apps/network/envoy-gateway/app/envoy.yaml232-235

Lua-based Extensions

The EnvoyExtensionPolicy is used to inject custom Lua scripts into the Envoy filter chain kubernetes/apps/network/envoy-gateway/app/envoy.yaml166-199 This is specifically used to implement user-agent blocking by referencing a Lua script stored in a ConfigMapkubernetes/apps/network/envoy-gateway/app/kustomization.yaml11-17

Sources: kubernetes/apps/network/envoy-gateway/app/envoy.yaml166-248kubernetes/apps/network/envoy-gateway/app/kustomization.yaml11-17

Global Routing Rules

HTTPS Redirect

A global HTTPRoute named https-redirect is applied to both the envoy-external and envoy-internal gateways kubernetes/apps/network/envoy-gateway/app/envoy.yaml251-263 It uses a RequestRedirect filter to automatically upgrade all port 80 (HTTP) traffic to port 443 (HTTPS) kubernetes/apps/network/envoy-gateway/app/envoy.yaml268-269

TCP and SSH Routing

Both gateways support TCP routing for SSH. The ssh listener on port 22 is restricted to the dev namespace using a namespace selector kubernetes/apps/network/envoy-gateway/app/envoy.yaml69-79kubernetes/apps/network/envoy-gateway/app/envoy.yaml129-139

Sources: kubernetes/apps/network/envoy-gateway/app/envoy.yaml69-269

Autoscaling and Observability

KEDA Integration

The Envoy Gateway and the individual Envoy proxy deployments are managed by KEDA (Kubernetes Event-driven Autoscaling) kubernetes/apps/network/envoy-gateway/app/scaledobject.yaml1-87

• Triggers: Scaling is driven by Prometheus queries that monitor the success rate of external endpoints via Gatus metrics kubernetes/apps/network/envoy-gateway/app/scaledobject.yaml24-28
• Scale to Zero: The configuration allows minReplicaCount: 1kubernetes/apps/network/envoy-gateway/app/scaledobject.yaml12

Monitoring

Envoy metrics are exported to Prometheus with Zstd compression enabled kubernetes/apps/network/envoy-gateway/app/envoy.yaml26-30 Multiple Grafana dashboards are provisioned to visualize gateway health:

• network-envoy: General status kubernetes/apps/network/envoy-gateway/app/grafanadashboard.yaml7
• network-envoy-overview: High-level traffic stats kubernetes/apps/network/envoy-gateway/app/grafanadashboard.yaml27
• network-envoy-upstream/downstream: Detailed connection and latency metrics kubernetes/apps/network/envoy-gateway/app/grafanadashboard.yaml45-63

Ingress Data Flow

sequenceDiagram
    participant Client
    participant LB["Cilium LoadBalancer (10.10.30.26)"]
    participant Envoy["Envoy Proxy (envoy-external)"]
    participant App["Target Application Pod"]
    participant LB
    participant Envoy
    participant App
    Client->>LB: Request (HTTPS/443)
    LB->>Envoy: Proxy Traffic
    Note over Envoy: Apply ClientTrafficPolicy (TLS 1.2+)
    Note over Envoy: Run Lua Extension (User-Agent Block)
    Envoy->>App: Forward to Backend Service
    Note over Envoy: Apply BackendTrafficPolicy (Compression/Retry)
    App-->>Envoy: Response
    Envoy-->>Client: Compressed Response

Sources: kubernetes/apps/network/envoy-gateway/app/envoy.yaml49-248kubernetes/apps/network/envoy-gateway/app/scaledobject.yaml60-87

Deployment and Lifecycle

Envoy Gateway is deployed via a HelmRelease using an OCI repository kubernetes/apps/network/envoy-gateway/app/helmrelease.yaml3-10 The provider is set to Kubernetes with the GatewayNamespace deployment type, meaning Envoy proxies are deployed into the same namespace as the Gateway resources kubernetes/apps/network/envoy-gateway/app/helmrelease.yaml16-20

During cluster bootstrap, Envoy Gateway CRDs are applied using Server-Side Apply via the apply_crds function in the bootstrap script scripts/bootstrap-cluster.sh115-132 This ensures the large Envoy-related CRDs are correctly handled before the controller starts scripts/bootstrap-cluster.sh123

Sources: kubernetes/apps/network/envoy-gateway/app/helmrelease.yaml1-23scripts/bootstrap-cluster.sh115-132