Cloudjur

Multus-CNI-and-VLAN-Networking

3 min read

Multus CNI and VLAN Networking

Relevant source files

The home-ops cluster utilizes Multus CNI to enable multi-homed networking for specific workloads, primarily Home Assistant. While Cilium provides the primary CNI for pod-to-pod communication, Multus acts as a “meta-plugin” that allows pods to attach to multiple physical or virtual network interfaces. This is critical for home automation scenarios where a pod must reside simultaneously on the primary cluster network and various isolated VLANs (IoT, Trusted, or No-Internet) to discover and control local hardware.

Multus CNI Architecture

Multus is deployed via a HelmRelease using the app-template chart from the bjw-s repository kubernetes/apps/network/multus/app/helmrelease.yaml5-10 It configures the standard CNI binary and network configuration paths on the Talos nodes kubernetes/apps/network/multus/app/helmrelease.yaml12-14

Data Flow: Multi-Interface Attachment

The following diagram illustrates how Multus orchestrates the attachment of multiple interfaces to a Pod using NetworkAttachmentDefinitions.

Diagram: Multus Interface Orchestration

[Flowchart Diagram]

Sources:kubernetes/apps/network/multus/config/net-attach-iot.yaml11-15kubernetes/apps/network/multus/config/net-attach-trust.yaml11-15kubernetes/apps/network/multus/config/net-attach-not.yaml11-15

NetworkAttachmentDefinitions

The cluster defines several NetworkAttachmentDefinition (NAD) resources to bridge pods into specific VLANs. These use the macvlan plugin in bridge mode, allowing the pod to appear as a physical device on the specified host interface kubernetes/apps/network/multus/config/net-attach-iot.yaml13-15

NAD NameHost InterfaceVLAN PurposeGatewaySubnet
multus-ioteth3IoT Devices10.10.33.110.10.33.0/24
multus-trusteth4Trusted Devices10.10.10.110.10.10.0/24
multus-noteth2No-Internet (Isolated)10.10.32.110.10.32.0/24

Source-Based Routing (SBR)

Each NAD includes the sbr (Source-Based Routing) plugin kubernetes/apps/network/multus/config/net-attach-iot.yaml33-35 This is essential for multi-homed pods to ensure that traffic entering a specific interface (e.g., net1 for IoT) is responded to via that same interface, rather than trying to use the default gateway on eth0.

Sources:kubernetes/apps/network/multus/config/net-attach-iot.yaml1-36kubernetes/apps/network/multus/config/net-attach-trust.yaml1-37kubernetes/apps/network/multus/config/net-attach-not.yaml1-36

Home Assistant Implementation

Home Assistant is the primary consumer of these Multus interfaces. It is configured via pod annotations in its HelmReleasekubernetes/apps/home-automation/home-assistant/app/helmrelease.yaml38-54

Pod Configuration

The k8s.v1.cni.cncf.io/networks annotation defines the requested interfaces and assigns static IP addresses within the respective VLAN subnets:

Trusted Proxies

Because Home Assistant receives traffic from the envoy-external ingress controller, it must trust the internal CIDRs to correctly identify client IPs. The configuration includes HASS_HTTP_TRUSTED_PROXY_1 (10.0.0.0/8) and HASS_HTTP_TRUSTED_PROXY_2 (10.10.30.0/16) kubernetes/apps/home-automation/home-assistant/app/helmrelease.yaml68-69

Sources:kubernetes/apps/home-automation/home-assistant/app/helmrelease.yaml33-70

External Service Proxying

While Multus handles pod-to-VLAN connectivity, the cluster also manages connectivity to legacy hardware and hypervisors that exist outside the Kubernetes cluster but within the same physical network.

Service and EndpointSlice Pattern

For external entities like the Proxmox host (pve) or the NAS, the cluster uses a combination of a headless Service and a manual EndpointSlice to route internal traffic to external IPs kubernetes/apps/network/external-services/app/nas.yaml2-35

Diagram: External Service Routing

[Flowchart Diagram]

TLS Passthrough

For management interfaces like Proxmox and the NAS, TLSRoute resources are used with envoy-internal to perform TLS passthrough kubernetes/apps/network/external-services/app/proxmox.yaml38-55 This allows the external services to manage their own SSL certificates while being accessible via internal cluster hostnames like pve.cloudjur.comkubernetes/apps/network/external-services/app/proxmox.yaml46

Sources:kubernetes/apps/network/external-services/app/nas.yaml1-55kubernetes/apps/network/external-services/app/proxmox.yaml1-55kubernetes/apps/network/external-services/app/ap.yaml1-45

Graph View

Backlinks