Part 4: Advanced VM Networking & MAC Address Governance

Containers thrive on ephemeral, dynamic IP addresses. If a container dies, Kubernetes replaces it with a new pod, a new IP, and services automatically route traffic via kube-proxy or OVN-Kubernetes. Virtual Machines, however, come from an entirely different enterprise heritage. Legacy enterprise workloads running inside VMs often require: Static, predictable Layer-2 IP addresses. Direct connectivity to external corporate VLANs without NAT or Ingress proxies. Multiple Network Interface Cards (NICs) separating management, storage, and application traffic. Persistent MAC addresses that survive reboots, live migrations, and cluster restarts. In this fourth part of our series, we dive deep into OpenShift Virtualization Networking, uncovering how Multus CNI bridges the gap between Kubernetes and physical networks, and how KubeMacPool prevents devastating Layer-2 MAC address collisions. 🏗️ Dual-Layer Architecture: Pod Network vs. Secondary Networks OpenShift Virtualization solves the networking dilemma by offering a flexible, dual-layer networking model: The two networking layers operate simultaneously: The Default Pod Network (Primary): Connects the VM to the internal cluster SDN (OVN-Kubernetes) using Masquerade (NAT) mode. The VM can reach Kubernetes Services, DNS, and Ingress routers. Secondary Physical Networks (Multus CNI): Bypasses the cluster SDN entirely, plugging the VM directly into an external VLAN via a Linux Bridge or Open vSwitch (OVS). 🔌 Connecting VMs Directly to External VLANs To attach a virtual machine directly to an existing physical corporate network (e.g., VLAN 100 on subnet 192.168.10.0/24), we follow a three-step configuration: 1. Configure the Physical Bridge on Worker Nodes (NMState) Using the Kubernetes NMState Operator, we declare a NodeNetworkConfigurationPolicy (NNCP) that creates a Linux Bridge on the physical NIC of worker nodes: apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: br-vlan100-policy spec: nodeSelector: node-role.kubernetes.io/worker: "" desiredState: interfaces: - name: br-vlan100 type: linux-bridge state: up bridge: options: stp: enabled: false port: - name: eth1 2. Define the NetworkAttachmentDefinition (NAD) Multus CNI requires a NetworkAttachmentDefinition in the workload namespace to make the bridge available: apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: vlan100-net namespace: vm-workloads spec: config: '{ "cniVersion": "0.3.1", "name": "vlan100-net", "type": "cnv-bridge", "bridge": "br-vlan100", "macspoofchk": true }' 3. Attach Dual NICs to the VirtualMachine Now, in your VirtualMachine manifest, declare both interfaces under devices.interfaces and map them under networks: apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: dual-nic-enterprise-vm namespace: vm-workloads spec: running: true template: spec: domain: devices: interfaces: - name: default-pod-net masquerade: {} # Cluster SDN access - name: physical-vlan # Direct external VLAN access bridge: {} resources: requests: memory: 4Gi cpu: 2 networks: - name: default-pod-net pod: {} - name: physical-vlan multus: networkName: vlan100-net Inside the guest OS, you will now see two physical network adapters (eth0 and eth1). 🏷️ MAC Address Governance: Enter KubeMacPool In a standard Kubernetes cluster, pods receive random MAC addresses upon startup because Layer-2 MAC uniqueness rarely matters across routed overlay networks. However, when multiple VMs connect directly to an external Layer-2 bridge: If two VMs accidentally receive the same MAC address, an enterprise network collision occurs, dropping packets across the physical switch. If a VM loses its MAC address after a live migration or restart, external DHCP reservations and ARP tables break. How KubeMacPool Works KubeMacPool is an intelligent admission controller deployed automatically with OpenShift Virtualization: It manages a dedicated, configured pool of unique IEEE MAC addresses (e.g., 02:00:00:00:00:00 to 02:FF:FF:FF:FF:FF). When a VirtualMachine is submitted, KubeMacPool's mutating webhook intercepts the creation request and injects a guaranteed-unique static MAC address into the manifest. It tracks allocated MAC addresses in real-time, preventing duplicate assignments across all namespaces. ⚠️ Battle-Tested Gotchas & Production Traps 1. The Backup/Restore MAC Collision Trap When restoring a VM backup (via OADP/Velero) into a new namespace for disaster recovery testing, KubeMacPool will intercept the restore request and block it with a webhook validation error: MAC address already allocated to another VM. 🛠️ The Solution: Temporarily instruct KubeMacPool to ignore the target restore namespace using an exclusion label: oc label namespace vm-workloads-restored \ mutatevirtualmachines.kubemacpool.io=ignore --overwrite 2. Promiscuous Mode on Nested Hypervisors If your OpenShift cluster is running inside a nested virtualization platform (such as VMware vSphere or OpenStack): The physical Linux bridge needs to pass foreign MAC addresses through the underlying vSwitch. Ensure Promiscuous Mode and MAC Address Changes are set to "Accept" on the host port group / security group, otherwise incoming traffic will be silently discarded by the hypervisor host. 3. MTU Mismatch If your cluster SDN uses an MTU of 1450 (Geneve encapsulation) but your external VLAN uses standard Ethernet MTU 1500 or Jumbo Frames 9000: Ensure your NetworkAttachmentDefinition specifies the correct MTU matching the physical switch port to avoid silent TCP packet fragmentation and packet drops. 🎯 Key Takeaways Default Pod Network (masquerade) is ideal for Kubernetes-internal communications and standard Ingress routing. Multus CNI (cnv-bridge) allows seamless, direct connection to physical enterprise VLANs and legacy subnets. KubeMacPool ensures carrier-grade Layer-2 stability by preventing duplicate MAC addresses across your cluster fleet. ⏭️ Next Up: Enterprise Backup & Disaster Recovery (DR) Now that our VMs have reliable compute, high-performance storage, and enterprise networking, how do we protect them against catastrophic failures? In Part 5, we will construct an end-to-end Enterprise Disaster Recovery strategy using OADP (Velero) and NooBaa (S3)—including how to configure QEMU Guest Agent hooks (fsfreeze / fsthaw) for 100% application-consistent snapshots. 💬 How are you handling VM VLANs and IP management in your container platform? Let us know in the comments below!