A banner image for a blog post with a dramatic and tech-themed design. The image should depict a broken or malfunctioning network interface card (NIC)

Newly Installed NIC Not Working? Here’s How I Fixed Mine on Proxmox!

2024-09-09 / 3 min read
Last updated: 2024-09-09

Guides

If you installed a new NIC in your Proxmox server and it's not working as expected, don't give up just yet! The solution might be simpler than you think, thanks to IOMMU grouping adjustments.

I recently bought a used 10 GbE dual SFP port PCI card off eBay, and when it didn’t work right away, I thought it might be because it was second-hand hardware. But I didn't give up easily, and I hope my solution can help you too.

Common Symptoms of IOMMU Group Conflicts in relation to a PCI NIC

  • Network connections drop or become unstable when starting a VM.
  • Devices that are not passed through still behave erratically.
  • Unexpected resets or resource conflicts appear in dmesg.

To make this work, you need to have IOMMU enabled on your system. If you haven't set it up yet, check out my Proxmox setup guide with IOMMU instructions.

Step-by-Step Solution: Using ACS Override on Proxmox

Step 1: Identify the Devices and Their IOMMU Groups

Start by identifying which devices are causing the conflict by listing all IOMMU groups:

find /sys/kernel/iommu_groups/ -type l

Identify the conflicting devices by checking their PCI addresses:

lspci -nn | grep -i ethernet
lspci -nn | grep -i sata

Step 2: Apply ACS Override to Separate Devices

The ACS (Access Control Services) override patch can force the kernel to separate devices into their own IOMMU groups. This is particularly useful when dealing with multifunction devices and PCI bridges.

Edit the Kernel Command Line: Modify the kernel command line parameters in /etc/kernel/cmdline:

nano /etc/kernel/cmdline

Add the following parameters:

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction

Refresh the Boot Configuration: After editing the file, update the boot entries using:

proxmox-boot-tool refresh

Reboot to Apply Changes: Reboot your system to apply the new settings:

sudo reboot

Step 3: Verify the IOMMU Group Separation

After rebooting, verify that the devices are now in separate groups:

find /sys/kernel/iommu_groups/ -type l

If successful, the previously conflicting devices should now be isolated in their own IOMMU groups.

Alternative Solution: Using vfio-pci to Bind Devices

If ACS override does not work or is not supported by your hardware, you can use vfio-pci to bind specific devices to a different driver, isolating them from the host system.

Identify the Device IDs: Use lspci -nn to find the vendor and device IDs:

lspci -nn | grep -i sata

Configure vfio-pci to Bind the Device: Create a configuration file to bind the device to vfio-pci:

echo "options vfio-pci ids=1b21:0612" > /etc/modprobe.d/vfio.conf

Load the vfio-pci Module: Ensure vfio-pci is loaded at boot:

echo "vfio-pci" >> /etc/modules

Update Initramfs: Apply the changes by updating the initramfs:

update-initramfs -u

Reboot and Verify: Reboot your system and check if the device is correctly bound to vfio-pci:

reboot
lspci -k -s 03:00.0

Conclusion

Dealing with IOMMU group conflicts can be challenging, but with the right approach, you can isolate devices effectively to ensure smooth operation of your Proxmox VMs. Whether using ACS override or binding devices with vfio-pci, these methods provide robust solutions to common passthrough issues. If you found this guide helpful or have additional tips, feel free to share your experience!

Looking for a new 10 GBe NIC? Try this one, I installed this one in my second system and works flawless!