Unable to boot after turning on IOMMU in the bios

vepar · September 21, 2023, 2:55pm

Hi!

I’m not sure what’s happening here:

This only happens after i turn on IOMMU in the BIOS. If it’s disabled, the system boots normally, and all the PCI devices are recognized.

This is an ASrock B550m PRO4 motherboard, Ryzen 5 5600g CPU and 2 GPUS, one in each PCIE slot, a Nvidia 1060 3GB, and an AMD R7 260x 1GB…

I used to run this setup but without the nvidia card, the iGPU for the host, and the PCIE card for the guest. Worked just fine.
Now i’ve put the nvidia card as my main one for the host, and the second PCIE slot for the guest (doesn’t need full lanes, just for so Affinity can run), and i get this error.

If it boots, all the USB devices are disabled.
WIthout IOMMU, everything’s fine.

I flashed the latest BIOS.
The bootloader is GRUB, but this happeens to systemd-boot as well when i tried it.

How do i fix this? I kinda need the virtualization to work with the GPU passthrough.

Thanks for reading!

milkytwix · September 21, 2023, 3:30pm

My bet is on: Bad wiring of IOMMU groups. Can I see the output of

sudo dmesg |grep -e group |awk '{print $NF" "$0}' |sort -n

What you should also test ist to completely deactivate USB everywhere and then boot with both GPUs, or boot in different PCIe slots. Also a question: The iGPU is disabled in the BIOS?

vepar · September 21, 2023, 3:36pm

Here’s the output of dmesg:

groups [    0.092326] ftrace: allocated 182 pages with 5 groups
8224593 [    0.042666] Built 1 zonelists, mobility grouping on.  Total pages: 8224593

How would i completely deactivate USB?
I’d rather not switch up GPUs, for now, they both work, and are recognized in lspci, must be something wrong with the IOMMU groups like you mentioned maybe…

And yes, the iGPU is disabled.

milkytwix · September 21, 2023, 3:42pm

That isd all? I don’t see any group at all - here is mine for copmarision:

sudo dmesg |grep -e group |awk '{print $NF" "$0}' |sort -n
0 [    0.631661] pci 0000:00:01.0: Adding to iommu group 0
groups [    0.144927] ftrace: allocated 190 pages with 6 groups
1 [    0.631674] pci 0000:00:01.1: Adding to iommu group 1
2 [    0.631687] pci 0000:00:01.2: Adding to iommu group 2
3 [    0.631707] pci 0000:00:02.0: Adding to iommu group 3
4 [    0.631726] pci 0000:00:03.0: Adding to iommu group 4
5 [    0.631738] pci 0000:00:03.1: Adding to iommu group 5
6 [    0.631757] pci 0000:00:04.0: Adding to iommu group 6
7 [    0.631775] pci 0000:00:05.0: Adding to iommu group 7
8 [    0.631794] pci 0000:00:07.0: Adding to iommu group 8
9 [    0.631807] pci 0000:00:07.1: Adding to iommu group 9
10 [    0.631826] pci 0000:00:08.0: Adding to iommu group 10
11 [    0.631840] pci 0000:00:08.1: Adding to iommu group 11
12 [    0.631865] pci 0000:00:14.0: Adding to iommu group 12
12 [    0.631877] pci 0000:00:14.3: Adding to iommu group 12
13 [    0.631942] pci 0000:00:18.0: Adding to iommu group 13
13 [    0.631954] pci 0000:00:18.1: Adding to iommu group 13
13 [    0.631968] pci 0000:00:18.2: Adding to iommu group 13
13 [    0.631980] pci 0000:00:18.3: Adding to iommu group 13
13 [    0.631992] pci 0000:00:18.4: Adding to iommu group 13
13 [    0.632005] pci 0000:00:18.5: Adding to iommu group 13
13 [    0.632017] pci 0000:00:18.6: Adding to iommu group 13
13 [    0.632031] pci 0000:00:18.7: Adding to iommu group 13
14 [    0.632043] pci 0000:01:00.0: Adding to iommu group 14
15 [    0.632056] pci 0000:20:00.0: Adding to iommu group 15
16 [    0.632092] pci 0000:21:01.0: Adding to iommu group 16
17 [    0.632127] pci 0000:21:02.0: Adding to iommu group 17
18 [    0.632146] pci 0000:21:08.0: Adding to iommu group 18
18 [    0.632259] pci 0000:2a:00.0: Adding to iommu group 18
18 [    0.632262] pci 0000:2a:00.1: Adding to iommu group 18
18 [    0.632266] pci 0000:2a:00.3: Adding to iommu group 18
19 [    0.632166] pci 0000:21:09.0: Adding to iommu group 19
19 [    0.632270] pci 0000:2b:00.0: Adding to iommu group 19
20 [    0.632185] pci 0000:21:0a.0: Adding to iommu group 20
20 [    0.632273] pci 0000:2c:00.0: Adding to iommu group 20
21 [    0.632220] pci 0000:23:00.0: Adding to iommu group 21
22 [    0.632254] pci 0000:24:00.0: Adding to iommu group 22
23 [    0.632287] pci 0000:2d:00.0: Adding to iommu group 23
24 [    0.632300] pci 0000:2e:00.0: Adding to iommu group 24
25 [    0.632316] pci 0000:2f:00.0: Adding to iommu group 25
26 [    0.632332] pci 0000:2f:00.1: Adding to iommu group 26
27 [    0.632348] pci 0000:2f:00.2: Adding to iommu group 27
28 [    0.632365] pci 0000:2f:00.3: Adding to iommu group 28
29 [    0.632377] pci 0000:30:00.0: Adding to iommu group 29
30 [    0.632390] pci 0000:31:00.0: Adding to iommu group 30
31 [    0.632404] pci 0000:31:00.1: Adding to iommu group 31
32 [    0.632418] pci 0000:31:00.3: Adding to iommu group 32

This could give you a hint - together with lspci -vvv - if there is a group collision.
Switching of USB depends on the BIOS / UEFI. I can switch my onboard controllers completely off in mine. Did you enable extended memory on top of VT-D (or whatever the AMD thingy is named)? Can you give

inxi -F

for more information?

vepar · September 21, 2023, 3:54pm

Yup, that’s all it gives me.

It’s probably because IOMMU is disabled in BIOS. But i can’t enable it because it won’t boot then, and i won’t be able to write anyhting into the terminal anyway to see if the groups are correct or not.

inxi -F

System:
  Host: eos Kernel: 6.1.54-1-lts arch: x86_64 bits: 64 Desktop: Xfce v: 4.18.1
    Distro: EndeavourOS
Machine:
  Type: Desktop Mobo: ASRock model: B550M Pro4 serial: <superuser required>
    UEFI: American Megatrends LLC. v: P2.90 date: 08/01/2023
CPU:
  Info: 6-core model: AMD Ryzen 5 5600G with Radeon Graphics bits: 64
    type: MT MCP cache: L2: 3 MiB
  Speed (MHz): avg: 1427 min/max: 1400/4464 cores: 1: 1400 2: 1400 3: 1400
    4: 1400 5: 1400 6: 1732 7: 1400 8: 1400 9: 1400 10: 1400 11: 1400 12: 1400
Graphics:
  Device-1: NVIDIA GP106 [GeForce GTX 1060 3GB] driver: nvidia v: 535.104.05
  Device-2: AMD Bonaire XTX [Radeon R7 260X/360] driver: radeon v: kernel
  Display: x11 server: X.Org v: 21.1.8 driver: X: loaded: modesetting,nvidia
    unloaded: radeon dri: radeonsi gpu: nvidia,nvidia-nvswitch,radeon
    resolution: 1: 2560x1440~60Hz 2: 2560x1440~60Hz 3: N/A
  API: OpenGL v: 4.6.0 NVIDIA 535.104.05 renderer: NVIDIA GeForce GTX 1060
    3GB/PCIe/SSE2
Audio:
  Device-1: NVIDIA GP106 High Definition Audio driver: snd_hda_intel
  Device-2: AMD Tobago HDMI Audio [Radeon R7 360 / R9 OEM]
    driver: snd_hda_intel
  Device-3: AMD Renoir Radeon High Definition Audio driver: snd_hda_intel
  Device-4: AMD Family 17h/19h HD Audio driver: snd_hda_intel
  Device-5: Focusrite-Novation Scarlett 2i4 USB driver: snd-usb-audio
    type: USB
  API: ALSA v: k6.1.54-1-lts status: kernel-api
  Server-1: PipeWire v: 0.3.80 status: active
Network:
  Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    driver: r8169
  IF: enp6s0 state: up speed: 1000 Mbps duplex: full mac: 70:85:c2:9c:6a:d5
Drives:
  Local Storage: total: 1.14 TiB used: 7.53 GiB (0.6%)
  ID-1: /dev/nvme0n1 vendor: Kingston model: SNV2S500G size: 465.76 GiB
  ID-2: /dev/nvme1n1 vendor: Kingston model: SNV2S500G size: 465.76 GiB
  ID-3: /dev/sda vendor: Samsung model: SSD 860 EVO 250GB size: 232.89 GiB
Partition:
  ID-1: / size: 464.78 GiB used: 7.53 GiB (1.6%) fs: btrfs dev: /dev/nvme1n1p2
  ID-2: /boot/efi size: 998 MiB used: 576 KiB (0.1%) fs: vfat
    dev: /dev/nvme1n1p1
  ID-3: /home size: 464.78 GiB used: 7.53 GiB (1.6%) fs: btrfs
    dev: /dev/nvme1n1p2
  ID-4: /var/log size: 464.78 GiB used: 7.53 GiB (1.6%) fs: btrfs
    dev: /dev/nvme1n1p2
Swap:
  ID-1: swap-1 type: file size: 512 MiB used: 0 KiB (0.0%)
    file: /swap/swapfile
Sensors:
  System Temperatures: cpu: 51.2 C mobo: N/A
  Fan Speeds (rpm): N/A
  GPU: device: nvidia screen: :0.0 temp: 49 C fan: 27% device: radeon
    temp: 40.0 C
Info:
  Processes: 300 Uptime: 0m Memory: total: 32 GiB available: 31.21 GiB
  used: 1.86 GiB (5.9%) Shell: Bash inxi: 3.3.29

lspci -vvv

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne Root Complex
	Subsystem: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne Root Complex
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge (prog-if 00 [Normal decode])
	Subsystem: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 25
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: f000-ffff [size=4K] [16-bit]
	Memory behind bridge: fb000000-fc0fffff [size=17M] [32-bit]
	Prefetchable memory behind bridge: b0000000-c1ffffff [size=288M] [32-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge (prog-if 00 [Normal decode])
	Subsystem: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 26
	Bus: primary=00, secondary=02, subordinate=06, sec-latency=0
	I/O behind bridge: d000-efff [size=8K] [16-bit]
	Memory behind bridge: fc600000-fc9fffff [size=4M] [32-bit]
	Prefetchable memory behind bridge: d0000000-e07fffff [size=264M] [32-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge (prog-if 00 [Normal decode])
	Subsystem: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 27
	Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled] [32-bit]
	Memory behind bridge: fca00000-fcafffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus (prog-if 00 [Normal decode])
	Subsystem: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 28
	Bus: primary=00, secondary=08, subordinate=08, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled] [32-bit]
	Memory behind bridge: fc200000-fc5fffff [size=4M] [32-bit]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51)
	Subsystem: ASRock Incorporation FCH SMBus Controller
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: piix4_smbus
	Kernel modules: i2c_piix4, sp5100_tco

00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
	Subsystem: ASRock Incorporation FCH LPC Bridge
	Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 0
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 2
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 3
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Kernel driver in use: k10temp
	Kernel modules: k10temp

00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 4
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 5
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 6
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 7
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

01:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 83
	Region 0: Memory at fb000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at b0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at c0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at f000 [size=128]
	Expansion ROM at fc000000 [virtual] [disabled] [size=512K]
	Capabilities: <access denied>
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia

01:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
	Subsystem: NVIDIA Corporation GP106 High Definition Audio Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 85
	Region 0: Memory at fc080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset USB 3.1 XHCI Controller (prog-if 30 [XHCI])
	Subsystem: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 31
	Region 0: Memory at fc9a0000 (64-bit, non-prefetchable) [size=32K]
	Capabilities: <access denied>
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset SATA Controller (prog-if 01 [AHCI 1.0])
	Subsystem: ASMedia Technology Inc. ASM1062 Serial ATA Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 36
	Region 5: Memory at fc980000 (32-bit, non-prefetchable) [size=128K]
	Expansion ROM at fc900000 [disabled] [size=512K]
	Capabilities: <access denied>
	Kernel driver in use: ahci

02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset Switch Upstream Port (prog-if 00 [Normal decode])
	Subsystem: ASMedia Technology Inc. 500 Series Chipset Switch Upstream Port
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 30
	Bus: primary=02, secondary=03, subordinate=06, sec-latency=0
	I/O behind bridge: d000-efff [size=8K] [16-bit]
	Memory behind bridge: fc600000-fc8fffff [size=3M] [32-bit]
	Prefetchable memory behind bridge: d0000000-e07fffff [size=264M] [32-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea (prog-if 00 [Normal decode])
	Subsystem: ASMedia Technology Inc. Device 3308
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 32
	Bus: primary=03, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: e000-efff [size=4K] [16-bit]
	Memory behind bridge: fc800000-fc8fffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: d0000000-e07fffff [size=264M] [32-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea (prog-if 00 [Normal decode])
	Subsystem: ASMedia Technology Inc. Device 3308
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 33
	Bus: primary=03, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 0000f000-00000fff [disabled] [32-bit]
	Memory behind bridge: fc700000-fc7fffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea (prog-if 00 [Normal decode])
	Subsystem: ASMedia Technology Inc. Device 3308
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 34
	Bus: primary=03, secondary=06, subordinate=06, sec-latency=0
	I/O behind bridge: d000-dfff [size=4K] [16-bit]
	Memory behind bridge: fc600000-fc6fffff [size=1M] [32-bit]
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] [64-bit]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XTX [Radeon R7 260X/360] (prog-if 00 [VGA controller])
	Subsystem: PC Partner Limited / Sapphire Technology Bonaire XTX [Radeon R7 260X/360]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 90
	Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at e0000000 (64-bit, prefetchable) [size=8M]
	Region 4: I/O ports at e000 [size=256]
	Region 5: Memory at fc800000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at fc840000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: radeon
	Kernel modules: radeon, amdgpu

04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tobago HDMI Audio [Radeon R7 360 / R9 360 OEM]
	Subsystem: PC Partner Limited / Sapphire Technology Tobago HDMI Audio [Radeon R7 360 / R9 360 OEM]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 86
	Region 0: Memory at fc860000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

05:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. Device 5017 (rev 03) (prog-if 02 [NVM Express])
	Subsystem: Kingston Technology Company, Inc. Device 5017
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 31
	NUMA node: 0
	Region 0: Memory at fc700000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: nvme
	Kernel modules: nvme

06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
	Subsystem: ASRock Incorporation Motherboard (one of many)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 31
	Region 0: I/O ports at d000 [size=256]
	Region 2: Memory at fc604000 (64-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fc600000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: r8169
	Kernel modules: r8169

07:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. Device 5017 (rev 03) (prog-if 02 [NVM Express])
	Subsystem: Kingston Technology Company, Inc. Device 5017
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 40
	NUMA node: 0
	Region 0: Memory at fca00000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: nvme
	Kernel modules: nvme

08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function (rev c9)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Zeppelin/Raven/Raven2 PCIe Dummy Function
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: <access denied>

08:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 88
	Region 0: Memory at fc508000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

08:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
	Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort+ <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 31
	Region 2: Memory at fc400000 (32-bit, non-prefetchable) [size=1M]
	Region 5: Memory at fc50c000 (32-bit, non-prefetchable) [size=8K]
	Capabilities: <access denied>
	Kernel driver in use: ccp
	Kernel modules: ccp

08:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 (prog-if 30 [XHCI])
	Subsystem: ASRock Incorporation Renoir/Cezanne USB 3.1
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 35
	Region 0: Memory at fc300000 (64-bit, non-prefetchable) [size=1M]
	Capabilities: <access denied>
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

08:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 (prog-if 30 [XHCI])
	Subsystem: ASRock Incorporation Renoir/Cezanne USB 3.1
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 29
	Region 0: Memory at fc200000 (64-bit, non-prefetchable) [size=1M]
	Capabilities: <access denied>
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

08:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller
	Subsystem: ASRock Incorporation Family 17h/19h HD Audio Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 89
	Region 0: Memory at fc500000 (32-bit, non-prefetchable) [size=32K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

What’s VT-D? I tried googling it but can’t find what AMDs equivalent is called. I’ll reboot and take some pictures of the bios.

EDIT: I also tried setting igpu to auto but it’s still the same.
I mean, this is probably a BIOS misconfig issue i’m having, but idk what exact settings conflict with amd IOMMU.

amd_iommu=on is in grub, and qemu-full is installed. It was the same issue without that installed and set.

vepar · September 21, 2023, 4:03pm

Here’s the BIOS settings, all the possibly relevant things i could find:

[Album] Imgur: The magic of the Internet

milkytwix · September 21, 2023, 4:06pm

So, for IOMMU to work you need three things:

correct Kernel parameters
Support within the UEFI
Isolation

Basically everything within https://pve.proxmox.com/wiki/PCI_Passthrough guide must be done. On top of it, to enable it within your UEFI you have to dig quite deep:
https://www.reddit.com/r/VFIO/comments/cxbnqn/protip_for_anyone_wondering_how_to_enable/
Hope that helps!
And btw, to test it just boot in the simplest of all configs (preferbly only the iGPU) and activate IOMMU. And then improve upon this with first and then second GPU.

vepar · September 21, 2023, 4:09pm

When only thie iGPU was installed, IOMMU worked fine. That’s the problem. I already know i have all the prerequisites, and it did work on my previous install, but now i can’t get it to work at all. I can’t even enable IOMMU in the bios because it doesn’t boot.

milkytwix · September 21, 2023, 4:21pm

Did you care for Isolation? Did you care for the extension of the memory adressing? Here you will find more:

And just because “some” parts of IOMMU were working (what did exactly work?) does not mean that all features are working - as you experience right now, unfortunately…

vepar · September 21, 2023, 4:28pm

Well, everything i needed was working.
Here was the setup:

iGPU for the host
R7 260x in the 1st PCIEx16 slot passed to the VM

now, all i did was:

Nvidia 1060 in the 1st PCIEx16 slot
R7 260x in the 2nd PCIE.16 slot

I mean, should work, but i can’t even start the configuration of the IOMMU because i can’t even enable it. I don’t see any groups, nothing because it’s disabled.
I’ll check the link.

ricklinux · September 21, 2023, 4:44pm

Have you tried it as a kernel parameter in the deafult grub command line and then run the update grub command?

amd_iommu=on

vepar · September 21, 2023, 4:46pm

@ricklinux
I did, with or without it, it failed.

But, i removed the R7 260x gpu, turned on IOMMU, and it boots…
So there’s something wrong with either that slot or the card or the IOMMU configuration.

The groups it listed with dmesg look weird.

0 [    0.307663] pci 0000:00:01.0: Adding to iommu group 0
0 [    0.307669] pci 0000:00:01.1: Adding to iommu group 0
0 [    0.307817] pci 0000:01:00.0: Adding to iommu group 0
0 [    0.307821] pci 0000:01:00.1: Adding to iommu group 0
groups [    0.091794] ftrace: allocated 182 pages with 5 groups
1 [    0.307683] pci 0000:00:02.0: Adding to iommu group 1
1 [    0.307689] pci 0000:00:02.1: Adding to iommu group 1
1 [    0.307696] pci 0000:00:02.2: Adding to iommu group 1
1 [    0.307825] pci 0000:02:00.0: Adding to iommu group 1
1 [    0.307829] pci 0000:02:00.1: Adding to iommu group 1
1 [    0.307832] pci 0000:02:00.2: Adding to iommu group 1
1 [    0.307837] pci 0000:03:04.0: Adding to iommu group 1
1 [    0.307841] pci 0000:03:08.0: Adding to iommu group 1
1 [    0.307845] pci 0000:04:00.0: Adding to iommu group 1
1 [    0.307849] pci 0000:05:00.0: Adding to iommu group 1
1 [    0.307854] pci 0000:06:00.0: Adding to iommu group 1
2 [    0.307706] pci 0000:00:08.0: Adding to iommu group 2
2 [    0.307713] pci 0000:00:08.1: Adding to iommu group 2
2 [    0.307868] pci 0000:07:00.0: Adding to iommu group 2
2 [    0.307873] pci 0000:07:00.1: Adding to iommu group 2
2 [    0.307877] pci 0000:07:00.2: Adding to iommu group 2
2 [    0.307881] pci 0000:07:00.3: Adding to iommu group 2
2 [    0.307885] pci 0000:07:00.4: Adding to iommu group 2
2 [    0.307890] pci 0000:07:00.6: Adding to iommu group 2
3 [    0.307725] pci 0000:00:14.0: Adding to iommu group 3
3 [    0.307731] pci 0000:00:14.3: Adding to iommu group 3
4 [    0.307757] pci 0000:00:18.0: Adding to iommu group 4
4 [    0.307765] pci 0000:00:18.1: Adding to iommu group 4
4 [    0.307772] pci 0000:00:18.2: Adding to iommu group 4
4 [    0.307782] pci 0000:00:18.3: Adding to iommu group 4
4 [    0.307790] pci 0000:00:18.4: Adding to iommu group 4
4 [    0.307798] pci 0000:00:18.5: Adding to iommu group 4
4 [    0.307805] pci 0000:00:18.6: Adding to iommu group 4
4 [    0.307812] pci 0000:00:18.7: Adding to iommu group 4

@milkytwix your had tons of groups, mine only shows 0-5, maybe that’s why something’s not working?

ricklinux · September 21, 2023, 4:51pm

I would say that is because the 5600G uses amdgpu and the R7 3260X is an older card that probably uses radeon.

vepar · September 21, 2023, 4:54pm

Well, it worked before with the igpu. It booted and everything.
Maybe now with nvidia, or that second PCIEx16 slot has something to do with it. Or the 5 iommu groups, seems like too few, but idk.

I’d pass the iGPU to the VM, but it’s listed as “dummy function” ever since i installed the nvidia card, and i can’t turn it on anymore in bios.

This is the iGPU:

07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function (rev c9)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Zeppelin/Raven/Raven2 PCIe Dummy Function

It only ever got recognized as the renoir gpu once, then bios shut it down. Weird…

ricklinux · September 21, 2023, 4:57pm

When you start changing hardware and the system is already installed you can expect to have issues. That card is an older Ati Radeon. I don’t do pass through to vm.

vepar · September 21, 2023, 4:59pm

Well, this is a freshly installed system.
But yeah, seems there’s a hardware conflict with that R7 260x. I’ll have to figure out some other way to do passthrough.

anon49550872 · September 21, 2023, 5:01pm

Have you tried to disable modesetting https://bbs.archlinux.org/viewtopic.php?id=270961 ?

vepar · September 21, 2023, 5:07pm

I haven’t. Wasn’t aware that has anything to do with IOMMU configuration.
There is nvidia-drm.modeset=1 in there though.

anon49550872 · September 21, 2023, 5:12pm

Same here, but the error in that link is similar to yours, and with nomodeset worked.
I think it worth the shot.

ricklinux · September 21, 2023, 5:27pm

I think you may be right.