Virt Manager/QEMU - VMs started crashing after system update

firefly · January 25, 2024, 2:24pm

Yesterday, after regular system update my VMs suddenly stopped working
(initial configuration is based on EOS wiki article)

I can open, power on VM, it boots normally until login screen but as soon as I enter password and hit enter, both Virt Manager & VM windows just close. Reopening Virt Manager shows that VM is still running but attempt to open it once again, leads to the exactly same behavior described above.
Also, when this happens, I noticed that each time crossed mic icon briefly shows-up in Plasma system tray saying “virt-manager is using the microphone”.

This happens on both latest kernels - lts 6.6.13 and 6.7.1. Switching Video model and Display type under VM setting doesn’t help either.

Anyone else noticed something similar or have an idea where to start looking in order to troubleshoot it?

Checking both QEMU and Arch bug trackers, as well as Googling, doesn’t show recent reports with similar issue

journalctl shows these errors when VM crashes:

Jan 25 13:15:08 Serenity systemd-coredump[1580]: [🡕] Process 1395 (virt-manager) of user 1000 dumped core.
                                             Stack trace of thread 1572:
                                             #0  0x000074b27019e00c n/a (n/a + 0x0)
                                             ELF object binary architecture: AMD x86-64
Jan 25 13:15:08 Serenity systemd[1]: systemd-coredump@0-1575-0.service: Deactivated successfully.
Jan 25 13:15:08 Serenity systemd[1]: systemd-coredump@0-1575-0.service: Consumed 2.042s CPU time.
Jan 25 13:15:08 Serenity kded5[816]: Service “:1.55” unregistered
Jan 25 13:15:08 Serenity virtqemud[1413]: libvirt version: 10.0.0
Jan 25 13:15:08 Serenity virtqemud[1413]: hostname: Serenity
Jan 25 13:15:08 Serenity virtqemud[1413]: End of file while reading data: Input/output error

ajgringo619 · January 25, 2024, 6:17pm

My last update was on the 21st and all my VMs are running fine. What does paclog --after=2024-01-24 return? When checking pacseek -u, I didn’t see any packages directly related to qemu or libvirt.

firefly · January 25, 2024, 8:24pm

Here are the packages I updated yesterday:

Summary

[2024-01-24T17:38:46+0100] [ALPM] transaction started
[2024-01-24T17:38:46+0100] [ALPM] warning: /etc/passwd installed as /etc/passwd.pacnew
[2024-01-24T17:38:46+0100] [ALPM] upgraded filesystem (2023.09.18-1 -> 2024.01.19-1)
[2024-01-24T17:38:46+0100] [ALPM] upgraded aom (3.8.0-2 -> 3.8.1-1)
[2024-01-24T17:38:46+0100] [ALPM] upgraded readline (8.2.007-1 -> 8.2.010-1)
[2024-01-24T17:38:46+0100] [ALPM] upgraded bash (5.2.021-1 -> 5.2.026-2)
[2024-01-24T17:38:46+0100] [ALPM] upgraded zlib (1:1.3-2 -> 1:1.3.1-1)
[2024-01-24T17:38:46+0100] [ALPM] upgraded btrfs-progs (6.6.3-1 -> 6.7-1)
[2024-01-24T17:38:46+0100] [ALPM] upgraded ca-certificates-mozilla (3.96.1-1 -> 3.97-1)
[2024-01-24T17:38:47+0100] [ALPM] upgraded glib2 (2.78.3-1 -> 2.78.4-1)
[2024-01-24T17:38:47+0100] [ALPM] upgraded libnghttp2 (1.58.0-1 -> 1.59.0-1)
[2024-01-24T17:38:47+0100] [ALPM] upgraded libcolord (1.4.6-1 -> 1.4.7-1)
[2024-01-24T17:38:47+0100] [ALPM] upgraded gtk3 (1:3.24.40-2 -> 1:3.24.41-1)
[2024-01-24T17:38:47+0100] [ALPM] upgraded nss (3.96.1-1 -> 3.97-1)
[2024-01-24T17:38:47+0100] [ALPM] upgraded xdg-utils (1.2.0r25+g0f49cf5-1 -> 1.2.0r28+g9b7d253-1)
[2024-01-24T17:38:48+0100] [ALPM] upgraded chromium (121.0.6167.71-1 -> 121.0.6167.85-1)
[2024-01-24T17:38:48+0100] [ALPM] upgraded ddcutil (2.1.0-3 -> 2.1.0-4)
[2024-01-24T17:38:48+0100] [ALPM] upgraded dracut (059-4 -> 059-5)
[2024-01-24T17:38:48+0100] [ALPM] upgraded orc (0.4.34-1 -> 0.4.35-1)
[2024-01-24T17:38:48+0100] [ALPM] upgraded eos-bash-shared (24.1-1 -> 24.4-1)
[2024-01-24T17:38:48+0100] [ALPM] upgraded fakeroot (1.32.2-1 -> 1.33-1)
[2024-01-24T17:38:48+0100] [ALPM] upgraded libplacebo (6.338.1-1 -> 6.338.2-1)
[2024-01-24T17:38:48+0100] [ALPM] upgraded serd (0.32.0-1 -> 0.32.2-1)
[2024-01-24T17:38:48+0100] [ALPM] upgraded lilv (0.24.22-1 -> 0.24.24-1)
[2024-01-24T17:38:49+0100] [ALPM] upgraded firefox (121.0.1-1 -> 122.0-1)
[2024-01-24T17:38:50+0100] [ALPM] upgraded firefox-developer-edition (122.0b9-1 -> 123.0b1-1)
[2024-01-24T17:38:50+0100] [ALPM] warning: directory permissions differ on /usr/share/polkit-1/rules.d/
filesystem: 750  package: 755
[2024-01-24T17:38:50+0100] [ALPM] upgraded fwupd (1.9.11-2 -> 1.9.12-1)
[2024-01-24T17:38:51+0100] [ALPM] upgraded imagemagick (7.1.1.26-2 -> 7.1.1.27-1)
[2024-01-24T17:38:51+0100] [ALPM] upgraded inxi (3.3.31.1-1 -> 3.3.31.1-3)
[2024-01-24T17:38:51+0100] [ALPM] upgraded iwd (2.13-2 -> 2.13-3)
[2024-01-24T17:38:51+0100] [ALPM] upgraded lib32-zlib (1.3-1 -> 1.3.1-1)
[2024-01-24T17:38:51+0100] [ALPM] upgraded lib32-glib2 (2.78.3-1 -> 2.78.4-1)
[2024-01-24T17:38:51+0100] [ALPM] upgraded lib32-nss (3.96.1-1 -> 3.97-1)
[2024-01-24T17:38:52+0100] [ALPM] upgraded libibus (1.5.29-2 -> 1.5.29-3)
[2024-01-24T17:38:52+0100] [ALPM] upgraded libpaper (2.1.2-1 -> 2.1.3-1)
[2024-01-24T17:38:53+0100] [ALPM] upgraded linux (6.7.arch3-1 -> 6.7.1.arch1-1)
[2024-01-24T17:38:56+0100] [ALPM] upgraded linux-headers (6.7.arch3-1 -> 6.7.1.arch1-1)
[2024-01-24T17:38:58+0100] [ALPM] upgraded linux-lts (6.6.12-1 -> 6.6.13-1)
[2024-01-24T17:39:01+0100] [ALPM] upgraded linux-lts-headers (6.6.12-1 -> 6.6.13-1)
[2024-01-24T17:39:01+0100] [ALPM] upgraded md4c (0.5.0-1 -> 0.5.1-1)
[2024-01-24T17:39:01+0100] [ALPM] upgraded minizip (1:1.3-2 -> 1:1.3.1-1)
[2024-01-24T17:39:02+0100] [ALPM] upgraded python-toolz (0.12.0-3 -> 0.12.1-1)
[2024-01-24T17:39:02+0100] [ALPM] upgraded virtiofsd (1.9.0-2 -> 1.10.1-1)
[2024-01-24T17:39:02+0100] [ALPM] upgraded welcome (24-1 -> 24.2-1)
[2024-01-24T17:39:02+0100] [ALPM] transaction completed

As an addition: the more I’m “playing” with it - the more puzzling it becomes to pinpoint certain pattern.

Tried to create a new fresh Debian-based VM. Went through all graphical install process without any problems whatsoever. Afterwards, right at the end when it finished installation and made final reboot - VM crashed again. This time during systemd booting stage.

ajgringo619 · January 25, 2024, 8:56pm

This is very strange. My 01-21 update included libvirtd, same version as yours. I’m also running the same linux-lts version. The only major difference is that my VMs are not using spice-vdagent for video (GPU pass-through instead).

If I have time later today, I’ll try installing another distro with spice.

ajgringo619 · January 25, 2024, 9:31pm

OK, just installed EOS w/KDE - no issues.

firefly · January 25, 2024, 9:44pm

Thanks for double-checking!
Can you please also share your config XML? (if it’s not all default)

In any case - looks like it’s just my particular system/setup not playing well with latest qemu. Will tinker with it a bit more and then just try my luck with virtualbox or vmware.

ajgringo619 · January 25, 2024, 10:02pm

Here you go:

<domain type='kvm'>
  <name>eos-kde</name>
  <uuid>ac230f09-8a46-4d69-a334-b561638f22ec</uuid>
  <title>EndeavourOS (KDE Plasma)</title>
  <description>No GPU pass-through</description>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://archlinux.org/archlinux/rolling"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-8.2'>hvm</type>
    <firmware>
      <feature enabled='no' name='enrolled-keys'/>
      <feature enabled='yes' name='secure-boot'/>
    </firmware>
    <loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd</loader>
    <nvram template='/usr/share/edk2/x64/OVMF_VARS.4m.fd'>/var/lib/libvirt/qemu/nvram/eos-kde_VARS.fd</nvram>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
    <smm state='on'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='2' dies='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' discard='unmap'/>
      <source file='/media/vms/kvm-images/eos-kde.qcow2'/>
      <target dev='sdb' bus='scsi'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x1a'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x1b'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x1c'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x1d'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:89:e3:5b'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'/>
    </tpm>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <audio id='1' type='spice'/>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <watchdog model='itco' action='reset'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </rng>
  </devices>
</domain>

firefly · January 26, 2024, 5:04pm

Just a small update:
created new vm with basically same settings (the only difference - by default it picked bios system instead of efi, so had to switch it) - no luck. Tried also Gnome Boxes to eliminate possible problems with Virt Manager, since it too uses qemu/libvirt under the hood - same vm crashes during boot.

At this point I’m giving up, cause I’m out of ideas. Granted, my 10y.o. system is quite old but from my understanding, if it is just a hardware incompatibility, I wouldn’t have been able to install guest or use it in a live mode at all

ajgringo619 · January 26, 2024, 5:15pm

If it was a hardware compatibility issue, I would have expected problems to show up much earlier. Let me know if VB/VMware work.

firefly · January 26, 2024, 8:02pm

So… problem solved on its own. Kind of.
Turned out orc was the culprit. How on earth it managed to “fry” all my VMs is still a mystery, but I’ve been doing frequent updates lately, followed by tests, and after the last one when only orc & protobuf were updated, all VMs started to boot as usual.

I guess the main lesson is: update system once every 2 weeks and not on a weekly basis

Thanks a lot for your time and your willingness to help - I really appreciate it

ajgringo619 · January 26, 2024, 8:28pm

Glad it’s working now, although I’m a little confused about protobuf. Mine hasn’t been updated since December; full update was run this morning (orc was updated).

firefly · January 26, 2024, 8:48pm

Sorry for the confusion - I only mentioned this package in the context of my latest update itself.
orc somehow messed whole thing up.
Some additional information I was able to track, in case you’re interested:

ajgringo619 · January 26, 2024, 9:00pm

That seems directly related to the 6.8 kernel.

Stagger_Lee · January 26, 2024, 9:03pm

Probably this bug (which was closed earlier today):

I’m guessing you’re running an older Intel CPU? This bug was that the AVX backend was requiring AVX2, which was introduced around 10 years ago and so was not present in older CPUs.

firefly · January 26, 2024, 9:18pm

Oh, thanks for shedding some light on this!

Indeed - it’s Core i5-3230M, which was introduced back in 2013

Stagger_Lee · January 26, 2024, 9:20pm

Yep, that’s an Ivy Bridge family CPU. AVX2 was introduced in the later Haswell processors.

system · January 28, 2024, 9:20pm

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.