Booting takes long time issue

General system Kernel, boot, graphics & hardware
1

Hi everyone

I’m new on the Forum as a member, but I’m a silent reader since some time already and because of the amazing community here, I decided to give it a try to reach out to seeking for help.

I’m not using EOS (yet? ;)) using Arch with the CachyOS Kernel since about 3 months, so I’m still pretty new to the game in General. Since around 1 - 1,5 months I am having the issue, that my System takes way longer to boot than it did before. Right now I’m about 35s for everything, where it was half as much before:

Startup finished in 13.972s (firmware) + 7.351s (loader) + 3.282s (kernel) + 11.021s (userspace) = 35.628s graphical.target reached after 11.021s in userspace.

according to systemd-blame and critical chain, it is caused by systemd-journald.service:

9.873s systemd-journald.service
 856ms \x2esnapshots.mount
 748ms NetworkManager.service
 313ms systemd-modules-load.service
 270ms dev-nvme1n1p2.device
 132ms upower.service
 114ms systemd-udev-trigger.service
 104ms user@1000.service

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

systemd-journald.service +9.873s
└─systemd-journald.socket @190ms
  └─system.slice @168ms
    └─-.slice @168ms

But according to journalctl it looks more likely to be a problem with my USB Hub:

[    4.758037] usb 1-6: New USB device found, idVendor=0b05, idProduct=1a52, bcdDevice= 0.05
[    4.758041] usb 1-6: New USB device strings: Mfr=3, Product=1, SerialNumber=0
[    4.758043] usb 1-6: Product: USB Audio
[    4.758045] usb 1-6: Manufacturer: Generic
[   10.348140] hid-generic 0003:0B05:1A52.0008: hiddev98,hidraw7: USB HID v1.11 Device [Generic USB Audio] on usb-0000:0a:00.0-6/input7
[   10.524319] usb 1-7: new full-speed USB device number 3 using xhci_hcd
[   10.887119] usb 1-7: config 1 has an invalid interface number: 2 but max is 1
[   10.887125] usb 1-7: config 1 has no interface number 1
[   10.898749] usb 1-7: New USB device found, idVendor=0b05, idProduct=19af, bcdDevice= 1.00

Inxi:

System:
  Kernel: 6.10.4-1-cachyos-eevdf arch: x86_64 bits: 64 compiler: gcc v: 14.2.1
  Desktop: KDE Plasma v: 6.1.4 tk: Qt v: N/A wm: kwin_wayland dm: SDDM
    Distro: Arch Linux
Machine:
  Type: Desktop System: ASUS product: N/A v: N/A serial: <superuser required>
  Mobo: ASUSTeK model: ROG STRIX B650E-F GAMING WIFI v: Rev 1.xx
    serial: <superuser required> part-nu: SKU UEFI: American Megatrends v: 3024
    date: 08/02/2024
CPU:
  Info: 8-core model: AMD Ryzen 7 7700X bits: 64 type: MT MCP arch: Zen 4
    rev: 2 cache: L1: 512 KiB L2: 8 MiB L3: 32 MiB
  Speed (MHz): avg: 1637 high: 5338 min/max: 545/5573 boost: enabled cores:
    1: 545 2: 545 3: 545 4: 5338 5: 4627 6: 5337 7: 4356 8: 545 9: 545 10: 545
    11: 545 12: 545 13: 545 14: 545 15: 545 16: 545 bogomips: 143723
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Graphics:
  Device-1: NVIDIA GA104 [GeForce RTX 3070 Lite Hash Rate] vendor: ASUSTeK
    driver: nvidia v: 560.31.02 arch: Ampere pcie: speed: 16 GT/s lanes: 16
    ports: active: none off: DP-2,HDMI-A-2 empty: DP-3,DP-4,HDMI-A-3
    bus-ID: 01:00.0 chip-ID: 10de:2488
  Device-2: AMD Raphael vendor: ASUSTeK driver: amdgpu v: kernel
    arch: RDNA-2 pcie: speed: 16 GT/s lanes: 16 ports: active: none
    empty: DP-1,HDMI-A-1,Writeback-1 bus-ID: 0c:00.0 chip-ID: 1002:164e
    temp: 52.0 C
  Display: wayland server: X.org v: 1.21.1.13 with: Xwayland v: 24.1.2
    compositor: kwin_wayland driver: X: loaded: modesetting,nvidia
    alternate: fbdev,nouveau,nv,vesa dri: radeonsi gpu: nvidia,amdgpu
    d-rect: 4480x2520 display-ID: 0
  Monitor-1: DP-2 pos: top-right res: 2560x1440 size: N/A
  Monitor-2: HDMI-A-2 pos: bottom-l res: 1920x1080 size: N/A
  API: EGL v: 1.5 platforms: device: 0 drv: nvidia device: 1 drv: radeonsi
    device: 3 drv: swrast gbm: drv: radeonsi surfaceless: drv: nvidia wayland:
    drv: nvidia x11: drv: nvidia inactive: device-2
  API: OpenGL v: 4.6.0 compat-v: 4.5 vendor: nvidia mesa v: 560.31.02
    glx-v: 1.4 direct-render: yes renderer: NVIDIA GeForce RTX 3070/PCIe/SSE2
    display-ID: :1.0
  API: Vulkan v: 1.3.285 surfaces: xcb,xlib,wayland device: 0
    type: discrete-gpu driver: nvidia device-ID: 10de:2488
Audio:
  Device-1: NVIDIA GA104 High Definition Audio vendor: ASUSTeK
    driver: snd_hda_intel v: kernel pcie: speed: 16 GT/s lanes: 16
    bus-ID: 01:00.1 chip-ID: 10de:228b
  Device-2: AMD Rembrandt Radeon High Definition Audio vendor: ASUSTeK
    driver: snd_hda_intel v: kernel pcie: speed: 16 GT/s lanes: 16
    bus-ID: 0c:00.1 chip-ID: 1002:1640
  Device-3: ASUSTek USB Audio driver: hid-generic,snd-usb-audio,usbhid
    type: USB rev: 2.0 speed: 480 Mb/s lanes: 1 bus-ID: 1-6:2 chip-ID: 0b05:1a52
  Device-4: Razer USA BlackShark V2 Pro 2.4
    driver: hid-generic,snd-usb-audio,usbhid type: USB rev: 1.1 speed: 12 Mb/s
    lanes: 1 bus-ID: 5-2.4:5 chip-ID: 1532:0555
  API: ALSA v: k6.10.4-1-cachyos-eevdf status: kernel-api
  Server-1: PipeWire v: 1.2.2 status: active with: 1: pipewire-pulse
    status: active 2: wireplumber status: active 3: pipewire-alsa type: plugin
    4: pw-jack type: plugin
Network:
  Device-1: Intel Ethernet I225-V vendor: ASUSTeK driver: igc v: kernel pcie:
    speed: 5 GT/s lanes: 1 port: N/A bus-ID: 08:00.0 chip-ID: 8086:15f3
  IF: eno1 state: up speed: 1000 Mbps duplex: full mac: <filter>
  Device-2: MEDIATEK MT7922 802.11ax PCI Express Wireless Network Adapter
    vendor: Foxconn driver: mt7921e v: kernel pcie: speed: 5 GT/s lanes: 1
    bus-ID: 09:00.0 chip-ID: 14c3:0616
  IF: wlan0 state: down mac: <filter>
Bluetooth:
  Device-1: Foxconn / Hon Hai Wireless_Device driver: btusb v: 0.8 type: USB
    rev: 2.1 speed: 480 Mb/s lanes: 1 bus-ID: 1-11:7 chip-ID: 0489:e0e2
  Report: rfkill ID: hci0 rfk-id: 1 state: down bt-service: disabled
    rfk-block: hardware: no software: no address: see --recommends
Drives:
  Local Storage: total: 2.75 TiB used: 231.33 GiB (8.2%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: MZVL21T0HCLR-00B00
    size: 953.87 GiB speed: 63.2 Gb/s lanes: 4 serial: <filter> temp: 43.9 C
  ID-2: /dev/nvme1n1 vendor: Western Digital model: WD BLACK SN850X 1000GB
    size: 931.51 GiB speed: 63.2 Gb/s lanes: 4 serial: <filter> temp: 35.9 C
  ID-3: /dev/sda vendor: Western Digital model: WDS100T2B0A-00SM50
    size: 931.51 GiB speed: 6.0 Gb/s serial: <filter>
Partition:
  ID-1: / size: 930.51 GiB used: 230.87 GiB (24.8%) fs: btrfs
    dev: /dev/nvme1n1p2
  ID-2: /boot size: 1022 MiB used: 470.2 MiB (46.0%) fs: vfat
    dev: /dev/nvme1n1p1
  ID-3: /home size: 930.51 GiB used: 230.87 GiB (24.8%) fs: btrfs
    dev: /dev/nvme1n1p2
  ID-4: /var/log size: 930.51 GiB used: 230.87 GiB (24.8%) fs: btrfs
    dev: /dev/nvme1n1p2
Swap:
  Alert: No swap data was found.
Sensors:
  System Temperatures: cpu: 61.4 C mobo: 42.0 C gpu: amdgpu temp: 52.0 C
  Fan Speeds (rpm): N/A
Info:
  Memory: total: 32 GiB note: est. available: 30.49 GiB used: 4.47 GiB (14.7%)
  Processes: 370 Power: uptime: 21m wakeups: 0 Init: systemd v: 256
    default: graphical
  Packages: pm: pacman pkgs: 1233 Compilers: clang: 18.1.8 gcc: 14.2.1
    Shell: Bash v: 5.2.32 running-in: kitty inxi: 3.3.35```

What I tried so far

  • tried to boot with no USB Devices at all, to the same issue
  • tried a different Kernel (LTS and Zen), same issue persists
  • tried with usbcore.autosuspend=-1, not better
  • tried to power cycle, since for other user this resolved their issue - with no change unfortunately
  • asked in the official Arch Linux Forum, where I got told, that they blame my “g4m0rz” stuff … where it wasn’t an issue before …

Since it happens when nothing is connected via USB, I strongly assume that it is an issue with an USB Hub … but I’m at a point, where I don’t know how to proceed further to narrow down the issue and maybe (hopefully) find a solution.

Any help or tips are very appreciated, thank you in advance for having a look into it and taking the time so crawl through it. If something’s missing or Logs are needed, feel free to let me know and I’ll provide them.

2

This particular service seems to be taking some excessive amount of time.

systemd-journald.service collects and stores logging data on Linux systems using systemd.

If the system journal has grown very large over time, it can slow down the boot process as systemd-journald needs to process and load this data.

If the size of the journal is the issue here, you may want to look into deleting some of it and limiting its size. Have a look at point 4 and 5 at: https://wiki.archlinux.org/title/Systemd/Journal#

Also, this seems to be taking some time. If I am not mistaken, this time represents the duration of the system’s firmware/BIOS initialization process before control is handed over to the bootloader.
A long firmware time could indicate outdated firmware.

That is how far I can stretch myself. Others may want to chime in with more ideas.

3

Thank you for taking the time!

If the system journal has grown very large over time, it can slow down the boot process as systemd-journald needs to process and load this data.

Thought on that as well, but it also happens directly after the first reboot on a completely fresh installed system … impossible that i grew that big only from installing …

could the software be then the USB Hub initialization thingy which is visible in the journal?

1 Like
4

It very well might be. Unfortunately I have no knowledge about how peripherals may interact with the boot process and slow it down as in your case. Hopefully more well-versed users will soon join up.

1 Like
5

Just a long shot idea that the HDD might cause the delay.
Is it possible to test booting without the HDD?

Also, can you show the contents of file /etc/fstab?

6

You might try unplugging all USB devices except mouse and keyboard and try booting, as a test.

The peripheral I noted as taking the time in your log was a USB audio device. If unplugging the USB devices results in a faster boot, you might want to try to isolate the specific peripheral causing issues, and connect it to a different type of USB port (eg: USB 1/2/3), with the hope of a fix.

7

Is it possible to test booting without the HDD?

possible for sure, but will take me some time to grab out some other SSD

Also, can you show the contents of file /etc/fstab?

Sure, here you go:

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme1n1p2
UUID=<filter>	/         	btrfs     	rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/@	0 0

# /dev/nvme1n1p2
UUID=<filter>	/home     	btrfs     	rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=257,subvol=/@home	0 0

# /dev/nvme1n1p2
UUID=<filter>	/var/log  	btrfs     	rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=258,subvol=/@log	0 0

# /dev/nvme1n1p2
UUID=<filter>	/var/cache/pacman/pkg	btrfs     	rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=259,subvol=/@pkg	0 0

# /dev/nvme1n1p2
UUID=<filter>	/.snapshots	btrfs     	rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=260,subvol=/@.snapshots	0 0

# /dev/nvme1n1p1
UUID=<filter>      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro0 2
8

I already tried booting without anything, even no mouse/keyboard, but I can surely try to cross peripherals out, even though I can remember, that it’s not always the same device taking this long in the journal, but it’s always the same hub.

I’ll try, gather some logs and report back!

1 Like
9

@manuel
just installed the OS on a spare SSD I luckily had lying around and the error occurs there as well :cry:

@Bink
this is the journal from the last boot - as you can see, this time it’s again the xhci host controller and not the audio taking longer…

Aug 14 01:35:51 NALX kernel: xhci_hcd 0000:0a:00.0: xHCI Host Controller
Aug 14 01:35:51 NALX kernel: xhci_hcd 0000:0a:00.0: new USB bus registered, assigned bus number 2
Aug 14 01:35:51 NALX kernel: xhci_hcd 0000:0a:00.0: Host supports USB 3.2 Enhanced SuperSpeed
Aug 14 01:35:55 NALX kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.10
Aug 14 01:35:55 NALX kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug 14 01:35:55 NALX kernel: usb usb1: Product: xHCI Host Controller
Aug 14 01:35:55 NALX kernel: usb usb1: Manufacturer: Linux 6.10.4-1-cachyos-eevdf xhci-hcd
Aug 14 01:35:55 NALX kernel: usb usb1: SerialNumber: <filter>

I had a look into dmesg as well and there somehow usb-audio is there again. Will do a boot with no peripherials at all tomorrow and paste that log as well…

[   13.334378] usbcore: registered new interface driver snd-usb-audio
[   34.245664] usb 5-2.4: new full-speed USB device number 3 using xhci_hcd
[   34.360046] usb 5-2.4: New USB device found, idVendor=1532, idProduct=0555, bcdDevice= 0.01
[   34.360053] usb 5-2.4: New USB device strings: Mfr=0, Product=2, SerialNumber=3
10

In your BIOS, you might:

  • Check whether XHCI hand-off is enabled/disabled. I believe enabled is usually desired for top performance, but you might toggle this to see if impacts the issue.
  • Disable Legacy USB.
11

Legacy USB was already off, wasn’t sure about the XHCI hand-off so haven’t tried until now - but unfortunately it doesn’t look like it’s doing much

9.281s systemd-journald.service
 793ms \x2esnapshots.mount
 777ms NetworkManager.service
 315ms systemd-modules-load.service
 252ms dev-nvme1n1p2.device
 118ms user@1000.service
 105ms systemd-udev-trigger.service
 100ms upower.service
  82ms systemd-vconsole-setup.service
  72ms systemd-tmpfiles-setup-dev-early.service
  69ms udisks2.service
  63ms systemd-tmpfiles-clean.service
  57ms systemd-hostnamed.service
  41ms systemd-tmpfiles-setup.service
  39ms systemd-tmpfiles-setup-dev.service
  34ms systemd-userdbd.service
  31ms systemd-journal-flush.service
  29ms dbus-broker.service
  29ms polkit.service
  29ms systemd-logind.service
  28ms systemd-udevd.service
  25ms systemd-random-seed.service
  25ms wpa_supplicant.service
  24ms systemd-rfkill.service
  22ms systemd-timesyncd.service
  20ms user-runtime-dir@1000.service
  18ms iptables.service
  15ms systemd-sysctl.service
  15ms boot.mount
   9ms home.mount
   8ms var-cache-pacman-pkg.mount
   8ms modprobe@dm_mod.service
   8ms var-log.mount
   8ms dev-hugepages.mount
12

Not sure how you set up btrfs? Curious about this entry in fstab?

Edit: I’m no expert on btrfs but i don’t have this in my fstab. Are you keeping snapshots on another external usb drive?

UUID=<filter>	/.snapshots	btrfs     	rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=260,subvol=/@.snapshots	0 0

Edit: also don’t have this one either so obviously you a have set it up differently than the standard install process for btrfs.

UUID=<filter>	/var/cache/pacman/pkg	btrfs     	rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=259,subvol=/@pkg	0 0

Edit: I just noticed you aren’t using EOS yet. Anyway this is all i noticed wondering?

13

iirc the /.snapshots is created by timeshift, i didn’t create it manually. It’s on the same drive, since BTRFS snapshots to an external disk are not supported (by timeshift).

for the /var/cache/pacman/pkg this is most like to be installed by pacman-contrib, see https://wiki.archlinux.org/title/Pacman#Cleaning_the_package_cache

I’m not using EOS, because my Arch works, haven’t found a reason yet (don’t change, what’s not broken) … considered trying, to see how it behaves there, but if its HW related I don’t expect it to behave different - also because EOS is said to be identical, once the system is running!?

But I’m a newb myself still, so maybe I got some things here and there wrong

14

I don’t use timeshift on endeavour. The fstab just caught my eye because it has some different entries. I know Arch sets up btrfs differently and I’m not the guy to explain it as i don’t have a huge understanding of all things btrrfs. I just know that i use it on two desktops. One with Ryzen and AMD Gpu. The other is an Intel with Nvidia. Both work flawlessly but I am using btfs, btrfs-assistant, snapper-support and btrfsmaintenance along with grub-btrfs.

Edit: Both desktops boot instantly without any delay.

15

I see.
on the other side, when tested on the “fresh” system, there was nothing like timeshift and the like installed, just btrfs setup and Plasma and nothing else, and the error still happening … while interesting, that the /etc/fstab or the handling or btrfs volumes and subvolumes is different, I don’t think that this might cause the delays I’m having…

1 Like
16

Did you look at

systemd-analyze critical-chain

systemd-analyze blame

Edit: Also does sudo dmesg give any hints?

17

As stated above, critical-chain and blame show systemd-journald.service as the „long taking one“, in dmesg and journalctl Im inly seeing about 10 seconds of nothing between usb device/hub recognition…

18

https://wiki.archlinux.org/title/Improving_performance/Boot_process

19

On my system …

240ms systemd-journald.service

20

That would save me 10 seconds and I‘d be again at 13-15 seconds to boot up, as it was before… still wondering what changed, but I‘ll work through the wiki link you pasted … thank you!

1 Like