Not booting after update, hanging on "create volatile files and directories"

After many attempts I managed to boot back after removing the nvidia drivers, previous attempts were getting stuck in a black screen, what can I do now that I’m booted to diagnose the problem?

using nvidia-inst again I noticed the following error in the post-transaction hooks:

(5/8) Install DKMS modules
==> dkms install --no-depmod nvidia/535.86.05 -k 6.4.6-arch1-1
Error! Bad return status for module build on kernel: 6.4.6-arch1-1 (x86_64)
Consult /var/lib/dkms/nvidia/535.86.05/build/make.log for more information.
==> WARNING: `dkms install --no-depmod nvidia/535.86.05 -k 6.4.6-arch1-1' exited 10
(6/8) Reloading system bus configuration...
(7/8) eos-nvidia-fix.hook

the log from /var/lib/dkms/nvidia/535.86.05/build/make.log is here: http://ix.io/4ByA

as far as I can see these are the snippets of interest:
in the log file I can see these snippets:

DKMS make.log for nvidia-535.86.05 for kernel 6.4.6-arch1-1 (x86_64)
mié 26 jul 2023 19:32:43 CEST
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: gcc (GCC) 13.1.1 20230714
  You are using:           cc (GCC) 13.1.1 20230714
  SYMLINK /var/lib/dkms/nvidia/535.86.05/build/nvidia/nv-kernel.o
  SYMLINK /var/lib/dkms/nvidia/535.86.05/build/nvidia-modeset/nv-modeset-kernel.o
  CC [M]  /var/lib/dkms/nvidia/535.86.05/build/nvidia/libspdm_ec.o
/var/lib/dkms/nvidia/535.86.05/build/nvidia/nv-mmap.c:303:5: warning: conflicting types for ‘nv_encode_caching’ due to enum/integer mismatch; have ‘int(pgprot_t *, NvU32,  nv_memory_type_t)’ {aka ‘int(struct pgprot *, unsigned int,  nv_memory_type_t)’} [-Wenum-int-mismatch]
  303 | int nv_encode_caching(
      |     ^~~~~~~~~~~~~~~~~
In file included from /var/lib/dkms/nvidia/535.86.05/build/common/inc/nv-linux.h:1767,
                 from /var/lib/dkms/nvidia/535.86.05/build/nvidia/nv-mmap.c:27:
/var/lib/dkms/nvidia/535.86.05/build/common/inc/nv-proto.h:46:13: note: previous declaration of ‘nv_encode_caching’ with type ‘int(pgprot_t *, NvU32,  NvU32)’ {aka ‘int(struct pgprot *, unsigned int,  unsigned int)’}
   46 | int         nv_encode_caching           (pgprot_t *, NvU32, NvU32);
      |             ^~~~~~~~~~~~~~~~~
  CC [M]  /var/lib/dkms/nvidia/535.86.05/build/nvidia-uvm/uvm_migrate_pageable.o
*** stack smashing detected ***: terminated
In file included from ./include/linux/quota.h:48,
                 from ./include/linux/fs.h:269,
                 from ./include/linux/huge_mm.h:8,
                 from ./include/linux/mm.h:978,
                 from /var/lib/dkms/nvidia/535.86.05/build/common/inc/nv-pgprot.h:30,
                 from /var/lib/dkms/nvidia/535.86.05/build/common/inc/nv-linux.h:33,
                 from /var/lib/dkms/nvidia/535.86.05/build/nvidia-uvm/uvm_linux.h:40,
                 from /var/lib/dkms/nvidia/535.86.05/build/nvidia-uvm/uvm_extern_decl.h:27,
                 from /var/lib/dkms/nvidia/535.86.05/build/nvidia-uvm/uvm_global.h:28,
                 from /var/lib/dkms/nvidia/535.86.05/build/nvidia-uvm/uvm_hopper_mmu.c:35:
./include/linux/projid.h: In function ‘projid_eq’:
./include/linux/projid.h:38:9: internal compiler error: Aborted
   38 |         return __kprojid_val(left) == __kprojid_val(right);
      |         ^~~~~~
  CC [M]  /var/lib/dkms/nvidia/535.86.05/build/nvidia-uvm/uvm_map_external.o
0x1901918 internal_error(char const*, ...)
	???:0
0x7efcee821a07 raise
	???:0
0x7efcee80a537 abort
	???:0
0x7efcee902d0a __fortify_fail
	???:0
0x7efcee902ce5 __stack_chk_fail
	???:0
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <https://bugs.archlinux.org/> for instructions.
make[2]: *** [scripts/Makefile.build:252: /var/lib/dkms/nvidia/535.86.05/build/nvidia-uvm/uvm_hopper_mmu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [Makefile:2032: /var/lib/dkms/nvidia/535.86.05/build] Error 2
make: *** [Makefile:82: modules] Error 2

I have no idea how to solve this, I’m assuming nvidia-inst is trying to build a bugged version of dkms? how can I prevent that? what are the next steps?

Could be that your system just needs updating before running nvidia-inst:

sudo pacman -Syu

Already made sure, so unfortunately it’s not that

Interesting. I cannot test the nvidia-dkms driver currently since I’m not at my Nvidia box. Hopefully someone with nvidia-dkms driver can chime in and check if the latest driver is bugging with dkms.