So, this is weird:
I did a clean reinstall, booted up on the LTS kernel (5.10) and then installed the nvidia 470 drivers, and not the 495 that I had installed before through dkms, and I still have the problem which makes no sense to me honestly. Next step is probably just install KDE and hope this is somehow a Gnome issue.