i do think the the main issue is the nouveau drivber for you related to your GPU Model.
I have a 3060 here and if i run this one on nouveau and vulkan it is running at big speeds fully usable even in Games.
The way to go for yor issue is (on Arch/based) currently to use the legacy driver from AUR nvidia-580xx-dkms nvidia-580xx-utils and indeed remove vulkan-nouveau xf86-video-nouveau
what you may hit is something like this too:
but if i were you i would first switch to 580 legacy drivers and see what happen.