High CPU usage, high VIRT, gameplay suffers

In htop I see that /usr/lib/Xorg /tmp/serverauth is peaking up to 19% of CPU but mostly being in 2-6% range. It also uses 24.2G of VIRT.

I got 16G RAM but only 8GB swap. Could it be I had to make 32G Swap and this wouldn’t be an issue?
How would I solve this? I notice gaming performance is subpar and I highly suspect that this /tmp/serverauth thingy using CPU and VIRT is an issue.