Ive thrown together a few pkgbuilds from some AUR pkgbuilds to make OBS work on Gnome3.38 wayland and fixed the pkgbuilds desktop file so it launches with wayland support on default. Ive also added the patches for enabling HEVC VAAPI support and re-enabled DMA-BUF on non intel GPUs for the current version of mutter.
Original AUR pkgbuilds
https://aur.archlinux.org/packages/obs-hevc-vaapi-git
https://aur.archlinux.org/packages/obs-studio-wayland/
https://aur.archlinux.org/packages/obs-xdg-portal-git
https://aur.archlinux.org/packages/mutter-performance
My edited PKGBUILDS Below
OBS-Way-hevc
https://gitlab.com/echoa/obs-way-hevc
obs-xdg-portal-git for capturing wayland windows/screen
https://gitlab.com/echoa/obs-xdg-portal-git
Patched Gnome Mutter with DMA-BUF
https://gitlab.com/echoa/mutter-perf-dma
Git Clone or Download as a zip and follow the build instructions
Enjoy