Which xlibre packages do you have installed? I only have the following on my Xfce system:
$ yay -Qs xlibre
local/xlibre-input-libinput 1.5.1.0-1 (xlibre-drivers)
XLibre fork of the generic input driver for the X.Org server based on libinput
local/xlibre-xserver 25.0.0.16-1 (xlibre)
XLibre fork of X.Org X server
local/xlibre-xserver-common 25.0.0.16-1 (xlibre)
XLibre fork of X.Org Xorg server common files
I also run my system with Xorg as non-root, using startxfce4 from the console (no display manager).