How to Disable xorg on GPU?

Id like my GPU solely for running LLMs and thus how do I update xorg/x11 from using it? Thanks!! << q

λ neofetch
OS: EndeavourOS Linux x86_64 
Kernel: 6.8.9-arch1-2 
Uptime: 55 mins 
Packages: 1484 (pacman) 
Shell: zsh 5.9 
Resolution: 3440x1440 
DE: Xfce 4.18 
WM: Xfwm4 
WM Theme: Gorilla 
Theme: Arc-Darker [GTK2/3] 
Icons: kora [GTK2], Qogir [GTK3] 
Terminal: xfce4-terminal 
Terminal Font: Source Code Pro 14 
CPU: AMD Ryzen 5 3600 (12) @ 3.600GHz
GPU: NVIDIA GeForce GTX 960 

λ inxi -Ga                                                                                          12:39:28
Graphics:
  Device-1: NVIDIA GM206 [GeForce GTX 960] vendor: Micro-Star MSI
    driver: nvidia v: 550.78 alternate: nouveau,nvidia_drm non-free: 545.xx+
    status: current (as of 2024-04; EOL~2026-12-xx) arch: Maxwell code: GMxxx
    process: TSMC 28nm built: 2014-2019 pcie: gen: 3 speed: 8 GT/s lanes: 16
    bus-ID: 09:00.0 chip-ID: 10de:1401 class-ID: 0300

λ nvidia-smi                                                                                        12:59:55
Sat May 11 13:00:02 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.78                 Driver Version: 550.78         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 960         Off |   00000000:09:00.0  On |                  N/A |
| 36%   46C    P0             30W /  130W |     675MiB /   2048MiB |     18%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A       789      G   /usr/lib/Xorg                                 542MiB |
|    0   N/A  N/A      1016      G   xfwm4                                           1MiB |
|    0   N/A  N/A      1791      G   ...erProcess --variations-seed-version         56MiB |
|    0   N/A  N/A      1844      G   ...yOnDemand --variations-seed-version         43MiB |
|    0   N/A  N/A      1908      G   ...erProcess --variations-seed-version         11MiB |
|    0   N/A  N/A      2127      G   ...ures=SpareRendererForSitePerProcess          3MiB |
|    0   N/A  N/A     22831      G   /usr/lib/firefox/firefox                        2MiB |
+-----------------------------------------------------------------------------------------+