Just bought a motherboard that supports multiple gpu. I’m using an AMD rx580 connected to my monitors, all fine. Connected an nvidia rtx3060 to second slot, it is detected in lspci, now I need cuda support to run an llm on it. What is the best approach today for this card/setup? Should I just yay nvidia-inst ?
No harm in running nvidia-inst in test mode to see what it suggests.
nvidia-inst -t
With respect to CUDA and such, it may depend on how you’re running the LLM. Some tools will depend on CUDA (CUDA specific versions etc), others may only optionally depend on CUDA and require you to have it installed already.