try it after getting your latencies setup, take a look at the piepwire documentation here
## Properties for the DSP configuration.
#default.clock.rate = 48000
#default.clock.allowed-rates = [ 48000 ]
#default.clock.quantum = 1024
default.clock.min-quantum = 16
#default.clock.max-quantum = 2048
#default.clock.quantum-limit = 8192
This stuff here is going to be primarily what youre looking to tweak to prevent that issue.