It seems like I just solved this issue via using different DisplayPort. It was sort of coincidence, because I needed longer cable to move my secondary monitor to a different position on my desk and since I switched it, the issue disappeared. I don’t really know if it was a faulty cable, because on Windows it worked just fine and the version of the DisplayPort itself shouldn’t be an issue as well, because it was 1.2, which is well off for 144Hz 1920x1080 monitor. However this new one is 1.4 and that’s also what my main monitor (the other one) needs, so there may be a bug in which drivers think both monitors need cables with the same throughput. At least that would explain why the workaround I mentioned in the original post seemed to work.
I can’t really explain it otherwise, so for the future readers - try switching to different cables or maybe even try to keep the same version of DisplayPort across your multi-monitor setup.