Middle mouse scroll button sometimes causes opposite scrolling

Hello,

this annoyes me quite a lot, I scroll in terminal up and suddenly it jumps several lines down and then it again scrolls up on next wheel turn. Does anyone know what causes this? Bad driver? Bad settings?

Sometimes if you have a command that is outputting text, it will act like that.
Which terminal emulator?