Several of the system tray icons in Xfce do not scale correctly. Notification and Pulseaudio both suddenly jump to full size icons if the panel height is more than 28 pixels, looking very weird compared to the other systray items that scale correctly.
I found a three year old post on the Xfce forums that just said “edit the CSS”. Which seems like the wrong kind of answer in the long run. Admittedly three years ago is basically just one release version ago, in Xfce terms, but…
So there is the panel preferences, appearance that has Icons …adjust size automatically. Then there is notification area that has maximum icon size (px) Also square icons or not.
Have you tried those settings?