So, to close this out: the problem is asymmetric routing: this causes the pfsense firewall to fail to establish proper state as it witnesses only part of the TCP handshake.
The linked documentation explains the issue, how to manually set rules to fix it, as well as the easy solution (if like me you’re a home with a simple subnet structure and no need to have internal rules) which is to enable “Bypass firewall rules for traffic on the same interface” in System/Advanced/Firewall&NAT.