OK, I figured it out. It is the ZFS ARC cache. It defaults to 50% of available memory which appears to include swap. While it automatically scales down when applications demand memory, that doesn’t help when applications take action based on available memory.
I lowered the cache to a more reasonable number for my use case(5GiB) and the impact was instant.
---------------------------------
17.3 GiB
=================================
>> free -m
total used free shared buff/cache available
Mem: 64217 17398 45109 547 1708 45589
Swap: 16383 6759 9624
I am a bit curious to see if there is any notable decrease in performance since I no longer have ~40GiB of data in cache.