Did you install with grub or systemd-boot?
With grub the slow decrypt is well known. grub does not use any CPU-related acceleration functions while decrypting.
You can look at this older post
@2000 mentioned a way to increase the speed in decryption. But you have to know that you will lower the level of security of your encryption.