It is exactly what @marknn is saying.
When /boot is encrypted, the luks decryption is done by grub which doesn’t have any access to hardware acceleration. When /boot(or really kernel/initramfs) isn’t encrypted then the decryption will be much faster.
The installer uses both key files and password-based keys. The password is used by grub and the keyfiles are used by the initramsfs so you don’t need to put your password in twice.
You can certainly convert your install to use an unencrypted /boot if that is your preference. If you do that, you should remove the keyfiles from your main luks volume.
That being said, there are some security implications to having a decrypted /boot. Mainly that an attacker can access your initramfs which contains information about your system.