Did you add a new password or change an existing one?
If you added a password to a new slot, unlocking will take longer because the unlocking process cycles through all your passphrase slots until it finds a match.
Using slot0 will provide the fastest unlocking time.
Check your slots with:
sudo cryptsetup luksDump <yourLuksDevice>
[Edit]
Here’s some information about manipulating luks keys.