I finally got my case for my new machine and I was able to switch from the 1070 to the 4080 GPU as it finally fits inside the case.
Everything seems to work just fine after I booted into EOS but I still wanted to ask just in case if I have to remove or re-install something, e.g. drivers? Using the nvidia-inst cli it seems that the driver/series is the same for 1070 and 4080.