@Richard Weaver Only you have enabled the "Deployment slot setting" the application setting should only be configured only for that slot. If you perform the swap operation then Deployment slot setting enabled application setting should not be swapped. I have tested the same at my end and couldn't observe the same behavior as you have observed.
I will suggest you to perform the deployment to the slot and first verify whether there are changes to your application setting on that slot as the setting might be updated/changed before you perform the swap operation and this could be one of reason for your behavior.
If you still observe the behavior then please share more details how you have performed the deployment to the slot and whether you are updating the configuration or not during deployment.