Figured this out. Settings with _EXTENSION_VERSIONS suffix are by default not swappable. ie: Even if the "Deployment slot setting" check box is cleared, this setting still won't swap without the app setting WEBSITE_OVERRIDE_STICKY_EXTENSION_VERSIONS set to 0 or false. The confusing part about all of this is that if the "Deployment slot setting" check box is checked, the aforementioned app setting won't have any affect. Therefore, if you are using an ARM template to set an _EXTENSION_VERSIONS setting, you have to explicitly set the "Deployment slot setting" to false in the template. Something like:
{
"name": "ApplicationInsightsAgent_EXTENSION_VERSION",
"value": "~2",
"slotSetting": false
}
Or clear the "Deployment slot setting" check box in the portal gui.
Hopefully this helps somebody else out that was as confused as I was.
Cheers.