Found what was wrong.
The configuration data stored has a limit of 1KB (1024bytes). And the OAuth token I was using was larger than that.
Reduce the amount of configuration data stored - All data entered in the Admin credentials section of the provisioning tab is stored in the same place as the SAML certificate. While it may not be possible to reduce the length of all of this data, some optional configuration fields like the Notification Email can be removed.