ADF randomly throw exception for long time Scope job

Question

Some Scope jobs were successfully submitted to Cosmos ADLA through ADF. The job continued running in the ADLA, but ADF failed with following exception:

Unable to get an access token - [GetTokenCredential] was called without a valid X509 certificate or service principal key.  Object was of type System.Collections.Generic.Dictionary`2[System.String,System.Object]

Similar ICM raised: Microsoft Internal Links Redacted by Caozhi Li.

Is there anyway to resolve it?.

Answer 1

Hi Jing Ou,

Thank you for reaching out, Microsoft QA. This issue has been resolved.

Starting December 21, transient GetTokenCredential authentication failures in Azure Data Factory caused long-running Scope jobs to retry and experience extended execution times. Although jobs were successfully submitted to ADLA and continued running, ADF failed during token refresh operations. The behavior was caused by a regression, combined with VM unresponsiveness and certificate retrieval issues in the Scope pools.

The Batch team reverted the changes that introduced the regression, and no job run failures have been observed since January 3. The issue is now mitigated. A small ADF deployment is planned to ensure the fix is consistently applied across environments. In rare cases, clusters that were already running at the time of the fix may require a restart to fully benefit from the resolution.

Please let us know if you observe any further issues.

Answer 2

The error message you are encountering, Unable to get an access token - [GetTokenCredential] was called without a valid X509 certificate or service principal key, typically indicates that there is an issue with the authentication setup in Azure Data Factory (ADF) when trying to access Azure Data Lake Analytics (ADLA).

To resolve this issue, you can take the following steps:

1. Check Service Principal Configuration: Ensure that the service principal you are using has the correct permissions and that it is properly configured with a valid X509 certificate or service principal key.
2. Validate Authentication Method: Confirm that the authentication method you are using in ADF is supported and correctly set up for accessing ADLA. If you are using a managed identity, ensure it is enabled and has the necessary permissions.
3. Retry the Operation: Sometimes, transient errors can occur. If you haven't already, try rerunning the job after a brief wait.
4. Review Azure Logs: Check the Azure logs for any additional error messages or warnings that might provide more context about the failure.
5. Consult Azure Support: If the issue persists, consider reaching out to Azure Support for further assistance, especially since you have an ICM (Incident Management) case raised regarding this issue.

By following these steps, you should be able to troubleshoot and potentially resolve the access token issue you're facing with ADF and ADLA.