Connection to JSON File via AzureBlob Storage via Dataflow doesn't work but through pipeline the dataset connection is working.

Question

I wanted to get your thoughts on error below. I am unable to get it working when using data flow module of Azure Data factory which is required to split the output into multiple files. This is a connection error. The error persists even if I use a simple csv file as a source or destination.

Dataflow

Connection failed
Spark job failed: { "text/plain": "{\"runId\":\"c1adb765-bdf0-4a61-84cb-73a1a6195b2f\",\"sessionId\":\"be254554-affc-4bdf-ba30-e9f9b967a626\",\"status\":\"Failed\",\"payload\":{\"statusCode\":400,\"shortMessage\":\"com.microsoft.dataflow.broker.InvalidOperationException: ServicePrincipal and MI auth are not supported if blob storage kind is Storage (general purpose v1)\",\"detailedMessage\":\"Failure 2023-03-22 07:52:34.974 failed DebugManager.processJob, run=c1adb765-bdf0-4a61-84cb-73a1a6195b2f, errorMessage=com.microsoft.dataflow.broker.InvalidOperationException: ServicePrincipal and MI auth are not supported if blob storage kind is Storage (general purpose v1)\"}}\n" } - RunId: c1adb765-bdf0-4a61-84cb-73a1a6195b2f

The same connection is working without issue when I use the ingest Pipeline of data factory.

PIPELINE

Based on the error message it says It does not support using a ServicePrincipal and MI auth on a general purpose v1 storage. But I am using a v2 Blob Storage.

I am using the default Azure Integration Runtime that comes with ADF. Any advise would be appreciated. Thanks.

Answer 1

Hi @DANCEL Christopher John ,

Welcome to Microsoft Q&A forum and thanks for reaching out here.

As per your explanation my understanding is that the same error message persists even if you are trying to connect to any other file type (i.e., non-Json file types) in the same storage account. This confirms that the issue is in establishing a success connection to the storage account itself from Mapping data flow. And I believe you are using either Service principal or MI authentication which is why you are receiving this error.

I would recommend reading through this troubleshooting guide to understand the actual root cause and possible resolution to overcome it.

Account Storage type (general purpose v1) doesn't support service principal and MI authentication

Symptoms

In data flows, if you use Azure Blob Storage (general purpose v1) with the service principal or MI authentication, you may encounter the following error message:

com.microsoft.dataflow.broker.InvalidOperationException: ServicePrincipal and MI auth are not supported if blob storage kind is Storage (general purpose v1)

Cause

When you use the Azure Blob linked service in data flows, the managed identity or service principal authentication is not supported when the account kind is empty or "Storage". This situation is shown in Image 1 and Image 2 below.

Image 1: The account kind in the Azure Blob Storage linked service

Image 2: Storage account page

Recommendation

To solve this issue, refer to the following recommendations:

• If the storage account kind is None in the Azure Blob linked service, specify the proper account kind, and refer to Image 3 shown below to accomplish it. Furthermore, refer to Image 2 to get the storage account kind, and check and confirm the account kind is not Storage (general purpose v1). Image 3: Specify the storage account kind in the Azure Blob Storage linked service
• If the account kind is Storage (general purpose v1), upgrade your storage account to the general purpose v2 or choose a different authentication (Use either account key or SAS token for the Azure Blob linked service). Image 4: Upgrade the storage account to general purpose v2

Hope this info helps.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.