Error updating projection in ADF Data flow

Question

I'm getting a strange error when trying to update a projection inside a data flow. I have an Azure Sql Server data set - testing the connection outside of the debug session inside the data flow works fine. This is the error:

Spark job failed: { "text/plain": "{"runId":"e576d30d-8fa2-4e71-a924-4cbcd22b60b7","sessionId":"581b4c48-afd2-4b3f-9e40-1de54e7b201b","status":"Failed","payload":{"statusCode":400,"shortMessage":"java.lang.Exception: DataFlowManagerClient.getPayload: 400","detailedMessage":"Failure 2024-06-18 10:51:02.633 failed DebugManager.processJob, run=e576d30d-8fa2-4e71-a924-4cbcd22b60b7, errorMessage=java.lang.Exception: DataFlowManagerClient.getPayload: 400"}}\n" } - RunId: e576d30d-8fa2-4e71-a924-4cbcd22b60b7

Any help would be much appreciated, thanks

Answer 1

Hello Rich-EA,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you are having spark job failed with error code 400 during updating projection in ADF Data flow.

Solution

The error you're encountering suggests a problem with the data flow execution, specifically with the interaction between the DataFlowManagerClient and the payload retrieval. The 400 status code indicates a bad request error. You will need to check through the followings:

Ensure that the Azure SQL Server dataset configuration is correct. Especially, Connection strings

Authentication details, and Schema and table names.

Inspect the data flow where the error occurs, verify that all transformations and projections are correctly configured and check for any recent changes that might have introduced the error.

Projection issues can sometimes be due to mismatched schemas. So, ensure that the schema of your source dataset matches the expected schema in the data flow, and all columns are correctly mapped and there are no missing or extra columns.

Since you mentioned that testing the connection outside of the debug session works fine, try previewing the data directly within the data flow to ensure the connection and data retrieval are functioning as expected by using the "Data Preview" feature to validate the dataset within the data flow.

Lasty, check your debug setting and validate the integration runtime used in debug mode also, ensure the necessary resources are allocated for the debug session.

References

For more reading and steps to resolve the issue, kindly use the following links as listed in the additional resources:

• Troubleshoot mapping data flows - Azure Data Factory
• Troubleshoot connector and format issues in mapping data flows - Azure Data Factory
• General troubleshooting - Microsoft Fabric
• Troubleshoot pipeline orchestration and triggers in Azure Data Factory - Azure Data Factory
• General Troubleshooting - Azure Data Factory & Azure Synapse

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam

Answer 2

Hi @Rich-EA

Thanks for the question and using MS Q&A platform.

You mentioned that you are getting an error when trying to update a projection inside a data flow. Based on the error message you provided, it seems like there is an issue with the data flow job.

One possible cause of this issue could be related to the connection string or key vault used in the Integration Runtime. As per my understanding, you were able to successfully create a dataset and test the connection using the Integration Runtime, and you were also able to preview the data. However, I would like to confirm whether you used a connection string in the Integration Runtime or if you used a key vault while testing the connection. One possibility is that you may have used a connection string while testing the Integration Runtime connection and then changed it to AKV before testing the connection in the data flow source.

To resolve this issue, I recommend recreating a new linked service and checking if you are still experiencing the same behavior. Some users have reported that they were able to resolve the issue after recreating the linked service.

Here is another thread where a similar issue has been reported - DataFlowManagerClient.getPayload: 400. You may find it helpful to refer to this thread for additional information on troubleshooting this issue.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.