Unexplained Failure in Data Flow Flatten Activity Post-May 29th?

Question

Dear Microsoft Support,

I am writing to report a persistent issue with an Azure Data Factory data flow that has been failing since May 29th. This data flow was previously functioning correctly without any changes on my end. The failure occurs during the flatten activity where I attempt to unroll multiple (six) nested arrays from a JSON source.

Upon investigation, I found that the error seems to be triggered when attempting to unroll more than three arrays simultaneously. I can unroll up to three columns successfully, but any attempt to process additional arrays results in the aforementioned error.

Given that this data flow was operating without issues before May 29th, I suspect there might have been an underlying change or update on the platform side that is causing this unexpected behavior.

I have spent considerable time debugging this issue, trying various configurations and verifying the integrity of the input data. Unfortunately, I have been unable to resolve the problem, and it continues to hinder our data processing workflows.

Could you please investigate this issue on your end? Any insights or solutions you could provide would be greatly appreciated as this is critical to our operations.

Thank you for your assistance.

Garrett Locke

Answer 1

Hello Garrett Locke,

Welcome to the Microsoft Q&A forum.

The issue was due to a recent Spark version upgrade.

The new version included updates to the flatten transformation logic to improve performance and functionality.

These updates were not fully backward compatible with the previous version, leading to breaking changes.

PG is aware of this issue and is working on a fix.

To unblock you, could you please raise a support case so that a support engineer can assist you from the backend by reverting back the changes for your workspace.

In case you don't have a support plan, please let me know so that I can provide a one-time free support request to work on this case.

I am looking forward to hearing from you.