ADF Pipeline failing with StatusCode : DFExecutorUserError

Question

I have create a data flow in ADF and I am running it via a pipeline. The pipeline is validated and there are no errors. I am also able to preview all the data in the sink in debug team. But on executing the pipeline I am getting the following error :

Operation on target Execute GeneratePlacementReport Pipeline failed: Operation on target PlacementReport_Dataflow_1 failed: {"StatusCode":"DFExecutorUserError","Message":"at Sink 'PlacementReportSink': java.util.NoSuchElementException: None.get","Details":"at Sink 'PlacementReportSink': java.util.NoSuchElementException: None.get"}

Answer 1

Hi Microsoft Q&A Platform,

I was able to resolve this issue.

I followed the following steps :

1. Reimported the projections from the source.
2. Checked if all my datatypes in the sink are the same as the destination database.

I had an issue with the schema mismatch between sink and the database.

Follow the given best practices to avoid this issue :

1. Clean up duplicate columns as often as possible after performing certain transforms
2. For cleaning up use select statements.
3. Import your schema projection at the source level every time there are major/minor schema changes.
4. If the database is pre-created make sure it matches the sink schema.