[Data Factory] Schema Drift and Unpivot transformation -> scala.MatchError

Question

Hi,

I try to use "Unpivot" transformation on a Source that don't have schema pre-defined (Schema Drift is activated).
The "unpivot" transformation generates the following error:

Error: DF-SYS-01 at : scala.MatchError: 1 (of class java.lang.Integer) - RunId: 700ee826-2737-4a4a-b9bb-daf65b52fb31

Could you help me understand the meaning of this error and how to fix it?
Is "unpivot" unable to work with schema drift?

For information, I put more detail about the other component below.

The input file looks like this:

The file format will change regularly, a new column will be added every month, so I want to use Schema Drift.

The source looks like this:
As you can see there is no schema prepared.

My select transformation looks like this:
I retrieve the product code by using "name" and I retrieve all columns prefixed by 20 (ex: 2020-01)

The unpivot is configured as follow, using "byName" to retrieve the product code column:

I create a column year_month which will contain the column headers that have been unpivoted.

I create a column allocation which will contain the values that have been unpivoted.

If you have any other suggestion I'm happy to hear too.

Thank you

Answer 1

This looks more complicated that it needs to be. You can do something as simple as

yearly derive(
Product_Code = byName("Product_Code")
) ~> yearlyDerived

yearlyDerived unpivot(output(
year_month as string,
allocation as string
),
ungroupBy(Product_Code),
lateral: false,
ignoreNullPivots: false) ~> unpivotYearly

---

Unpivots do not allow expressions or assignments in ungroupBy expressions. The validation error will be made more user friendly