I try to use "Unpivot" transformation on a Source that don't have schema pre-defined (Schema Drift is activated).
The "unpivot" transformation generates the following error:
Error: DF-SYS-01 at : scala.MatchError: 1 (of class java.lang.Integer) - RunId: 700ee826-2737-4a4a-b9bb-daf65b52fb31
Could you help me understand the meaning of this error and how to fix it?
Is "unpivot" unable to work with schema drift?
For information, I put more detail about the other component below.
The input file looks like this:
The file format will change regularly, a new column will be added every month, so I want to use Schema Drift.
The source looks like this:
As you can see there is no schema prepared.
My select transformation looks like this:
I retrieve the product code by using "name" and I retrieve all columns prefixed by 20 (ex: 2020-01)
The unpivot is configured as follow, using "byName" to retrieve the product code column:
I create a column year_month which will contain the column headers that have been unpivoted.
I create a column allocation which will contain the values that have been unpivoted.
If you have any other suggestion I'm happy to hear too.