[Data Factory] Schema Drift and Unpivot transformation -> scala.MatchError

Pestre Remi 41 Reputation points


I try to use "Unpivot" transformation on a Source that don't have schema pre-defined (Schema Drift is activated).
The "unpivot" transformation generates the following error:

Error: DF-SYS-01 at : scala.MatchError: 1 (of class java.lang.Integer) - RunId: 700ee826-2737-4a4a-b9bb-daf65b52fb31


Could you help me understand the meaning of this error and how to fix it?
Is "unpivot" unable to work with schema drift?

For information, I put more detail about the other component below.

The input file looks like this:


The file format will change regularly, a new column will be added every month, so I want to use Schema Drift.

The source looks like this:
As you can see there is no schema prepared.

My select transformation looks like this:
I retrieve the product code by using "name" and I retrieve all columns prefixed by 20 (ex: 2020-01)


The unpivot is configured as follow, using "byName" to retrieve the product code column:


I create a column year_month which will contain the column headers that have been unpivoted.


I create a column allocation which will contain the values that have been unpivoted.


If you have any other suggestion I'm happy to hear too.

Thank you

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,170 questions
{count} votes

Accepted answer
  1. Kiran-MSFT 691 Reputation points Microsoft Employee

    This looks more complicated that it needs to be. You can do something as simple as

    yearly derive(
    Product_Code = byName("Product_Code")
    ) ~> yearlyDerived

    yearlyDerived unpivot(output(
    year_month as string,
    allocation as string
    lateral: false,
    ignoreNullPivots: false) ~> unpivotYearly

    Unpivots do not allow expressions or assignments in ungroupBy expressions. The validation error will be made more user friendly

0 additional answers

Sort by: Most helpful