Dataflow in ADF is not able to read parquet files exported from ADX

Question

I have defined a continuous export to parquet files in ADX and then use these files to perform some operations on a dataflow in ADF. The dataflow was running for more than six months without errors and since a week ago it has not been able to read some files giving the following error:

java.lang.UnsupportedOperationException: Unsupported encoding: DELTA_BYTE_ARRAY

The properties of the continuous export are as follows:

  "WriteNativeParquetV2": true,
  "ParquetDatetimePrecision": 1,
  "isDisabled": false,
  "ReportDeltaLogResults": false

How should I handle this situation?

Answer 1

Like the error message indicates, you are reading data in Parquet format and writing to a (Delta I guess) table when you get a Parquet column cannot be converted error message.

The vectorized Parquet reader is decoding the type column to a binary format.

So if you have decimal type columns in your source data, you should disable the vectorized Parquet reader.

https://kb.databricks.com/scala/spark-job-fail-parquet-column-convert

https://stackoverflow.com/questions/75145412/unsupported-encoding-delta-byte-array-when-reading-from-kusto-using-kusto-spark

https://issues.apache.org/jira/browse/SPARK-26509