Edit

Share via

Known issue - Unsupported error for legacy timestamp in Fabric Runtime 1.3

  • Article

When using the native execution engine in Fabric Runtime 1.3, you might encounter an error if your data contains legacy timestamps. This issue arises due to compatibility challenges introduced when Spark 3.0 transitioned to the Java 8 date/time API, which uses the Proleptic Gregorian calendar (SQL ISO standard). Earlier Spark versions utilized a hybrid Julian-Gregorian calendar, resulting in potential discrepancies when processing timestamp data created by different Spark versions.

Status: Open

Product Experience: Data Engineering

Symptoms

When using legacy timestamp support in native execution engine for Fabric Runtime 1.3, you receive an error. The error message is similar to: Error Source: USER. Error Code: UNSUPPORTED. Reason: Reading legacy timestamp is not supported.

Solutions and workarounds

For more information about the feature that addresses this known issue, see the blog post on legacy timestamp support. To activate the feature, add the following to your Spark session: SET spark.gluten.legacy.timestamp.rebase.enabled = true. Dates that are post-1970 are unaffected, ensuring consistency without extra steps.

Next steps