Azure Data Factory Oracle to Parquet datatype mismatch

Jaco van Gelder 1 Reputation point
2021-10-04T11:19:05.047+00:00

Hi all,

Currently we’re extracting an Oracle on-premise source using Azure Data Factory and loading it into Azure Data Lake Storage as parquet files. The extracting and loading goes fine, except for the datatype mappings. Whatever we try, the datatype “decimal(p,s)” will always change to decimal (38,18) in the extracted parquet file.
In the mapping blade of the copy data task in ADF, automatically importing the schema’s using the “import Schemas” button will only map Decimal to Decimal, and not the precision and scale the source has.

I've dynamically mapped the precision and scale as dynamic content to each column, but if I check out my parquet file in a Spark dataframe I'm still getting a decimal(38,18) datatype.

I’ve tried doing the exact same extract and load using an MSSQL source, and that works fine. The data types get mapped and extracted in the right way. Is there currently no support for typecasting datatypes from Oracle to Parquet using Azure Data Factory, and if so, is there a workaround?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,106 questions
{count} votes

1 answer

Sort by: Most helpful
  1. HimanshuSinha-msft 19,381 Reputation points Microsoft Employee
    2022-03-02T06:16:55.807+00:00

    Hello @Jaco van Gelder ,
    Updating the thread from the support ticket information , so that other community are also benefited .

    • Currently ADF had limited support on decimal when copying from Oracle to parquet. We only support decimal of precision <= 38 and length of integer part <= 20 .As a workaround you can you may convert such columns into varchar2 as a workaround.

    Thanks
    Himanshu


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators