Share via

Azure Data Factory Oracle to Parquet datatype mismatch

Vaibhav 105 Reputation points
2024-05-09T10:06:20.1066667+00:00

Hi Team,

I have build a dynamic pipeline to load the data from Oracle tables to datalake as Parquet files.
Now, when I am trying to copy the data from source to sink through CopyActivity, then I am not able to convert the decimal datatype of the columns dynamically.

User's image

In the json, I have changed my source and sink type as Int32 for one column, but when I preview the sink data, then I see decimal values only for that column.

  1. Similar query was raised few years back , so wanted to know if we have any solution now.
    https://learn.microsoft.com/en-us/answers/questions/576333/azure-data-factory-oracle-to-parquet-datatype-mism
  2. Currently, I am passing my columns as parameter in dataflow derived column transformation to convert it into integer type. But, I don't think its a good solution if I have to do some other column datatype conversion.
    So, is there a better way where I can convert all the columns with new datatype in one go .
    User's image
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,804 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Smaran Thoomu 11,045 Reputation points Microsoft Vendor
    2024-05-10T11:27:40.9033333+00:00

    @Vaibhav Thank you for your response and clarifying your requirements.

    I understand that you have already implemented the solution I suggested using the Data Flow transformation in Azure Data Factory, but you are looking for a more dynamic solution where you can convert all the columns with new datatype in one go, similar to the Copy Activity dynamic mapping tab.
    However, as per my knowledge, there is no direct way to convert all the columns with a new datatype in one go using the Copy Activity dynamic mapping tab. The dynamic mapping tab only allows you to map columns between the source and sink datasets dynamically.

    If you want to convert all the columns with a new datatype in one go, you can use the Data Flow transformation in Azure Data Factory. You can use the Derived Column transformation to convert the decimal datatype columns to Int32, string to date, or any other datatype conversion.

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments