question

arkiboys avatar image
0 Votes"
arkiboys asked KranthiPakala-MSFT commented

string to date conversion

in dataflow, the select transform shows columns.
some of them are date columns but datapreview shows as string because they are coming in from .json files.
I would like to convert the date columns to date datatype
the sink at present is delta parquet

Question:
When I try to convert the date string to date, I see null in datapreview of derived column.
This is what I am using in derived column:

toDate(columnName, 'dd-MM-yyyy')

Any suggestions?

202640-image.png

202701-image.png


azure-data-factory
image.png (20.9 KiB)
image.png (18.1 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
1 Vote"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hello @arkiboys,

Thanks for the question and using MS Q&A platform.

From the above screenshots, looks like your source date format is yyyy-MM-dd (example date from your picture -> 2022-04-26) but in your derived column expression language you are using dd-MM-yyyy in toDate function which is why you are seeing nulls

Instead of toDate(columnName, 'dd-MM-yyyy') please try with toDate(columnName, 'yyyy-MM-dd') and it should resolve your issue.

Reference doc: Mapping data flow expressions usage#toDate

202951-image.png

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (19.7 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

this works and thank you.
Question:
As my sink is delta parquet, should I convert the string date fields to date datatype like the way you showed me or is it best to leave everything as string since the sink will be delta parquet?
Thank you

0 Votes 0 ·

Hey @arkiboys, it totally depends on your use case after writing to your sink delta tables but in general it would be good to have a defined schema for parquet unless you have a specific requirement to define all columns as string type in later part of your implementation.

Hope this clarifies.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.


0 Votes 0 ·