How to keep the order of the columns in a (with Azure Purview) scanned parquet file?

Question

How to keep the order of the columns in a (with Azure Purview) scanned parquet file?

Bernhard Lauber 41

I did a scan of Parquet files (StorageV2 (general purpose v2)). The discovered columns are shown not the original order but alphabetically. When I do a scan with a CSV file the original order is retained.

What can I do to keep the order of the columns in a scanned parquet file? Any hints? Do you have the same experience?

KranthiPakala-MSFT 46,827 Reputation points Microsoft Employee Moderator

2022-02-26T01:00:38.267+00:00

Hello @Bernhard Lauber ,

Thanks for the question and using MS Q&A platform.

My understanding is that the parquet files schema order is displayed in alphabetical order after scanning but when you compared with CSV/TXT file it is not the same case, they have retained the schema order but it is not the same case with Parquet file. Please correct me if I'm wrong.

I did try to reproduce the issue and noticed the same behavior as yours.

I'm reaching out to internal team to verify if it is an expected behavior parquet file and any specific reason for that behavior and will get back to you as soon as we hear back from the team.

Thank you
Bernhard Lauber 41 Reputation points

2022-02-28T13:46:48.523+00:00

Hello @KranthiPakala-MSFT ,

thanks for reproducing the issue. For me the expected behavior is: to get the orginal sort order of the columns (as it is in the source file). Please reach out to the internal team.

Thank you very much

1 answer

Your answer

KranthiPakala-MSFT 46,827 Reputation points Microsoft Employee Moderator

2022-02-26T01:00:38.267+00:00

Hello @Bernhard Lauber ,

Thanks for the question and using MS Q&A platform.

My understanding is that the parquet files schema order is displayed in alphabetical order after scanning but when you compared with CSV/TXT file it is not the same case, they have retained the schema order but it is not the same case with Parquet file. Please correct me if I'm wrong.

I did try to reproduce the issue and noticed the same behavior as yours.

I'm reaching out to internal team to verify if it is an expected behavior parquet file and any specific reason for that behavior and will get back to you as soon as we hear back from the team.

Thank you
Bernhard Lauber 41 Reputation points

2022-02-28T13:46:48.523+00:00

Hello @KranthiPakala-MSFT ,

thanks for reproducing the issue. For me the expected behavior is: to get the orginal sort order of the columns (as it is in the source file). Please reach out to the internal team.

Thank you very much

Answer 1

KranthiPakala-MSFT 46,827 Microsoft Employee Moderator

Hello @Bernhard Lauber ,

After having conversation with internal team and other online readings, it looks like this is a problem with parquet and not Azure Purview. PARQUET-188 suggests that column ordering is not part of the parquet spec, so it confirms that the column order is not honored while Parquet.

Here is another SO thread on similar discussion - Is there a possibility to keep column order when reading parquet?

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Bernhard Lauber 41 Reputation points

2022-03-04T16:58:30.903+00:00
Hello @KranthiPakala-MSFT ,

I don't think that this is an issue with parquet. Because I've run a crawler on AWS glue with the same parquet file. And AWS Glue is able to persist the original order of the columns.
Please have a look at the attachments:

Parquet file itself (please remove the ending .txt)

and a screenshot of AWS glue). So I think Azure Purview should be able to persist als the original order of the columns.

I've invesitgated the file also with this tool: https://github.com/mukunku/ParquetViewer
--> the order in the Parquet file is OK!!

I would like to ask you to investigate the issue again with the internal team and thank you in advance.

Best regards
Bernhard
KranthiPakala-MSFT 46,827 Reputation points Microsoft Employee Moderator

2022-03-04T18:00:59.733+00:00

Hello @Bernhard Lauber ,

Thanks for getting back. I had conversation with Azure Purview product team, and they have confirmed that it is an expected behavior currently and requested to log a feature request in IDEAS forum here - https://feedback.azure.com/d365community/forum/82d7bddb-fb24-ec11-b6e6-000d3a4f07b8.

Product team agreed that this can be a feature request and added to their list of future feature implementations list. But we do not have an ETA at the moment.

Once you log the feature request in Ideas forum, may I request you to please share it here, so that I can pass the link to Product team and document it well so that we don't lose track of it. Sharing it here also will help other readers with similar idea to up-vote and/or comment on it, which in turn help increase the priority of the request.

Do let me know if any further queries.

Thank you
Bernhard Lauber 41 Reputation points

2022-03-07T09:50:07.937+00:00

Hello @KranthiPakala-MSFT ,

thanks for your answer, this is a good suggestion. Yes, I will insert a feature request in IDEAS forum. When it is done I will post the link again here.

Thanks for your support!
KranthiPakala-MSFT 46,827 Reputation points Microsoft Employee Moderator

2022-03-07T17:28:19.76+00:00

Thank you so much @Bernhard Lauber !
Bernhard Lauber 41 Reputation points

2022-03-09T14:13:06.09+00:00

Hello @KranthiPakala-MSFT

I've inserted the feature request into the Ideas forum here: https://feedback.azure.com/d365community/idea/f4367ec2-b09f-ec11-a81c-000d3adfb8f5
Please pass the link to the Product team.

Thanks & Regards
Bernhard
KranthiPakala-MSFT 46,827 Reputation points Microsoft Employee Moderator

2022-03-09T17:50:51.327+00:00

Appreciate much for sharing the feedback link @Bernhard Lauber ! Will pass this details to respective product team and if I have any further updates from them, will keep you posted here.

Thank you again!
David Foster 0 Reputation points

2024-02-09T10:17:59.44+00:00

Has this been resolved yet?

Share via

How to keep the order of the columns in a (with Azure Purview) scanned parquet file?

1 answer

Your answer