Reading .sav files from ADLS Gen2 into Azure Synapse

Question

Reading .sav files from ADLS Gen2 into Azure Synapse

Johan á Rogvi-Hansen 106

Hi.

I have some pre-trained models saved as .sav files and uploaded to an ADLS Gen2 account.

To my knowledge the .sav format is not supported by spark.read. Is there another way to read files as these?

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-01-19T00:52:17.7233333+00:00

Hi Johan á Rogvi-Hansen,

We still have not heard back from you. Just wanted to check if the below suggestion was helpful? If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

2 answers

Your answer

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-01-19T00:52:17.7233333+00:00

Hi Johan á Rogvi-Hansen,

We still have not heard back from you. Just wanted to check if the below suggestion was helpful? If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Answer 1

Johan á Rogvi-Hansen 106

Hi,

I was able to do it by mounting the file location and using pickle. But, indeed, I needed to upload pyreadstat to the Spark pool. Specifically, after mounting:

with open(f'/synfs/{job_id}/mnt/<folder_path>', 'rb') as pickle_file:
    content = pickle.load(pickle_file)
    print(content)

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-01-21T00:21:43.5666667+00:00

Hi Johan á Rogvi-Hansen ,

Appreciate much for sharing your findings here and helping the community. :)

Answer 2

Hi Johan á Rogvi-Hansen ,

Thank you for using this forum and posting your query.

To read .sav file format (SPSS), you may explore pyreadstat python package (pip install pyreadstat). This python package is useful to read and write SAS (sas7bdat, sas7bcat, xport/xpt), SPSS (sav, zsav, por) and Stata (dta) files into/from pandas data frames.

Here are related documentations:

You may try loading the .sav file data from your ADLS gen2 to Pandas Data Frame and then push it to your desired destination from Pandas Data Frame.

Or you may explore similar approach as explained in this article: How to use the pyreadstat.read_sav function in pyreadstat

Other helpful threads: pyreadstat read and write spss without data loss

Hope this information helps.

Thank you

Please do consider clicking on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.

Share via

Reading .sav files from ADLS Gen2 into Azure Synapse

2 answers

Your answer