how to read and write special characters using pyspark

Question

HI - I have a file with pipe ( | ) delimiter values. However they have special characters like comma( , ) and double qutoes ( " ) in some columns. I am writing the data to another file using dataframe but these characters are not written there instead '/' and spaces are inserted in place of them. Please suggest how can I write them as it is to another file using pyspark in synapse analytics notebook.

thanks,
RR Netha

Answer 1

Hi @Erabathini, Rajendar ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As I understand your question, you are trying to write data from one csv file having pipe delimiter and symbols (i.e. comma and double quotes) into another csv file. However, in the target file, symbols are not retained and they are getting replaced by slash '/'. Please let me know if that is not the case.

'/' before " is expected in the target file if you have not explicitly defined quote other than double quote ("). Since delimited text files by default enclose string data with double quotes(eg: "ABC","india"), so in order to escape the " which lies in the data , it puts '/' automatically . So, Kindly make sure to include quote parameter having value anything other than double quote. You can use single quote in quote parameter like below :

df.write.csv('abfss://<file_system>@<account_name>.dfs.core.windows.net/<path>/<file_name>', quote="'")

Without quote param:

With quote param:

Hope it helps. Please do consider clicking Accept Answer as accepted answers help community as well. Also, please click on Yes for the survey 'Was the answer helpful'