inferSchema not working for json on Synapse

Question

I'm having an issue reading gzipped json files from ADLS in Azure Synapse using pyspark in a notebook on my workspace. I'm able to read individual files using pd.read_json(), but when running spark.read.json(url) it returns the error AnalysisException: Unable to infer schema for JSON. It must be specified manually.. I've tried spark.read.option("inferSchema", True).json(url) but get the same error. This is on multiple different files across my ADLS. I tested specifying a schema manually on dummy data I put into a json, and that only yielded an empty dataframe. Any suggestions on what else I can try to resolve this? Let me know if I can provide more context!

Answer 1

Hi @Ben Fisher ,

Thank you for posting query in Microsoft Q&A Platform.

Could you please share more details on what you mean by gzipped json files? Did you considered unzipping folder first and then try to take json from that unzipped path to read? If not, please try same.

I tried to read data from json file in Synapse Notebook. Its working fine for me. Kindly check below.

Code Used in above Image:

%%pyspark  
df = spark.read.load('abfss://******@storageName.dfs.core.windows.net/data/employees.json', format='json')  
df.printSchema()  

Hope this helps. Please let us know how it goes.