Unable to infer schema for JSON. It must be specified manually

Question

Hi,

I am facing issue while reading json file from adls location.

Error: Unable to infer schema for JSON. It must be specified manually

Actually i am trying to read the data generated from datafactory (insights-logs-activityruns) which is configured at diagnostic settings.

i am using the below way to read

df = spark.read.option("inferSchema",'true').json("dbfs:/mnt/insights-logs-activity/resourceId=/")

PFA

Answer 1

Hello @praveen kumar ,
Thanks for the question and using MS Q&A platform.

As we understand the ask here is how to get rid of the "Error: Unable to infer schema for JSON. It must be specified manually", please do let us know if its not accurate.
The path whcih you are refering the code is "insights-logs-activity/resourceId=/"

df = spark.read.option("inferSchema",'true').json("dbfs:/mnt/insights-logs-activity/resourceId=/")

But from the screenshot I think the JSON file is at "insights-logs-activity/resourceId=/subscript ...../m=00"

I suggest you to please update the path and try it again .

Please do let me if you have any queries.
Thanks
Himanshu

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Depending on what version you're using, you may be able to use RecursiveFileLookup

https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#recursive-file-lookup