Unable to infer schema for JSON. It must be specified manually

praveen kumar 1 Reputation point


I am facing issue while reading json file from adls location.

Error: Unable to infer schema for JSON. It must be specified manually

Actually i am trying to read the data generated from datafactory (insights-logs-activityruns) which is configured at diagnostic settings.

i am using the below way to read

df = spark.read.option("inferSchema",'true').json("dbfs:/mnt/insights-logs-activity/resourceId=/")

PFA 197957-log-activity-runs.png

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
993 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,392 questions
{count} votes

2 answers

Sort by: Most helpful
  1. HimanshuSinha-msft 19,196 Reputation points Microsoft Employee

    Hello @praveen kumar ,
    Thanks for the question and using MS Q&A platform.

    As we understand the ask here is how to get rid of the "Error: Unable to infer schema for JSON. It must be specified manually", please do let us know if its not accurate.
    The path whcih you are refering the code is "insights-logs-activity/resourceId=/"

    df = spark.read.option("inferSchema",'true').json("dbfs:/mnt/insights-logs-activity/resourceId=/")

    But from the screenshot I think the JSON file is at "insights-logs-activity/resourceId=/subscript ...../m=00"

    I suggest you to please update the path and try it again .

    Please do let me if you have any queries.

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

  2. Ryan Abbey 1,136 Reputation points

    Depending on what version you're using, you may be able to use RecursiveFileLookup


    0 comments No comments