Read JSON in ML Pipeline

Michael Kollegger 1 Reputation point
2021-09-10T07:35:08.947+00:00

We created a ML pipline that downloads data from an external CRM system, predicts certain things based on the new data and uploads the results to the external CRM. For the upload a json file, that contains information on the metadata is mandatory. The json file sits in our datastore. We need to open the json file and update its content based on the new predictions. Unfortunately we are not able to load and read the json file in our Python script.
First we create a datastore path --> datastore_paths = [(ds, 'Metadata/XXX_metadata.json')]
Then we create a FileDataset --> json_file = Dataset.File.from_files(path=datastore_paths)
Then we try to open the Json file -->
f=open(json_file)
data_json=json.load(f)
We get the following error message --> expected str, bytes or os.PathLike object, not FileDataset
So far we were not able to find any solution for our problem, Any help or input is highly appreciated. Many thanks in advance !!

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,552 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Ramr-msft 17,611 Reputation points
    2021-09-13T04:08:12.16+00:00

    @Michael Kollegger Thanks, Can you please share the code that you are trying. Please follow the to use FileDataset. Creating a FileDataset pointing to your json file in azure storage, then mount/download the filedataset to your compute target for reading and parsing. Mount will work even if the file size exceeds the storage of your compute instance.