Accessing Registered Data Assets or Workspace Blob Storage JSON in score.py of Azure ML Online Endpoint

Question

Is it possible to access our registered Data Assets or data stored in the workspace blob store from score.py of an online endpoint?
The file is a JSON, which cannot be read with pandas due to inconsistent column counts, so I would like to use AzureMachineLearningFileSystem (from azureml.fsspec import AzureMachineLearningFileSystem) to read it.

Attached code snippet:

fs = AzureMachineLearningFileSystem("path to data")
with fs.open("data.json") as f:
    data = json.load(f)

If this approach works, what should be used as the correct path?

Answer 1

Hi Issac Chan

The approach on using AzureMachineLearningFileSystem will need the datastore path which can be copied from "Browse Option"/Copy URL(select the azureml:/ one). Post that we can use "./" method to load the json file

from azureml.fsspec import AzureMachineLearningFileSystem

# instantiate file system using following URI
fs = AzureMachineLearningFileSystem('azureml://subscriptions/<subid>/resourcegroups/<rgname>/workspaces/<workspace_name>/datastore*s*/datastorename')

fs.ls() # list folders/files in datastore 'datastorename'

# output example:
# folder1
# folder2
# file3.csv

# use an open context
with fs.open('./folder1/file1.json') as f:     # do some process
    process_file(f)

You can also use pd.read_json to read json files

import pandas as pd
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

ml_client = MLClient.from_config(credential=DefaultAzureCredential())
data_asset = ml_client.data.get("testdatasetjson", version="1")

df = pd.read_json(data_asset.path)
df

Reference used -

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-data-interactive?view=azureml-api-2&tabs=adls

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-read-write-data-v2?view=azureml-api-2&tabs=python

Thank you.