TypeError when creating Data Asset with Python

Marvin Joseph Guico Agor 67

I'm trying to register data from a datastore programatically (via Python SDK v2). I need to do it this way since the final output will be a scheduled AzureML Pipeline that does preprocessing for an NLP module. Further, as time progresses, the source files will also be updated. I'm attempting to run the following script:

from azure.ai.ml.entities  import Data
from azure.ai.ml.constants import AssetTypes

data_path = "azureml://
/datastores/[name-of-datastore]/paths/[filename].csv
"


data_asset = Data(
    path = data_path,
    type = AssetTypes.URI_FILE,
    description= 'description',
    name = 'data_name'
)

ml_client.data.create_or_update(asset)

However, running that prompts me with the error

TypeError: Please refer to create_or_update docstring for valid input types.

I'm not sure why I'm getting this error and would like to ask for all the help I can get.

While the example shows an excel file, I have more files I wish to register this way, some being excel sheets.

Ramr-msft 17,606 Reputation points

2023-05-26T15:48:11.7266667+00:00

@Marvin Joseph Guico Agor Thanks for the question, Here is the sample for Data Asset in Python and supported datatypes.

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-read-write-data-v2?view=azureml-api-2&tabs=cli#supported-paths
Marvin Joseph Guico Agor 67 Reputation points

2023-05-27T02:25:40.2833333+00:00

thank you for the resource. I've used the link you've provided as the reference for my script that you see in the OP. While the sample you've provided is really good, I'm still prototyping this script and have not yet reached the automation state (i.e., command jobs), and cannot yet use it. The sample reads the data assets into a command job. Also, the later portion uploads the dataset to the datastore from a local folder. What I am trying to achieve at this step is to register the data asset programmatically from an existing datastore (that is, the data already exists in a storage account that was linked to a datastore).
Ramr-msft 17,606 Reputation points

2023-05-31T16:08:47.9+00:00
@Marvin Joseph Guico Agor Thanks for the details, The error message suggests that there is an issue with the input parameters for the create_or_update method. The create_or_update method expects a dictionary or a list of dictionaries as input, but you are passing a Data object instead.

To fix this issue, you can try converting the Data object to a dictionary and passing it to the create_or_update method. Here's an example of how you can do this:

from azure.ai.ml.entities import Data from azure.ai.ml.constants import AssetTypes data_path = "azureml://datastores/[name-of-datastore]/paths/[filename].csv" data_asset = Data( path=data_path, type=AssetTypes.URI_FILE, description='description', name='data_name' ) data_dict = data_asset.to_dict() ml_client.data.create_or_update(data_dict)

This should convert the Data object to a dictionary and pass it to the create_or_update method. Make sure to replace [name-of-datastore] and [filename] with the actual names of your datastore and file, respectively.

If you have multiple files to register, you can create a list of Data objects and convert them to a list of dictionaries using a loop, like this:

data_list = [] for file_path in file_paths: data_path = f"azureml://datastores/[name-of-datastore]/paths/{file_path}" data_asset = Data( path=data_path, type=AssetTypes.URI_FILE, description='description', name='data_name' ) data_dict = data_asset.to_dict() data_list.append(data_dict) ml_client.data.create_or_update(data_list)