Edit

Upgrade datastore management to SDK v2

  • Article

Azure Machine Learning Datastores securely keep the connection information to your data storage on Azure, so you don't have to code it in your scripts. V2 Datastore concept remains mostly unchanged compared with V1. The difference is we won't support SQL-like data sources via Azure Machine Learning Datastores. We'll support SQL-like data sources via Azure Machine Learning data import&export functionalities.

This article gives a comparison of scenario(s) in SDK v1 and SDK v2.

Create a datastore from an Azure Blob container via account_key

  • SDK v1

    blob_datastore_name='azblobsdk' # Name of the datastore to workspace
container_name=os.getenv("BLOB_CONTAINER", "<my-container-name>") # Name of Azure blob container
account_name=os.getenv("BLOB_ACCOUNTNAME", "<my-account-name>") # Storage account name
account_key=os.getenv("BLOB_ACCOUNT_KEY", "<my-account-key>") # Storage account access key

blob_datastore = Datastore.register_azure_blob_container(workspace=ws, 
                                                         datastore_name=blob_datastore_name, 
                                                         container_name=container_name, 
                                                         account_name=account_name,
                                                         account_key=account_key)

  • SDK v2

    from azure.ai.ml.entities import AzureBlobDatastore
from azure.ai.ml import MLClient

ml_client = MLClient.from_config()

store = AzureBlobDatastore(
    name="blob-protocol-example",
    description="Datastore pointing to a blob container using wasbs protocol.",
    account_name="mytestblobstore",
    container_name="data-container",
    protocol="wasbs",
    credentials={
        "account_key": "XXXxxxXXXxXXXXxxXXXXXxXXXXXxXxxXxXXXxXXXxXXxxxXXxxXXXxXxXXXxxXxxXXXXxxxxxXXxxxxxxXXXxXXX"
    },
)

ml_client.create_or_update(store)

Create a datastore from an Azure Blob container via sas_token

  • SDK v1

    blob_datastore_name='azblobsdk' # Name of the datastore to workspace
container_name=os.getenv("BLOB_CONTAINER", "<my-container-name>") # Name of Azure blob container
sas_token=os.getenv("BLOB_SAS_TOKEN", "<my-sas-token>") # Sas token

blob_datastore = Datastore.register_azure_blob_container(workspace=ws, 
                                                         datastore_name=blob_datastore_name, 
                                                         container_name=container_name, 
                                                         sas_token=sas_token)

  • SDK v2

    from azure.ai.ml.entities import AzureBlobDatastore
from azure.ai.ml import MLClient

ml_client = MLClient.from_config()

store = AzureBlobDatastore(
    name="blob-sas-example",
    description="Datastore pointing to a blob container using SAS token.",
    account_name="mytestblobstore",
    container_name="data-container",
    credentials=SasTokenCredentials(
        sas_token= "?xx=XXXX-XX-XX&xx=xxxx&xxx=xxx&xx=xxxxxxxxxxx&xx=XXXX-XX-XXXXX:XX:XXX&xx=XXXX-XX-XXXXX:XX:XXX&xxx=xxxxx&xxx=XXxXXXxxxxxXXXXXXXxXxxxXXXXXxxXXXXXxXXXXxXXXxXXxXX"
    ),
)

ml_client.create_or_update(store)

Create a datastore from an Azure Blob container via identity-based authentication

  • SDK v1
blob_datastore = Datastore.register_azure_blob_container(workspace=ws,
                                                      datastore_name='credentialless_blob',
                                                      container_name='my_container_name',
                                                      account_name='my_account_name')

  • SDK v2

    from azure.ai.ml.entities import AzureBlobDatastore
from azure.ai.ml import MLClient

ml_client = MLClient.from_config()

store = AzureBlobDatastore(
    name="",
    description="",
    account_name="",
    container_name=""
)

ml_client.create_or_update(store)

Get datastores from your workspace

  • SDK v1

    # Get a named datastore from the current workspace
datastore = Datastore.get(ws, datastore_name='your datastore name')

    # List all datastores registered in the current workspace
datastores = ws.datastores
for name, datastore in datastores.items():
    print(name, datastore.datastore_type)

  • SDK v2

    from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

#Enter details of your Azure Machine Learning workspace
subscription_id = '<SUBSCRIPTION_ID>'
resource_group = '<RESOURCE_GROUP>'
workspace_name = '<AZUREML_WORKSPACE_NAME>'

ml_client = MLClient(credential=DefaultAzureCredential(),
                     subscription_id=subscription_id, 
                     resource_group_name=resource_group)

datastore = ml_client.datastores.get(name='your datastore name')

Mapping of key functionality in SDK v1 and SDK v2

Storage types in SDK v1 Storage types in SDK v2
azureml_blob_datastore azureml_blob_datastore
azureml_data_lake_gen1_datastore azureml_data_lake_gen1_datastore
azureml_data_lake_gen2_datastore azureml_data_lake_gen2_datastore
azuremlml_sql_database_datastore Will be supported via import & export functionalities
azuremlml_my_sql_datastore Will be supported via import & export functionalities
azuremlml_postgre_sql_datastore Will be supported via import & export functionalities

Next steps

For more information, see: