data_transfer Package

Classes

DataTransferCopy	Note This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information. Base class for data transfer copy node. You should not instantiate this class directly. Instead, you should create from builder function: copy_data.
DataTransferCopyComponent	Note This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information. DataTransfer copy component version, used to define a data transfer copy component.
DataTransferExport	Note This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information. Base class for data transfer export node. You should not instantiate this class directly. Instead, you should create from builder function: export_data.
DataTransferExportComponent	Note This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information. DataTransfer export component version, used to define a data transfer export component.
DataTransferImport	Note This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information. Base class for data transfer import node. You should not instantiate this class directly. Instead, you should create from builder function: import_data.
DataTransferImportComponent	Note This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information. DataTransfer import component version, used to define a data transfer import component.
Database	Define a database class for a DataTransfer Component or Job.
FileSystem	Define a file system class of a DataTransfer Component or Job. e.g. source_s3 = FileSystem(path='s3://my_bucket/my_folder', connection='azureml:my_s3_connection')

Functions

copy_data

Note

This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Create a DataTransferCopy object which can be used inside dsl.pipeline as a function.

copy_data(*, name: str | None = None, description: str | None = None, tags: Dict | None = None, display_name: str | None = None, experiment_name: str | None = None, compute: str | None = None, inputs: Dict | None = None, outputs: Dict | None = None, is_deterministic: bool = True, data_copy_mode: str | None = None, **kwargs: Any) -> DataTransferCopy

Keyword-Only Parameters

Name	Description
name	str The name of the job.
description	str Description of the job.
tags	dict[str, str] Tag dictionary. Tags can be added, removed, and updated.
display_name	str Display name of the job.
experiment_name	str Name of the experiment the job will be created under.
compute	str The compute resource the job runs on.
inputs	dict Mapping of inputs data bindings used in the job.
outputs	dict Mapping of outputs data bindings used in the job.
is_deterministic	bool Specify whether the command will return same output given same input. If a command (component) is deterministic, when use it as a node/step in a pipeline, it will reuse results from a previous submitted job in current workspace which has same inputs and settings. In this case, this step will not use any compute resource. Default to be True, specify is_deterministic=False if you would like to avoid such reuse behavior.
data_copy_mode	str data copy mode in copy task, possible value is "merge_with_overwrite", "fail_if_conflict".

Returns

Type	Description
DataTransferCopyComponent	A DataTransferCopy object.

export_data

Note

This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Create a DataTransferExport object which can be used inside dsl.pipeline.

export_data(*, name: str | None = None, description: str | None = None, tags: Dict | None = None, display_name: str | None = None, experiment_name: str | None = None, compute: str | None = None, sink: Dict | Database | FileSystem | None = None, inputs: Dict | None = None, **kwargs: Any) -> DataTransferExport

Keyword-Only Parameters

Name	Description
name	str The name of the job.
description	str Description of the job.
tags	dict[str, str] Tag dictionary. Tags can be added, removed, and updated.
display_name	str Display name of the job.
experiment_name	str Name of the experiment the job will be created under.
compute	str The compute resource the job runs on.
sink	Union[ Dict, Database, FileSystem] The sink of external data and databases.
inputs	dict Mapping of inputs data bindings used in the job.

Returns

Type	Description
<xref:azure.ai.ml.entities._job.pipeline._component_translatable.DataTransferExport>	A DataTransferExport object.

Exceptions

Type	Description
ValidationException	If sink is not provided or exporting file system is not supported.

import_data

Note

This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Create a DataTransferImport object which can be used inside dsl.pipeline.

import_data(*, name: str | None = None, description: str | None = None, tags: Dict | None = None, display_name: str | None = None, experiment_name: str | None = None, compute: str | None = None, source: Dict | Database | FileSystem | None = None, outputs: Dict | None = None, **kwargs: Any) -> DataTransferImport

Keyword-Only Parameters

Name	Description
name	str The name of the job.
description	str Description of the job.
tags	dict[str, str] Tag dictionary. Tags can be added, removed, and updated.
display_name	str Display name of the job.
experiment_name	str Name of the experiment the job will be created under.
compute	str The compute resource the job runs on.
source	Union[Dict, Database, FileSystem] The data source of file system or database.
outputs	dict Mapping of outputs data bindings used in the job. The default will be an output port with the key "sink" and type "mltable".

Returns

Type	Description
<xref:azure.ai.ml.entities._job.pipeline._component_translatable.DataTransferImport>	A DataTransferImport object.