Dataset Class

Reference

Represents a resource for exploring, transforming, and managing data in Azure Machine Learning.

A Dataset is a reference to data in a Datastore or behind public web urls.

For methods deprecated in this class, please check AbstractDataset class for the improved APIs.

The following Datasets types are supported:

TabularDataset represents data in a tabular format created by parsing the provided file or list of files.
FileDataset references single or multiple files in datastores or from public URLs.

To get started with datasets, see the article Add & register datasets, or see the notebooks https://aka.ms/tabulardataset-samplenotebook and https://aka.ms/filedataset-samplenotebook.

Initialize the Dataset object.

To obtain a Dataset that has already been registered with the workspace, use the get method.

Inheritance: builtins.object

Dataset

Constructor

Dataset(definition, workspace=None, name=None, id=None)

Parameters

definition: <xref:azureml.data.DatasetDefinition>

Required

The Dataset definition.

workspace: Workspace

Required

The workspace in which the Dataset exists.

name: str

Required

The name of the Dataset.

id: str

Required

The unique identifier of the Dataset.

Remarks

The Dataset class exposes two convenience class attributes (File and Tabular) you can use for creating a Dataset without working with the corresponding factory methods. For example, to create a dataset using these attributes:

Dataset.Tabular.from_delimited_files()
Dataset.File.from_files()

You can also create a new TabularDataset or FileDataset by directly calling the corresponding factory methods of the class defined in TabularDatasetFactory and FileDatasetFactory.

The following example shows how to create a TabularDataset pointing to a single path in a datastore.


   from azureml.core import Dataset
   dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, 'train-dataset/tabular/iris.csv')])

   # preview the first 3 rows of the dataset
   dataset.take(3).to_pandas_dataframe()

Full sample is available from https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/work-with-data/datasets-tutorial/train-with-datasets/train-with-datasets.ipynb

Variables

azureml.core.Dataset.File

A class attribute that provides access to the FileDatasetFactory methods for creating new FileDataset objects. Usage: Dataset.File.from_files().

azureml.core.Dataset.Tabular

A class attribute that provides access to the TabularDatasetFactory methods for creating new TabularDataset objects. Usage: Dataset.Tabular.from_delimited_files().

Methods

archive	Archive an active or deprecated dataset. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
auto_read_files	Analyzes the file(s) at the specified path and returns a new Dataset. Note This method is deprecated and will no longer be supported. Recommend to use the Dataset.Tabular.from_* methods to read files. For more information, see https://aka.ms/dataset-deprecation.
compare_profiles	Compare the current Dataset's profile with another dataset profile. This shows the differences in summary statistics between two datasets. The parameter 'rhs_dataset' stands for "right-hand side", and is simply the second dataset. The first dataset (the current dataset object) is considered the "left-hand side". Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
create_snapshot	Create a snapshot of the registered Dataset. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
delete_snapshot	Delete snapshot of the Dataset by name. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
deprecate	Deprecate an active dataset in a workspace by another dataset. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
diff	Diff the current Dataset with rhs_dataset. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
from_binary_files	Create an unregistered, in-memory Dataset from binary files. Note This method is deprecated and will no longer be supported. Recommend to use Dataset.File.from_files instead. For more information, see https://aka.ms/dataset-deprecation.
from_delimited_files	Create an unregistered, in-memory Dataset from delimited files. Note This method is deprecated and will no longer be supported. Recommend to use Dataset.Tabular.from_delimited_files instead. For more information, see https://aka.ms/dataset-deprecation. `# Create a dataset from delimited files with header option as ALL_FILES_HAVE_SAME_HEADERS dataset = Dataset.Tabular.from_delimited_files(path=(datastore, 'data/crime-spring.csv'), header='ALL_FILES_HAVE_SAME_HEADERS') df = dataset.to_pandas_dataframe()`
from_excel_files	Create an unregistered, in-memory Dataset from Excel files. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
from_json_files	Create an unregistered, in-memory Dataset from JSON files. Note This method is deprecated and will no longer be supported. Recommend to use Dataset.Tabular.from_json_lines_files instead to read from JSON lines file. For more information, see https://aka.ms/dataset-deprecation.
from_pandas_dataframe	Create an unregistered, in-memory Dataset from a pandas dataframe. Note This method is deprecated and will no longer be supported. Recommend to use Dataset.Tabular.register_pandas_dataframe instead. For more information, see https://aka.ms/dataset-deprecation.
from_parquet_files	Create an unregistered, in-memory Dataset from parquet files. Note This method is deprecated and will no longer be supported. Recommend to use Dataset.Tabular.from_parquet_files instead. For more information, see https://aka.ms/dataset-deprecation.
from_sql_query	Create an unregistered, in-memory Dataset from a SQL query. Note This method is deprecated and will no longer be supported. Recommend to use Dataset.Tabular.from_sql_query instead. For more information, see https://aka.ms/dataset-deprecation.
generate_profile	Generate new profile for the Dataset. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
get	Get a Dataset that already exists in the workspace by specifying either its name or ID. Note This method is deprecated and will no longer be supported. Recommend to use get_by_name and get_by_id instead. For more information, see https://aka.ms/dataset-deprecation.
get_all	Get all the registered datasets in the workspace.
get_all_snapshots	Get all snapshots of the Dataset. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
get_by_id	Get a Dataset which is saved to the workspace.
get_by_name	Get a registered Dataset from workspace by its registration name.
get_definition	Get a specific definition of the Dataset. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
get_definitions	Get all the definitions of the Dataset. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
get_profile	Get summary statistics on the Dataset computed earlier. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
get_snapshot	Get snapshot of the Dataset by name. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
head	Pull the specified number of records specified from this Dataset and returns them as a DataFrame. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
list	List all the Datasets in the workspace, including ones with `is_visible` property equal to False. Note This method is deprecated and will no longer be supported. Recommend to use get_all instead. For more information, see https://aka.ms/dataset-deprecation.
reactivate	Reactivate an archived or deprecated dataset. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
register	Register the Dataset in the workspace, making it available to other users of the workspace. Note This method is deprecated and will no longer be supported. Recommend to use register instead. For more information, see https://aka.ms/dataset-deprecation.
sample	Generate a new sample from the source Dataset, using the sampling strategy and parameters provided. Note This method is deprecated and will no longer be supported. Create a TabularDataset by calling the static methods on Dataset.Tabular and use the take_sample method there. For more information, see https://aka.ms/dataset-deprecation.
to_pandas_dataframe	Create a Pandas dataframe by executing the transformation pipeline defined by this Dataset definition. Note This method is deprecated and will no longer be supported. Create a TabularDataset by calling the static methods on Dataset.Tabular and use the to_pandas_dataframe method there. For more information, see https://aka.ms/dataset-deprecation.
to_spark_dataframe	Create a Spark DataFrame that can execute the transformation pipeline defined by this Dataset definition. Note This method is deprecated and will no longer be supported. Create a TabularDataset by calling the static methods on Dataset.Tabular and use the to_spark_dataframe method there. For more information, see https://aka.ms/dataset-deprecation.
update	Update the Dataset mutable attributes in the workspace and return the updated Dataset from the workspace. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.
update_definition	Update the Dataset definition. Note This method is deprecated and will no longer be supported. For more information, see https://aka.ms/dataset-deprecation.

auto_read_files

Analyzes the file(s) at the specified path and returns a new Dataset.

Note

This method is deprecated and will no longer be supported.

Recommend to use the Dataset.Tabular.from_* methods to read files. For more information, see https://aka.ms/dataset-deprecation.

static auto_read_files(path, include_path=False, partition_format=None)

Parameters

path: DataReference or str

Required

A data path in a registered datastore, a local path, or an HTTP URL(CSV/TSV).

include_path: bool

Required

Whether to include a column containing the path of the file from which the data was read. Useful when reading multiple files, and want to know which file a particular record originated from. Also useful if there is information in file path or name that you want in a column.

partition_format: str

Required

Specify the partition format in path and create string columns from format '{x}' and datetime column from format '{x:yyyy/MM/dd/HH/mm/ss}', where 'yyyy', 'MM', 'dd', 'HH', 'mm' and 'ss' are used to extrat year, month, day, hour, minute and second for the datetime type. The format should start from the postition of first partition key until the end of file path. For example, given a file path '../Accounts/2019/01/01/data.csv' where data is partitioned by department name and time, we can define '/{Department}/{PartitionDate:yyyy/MM/dd}/data.csv' to create columns 'Department' of string type and 'PartitionDate' of datetime type.

Returns

Dataset object.

Return type

Dataset

Remarks

Use this method when to have file formats and delimiters detected automatically.

After creating a Dataset, you should use get_profile to list detected column types and summary statistics for each column.

The returned Dataset is not registered with the workspace.

compare_profiles

Compare the current Dataset's profile with another dataset profile.

This shows the differences in summary statistics between two datasets. The parameter 'rhs_dataset' stands for "right-hand side", and is simply the second dataset. The first dataset (the current dataset object) is considered the "left-hand side".

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

compare_profiles(rhs_dataset, profile_arguments={}, include_columns=None, exclude_columns=None, histogram_compare_method=HistogramCompareMethod.WASSERSTEIN)

Parameters

rhs_dataset: Dataset

Required

A second Dataset, also called a "right-hand side" Dataset for comparision.

profile_arguments: dict

Required

Arguments to retrive specific profile.

include_columns: list[str]

Required

List of column names to be included in comparison.

exclude_columns: list[str]

Required

List of column names to be excluded in comparison.

histogram_compare_method: HistogramCompareMethod

Required

Enum describing the comparison method, ex: Wasserstein or Energy

Returns

Difference between the two dataset profiles.

Return type

<xref:azureml.dataprep.api.engineapi.typedefinitions.DataProfileDifference>

Remarks

This is for registered Datasets only. Raises an exception if the current Dataset's profile does not exist. For unregistered Datasets use profile.compare method.

create_snapshot

Create a snapshot of the registered Dataset.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

create_snapshot(snapshot_name, compute_target=None, create_data_snapshot=False, target_datastore=None)

Parameters

snapshot_name: str

Required

The snapshot name. Snapshot names should be unique within a Dataset.

compute_target: Union[ComputeTarget, str]

Required

Optional compute target to perform the snapshot profile creation. If omitted, the local compute is used.

create_data_snapshot: bool

Required

If True, a materialized copy of the data will be created.

target_datastore: Union[AbstractAzureStorageDatastore, str]

Required

Target datastore to save snapshot. If omitted, the snapshot will be created in the default storage of the workspace.

Returns

Dataset snapshot object.

Return type

DatasetSnapshot

Remarks

Snapshots capture point in time summary statistics of the underlying data and an optional copy of the data itself. To learn more about creating snapshots, go to https://aka.ms/azureml/howto/createsnapshots.

delete_snapshot

Delete snapshot of the Dataset by name.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

delete_snapshot(snapshot_name)

Parameters

snapshot_name: str

Required

The snapshot name.

Returns

None.

Return type

None

Remarks

Use this to free up storage consumed by data saved in snapshots that you no longer need.

deprecate

Deprecate an active dataset in a workspace by another dataset.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

deprecate(deprecate_by_dataset_id)

Parameters

deprecate_by_dataset_id: str

Required

The Dataset ID which is the intended replacement for this Dataset.

Returns

None.

Return type

None

Remarks

Deprecated Datasets will log warnings when they are consumed. Deprecating a dataset deprecates all its definitions.

Deprecated Datasets can still be consumed. To completely block a Dataset from being consumed, archive it.

If deprecated by accident, reactivate will activate it.

diff

Diff the current Dataset with rhs_dataset.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

diff(rhs_dataset, compute_target=None, columns=None)

Parameters

rhs_dataset: Dataset

Required

Another Dataset also called right hand side Dataset for comparision

compute_target: Union[ComputeTarget, str]

Required

compute target to perform the diff. If omitted, the local compute is used.

columns: list[str]

Required

List of column names to be included in diff.

Returns

Dataset action run object.

Return type

DatasetActionRun

from_binary_files

Create an unregistered, in-memory Dataset from binary files.

Note

This method is deprecated and will no longer be supported.

Recommend to use Dataset.File.from_files instead. For more information, see https://aka.ms/dataset-deprecation.

static from_binary_files(path)

Parameters

path: DataReference or str

Required

A data path in a registered datastore or a local path.

Returns

The Dataset object.

Return type

Dataset

Remarks

Use this method to read files as streams of binary data. Returns one file stream object per file read. Use this method when you're reading images, videos, audio or other binary data.

get_profile and create_snapshot will not work as expected for a Dataset created by this method.

The returned Dataset is not registered with the workspace.

from_delimited_files

Create an unregistered, in-memory Dataset from delimited files.

Note

This method is deprecated and will no longer be supported.

Recommend to use Dataset.Tabular.from_delimited_files instead. For more information, see https://aka.ms/dataset-deprecation.


   # Create a dataset from delimited files with header option as ALL_FILES_HAVE_SAME_HEADERS
   dataset = Dataset.Tabular.from_delimited_files(path=(datastore, 'data/crime-spring.csv'),
       header='ALL_FILES_HAVE_SAME_HEADERS')

   df = dataset.to_pandas_dataframe()

static from_delimited_files(path, separator=',', header=PromoteHeadersBehavior.ALL_FILES_HAVE_SAME_HEADERS, encoding=FileEncoding.UTF8, quoting=False, infer_column_types=True, skip_rows=0, skip_mode=SkipLinesBehavior.NO_ROWS, comment=None, include_path=False, archive_options=None, partition_format=None)

Parameters

path: DataReference or str

Required

A data path in a registered datastore, a local path, or an HTTP URL.

separator: str

Required

The separator used to split columns.

header: PromoteHeadersBehavior

Required

Controls how column headers are promoted when reading from files.

encoding: FileEncoding

Required

The encoding of the files being read.

quoting: bool

Required

Specify how to handle new line characters within quotes. The default (False) is to interpret new line characters as starting new rows, irrespective of whether the new line characters are within quotes or not. If set to True, new line characters inside quotes will not result in new rows, and file reading speed will slow down.

infer_column_types: bool

Required

Indicates whether column data types are inferred.

skip_rows: int

Required

How many rows to skip in the file(s) being read.

skip_mode: SkipLinesBehavior

Required

Controls how rows are skipped when reading from files.

comment: str

Required

Character used to indicate comment lines in the files being read. Lines beginning with this string will be skipped.

include_path: bool

Required

Whether to include a column containing the path of the file from which the data was read. This is useful when you are reading multiple files, and want to know which file a particular record originated from, or to keep useful information in file path.

archive_options: <xref:azureml.dataprep.ArchiveOptions>

Required

Options for archive file, including archive type and entry glob pattern. We only support ZIP as archive type at the moment. For example, specifying


   archive_options = ArchiveOptions(archive_type = ArchiveType.ZIP, entry_glob = '*10-20.csv')

reads all files with name ending with "10-20.csv" in ZIP.

partition_format: str

Required

Returns

Dataset object.

Return type

Dataset

Remarks

Use this method to read delimited text files when you want to control the options used.

After creating a Dataset, you should use get_profile to list detected column types and summary statistics for each column.

The returned Dataset is not registered with the workspace.

from_excel_files

Create an unregistered, in-memory Dataset from Excel files.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

static from_excel_files(path, sheet_name=None, use_column_headers=False, skip_rows=0, include_path=False, infer_column_types=True, partition_format=None)

Parameters

path: DataReference or str

Required

A data path in a registered datastore or a local path.

sheet_name: str

Required

The name of the Excel sheet to load. By default we read the first sheet from each Excel file.

use_column_headers: bool

Required

Controls whether to use the first row as column headers.

skip_rows: int

Required

How many rows to skip in the file(s) being read.

include_path: bool

Required

infer_column_types: bool

Required

If true, column data types will be inferred.

partition_format: str

Required

Specify the partition format in path and create string columns from format '{x}' and datetime column from format '{x:yyyy/MM/dd/HH/mm/ss}', where 'yyyy', 'MM', 'dd', 'HH', 'mm' and 'ss' are used to extrat year, month, day, hour, minute and second for the datetime type. The format should start from the postition of first partition key until the end of file path. For example, given a file path '../Accounts/2019/01/01/data.xlsx' where data is partitioned by department name and time, we can define '/{Department}/{PartitionDate:yyyy/MM/dd}/data.xlsx' to create columns 'Department' of string type and 'PartitionDate' of datetime type.

Returns

Dataset object.

Return type

Dataset

Remarks

Use this method to read Excel files in .xlsx format. Data can be read from one sheet in each Excel file. After creating a Dataset, you should use get_profile to list detected column types and summary statistics for each column. The returned Dataset is not registered with the workspace.

from_json_files

Create an unregistered, in-memory Dataset from JSON files.

Note

This method is deprecated and will no longer be supported.

Recommend to use Dataset.Tabular.from_json_lines_files instead to read from JSON lines file. For more information, see https://aka.ms/dataset-deprecation.

static from_json_files(path, encoding=FileEncoding.UTF8, flatten_nested_arrays=False, include_path=False, partition_format=None)

Parameters

path: DataReference or str

Required

The path to the file(s) or folder(s) that you want to load and parse. It can either be a local path or an Azure Blob url. Globbing is supported. For example, you can use path = "./data*" to read all files with name starting with "data".

encoding: FileEncoding

Required

The encoding of the files being read.

flatten_nested_arrays: bool

Required

Property controlling program's handling of nested arrays. If you choose to flatten nested JSON arrays, it could result in a much larger number of rows.

include_path: bool

Required

Whether to include a column containing the path from which the data was read. This is useful when you are reading multiple files, and might want to know which file a particular record originated from, or to keep useful information in file path.

partition_format: str

Required

Specify the partition format in path and create string columns from format '{x}' and datetime column from format '{x:yyyy/MM/dd/HH/mm/ss}', where 'yyyy', 'MM', 'dd', 'HH', 'mm' and 'ss' are used to extrat year, month, day, hour, minute and second for the datetime type. The format should start from the postition of first partition key until the end of file path. For example, given a file path '../Accounts/2019/01/01/data.json' and data is partitioned by department name and time, we can define '/{Department}/{PartitionDate:yyyy/MM/dd}/data.json' to create columns 'Department' of string type and 'PartitionDate' of datetime type.

Returns

The local Dataset object.

Return type

Dataset

from_pandas_dataframe

Create an unregistered, in-memory Dataset from a pandas dataframe.

Note

This method is deprecated and will no longer be supported.

Recommend to use Dataset.Tabular.register_pandas_dataframe instead. For more information, see https://aka.ms/dataset-deprecation.

static from_pandas_dataframe(dataframe, path=None, in_memory=False)

Parameters

dataframe: DataFrame

Required

The Pandas DataFrame.

path: Union[DataReference, str]

Required

A data path in registered datastore or local folder path.

in_memory: bool

Required

Whether to read the DataFrame from memory instead of persisting to disk.

Returns

A Dataset object.

Return type

Dataset

Remarks

Use this method to convert a Pandas dataframe to a Dataset object. A Dataset created by this method can not be registered, as the data is from memory.

If in_memory is False, the Pandas DataFrame is converted to a CSV file locally. If pat is of type DataReference, then the Pandas frame will be uploaded to the data store, and the Dataset will be based off the DataReference. If ``path` is a local folder, the Dataset will be created off of the local file which cannot be deleted.

Raises an exception if the current DataReference is not a folder path.

from_parquet_files

Create an unregistered, in-memory Dataset from parquet files.

Note

This method is deprecated and will no longer be supported.

Recommend to use Dataset.Tabular.from_parquet_files instead. For more information, see https://aka.ms/dataset-deprecation.

static from_parquet_files(path, include_path=False, partition_format=None)

Parameters

path: DataReference or str

Required

A data path in a registered datastore or a local path.

include_path: bool

Required

partition_format: str

Required

Specify the partition format in path and create string columns from format '{x}' and datetime column from format '{x:yyyy/MM/dd/HH/mm/ss}', where 'yyyy', 'MM', 'dd', 'HH', 'mm' and 'ss' are used to extrat year, month, day, hour, minute and second for the datetime type. The format should start from the postition of first partition key until the end of file path. For example, given a file path '../Accounts/2019/01/01/data.parquet' where data is partitioned by department name and time, we can define '/{Department}/{PartitionDate:yyyy/MM/dd}/data.parquet' to create columns 'Department' of string type and 'PartitionDate' of datetime type.

Returns

Dataset object.

Return type

Dataset

Remarks

Use this method to read Parquet files.

After creating a Dataset, you should use get_profile to list detected column types and summary statistics for each column.

The returned Dataset is not registered with the workspace.

from_sql_query

Create an unregistered, in-memory Dataset from a SQL query.

Note

This method is deprecated and will no longer be supported.

Recommend to use Dataset.Tabular.from_sql_query instead. For more information, see https://aka.ms/dataset-deprecation.

static from_sql_query(data_source, query)

Parameters

data_source: AzureSqlDatabaseDatastore

Required

The details of the Azure SQL datastore.

query: str

Required

The query to execute to read data.

Returns

The local Dataset object.

Return type

Dataset

generate_profile

Generate new profile for the Dataset.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

generate_profile(compute_target=None, workspace=None, arguments=None)

Parameters

compute_target: Union[ComputeTarget, str]

Required

An optional compute target to perform the snapshot profile creation. If omitted, the local compute is used.

workspace: Workspace

Required

Workspace, required for transient(unregistered) Datasets.

arguments: dict[str, object]

Required

Profile arguments. Valid arguments are:

'include_stype_counts' of type bool. Check if values look like some well known semantic types such as email address, IP Address (V4/V6), US phone number, US zipcode, Latitude/Longitude. Enabling this impacts performance.
'number_of_histogram_bins' of type int. Represents the number of histogram bins to use for numeric data. The default value is 10.

Returns

Dataset action run object.

Return type

DatasetActionRun

Remarks

Synchronous call, will block till it completes. Call get_result to get the result of the action.

get

Get a Dataset that already exists in the workspace by specifying either its name or ID.

Note

This method is deprecated and will no longer be supported.

Recommend to use get_by_name and get_by_id instead. For more information, see https://aka.ms/dataset-deprecation.

static get(workspace, name=None, id=None)

Parameters

workspace: Workspace

Required

The existing AzureML workspace in which the Dataset was created.

name: str

Required

The name of the Dataset to be retrieved.

id: str

Required

A unique identifier of the Dataset in the workspace.

Returns

The Dataset with the specified name or ID.

Return type

Dataset

Remarks

You can provide either name or id. An exception is raised if:

both name and id are specified but don't match.
the Dataset with the specified name or id cannot be found in the workspace.

get_all

Get all the registered datasets in the workspace.

get_all()

Parameters

workspace: Workspace

Required

The existing AzureML workspace in which the Datasets were registered.

Returns

A dictionary of TabularDataset and FileDataset objects keyed by their registration name.

Return type

dict[str, Union[TabularDataset, FileDataset]]

get_all_snapshots

Get all snapshots of the Dataset.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

get_all_snapshots()

Returns

List of Dataset snapshots.

Return type

list[DatasetSnapshot]

get_by_id

Get a Dataset which is saved to the workspace.

get_by_id(id)

Parameters

workspace: Workspace

Required

The existing AzureML workspace in which the Dataset is saved.

id: str

Required

The id of dataset.

Returns

The dataset object. If dataset is registered, its registration name and version will also be returned.

Return type

Union[TabularDataset, FileDataset]

get_by_name

Get a registered Dataset from workspace by its registration name.

get_by_name(name, version='latest')

Parameters

workspace: Workspace

Required

The existing AzureML workspace in which the Dataset was registered.

name: str

Required

The registration name.

version: int

Required

The registration version. Defaults to 'latest'.

Returns

The registered dataset object.

Return type

Union[TabularDataset, FileDataset]

get_definition

Get a specific definition of the Dataset.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

get_definition(version_id=None)

Parameters

version_id: str

Required

The version ID of the Dataset definition

Returns

The Dataset definition.

Return type

DatasetDefinition

Remarks

If version_id is provided, then Azure Machine Learning tries to get the definition corresponding to that version. If that version does not exist, an exception is thrown. If version_id is omitted, then the latest version is retrieved.

get_definitions

Get all the definitions of the Dataset.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

get_definitions()

Returns

A dictionary of Dataset definitions.

Return type

dict[str, DatasetDefinition]

Remarks

A Dataset registered in an AzureML workspace can have multiple definitions, each created by calling update_definition. Each definition has an unique identifier. The current definition is the latest one created.

For unregistered Datasets, only one definition exists.

get_profile

Get summary statistics on the Dataset computed earlier.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

get_profile(arguments=None, generate_if_not_exist=True, workspace=None, compute_target=None)

Parameters

arguments: dict[str, object]

Required

Profile arguments.

generate_if_not_exist: bool

Required

Indicates whether to generate a profile if it doesn't exist.

workspace: Workspace

Required

Workspace, required for transient(unregistered) Datasets.

compute_target: Union[ComputeTarget, str]

Required

A compute target to execute the profile action.

Returns

DataProfile of the Dataset.

Return type

<xref:azureml.dataprep.DataProfile>

Remarks

For a Dataset registered with an Azure Machine Learning workspace, this method retrieves an existing profile that was created earlier by calling get_profile if it is still valid. Profiles are invalidated when changed data is detected in the Dataset or the arguments to get_profile are different from the ones used when the profile was generated. If the profile is not present or invalidated, generate_if_not_exist will determine if a new profile is generated.

For a Dataset that is not registered with an Azure Machine Learning workspace, this method always runs generate_profile and returns the result.

get_snapshot

Get snapshot of the Dataset by name.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

get_snapshot(snapshot_name)

Parameters

snapshot_name: str

Required

The snapshot name.

Returns

Dataset snapshot object.

Return type

DatasetSnapshot

head

Pull the specified number of records specified from this Dataset and returns them as a DataFrame.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

head(count)

Parameters

count: int

Required

The number of records to pull.

Returns

A Pandas DataFrame.

Return type

DataFrame

list

List all the Datasets in the workspace, including ones with is_visible property equal to False.

Note

This method is deprecated and will no longer be supported.

Recommend to use get_all instead. For more information, see https://aka.ms/dataset-deprecation.

static list(workspace)

Parameters

workspace: Workspace

Required

The workspace for which you want to retrieve the list of Datasets.

Returns

A list of Dataset objects.

Return type

list[Dataset]

reactivate

Reactivate an archived or deprecated dataset.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

reactivate()

Returns

None.

Return type

None

register

Note

This method is deprecated and will no longer be supported.

Recommend to use register instead. For more information, see https://aka.ms/dataset-deprecation.

register(workspace, name, description=None, tags=None, visible=True, exist_ok=False, update_if_exist=False)

Parameters

workspace: Workspace

Required

The AzureML workspace in which the Dataset is to be registered.

name: str

Required

The name of the Dataset in the workspace.

description: str

Required

A description of the Dataset.

tags: dict[str, str]

Required

Tags to associate with the Dataset.

visible: bool

Required

Indicates whether the Dataset is visible in the UI. If False, then the Dataset is hidden in the UI and available via SDK.

exist_ok: bool

Required

If True, the method returns the Dataset if it already exists in the given workspace, else error.

update_if_exist: bool

Required

If exist_ok is True and update_if_exist is True, this method will update the definition and return the updated Dataset.

Returns

A registered Dataset object in the workspace.

Return type

Dataset

sample

Generate a new sample from the source Dataset, using the sampling strategy and parameters provided.

Note

This method is deprecated and will no longer be supported.

Create a TabularDataset by calling the static methods on Dataset.Tabular and use the take_sample method there. For more information, see https://aka.ms/dataset-deprecation.

sample(sample_strategy, arguments)

Parameters

sample_strategy: str

Required

Sample strategy to use. Accepted values are "top_n", "simple_random", or "stratified".

arguments: dict[str, object]

Required

A dictionary with keys from the "Optional argument" in the list shown above, and values from tye "Type" column. Only arguments from the corresponding sampling method can be used. For example, for a "simple_random" sample type, you can only specify a dictionary with "probability" and "seed" keys.

Returns

Dataset object as a sample of the original dataset.

Return type

Dataset

Remarks

Samples are generated by executing the transformation pipeline defined by this Dataset, and then applying the sampling strategy and parameters to the output data. Each sampling method supports the following optional arguments:

top_n
- Optional arguments
  - n, type integer. Select top N rows as your sample.
simple_random
- Optional arguments
  - probability, type float. Simple random sampling where each row has equal probability of being selected. Probability should be a number between 0 and 1.
  - seed, type float. Used by random number generator. Use for repeatability.
stratified
- Optional arguments
  - columns, type list[str]. List of strata columns in the data.
  - seed, type float. Used by random number generator. Use for repeatability.
  - fractions, type dict[tuple, float]. Tuple: column values that define a stratum, must be in the same order as column names. Float: weight attached to a stratum during sampling.

The following code snippets are example design patterns for different sample methods.


   # sample_strategy "top_n"
   top_n_sample_dataset = dataset.sample('top_n', {'n': 5})

   # sample_strategy "simple_random"
   simple_random_sample_dataset = dataset.sample('simple_random', {'probability': 0.3, 'seed': 10.2})

   # sample_strategy "stratified"
   fractions = {}
   fractions[('THEFT',)] = 0.5
   fractions[('DECEPTIVE PRACTICE',)] = 0.2

   # take 50% of records with "Primary Type" as THEFT and 20% of records with "Primary Type" as
   # DECEPTIVE PRACTICE into sample Dataset
   sample_dataset = dataset.sample('stratified', {'columns': ['Primary Type'], 'fractions': fractions})

to_pandas_dataframe

Create a Pandas dataframe by executing the transformation pipeline defined by this Dataset definition.

Note

This method is deprecated and will no longer be supported.

Create a TabularDataset by calling the static methods on Dataset.Tabular and use the to_pandas_dataframe method there. For more information, see https://aka.ms/dataset-deprecation.

to_pandas_dataframe()

Returns

A Pandas DataFrame.

Return type

DataFrame

Remarks

Return a Pandas DataFrame fully materialized in memory.

to_spark_dataframe

Create a Spark DataFrame that can execute the transformation pipeline defined by this Dataset definition.

Note

This method is deprecated and will no longer be supported.

Create a TabularDataset by calling the static methods on Dataset.Tabular and use the to_spark_dataframe method there. For more information, see https://aka.ms/dataset-deprecation.

to_spark_dataframe()

Returns

A Spark DataFrame.

Return type

DataFrame

Remarks

The Spark Dataframe returned is only an execution plan and does not actually contain any data, as Spark Dataframes are lazily evaluated.

update

Update the Dataset mutable attributes in the workspace and return the updated Dataset from the workspace.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

update(name=None, description=None, tags=None, visible=None)

Parameters

name: str

Required

The name of the Dataset in the workspace.

description: str

Required

A description of the data.

tags: dict[str, str]

Required

Tags to associate the Dataset with.

visible: bool

Required

Indicates whether the the Dataset is visible in the UI.

Returns

An updated Dataset object from the workspace.

Return type

Dataset

update_definition

Update the Dataset definition.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

update_definition(definition, definition_update_message)

Parameters

definition: DatasetDefinition

Required

The new definition of this Dataset.

definition_update_message: str

Required

The definition update message.

Returns

An updated Dataset object from the workspace.

Return type

Dataset

Remarks

To consume the updated Dataset, use the object returned by this method.

Attributes

definition

Return the current Dataset definition.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

Returns

The Dataset definition.

Return type

DatasetDefinition

Remarks

A Dataset definition is a series of steps that specify how to read and transform data.

A Dataset registered in an AzureML workspace can have multiple definitions, each created by calling update_definition. Each definition has an unique identifier. Having multiple definitions allows you to make changes to existing Datasets without breaking models and pipelines that depend on the older definition.

For unregistered Datasets, only one definition exists.

definition_version

Return the version of the current definition of the Dataset.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

Returns

The Dataset definition version.

Return type

str

Remarks

A Dataset definition is a series of steps that specify how to read and transform data.

For unregistered Datasets, only one definition exists.

description

Return the description of the Dataset.

Returns

The Dataset description.

Return type

str

Remarks

Specifying a description of the data in the Dataset enables users of the workspace to understand what the data represents, and how they can use it.

id

If the Dataset was registered in a workspace, return the ID of the Dataset. Otherwise, return None.

Returns

The Dataset ID.

Return type

str

is_visible

Control the visibility of a registered Dataset in the Azure ML workspace UI.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

Returns

The Dataset visibility.

Return type

bool

Remarks

Values returned:

True: Dataset is visible in workspace UI. Default.
False: Dataset is hidden in workspace UI.

Has no effect on unregistered Datasets.

name

Return the Dataset name.

Returns

The Dataset name.

Return type

str

state

Return the state of the Dataset.

Note

This method is deprecated and will no longer be supported.

For more information, see https://aka.ms/dataset-deprecation.

Returns

The Dataset state.

Return type

str

Remarks

The meaning and effect of states are as follows:

Active. Active definitions are exactly what they sound like, all actions can be performed on active definitions.
Deprecated. deprecated definition can be used, but will result in a warning being logged in the logs everytime the underlying data is accessed.
Archived. An archived definition cannot be used to perform any action. To perform actions on an archived definition, it must be reactivated.

workspace

If the Dataset was registered in a workspace, return that. Otherwise, return None.

Returns

The workspace.

Return type

Workspace