Edit

Share via

New-AzDataFactoryDataset

Module:
Az.DataFactory Module

Creates a dataset in Data Factory.

Syntax

ByFactoryName (Default) 

New-AzDataFactoryDataset
    [-DataFactoryName] <String>
    [[-Name] <String>]
    [-File] <String>
    [-ResourceGroupName] <String>
    [-Force]
    [-DefaultProfile <IAzureContextContainer>]
    [-WhatIf]
    [-Confirm]
    [<CommonParameters>]

ByFactoryObject 

New-AzDataFactoryDataset
    [-DataFactory] <PSDataFactory>
    [[-Name] <String>]
    [-File] <String>
    [-Force]
    [-DefaultProfile <IAzureContextContainer>]
    [-WhatIf]
    [-Confirm]
    [<CommonParameters>]

Description

The New-AzDataFactoryDataset cmdlet creates a dataset in Azure Data Factory. If you specify a name for a dataset that already exists, this cmdlet prompts you for confirmation before it replaces the dataset. If you specify the Force parameter, the cmdlet replaces the existing dataset without confirmation. Perform these operations in the following order:

  • Create a data factory.
  • Create linked services.
  • Create datasets.
  • Create a pipeline. If a dataset with the same name already exists in the data factory, this cmdlet prompts you to confirm whether to overwrite the existing dataset with the new dataset. If you confirm to overwrite the existing dataset, the dataset definition is also replaced.

Examples

Example 1: Create a dataset

New-AzDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File "C:\\samples\\WikiSample\\DA_WikipediaClickEvents.json"

DatasetName         : DAWikipediaClickEvents
ResourceGroupName : ADF
DataFactoryName   : WikiADF
Availability      : Microsoft.DataFactories.Availability
Location          : Microsoft.DataFactories.AzureBlobLocation
Policy            : Microsoft.DataFactories.Policy
Structure         : {}

This command creates a dataset named DA_WikipediaClickEvents in the data factory named WikiADF. The command bases the dataset on information in the DAWikipediaClickEvents.json file.

Example 2: View availability for a new dataset

$Dataset = New-AzDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File "C:\\samples\\WikiSample\\DA_WikipediaClickEvents.json"
$Dataset.Availability

AnchorDateTime :
Frequency      : Hour
Interval       : 1
Offset         :
WaitOnExternal : Microsoft.DataFactories.WaitOnExternal

The first command creates a dataset named DA_WikipediaClickEvents, as in a previous example, and then assigns that dataset to the $Dataset variable. The second command uses standard dot notation to display details about the Availability property of the dataset.

Example 3: View location for a new dataset

$Dataset = New-AzDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File "C:\\samples\\WikiSample\\DA_WikipediaClickEvents.json"
$Dataset.Location

BlobPath          : wikidatagateway/wikisampledatain/
FilenamePrefix    :
Format            :
LinkedServiceName : LinkedServiceWikipediaClickEvents
PartitionBy       : {}

The first command creates a dataset named DA_WikipediaClickEvents, as in a previous example, and then assigns that dataset to the $Dataset variable. The second command displays details about the Location property of the dataset.

Example 4: View validation rules for a new dataset

$Dataset = New-AzDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File "C:\\samples\\WikiSample\\DA_WikipediaClickEvents.json"
$Dataset.Policy.Validation | Format-List $dataset.Location

BlobPath          : wikidatagateway/wikisampledatain/
FilenamePrefix    :
Format            :
LinkedServiceName : LinkedServiceWikipediaClickEvents
PartitionBy       : {}

MinimumRows   :
MinimumSizeMB : 1

The first command creates a dataset named DA_WikipediaClickEvents, as in a previous example, and then assigns that dataset to the $Dataset variable. The second command gets details about the validation rules for the dataset, and then passes them to the Format-List cmdlet by using the pipeline operator. That Windows PowerShell cmdlet formats the results. For more information, type Get-Help Format-List.

Parameters

-Confirm

Prompts you for confirmation before running the cmdlet.

Parameter properties

Type:SwitchParameter
Default value:False
Supports wildcards:False
DontShow:False
Aliases:cf

Parameter sets

(All)
Position:Named
Mandatory:False
Value from pipeline:False
Value from pipeline by property name:False
Value from remaining arguments:False

-DataFactory

Specifies a PSDataFactory object. This cmdlet creates a dataset in the data factory that this parameter specifies.

Parameter properties

Type:PSDataFactory
Default value:None
Supports wildcards:False
DontShow:False

Parameter sets

ByFactoryObject
Position:0
Mandatory:True
Value from pipeline:False
Value from pipeline by property name:True
Value from remaining arguments:False

-DataFactoryName

Specifies the name of a data factory. This cmdlet creates a dataset in the data factory that this parameter specifies.

Parameter properties

Type:String
Default value:None
Supports wildcards:False
DontShow:False

Parameter sets

ByFactoryName
Position:1
Mandatory:True
Value from pipeline:False
Value from pipeline by property name:True
Value from remaining arguments:False

-DefaultProfile

The credentials, account, tenant, and subscription used for communication with azure

Parameter properties

Type:IAzureContextContainer
Default value:None
Supports wildcards:False
DontShow:False
Aliases:AzContext, AzureRmContext, AzureCredential

Parameter sets

(All)
Position:Named
Mandatory:False
Value from pipeline:False
Value from pipeline by property name:False
Value from remaining arguments:False

-File

Specifies the full path of the JavaScript Object Notation (JSON) file that contains the description of the dataset.

Parameter properties

Type:String
Default value:None
Supports wildcards:False
DontShow:False

Parameter sets

(All)
Position:3
Mandatory:True
Value from pipeline:False
Value from pipeline by property name:False
Value from remaining arguments:False

-Force

Indicates that this cmdlet replaces an existing dataset without prompting you for confirmation.

Parameter properties

Type:SwitchParameter
Default value:None
Supports wildcards:False
DontShow:False

Parameter sets

(All)
Position:Named
Mandatory:False
Value from pipeline:False
Value from pipeline by property name:False
Value from remaining arguments:False

-Name

Specifies the name of the dataset to create.

Parameter properties

Type:String
Default value:None
Supports wildcards:False
DontShow:False

Parameter sets

(All)
Position:2
Mandatory:False
Value from pipeline:False
Value from pipeline by property name:True
Value from remaining arguments:False

-ResourceGroupName

Specifies the name of an Azure resource group. This cmdlet creates a dataset in the group that this parameter specifies.

Parameter properties

Type:String
Default value:None
Supports wildcards:False
DontShow:False

Parameter sets

ByFactoryName
Position:0
Mandatory:True
Value from pipeline:False
Value from pipeline by property name:True
Value from remaining arguments:False

-WhatIf

Shows what would happen if the cmdlet runs. The cmdlet is not run.

Parameter properties

Type:SwitchParameter
Default value:False
Supports wildcards:False
DontShow:False
Aliases:wi

Parameter sets

(All)
Position:Named
Mandatory:False
Value from pipeline:False
Value from pipeline by property name:False
Value from remaining arguments:False

CommonParameters

This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutBuffer, -OutVariable, -PipelineVariable, -ProgressAction, -Verbose, -WarningAction, and -WarningVariable. For more information, see about_CommonParameters.

Inputs

PSDataFactory

String

Outputs

PSDataset

Notes

  • Keywords: azure, azurerm, arm, resource, management, manager, data, factories