What's new archive for Azure Data Factory
Azure Data Factory is improved on an ongoing basis. To stay up to date with the most recent developments, refer to the current What's New page, which provides you with information about:
- The latest releases.
- Known issues.
- Bug fixes.
- Deprecated functionality.
- Plans for changes.
This archive page retains updates from older months.
Check out our What's New video archive for all of our monthly updates.
September 2023
Pipelines
Added support for metadata driven pipelines for dynamic full and incremental processing in Azure SQL Learn more
August 2023
Change Data Capture
- Azure Synapse Analytics target availability in top-level CDC resource Learn more
- Snowflake connector in Mapping Data Flows support for Change Data Capture in public preview Learn more
Data flow
- Integer type available for pipeline variables Learn more
- Snowflake CDC source connector available in top-level CDC resource Learn more
- Native UI support of parameterization for more linked services Learn more
Data movement
- Managed private endpoints support for Application Gateway and MySQL Flexible Server Learn more
- Managed virtual network time-to-live (TTL) general availability Learn more
Integration runtime
Self-hosted integration runtime now supports self-contained interactive authoring (Preview) Learn more
July 2023
Change Data Capture
Top-level CDC resource now supports schema evolution. Learn more
Data flow
Merge schema option in delta sink now supports schema evolution in Mapping Data Flows. Learn more
Data movement
- Comment Out Part of Pipeline with Deactivation. Learn more
- Pipeline return value is now generally available. Learn more
Developer productivity
Documentation search now included in the Azure Data Factory search toolbar. Learn more
June 2023
Continuous integration and continuous deployment
npm package now supports pre-downloaded bundle for building ARM templates. If your firewall setting is blocking direct download for your npm package, you can now pre-load the package upfront, and let npm package consume local version instead. This is a super boost for your CI/CD pipeline in a firewalled environment.
Region expansion
Azure Data Factory is now available in Sweden Central. You can co-locate your ETL workflow in this new region if you are utilizing the region for storing and managing your modern data warehouse. Learn more
Data movement
Securing outbound traffic with Azure Data Factory's outbound network rules is now supported. Learn more
Connectors
The Amazon S3 connector is now supported as a sink destination using Mapping Data Flows. Learn more
Data flow
We introduce optional Source settings for DelimitedText and JSON sources in top-level CDC resource. The top-level CDC resource in data factory now supports optional source configurations for Delimited and JSON sources. You can now select the column/row delimiters for delimited sources and set the document type for JSON sources. Learn more
May 2023
Data Factory in Microsoft Fabric
Data factory in Microsoft Fabric provides cloud-scale data movement and data transformation services that allow you to solve the most complex data factory and ETL scenarios. It's intended to make your data factory experience easy to use, powerful, and truly enterprise-grade.
April 2023
Data flow
Easily unroll multiple arrays in ADF data flows. ADF updated the Flatten transformation that now makes it super easy to unroll multiple arrays from a single Flatten transformation step. Learn more
Continuous integration and continuous deployment
You can customize the commit message in Git mode now. Type in a detailed description about the changes you make, and we will save it to Git repository.
Connectors
The Azure Blob Storage connector now supports anonymous authentication. Learn more
March 2023
Connectors
Azure Data Lake Storage Gen2 connector now supports shared access signature authentication. Learn more
February 2023
Data movement
- Anonymous authentication type supported for Azure Blob storage Learn more
- Updated SAP template to easily move SAP data to ADLSGen2 in Delta format Learn more
Orchestration
- Set pipeline output value (Public preview) Learn more
- Workflow Orchestration Manager (Public preview) Learn more
Developer productivity
Dark theme support added (Public preview) Learn more
January 2023
Data flow
- Flowlets now support schema drift Learn more
- SQL CDC incremental extract now supports numeric columns Learn more
Data movement
New top-level CDC resource - native CDC configuration in 3 simple steps Learn more
Orchestration
Orchestrate Synapse notebooks and Synapse spark job definitions natively Learn more
Region expansion
- Continued region expansion - China North 3 now supported Learn more
- Continued region expansion - Switzerland West now supported Learn more
SQL Server Integration Services (SSIS)
Express virtual network injection for SSIS now generally available Learn more
Developer productivity
- Added support for favoriting resources on home page
- Save column width for pipeline monitoring
- Monitor filter updates for faster searches
- Directly launch Pipeline Template Gallery through Azure portal
December 2022
Data flow
SQL change data capture (CDC) incremental extract - supports numeric columns in mapping dataflow Learn more
Data movement
Express virtual network injection for SSIS in Azure Data Factory is generally available Learn more
Region expansion
Continued region expansion - Azure Data Factory is now available in China North 3 Learn more
November 2022
Data flow
- Incremental only is available in SAP CDC - get changes only from SAP system without initial full load Learn more
- Source partitions in initial full data load of SAP CDC to improve performance Learn more
- A new pipeline template - Load multiple objects with big amounts from SAP via SAP CDC Learn more
Data Movement
- Support to Azure Databricks through private link from a Data Factory managed virtual network Learn more
User Interface
2 Pipeline designer enhancements added to ADF Studio preview experience
- Dynamic content flyout - make it easier to set dynamic content in your pipeline activities without using the expression builder Learn more
- Error message relocation to status column - make it easier for you to view errors when you see a Failed pipeline run Learn more
Continuous integration and continuous deployment
In auto publish config, disable publish button is available to void overwriting the last automated publish deployment Learn more
October 2022
Video summary
Data flow
- Export up to 1000 rows from data flow preview Learn more
- SQL CDC in Mapping Data Flows now available (Public Preview) Learn more
- Unlock advanced analytics with Microsoft 365 Mapping Data Flow Connector Learn more
- SAP Change Data Capture (CDC) in now generally available Learn more
Developer productivity
- Now accepting community contributions to Template Gallery Learn more
- New design in Azure portal - easily discover how to launch ADF Studio Learn more
- Learning Center now available in the Azure Data Factory studio Learn more
- One-click to try Azure Data Factory Learn more
Orchestration
- Granular billing view available for ADF - see detailed billing information by pipeline (Public Preview) Learn more
- Script activity execution timeout now configurable Learn more
Region expansion
Continued region expansion - Qatar Central now supported Learn more
Continuous integration and continuous deployment
Exclude pipeline triggers that did not change in deployment now generally available Learn more
September 2022
Video summary
Data flow
- Amazon S3 source connector added Learn more
- Google Sheets REST-based connector added as Source (Preview) Learn more
- Maximum column optimization in dataflow Learn more
- SAP Change Data Capture capabilities in Mapping Data Flow (Preview) - Extract and transform data changes from SAP systems for a more efficient data refresh Learn more
- Writing data to a lookup field via alternative keys supported in Dynamics 365/CRM connectors for mapping data flows Learn more
Data movement
Support to convert Oracle NUMBER type to corresponding integer in source Learn more
Monitoring
- Additional monitoring improvements in Azure Data Factory Learn more
- Monitoring loading improvements - pipeline re-run groupings data fetched only when expanded
- Pagination added to pipeline activity runs view to show all activity records in pipeline run
- Monitoring consumption improvement - loading icon added to know when consumption report is fully calculated
- Additional sorting columns in monitoring - sorting added for Pipeline name, Run End, and Status
- Time-zone settings now saved in monitoring
- Gantt chart view now supported in IR monitoring Learn more
Orchestration
DELETE method in the Web activity now supports sending a body with HTTP request Learn more
User interface
- Native UI support of parameterization added for 6 additional linked services - SAP ODP, ODBC, Microsoft Access, Informix, Snowflake, and DB2 Learn more
- Pipeline designer enhancements added in Studio Preview experience - users can view workflow inside pipeline objects like For Each, If Then, etc. Learn more
August 2022
Video summary
Data flow
- Appfigures connector added as Source (Preview) Learn more
- Cast transformation added - visually convert data types Learn more
- New UI for inline datasets - categories added to easily find data sources Learn more
Data movement
Service principal authentication type added for Azure Blob storage Learn more
Developer productivity
- Default activity time-out changed from 7 days to 12 hours Learn more
- New data factory creation experience - one click to have your factory ready within seconds Learn more
- Expression builder UI update - categorical tabs added for easier use Learn more
Continuous integration and continuous delivery (CI/CD)
When CI/CD integrating ARM template, instead of turning off all triggers, it can exclude triggers that didn't change in deployment Learn more
July 2022
Video summary
Data flow
- Asana connector added as source Learn more
- Three new data transformation functions now supported Learn more
- collectUnique() - Create a new collection of unique values in an array.
- substringIndex() - Extract the substring before n occurrences of a delimiter.
- topN() - Return the top n results after sorting your data.
- Refetch from source available in Refresh for data source change scenarios Learn more
- User defined functions (GA) - Create reusable and customized expressions to avoid building complex logic over and over Learn more Video
- Easier configuration on data flow runtime - choose compute size among Small, Medium and Large to pre-configure all integration runtime settings Learn more
Continuous integration and continuous delivery (CI/CD)
Include Global parameters supported in ARM template. Learn more
Developer productivity
Be a part of Azure Data Factory studio preview features - Experience the latest Azure Data Factory capabilities and be the first to share your feedback Learn more
June 2022
Video summary
| Service category | Service improvements | Details |
| Data flow | Fuzzy join supported for data flows | Fuzzy join is now supported in Join transformation of data flows with configurable similarity score on join conditions. Learn more |
| Editing capabilities in source projection | Editing capabilities in source projection is available in Dataflow to make schemas modifications easily Learn more | |
| Assert error handling | Assert error handling are now supported in data flows for data quality and data validation Learn more | |
| Data Movement | Parameterization natively supported in additional 4 connectors | We added native UI support of parameterization for the following linked services: Azure Data Lake Gen1, Azure PostgreSQL, Google AdWords and PostgreSQL. Learn more |
| SAP Change Data Capture (CDC) capabilities in the new SAP ODP connector (Public Preview) | SAP Change Data Capture (CDC) capabilities are now supported in the new SAP ODP connector. Learn more | |
| Integration Runtime | Time-To-Live in managed VNET (Public Preview) | Time-To-Live can be set to the provisioned computes in managed VNET. Learn more |
| Monitoring | Rerun pipeline with new parameters | You can now rerun pipelines with new parameter values in Azure Data Factory. Learn more |
| Orchestration | ‘turnOffAsync' property is available in Web activity | Web activity supports an async request-reply pattern that invokes HTTP GET on the Location field in the response header of an HTTP 202 Response. It helps web activity automatically poll the monitoring end-point till the job runs. ‘turnOffAsync' property is supported to disable this behavior in cases where polling isn't needed Learn more |
May 2022
| Service category | Service improvements | Details |
| Data flow | User Defined Functions for mapping data flows | Azure Data Factory introduces in public preview user defined functions and data flow libraries. A user defined function is a customized expression you can define to be able to reuse logic across multiple mapping data flows. User defined functions live in a collection called a data flow library to be able to easily group up common sets of customized functions. Learn more |
April 2022
| Service category | Service improvements | Details |
| Data flow | Data preview and debug improvements in mapping data flows | Debug sessions using the AutoResolve Azure integration runtime (IR) will now start up in under 10 seconds. There are new updates to the data preview panel in mapping data flows. Now you can sort the rows inside the data preview view by selecting column headers. You can move columns around interactively. You can also save the data preview results as a CSV by using Export CSV. Learn more |
| Dataverse connector is available for mapping data flows | Dataverse connector is available as source and sink for mapping data flows. Learn more | |
| Support for user database schemas for staging with the Azure Synapse Analytics and PostgreSQL connectors in data flow sink | Data flow sink now supports using a user database schema for staging in both the Azure Synapse Analytics and PostgreSQL connectors. Learn more | |
| Monitoring | Multiple updates to Data Factory monitoring experiences | New updates to the monitoring experience in Data Factory include the ability to export results to a CSV, clear all filters, and open a run in a new tab. Column and result caching is also improved. Learn more |
| User interface | New regional format support | Choosing your language and the regional format in settings will influence the format of how data such as dates and times appear in the Azure Data Factory Studio monitoring. For example, the time format in Monitoring will appear like "Apr 2, 2022, 3:40:29 pm" when choosing English as the regional format, and "2 Apr 2022, 15:40:29" when choosing French as regional format. These settings affect only the Azure Data Factory Studio user interface and do not change/ modify your actual data and time zone. |
March 2022
Video summary
| Service category | Service improvements | Details |
| Data flow | ScriptLines and parameterized linked service support added mapping data flows | It's now easy to detect changes to your data flow script in Git with ScriptLines in your data flow JSON definition. Parameterized linked services can now also be used inside your data flows for flexible generic connection patterns. Learn more |
| Flowlets general availability (GA) | Flowlets is now generally available to create reusable portions of data flow logic that you can share in other pipelines as inline transformations. Flowlets enable extract-transform-and-load (ETL) jobs to be composed of custom or common logic components. Learn more | |
| Change Feed connectors are available in five data flow source transformations | Change Feed connectors are available in data flow source transformations for Azure Cosmos DB, Azure Blob Storage, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, and the common data model (CDM). Learn more | |
| Data preview and debug improvements in mapping data flows | New features were added to data preview and the debug experience in mapping data flows. Learn more | |
| SFTP connector for mapping data flow | SFTP connector is available for mapping data flow as both source and sink. Learn more | |
| Data movement | Support Always Encrypted for SQL-related connectors in Lookup activity under Managed virtual network | Always Encrypted is supported for SQL Server, Azure SQL Database, Azure SQL Managed Instance, and Synapse Analytics in the Lookup activity under managed virtual network. Learn more |
| Integration runtime | New UI layout in Azure IR creation and edit page | The UI layout of the IR creation and edit page now uses tab style for Settings, Virtual network, and Data flow runtime. Learn more |
| Orchestration | Transform data by using the Script activity | You can use a Script activity to invoke a SQL script in SQL Database, Azure Synapse Analytics, SQL Server, Oracle, or Snowflake. Learn more |
| Web activity timeout improvement | You can configure response timeout in a Web activity to prevent it from timing out if the response period is more than one minute, especially in the case of synchronous APIs. Learn more |
February 2022
Video summary
| Service category | Service improvements | Details |
| Data flow | Parameterized linked services supported in mapping data flows | You can now use your parameterized linked services in mapping data flows to make your data flow pipelines generic and flexible. Learn more |
| SQL Database incremental source extract available in data flow (public preview) | A new option has been added on mapping data flow SQL Database sources called Enable incremental extract (preview). Now you can automatically pull only the rows that have changed on your SQL Database sources by using data flows. Learn more | |
| Four new connectors available for mapping data flows (public preview) | Data Factory now supports four new connectors (public preview) for mapping data flows: Quickbase connector, Smartsheet connector, TeamDesk connector, and Zendesk connector. Learn more | |
| Azure Cosmos DB (SQL API) for mapping data flow now supports inline mode | Azure Cosmos DB (SQL API) for mapping data flow can now use inline datasets. Learn more | |
| Data movement | Get metadata-driven data ingestion pipelines on the Data Factory Copy Data tool within 10 minutes (GA) | You can build large-scale data copy pipelines with a metadata-driven approach on the Copy Data tool within 10 minutes. Learn more |
| Data Factory Google AdWords connector API upgrade available | The Data Factory Google AdWords connector now supports the new AdWords API version. No action is required for the new connector user because it's enabled by default. Learn more | |
| Continuous integration and continuous delivery (CI/CD) | Cross tenant Azure DevOps support | Configure a repository using Azure DevOps Git not in the same tenant as the Azure Data Factory。 Learn more |
| Region expansion | Data Factory is now available in West US3 and Jio India West | Data Factory is now available in two new regions: West US3 and Jio India West. You can colocate your ETL workflow in these new regions if you're using these regions to store and manage your modern data warehouse. You can also use these regions for business continuity and disaster recovery purposes if you need to fail over from another region within the geo. Learn more |
| Security | Connect to an Azure DevOps account in another Entra ID tenant | You can connect your Data Factory instance to an Azure DevOps account in a different Azure AD tenant for source control purposes. Learn more |
January 2022
| Service category | Service improvements | Details |
| Data flow | Quick reuse is now automatic in all Azure IRs that use Time to Live (TTL) | You no longer need to manually specify "quick reuse." Data Factory mapping data flows can now start up subsequent data flow activities in under five seconds after you set a TTL. Learn more |
| Retrieve your custom Assert description | In the Assert transformation, you can define your own dynamic description message. You can use the new function assertErrorMessage() to retrieve the row-by-row message and store it in your destination data. Learn more | |
| Automatic schema detection in Parse transformation | A new feature added to the Parse transformation makes it easy to automatically detect the schema of an embedded complex field inside a string column. Select the Detect schema button to set your target schema automatically. Learn more | |
| Support Dynamics 365 connector as both sink and source | You can now connect directly to Dynamics 365 to transform your Dynamics data at scale by using the new mapping data flow connector for Dynamics 365. Learn more | |
| Always Encrypted SQL connections now available in data flows | Always Encrypted can now source transformations in SQL Server, SQL Database, SQL Managed Instance, and Azure Synapse when you use data flows. Learn more | |
| Data movement | Data Factory Azure Databricks Delta Lake connector supports new authentication types | Data Factory Databricks Delta Lake connector now supports two more authentication types: system-assigned managed identity authentication and user-assigned managed identity authentication. Learn more |
| Data Factory Copy activity supports upsert in several more connectors | Data Factory Copy activity now supports upsert while it sinks data to SQL Server, SQL Database, SQL Managed Instance, and Azure Synapse. Learn more |
December 2021
| Service category | Service improvements | Details |
| Data flow | Dynamics connector as native source and sink for mapping data flows | The Dynamics connector is now supported as source and sink for mapping data flows. Learn more |
| Native change data capture (CDC) is now natively supported | CDC is now natively supported in Data Factory for Azure Cosmos DB, Blob Storage, Data Lake Storage Gen1, Data Lake Storage Gen2, and CDM. Learn more | |
| Flowlets public preview | The flowlets public preview allows data flow developers to build reusable components to easily build composable data transformation logic. Learn more | |
| Map Data public preview | The Map Data preview enables business users to define column mapping and transformations to load Azure Synapse lake databases. Learn more | |
| Multiple output destinations from Power Query | You can now map multiple output destinations from Power Query in Data Factory for flexible ETL patterns for citizen data integrators. Learn more | |
| External Call transformation support | Extend the functionality of mapping data flows by using the External Call transformation. You can now add your own custom code as a REST endpoint or call a curated third-party service row by row. Learn more | |
| Enable quick reuse by Azure Synapse mapping data flows with TTL support | Azure Synapse mapping data flows now support quick reuse by setting a TTL in the Azure IR. Using a setting enables your subsequent data flow activities to execute in under five seconds. Learn more | |
| Assert transformation | Easily add data quality, data domain validation, and metadata checks to your Data Factory pipelines by using the Assert transformation in mapping data flows. Learn more | |
| IntelliSense support in expression builder for more productive pipeline authoring experiences | IntelliSense support in expression builder and dynamic content authoring makes Data Factory and Azure Synapse pipeline developers more productive while they write complex expressions in their data pipelines. Learn more |
November 2021
| Service category | Service improvements | Details |
| Continuous integration and continuous delivery (CI/CD) | GitHub integration improvements | Improvements in Data Factory and GitHub integration remove limits on 1,000 Data Factory resources per resource type, such as datasets and pipelines. For large data factories, this change helps mitigate the impact of the GitHub API rate limit. Learn more |
| Data flow | Set a custom error code and error message with the Fail activity | Fail activity enables ETL developers to set the error message and custom error code for a Data Factory pipeline. Learn more |
| External call transformation | Mapping data flows External Call transformation enables ETL developers to use transformations and data enrichments provided by REST endpoints or third-party API services. Learn more | |
| Synapse quick reuse | When you execute data flow in Synapse Analytics, use the TTL feature. The TTL feature uses the quick reuse feature so that sequential data flows will execute within a few seconds. You can set the TTL when you configure an Azure IR. Learn more | |
| Data movement | Copy activity supports reading data from FTP or SFTP without chunking | Automatically determine the file length or the relevant offset to be read when you copy data from an FTP or SFTP server. With this capability, Data Factory automatically connects to the FTP or SFTP server to determine the file length. After the length is determined, Data Factory divides the file into multiple chunks and reads them in parallel. Learn more |
| UTF-8 without BOM support in Copy activity | Copy activity supports writing data with encoding the type UTF-8 without BOM for JSON and delimited text datasets. | |
| Multicharacter column delimiter support | Copy activity supports using multicharacter column delimiters for delimited text datasets. | |
| Integration runtime | Run any process anywhere in three steps with SQL Server Integration Services (SSIS) in Data Factory | Learn how to use the best of Data Factory and SSIS capabilities in a pipeline. A sample SSIS package with parameterized properties helps you get a jump-start. With Data Factory Studio, the SSIS package can be easily dragged and dropped into a pipeline and used as part of an Execute SSIS Package activity. This capability enables you to run the Data Factory pipeline with an SSIS package on self-hosted IRs or SSIS IRs. By providing run-time parameter values, you can use the powerful capabilities of Data Factory and SSIS capabilities together. This article illustrates three steps to run any process, which can be any executable, such as an application, program, utility, or batch file, anywhere. Learn more |
October 2021
| Service category | Service improvements | Details |
| Data flow | Azure Data Explorer and Amazon Web Services (AWS) S3 connectors | The Microsoft Data Integration team has released two new connectors for mapping data flows. If you're using Azure Synapse, you can now connect directly to your AWS S3 buckets for data transformations. In both Data Factory and Azure Synapse, you can now natively connect to your Azure Data Explorer clusters in mapping data flows. Learn more |
| Power Query activity leaves preview for GA | The Data Factory Power Query pipeline activity is now generally available. This new feature provides scaled-out data prep and data wrangling for citizen integrators inside the Data Factory browser UI for an integrated experience for data engineers. The Power Query data wrangling feature in Data Factory provides a powerful, easy-to-use pipeline capability to solve your most complex data integration and ETL patterns in a single service. Learn more | |
| New Stringify data transformation in mapping data flows | Mapping data flows adds a new data transformation called Stringify to make it easy to convert complex data types like structs and arrays into string form. These data types then can be sent to structured output destinations. Learn more | |
| Integration runtime | Express virtual network injection for SSIS IR (public preview) | The SSIS IR now supports express virtual network injection. Learn more: Overview of virtual network injection for SSIS IR Standard vs. express virtual network injection for SSIS IR Express virtual network injection for SSIS IR |
| Security | Azure Key Vault integration improvement | Key Vault integration now has dropdowns so that users can select the secret values in the linked service. This capability increases productivity because users aren't required to type in the secrets, which could result in human error. |
| Support for user-assigned managed identity in Data Factory | Credential safety is crucial for any enterprise. The Data Factory team is committed to making the data engineering process secure yet simple for data engineers. User-assigned managed identity (preview) is now supported in all connectors and linked services that support Azure AD-based authentication. Learn more |
September 2021
| Service category | Service improvements | Details |
| Continuous integration and continuous delivery | Expanded CI/CD capabilities | You can now create a new Git branch based on any other branch in Data Factory. Learn more |
| Data movement | Amazon Relational Database Service (RDS) for Oracle sources | The Amazon RDS for Oracle sources connector is now available in both Data Factory and Azure Synapse. Learn more |
| Amazon RDS for SQL Server sources | The Amazon RDS for the SQL Server sources connector is now available in both Data Factory and Azure Synapse. Learn more | |
| Support parallel copy from Azure Database for PostgreSQL | The Azure Database for PostgreSQL connector now supports parallel copy operations. Learn more | |
| Data flow | Use Data Lake Storage Gen2 to execute pre- and post-processing commands | Hadoop Distributed File System pre- and post-processing commands can now be executed by using Data Lake Storage Gen2 sinks in data flows. Learn more |
| Edit data flow properties for existing instances of the Azure IR | The Azure IR has been updated to allow editing of data flow properties for existing IRs. You can now modify data flow compute properties without needing to create a new Azure IR. Learn more | |
| TTL setting for Azure Synapse to improve pipeline activities execution startup time | Azure Synapse has added TTL to the Azure IR to enable your data flow pipeline activities to begin execution in seconds, which greatly minimizes the runtime of your data flow pipelines. Learn more | |
| Integration runtime | Data Factory managed virtual network GA | You can now provision the Azure IR as part of a managed virtual network and use private endpoints to securely connect to supported data stores. Data traffic goes through Azure Private Links, which provides secured connectivity to the data source. It also prevents data exfiltration to the public internet. Learn more |
| Orchestration | Operationalize and provide SLA for data pipelines | The new Elapsed Time Pipeline Run metric, combined with Data Factory alerts, empowers data pipeline developers to better deliver SLAs to their customers. Now you can tell us how long a pipeline should run, and we'll notify you proactively when the pipeline runs longer than expected. Learn more |
| Fail activity (public preview) | The new Fail activity allows you to throw an error in a pipeline intentionally for any reason. For example, you might use the Fail activity if a Lookup activity returns no matching data or a custom activity finishes with an internal error. Learn more |
August 2021
| Service category | Service improvements | Details |
| Continuous integration and continuous delivery | CI/CD improvements with GitHub support in Azure Government and Microsoft Azure operated by 21Vianet | We've added support for GitHub in Azure for US Government and Azure operated by 21Vianet. Learn more |
| Data movement | The Azure Cosmos DB API for MongoDB connector supports versions 3.6 and 4.0 in Data Factory | The Data Factory Azure Cosmos DB API for MongoDB connector now supports server versions 3.6 and 4.0. Learn more |
| Enhance using COPY statement to load data into Azure Synapse | The Data Factory Azure Synapse connector now supports staged copy and copy source with *.* as wildcardFilename for the COPY statement. Learn more | |
| Data flow | REST endpoints are available as source and sink in data flow | Data flows in Data Factory and Azure Synapse now support REST endpoints as both a source and sink with full support for both JSON and XML payloads. Learn more |
| Integration runtime | Diagnostic tool is available for self-hosted IR | A diagnostic tool for self-hosted IR is designed to provide a better user experience and help users to find potential issues. The tool runs a series of test scenarios on the self-hosted IR machine. Every scenario has typical health check cases for common issues. Learn more |
| Orchestration | Custom event trigger with advanced filtering option GA | You can now create a trigger that responds to a custom topic posted to Azure Event Grid. You can also use advanced filtering to get fine-grain control over what events to respond to. Learn more |
July 2021
| Service category | Service improvements | Details |
| Data movement | Get metadata-driven data ingestion pipelines on the Data Factory Copy Data tool within 10 minutes (public preview) | Now you can build large-scale data copy pipelines with a metadata-driven approach on the Copy Data tool (public preview) within 10 minutes. Learn more |
| Data flow | New map functions added in data flow transformation functions | A new set of data flow transformation functions enables data engineers to easily generate, read, and update map data types and complex map structures. Learn more |
| Integration runtime | Five new regions are available in Data Factory managed virtual network (public preview) | Five new regions, China East2, China North2, US Government Arizona, US Government Texas, and US Government Virginia, are available in the Data Factory managed virtual network (public preview). |
| Developer productivity | Data Factory home page improvements | The Data Factory home page has been redesigned with better contrast and reflow capabilities. A few sections are introduced on the home page to help you improve productivity in your data integration journey. Learn more |
| New landing page for Data Factory Studio | The landing page for the Data Factory pane in the Azure portal. Learn more |
June 2021
| Service category | Service improvements | Details |
| Data movement | New user experience with Data Factory Copy Data tool | The redesigned Copy Data tool is now available with improved data ingestion experience. Learn more |
| MongoDB and MongoDB Atlas are supported as both source and sink | This improvement supports copying data between any supported data store and MongoDB or MongoDB Atlas database. Learn more | |
| Always Encrypted is supported for SQL Database, SQL Managed Instance, and SQL Server connectors as both source and sink | Always Encrypted is available in Data Factory for SQL Database, SQL Managed Instance, and SQL Server connectors for the Copy activity. Learn more | |
| Setting custom metadata is supported in Copy activity when sinking to Data Lake Storage Gen2 or Blob Storage | When you write to Data Lake Storage Gen2 or Blob Storage, the Copy activity supports setting custom metadata or storage of the source file's last modified information as metadata. Learn more | |
| Data flow | SQL Server is now supported as a source and sink in data flows | SQL Server is now supported as a source and sink in data flows. Follow the link for instructions on how to configure your networking by using the Azure IR managed virtual network feature to talk to your SQL Server on-premises and cloud VM-based instances. Learn more |
| Dataflow Cluster quick reuse now enabled by default for all new Azure IRs | The popular data flow quick startup reuse feature is now generally available for Data Factory. All new Azure IRs now have quick reuse enabled by default. Learn more | |
| Power Query (public preview) activity | You can now build complex field mappings to your Power Query sink by using Data Factory data wrangling. The sink is now configured in the pipeline in the Power Query (public preview) activity to accommodate this update. Learn more | |
| Updated data flows monitoring UI in Data Factory | Data Factory has a new update for the monitoring UI to make it easier to view your data flow ETL job executions and quickly identify areas for performance tuning. Learn more | |
| SQL Server Integration Services | Run any SQL statements or scripts anywhere in three steps with SSIS in Data Factory | This post provides three steps to run any SQL statements or scripts anywhere with SSIS in Data Factory.
|