is there any limitation of writing billions of record from Azure Data Factory

Samy Abdul 3,376 Reputation points
2021-08-09T14:25:58.377+00:00

Hi All, I have gone through the same question processing billion records in databricks in to Data warehouse.

https://learn.microsoft.com/en-us/answers/questions/120575/is-there-any-limitation-of-writing-billions-of-rec.html

I tried to find out the approach to do similar task in Azure Data Factory but I couldn't able to find any solution for the same. I will be grateful if you could please explain how you would design a pipeline to process billion records and that should provide a near real time analytics. Thanks a lot.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,827 questions
0 comments No comments
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 38,456 Reputation points Microsoft Employee
    2021-08-10T10:40:10.61+00:00

    Hi @Samy Abdul ,

    Thank you for posting your query on Microsoft Q&A Platform.

    Azure data factory can perform Bigdata ETL operations. You can use Copy activity to perform your data movement. It will support billion rows as well.

    Click here to know how to bulk copy multiple tables using Azure data factory
    Click here to know how to bulk copy files to database

    If your Sink is Azure Synapse then you can use Polybase approach to load data. Click here to know more about this.

    Azure Data Factory enriches PolyBase support for loading data into SQL DW. Check below Article for more details.
    https://azure.microsoft.com/en-in/updates/azure-data-factory-enriched-polybase-support-for-loading-data-into-sql-dw/

    Hope this will help. Please let us know if any further queries. Thank you.

    ------------------------------------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.
    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.