Share via

Ingest data from 80 databases

Anshal 2,251 Reputation points
2023-10-13T09:22:53.72+00:00

My client has over 80 da My client has over 80 databases and each ingest 1GB of Data daily and has over 80GB data each. can you help design a pipeline that is scalable and high performance.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator
    2023-10-18T17:24:16.9433333+00:00

    Hello Anshal,

    One of the method is by using the metadata-driven approach in ADF to design a pipeline that is scalable and high performance. With this you can copy huge amounts of objects or load data from different sources

    I hope this helps.

    https://learn.microsoft.com/en-us/azure/data-factory/copy-data-tool-metadata-driven

    User's image

    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.