Data pipeline

Question

I want to set up a data pipeline in my company. I was thinking using azure data factory to collect all data from different sources and connect it to powerbi. However, i see different pipelines set ups went looking for information on the web. Examples are Azure datafactory-SSIS-Data lake -DWH-Powerbi or Azure datafactory-Databricks-datalake-Powerbi or just azure datafactory-databricks-powerbi. Assuming that we want to take it full to the cloud which set up would be the best? Which tools are commonly connected to each other?

Answer 1

Hello @Lindsey Utomo ,

Welcome to the Microsoft Q&A platform.

Azure Synapse Analytics is a limitless analytics service that brings together data integration, enterprise data warehousing and big data analytics. It gives you the freedom to query data on your terms, using either serverless or dedicated resources—at scale. Azure Synapse brings these worlds together with a unified experience to ingest, explore, prepare, manage and serve data for immediate BI and machine learning needs.

• Azure Synapse Contains industry leading SQL and Apache Spark.
• Synapse SQL is a distributed query system for T-SQL that enables data warehousing and data virtualization scenarios and extends T-SQL to address streaming and machine learning scenarios.
• Apache Spark for Azure Synapse deeply and seamlessly integrates Apache Spark--the most popular open-source big data engine used for data preparation, data engineering, ETL, and machine learning.
• Azure Synapse removes the traditional technology barriers between using SQL and Spark together. You can seamlessly mix and match based on your needs and expertise with your Data Lake.
• Azure Synapse contains the same Data Integration engine and experiences as Azure Data Factory, allowing you to create rich at-scale ETL pipelines without leaving Azure Synapse Analytics.
• Also, you can connect a Power BI workspace to an Azure Synapse Analytics workspace to create new Power BI reports and datasets from Synapse Studio.

So finally, Azure Synapse Analytics is the best option for all your needs.

For more information, refer to What is Azure Synapse Analytics?

This example scenario demonstrates how to use the extensive family of Azure Data Services to build a modern data platform capable of handling the most common data challenges in an organization.

For more details, refer to Analytics end-to-end with Azure Synapse.

Hope this helps. Do let us know if you any further queries.

---------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.