ETL migration of cloud

Anshal 2,251 Reputation points
2024-03-29T13:27:32.8266667+00:00

Hi friends, we have to migrate the on-prem ETL solution (Informatica) to Azure. There are 100s of pipelines with business logic and transformations. How to initiate the migration journey do we need to go to each individual pipeline and develop corresponding ETL logic in ADF or Synapse or do we start with fresh business requirements and develop entirely new solutions? Business stakeholders want exactly the same solution which is working fine for time so far on the cloud. Please provide a detailed approach in this case.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,997 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,843 questions
0 comments No comments
{count} votes

Accepted answer
  1. Amira Bedhiafi 26,186 Reputation points
    2024-03-29T15:07:17.7266667+00:00

    Your question is very general and I understand that the concern of business stakeholders may be related to budgets and costs and you need to keep in mind that the costs are not only including the infrastructure (human ressources capacities and readiness are included for example)

    1. Assessment and Planning

    • Begin by cataloging all existing ETL pipelines, including their sources, destinations, transformations, schedules, and dependencies. Azure Migrate can help you assessing your on-premises workloads and their cloud "readiness".
    • Decide on a migration strategy that aligns with your business needs. I think in your case the "Lift and Shift" approach might seem appealing but assess if rearchitecting (for example if you need to modernize certain aspects) could offer benefits (long-term at least).
    • ADF and Synapse serve different but complementary roles in Azure data integration and analytics landscape: I think in this case you need to decide based on your pipelines complexity, data processing needs, and orchestration requirements.

    2. Environment Setup

    • Set up your Azure environment with subscriptions, resource groups, and configuring network security.
    • Then you will be able to provision instances of Azure Data Factory and Azure Synapse Analytics.

    3. Pilot Migration

    • Identify a subset of ETL pipelines that are representative of your broader portfolio in terms of complexity and criticality.
    • Use ADF or Synapse to recreate the selected pipelines (involves mapping Informatica transformations to their Azure equivalents, configuring data sources/sinks, and setting up pipeline orchestration).

    Don't forget that your new Azure solution adheres to all relevant governance and compliance requirements. Azure Policy and Azure Blueprints can help enforce organizational standards and regulatory compliance

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.