ETL migration of cloud

Question

Hi friends, we have to migrate the on-prem ETL solution (Informatica) to Azure. There are 100s of pipelines with business logic and transformations. How to initiate the migration journey do we need to go to each individual pipeline and develop corresponding ETL logic in ADF or Synapse or do we start with fresh business requirements and develop entirely new solutions? Business stakeholders want exactly the same solution which is working fine for time so far on the cloud. Please provide a detailed approach in this case.

Answer 1

Your question is very general and I understand that the concern of business stakeholders may be related to budgets and costs and you need to keep in mind that the costs are not only including the infrastructure (human ressources capacities and readiness are included for example)

1. Assessment and Planning

• Begin by cataloging all existing ETL pipelines, including their sources, destinations, transformations, schedules, and dependencies. Azure Migrate can help you assessing your on-premises workloads and their cloud "readiness".
• Decide on a migration strategy that aligns with your business needs. I think in your case the "Lift and Shift" approach might seem appealing but assess if rearchitecting (for example if you need to modernize certain aspects) could offer benefits (long-term at least).
• ADF and Synapse serve different but complementary roles in Azure data integration and analytics landscape: I think in this case you need to decide based on your pipelines complexity, data processing needs, and orchestration requirements.

2. Environment Setup

• Set up your Azure environment with subscriptions, resource groups, and configuring network security.
• Then you will be able to provision instances of Azure Data Factory and Azure Synapse Analytics.

3. Pilot Migration

• Identify a subset of ETL pipelines that are representative of your broader portfolio in terms of complexity and criticality.
• Use ADF or Synapse to recreate the selected pipelines (involves mapping Informatica transformations to their Azure equivalents, configuring data sources/sinks, and setting up pipeline orchestration).

Don't forget that your new Azure solution adheres to all relevant governance and compliance requirements. Azure Policy and Azure Blueprints can help enforce organizational standards and regulatory compliance