ETL Framework In Production

Question

Uma 511

Hello,

I am leveraging Python within SQL Server Integration Services (SSIS) packages, primarily through the Execute Process Task.

I am extracting Data from E2 SQL to Azure SQL Server.

The connection has already been established.

I am building complete ETL using Function

The entire execution will be drive using Driver table or Configuration table.

May someone please help me how can we achieve this parameter based ETL framework.

do I need to create any config table or any metadata table how to pass value like .

Please suggest if any production grade ETL Framework, which helps to achieve this.

Appreciate your help.

Thanks a lot

Anonymous

2025-10-31T11:36:07.2166667+00:00
Hi @Uma,

Good question, since you’re using Python within SSIS, this setup belongs to the SQL Server → Integration Services (SSIS) category.

To build a parameter-driven ETL framework:

Create a Config/Driver table to store task name, source, target, and load type (Full/Incremental).

Use SSIS variables to read those parameters and pass them to your Python script.

Capture audit info (start time, end time, row count, status) in a log table for monitoring.

You can then control the full ETL flow dynamically from that driver table.

Thanks,
Akhil.
Uma 511 Reputation points

2025-11-03T10:43:19.2266667+00:00

@Anonymous : Any example link? Appreciate your help very much.
Anonymous

2026-01-23T09:27:09.1066667+00:00

Hi @Uma ,

Just checking back to see if the official metadata-driven SSIS ETL references shared earlier were helpful. Please let us know if you’re still facing issues accessing the links or if you need a specific Microsoft Docs example (tables, parameters, or control flow).

Happy to help further.

Thanks,
Akhil.

Answer recommended by moderator

0 additional answers

Anonymous

2025-10-31T11:36:07.2166667+00:00

Hi @Uma,

Good question, since you’re using Python within SSIS, this setup belongs to the SQL Server → Integration Services (SSIS) category.

To build a parameter-driven ETL framework:

Create a Config/Driver table to store task name, source, target, and load type (Full/Incremental).

Use SSIS variables to read those parameters and pass them to your Python script.

Capture audit info (start time, end time, row count, status) in a log table for monitoring.

You can then control the full ETL flow dynamically from that driver table.

Thanks,
Akhil.
Uma 511 Reputation points

2025-11-03T10:43:19.2266667+00:00

@Anonymous : Any example link? Appreciate your help very much.
Anonymous

2026-01-23T09:27:09.1066667+00:00

Hi @Uma ,

Just checking back to see if the official metadata-driven SSIS ETL references shared earlier were helpful. Please let us know if you’re still facing issues accessing the links or if you need a specific Microsoft Docs example (tables, parameters, or control flow).

Happy to help further.

Thanks,
Akhil.

Answer 1

Anonymous

Hi @Uma,

Glad that helped. You can check these examples to get started:

These show how to use config tables and dynamic control flow you can extend that with your Python logic for extraction and transformation.

Thanks,
Akhil.

0 comments