Workflow that logs the completion of certain pipelines into a table

Question

I'm having a lot of difficulty implementing the solutions I had in mind. Previously, I asked for help to create an architecture that would be efficient, easy to maintain, and cost-effective. I received various suggestions, but I can't decide which one to use.

The goal is to create a workflow that logs the completion of certain pipelines into a table. This table should have the following schema:

Column	Type
run_name	string
params	string
run_id	bigint
run_duration	double
start_time	string
end_time	string
state_message_error	string
state	string

My biggest challenge is deciding if this monitoring should be in real-time, which might be more expensive. I need to process around 180 executions per day, and I only want to log pipelines that have the name prefix "pip_premium_".

In summary:

I need to capture the logs of completed pipelines with names starting with "pip_premium_", and insert them into a pre-created Delta table with a specified schema. Additionally, I want this table to have a retention period of 2 to 3 days, as it will be used for monitoring purposes only.

I want to use Azure tools exclusively and ensure that the architecture is clear and understandable to anyone.

Can you help me design this architecture? I would greatly appreciate it!