Logging Pipeline exection

Question

I am creating a parameterized pipeline to load data from blob storage to Cosmos DB . Trying to implement a logging framework for the pipeline execution details. The logging framework which everyone proposes is based on tables in Azure SQL Databases .(I don't have SQL databases)
Please help with information on creating the logging frameworks in Cosmos DB (Pipeline execution start ,end duration ,errors if any ,status etc)

What is the preferred way to store configuration information required for pipeline (File name,destination collection etc)

Answer 1

Hi @Litty Kuriakose ,please go through below links, it might help. Thank you.

https://learn.microsoft.com/en-us/azure/cosmos-db/cosmosdb-monitor-resource-logs

https://medium.com/codex/azure-cosmos-db-the-power-of-logs-troubleshooting-64b619170d63

https://stackoverflow.com/questions/59808079/read-log-files-in-cosmos-db

https://stackoverflow.com/questions/59246715/log-the-conversation-to-azure-cosmosdb-with-nodejs