Data Factory monitoring by inserting data table

Question

Hello, I would like to know the best way to insert Datafactory activity logs into my Databricks delta table, so that I can use dashbosrd and create monitoring in Databricks itself , can you help me? I would like every 5 minutes for all activity logs in the data factory to be inserted into the Databricks delta table, that is, if 10 pipelines are completed, the logs of these 10 are inserted into the delta. Please note: no logs cannot be missing. I want a solution that is considered good practice, economical and efficient, can you help me with this?

Answer 1

Hello @Hanna

One way to achieve this is by using Azure Stream Analytics to stream the Data Factory activity logs to Databricks Delta table. Here are the high-level steps to achieve this:

1. Create a Delta table in Databricks to store the Data Factory activity logs.
2. Create an Azure Stream Analytics job to stream the Data Factory activity logs to the Delta table.
3. Configure the Stream Analytics job to run every 5 minutes.
4. Verify that the logs are being inserted into the Delta table. Here is an example of how you can create a Delta table in Databricks:

CREATE TABLE activity_logs ( eventTime timestamp, operationName string, status string, activityId string, pipelineName string, runId string, message string ) USING DELTA LOCATION '/mnt/delta/activity_logs';

Once you have created the Delta table, you can use Azure Stream Analytics to stream the Data Factory activity logs to the table. Here is an example of how you can configure the Stream Analytics job:

5. In the Azure portal, navigate to your Stream Analytics job.
6. Click on Inputs and then click on Add.
7. Select Data Factory as the input source and configure the input settings.
8. Click on Outputs and then click on Add.
9. Select Databricks Delta as the output sink and configure the output settings.
10. Click on Query and then enter the following query:

SELECT CAST(eventTime AS timestamp) AS eventTime, operationName, status, activityId, pipelineName, runId, message INTO activity_logs FROM DataFactoryInput

11. Click on Save and then click on Start to start the Stream Analytics job. Once the job is running, it will stream the Data Factory activity logs to the Delta table every 5 minutes. You can then use Databricks to create a dashboard and monitor the activity logs in real-time.

---

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.