Hello @Rohit Boddu ,
Unfortunately, Azure Databricks does not natively support sending log data to Azure monitor.
To get metric data related to an Apache Spark job, stages of the job, and tasks, you need to use a library for this functionality is available in GitHub.
You can use the library enables logging of Azure Databricks service metrics as well as Apache Spark structure streaming query event metrics. Once you've successfully deployed this library to an Azure Databricks cluster, you can further deploy a set of Grafana dashboards that you can deploy as part of your production environment.
This article shows how to set up a Grafana dashboard to monitor Azure Databricks jobs for performance issues.
Hope this helps. Do let us know if you any further queries.