Hi Vineet, Welcome to MS Q&A
To automatically recognize and monitor the CPU usage of new VMs added to a subscription and load this data into an Azure Data Factory or Databricks pipeline, you can use Azure Monitor auto scale.
Autoscaling rules that use a detection mechanism based on a measured trigger attribute (such as CPU usage) use an aggregated value over time, rather than instantaneous values, to trigger an autoscaling action. By default, the aggregate is an average of the values. This prevents the system from reacting too quickly or causing rapid oscillation. It also allows time for new instances that are automatically started to settle into running mode, preventing additional autoscaling actions from occurring while the new instances are starting up.
You can configure the autoscaling rules and then monitor the performance of your application over time. Use the results of this monitoring to adjust the way in which the system scales if necessary. However, keep in mind that autoscaling is not an instantaneous process. It takes time to react to a metric such as average CPU utilization exceeding (or falling below) a specified threshold.
For more information, you can refer to the following Azure documentation:
Kindly check and let us know if any further questions
Please accept answer , if it helps
Thanks
Deepanshu