Thanks for reaching out to Q&A.
Yes. You can scaledown the App service plan safely. There might be an app restart as the backend instances will change during this process. You can observe this behavior in the App restart detector present in Diagnose and Solve problems blade.
In future, I would suggest you to configure Autoscale rule so that App service plan scales out automatically when there is high CPU usage
Autoscaling : https://learn.microsoft.com/en-us/azure/architecture/best-practices/auto-scaling
I hope this helps!
Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.