I'm afraid there is no setting for that (I also find this experience quite annoying). Best thing you can do is to post a suggestion over at the Feedback portal, and hope that enough people vote for it for Microsoft to consider adding an option to control it. As the new Calendar experience "borrows" from Outlook, I believe it's best to post under the Outlook forum: https://feedbackportal.microsoft.com/feedback/forum/89a8afa3-2e1c-ec11-b6e7-0022481f8472
Just in case, here's also the Teams one: https://feedbackportal.microsoft.com/feedback/forum/ad198462-1c1c-ec11-b6e7-0022481f8472