Where do you schedule Teams meeting?
If user schedules a meeting in some channels, it will be sent to all the members in the channel. You could delete the channel from the red box and add required attendees in the orange box as needed.

Regarding the meeting invitation sent to all users in your organization, it is due to this team is an org-wide team, which will automatically add everyone in your organization. So, the channel of this team contains all the users in your organization.
Referring to this article when user just wants to schedule a meeting with specific attendees but not all the members in certain channel.
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.