No way to disable it, you should handle who can see the videos and the videos that will appear for each user on trending on the permissions
Videos in Microsoft Stream are uploaded to Stream, not to groups or channels.
They are linked to groups or channels or people.
- Verify-in the permission tab of the video properties that only the wanted group is the owner or can display the Video
- Allow everyone in your company to view this video" is off.