Please remove the default permissions after you break the inheritance for both libraries. After that, assign the unique permission for the AD group. This should remove all unwanted permission at library level.
Modifying the permissions on one library should not affect the ones of another after you break permission inheritance for any one of them.
Any existing permissions will need to be manually removed.
If an Answer is helpful, please click "Accept Answer" and upvote it.
**Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread. **