You could set unique permission for document library to make users can access the site but cannot access the document library.
Here are steps:
1.Go to Site permissions -> Grant users with read permission.
2.Go to the Library settings -> Permissions for this document library -> Stop Inheriting Permissions -> Remove users except the site owners or site administrators.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.