Hi @martinn tan ,
If you don't want users to see a certain library in the left panel, you only need to perform the permission operation.
Settings-->Library settings-->Permissions for this document library(under Permissions and Management section)-->Stop inheriting Permissions, select users who you do not want to see this library, select Remove User Permissions. In this way, the user cannot access the library, and naturally cannot see the library in the left panel.
For my test, below Amy Jiang is site owner, he can see library1 in the left panel.
Amy test has read permission for this site. And when I remove the permission for library1, he cannot see the library1 in the left panel.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.