Hi yasukoji,
Thank you for posting your query in our community.
Based on your description, you want to remove and add shared library under Quick access part in your OneDrive.
Generally, the Shared Libraries in the OneDrive for Business will be displayed based on the Frequent sites and Sites which are Followed. So, once you follow the SharePoint site or frequently go to a site, the Document Library will be displayed in the OneDrive for Business after some time.
So, you may go to this SharePoint site to check if it is your followed site now. If yes, you can unfollow it.
If you are not following the site, you may wait for several days, and it may automatically disappear form your Frequent sites section. So, it will also disappear form the OneDrive navigation.
Given this situation, we do understand the inconvenience the problem caused for you. We recommend that you submit requirement for this feature in this OneDrive Feedback portal and many members who has the same requirement with you will vote it in the future.
Appreciate your understanding.
Best regards,
Jazlyn