Hi Kai Deian,
Welcome to community.
I go through the post carefully. It seems like you want to remove user access to file/folder shared through sharing link (Share>Send link>enter user name>Send) in SharePoint.
Unfortunately I'm afraid there is no out of box way in SharePoint to remove the sharing link shared by this way. And I also understand "specific people" link is not a suitable method for the scenario on your side. I do understand it has affected your daily work and feel very sorry about it.
Given this, I sincerely recommend sharing your valuable feedback for this behavior to Tech Community, it is a specific platform to let SharePoint developers know your requirements and comments, help to improve the product and make it better for you and others.
Sorry for the inconvenience again. Your understanding is highly appreciated.
Best regards,
Tina