I have tested the situation you provided and check in/check out feature works well in my document library.
Generally, such issue may happen due to unexpected closing of the file and the file cache is still stored in system or browser showing the file is still in use.
Please check the following tips and share the result.
1.Go to Library Settings -> Manage files which have no checked in versions under Permissions and Management -> Check whether the file is there.
2.Check whether the file is shared, other users open it and don’t close it. If yes, let them close it and re-sign into the site library to do a test.
3.Clear the browser cache or use another browser to test.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.