Why SafeHandle use reference counting?

Question

I'm reading a book (CLR via C#), below is the quote from the book:

SafeHandle-derived classes can prevent someone from trying to exploit a potential security hole. The problem is that one thread could be trying to use a native resource.while another thread tries to free the resource. This could manifest itself as a handle-recycling exploit. The SafeHandle class prevents this security vulnerability by using reference counting. Internally, the SafeHandle class defines a private field that maintains a count. When a SafeHandle-derived object is set to a valid handle, the count is set to 1. Whenever a SafeHandle-derived object is passed as an argument to a native method, the CLR knows to automatically increment the counter. Likewise, when the native method returns to managed code, the CLR knows to decrement the counter.

How does this improve security? Well, if another thread tries to release the native resource wrapped by the SafeHandle object, the CLR knows that it cannot actually release it because the resource is being used by a native function. When the native function returns, the counter is decremented to 0, and the resource will be released.

I'm a little bit confused, let's say we have a thread called threadA, and threadA is executing the C# program that use SafeHandle to read/write a file. Let's say another thread-threadB try to close the file using the IntPtr that associates with the file. So how does the CLR in threadA can stop this happening even though the SafeHandle uses reference counting when threadB already know that IntPtr, threadB can close the file using this IntPtr and threadA can't stop it. Maybe the author means threadB tries to close the file by using SafeHandle in threadA, but SafeHandle in threadA is not visible to threadB. So why SafeHandle use reference counting?