Dear respected Samadhan Kshirsagar,
Good day! Thank you for posting to Microsoft Community. We are happy to help you.
As per your description, as you know that there is a limit on the number of unique SharePoint online permissions you can set of 50,000 items per list or library. And this limit is officially presented by Microsoft. It is referred to as a "Security scope" limit for lists and libraries.
To answer of your question, "What happen when SharePoint online list Unique permission limit crossed 5000 limit?" when you try to go over the 50,000 limits, SharePoint online will raise you an error "You cannot break inheritance for this item because there are too many items with unique permissions in this list". You can't break permissions inheritance on library or folder. You also can't re-inherit permissions on it. Microsoft also indicates that if you try to declare unique permissions after this limit has been reached you will be blocked from doing so.
The second aspect of this problem is the performance decrease. As the number of unique permissions in a list or library increases, you will notice SharePoint performance decrease even if you are not close to the limit yet. This might happen from around 5,000 unique permissions, although that will depend on your specific SharePoint implementation.
This affects any SharePoint environment that massively uses unique items permissions.
Try to organize your documents differently. The limit is set per list / library, so if you have more libraries, you are less likely to hit the limit.
Appreciate your patience and understanding and thank you for your time and cooperation.
Sincerely,
Waqas Muhammad