@Dhruvika G , From your description ,I know we want to monitor a URL with pop up. Could you confirm if the pop up is used for authentication?
If yes, we can configure the web application transaction monitor’s authentication setting and choose the authentication method used by the web application. We can see more details in the following link (The configuration can also be applied to SCOM 2016):
Meanwhile, we can also see if URLGenie Management Pack meet your request:
Note: Non-Microsoft link, just for the reference.
Hope it can help.
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.