Hi @Revanth T
For user contexts, more granular access permissions have not yet been released. That is, you can only restrict the permission to specific users but cannot restrict the access scope of the permission to specific sites or folders.
Hope this helps.
If the reply is helpful, please click Accept Answer and kindly upvote it. If you have additional questions about this answer, please click Comment.