Hello @Shrikant Bhagwat
I understand that you want to call the SharePoint API using Microsoft Graph and want the application to access files only in a specific SharePoint site.
To achieve it, you need to grant only Sites.Selected API permission to the Microsoft Entra ID application:

Now, create the permission so that the application so that the app can access only one site not all:
POST https://graph.microsoft.com/v1.0/sites/SiteID/permissions
Content-Type: application/json
{
"roles": ["write"],
"grantedToIdentities": [{
"application": {
"id": "ClientIdofApp",
"displayName": "Sharepointapp"
}
}]
}

Generate the access token:
POST https://login.microsoftonline.com/TenantID/oauth2/v2.0/token
client_id: ClientID
client_secret: Secret
scope: https://graph.microsoft.com/.default
grant_type:client_credentials

Using the above access token, I am able to access the drive of the particular site successfully:
GET https://graph.microsoft.com/v1.0/sites/SiteID/drive

If I try to access other SharePoint Site, for which the application does not have access, 403 error occurs:

To download files, refer: Download driveItem content - Microsoft Graph v1.0 | Microsoft
Let me know if you’re still running into issues or have additional questions.
Hope this helps!
If this answers your query, do click Accept Answer and Yes for was this answer helpful, which may help members with similar questions.

If you have any other questions or are still experiencing issues, feel free to ask in the "comments" section, and I'd be happy to help.