We discussed this with the engineering team and got the below reply -
The permission is granted to the application instead of user. So we should use application identity to get the token to call graph API instead of user. Here is the link to use application credential https://github.com/OfficeDev/TeamsFx/tree/main/packages/sdk#invoke-graph-api-without-user-application-identity . One thing needs to be noticed is since we are using application credential, the token could not be got from front end. There should be an backend service to store the app credential and use the credential to exchange token.
FYI - Could you please post the Teams toolkit or teamfx questions on https://github.com/OfficeDev/TeamsFx/issues