That's not how it works though. While you can certainly add permissions on the application object, said permissions need to be consented to either by the user, or an admin. When consenting to permissions, they are stamped on the corresponding service principal object, not the application object.
So you're effectively "describing" what permissions the app should get (via the application object), but not making the actual decision to grant them (on the matching service principal). And it's the whole point behind the consent process.
You can always check the set of scopes within the user's access token and if certain permission is missing (or is "extra"), you can block/grant access on the app layer itself. But revoking permissions on the service principal object is a directory-level operation, which you can only perform if the app has been granted the necessary permissions (DelegatedPermissionGrant.ReadWrite.All).