
Yes, the Imap.Accessasapp application permission allows the service principal to access each and every mailbox within the tenant. By default, application permissions are unscoped, i.e. they cover all available resources. In contrast, delegate permissions are limited in scope - they only cover what the logged in user has access to.
If you want to restrict the scope of Imap.Accessasapp or similar permissions, use the RBAC for applications functionality: https://learn.microsoft.com/en-us/exchange/permissions-exo/application-rbac
That's the theory at least, as I haven't tested the IMAP scenario (and there is suspicious lack of mention for anything IMAP-related in the documentation above). I've pinged the PM on this just in case, will let you know if my answer above needs to be corrected.