Hi @Kristian Haahr ,
Personally, I never liked authentication because it's so complex and tiresome. Having said, I can tell what I do know it hopefully it helps.
For your first concern, I wouldn't go the route of creating a client secret per customer. Utilize the client secret per app (service) as stated in best practices. Through the App Registration, you can control who can or can't login through the provider (screenshot below) through the provider. Which means you can utilize delegate permissions for signed in users. Since the delegate permission will use the user to get the access token, you can restrict users that are allowed to receive that token.
As for your second concern, if by client you mean someone running a desktop application, you can leverage the Azure Identity libraries to access the Key Vault. That SDK will utilize a user that has signed into the AD tenant to access the key vault. This does assume that user identity is associated with an access policy. You can place all client id, secret, etc. in the key vault and all the application needs is the KeyVault URI and an identity with said access policy on it. It's the route I would take to avoid storing sensitive information with the app. Another bonus side is you can change app registrations, secret values, etc. without even having to change application code or update application settings.
Kick it up another notch by leveraging App Configuration to retrieve Key Vault reference values based on certain feature flags. That way, all you app knows is an app configuration endpoint and key vault values are retrieved at runtime.