This isn't offered with every identity provider, but it does look like Azure has a guide for setting up Azure Active Directory single sign-on (SSO) integration with AWS Single-Account Access. This allows your users to be automatically signed-in to AWS Single-Account Access with their Azure AD accounts.
AWS appears to have a corresponding guide.
Okta appears to have a guide for making Azure AD an identity provider but there is no corresponding documentation from the Azure side.