No, you cannot configure password complexity, best you can do is toggle the "strong password" requirements, but that still leaves you at 8 char minimum length. You can however redirect the auth process to on-premises AD (pass-trough authentication/AD FS) or external IdP (federation) and have the complexity policies therein apply.
Enabling/forcing multi-factor authentication should alleviate password complexity concerns though, and I strongly recommend you look into implementing MFA, if you haven't done so already.
For the sake of completeness, here are the minimum requirements for Azure AD passwords: https://learn.microsoft.com/en-us/azure/active-directory/authentication/concept-sspr-policy#password-policies-that-only-apply-to-cloud-user-accounts