First of all, if the goal is to have SSO with Azure AD (and Office 365), you don't even need ADFS to make it work seamlessly from outside the organization. So if that's the case, let us know we'll explain!
Then, you can use different authentication methods to log on to ADFS. For example, you could use certificate based authentication as a primary authentication method.
Or, in the same spirit, you could use Azure MFA as a first factor for authentication too. And when you connect to ADFS, you will just have to accept the notification on your phone (no password involved).