Yes you can, that is how it works by default... the token is used for the 2nd factor (instead of phone) after entering user/password. Make sure the hardware token option is set as the default method under the user settings at https://myaccount.microsoft.com and the hardware token option is enabled in the MFA settings portal: https://account.activedirectory.windowsazure.com/UserManagement/MfaSettings.aspx
Do you need ADFS for 365 or is it just there 'because'? I would look at running AD Connect and moving to password hash or pass-through auth for 365 (in a planned way of course!). You can still use ADFS if you require it for other apps. You can even create an Azure AD group and do a staged migration which allows you to put some test users into the group so they can test the change before you move all users across. https://portal.azure.com/#blade/Microsoft_AAD_IAM/StagedRolloutEnablementBladeV2
If you have to keep ADFS make sure it is not using on-premises MFA server, move to using Azure MFA:
https://learn.microsoft.com/en-us/azure/active-directory/authentication/how-to-migrate-mfa-server-to-azure-mfa-user-authentication