If you want to invite only one user from specific organization, you can directly invite user as guest to your tenant.
This way you can just enter the email address of external users whom you want to invite to your Azure AD.
Below article explains on how you can invite a guest user to your Azure active directory,
https://learn.microsoft.com/en-us/azure/active-directory/external-identities/b2b-quickstart-add-guest-users-portal
Do let me know if you have any further questions on this.
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.