@Srinivas Rautwar Thanks for reaching out. This is not possible currently. the conditional access cannot allow access to office 365 just from Azure VDI.
You still have options to secure office 365 access by device state (Hybrid AAD joined and compliant devices), and approved apps from conditional access.
But if you are looking for particularly Azure VDI, no option as such.
-----------------------------------------------------------------------------------------------------------------
If the suggested response helped you resolve your issue, please do not forget to accept the response as Answer and "Up-Vote" for the answer that helped you for benefit of the community.