Hello @xyz_chatter and welcome to Microsoft Q&A.
The only possible disadvantage I can think of, is in some scenarios using ADLS as staging can result in faster loads when using polybase.
Other than that, I think you should be fine skipping ADLS.
Please let me know if this answered your query, or if you have other concerns. Thank you!