If you have Windows devices with Pro OEM preinstalled then attaching the Enterprise E3 license to the primary users of these devices is enough to access virtual Windows instances, e.g. those hosted on AVD, W365, on-premises VDI.
In case these users do NOT have a primary device with Windows Pro OEM preinstalled, then you need to purchase the Windows VDA E3 license for them.
This is higher priced than the Windows Enterprise E3 because as you have already mentioned in the question as it is just an upgrade license for a qualifying OS.
If the latter is not present the Windows VDA E3 includes the Windows Pro and the Enterprise upgrade licenses.
One more thing to add here is, that if you want to run Office applications in the hosted Windows desktop environment you need Microsoft 365 Apps licenses in addition. For AVD additional charges apply for Azure consumption like compute, storage, network etc.
Hope this helps.
Please 'Accept as answer' if the provided information is helpful, so that it can help others in the community looking for help on similar topics.