@Thomas Gusset Thank you for reaching out to us, As I understand you are looking for details on why public key of EC key pair cant be downloaded from the portal.
Discussed your ask with my team, here is the update I have - You don't see a download option on GUI is expected, EC keys aren't fully supported through GUI which means some operations are only available through other interfaces such as PSH/CLI. you may download the public key via CLI - https://learn.microsoft.com/en-us/cli/azure/keyvault/key?view=azure-cli-latest#az-keyvault-key-download
Let me know if you have any further questions, feel free to post back.
Please remember to "Accept Answer" if answer helped, so that others in the community facing similar issues can easily find the solution.