Thank you for providing the confirmation above. Based on my understanding from your issue above.
Question 1: Is this technically feasible for my above requirement?
I think to make this type of connectivity work you will have to create two S2S VPN connections one for Prod env and another one for Non-Prod env. To establish this type of connectivity with a single VPN Gateway you can use the Multi-Site VPN which is a variation of a site-to-site configuration that allows you to connect multiple on-premises sites to a virtual network. You can add additional Site-to-Site (S2S) connections to a VPN gateway that has an existing connection. To use Multi-Site VPN your VPN Gateway should be route based. When you create multiple connections to the same VPN gateway, all VPN tunnels share the available gateway bandwidth.
Question 2: When I read MSFT document says "the traffic from your Azure virtual network to your on-premises network will be routed through both tunnels simultaneously" what does it meant, if so when Zscaler side(3rd party) prod proxy receives the non prod traffic it will drop.
As mentioned in the document you have referenced above the active-active functionality of Azure VPN Gateway is used to provide high availability and although there are two tunnels deployed, Azure VPN Gateway treats it as single connection only and Equal-cost multi-path routing (ECMP) routing is required. Based on my comment above if you can create two connections for each for Prod and Non-prod the traffic will not get exchanged between them.
Question 3: What sort of (conceptually) I would as Zscaler to perform such distinct routing.
I am not familiar with Zscaler, but from Azure VPN perspective if you are using active-active mode then for two connections you will create 4 tunnels in total which will require 4 Local Network gateways and for Multi-Site VPN you need to configure BGP to establish connectivity. We do not have a document specific for Zscaler but I think you can refer to this article related to S2S VPN between Azure and AWS for reference.
Hope this helps! Please let me know if you have any questions. Thank you!