You would need to put a service in front of SharePoint to do this. Windows Integrated Authentication naturally requires the WIA popup. Forms-based authentication uses a method other than Windows authentication.
One suggestion to look at would be to use AD FS and Web Application Proxy. This solution would require you to configure Kerberos Constrained Delegation for the SharePoint Web Application. In addition, the WAP server(s) need to be joined to Active Directory. Finally, on AD FS you would set up a non-claims aware relying party.
DNS for your Web Application would be pointed at the WAP server (or load balancer routing to WAP) and users would log in there using an FBA experience but still using Windows auth on the back end.