I finally found real reason as to why it was happening. This was happening in IE:
http://web.archive.org/web/20150906055648/http://bugs.jquery.com/ticket/10660
Solution is to use following before your AJAX request.
$.support.cors = true;
Also your request and Web API must use same protocol i.e. you cannot call HTTP while you are on HTTPS. So in my case I changed my Web API to be hosted on HTTPS and now it works.