I got it working through a tweak. I changed the code to directly call the read API version 3.2 through REST call and then use computer vision client library methods to read the results using the operation location received from the read rest call. So it is working with a mix of both i.e directly hitting Read REST API call (not using library methods) and using the analyse method provided by the library.
I didn't want to use the REST call directly because I thought the library should take care of the all heavy lifting but seems like the library is having some issue with Read API calls.
I used the same byte array for the REST APIs and it works.