Alright, after MUCH further digging, I can confirm the bottleneck is from serving files from the local Azure Storage Blob Emulator running on my local machine.
The typical time to retrieve and serve the file from Azure Local Emulator is about 120-150ms. Whereas if I serve files directly from disk, the time drops to 15-20ms. Running some profiling the majority of those 15-20ms are spent in EntityFramework pulling the information required to assert authorization to view the file.
I sure hope this improves when actually deployed to production. I think I read somewhere that the emulator is going away in favor of a new cross platform version.
In any case, I am satisfied with all my research here now and have the answer. The problem was not ASP.NET and in fact, it was working very well and very intuitively. I can not say enough great things about this product and the team behind it.