To let anyone who may come across this while trying to generate their own OData serialisation.
The problem turned out to be nothing to do with the textual format of my response. I had that right.
However, I had missed that the standard says that OData responses SHOULD include a "OData-Version" HTTP header (odata-v4.01-part1-protocol.html)
It appears the Mashup engine treats that as MUST (anyone who's ever had to deal with RFCs will know the importance of the phrasing)
Once the Http header is added and I checked the "Include open type columns" checkbox in the OData feed advanced options I was able to access the data in PowerQuery (still need to test in PowerBI and Dynamics, but feeling a lot more confident now)
If you've got a reference in your code to Microsoft OData nugets you should be able to handle it with the following code:
response.Headers.TryAddWithoutValidation( "OData-Version", ODataUtils.ODataVersionToString(ODataVersion.V4));
Still some more testing to do, but hopefully this helps others in the future
All the best
Mark