Invalid data in monthly cost data export

Question

Invalid data in monthly cost data export

Erik Post 25

Hello,

Maybe someone here can help me out. I'm generating a monthly cost export via the Export function in Cost Management in Azure Portal. This export generates a CSV containing last month's cost data.

The file is in UTF-8 format, which is good. This can be verified by the fact that the file starts with the U+FEFF character, which is EF BB BF encoded in UTF-8 bytes.

However, this same character (and UTF-8 encoded bytes) is present at the start of the first row of data (ie second line of the file).

This has the effect that the first invoice ID in the file becomes invalid: 11 characters and starting with a for users invisible ZERO WIDTH NON-BREAK SPACE.

Is this a bug in the Cost Export functionality, or is there some setting I need to change? I asked Microsoft Azure Support but they are blaming it on the tooling I'm using to process the CSV.

Answer 1

SadiqhAhmed-MSFT 27,606 Microsoft Employee

Hello @Erik Post Thank you for your post and I apologize for the delayed response.

The characters (ZERO WIDTH NON-BREAK SPACE ) you see at the beginning of the file are BOM characters. Although BOM characters are not required for UTF-8 encoding, the exported CSV files has them to aid interoperability with tools such as Excel etc. This is by design.

If you want to get rid of BOM characters, you can open the file in Notepad and save it as UTF-8 only. Notepad supports saving text files in both UTF-8 and UTF-8 with BOM. Excel requires CSV files to have the BOM character to be able to open it.
User's image

Hope this helps. If you have any questions, please write back to us!

If the response helped, do "Accept Answer" and up-vote it

Erik Post 25 Reputation points

2023-05-31T12:48:31.35+00:00

Hi, thanks for your answer. I'm aware of the BOM character and its purpose.

The issue I'm seeing though is that there is a second BOM in the export CSV file, at the start of the second line in the file. So there's a BOM at the start of the file (which is good), but there's also a BOM at the start of the second line in the file (which is not good).

This leads to errors in the automatic processing I'm doing with the CSV file, because the second BOM is interpreted as an extra character (ZERO WIDTH NON-BREAK SPACE) in the invoice id. For human readable purposes this wouldn't be an issue, but for automatich processing it is since it's an extra character that shouldn't be there.

I would attach a sample file displaying the issue, but since it contains private data that's not an option unfortunately :(
SadiqhAhmed-MSFT 27,606 Reputation points Microsoft Employee

2023-05-31T12:51:47.1966667+00:00

Would it be possible to only share the first line and second line BOM characters screenshot with us?
Erik Post 25 Reputation points

2023-05-31T12:59:45.31+00:00

I've added a small screenshot from VS Code with the export file opened.

In the status bar you can see that the file is correctly identified as UTF-8 with BOM (due to the BOM being the first character of the file).

But you can also see the first character at the second line, which is also the BOM. When looking at a binary view of the file this character is encoded as EF BB BF, which is the BOM character.
SadiqhAhmed-MSFT 27,606 Reputation points Microsoft Employee

2023-05-31T13:12:34.4533333+00:00

I shall look into it and get back to you soon.
SadiqhAhmed-MSFT 27,606 Reputation points Microsoft Employee

2023-05-31T18:37:00.86+00:00

@Erik Post You’re correct that this does not look right. We are investigating this issue and we will reach out to you with the investigation result soon.

Meanwhile, can you check if there are more BOM characters in the file? I am suspecting that if your file size is bigger than 1 GB you may see more than one BOM characters (this may be the bug).
Erik Post 25 Reputation points

2023-06-01T09:02:00.92+00:00

@SadiqhAhmed-MSFT thanks for looking into this.

I checked the export file, and it indeed does contain multiple superfluous BOM characters (didn't notice that before). There are 3 superfluous BOMs in total (excluding the correct one at the start of the file). The file is 127MB though.

The first one is at line 2 of the file, the second one at line 85233 and the third one at line 102882. At first glance I cannot find any consistency in when they appear.

It almost looks like multiple UTF-8 BOM encoded character streams are joined without taking care of the BOM at the start of each stream.
SadiqhAhmed-MSFT 27,606 Reputation points Microsoft Employee

2023-06-02T18:58:00.2066667+00:00

@Erik Post We are working on a fix and will update you as soon as this is resolved.
Erik Post 25 Reputation points

2023-06-02T19:55:44.6133333+00:00

Great, much appreciated!

Invalid data in monthly cost data export

1 answer

Activity