Hello, Maybe someone here can help me out. I'm generating a monthly cost export via the Export function in Cost Management in Azure Portal. This export generates a CSV containing last month's cost data. The file is in UTF-8 format, which is good. This can be verified by the fact that the file starts with the U+FEFF character, which is EF BB BF encoded in UTF-8 bytes. However, this same character (and UTF-8 encoded bytes) is present at the start of the first row of data (ie second line of the file). This has the effect that the first invoice ID in the file becomes invalid: 11 characters and starting with a for users invisible ZERO WIDTH NON-BREAK SPACE. Is this a bug in the Cost Export functionality, or is there some setting I need to change? I asked Microsoft Azure Support but they are blaming it on the tooling I'm using to process the CSV.

Hello @ Erik Post Thank you for your post and I apologize for the delayed response. The characters (ZERO WIDTH NON-BREAK SPACE ) you see at the beginning of the file are BOM characters. Although BOM characters are not required for UTF-8 encoding, the exported CSV files has them to aid interoperability with tools such as Excel etc. This is by design. If you want to get rid of BOM characters, you can open the file in Notepad and save it as UTF-8 only. Notepad supports saving text files in both UTF-8 and UTF-8 with BOM. Excel requires CSV files to have the BOM character to be able to open it. Update : @ Erik Post The fix has been deployed across all the regions. I suggest you try and please confirm if you are able to see the success with fix to BOM characters issue. Please write back to us if you have any concerns in this matter! If the response helped, do " Accept Answer " and up-vote it

Invalid data in monthly cost data export

Accepted answer

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-05-31T09:55:22.04+00:00

Hello @Erik Post Thank you for your post and I apologize for the delayed response.

The characters (ZERO WIDTH NON-BREAK SPACE ) you see at the beginning of the file are BOM characters. Although BOM characters are not required for UTF-8 encoding, the exported CSV files has them to aid interoperability with tools such as Excel etc. This is by design.

If you want to get rid of BOM characters, you can open the file in Notepad and save it as UTF-8 only. Notepad supports saving text files in both UTF-8 and UTF-8 with BOM. Excel requires CSV files to have the BOM character to be able to open it.

Update:

@Erik Post The fix has been deployed across all the regions. I suggest you try and please confirm if you are able to see the success with fix to BOM characters issue.

Please write back to us if you have any concerns in this matter!

If the response helped, do "Accept Answer" and up-vote it
Please sign in to rate this answer.

1 person found this answer helpful.
Erik Post 70 Reputation points

2023-05-31T12:48:31.35+00:00

Hi, thanks for your answer. I'm aware of the BOM character and its purpose.

The issue I'm seeing though is that there is a second BOM in the export CSV file, at the start of the second line in the file. So there's a BOM at the start of the file (which is good), but there's also a BOM at the start of the second line in the file (which is not good).

This leads to errors in the automatic processing I'm doing with the CSV file, because the second BOM is interpreted as an extra character (ZERO WIDTH NON-BREAK SPACE) in the invoice id. For human readable purposes this wouldn't be an issue, but for automatich processing it is since it's an extra character that shouldn't be there.

I would attach a sample file displaying the issue, but since it contains private data that's not an option unfortunately :(

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-05-31T12:51:47.1966667+00:00

Would it be possible to only share the first line and second line BOM characters screenshot with us?

Erik Post 70 Reputation points

2023-05-31T12:59:45.31+00:00

I've added a small screenshot from VS Code with the export file opened.

In the status bar you can see that the file is correctly identified as UTF-8 with BOM (due to the BOM being the first character of the file).

But you can also see the first character at the second line, which is also the BOM. When looking at a binary view of the file this character is encoded as EF BB BF, which is the BOM character.

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-05-31T13:12:34.4533333+00:00

I shall look into it and get back to you soon.

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-05-31T18:37:00.86+00:00

@Erik Post You’re correct that this does not look right. We are investigating this issue and we will reach out to you with the investigation result soon.

Meanwhile, can you check if there are more BOM characters in the file? I am suspecting that if your file size is bigger than 1 GB you may see more than one BOM characters (this may be the bug).

Erik Post 70 Reputation points

2023-06-01T09:02:00.92+00:00

@SadiqhAhmed-MSFT thanks for looking into this.

I checked the export file, and it indeed does contain multiple superfluous BOM characters (didn't notice that before). There are 3 superfluous BOMs in total (excluding the correct one at the start of the file). The file is 127MB though.

The first one is at line 2 of the file, the second one at line 85233 and the third one at line 102882. At first glance I cannot find any consistency in when they appear.

It almost looks like multiple UTF-8 BOM encoded character streams are joined without taking care of the BOM at the start of each stream.

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-06-02T18:58:00.2066667+00:00

@Erik Post We are working on a fix and will update you as soon as this is resolved.

Erik Post 70 Reputation points

2023-06-02T19:55:44.6133333+00:00

Great, much appreciated!

Erik Post 70 Reputation points

2023-06-15T11:48:02.6233333+00:00

Hi @SadiqhAhmed-MSFT , I was wondering if there's any update on the progress of the fix for this issue?

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-06-16T11:11:09.1933333+00:00

@Erik Post Not yet. We will start the deployment train this week or next week. Will update the thread once it gets deployed to all regions.

Erik Post 70 Reputation points

2023-06-16T11:12:30.8533333+00:00

Thanks, great to hear a fix will be deployed for this issue!

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-06-27T17:35:00.9633333+00:00

@Erik Post We are in a deployment freeze until July 5<sup>th</sup>. After which we will start deployment.

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-06-27T17:35:22.03+00:00

Hello @Erik Post,

We noticed your recent experience on the Q&A community platform was rated "Low". Thank you for taking time to share your feedback.

We are eager to know what could have been done better to evolve your experience to a "High" rating.

Your candid feedback is very important to us.

Looking forward to your reply.

Erik Post 70 Reputation points

2023-06-29T08:11:08.57+00:00

Hi @SadiqhAhmed-MSFT , I marked the initial reply as not solving my problem, because it did not solve my problem :) But the follow up was good and I'm glad a fix is in the works.

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-07-24T20:05:30.6+00:00

@Erik Post, The fix has been deployed across all the regions. I suggest you try and please confirm if you are able to see the success with fix to BOM characters issue.

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-07-26T11:23:15.1933333+00:00

@Erik Post We haven't heard back from you. Please reply if you have any questions in this matter and we will gladly continue the discussion.

If the issue is resolved, please take a moment to "Accept Answer" and "Up-vote" the post and rate the survey.

Erik Post 70 Reputation points

2023-08-10T11:52:34.5233333+00:00

Hi, apologies for the delay in replying, I was on vacation.

I can confirm the export is correct now, as it only contains a BOM at the start of the file. Thanks for having the issue fixed.

SadiqhAhmed-MSFT 36,881 Reputation points Microsoft Employee

2023-08-11T09:46:04.04+00:00

@Erik Post Thanks for confirming!
Sign in to comment

Invalid data in monthly cost data export

0 additional answers