For the past couple of months, a Blob Storage Account on my personal Subscription has been hit with transactions that aren't mine, around 300K GetBlobProperties per time. Usually, they're every seven hours, but they occasionally stop for a couple of days at a time, and have recently stopped for seven days, but they've started again. It's a small, mostly Archive Storage Account with 1.5 TB and around 300K blobs. I've tried everything that I can think of to stop these transactions, as they're definintely unwanted (and costing me a comparatively lot of money): Renew the Storage Keys (and haven't given them to anything) Limited access to specific subnets (and then didn't configure any) Disabled access even to Azure exception services The Storage Account is the target of my laptop's backup job (MSP360), but that laptop hasn't been on in months, so can't be the source. I don't have any VMs or any other resources, and I don't have any special tracking configured (to the best of my knowledge). My reading of the logs (when I enabled them) was that the calls are to .SuccessE2ELatency and .SuccessServerLatency, but I've not found anything online to hint at what it could be. The source IPs were 10.218.0.13,14,15, but those are non-routable so not much help. Any ideas for tracking down the root cause would be appreciated! I'm relatively new to Azure, but have a general idea about cloud (I've done the Fundamentals course). Could it be another one of my work-linked accounts getting in somehow? I can't see anything in the permissions...

I assume you are using the StorageBlobLogs as you mentioned that you enabled logging. Have you checked the RequesterAppId and RequesterObjectId columns to determine what application might be accessing it? As the ip is a private ip-address range this is most likely an Azure Service. You mentioned that you disabled access even to Azure exception services. Do you mean that you removed the checkbox of "Allow Azure services on the trusted services list to access this storage account"?

Random Storage Account GetBlobProperties calls

Andrew Gough 1

For the past couple of months, a Blob Storage Account on my personal Subscription has been hit with transactions that aren't mine, around 300K GetBlobProperties per time. Usually, they're every seven hours, but they occasionally stop for a couple of days at a time, and have recently stopped for seven days, but they've started again. It's a small, mostly Archive Storage Account with 1.5 TB and around 300K blobs. I've tried everything that I can think of to stop these transactions, as they're definintely unwanted (and costing me a comparatively lot of money):

Renew the Storage Keys (and haven't given them to anything)
Limited access to specific subnets (and then didn't configure any)
Disabled access even to Azure exception services

The Storage Account is the target of my laptop's backup job (MSP360), but that laptop hasn't been on in months, so can't be the source.

I don't have any VMs or any other resources, and I don't have any special tracking configured (to the best of my knowledge). My reading of the logs (when I enabled them) was that the calls are to .SuccessE2ELatency and .SuccessServerLatency, but I've not found anything online to hint at what it could be. The source IPs were 10.218.0.13,14,15, but those are non-routable so not much help.

Any ideas for tracking down the root cause would be appreciated! I'm relatively new to Azure, but have a general idea about cloud (I've done the Fundamentals course). Could it be another one of my work-linked accounts getting in somehow? I can't see anything in the permissions...

1 answer

Ruud van den Hooff 81 Reputation points Microsoft Employee

2022-01-26T11:01:16.317+00:00

I assume you are using the StorageBlobLogs as you mentioned that you enabled logging. Have you checked the RequesterAppId and RequesterObjectId columns to determine what application might be accessing it?

As the ip is a private ip-address range this is most likely an Azure Service. You mentioned that you disabled access even to Azure exception services. Do you mean that you removed the checkbox of "Allow Azure services on the trusted services list to access this storage account"?
Please sign in to rate this answer.
Andrew Gough 1 Reputation point

2022-01-26T13:05:52.73+00:00

Thanks for replying! I've turned on various logging, including in Monitoring and Monitoring (classic) using Diagnostic Settings, but I can't see a RequesterAppId in the logs. To narrow it down, which logging records that? I have the IPs in a file with the name type 000004.log (stored in $logs$ in the Storage Account itself) and found the Success[...]Latency entries in logs archived to another Storage Account in the format PT1H.json. There are so many options, I'm getting a bit confused with what's coming from where...

As for the checkbox, yes "Allow Azure services on the trusted services list to access this storage account" is what I disabled and I still got the transactions. I've since reenabled it. What's weird is that it changed from every 7 hours or so, with only occasional misses over the weeks, it paused between the 16th and 25th February.

Curiously, I note that the two transaction types are those listed in Performance under Insights for the Storage Account. I don't know if this is a coincidence or if there's a way to reign that in (or if that's just passively monitoring whatever else is happening).

Again, I appreciate your taking time to have a think about this.

Ruud van den Hooff 81 Reputation points Microsoft Employee

2022-01-26T13:46:26.473+00:00

You can use the diagnostic settings as well to send to log analytics and from there you might have a bit more information.
This is a document that goes into detail how to do that : https://learn.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage?tabs=azure-portal#creating-a-diagnostic-setting
You don't need the metrics, in your case the log is enough.
When the data is eventually logged in the StorageBlobLogs table you can use this section to find out how you can create some queries : https://learn.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage?tabs=azure-portal#accessing-logs-in-a-log-analytics-workspace

In your case it would be easy to find the records as you say there shouldn't be anything actually accessing the data and therefore you don't need to filter it. This means that the below kusto query should be more than enough at first:

StorageBlobLogs | where TimeGenerated > ago(1d) | limit 10

This is the reference to the StorageBlobLogs table and all of the available columns. That gives you a better idea what the data is: https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/storagebloblogs

Good luck with that. I am actually curious what it is so please update us.

Andrew Gough 1 Reputation point

2022-01-26T16:03:07.257+00:00

Thanks for the idea! I've added a Log Analytics workspace to the diagnostics and will sit back to see what happens... Hopefully I won't have too long to wait!

All the best for now :-)

Andrew Gough 1 Reputation point

2022-02-03T15:00:42.197+00:00

I ran the query in Monitor > Logs at 14:30 today (3rd Feb) and the latest log entries are all for GetContainerProperties, of which my Storage Account Insights show I've had around 30 transactions in the past 4 hours. These transactions are showing that they were authorised by Account Key despite my having refreshed it and not distributed it, and which come up blank for the RequesterAppID and ObjectID columns. But there's no mention of the 476000 GetBlobProperties transactions indicated on Insights that occurred between 13:57 and 14:28. I've modified the "ago" check to 1 hour, and set the limit to 100, but only 16 rows are returned, so I don't know the GetBlobProperties are.

The URI for the GetContainerProperties fields is https://[storageaccount].blob.core.windows.net/$root?restype=container and come from a variety of IPs (eg 10.218.0.15:59781, 100.77.168.60:59846, 10.218.0.13:61219, 10.72.124.26:52241).

Do you have any other ideas for investigating? Either way, thank you for your help! If I can't sort this, I'm going to have to burn the Subscription down and start again (and considering that lots of this data is <180 days old sat in Archive, it's going to cost me just to delete it!

Ruud van den Hooff 81 Reputation points Microsoft Employee

2022-02-04T07:52:11.603+00:00

All of the IP-addresses you see are private IP address ranges or a bogon IP address (100.77.168.60) . Meaning that these are most likely Microsoft Azure resources itself, also the AppId and ObjectId showing blank indicates that a bit more. I would recommend in raising a support case with Microsoft in getting more details here as your logs do not provide it. The only one in providing more insights are them. Keep us posted, I am very interested in what this could be!

Andrew Gough 1 Reputation point

2022-02-07T09:57:01.407+00:00

I've only got a basic account, so MS's response was "You have everything you need in the logs. If you no longer require the Storage Account due to the cost, please delete it." (Paraphrasing.) My feeling is that this is a bug of some kind, and it's costing me money!

So, I've an option to pay £29 (2x months of these erroneous charges) and I might get a resolution, or delete everything including the subscription, but that's a hassle as that's where my personal backups are stored!

I need to have a think about how much I care about getting to the bottom of this. If I do take them up on support, I will post the outcome back here, but until then I very much appreciate your ideas: I've learned things about logging and analysis :-)
Sign in to comment