Why is the public ip range download blocked?

karolyi 0

Hey,

I use a script to download the public ip ranges from azure (https://www.microsoft.com/en-us/download/details.aspx?id=56519).

If I view that URL in my browser, it displays and I am able to download the JSON.

However, if I run my script that would emulate a browser and thus download the ip range (because for some reason the public IP list isn't available under a stable URL), my script gets 404.

There seems no way to reach MS with this issue, so I'm trying the only available avenue which is this one. Does anyone have a clue what's going on here, and why is MS/Azure intentionally making the downloading their IP ranges harder?

3 answers

Ben Gimblett 3,580 Microsoft Employee

Hi Thanks for the question The Download link is open, for example in PS1 script (apologies a bit clunky - but it proves the point)

$downloadPage= Invoke-WebRequest -uri "https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519"

 ($downloadPage.Links | Where-Object {$_.href -Like "*download*" -and $_.href -Like "*ServiceTags_Public*" -and $_.href -Like "*.json"}).outerHTML[0]

Would give you the anchor with "this months" download link As of just now that would be

<a href="https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20240422.json"><span class="loc" locid="b9cd6c93-1571-f3e9-5a11-d35e3d5b5cad" srcid="b9 cd6c93-1571-f3e9-5a11-d35e3d5b5cad">Click here</span></a>

But a better solution is to use the API which is a lot easier than downloading and parsing the file https://learn.microsoft.com/en-us/azure/virtual-network/service-tags-overview#use-the-service-tag-discovery-api - documented caveats notwithstanding !

karolyi 0 Reputation points

2024-05-01T16:29:30.04+00:00

Hey,

I've built a python script for downloading and parsing it, that basically does the same. Yet, all I get is 404 from my server's IP address (IPv6), even on retries.

Not sure if you're in a position of being able to check if various IP addresses are blocked from downloading, but it seems mines are, yet I never abused the downloads. I only download them once a day, started from cron.

AFAIK the API requests are for paid users which I'm not, since I don't use azure, I just need the public IP address list, so that doesn't seem to be a solution either.
karolyi 0 Reputation points

2024-05-02T10:18:30.5233333+00:00

Still no answer, so I'll add my reply as a comment:

I've built a python script for downloading and parsing it, that basically does the same. Yet, all I get is 404 from my server's IP address (IPv6), even on retries.

Not sure if you're in a position of being able to check if various IP addresses are blocked from downloading, but it seems mines are, yet I never abused the downloads. I only download them once a day, started from cron.

AFAIK the API requests are for paid users which I'm not, since I don't use azure, I just need the public IP address list, so that doesn't seem to be a solution either.
Ben Gimblett 3,580 Reputation points Microsoft Employee

2024-05-02T10:36:18.1766667+00:00

The public IP range is not blocked. The download site is public-anonymous and the REST API site is public but (if memory serves) requires a token as it's "management plane" operation

If you're getting a 404 please check your location , for example if you work on a corporate machine within an office or connected work-from-home environment then you may be subject to forward proxy or firewall rules and restrictions

karolyi 0

Ben,

my server IP is a rented bare metal machine at Hetzner Germany, has an IPv6 /64 range and an IPv4 address, so none of your assumptions apply.

Yet, all I get is a HTTPError from urllib.

If you are eager to read some tracebacks, here's one for trying to download the aforementioned URL from my server:

HTTP Error 404: Not Found

  File "/usr/local/src/project/backend/utils/ip_collector/azure.py", line 104, in _fetch_with_retries
    with urlopen(url=req) as fd:
  File "/usr/local/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/local/lib/python3.10/urllib/request.py", line 557, in error
    result = self._call_chain(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/local/lib/python3.10/urllib/request.py", line 557, in error
    result = self._call_chain(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/local/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

It would be nice to have a freely and easily available URL to download the Azure/MS IP ranges, other cloud providers don't hide them as much as MS does.I can reproduce this 404 with using curl, or any other tool, but here's one with curl, for clarification's sake:

# curl -v 'https://www.microsoft.com/en-us/download/details.aspx?id=56519'
* Host www.microsoft.com:443 was resolved.
* IPv6: 2a02:26f0:ab00:385::356e, 2a02:26f0:ab00:383::356e, 2a02:26f0:ab00:3ab::356e, 2a02:26f0:ab00:3ac::356e, 2a02:26f0:ab00:3ad::
356e
* IPv4: 2.19.217.218
*   Trying [2a02:26f0:ab00:385::356e]:443...   
* Connected to www.microsoft.com (2a02:26f0:ab00:385::356e) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):  
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20): 
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=www.microsoft.com
*  start date: Sep 14 17:24:20 2023 GMT
*  expire date: Sep  8 17:24:20 2024 GMT
*  subjectAltName: host "www.microsoft.com" matched cert's "www.microsoft.com"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure RSA TLS Issuing CA 07
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha384WithRSAEncryption
*   Certificate level 1: Public key type RSA (4096/152 Bits/secBits), signed using sha384WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.microsoft.com/en-us/download/details.aspx?id=56519
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.microsoft.com] 
* [HTTP/2] [1] [:path: /en-us/download/details.aspx?id=56519]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /en-us/download/details.aspx?id=56519 HTTP/2
> Host: www.microsoft.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 302
< content-length: 154
< content-type: text/html
< location: https://www.microsoft.com/en-us/download/404Error.aspx
< set-cookie: ApplicationGatewayAffinityCORS=15a834ef837f6653b104fa7d93c22ebc; Path=/; SameSite=None; Secure
< set-cookie: bStore=Y; expires=Thu, 02-May-2024 14:44:34 GMT
< set-cookie: ApplicationGatewayAffinity=15a834ef837f6653b104fa7d93c22ebc; Path=/
< x-dispatcher: dispatcher1westeurope
< x-rtag: ARRPrd
< ms-cv-esi: CASMicrosoftCV1ce2a027.0
< ms-cv: CASMicrosoftCV1ce2a027.0
< strict-transport-security: max-age=31536000  
< tls_version: tls1.3
< ak-forward-host: publish.adobeprod.microsoft.com
< x-edgeconnect-origin-mex-latency: 11
< x-edgeconnect-midmile-rtt: 0
< x-frame-options: SAMEORIGIN
< ms-commit-id: b2bfa6e
< x-content-type-options: nosniff
< accept-ch: Sec-CH-UA-Platform-Version
< x-vhost: publish_microsoft_s
< date: Thu, 02 May 2024 14:44:29 GMT
<
<HTML>
<HEAD>
<TITLE>Error Page</TITLE>
</HEAD>
<BODY>
An error (302 Moved Temporarily) has occurred in response to this request.
</BODY>
</HTML>

Ben Gimblett 3,580 Reputation points Microsoft Employee

2024-05-02T18:51:04.07+00:00

Hi - thanks for the follow-up comments. If you follow the curl output carefully you'll see the DNS resolved correct, the socket connects, the SSL handshake is successfull ....all this indicates your client on your Hetzna machine can connect successfully. Nothing is blocked. If you follow this further you'll see the URL you request is being answered by a 302 (redirect) to an error page . I am not 100% sure why - if I open the same fqdn on my browser I also get a redirect but to a landing page. The download fqdn you need is the one I demo in the answer above and you can derive this from the download page links object
karolyi 0 Reputation points

2024-05-03T11:02:04.07+00:00

Ben,

The download fqdn you need is the one I demo in the answer above and you can derive this from the download page links object

Here's the problem: if I can't load the page that will generate that link (because it only generates a 404), I can't get the generated link.

Are you able to reach out to your networking/devops dept to get some information as to what could be happening here?
Ben Gimblett 3,580 Reputation points Microsoft Employee

2024-05-03T11:15:50.7266667+00:00

Try using the confirmation page link instead of details

I think the PS1 Invoke-WebRequest command is doing more or behaving differently - as I can get that to work with the "details" page, but with curl it does not, I had to use the "confirmation" page instead
Ben Gimblett 3,580 Reputation points Microsoft Employee

2024-05-03T11:18:59.8766667+00:00

PS the SO answer I used for the CURL GREP is here https://stackoverflow.com/questions/28798014/is-there-a-way-to-automatically-and-programmatically-download-the-latest-ip-rang

karolyi 0 Reputation points

2024-05-06T10:53:58.07+00:00

Hello,

using the confirmation.aspx link instead of details.aspx seems to work — for now.

I'll open up another issue when it stops working. Thanks for your help.
Please sign in to rate this answer.
PRADEEPCHEEKATLA-MSFT 79,141 Reputation points Microsoft Employee

2024-05-20T07:24:36.6533333+00:00

@karolyi - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .
Sign in to comment
PRADEEPCHEEKATLA-MSFT 79,141 Reputation points Microsoft Employee

2024-05-20T07:24:10.3466667+00:00

@karolyi - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: Why is the public ip range download blocked?

Solution: The issue is resolved using the confirmation.aspx link instead of details.aspx seems to work — for now.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Why is the public ip range download blocked?

3 answers