Share via

Why is the public ip range download blocked?

karolyi 0 Reputation points
2024-05-01T11:40:10.79+00:00

Hey,

I use a script to download the public ip ranges from azure (https://www.microsoft.com/en-us/download/details.aspx?id=56519).

If I view that URL in my browser, it displays and I am able to download the JSON.

However, if I run my script that would emulate a browser and thus download the ip range (because for some reason the public IP list isn't available under a stable URL), my script gets 404.

There seems no way to reach MS with this issue, so I'm trying the only available avenue which is this one. Does anyone have a clue what's going on here, and why is MS/Azure intentionally making the downloading their IP ranges harder?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,690 questions
0 comments

5 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Ben Gimblett 3,575 Reputation points Microsoft Employee
    2024-05-01T16:02:58.44+00:00

    Hi Thanks for the question The Download link is open, for example in PS1 script (apologies a bit clunky - but it proves the point)

    $downloadPage= Invoke-WebRequest -uri "https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519"

     ($downloadPage.Links | Where-Object {$_.href -Like "*download*" -and $_.href -Like "*ServiceTags_Public*" -and $_.href -Like "*.json"}).outerHTML[0]

    Would give you the anchor with "this months" download link As of just now that would be 

    <a href="https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20240422.json"><span class="loc" locid="b9cd6c93-1571-f3e9-5a11-d35e3d5b5cad" srcid="b9 cd6c93-1571-f3e9-5a11-d35e3d5b5cad">Click here</span></a>

    But a better solution is to use the API which is a lot easier than downloading and parsing the file https://learn.microsoft.com/en-us/azure/virtual-network/service-tags-overview#use-the-service-tag-discovery-api - documented caveats notwithstanding !

  2. karolyi 0 Reputation points
    2024-05-02T10:18:30.5233333+00:00

    Still no answer, so I'll add my reply as a comment:

    I've built a python script for downloading and parsing it, that basically does the same. Yet, all I get is 404 from my server's IP address (IPv6), even on retries.

    Not sure if you're in a position of being able to check if various IP addresses are blocked from downloading, but it seems mines are, yet I never abused the downloads. I only download them once a day, started from cron.

    AFAIK the API requests are for paid users which I'm not, since I don't use azure, I just need the public IP address list, so that doesn't seem to be a solution either.

  3. karolyi 0 Reputation points
    2024-05-02T14:35:08.6966667+00:00

    Ben,

    my server IP is a rented bare metal machine at Hetzner Germany, has an IPv6 /64 range and an IPv4 address, so none of your assumptions apply.

    Yet, all I get is a HTTPError from urllib.

    If you are eager to read some tracebacks, here's one for trying to download the aforementioned URL from my server:

    HTTP Error 404: Not Found

  File "/usr/local/src/project/backend/utils/ip_collector/azure.py", line 104, in _fetch_with_retries
    with urlopen(url=req) as fd:
  File "/usr/local/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/local/lib/python3.10/urllib/request.py", line 557, in error
    result = self._call_chain(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/local/lib/python3.10/urllib/request.py", line 557, in error
    result = self._call_chain(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/local/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

    It would be nice to have a freely and easily available URL to download the Azure/MS IP ranges, other cloud providers don't hide them as much as MS does.I can reproduce this 404 with using curl, or any other tool, but here's one with curl, for clarification's sake:

    # curl -v 'https://www.microsoft.com/en-us/download/details.aspx?id=56519'
* Host www.microsoft.com:443 was resolved.
* IPv6: 2a02:26f0:ab00:385::356e, 2a02:26f0:ab00:383::356e, 2a02:26f0:ab00:3ab::356e, 2a02:26f0:ab00:3ac::356e, 2a02:26f0:ab00:3ad::
356e
* IPv4: 2.19.217.218
*   Trying [2a02:26f0:ab00:385::356e]:443...   
* Connected to www.microsoft.com (2a02:26f0:ab00:385::356e) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):  
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20): 
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=www.microsoft.com
*  start date: Sep 14 17:24:20 2023 GMT
*  expire date: Sep  8 17:24:20 2024 GMT
*  subjectAltName: host "www.microsoft.com" matched cert's "www.microsoft.com"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure RSA TLS Issuing CA 07
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha384WithRSAEncryption
*   Certificate level 1: Public key type RSA (4096/152 Bits/secBits), signed using sha384WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.microsoft.com/en-us/download/details.aspx?id=56519
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.microsoft.com] 
* [HTTP/2] [1] [:path: /en-us/download/details.aspx?id=56519]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /en-us/download/details.aspx?id=56519 HTTP/2
> Host: www.microsoft.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 302
< content-length: 154
< content-type: text/html
< location: https://www.microsoft.com/en-us/download/404Error.aspx
< set-cookie: ApplicationGatewayAffinityCORS=15a834ef837f6653b104fa7d93c22ebc; Path=/; SameSite=None; Secure
< set-cookie: bStore=Y; expires=Thu, 02-May-2024 14:44:34 GMT
< set-cookie: ApplicationGatewayAffinity=15a834ef837f6653b104fa7d93c22ebc; Path=/
< x-dispatcher: dispatcher1westeurope
< x-rtag: ARRPrd
< ms-cv-esi: CASMicrosoftCV1ce2a027.0
< ms-cv: CASMicrosoftCV1ce2a027.0
< strict-transport-security: max-age=31536000  
< tls_version: tls1.3
< ak-forward-host: publish.adobeprod.microsoft.com
< x-edgeconnect-origin-mex-latency: 11
< x-edgeconnect-midmile-rtt: 0
< x-frame-options: SAMEORIGIN
< ms-commit-id: b2bfa6e
< x-content-type-options: nosniff
< accept-ch: Sec-CH-UA-Platform-Version
< x-vhost: publish_microsoft_s
< date: Thu, 02 May 2024 14:44:29 GMT
<
<HTML>
<HEAD>
<TITLE>Error Page</TITLE>
</HEAD>
<BODY>
An error (302 Moved Temporarily) has occurred in response to this request.
</BODY>
</HTML>
  4. karolyi 0 Reputation points
    2024-05-03T11:02:04.07+00:00

    Ben,

    The download fqdn you need is the one I demo in the answer above and you can derive this from the download page links object

    Here's the problem: if I can't load the page that will generate that link (because it only generates a 404), I can't get the generated link.

    Are you able to reach out to your networking/devops dept to get some information as to what could be happening here?

  5. karolyi 0 Reputation points
    2024-05-06T10:53:58.07+00:00

    Hello,

    using the confirmation.aspx link instead of details.aspx seems to work — for now.

    I'll open up another issue when it stops working. Thanks for your help.

    0 comments