{"The remote server returned an error: (403) Forbidden."}

Question

I'm writing an .NET application that uses HttpClient to connect to the API and retrieve video.
The Documentation provided for "Cookie-based authentication" details the following to login:

1. Call GET /api/getNonce
2. In response you'll get a JSON object with realm and nonce (nonce is a session key for this user)
3. Calculate authentication hash auth_digest, using realm and nonce (see algorithm below)
4. Call and pass the "auth" parameter in the json request body GET /api/cookieLogin
5. Server will check authentication and set session cookies

Which then expands on "Calculating Authentication Hash" with the following steps:

• Call GET /api/getNonce
• In response you'll get a JSON object with realm and nonce
• Translate user's username to the lower case
• Check the required method ("GET" for HTTP GET requests, "POST" for HTTP POST requests, "PLAY" for RTSP etc) of the API call that requires the authentication hash.
• digest = md5_hex(user_name + ":" + realm + ":" + password)
• partial_ha2 = md5_hex(method + ":")
• simplified_ha2 = md5_hex(digest + ":" + nonce + ":" + partial_ha2)
• auth_digest = base64(user_name + ":" + nonce + ":" + simplified_ha2)
• Here auth_digest is the required authentication hash

I use this Article written in C# as a reference, converted it to VB.NET, change a little bit of code in it according to the requirement above and then implemented it into my program. But I always get an error like this;

"The remote server returned an error: (403) Forbidden."

When I debug the program, the problem appears when reading DigestAuthFixer.vb class in GrabResponse Function on this line response = CType(request2.GetResponse(), HttpWebResponse) . I suspect that "auth_digest" didn't return the correct value so the web service to deny the request with a 403 error. My question is how to properly implement digest authentication using VB.Net ? Are there any standard methods or do I have to do it from scratch? Thanks in advance.

This is the code that I use to connect to the API and retrieve the video:

Download HTTPClient

Private Shared _host As String = "http://127.0.0.1:7001"  
Private Shared _username As String = "user"  
Private Shared _password As String = "password"  
Shared ReadOnly client As HttpClient = New HttpClient()  
  
Public Async Function Download_HttpClient(ByVal cameraID As String _  
                                          , ByVal videoDT As String _  
                                          , ByVal duration As String _  
                                          , ByVal resolution As String _  
                                          , ByVal path As String) As Task(Of String)  
  
        Dim cookieContainer As New CookieContainer  
        Dim httpClientHandler As New HttpClientHandler  
        httpClientHandler.AllowAutoRedirect = True  
        httpClientHandler.UseCookies = True  
        httpClientHandler.CookieContainer = cookieContainer  
        Dim httpClient As New HttpClient(httpClientHandler)  
        Dim err As Boolean = False  
  
httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.53")  
httpClient.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml")  
httpClient.DefaultRequestHeaders.Add("Accept-Charset", "ISO-8859-1")  
  
Try  
  
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 Or SecurityProtocolType.Tls12 Or SecurityProtocolType.Tls11 Or SecurityProtocolType.Tls  
            'CALL GET /api/getNonce  
            Dim login_url As String = _host + "/api/getNonce"  
            Dim login_parameter As String = "?user_name =" + _username + "&password=" + _password  
            Dim login_response As HttpResponseMessage = Await httpClient.GetAsync(login_url + login_parameter)  
            Dim login_contents As String = Await login_response.Content.ReadAsStringAsync()  
  
            'CALCULATE AUTHENTICATION HASH  
            Dim dir As String = "/media/" + cameraID + ".mkv?pos=" + videoDT + "&duration=" + duration + "&resolution=" + resolution '"/api/getNonce"   
            Dim digest As NUI.DigestAuthFixer = New NUI.DigestAuthFixer(_host, _username, _password)  
            Dim auth As String = digest.GrabResponse(Dir)  
  
            'CALL POST /api/cookieLogin  
            Dim client As HttpClient = New HttpClient  
            client.DefaultRequestHeaders.Accept.Add(New System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"))  
            Dim postData As New List(Of KeyValuePair(Of String, String))  
            postData.Add(New KeyValuePair(Of String, String)("auth_digest ", auth))  
            Dim content As New FormUrlEncodedContent(postData)  
            content.Headers.ContentType = New Headers.MediaTypeHeaderValue("application/x-runtime-guid")  
  
            Dim response As HttpResponseMessage = client.PostAsync(New Uri(_host + "/api/cookieLogin"), content).Result  
            If response.IsSuccessStatusCode Then  
                MsgBox("Call CookieLogin Successfully")  
            Else  
                MsgBox(response)  
            End If  
  
        Catch ex As Exception  
            MessageBox.Show("Invalid response from the server due to connection limitation or firewall blocking your request")  
            Return ex.ToString()  
        End Try  
  
        Return "Download test"  
  
    End Function  

DigestAuthFixer.vb class

Imports System  
Imports System.Collections.Generic  
Imports System.Linq  
Imports System.Text  
Imports System.Security.Cryptography  
Imports System.Text.RegularExpressions  
Imports System.Net  
Imports System.IO  
Imports System.Net.Http  
  
Namespace NUI  
    Public Class DigestAuthFixer  
  
        Private Shared _host As String  
        Private Shared _username As String  
        Private Shared _password As String  
        Private Shared _realm As String  
        Private Shared _nonce As String  
        Private Shared _qop As String  
        Private Shared _cnonce As String  
        Private Shared _cnonceDate As DateTime  
        Private Shared _nc As Integer  
  
        Public Sub New(ByVal host As String, ByVal username As String, ByVal password As String)  
            _host = host  
            _username = username  
            _password = password  
        End Sub  
  
        Private enc As Encoding  
  
        Private Function CalculateMd5Hash(ByVal input As String) As String  
            Dim inputBytes As Byte() = System.Text.Encoding.UTF8.GetBytes(input)  
            Dim hash As Byte() = MD5.Create().ComputeHash(inputBytes)  
            Dim sb As New StringBuilder()  
            For Each b In hash  
                sb.Append(b.ToString("x2"))  
            Next  
  
            Return sb.ToString()  
        End Function  
  
        Private Function GrabHeaderVar(ByVal varName As String, ByVal header As String) As String  
            Dim regHeader = New Regex(String.Format("{0}=""([^""]*)""", varName))  
            Dim matchHeader = regHeader.Match(header)  
            If matchHeader.Success Then Return matchHeader.Groups(1).Value  
            Throw New ApplicationException(String.Format("Header {0} not found", varName))  
        End Function  
  
        Private Function GetDigestHeader(ByVal dir As String) As String  
            Dim login = _username.ToLower()  
            Dim digest = CalculateMd5Hash(login + ":" + _realm + ":" + _password)  
            Dim partial_ha2 = CalculateMd5Hash("GET" + ":")  
            Dim simplified_ha2 = CalculateMd5Hash(digest + ":" + _nonce + ":" + partial_ha2)  
            Dim auth_digest = (login + ":" + _nonce + ":" + simplified_ha2)  
            Return Convert.ToBase64String(System.Text.Encoding.GetEncoding("ISO-8859-1").GetBytes(login + ":" + _nonce + ":" + simplified_ha2))  
  
        End Function  
  
        Public Function GrabResponse(ByVal dir As String) As String  
            Dim url = _host & dir  
            Dim uri = New Uri(url)  
            Dim request = CType(WebRequest.Create(uri), HttpWebRequest)  
  
            If Not String.IsNullOrEmpty(_cnonce) AndAlso DateTime.Now.Subtract(_cnonceDate).TotalHours < 1.0 Then  
                request.Headers.Add("Authorization", GetDigestHeader(dir))  
            End If  
  
            Dim response As HttpWebResponse  
  
            Try  
                response = CType(request.GetResponse(), HttpWebResponse)  
            Catch ex As WebException  
  
                If ex.Response Is Nothing OrElse (CType(ex.Response, HttpWebResponse)).StatusCode <> HttpStatusCode.Unauthorized Then Throw  
                Dim wwwAuthenticateHeader = ex.Response.Headers("WWW-Authenticate")  
                _realm = GrabHeaderVar("realm", wwwAuthenticateHeader)  
                _nonce = GrabHeaderVar("nonce", wwwAuthenticateHeader)  
                _qop = "auth"  
                _nc = 0  
                _cnonce = New Random().[Next](123400, 9999999).ToString()  
                _cnonceDate = DateTime.Now  
                Dim request2 = CType(WebRequest.Create(uri), HttpWebRequest)  
  
                request2.KeepAlive = True  
                request2.Referer = url  
                request2.ContentType = "application/json" '"application/x-www-form-urlencoded"  
                request2.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.53"  
                request2.Method = "GET"  
                request2.UseDefaultCredentials = True  
                request2.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials  
  
                Dim cache As CredentialCache = New CredentialCache()  
                Dim prefix As Uri = New Uri(url)  
                cache.Add(prefix, "Basic", New NetworkCredential(_username, _password))  
                Dim wc As WebClient = New WebClient()  
                wc.Credentials = cache  
  
                request2.Headers.Add("Authorization", GetDigestHeader(url))  
                response = CType(request2.GetResponse(), HttpWebResponse)  
            End Try  
  
            Dim reader = New StreamReader(response.GetResponseStream())  
            Return reader.ReadToEnd()  
  
        End Function  
  
    End Class  
End Namespace