Upload data stream from an api call to azure adls gen2 using c#

Question

Greetings!!!

I'm working on reading data from an api call and write data to Azure Data Lake Storage Gen 2 using the code below. When I try to pass the ADLS Gen 2 file path I am running into error as its not able to recognize the file path. For instance the file path I have is something like this "https://storageaccount.dfs.core.windows.net/adlsContainer/test/apidata.json".

C# code:

public static uploadAPIDataToADLS(string targetPath, string fileName, string fileStream)
{
    var baseAddressString = $"https://{_adlsStorage}.{_adlsSuffix}/{_rootFileSystem}";
    var copyPath = $"{targetPath}/{fileName}";
    var httpClient = new HttpClient();
 string filePath = "https://storageaccount.dfs.core.windows.net/adlsContainer/test/{fileStream}"; //Data from an api call is read into fileStream 
    string fileLength;

    var auth = await GetAuthTokenForDatalake();
    var createMessage = new HttpRequestMessage
    {
        Method = HttpMethod.Put,
        RequestUri = new Uri($"{baseAddressString}/{copyPath}?resource=file"),
    };
    createMessage.Headers.Add("Authorization", auth);
    createMessage.Headers.Add("x-ms-version", "2018-11-09");
    var response = await httpClient.SendAsync(createMessage);

    if (response.IsSuccessStatusCode)
    {
        using (FileStream filestream = File.OpenRead(filePath)) //Running into error trying to pass ADLS file path
        {
            var streamContent = new StreamContent(filestream);
            fileLength = filestream.Length.ToString();
            var appendMessage = new HttpRequestMessage
            {
                Method = HttpMethod.Patch,
                RequestUri = new Uri($"{baseAddressString}/{copyPath}?action=append&position=0"),
                Content = streamContent
            };
            appendMessage.Headers.Add("Authorization", auth);
            appendMessage.Headers.Add("x-ms-version", "2018-11-09");
            appendMessage.Content.Headers.Add("Content-Length", fileLength);
            appendMessage.Content.Headers.Add("Content-Type", "application/octet-stream");

            response = await httpClient.SendAsync(appendMessage);
        }

        if (response.IsSuccessStatusCode)
        {
            var flushMessage = new HttpRequestMessage
            {
                Method = HttpMethod.Patch,
                RequestUri = new Uri($"{baseAddressString}/{copyPath}?action=flush&position={fileLength}"),
            };
            flushMessage.Headers.Add("Authorization", auth);
            flushMessage.Headers.Add("x-ms-version", "2018-11-09");
            response = await httpClient.SendAsync(flushMessage);
        }
    }
}

Error:

Invalid file path.

Please guide me on how to pass the file path.

Thank you in advance

Answer

Hi @Raj D ,

Thanks for posting query in Microsoft Q&A Platform.

File.OpenRead() function will try to look for given path in local system not on ADLS gen2.

Kindly check below link where .NET SDK examples discussed to interact with ADLS gen2 to read file content or to load files to it. Thank you.
https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-dotnet

Please let us know how it goes.

Share via

Upload data stream from an api call to azure adls gen2 using c#

1 answer