Upload data stream from an api call to azure adls gen2 using c#

Raj D 581 Reputation points
2022-04-08T03:29:42.487+00:00

Greetings!!!

I'm working on reading data from an api call and write data to Azure Data Lake Storage Gen 2 using the code below. When I try to pass the ADLS Gen 2 file path I am running into error as its not able to recognize the file path. For instance the file path I have is something like this "https://storageaccount.dfs.core.windows.net/adlsContainer/test/apidata.json".

C# code:

public static uploadAPIDataToADLS(string targetPath, string fileName, string fileStream)
{
    var baseAddressString = $"https://{_adlsStorage}.{_adlsSuffix}/{_rootFileSystem}";
    var copyPath = $"{targetPath}/{fileName}";
    var httpClient = new HttpClient();
 string filePath = "https://storageaccount.dfs.core.windows.net/adlsContainer/test/{fileStream}"; //Data from an api call is read into fileStream 
    string fileLength;

    var auth = await GetAuthTokenForDatalake();
    var createMessage = new HttpRequestMessage
    {
        Method = HttpMethod.Put,
        RequestUri = new Uri($"{baseAddressString}/{copyPath}?resource=file"),
    };
    createMessage.Headers.Add("Authorization", auth);
    createMessage.Headers.Add("x-ms-version", "2018-11-09");
    var response = await httpClient.SendAsync(createMessage);

    if (response.IsSuccessStatusCode)
    {
        using (FileStream filestream = File.OpenRead(filePath)) //Running into error trying to pass ADLS file path
        {
            var streamContent = new StreamContent(filestream);
            fileLength = filestream.Length.ToString();
            var appendMessage = new HttpRequestMessage
            {
                Method = HttpMethod.Patch,
                RequestUri = new Uri($"{baseAddressString}/{copyPath}?action=append&position=0"),
                Content = streamContent
            };
            appendMessage.Headers.Add("Authorization", auth);
            appendMessage.Headers.Add("x-ms-version", "2018-11-09");
            appendMessage.Content.Headers.Add("Content-Length", fileLength);
            appendMessage.Content.Headers.Add("Content-Type", "application/octet-stream");

            response = await httpClient.SendAsync(appendMessage);
        }

        if (response.IsSuccessStatusCode)
        {
            var flushMessage = new HttpRequestMessage
            {
                Method = HttpMethod.Patch,
                RequestUri = new Uri($"{baseAddressString}/{copyPath}?action=flush&position={fileLength}"),
            };
            flushMessage.Headers.Add("Authorization", auth);
            flushMessage.Headers.Add("x-ms-version", "2018-11-09");
            response = await httpClient.SendAsync(flushMessage);
        }
    }
}

Error:

Invalid file path.

Please guide me on how to pass the file path.

Thank you in advance

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,394 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,481 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 38,301 Reputation points Microsoft Employee
    2022-04-12T09:25:38.43+00:00

    Hi @Raj D ,

    Thanks for posting query in Microsoft Q&A Platform.

    File.OpenRead() function will try to look for given path in local system not on ADLS gen2.

    Kindly check below link where .NET SDK examples discussed to interact with ADLS gen2 to read file content or to load files to it. Thank you.
    https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-dotnet

    Please let us know how it goes.