Write file to ADLS Gen2 container

Raj D 616 Reputation points
2022-07-26T20:40:23.96+00:00

Hi, I'm working on writing the json payload from an api call to Azure Datalake Storage Gen2 container using powershell script. I would like to save the json file from api call to the container path "MyContainer/MyDir/MyData.json". I have the code below to make the api call.

Could you please guide on how to write the json file to the container path.

container-name : MyContainer
directory-name : MyDir
file-name : MyData.json
file-path : MyContainer/MyDir/MyData.json

PoSh:

try {  
        $clientSecret = ''  
        $clientId = ''  
        $tenantId = ''  
        # Construct URI  
        $uri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"  
        # Construct Body  
        $body = @{  
            client_id = $clientId  
            client_secret = $clientSecret  
            scope = 'https://graph.microsoft.com/.default'  
            grant_type = 'client_credentials'  
        }  
        $Uri = 'https://apiserver.com/v1/data'  
        # Get OAuth 2.0 Token  
        $tokenRequest = Invoke-WebRequest -Method Post -Uri $uri -ContentType 'application/x-www-form-urlencoded' -Body $body -UseBasicParsing  
        # Access Token  
        $token = ($tokenRequest.Content | ConvertFrom-Json).access_token  
        $api = Invoke-RestMethod -Method Get -Uri $Uri -ContentType 'application/json' -Headers @{Authorization = "Bearer $token"} -ErrorAction Stop  
    }  
catch {  
        "Error"   
        Write-Host "StatusCode:" $_.Exception.Response.StatusCode.value__  
        Write-Host "StatusDescription:" $_.Exception.Response.StatusDescription  
        Write-Host "ErrorMessage:" $_.ErrorDetails.Message  
    }  

PoSh_Con

$subscription = 'test_sub'  
$resourceGroupName = 'test_rg'  
$storageAccountName = 'MyStorageAccount'  
$containerName = 'MyContainer'  
  
Set-AzContext -Subscription $subscription  
  
$storageAccountKey = Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $storageAccountName  
$primaryKey = $storageAccountKey | Where-Object keyname -eq 'key1' | Select-Object -ExpandProperty value  
  
$storageContext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $primaryKey  
  
try {  
    New-AzStorageContainer -Name $containerName -Context $storageContext -ErrorAction Stop  
}  
catch [Microsoft.WindowsAzure.Commands.Storage.Common.ResourceAlreadyExistException] {  
    Write-Output ('Container {0} already exists in Storage Account {1}' -f $containerName, $storageAccountName)  
}  

I'm trying to run this powershell script in Azure automation runbook I tried storage account key & managed identity for access onto container and nothing seems to work.

Thank you.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,562 questions
Azure Automation
Azure Automation
An Azure service that is used to automate, configure, and install updates across hybrid environments.
1,368 questions
Windows for business | Windows Server | User experience | PowerShell
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,236 Reputation points
    2022-07-27T19:32:01.057+00:00

    Hello @Raj D ,
    Thanks for the question and using MS Q&A platform.

    It sounds like you are trying to upload to ADLS Gen2 using powershell. However, the code you have shared only tries to create a new container. It does not show any commands for uploading.

    New-AzDataLakeGen2Item -Context $storageContext -FileSystem MyContainer -Path MyDir/MyData.json -Source MyData.json -Force

    New-AzDataLakeGen2Item -Context $storageContext -FileSystem $containerName -Path ($directory-name + "/" + $file-name) -Source $file-name -Force

    Example code sourced from data-lake-storage-directory-file-acl-powershell: uploading a file

    $localSrcFile =  "upload.txt"  
    $filesystemName = "my-file-system"  
    $dirname = "my-directory/"  
    $destPath = $dirname + (Get-Item $localSrcFile).Name  
    New-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $destPath -Source $localSrcFile -Force  
    

    Also see example from powershell reference

    $task = New-AzDataLakeGen2Item  -FileSystem "testfilesystem" -Path "dir1/dir2/file1" -Source "c:\sourcefile.txt" -Force -asjob  
    $task | Wait-Job  
    $task.Output  
    
       FileSystem Name: filesystem1  
    
    Path                 IsDirectory  Length          LastModified         Permissions  Owner                Group                  
    ----                 -----------  ------          ------------         -----------  -----                -----                 
    dir1/dir2/file1      False        14400000        2020-03-23 09:19:13Z rw-r-----    $superuser           $superuser  
    

    With the missing command out of the way, there seems to be another issue you mentioned

    I'm trying to run this powershell script in Azure automation runbook I tried storage account key & managed identity for access onto container and nothing seems to work.

    You haven't mentioned what precisely went wrong, or any error messages. So I am uncertain whether you are facing an authentication issue, or a container creation issue, or you just got confused between uploading and creating container.

    Please do let me if you have any queries.

    Thanks
    Martin


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.