deploy git artifacts to azure datalake via azure devops

Question

We have some critical files we currently store in a git repo to version control them. We would like to set up continuous deployment via azure devops to deploy these files to an azure data lake store gen1, including permissions.

I have written a powershell script that does exactly this, which accessing the ADLS, grabs the permissions of existing files/folders, deletes them if they also happen to exist in the git repo, then deploys the contents of a folder in that git repo, creating new folders as necessary, then the files, then re-applies the permissions (ownership as well as special other permissions (rwx)).

great, works in powershell.

but fails in azure devops release pipelines. Obviously, the agent doesn't have permissions into the datalake, nor can I give it permissions, as the user it executes as changes ($hostname\vssadministrator, where $Hostname changes). I'm fairly sure I could make it work locally using an existing VM, but would rather not if I could avoid it.

I know the various azure subscriptions the agent uses has various roles that allow it to deploy to various azure resources, but access to an azure datalake is not one of them (let alone updating the contents of it), and I would rather not grant it blanket permissions to modify any old datalake, I'd rather do it as a member of a group, with that group having explicit role permissions, then add members of that group in the appropriate datalake subfolder, but I don't think will work neither.

I didn't want to do this via a datafactory neither, besides, afaik, datafactories can't contact git repos that aren't hosted in github (our git repos are in azure devops). weird, I also don't see an azure devops tag...

Answer 1

FYI, adding it as a role assignment doesn't work neither. I created an AD group, gave it read permissions into the ADLS at the root level (and path to the folder I want it to write to), and full permissions in the appropriate subfolder. I can see in my group, as a azure role assignment, there's the 'contributor' role for the subscription (assigned to this group).

but, running the release pipeline... LISTSTATUS failed with HttpStatus:Forbidden RemoteException: AccessControlException LISTSTATUS failed with error 0x83090aa2 (Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation.). [<uid>] failed with error 0x83090aa2 (Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation

obviously the resource exists... but Get-AzDataLakeStoreChildItem -Account $dlAccount -Path $datalakePath fails...

And FYI, part of this script tests to see if the datalake exists, and to error out of it doesn't (like wrong subscription or tenant, or the datalake doesn't exist).

That code itself looks like

$datalakeExists = Test-AzDataLakeStoreAccount -Name $DataLakeName
if (! $datalakeExists) {
Write-Warning "ERROR: DataLake '$DataLakeName' does NOT exist"
Write-Warning " - Check your name of the datalake, it might be wrong"
Write-Warning " - Check your subscription, it might be in a different subscription"
Write-Warning " - Check your tenant/directory, it might be in a different tenant"
Write-Warning ""
Write-Warning "Exiting..."
exit 1
}

script doesn't fail on that, it just keeps going, till it actually tries to access the datalake.

Answer 2

FYI, I have my answer, figured it out on my own. doesn't involve any role modifications nor funky locally hosted VMs. So, close this out.

Answer 3

Hi could you please explain what you did. I have exactly the same problem.