Azure data factory run Python to access Azure blob storage failed

Question

I follow this document and try to "download a file, make some modification, and upload the file"

In "Create a Batch pool using Batch Explorer" section:
I change the OS from Windows to Linux

Setup commands for my python module installation

Here is my sample code

I even use the BatchExplorer tool to SSH into the node, and run my Python script, which works fine.

And then I follow the ADF setup in the tutorial:

The batch account also links to the storage account that I want to perform read write task.
(Maybe it has nothing to do with the storage I want to access? All I need is the connection string?)

Python script is placed at the corresponding location
(In fact, if I replace the "azure blob storage stuff" with "local file read & write". It passed the debug process successfully.)

And last, the error occurred. I collect all messages into a file.
140403-error-details.log

Any suggestion will be appreciated! Thanks!

Answer 1

Hello @Han Shih 施學翰 ,
Thanks for the ask and using Microsoft Q&A platform .
The example which you have refered in the ask uses the windows server and so the below command does make sense .

but in Ubuntu there is no "cmd" , I think if you just call
pip# install XXXXX
It should work .

Please do let me know how it goes .
Thanks
Himanshu

-------------------------------------------------------------------------------------------------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

@HimanshuSinha-msft , thanks for your reply

I modified the start task as you advised:

First, I tried a simple script. And it worked as I expected.

Second, I changed to another script interacting with Azure blob storage.
I found that once I add this line from azure.storage.blob import BlobClient, BlobServiceClient, ContainerClient, the same error occurred.

So, I think there might be something wrong in my task pool settings.
In 'start task' section, I saw this message:

To allow tasks to run in containers, the container configuration must be specified for the task's pool. Please ensure that pool container configuration is set up and selected for this task.

However, I can not find a way to deal with it.
Maybe I have to some configurations about 'start task' do not set correctly?

Answer 3

@HimanshuSinha-msft

Here is my script

Pipeline settings

Start task in Azure Batch

As you say, the reason might be:
1.module installation failed
2.module installed, but cannot be loaded in execution

However, the output of the execution seems confusing

{
"exitcode": 1,
"outputs": [
" https://sysgeneral.blob.core.windows.net/adfjobs/fc915f8d-d513-4683-a0b9-e64ebb85b8d7/output/stdout.txt ",
" https://sysgeneral.blob.core.windows.net/adfjobs/fc915f8d-d513-4683-a0b9-e64ebb85b8d7/output/stderr.txt "
],
"errorCategory": 0,
"code": "FailureExitCode",
"message": "The task exited with an exit code representing a failure",
"details": [
{
"Name": "Message",
"Value": "The task process exited with an unexpected exit code"
},
{
"Name": "AdditionalErrorCode",
"Value": "FailureExitCode"
}
],
"computeInformation": "{\"account\":\"dsubatch\",\"poolName\":\"test_pool\",\"vmSize\":\"standard_a1_v2\"}",
"effectiveIntegrationRuntime": "DefaultIntegrationRuntime (Japan East)",
"executionDuration": 16,
"durationInQueue": {
"integrationRuntimeQueue": 1
},
"billingReference": {
"activityType": "ExternalActivity",
"billableDuration": [
{
"meterType": "AzureIR",
"duration": 0.016666666666666666,
"unit": "Hours"
}
]
}
}

Is there a better to find out the root cause?

Answer 4

I found the root cause,

Whenever "start task" is modified, all the nodes should be rebooted.

Thanks for your time ~