Issue when using Promptflow in Azure Machine Learning Studio
Trying to use the new prompt flow feature in Azure Machine Learning Studio but I get an error when I do. In prompt flow when i would attempt to run anything just to see if the system is working I get the following error message:
'''
Run failed:
[Errno 2] No such file or directory: '/mnt/cloud/code/Users/cayden.dunn/promptflow/Flow-created-on-10-31-2023-16-35-28'
'''
I went into the associated managed blob store to track down the error log and it is as follows:
'''
2023-10-31 19:37:50 +0000 75 promptflow-runtime INFO [01255e4b-4a6a-49af-8eac-24cfeba22e9b] Receiving v2 flow request 1364d437-3978-4f3c-a6a5-c4324b25ca59: {"flow_id": "baa53d94-016d-4c12-a20a-abb1a51b0ec5", "flow_run_id": "01255e4b-4a6a-49af-8eac-24cfeba22e9b", "flow_source": {"flow_source_info": {"working_dir": "Users/cayden.dunn/promptflow/TestFlow-created-on-10-27-2023-11-37-57"}, "flow_dag_file": ".runs/01255e4b-4a6a-49af-8eac-24cfeba22e9b/flow.dag.yaml"}, "connections": "data_scrubbed", "log_path": "https://cfmlws1storage.blob.core.windows.net/azureml/PromptflowLogs/data_scrubbed/data_scrubbed/01255e4b-4a6a-49af-8eac-24cfeba22e9b.txt?sv=2021-10-04&se=2023-11-07T19%3A37%3A50Z&sr=b&sp=rw&sig=data_scrubbed "app_insights_instrumentation_key": "InstrumentationKey=data_scrubbed;IngestionEndpoint=https://eastus2-4.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus2.livediagnostics.monitor.azure.com/", "inputs": {"topic": "atom"}}
2023-10-31 19:37:50 +0000 75 promptflow-runtime INFO Runtime version: 20231024.v1. PromptFlow version: 0.1.0b8.dev5
2023-10-31 19:37:50 +0000 97 promptflow-runtime INFO [75--97] Start processing flowV2......
2023-10-31 19:37:51 +0000 75 promptflow-runtime INFO Process 97 finished
2023-10-31 19:37:51 +0000 75 promptflow-runtime ERROR Hit exception when execute request:
{
"message": "[Errno 2] No such file or directory: '/mnt/cloud/code/Users/cayden.dunn/promptflow/TestFlow-created-on-10-27-2023-11-37-57'",
"messageFormat": "",
"messageParameters": {},
"code": "SystemError",
"innerError": {
"code": "FileNotFoundError",
"innerError": null
},
"debugInfo": {
"type": "FileNotFoundError",
"message": "[Errno 2] No such file or directory: '/mnt/cloud/code/Users/cayden.dunn/promptflow/TestFlow-created-on-10-27-2023-11-37-57'",
"stackTrace": "Traceback (most recent call last):\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/runtime/utils/_utils.py\", line 171, in multi_processing_exception_wrapper\n yield\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/runtime/runtime.py\", line 232, in execute_flow_request_multiprocessing_impl\n result = execute_flow_func(config, request)\n File \"/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/runtime/runtime.py\", line 397, in execute_flow_request\n os.chdir(working_dir)\n",
"innerException": null
}
}
Traceback (most recent call last):
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/runtime/app.py", line 250, in submit_flow
result = runtime.execute_flow(req, execute_flow_request)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/runtime/runtime.py", line 90, in execute_flow
result = execute_flow_request_multiprocessing(self.config, request, execute_flow_func)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/runtime/runtime.py", line 295, in execute_flow_request_multiprocessing
raise exception
promptflow._utils.exception_utils.JsonSerializedPromptflowException: {"message": "[Errno 2] No such file or directory: '/mnt/cloud/code/Users/cayden.dunn/promptflow/TestFlow-created-on-10-27-2023-11-37-57'", "messageFormat": "", "messageParameters": {}, "code": "SystemError", "innerError": {"code": "FileNotFoundError", "innerError": null}, "debugInfo": {"type": "FileNotFoundError", "message": "[Errno 2] No such file or directory: '/mnt/cloud/code/Users/cayden.dunn/promptflow/TestFlow-created-on-10-27-2023-11-37-57'", "stackTrace": "Traceback (most recent call last):\n File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/runtime/utils/_utils.py", line 171, in multi_processing_exception_wrapper\n yield\n File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/runtime/runtime.py", line 232, in execute_flow_request_multiprocessing_impl\n result = execute_flow_func(config, request)\n File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/runtime/runtime.py", line 397, in execute_flow_request\n os.chdir(working_dir)\n", "innerException": null}}
2023-10-31 19:37:51 +0000 75 promptflow-runtime ERROR Submit flow request failed Code: 500 InnerException type: FileNotFoundError Exception type hierarchy: SystemError/FileNotFoundError
'''
I would assume this error is arising from my compute cluster not successfully mounting the flow I'm currently trying to run into is scope of files its capable of reading and executing.
As for testing this hypothesis/ ensuring a proper mount the low code architecture of prompt flow does not allow me to peak under the hood to address what is going on i don't think. All i am given is a box to select my compute(see attached image).
Not really sure what to do from here and i could not find anything on the web that matched the problem i'm facing so hoping to get some feedback.
One note that might be helpful i did recently cycle some access keys for the managed storage that Azure machine learning studio uses but I updated every key and it looks like AML has access to all the datastores it needs. not to sure where the promptflow files are stored but thought this might be causing the issue.
Any feedback is greatly appreciated,
Thank you,
Cayden Dunn