Azure job stucks on running status with HF trainer.train()

Question

Azure job stucks on running status with HF trainer.train()

Luigi Montaleone 0

Hi,

I'm facing a problem in Azure ML when launching a job about fine tuning a transformer. I'm fine tuning Whisper model from OpenAI for a speech-to-text task, and I am following this guide carefully: https://huggingface.co/blog/fine-tune-whisper

I created the .py and the last instruction is:

trainer.train()

and I launched an Azure ML job in this way:

job = command(
    code="./src",  # location of source code
    command="python main.py",
    display_name="whisper_ft",
    compute="gpu-compute",
    environment="pytorch_cuda_env_custom@latest"

)

ml_client.create_or_update(job)

but the job stucks on running status, even if the training ends successfully (I tried it without job execution). Probably the problem is on trainer.train() instrunction since if I comment it the job goes in Completed status.

Anyone else with the same problem?

santoshkc 15,600 Reputation points Microsoft External Staff Moderator

2024-10-03T13:09:04.0633333+00:00

Hi @Luigi Montaleone,

Thank you for reaching out to Microsoft Q&A forum!

The job likely gets stuck due to blocking behavior from trainer.train() or resource issues. Check the logs for errors, verify that your GPU resources are sufficient, and ensure all dependencies in the environment are properly set.
See: Configure and submit training jobs.

Thank you.
Luigi Montaleone 0 Reputation points

2024-10-03T15:08:06.6033333+00:00

Hi @santoshkc ,

thanks for your answer!

I think GPU resources of the compute instance are sufficient, since if I try to run the script from the compute bash, it works and ends properly. I think this is strictly related to the method train(), but I haven't been able to solve it yet.

P.S. I'm using v2 Azure Python SDK. Maybe I can try with v1?
santoshkc 15,600 Reputation points Microsoft External Staff Moderator

2024-10-04T08:16:45.1933333+00:00
Hi @Luigi Montaleone,

Since your Whisper model runs successfully outside of Azure ML but gets stuck when using trainer.train() in an Azure ML job, the issue likely isn't with GPU resources or the training script itself. Instead, it might be how Azure ML handles long-running jobs or waits for logs during the job execution.

Ensure you're capturing log output frequently within the trainer.train() call. This helps Azure ML know the job is still active, preventing it from being perceived as stuck.

If you suspect the problem lies with the v2 SDK, switching back to the v1 SDK can be worth trying. Some job handling features in v1 might manage the process differently, especially around job completion statuses.

I hope you understand. Thank you.
santoshkc 15,600 Reputation points Microsoft External Staff Moderator

2024-10-07T08:00:06.2333333+00:00

Hi @Luigi Montaleone,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Luigi Montaleone 0

Hi @santoshkc ,

sorry, for the delay.

The output of

trainer.train()

function is already frequently captured so It's possible to know when the trainer finishes his job.

I tried v1 SDK following the official tutorial but the problem remains. Source code:


ws = Workspace(subscription_id = subscription_id, resource_group = resource_group, workspace_name = workspace)



experiment = Experiment(workspace=ws, name="myexp")
myenv = Environment.get(workspace=ws, name="pytorch_cuda_env")

src = ScriptRunConfig(source_directory="./src",
                      script='main.py',
                      compute_target="gpu-compute",
                      environment=myenv)

run = experiment.submit(config=src)

I really don't know what I can do more!

santoshkc 15,600 Reputation points Microsoft External Staff Moderator

2024-10-10T14:43:14.8866667+00:00
Hi @Luigi Montaleone,

Sorry for the trouble you are facing and I recommend reporting this issue to the Azure support team. They will be able to investigate the issue further and provide a more targeted solution. You can report the issue by following these steps:

Go to the Azure portal and navigate to your Azure ML Services resource.

Click on the "Support + troubleshooting" tab.

Fill out the required information, including a detailed description of the issue and any steps you have taken to troubleshoot it.

Submit the support request.

The Azure support team will review your request and provide assistance as soon as possible.
Luigi Montaleone 0 Reputation points

2024-10-14T08:14:50.48+00:00

Hi @santoshkc ,

thanks for the answer, but I think that a different support plan is necessary for technical assitance.

I'll try to find another solution. Thanks a lot for your time!

Your answer

santoshkc 15,600 Reputation points Microsoft External Staff Moderator

2024-10-03T13:09:04.0633333+00:00

Hi @Luigi Montaleone,

Thank you for reaching out to Microsoft Q&A forum!

The job likely gets stuck due to blocking behavior from trainer.train() or resource issues. Check the logs for errors, verify that your GPU resources are sufficient, and ensure all dependencies in the environment are properly set.
See: Configure and submit training jobs.

Thank you.
Luigi Montaleone 0 Reputation points

2024-10-03T15:08:06.6033333+00:00

Hi @santoshkc ,

thanks for your answer!

I think GPU resources of the compute instance are sufficient, since if I try to run the script from the compute bash, it works and ends properly. I think this is strictly related to the method train(), but I haven't been able to solve it yet.

P.S. I'm using v2 Azure Python SDK. Maybe I can try with v1?
santoshkc 15,600 Reputation points Microsoft External Staff Moderator

2024-10-04T08:16:45.1933333+00:00

Hi @Luigi Montaleone,

Since your Whisper model runs successfully outside of Azure ML but gets stuck when using trainer.train() in an Azure ML job, the issue likely isn't with GPU resources or the training script itself. Instead, it might be how Azure ML handles long-running jobs or waits for logs during the job execution.

Ensure you're capturing log output frequently within the trainer.train() call. This helps Azure ML know the job is still active, preventing it from being perceived as stuck.

If you suspect the problem lies with the v2 SDK, switching back to the v1 SDK can be worth trying. Some job handling features in v1 might manage the process differently, especially around job completion statuses.

I hope you understand. Thank you.
santoshkc 15,600 Reputation points Microsoft External Staff Moderator

2024-10-07T08:00:06.2333333+00:00

Hi @Luigi Montaleone,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.
Luigi Montaleone 0 Reputation points

2024-10-10T08:33:27.1766667+00:00

Hi @santoshkc ,

sorry, for the delay.

The output of

trainer.train()

function is already frequently captured so It's possible to know when the trainer finishes his job.

I tried v1 SDK following the official tutorial but the problem remains. Source code:

ws = Workspace(subscription_id = subscription_id, resource_group = resource_group, workspace_name = workspace) experiment = Experiment(workspace=ws, name="myexp") myenv = Environment.get(workspace=ws, name="pytorch_cuda_env") src = ScriptRunConfig(source_directory="./src", script='main.py', compute_target="gpu-compute", environment=myenv) run = experiment.submit(config=src)

I really don't know what I can do more!
santoshkc 15,600 Reputation points Microsoft External Staff Moderator

2024-10-10T14:43:14.8866667+00:00

Hi @Luigi Montaleone,

Sorry for the trouble you are facing and I recommend reporting this issue to the Azure support team. They will be able to investigate the issue further and provide a more targeted solution. You can report the issue by following these steps:

Go to the Azure portal and navigate to your Azure ML Services resource.

Click on the "Support + troubleshooting" tab.

Fill out the required information, including a detailed description of the issue and any steps you have taken to troubleshoot it.

Submit the support request.

The Azure support team will review your request and provide assistance as soon as possible.
Luigi Montaleone 0 Reputation points

2024-10-14T08:14:50.48+00:00

Hi @santoshkc ,

thanks for the answer, but I think that a different support plan is necessary for technical assitance.

I'll try to find another solution. Thanks a lot for your time!

Share via

Azure job stucks on running status with HF trainer.train()

Your answer