Pipeline run stucked on queue

Question

Pipeline run stucked on queue

Gabriel Santos 0

I have some pipelines that ingest data from Oracle and write directly to Delta in a lakehouse on Fabric.

I have other pipelines that ingest data from Oracle and write to a .parquet file in a lakehouse on Fabric.

The pipelines that write directly to Delta are staying in the Azure Data Factory queue for an eternity and don’t even start as "In Progress."

At the source, I run a query, and when I click on "View Data," I see the full data. In the collector, the connection is successful.

This problem of staying in the queue started after the Fabric capacity was exceeded, but the writing to .parquet files is running normally.

When i copy the pipeline id and run diagnostics, its returns the messagem attached in the picture bellow.

User's image

There is nothing running in the azure data factory, still all the integration runtime available only for this pipeline that is very light, lessa than 100 lines, no treatment, only a copy activity.

Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-16T16:02:26.45+00:00

@Gabriel Santos -Thanks for the question and using MS Q&A platform.

MS Fabric is currently not supported in the MS Q&A forums, the supported products are listed over here Supported products on MS Q&A (more to be added later on).

You can ask the experts in the dedicated MS Fabric community forum.

Can you please open as a new thread in MS Fabric community: https://community.fabric.microsoft.com/t5/Power-BI-forums/ct-p/powerbi

Hope this helps. Do let us know if you have any further queries.

------------

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Gabriel Santos 0 Reputation points

2024-09-16T16:40:45.29+00:00

This problem is not about Microsoft Fabric, i'm facing the problem with AZURE DATA FACTORY. In this pipeline the sink is the microsoft fabric, so, is very relevant to mention that.
Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-16T18:20:27.48+00:00
Gabriel Santos I apologize for the confusion. Thank you for clarifying that the issue is with Azure Data Factory and not Microsoft Fabric.

To better understand your setup and provide a suitable solution, could you please provide more details on the following:

Can you share more information about your current pipeline configuration in ADF, including the trigger type and any error messages you've encountered?

What is the current capacity of your Fabric environment, and have you experienced any issues with pipeline execution or data movement in the past?

Are there any specific requirements or constraints for the data movement pipelines, such as data volume, frequency, or priority?

Additionally, it would be helpful to know more about the error message you're seeing when the pipeline gets stuck. Could you please provide a detailed error message or a screenshot of the error?

Once I have more information about your setup and the issue you're facing, I'll be happy to provide guidance on how to move forward.

I hope this helps. Let me know if you have any further questions.
Gabriel Santos 0 Reputation points

2024-09-17T16:08:35.8633333+00:00

Hello, follow the current configuration, is only a pipeline with a copy data activity, the data should be written in the Fabric Delta Tables.

In the source tab, if i click in preview data, i can see the data on the source.

In the sink tab, if i edit the dataset, i can preview the data in the sink.

Mapping,

Configuration,

the trigger:

With trigger or manually the error persists.

I'm facing theses errors:

Nice to mention that i have some parquet files landing in the microsoft fabric lakehouse and is working propperly.

This pipeline is running there is more than 6 months without any problem, just after a exceed capacity on my microsoft fabric that the error started and now even with anything normalized still with problem.

I was using a F8 capacity, now i'm using again the trial capacity (f64) and utilizin only 10% of it.

Very stranger situation, waiting for a possible solution.
Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-18T15:39:37.91+00:00

@Gabriel Santos - Thank you for sharing more details about your Azure Data Factory pipeline setup and the issues with writing data to Fabric Delta Tables.

I have reviewed the images you provided, it seems that the issue is related to the Fabric capacity, which is preventing the pipeline from acquiring the necessary compute resources. If switching to a different capacity resolves the issue, then this confirms the capacity limitation is the cause.

At this point, the best solution would be to choose a different Fabric capacity that suits your needs. Unfortunately, there isn’t an alternative fix for this problem beyond adjusting the capacity.

I recommend reaching out to the Fabric support forum for further assistance and to discuss capacity planning options. They can provide more specific guidance based on your environment and requirements.

Hope this helps. Do let us know if you any further queries.

If this answers your query, please do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Gabriel Santos 0 Reputation points

2024-09-19T17:06:18.71+00:00

@Smaran Thoomu , the fabric capacity was already changed multiple times to check if the problem is the fabric capacity, but still with problem. We have more than 80% of availability of our capacity and still with this problem. Very stranger
Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-19T20:14:14.4533333+00:00

@Gabriel Santos I agree that this issue looks strange, and I wasn't able to reproduce this issue. If you have a support plan, could you please file a support ticket for deeper investigation and do share the SR# with us? In case if you don't have a support plan please let us know here.
Gabriel Santos 0 Reputation points

2024-09-19T20:55:01.3266667+00:00

@Smaran Thoomu I dont have a support plan.

1 answer

Your answer

Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-16T16:02:26.45+00:00

@Gabriel Santos -Thanks for the question and using MS Q&A platform.

MS Fabric is currently not supported in the MS Q&A forums, the supported products are listed over here Supported products on MS Q&A (more to be added later on).

You can ask the experts in the dedicated MS Fabric community forum.

Can you please open as a new thread in MS Fabric community: https://community.fabric.microsoft.com/t5/Power-BI-forums/ct-p/powerbi

Hope this helps. Do let us know if you have any further queries.

------------

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Gabriel Santos 0 Reputation points

2024-09-16T16:40:45.29+00:00

This problem is not about Microsoft Fabric, i'm facing the problem with AZURE DATA FACTORY. In this pipeline the sink is the microsoft fabric, so, is very relevant to mention that.
Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-16T18:20:27.48+00:00

Gabriel Santos I apologize for the confusion. Thank you for clarifying that the issue is with Azure Data Factory and not Microsoft Fabric.

To better understand your setup and provide a suitable solution, could you please provide more details on the following:

Can you share more information about your current pipeline configuration in ADF, including the trigger type and any error messages you've encountered?

What is the current capacity of your Fabric environment, and have you experienced any issues with pipeline execution or data movement in the past?

Are there any specific requirements or constraints for the data movement pipelines, such as data volume, frequency, or priority?

Additionally, it would be helpful to know more about the error message you're seeing when the pipeline gets stuck. Could you please provide a detailed error message or a screenshot of the error?

Once I have more information about your setup and the issue you're facing, I'll be happy to provide guidance on how to move forward.

I hope this helps. Let me know if you have any further questions.
Gabriel Santos 0 Reputation points

2024-09-17T16:08:35.8633333+00:00

Hello, follow the current configuration, is only a pipeline with a copy data activity, the data should be written in the Fabric Delta Tables.

In the source tab, if i click in preview data, i can see the data on the source.

In the sink tab, if i edit the dataset, i can preview the data in the sink.

Mapping,

Configuration,

the trigger:

With trigger or manually the error persists.

I'm facing theses errors:

Nice to mention that i have some parquet files landing in the microsoft fabric lakehouse and is working propperly.

This pipeline is running there is more than 6 months without any problem, just after a exceed capacity on my microsoft fabric that the error started and now even with anything normalized still with problem.

I was using a F8 capacity, now i'm using again the trial capacity (f64) and utilizin only 10% of it.

Very stranger situation, waiting for a possible solution.
Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-18T15:39:37.91+00:00

@Gabriel Santos - Thank you for sharing more details about your Azure Data Factory pipeline setup and the issues with writing data to Fabric Delta Tables.

I have reviewed the images you provided, it seems that the issue is related to the Fabric capacity, which is preventing the pipeline from acquiring the necessary compute resources. If switching to a different capacity resolves the issue, then this confirms the capacity limitation is the cause.

At this point, the best solution would be to choose a different Fabric capacity that suits your needs. Unfortunately, there isn’t an alternative fix for this problem beyond adjusting the capacity.

I recommend reaching out to the Fabric support forum for further assistance and to discuss capacity planning options. They can provide more specific guidance based on your environment and requirements.

Hope this helps. Do let us know if you any further queries.

If this answers your query, please do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Gabriel Santos 0 Reputation points

2024-09-19T17:06:18.71+00:00

@Smaran Thoomu , the fabric capacity was already changed multiple times to check if the problem is the fabric capacity, but still with problem. We have more than 80% of availability of our capacity and still with this problem. Very stranger
Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-19T20:14:14.4533333+00:00

@Gabriel Santos I agree that this issue looks strange, and I wasn't able to reproduce this issue. If you have a support plan, could you please file a support ticket for deeper investigation and do share the SR# with us? In case if you don't have a support plan please let us know here.
Gabriel Santos 0 Reputation points

2024-09-19T20:55:01.3266667+00:00

@Smaran Thoomu I dont have a support plan.

Answer 1

Smaran Thoomu 34,070 Microsoft External Staff Moderator

Hi @Gabriel Santos

I found out that there was an issue with the Fabric Lakehouse, but it has now been fixed. Can you please check and let me know if you're still facing the problem? I believe the issue should be resolved now.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-24T14:35:02.0566667+00:00

@Gabriel Santos Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Gabriel Santos 0 Reputation points

2024-09-24T16:28:12.6866667+00:00

@Smaran Thoomu , the problem was fixed without any actions by my side. Idk what happened, can be by fabric side because the platform still with a lot of bugs.
But, for now, thats it. Thank you for your time.
Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-24T16:57:12.3466667+00:00

Gabriel Santos I'm glad to hear that the issue has been resolved, even without requiring any action from your side. Please note that I previously mentioned the Fabric Lakehouse issue as a potential cause of the problem. I'm glad that my assessment was correct.

If this response has been helpful, please click Accept Answer and Yes for 'Was this answer helpful?' This will help others who may encounter similar issues.
Smaran Thoomu 34,070 Reputation points Microsoft External Staff Moderator

2024-09-25T18:24:03.4433333+00:00

Gabriel Santos - Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Pipeline run stucked on queue

1 answer

Your answer