Logic App Standard Stateful Service Bus Trigger (Peek-Lock + Split-On + Workflow Concurrency) intermittently fails to instantiate workflow runs for received messages. Messages reach MaxDeliveryCount and DLQ without any corresponding workflow failure

Question

Logic App Standard Stateful Service Bus Trigger (Peek-Lock + Split-On + Workflow Concurrency) intermittently fails to instantiate workflow runs for received messages. Messages reach MaxDeliveryCount and DLQ without any corresponding workflow failure

Abhishake Jaiswal 160

I have a logic app standard with Service bus trigger peek lock with split on enabled and workflow concurrency enabled.

We noticed that messages are going to dead letter after 3 retries but the WF never gets triggered. There are no Workflow failures.
the Message lock time is set to 5 min in service bus topic.

and when we retry the same message from azure portal service bus explorer , we get similar behaviour. the WF never triggers.

and a dead letter count keeps on increasing.
any pointers what could be wrong here ?

Pravallika KV 18,850 Reputation points Microsoft External Staff Moderator

2026-07-23T03:03:45.3+00:00
Hi @Abhishake Jaiswal ,

This is a known interaction between peek-lock + Split-On + trigger concurrency on the Service Bus trigger. When you enable Split-On together with concurrency, the trigger switches to a "waiting runs" model: it eagerly fetches messages and places the excess into a waiting queue and the peek-lock timer starts while the run is still waiting, not when it begins executing.

So the sequence is:

The trigger fetches messages and queues waiting runs.

The 5-minute lock is already counting down on those waiting messages.

If a message waits longer than the lock duration (max 5 min) before its run starts, the lock expires and the message returns to the subscription.

Each lock expiry increments the delivery count but no workflow run ever executes.

After MaxDeliveryCount (3), the message is dead-lettered with no run and no failure, because settlement (Complete/Abandon) was never reached.

That's why you see the delivery count climb and messages hit the DLQ while run history stays empty the runs died in the waiting state before starting. Retrying from Service Bus Explorer re-enters the same path, so the behavior repeats.

The Logic Apps Product Team has confirmed this is by-design for Split-On concurrency, and that maximumWaitingRuns cannot be set to 0 precisely because Split-On/batch scenarios rely on the waiting queue.

How to fix it

The core problem is messages expiring their lock while waiting. Options, most reliable first:

Reduce/eliminate waiting runs. Lower the trigger concurrency degree and, if you don't need debatching in the trigger, turn Split-On off and split inside the workflow instead — this avoids the waiting-runs code path that starts the lock timer early.

Reduce the batch size so far fewer messages are pulled per trigger cycle and none sit long enough to expire. On the built-in connector, set maxMessageBatchSize (e.g. 1) in host.json:

{ "extensions": { "serviceBus": { "maxMessageBatchSize": 1, "minMessageBatchSize": 1 } } }

Keep the total in-flight well under the point where any message could wait past the lock. (Note: with concurrency on, Split-On debatch is capped at 100 and 100+ queued messages can be routed to the DLQ.)

Increase MaxDeliveryCount on the subscription as a safety buffer so transient waiting doesn't DLQ real messages but this is a mitigation, not the fix.

If you require peek-lock durability at higher throughput, consider switching to the pattern of a Recurrence trigger + "Get messages" action (which exposes an explicit max message count) so you control exactly how many messages are pulled and settled per cycle avoiding the eager waiting-run behavior entirely.

Important: the waiting-run behavior can persist even after you disable Split-On. If it doesn't clear, you may need to redeploy the workflow/Logic App fresh to reset it to the non-waiting code path.

To confirm the diagnosis, check the subscription's active vs. lock-expiry metrics and DeliveryCount progression against your throughput if messages are DLQ'ing at exactly MaxDeliveryCount with no matching run IDs, it's the lock-while-waiting expiry described above.

If this doesn't fully resolve it at your message volumes, please share your trigger concurrency degree, Split-On batch size, subscription MaxDeliveryCount, and approximate incoming message rate, and whether you're on the built-in or managed Service Bus connector that will let us tune the exact numbers.
Abhishake Jaiswal 160 Reputation points

2026-07-23T03:48:20.12+00:00

hi @Pravallika KV
thanks for the quick feedback. Let me add some history to this issue.

The WF triggers from in-app connector (When messages are available in a topic subscription (peek-lock)). There was no concurrency limit set and Split-on was enabled on triggerOutputs()>['body'] Array.
it worked fine when we messages were coming at low frequency but at a certain time, we get around 150-300 messages in one go and the cpu utilization would shoot up to 99% on WS3 instance. Though there were no dead letters at that time.
In order to keep it in control, we introduced concurrency control of 10 with 10 waiting runs with max message batch size as 10 and prefetch count as 10. we started seeing these silent dead letters.
We increased the WF run concurrency to 50 with 50 waiting runs and updated redelivery count to 3 . It reduced the silent dead letters count but there were still few.
Finally, we increased it to 100 with 100 waiting runs, still we see few silent dead letters come.
Now when We manually retry those dead letters from azure service portal at a time when there is no incoming traffic at all, it still fails silently and dead letter message count increases.
There are no failures in azure logic app WF and WF never triggers even on retrying the dead letter message manually.
can you help us understand this behavior?

Now as you mentioned

To confirm the diagnosis, check the subscription's active vs. lock-expiry metrics and DeliveryCount progression against your throughput if messages are DLQ'ing at exactly MaxDeliveryCount with no matching run IDs, it's the lock-while-waiting expiry described above.

I am not sure how to do that. if you can advise on steps to do so.
we identified that these are silent dead letters by comparing no of messages sent by upstream and no of WF runs processed in time window. the missing number always matches the dead letter count.

Quick edit - the average completion time for each logic app WF run is 3-5 seconds as seen from portal.

2 answers

Your answer

Answer 1

Hi @Abhishake Jaiswal ,

If you're seeing messages land in the dead-letter queue with no corresponding workflow run and no visible failure and it reproduces deterministically on a single message at zero load this is almost always a Split-On debatching issue, not a lock/timeout problem.

The Standard peek-lock Service Bus trigger relies on the workflow run to settle the message (Complete or Abandon). When you enable Split-On (@triggerOutputs()?['body']), the trigger debatches the array and creates one run per item.

If, for a given message, that expression doesn't resolve to a valid, non-empty array i.e. it's an empty array, a single object / non-array, or a shape that doesn't match under the batched (concurrency-on) output then no run is created for that message. And because no run owns the message:

Nothing calls Complete or Abandon → the peek-lock silently lapses.
DeliveryCount increments on each redelivery.
After MaxDeliveryCount (default 3), the message is dead-lettered with DeadLetterReason = MaxDeliveryCountExceeded.

No run ever executed, so there's no workflow failure to see. That's why it reproduces every time on one specific message the message's own content is what makes Split-On yield nothing.

Why enabling concurrency triggered it:

Two documented behaviors reinforce this:

Turning trigger concurrency on reduces the Split-On limit from 100,000 to 100 items, and "if the number of items exceeds this limit, the Split on capability is disabled." (Limits reference)
The Service Bus trigger is documented to route messages to the DLQ under concurrency + batching edge cases. (Service Bus connector considerations)

So the typical history matches:

Concurrency off → Split-On limit 100,000, every message got a run (often CPU-saturating), everything settled → no DLQ.
Concurrency on → Split-On limit dropped to 100 and the batched output shape changed → some messages debatch to nothing → silent DLQ.
Raising concurrency/waiting-run values only helped the load-induced portion; the deterministic DLQs are debatch failures that tuning can't fix.

Follow below steps to confirm:

Service Bus Explorer → Peek Mode → the DeadLetter subqueue (non-destructive). On a dead-lettered message, check DeadLetterReason = MaxDeliveryCountExceeded and DeliveryCount = 3, then inspect the message body the failing ones will be an empty / non-array / differently-shaped payload compared to the ones that succeed.
Metrics (filter EntityName = your subscription, aggregation = Sum): during failures you'll see Incoming rises, Dead-lettered rises, but Completed stays flat messages arrive and dead-letter without ever being completed. Quick count check:

az servicebus topic subscription show \
  --resource-group <rg> --namespace-name <ns> \
  --topic-name <topic> --name <sub> --query countDetails

Fix:

Turn Split-On off and debatch inside the workflow using a For each over @triggerBody(). A real run is always created, so the message is always settled an empty or odd payload just results in a run that does nothing, instead of silently dead-lettering.

Additional notes:

If you keep Split-On, make sure no debatched array exceeds 100 items while concurrency is on, otherwise Split-On is disabled for that batch.
Always use the ? operator (@triggerOutputs()?['body']) so a missing property doesn't hard-fail.
Trigger concurrency is irreversible once enabled plan accordingly.
If high throughput is the goal, consider a Recurrence trigger + "Get messages" action pattern instead, where you control the pull count and settlement explicitly (Recurrence also avoids waiting-run/lock-expiry issues). Hope this helps!

If the resolution was helpful, kindly take a moment to click on User's image and click on Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

The fact that no run is created is important: the failure is in the trigger or host path before the workflow instance starts, so it will not appear as a failed run. Check trigger history, Logic App host logs, and the Service Bus dead-letter reason using the same message ID and timestamps. With Peek-Lock, Split On, and workflow concurrency enabled together, keep the trigger batch size within the Split On concurrency limit and leave enough waiting-run capacity; otherwise, deliveries can retry while no run is admitted. Temporarily disable concurrency or Split On, set a small batch, and retest with the V2 Service Bus trigger. Confirm the lock duration or renewal window exceeds processing time and the connection has Listen permission. If the host still abandons messages before creating runs, collect message IDs, trigger-history entries, and runtime logs for Microsoft Support; only platform-side logs can show the pre-run exception.