Help me understand why activity functions should be idempotent in durable functions

Question

Documentation says

Because activity functions only guarantee at least once execution, we recommend you make your activity function logic idempotent whenever possible.

We have a case where SubOrchestration1 calls Activity1, which returns a list of Ids representing created entities. The above documentation indicates that Activity1 could be invoked multiple times. This leaves me with questions:

1. Under what conditions would Activity1 to be called multiple times?
2. If Activity1 is successfully called twice and returns a different list of Ids each time, which list does SubOrchestration1 end up receiving?
3. If Activity1 is called twice, could the 2nd call be running while the 1st has not yet completed?
4. How common are multiple invocations? Is this the kind of thing which will happen once in a blue moon or should we expect most functions to be invoked more than once?

In general, I wish I understood the mechanisms that could cause a function to be invoked multiple times so that I can better account for this in my design. Thank you!

Answer 1

1. Activity Triggers internally listens on the underlying queue that is used by the durable functions extensions.
2. When an activity function completes, its output is written to the orchestration history table
3. Like a queue triggered function, if the first activity function is stuck before renewing the visibility timeout on the message
4. Again would depend on the underlying queue storage

The guidance is primarily based on the underlying queue storage which also guarantee at least once execution.