azure durable functions support your case:
Implementing parallel processing of sequential tasks
I am having a asp.net core web application hosted as app service on Azure. This application takes the files uploaded by users as input. The records in this file (which can be in thousands) has to be processed by 3 separate processes. These 3 processes are time taking processes which will run in background as windows services as these services needs to access on-premise file system and they must run in sequence for each user to process the records in the uploaded file.
Let's say process from file upload till last process(Process1, Process2, Process3) as a work. So, inside single user the processes needs to run in sequential order but works should run in parallel.
User1 ----> File Upload > Process1(if File Upload is success) > Process2(if Process1 is success) > Process3(if Process2 is success) [In sequential order]
User2 ----> File Upload > Process1(if File Upload is success) > Process2(if Process1 is success) > Process3(if Process2 is success) [In sequential order]
User3 ----> File Upload > Process1(if File Upload is success) > Process2(if Process1 is success) > Process3(if Process2 is success) [In sequential order]
User1, User1 and User3 can be parallel.
When user uploads a file with 1 record, this web application will insert 1 message in azure service bus queue(which is session enabled queue). Each message in queue must be processed by all 3 windows services(Process1, Process2 and Process3) in sequential order.
But for different users the execution should be in parallel so that execution process of user1 must not delay the execution for user2. For that I am trying to use concurrent sessions of Azure service bus.
Not sure if this is correct use case for Azure service bus or not, hence I need if anyone could put me in right direction.
I am looking help for design solution, appreciate . Any ideas and implementation suggestions are welcome.
I can also provide code which I am using to send and receive concurrent messages of Azure service bus session enabled queue.