Implementing parallel processing of sequential tasks

Anirud Thapliyal 96 Reputation points
2021-07-07T17:05:21.72+00:00

I am having a asp.net core web application hosted as app service on Azure. This application takes the files uploaded by users as input. The records in this file (which can be in thousands) has to be processed by 3 separate processes. These 3 processes are time taking processes which will run in background as windows services as these services needs to access on-premise file system and they must run in sequence for each user to process the records in the uploaded file.

Let's say process from file upload till last process(Process1, Process2, Process3) as a work. So, inside single user the processes needs to run in sequential order but works should run in parallel.

User1 ----> File Upload > Process1(if File Upload is success) > Process2(if Process1 is success) > Process3(if Process2 is success) [In sequential order]
User2 ----> File Upload > Process1(if File Upload is success) > Process2(if Process1 is success) > Process3(if Process2 is success) [In sequential order]
User3 ----> File Upload > Process1(if File Upload is success) > Process2(if Process1 is success) > Process3(if Process2 is success) [In sequential order]

User1, User1 and User3 can be parallel.

When user uploads a file with 1 record, this web application will insert 1 message in azure service bus queue(which is session enabled queue). Each message in queue must be processed by all 3 windows services(Process1, Process2 and Process3) in sequential order.
But for different users the execution should be in parallel so that execution process of user1 must not delay the execution for user2. For that I am trying to use concurrent sessions of Azure service bus.

Not sure if this is correct use case for Azure service bus or not, hence I need if anyone could put me in right direction.
I am looking help for design solution, appreciate . Any ideas and implementation suggestions are welcome.
I can also provide code which I am using to send and receive concurrent messages of Azure service bus session enabled queue.

Azure Service Bus
Azure Service Bus
An Azure service that provides cloud messaging as a service and hybrid integration.
700 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
8,930 questions
Developer technologies C#
{count} votes

1 answer

Sort by: Most helpful
  1. Bruce Barker 801 Reputation points
    2021-07-07T22:23:05.217+00:00

    azure durable functions support your case:

    durable-functions-overview

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.