.NET Core Worker Service Deployment

Keyur Narkhi 1

I have long running task which is reading large file(3GB,5GB) excel,txt from server and dump to SQL. So I have created a Worker service and deployed that to Kubernetes Cluster and POD. I have also applied Horizontal Pod scaling but my concern is if file is half way and if POD is not able to load that and due to scaling if new POD replica is created then how I can manage the file which is already processed half way and start from their only.

Can you help me with best solution.

Thanks
Keyur

shiva patpi 13,171 Reputation points Microsoft Employee

2021-11-23T23:46:49.09+00:00

Hello @Keyur Narkhi ,
I am trying to understand your current application design .
what exactly that excel file contains ? Is that a set of records (rows,columns)?
Does that application read each row at one time and insert into SQL?
Let's say if the excel file has got 100 records , after reading 50 records and inserting to SQL , if the POD crashes and spins up the new POD - you are asking - how the new POD can continue reading from 51st record instead of from start ?

Regards,
Shiva.
Keyur Narkhi 1 Reputation point

2021-11-24T06:58:10.623+00:00

Hi Thanks for the reply.

I have bunch rows with columns in excel. I am reading the file and then using bulkinsert dumipng to database. so I might need to change and use the batch process to perform small number of records to insert and when load increase how newly created POD handle the state ? that is my concern.

Share via

.NET Core Worker Service Deployment