How to handle response in Parallel.ForEachAsync?

Question

Hi folks,

I have written a ASP.NET Core Web API that retrieves game codes from another external API. The process works like this: the game code and the number of codes to be retrieved are sent in the request. However, there is a restriction in the external API I am retrieving the game codes from, which is that only 10 game codes can be retrieved in one request. This process is currently being done in various chain stores' cash registers. Only one game code purchase transaction is made from the cash register.

However, there can be customers who want to bulk retrieve thousands of game codes online. To achieve this, I added a new method to the API to enable multiple purchases by looping. The requests are sent to the external API in small pieces, with 10 being the requested amount of game codes. This process works without any problems because each successful request and response is recorded in the database. This process is carried out through the ASP.NET Core interface and has a limitation: if the user inputs the amount of game codes requested through the interface, it takes a long time to retrieve thousands of game codes as the maximum is 100 (to avoid time-out issues, etc.).

To improve this situation, I created a worker service that operates in the background. The user inputs the total request through the web interface, which is converted into 100s and recorded in the database. The worker service retrieves these requests one by one randomly, then sends the requests to the API I created and then to the external API. The new process in the worker service is as follows: when 100 game code requests are made, the maximum parallelism is 2 and they are sent in Parallel.ForEachAsync, divided into 50/50. The requests are processed in the manner described in 10s, as previously mentioned. My concern here is if 100 game codes are successfully sent and retrieved, I update the related record in the database. However, if an error occurs somewhere in the process of processing the external API in 10s, my API will return a 500 error. I'm not exactly sure whether the Parallel.ForEachAsync will continue processing the other requests or if the operation will be cancelled. I was unable to test this scenario. What logic would be appropriate to construct here? How to mock the service in order to get error once ina while and check the update logic.

var num = amount;
var firstNum = 50;
var secondNum = 50;

if (num < 100)
{
  firstNum = (num + 1) / 2;
  secondNum = num - firstNum;
}
var quantities = new List<int> { firstNum, secondNum};
                    var cts = new CancellationTokenSource();
                    ParallelOptions parallelOptions = new()
                    {
                        MaxDegreeOfParallelism = 2,
                        CancellationToken = cts.Token
                    };
                    try
                    {
                        await Parallel.ForEachAsync(quantities, parallelOptions, async (quantity, ct) =>
                        {
                            var content = new FormUrlEncodedContent(new[]
                            {
                                new KeyValuePair<string, string>("productCode", productCode),
                                new KeyValuePair<string, string>("quantity", quantity.ToString()),
                                new KeyValuePair<string, string>("clientTrxRef", bulkId.ToString())
                            });

                            using var response =
                                await httpClient.PostAsync(_configuration["Razer:Production"], content, ct);

                            if ((int)response.StatusCode == 200)
                            {
                                var coupon = await response.Content.ReadFromJsonAsync<Root>(cancellationToken: ct);

                                _logger.LogInformation("REFERENCE ID: {referenceId}", coupon.ReferenceId);

                                await UpdateData(id);
                            }
                            else
                            {
                                _logger.LogError("Purchase ServiceError: {statusCode}",
                                    (int)response.StatusCode);
                            }
                        });
                    }
                    catch (OperationCanceledException ex)
                    {
                        _logger.LogError("Operation canceled: {Message}",
                            ex.Message);
                    }

My web API is cut into small pieces (10) and calls this 3rd party web API.

    
public async Task<HttpResponseMessage> CallRazerServiceMultiple(RequestDto requestDto)
        {
            HttpResponseMessage response = null;
            var customerId = int.Parse(_http.HttpContext.User.Claims.FirstOrDefault(x => x.Type == "UserId")?.Value ?? throw new InvalidOperationException());
            var coupons = new List<Coupon>();
            GameConfirmResponse gameConfirmResponse = null;

            var requestedAmount = requestDto.quantity;
            var requestClientCode = requestDto.productCode;

            //ProductCode Conversion
            var productCode = _unitOfWork.ProductCodeRepository.GetByCode(p => p.clientCode == requestDto.productCode);
            if (productCode != null)
            {
                requestDto.productCode = productCode.gameCode;
            }
            
            while (requestedAmount > 0)
            {
                var gameRequest = _mapper.Map<RequestDto, GameRequest>(requestDto);

                var count = Math.Min(requestedAmount, 10);
                gameRequest.quantity = count;

                //Unique reference ID
                gameRequest.referenceId = Guid.NewGuid();
                
                var gameRequestDto = _mapper.Map<GameRequest, GameRequestDto>(gameRequest);

                //Create signature
                gameRequest = Utilities.Utilities.CreateSignature(gameRequestDto, RequestType.Initiate,_configuration);

                //Set service
                gameRequest.service = "RAZER";
                gameRequest.customerID = customerId; //Get from token

                //Add initiation request into database
                _unitOfWork.GameRepository.Insert(gameRequest);

                #region Call Razer initiate/confirm

                //Call Razer for initiation
                response = await RazerApiCaller.CallRazer(gameRequest, "purchaseinitiation",_configuration);

                //Read response
                var htmlResponse = await response.Content.ReadAsStringAsync();

                var gameResponse = JsonConvert.DeserializeObject<GameResponse>(htmlResponse);

                //Adding initiation response into database
                _unitOfWork.GameResponseRepository.Insert(gameResponse);

                if (gameResponse.initiationResultCode == "00")
                {
                    gameRequestDto = _mapper.Map<GameRequest, GameRequestDto>(gameRequest);

                    gameRequestDto.validatedToken = gameResponse.validatedToken;
                    //Create signature
                    var gameConfirmRequest = Utilities.Utilities.CreateSignature(gameRequestDto, RequestType.Confirm,_configuration);

                    //Transform DTO into GameRequest for calling Razer Initiate

                    var gameConfirmRequests = _mapper.Map<GameRequest, GameConfirmRequest>(gameConfirmRequest);

                    //Add confirm request into database
                    _unitOfWork.GameConfirmRequestRepository.Insert(gameConfirmRequests);

                    //Call Razer for confirm
                    response = await RazerApiCaller.CallRazer(gameConfirmRequest, "purchaseconfirmation",_configuration);

                    //Read response
                    htmlResponse = await response.Content.ReadAsStringAsync();
                    //Convert UTC to Local Time
                    var settings = new JsonSerializerSettings { DateTimeZoneHandling = DateTimeZoneHandling.Local };

                    gameConfirmResponse = JsonConvert.DeserializeObject<GameConfirmResponse>(htmlResponse, settings);

                    //add coupons
                    coupons.AddRange(gameConfirmResponse.coupons);

                    //Set service and client code
                    gameConfirmResponse.service = "RAZER";
                    gameConfirmResponse.productCode = requestClientCode;
                    gameConfirmResponse.status = 1; //Default Confirmed!

                    //Add confirm response into database
                    _unitOfWork.GameConfirmResponseRepository.Insert(gameConfirmResponse);

                    //Add Confirm Cancel request into database - Confirmed!
                    
                    var confirmCancel = new ConfirmCancel { referenceId = gameConfirmResponse.referenceId, status = 1, customerID = customerId};
                    _unitOfWork.ConfirmCancelRepository.Insert(confirmCancel);
                }
                
                requestedAmount -= count;

            }
            #endregion Call Razer initiate/confirm

            await _unitOfWork.SaveAsync();

            if (gameConfirmResponse == null) return response;
            gameConfirmResponse.coupons = coupons;
            gameConfirmResponse.quantity = requestDto.quantity;
            gameConfirmResponse.totalPrice = gameConfirmResponse.unitPrice * requestDto.quantity;

            //Manipulate Response in order to send Client Product Code
            var resultResponse = JsonConvert.SerializeObject(gameConfirmResponse, Formatting.Indented,
                new JsonSerializerSettings()
                {
                    ReferenceLoopHandling = ReferenceLoopHandling.Ignore
                });
            response = new HttpResponseMessage
            {
                StatusCode = System.Net.HttpStatusCode.OK,
                Content = new StringContent(resultResponse, System.Text.Encoding.UTF8, "application/json"),
            };

            return response;


        }

I couldn't find how to check if there is a problem with one of your requests in the 3rd party web API. How should I handle it in Parallel.ForEachAsync? I mean let's say I want 100 codes (quantity), I am sending the requests to the 3rd party service 10 by 10. How to deal with an error let's say in the 4th request in the while loop?

Thank you.