Exchange 2016 mass inbound email and CPU

Anonymous
2021-11-23T16:26:09.647+00:00

Hello, we have a single Exchange 2016 server running as a VM allocated with 6vCPU and 28Gb RAM. 2 databases and about 1000 mailboxes. We have a smart host appliance provided by Barracuda at the perimeter responsible for handline all inbound/outbound mail. Yesterday one of our vendors sent a mass email to all employees and the CPU of our mail server spiked to 100%. This caused some Outlook clients to drop offline intermittently and the smart host started spooling mail. I saw two different SMTP responses in the smart host for deferred messages that could not be delivered to our Exchange server at this time..

  1. refused to talk to me: 421 4.3.2 The maximum number of concurrent server connections has exceeded a per-source limit, closing transmission channel
  2. I don't have the exact error unfortunately for the second, it was something about insufficient resources to service the request.

Anyway, this all eventually worked out. The messages spooled on the appliance and as Exchange CPU usage reduced all messages were eventually delivered successfully.

Question is, what can be done to prevent this scenario in the future. This mass email from our vendor resulted in a mini DDoS attack essentially. I am very surprised there is no mechanism in our Barracuda smart host or Exchange to deal with something like this. Some kind of sending domain rate limit mechanism maybe? Is there any feature of Exchange that could help here?

Exchange Server Management
Exchange Server Management
Exchange Server: A family of Microsoft client/server messaging and collaboration software.Management: The act or process of organizing, handling, directing or controlling something.
6,096 questions
{count} votes

4 answers

Sort by: Most helpful
  1. Andy David - MVP 110.7K Reputation points MVP
    2021-11-23T16:37:51.743+00:00

    Well, actually there was. Exchange backed off the connection (421 4.3.2) and forced the barracuda to essentially slow down.

    The bigger issue is the mail delivery on Exchange itself caused issues. Was this server sized with the mailbox calculator?

    https://techcommunity.microsoft.com/t5/exchange-team-blog/released-exchange-server-role-requirements-calculator-8-3/ba-p/605559

    6 vCPU and 28GB RAM seems really undersized to handle spikes.


  2. Andy David - MVP 110.7K Reputation points MVP
    2021-11-23T20:24:20.77+00:00

    well, the proof is in the pudding :)
    You have seen that it cant handle the spike when it happened. All I'm saying is that if 99% of the time Exchange is running fine, then not sure I would worry about the specs, however if you want to be ready for the next time this happens, then you have to either scale up or out. So either add more resources to the existing server or add more servers ( And build a DAG) and spread the load across multiple servers.
    I've actually seen a few people complain about this same issue. I personally have never seen it because when I always scale out and try to have more than I need server/resource-wise.
    However, I get there are costs with either solution, but that is really what is needed.

    As for preventing unlimited messages , there is throttling in Exchange that prevents that backs off connections by IP and time etc...
    The defaults are set in the receive connectors and transport service.
    https://learn.microsoft.com/en-us/powershell/module/exchange/set-receiveconnector?view=exchange-ps
    The MaxInboundConnectionPerSource parameter specifies the maximum number of connections that this Receive connector serves at the same time from a single IP address.
    A valid value is from 1 to 10000, or the value unlimited. The default value is 20.
    To disable the inbound connection per source limit on a Receive connector, enter a value of unlimited.

    But based on your description, that wasnt the issue you faced, since you saw this error "421 4.3.2 The maximum number of concurrent server connections has exceeded a per-source limit, closing transmission channel"

    it sounds to me like Exchange throttled the connection, but what it couldnt handle was actually processing all those messages once they were received.


  3. Andy David - MVP 110.7K Reputation points MVP
    2021-11-23T20:32:20.967+00:00

    Correct, it doesnt look at the original sending IP, just the server connecting directly to it.

    Yes, Load Balancer is recommended. To deliver you can have the smart host deliver to the load balanced IP or FQDN if it cant handle multiple IPs


  4. Andy David - MVP 110.7K Reputation points MVP
    2021-11-23T20:50:53.997+00:00

    f5 has always been a good choice, but its pricey.

    Kemp is a good solution:
    https://kemptechnologies.com/microsoft-load-balancing/load-balancing-microsoft-exchange/

    However, if you are planning to move to 365, I'd just beef up that existing server until you do.

    No comments