The simplest way is to leverage a Network Security Group on the Application Gateway subnet, create an inbound rule to specify the IP and another rule to deny other traffic.
As you have Azure Firewall in your architecture, you could also leverage this by creating a rule.