Depends on the firewall. The built in firewall of Windows is an application-level firewall. Web browser traffic is automatically allowed through and isn't managed here. You can however add or deny specific applications from the firewall.
For a network level firewall you need to use the client built into the firewall. For example if you use Cisco then they have a UI that allows you to specify what DNS entries are allowed to be called from the machine you configure it against.