SCCM Wake Up specify which IP to use

I would like to use SCCM’s wake up feature to wake our PCs from sleep. We use virtual box and thus our PCs have a second NIC that they use to communicate with the VMs. Unfortunately, the wake up feature does not want to work and in the BgbServer.log I can see that the IP it uses is the one of the virtual box NIC, which obviously wont work. In the properties pane of the device under "IP Addresses" I can see both, the virtual box NIC IP and the IP of the actual NIC. The same is the case for MAC addresses.
Is there some way I can get rid of the virtual box IP/MAC or somehow tell SCCM which one to use (or to try both?).
Thanks!