You can't specifically disable it no, but if the port is not open, clients will fallback to 80 or 443 for client notification. This will cause an increased load on the MP(s) though.
I would seek an exception here though because in this case, although the cert is self-signed, it is created and controlled by ConfigMgr and not just a randomly generated cert.