This looks to be fixed now. The short answer is that the Purview preview connector only works with JDK 11 - not higher or lower. The error I received (Error: (3913) JavaException: Must have Java 8 or newer installed.) is therefore a misleading error, likely carried over as-is from Data Factory (which Purview is seemingly built from) and hasn't yet been updated. It should read something like "Requires Java 11 to work".
For my situation, Erwin has Java 8 included within its package which is why the original error was strange. It's not possible to upgrade that version of Java, so instead I had installed JDK 17 (the latest) for the IR to use. Therefore my fix was to uninstall Java 17 and replace it with Java 11.
Assumedly once this connector is out of preview then it will support higher/latest versions of Java, but that's the solution for now.