Hello,
Welcome to our Microsoft Q&A platform!
In UWP, you can not directly reference the .NET framework library, you can try to create a .NET Standard library project that can reference your dll and then reference this .NET Standard project to your UWP app.
In addition, here is a IKVM Nuget package, you could try to download it and directly use it in UWP app to see if it could interact with java.