Yes we have UI tests running on a CI pipeline without using AppCenter.
But no not with Octopus, but the process should be similar.
I found a blog post on how to do this with Android and Azure DevOps, I'll see if I can dig it out and post the link here. You will then need to modify the process to work for your CI setup.
EDIT: Here is the link https://medium.com/genetec-tech/android-ui-testing-in-azure-devops-81bbe7cea9fd
Ignore the setup caching until the end (its optional), but using that post you should be able to download and install an AVD of your choice, run the UI tests and publish the results with screenshots, if required.