@Timothy
Thank you for your post and I apologize for the delayed response on this!
Based off your issue, App Source is currently not supported on the Q&A forums. However, I'd recommend reaching out to the App Source support team to see how you can make your Office Add-in "unlisted".
Thank you for your time and patience throughout this issue.
----------
Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.