@Daniel , Adding to AgaveJoe, If the library is something the main app uses it likely is already a reference in the WebApp project and gets copied with the WebApp. Publishing the WebApp only in this case should be sufficient for your scenario. If for some reason the library is not referenced by the WeApp and you are loading some other way, could you share more details on how/why you might be doing that?
See these Azure App Service docs for more info:
File structure on azure
Customizing deployments
Deployment vs runtime issues
P.S. I'd added additional tags to receive insights from the targeted SMEs/community.