if site.js is working, than static files are being returned. use the browsers network tool to verify the path used to access the files.
note: builder.UseStaticWebAssests() is used to support static files that are part of a Razor Class Library. Basically stored as a resource instead of a file.