Yes, you can restrict access to specific site(s) only by leveraging the method detailed here: https://devblogs.microsoft.com/microsoft365dev/controlling-app-access-on-specific-sharepoint-site-collections/
If you need more detailed steps, I've put some here: https://www.michev.info/blog/post/3256/limiting-access-to-sharepoint-online-resources-via-the-graph-api