It is possible to create a gallery app with just user provisioning enabled. If group provisioning is enabled afterwards for the app, it can either be added as default enabled or default disabled for everyone.
It's worth noting that there is currently a backlog of requests for new SCIM provisioning connectors, so you may have time to implement both users and groups before the gallery integration is developed.
For questions of this nature (integrating with the Azure AD Enterprise Application Gallery), I'd recommend reaching out to the email aliases listed in the documentation for onboarding to the gallery rather than Microsoft Q&A.