By defulat, I believe that all users with Databricks SQL access also have browse access to the Catalog Explorer.
You can use role-based access control (RBAC) to grant users only the permissions they need. For example, you could create a role that grants users the ability to run SQL queries on Delta tables, but not to browse the Catalog Explorer. Then, you could assign that role to the users who need it.
Or use Unity Catalog ACLs to restrict access to specific tables and other objects where you could grant users read access to specific Delta tables, but not to the entire catalog.