Do I only need to give site server access to database only?
That depends on which roles are hosted on any additional site systems. The primary site server itself definitely needs access.
Do I need give All SCCM servers (e.g. DP, MP) to database
No. As noted, it's about roles. If you review the port documentation (at https://learn.microsoft.com/en-us/mem/configmgr/core/plan-design/hierarchy/ports), it lists all roles that need access to site's DB:
- Datawarehouse service point
- Endpoint Protection point
- Management point
- Reporting Service point
- SMS Provider
- State migration point
Note that there are two additional roles that require access, but they are for deprecated functionality (AI Sync point and Enrollment point). Also note that your SUP instances require access to the WSUS DB whever that may sit as this may be separate from the site's DB.
Do client also need access to database?
No.