
Sorry for the above misleading answer. Here is a new answer for correction and clarify.
1.The Search settings in the Site settings (subsite) only effect classic search. If you use modern search, the search result page is designed, and cannot be customized.
After setting Search settings in the Site settings (subsite), please go to Site contents -> Return to classic SharePoint, then do classic search, it redirects to custom search result page you have set in the Search settings.
Note: You should wait for a few time, at least 30 minutes.
2.Your requirement is to make users in the subsite can only search contents in the subsite. Is it right? It can be achieved through Permission settings. Users who have permissions for the subsite but not for the site collection can only search the contents in the subsite.
(1)Go to Site settings of sub site -> Site permissions -> Stop Inheriting Permissions -> Create a new group.
2.Re-grant permissions to users in the subsite groups again.
3.Remove all site collection groups.
4.Go to Site settings of site collection -> Site permissions -> Remove users that don't need permission for the site collection.
5.Do a full crawl.
After above steps, when users search in the subsite, they can only search the contents in the subsite.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.