Adding the workarounds I found for people who might land here.
I found 2 work arounds and I am using both of them for different sets of users.
1) First workaround is setting 'Restricted Read' as permission level of the User. With this users cannot access documents library but the CEWP easily reads the files within it and renders on site page when those users open it. The issue with this solution is it stops the user from accessing other files too which I want them to be able to open.
2) The second solution is adding a script snippet in my MasterPage which redirects the user to home page of the SharePoint Site if the User tries to access specific files. The Code reads permission level of the user, and then matches the current URL of the page to the URL of the document I want to restrict. It redirects the user to homepage site if the URL is that of the document.
Hope what I wrote is intelligible 😅