After searching around I noticed adding them to the SQLAgentUserRole within msdb looks like pretty much does this just for the DB(s) that they own.
In testing around looks like a regular Login-User cannot make any configuration changes to SQL Server Agent which is good (or please let me know if I missed something that they can change which is undesirable).
But when I tried to Stop-Start-Restart services for SQL Server Agent I was able to logged in as a regular Login-User which is not good...is there a way to prevent regular Login-User to not be able to