The backup cleanup task does not delete based on the files, it deletes based on the backup log. If the log does not exist for the backup file, it does not delete it. If you moved the files from another location, it will never delete them.
What happens if you run:
EXECUTE master.dbo.xp_delete_file 0,N'{yourdirectory}',N'bak',N'{3daysago}'
