So I found this:
https://techcommunity.microsoft.com/t5/azure-active-directory-identity/smartcard-logon-considerations-or-how-i-learned-to-love/ba-p/243066
In a normal Windows logon situation, the user logs onto a domain client by providing a username (giving their uniqueness as a principal), the domain (read realm) that principal resides in, and their password. This information is used to request a TGT, and whatever service tickets that principal needs to get access to requested resources. This information (username, password, TGT, TGSs) are locally cached and provided as needed “behind the scenes” in Windows.
With smartcard logon there is a slightly different, and more secure, way of doing things. This can lead to unexpected things at times. One of the behaviors that makes Kerberos authentication secure in that way we love so much is the (configurable) renewal and renewal lifetime for the ticket granting ticket. Our customers who have begun a transition to a smartcard logon only environment have noticed the following scenario.
The user logs on to the computer using his or her smartcard. Once logged on and the user environment created, the user starts some long-running task (like a SQL query) that takes a very long time. Then the user’s day is over and he or she goes home to do whatever it is real people do when they are not using Microsoft products. I won’t speculate about what that may be, I can’t conceive of it. Let’s forge ahead. When they come back into the office the next morning they discover that the job they had running has failed with an error 5 (access denied). Why would that be? The credentials were correct at logon or it wouldn’t have ever successfully started, not to mention the fact that the access denied would have been thrown in the user’s face at the beginning of the job, not at some indeterminate time later. What gives?
In the above situation the situation was a result of the ticket renewal process whereby the TGT is renewed at default 10 hour intervals. Picture the user logging on, working their typical eight hour or whatever day and running this job which takes, say, 12 hours to complete. Sometime in this 12 hour interval, after logging on and after starting the job, the TGT needed to be renewed (seen as the End Time in kerbtray and other tools). In Windows 2000 and XP, this renewal required the smartcard to be present in order for it to succeed. Since the user requires the smartcard for everything at the office, perhaps even opening doors, they naturally took it with them with they left, so the smartcard was not present at renewal time. Result: failed TGT renewal, and since the TGT was not renewed service tickets-like those being used for that long-running job-would be expired as well. For those out there who are wondering, you should not see this with Windows 2003 SP1 or Windows Vista. I’ll add a little caveat there since some of this could be altered by the CSP being used or the application which that job is and how it utilizes the SSPI. Use your tools (system information file or debugger to ascertain CSP, kerbtray et cetera) to find out more if you see other results.
This details exactly what I am experiencing!
The only problem is that the author writes:
"For those out there who are wondering, you should not see this with Windows 2003 SP1 or Windows Vista. "
But we are seeing this with the latest builds of Windows 10...
So according to the above, we increased the time of the TGT value in the Default Domain Policy and that solved the issue, but it seems like the value is really dependent on use case, so if I increase the value to 24 hours, that will only help a user who is away OOO for 48 hours, or 72... basically, I need to increase this value for the maximum duration possible, which could be limitless in certain cases.