@Laco It seems like your SSMS session is getting disconnected after a period of inactivity.
It needs to establish the connection first before executing the query resulting in longer overall duration.
Also, I notice that Execution Plan is included with the results, turning it OFF will improve execution time
----------
If an answer is helpful, please "Accept answer" or "Up-Vote" which might help other community members reading this thread.