I don't know why this fails. Or even how to troubleshoot it. The challenge is that for such a task you need to know both sides. And I only know SQL Server.
Where is the Oracle database? On-prem? Another cloud? At least, it is not in Azure. And for communication from/to Azure, you will always have to count with transient network issues.
I note that you say that this happens on-prem too. But if the Oracle database is another cloud, you still have the same situation. And even if the Oracle database is local, there may still be network issues.
It may be that you just has to accept things as they are and design your solution to handle. That is, retrieve row in reasonably sized batches, and if a batch files retry that batch. I can see that this seems like an overkill for something that runs on-prem only. But as soon as you involve a cloud, you more or less has do to this.