Hello and welcome to Microsoft Q&A @Pär Adeen .
It looks like you are doing a non-standard organization of Factories and Git. I would like to get a little more clarity on the goals here.
Usually there is one Factory for each environment (dev, test , prod). Here you are "... moving the development environment from one Factory to another Factory ...". So you want to have your development branch/environment straddling 2 Factories? Can you tell me why, or what this is in pursuit of?
During a "Publish" action, the collaboration branch is pushed to the Factory's "Live Mode" and "adf_publish" branch. This action keeps the Live Mode and adf_publish in synch. However if there are two Factories reading from the same Git, only one of them would receive the benefit of this synchronization. The one not involved in the Publish action will remain with outdated "Live Mode".