Hi @Rohan Lukose , welcome to Microsoft Q&A forum.
You are right in mentioning that if failover is initiated from Server A to Server B, geo-replicated replicas will also have their primary changed to Server B. You can read the same in the below article:
Creating and using active geo-replication - Azure SQL Database
Also while doing the manual failover also, we get the same message as shown in below screenshot:
Please let me know if this helps or else we can discuss further.
----------
If answer helps, you can mark it 'Accept Answer'