There is no requirement for the FSW to be a resource of the AG and you should see no detrimental affect from removing it. The FSW will remain a resource of the CCRG.
Here is a guide I have used in the past for deploying an FSW:
An FSW can be used on multiple clusters for high availability.
I hope this answers your question.
--If the reply is helpful, please Upvote and Accept as answer--