Expand Operator is documented here expand parameter which explains how the operator works.
Ther query should work with balanced brackets. I was not able top produce a working Graph Query with unbalanced brackets. Take the query below:
GET /groups?expand=members($select=id,userPrincipalName,displayName)
Its a valid query and thats how the expand operator should work. If you have a complete query that behaves differently, please share with us including request id and timestamp in headers to help track.