Yes, you will have to register and define IPV6 addresses in Azure AD conditional access named location.
You can have both IPv4 and IPv6 defined under named locations. Azure AD will consider both the IP ranges and will act accordingly to the policy where this specific named location in used in conditional access policy.
We created an easy to remember link where we’ll continue to share additional guidance on IPv6 enablement in Azure AD. Access these details here: https://aka.ms/azureadipv6.
Let me know if you have any further questions.
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.