How to extracting all user records from AAD via MSgraph and Azure Data Factory Copy Data Activity Very large user pool

Question

I am trying to export all the User records from our AAD, there are around 700,000 records expected. But I can't get it to return more than a few thousand at best.

I have a linked service set up to MSGraph API that I know works as I get other data from AAD via the same service.

I've created a pipeline in ADF with a Copy Data activity, I've added ConsistencyLevel=eventual to the header.

the Relative URL is: users?$top=999&$count=true

When I debug this today, I got only 800 records instead of the expected 800K, strangely when I ran it yesterday, I consistently got 13666 records.

The users are students, so I thought I'd try to extract users from just one school, onPremisesExtensionAttributes/extensionAttribute5 holds the school number so I added a $filter, I also added a $select to the query string.

users?$top=999&$count=true&$filter=onPremisesExtensionAttributes/extensionAttribute5 eq '8897'&$select=accountEnabled,createdDateTime,department,displayName,givenName,jobTitle,mail,mailNickname,onPremisesExtensionAttributes,surname,userPrincipalName

I expected 2100 records but only got 3.

I'm sure I'm doing something wrong or stupid here but can't figure it out. Any help would be very much appreciated.