School Data Sync
Frequently asked questions
What is School Data Sync (SDS)?
SDS is a free service in Office 365 Education that reads the rosters from your Student Information System (SIS) / Student Management System (SMS). It creates classes for Microsoft Teams, Intune for Education, and third-party applications. Microsoft Teams brings conversations, content, and apps together in Office 365 for Education.
What SIS/SMS vendors does SDS support?
SDS supports importing data in CSV (Comma Separated Value) files therefore it supports virtually every SIS / SMS on the market. SDS also supports importing roster data via the industry standard OneRoster API (Application Programming Interface).
For more information about supported API vendors, see OneRoster API Provider Overview.
What happened to SDS for Insights?
SDS for Insights was the first building blocks of the bridge between SDS (Classic) and sharing data to Insights and Analytics experiences. From the beginning, it was built to allow us to extend it beyond just supporting sharing data for Insights & Analytics experiences. As part of growing the platform to now support SDS (Classic) provisioning experiences to manage Microsoft 365 Users and Groups, we've rebranded to SDS.
Where is SDS available?
SDS is currently available in all regions worldwide except for China and Germany.
Sync runs continuously after the inbound flow is created. - For OneRoster API, the connection to the SIS is continuous and always polling for changes in data to be synced. - For CSV, changes within your data can be synchronized by uploading new CSV files that contain the data changes.
What apps work with SDS?
SDS imports organization, users, user role association to organizations, and roster data from a SIS / SMS to Office 365 so numerous first party and third party applications can use it. To see a list of the third party education apps that use Office 365 and SDS data for single sign-on and Rostering integration, visit https://sds.microsoft.com.
Will SDS automatically synchronize changes, or do we have to restart sync to synchronize changes as they occur?
If you're synchronizing via API, an automated run will occur every 12 hours.
If you're synchronizing via CSV file, a run will occur upon the initial upload, and trigger a run again when new CSV files are uploaded. Keep in mind that an automated run every 12 hours will still occur to process Azure AD changes or to support Insights and Analytics scenarios.
For more information monitoring and handling of data errors, warnings, and troubleshooting see Health and Monitoring
What are the permission requirements for accessing and managing SDS?
To access and manage SDS, your account must be a global administrator within the tenant.
How can we export data from our SIS / SMS to Microsoft’s required CSV format?
Since each SIS/SMS is different, we encourage SDS customers to contact their SIS/SMS vendor for support and assistance with building the appropriate export. SDS V2.1 CSV file format.
What is the proper format for the StartDate and EndDate values?
SDS requires data values to be in ISO 8601 format (YYYY-MM-DD).
What is the proper format for the Phone or SMS values?
SDS requires Phone and SMS to be in E.164 and + must be included (+1234567890).
Can I export the errors / warnings generated by School Data Sync?
Yes, you can export the list of errors and warnings generated from a run within the SDS UI. For more information monitoring and handling of data errors, warnings, and troubleshooting, see Health and Monitoring
Does Microsoft provide extractor tools for my SIS/SMS data?
Microsoft doesn't build or maintain extractor tools for any SIS/SMS vendor. Many SIS’s/SMS’s have data extraction tools built into the SIS/SMS already. If your SIS/SMS doesn't include an extraction tool, and you need assistance extracting data from your SIS/SMS into our SDS V2.1 CSV file format, contact your SIS/SMS vendor for support.
Why is there a character limitation on email addresses (or username values that are email addresses) in SDS?
Email addresses for all objects in Office 365 must adhere to several RFC standards for internet email addressing. SDS is aligned to the character limitations within each of the core Office 365 services, including SharePoint Online, Exchange Online, and Azure AD.
How many Inbound data flows do I need to create when setting up School Data Sync?
SDS today supports a single SIS/SMS Inbound data flow per tenant.
More than 2 million rows in a csv file – SDS no longer has the limit that SDS (Classic) has with ingesting source data that contains over 2 million rows in a csv file.
Multiple sync methods or source directories - If you have multiple SIS/SMS sources going into the same tenant, then you'll need to perform some pre-processing before making the data available to SDS. Contact our team.
Single source directory that needs to go to multiple tenants - If you have single SIS/SMS source going into multiple tenants, then you'll need to perform some pre-processing before making the data available to SDS for import. Contact our team.
Multiple domains for user identity matching – When configuring SDS you must match users from your source directory to users in Azure AD. See User Identity Matching.
Mix of "create new users" and "sync existing users" – As part of managing users between your source data and Azure AD and the mix can be done with a single Microsoft 365 managed users outbound flow.
Matching existing users occurs as part of the inbound data from the SIS/SMS and is written to the data lake. Until you create a Microsoft 365 managed users outbound flow, this link isn't pushed to Azure AD.
When setting up your Microsoft 365 managed users outbound flow by default, all matched users will have the link between the source user and Azure AD written to the users Azure AD object. If you wish to also create unmatched users, you can enable the option and specify the user creation rules.
In SDS (Classic) I also needed to upload empty classes and enrollment CSV files if only synchronizing users. Is this still true?
No. If you're only synchronizing users, you only need to provide the organizations (orgs.csv), users (users.csv), and user role and associations to organizations (roles.csv) files. See SDS V2.1 CSV file format.
Can I have extra headers and columns in my CSV files beyond what I intend to sync?
No. As part of basic validation, when the file names and headers are validated, the files must be formatted correctly. See SDS V2.1 CSV file format and Data Ingestion with SDS v2.1 CSV. If not, the UI won't allow progression through the "create inbound flow" or the Update Source Data with SDS v2.1 CSV scenarios. If automating the CSV upload, the files will be rejected and not moved over for SDS to pick up for the next run. They'll be removed from the cache after two days.
What happens if I select or upload other files beyond what the CSV format needs?
These files will be ignored and not moved over for SDS to pick up for the next run. They'll be removed from the cache after two days.
In SDS (Classic) I only needed to upload CSV files that have data changes in them. Is this still the same in SDS?
No. You MUST upload all the same files that were provided with prior uploads. If not the data associated with the missing files and their records will be identified as no longer active for the current academic year in the next run. (If you need to, you can upload all data files again to process for the next run). The SDS platform architecture supported by the data lake and how longitudinal data is stored is in support of Insights and Analytics capabilities. For more information see, section Determining data awareness and active status in Health and Monitoring.
If we manually remove a user or section, will they reappear when we synchronize again?
Manually updating a class roster won't be overwritten during the following run. SDS makes changes to the roster based on the last run, and not based on manual changes. The only exception is for Reset or Deleting and recreating the Microsoft 365 groups and Teams outbound flow. Aside from those exceptions, here's how SDS treats manual additions and deletions.
- A class is synchronized with a teacher and students
- The teacher goes to add a co-teacher to the class
- The class is later synchronized with no changes to teacher enrollment
- (Correct behavior) The co-teacher's membership is unaffected
- The class is later synchronized with the co-teacher added
- (Correct behavior) the co-teacher's membership is unaffected
- The class is later synced w/ the co-teacher removed
- (Correct behavior) the co-teacher is removed from the class
- A class is synchronized with a teacher and students
- The teacher goes to remove a student from the class
- The class is later synchronized with no change to the student enrollment
- (Correct behavior) The student's non-membership is unaffected
- The class is later synchronized with the student removed
- (Correct behavior) the student's non-membership is unaffected
- The class is later synced w/ the student readded
- (Correct behavior) the student is readded to the class
What do Azure AD Connect and SDS do and how can they work together?
Azure Active Directory Connect (Azure AD Connect) syncs on-premises Azure AD users, groups, and objects to Azure AD in Office 365.
SDS synchronizes other student and teacher attributes from the Student Information System (SIS) with existing users already synchronized and created by Azure AD Connect. Adding student and teacher attributes evolves the identity and enables apps to provide richer user experiences based on these distinguishable attributes and education personas. SDS allows you to automatically create class groups and Teams within Teams for Education, OneNote Class Notebooks, and class rostering for third party application integration. Azure AD Connect and SDS will never conflict, as SDS won't synchronize or overwrite any attribute managed by Azure AD Connect.
What special characters aren't supported by SDS?
There are several special characters that aren't supported within Azure AD. During outbound flow creation, you'll have ability to have SDS automatically replace unsupported special characters found with an “_”. Learn more about unsupported characters.
I'm not seeing my data for the next academic year. Is there anything I need to do?
When it's time to transition to the next academic school year you'll need to start the transition process, including creating a new inbound flow configured for the next academic school year. These steps are similar to when you initially onboarded and created your prior inbound flow.
The process doesn't require you to re-create new Microsoft 365 outbound flows. The existing outbound flow(s) and configurations will persist after the steps have been completed for your new inbound flow for the next academic school year. The next run will start processing the new academic year data beginning with inbound flow and continue processing the data with your existing outbound flows.
For more information, see Academic Year Transition.
In SDS (Classic) I currently use the SDS V1 file format, UK file format, and/or Clever file format. I want to move to SDS with SDS V2.1 CSV. What should I do?
Switching between formats should only be performed as part of back-to-school or academic year transition events and shouldn't be done within an active school year to avoid potential user disruptions.
User matching for existing users is done as part of the inbound flow and not immediately written to the Azure AD user object, follow the steps for Data Ingestion with SDS v2.1 CSV
In SDS (Classic) I currently use the SDS V2 file format. I want to move to SDS with SDS V2.1 CSV. What should I do?
To transition from V2 CSV to V2.1 CSV, tenants should provide data in the SDS V2.1 CSV file format
Next, follow the steps for Data Ingestion with SDS v2.1 CSV.
Continue with any top actions as needed based on the scenarios you need SDS to support.
In SDS (Classic) I was able to reuse a class between academic sessions or academic years. Am I still able to do this with SDS?
- There's support but there are limitations and caveats.
- Data in the Education data lake for Roster data is segmented with at least an Academic Year association based on the Admin configured Academic Year for the active Inbound flow. Supplying Academic Session data will still be bound to the Admin configured Academic Year for the active Inbound flow.
- Manipulation of supplied Academic Session data and dates will still be associated to the defined Academic Year and could have unsupported downstream effects.
- If the source system supplies the same sourcedId value for a Class between Academic Sessions and / or Academic Years, the platforms will reuse the same Azure AD Group, as long as the Azure AD Group hasn't been archived / processed to mark Classes expired with Group / Section cleanup options. When creating and managing the Azure AD Group, based on the supplied Class record from the source, the Azure AD Group attribute will be stamped with the SIS ID value, that will match the sourcedId value, for the corresponding Class record. This will allow us to link the Class to reuse the same Azure AD Group.
- As Class Group Memberships / Enrollments are processed during the same SDS Academic Year, changes will be processed as data is supplied. If any manual changes are made, but not present in the source data, those memberships will be retained only in the Azure AD Group membership data and not be reflected in the SDS Lake for Roster data.
- As Class Group Memberships / Enrollments are processed when transitioning to a new SDS Academic Year, any manual changes made, and weren't recognized in the data coming from the source system data, won't be recognized in the Education data lake for Roster data and will reflect what is defined in the source system. The manual membership changes will be retained in the Azure AD Group membership data.
- It's always recommended that the source system contains all the membership changes and are processed through SDS rather than relying on performing them manually.
- If the need is to have the Class Group Memberships / Enrollments that are from the source system, and not persisted from one Academic Year to the next for a Class / Azure AD Group, and you want them removed, the source data must not have the enrollments records the Class Group Memberships / Enrollments must be removed from the supplied source data and processed before the SDS Inbound flow expires for the previous Academic Year. If not you'll need to manually remove the records from the Azure AD Group.
I handle the provisioning of users and classes without SDS (manually, through Graph API, or using another solution), and I would like to synchronize my organizational data with Insights Premium dashboards. What should I do?
Your tenant will need to use SDS to supply the organizational data.
If going with CSV files, you don't need to provide data for classes and enrollments.
When configuring your inbound flow, you'll need to define how the application should match your users between the user data provided and what is in your Azure AD. See User Identity Matching.
When the user match runs, SDS will find the match and establish it based on the defined rule.
After the run, you'll need to understand your data health and correct any errors and warnings as needed. See Health and Monitoring.
When satisfied with your data health you'll need to Enable Sync to Insights Premium.
I use SDS V2.1 CSV format provision part of the users and classes of my tenant through SDS, but I would like to see data in Education Insights experiences for the entire organization. What should I do?
If your tenant has any users in its active directory that are using supported Microsoft apps for providing activity data to power Education Insights but aren't in the data uploaded to SDS, you won't be able to link those users and their activity in the Education Insights experiences.
If your tenant wants to link users to activity in the Education Insights experiences, you'll need to add the data to be sent through SDS. The tenant doesn't need to add specific class and enrollment information but must expand the data records it sends for organizations (orgs.csv), users (users.csv), and user role and associations to organizations (roles.csv).
The tenant can add the data to an existing extract to be processed with the next upload.
Some of my students and staff are enrolled in two academic units. How do I make sure that their data is reflected in both units, especially for Education Insights experiences?
With SDS V2.1 CSV format and OneRoster API, a user can be associated with more than one organization unit (for example, students who are registered in a special program at another facility). For SDS V2.1 CSV format, the tenant will need to provide two lines in the roles.csv file for that student – one for each organization association.
SDS V2.1 CSV format will also support users that have different roles at different organizations. Provide two lines in the roles.csv for the user, organization, and role combination and mark one of the roles as isPrimary = true.
In SDS (Classic) Settings page there are toggle options to manage Insights data. Where are these managed now?
In SDS (Classic) there were two toggle options available, under the Manage Education Insights section. One for Collect activity data for Insights and the other Allow Advanced Inferences. Both toggles are being moved from managed as part of SDS to the Insights Admin experience. Until the move is complete and the settings are retired from the SDS (Classic) settings page, if you manage the settings in either location, the last change made from either location will be used.
How is the isPrimary value processed with Inbound data?
For SDS V2.1 CSV
isPrimary indicates if this role is the primary (true) or secondary (false) role for that organization. Default value is false.
If passing multiple roles, for the same user and same org, there should be one ‘active’ primary role for the organization.
If isPrimary is set to true on multiple records (same user, multiple roles, same organization).
Write the data as received but identify the records as warnings.
If tenant doesn't like the handling, they'll need to provide the desired isPrimary role / record from the source data.
If multiple roles are passed for the same organization, and isPrimary isn't set to ‘true’ for one of the roles.
Write the data as received but identify the records as warnings.
If tenant doesn't like the handling, they'll need to provide the desired isPrimary role / record from the source data.
- Specification only allows passing one role per user.
- They can pass in the same role for a user to multiple organizations.
- Since one role per organization then records are marked as isPrimary ‘True’.
How are user identity match rules determined when multiple roles and/or organizations are associated to a user?
When a user has multiple roles, the following rules are used to determine what staff or student match rules should be used between the user record and the Azure AD user object.
- If isPrimary is set for all student roles, even if association to a staff role exists, the match is made based on the student role.
- If isPrimary is set for any staff role, even if association to a student role exists, the match is made based on the staff role.
- If isPrimary is set for both staff and student role, the match is made based on the staff role.
- If isPrimary isn't set for any roles, especially with a mix for both staff and student roles, the match is based on the staff role.
If the user is also associated with multiple organizations, the following is also used to determine the value when writing the role to the Azure AD user object.
- Utilize the Organization Role Sort Order value for the highest organization role to set to the Azure AD user object.
Why am I seeing errors and/or warnings raised in SDS from data runs?
For more information See Health and Monitoring.
What is the default List of Values supported?
For more information, see Default List of Values.
I was using SDS for Insights, can I still use SDS (Classic) for provisioning?
Yes. During your academic year transition, you can still continue to select the option "Ingest from my SDS (Classic) Sync Profiles." See Academic Year Transition.
This data source option won't be available for new Inbound flows starting June 2023.
I was using SDS for Insights, but I want to transition from SDS (Classic) to SDS to manage my Microsoft 365 user and groups. Is this possible?
Yes. See Transition from SDS (Classic). The following is a brief overview of the steps:
During your academic year transition, you'll need to select the option Connect to my data, then select Create new API or CSV source, then the data source format ‘CSV’ or ‘API’ and complete the remaining Inbound Flow wizard. See Academic Year Transition for more information.
After the first run, investigate your data health. See Health and Monitoring
When satisfied with your data health, follow the top actions on the home page to continue to set up your Outbound flows to manage your Microsoft 365 user and groups.
The Education Data Lake Export is showing 'Critical error' for the status. What should I do?
If the Education Data Lake Export status card shows 'Critical error' for the status, this means there's a problem when processing data. If it's been more than 24 hours and you continue to see the same status, contact support.
I created my Microsoft 365 groups and Teams flow right after my Microsoft 356 user flow, but the status shows 'Skipped' on the groups flow. What does this mean?
If you see a status of Skipped, it means that the flow wasn't included in the current or last Run. The scenario is seen when there's an active run and a request was made to create a new outbound flow. The new outbound flow will be included in the next run.
I see a prompt for 'Ready for next year?' but it shows all of my Top Actions have been completed. Can I create a new Microsoft 365 or Insights outbound flow?
Any remaining Top Actions will appear or setup buttons will be enabled after completing the steps to get ready for the next year. Select Start transition to begin when you're ready with your new data for the next Academic year.
After I onboarded to SDS and created a Microsoft 365 user flow, why are my SDS (Classic) sync profiles stopped?
Sync Profiles are stopped to prevent issues where both SDS and SDS (Classic) are managing a tenants Microsoft 365 users, groups and class Teams. Not doing so would result in creating data conflicts or overwriting data. The system will detect if there are active SDS (Classic) sync profiles and will stop them from running. Additionally the ability to interact with the SDS (Classic) sync profiles is prevented. The only interactions that are allowed are to select and view and SDS (Classic) sync profile and delete.
I have onboarded SDS and want to delete my SDS (Classic) sync profiles, but I see portions of the interface don't allow for any interactions. How do I delete them?
There are two ways to delete your SDS (Classic) profiles. This process doesn't interact with SDS or the data that SDS is managing in any way.
After navigating to SDS (Classic), you can navigate to each sync profile individually by using the navigation on the left. After selecting a sync profile, a dialog box is presented with an option to delete the sync profile. Select the option and follow the remaining prompts to complete the process. Repeat the step for each of your SDS (Classic) sync profiles.
After navigating to SDS (Classic), you can navigate to the Settings page. A dialog box is presented with an option to delete all SDS (Classic) sync profiles. Select the option and follow the remaining prompts to complete the process.
I onboarded SDS and created Microsoft 365 outbound flows, but I'm not happy with the current capabilities supported and I'm not able to interact with SDS (Classic). How can I go back to SDS (Classic) to provision and manage my user and group data?
First, make sure to work with your deployment manager or support if there are any concerns. If you still want to go back, for more information see Reverting back to SDS (Classic).