School Data Sync

Frequently asked questions

What is School Data Sync (SDS)?

SDS is a free service in Office 365 Education that reads the rosters from your Student Information System (SIS) / Student Management System (SMS). It creates classes for Microsoft Teams, Intune for Education, and third-party applications. Microsoft Teams brings conversations, content, and apps together in Office 365 for Education.

What SIS/SMS vendors does SDS support?

SDS supports importing data in CSV (Comma Separated Value) files therefore it supports virtually every SIS / SMS on the market. SDS also supports importing roster data via the industry standard OneRoster API (Application Programming Interface).

What happened to SDS for Insights?

SDS for Insights was the first building blocks of the bridge between SDS (Classic) and sharing data to Insights and Analytics experiences. From the beginning, it was built to allow us to extend it beyond just supporting sharing data for Insights & Analytics experiences. As part of growing the platform to now support SDS (Classic) provisioning experiences to manage Microsoft 365 Users and Groups, we've rebranded to SDS.

Where is SDS available?

SDS is currently available in all regions worldwide except for China and Germany.

Sync runs continuously after the inbound flow is created.

  • For OneRoster API, the connection to the SIS is continuous and always polling for changes in data to be synced.
  • For CSV, changes within your data can be synchronized by uploading new CSV files that contain the data changes.

What apps work with SDS?

SDS imports organization, users, user role association to organizations, and roster data from a SIS / SMS to Office 365 so numerous first party and third party applications can use it. To see a list of the third party education apps that use Office 365 and SDS data for single sign-on and Rostering integration, visit https://sds.microsoft.com.

Will SDS automatically synchronize changes, or do we have to restart sync to synchronize changes as they occur?

  • If you're synchronizing via API, an automated run occurs every 12 hours.

  • If you're synchronizing via CSV file, a run occurs upon the initial upload, and triggers a run again when new CSV files are uploaded. Keep in mind that an automated run every 12 hours will still occur to process Azure AD changes or to support Insights and Analytics scenarios.

  • For more information monitoring and handling of data errors, warnings, and troubleshooting see Health and Monitoring

What are the permission requirements for accessing and managing SDS?

To access and manage SDS, your account must be a global administrator within the tenant.

How can we export data from our SIS / SMS to Microsoft’s required CSV format?

Since each SIS/SMS is different, we encourage SDS customers to contact their SIS/SMS vendor for support and assistance with building the appropriate export. SDS V2.1 CSV file format.

What is the proper format for the StartDate and EndDate values?

SDS requires data values to be in ISO 8601 format (YYYY-MM-DD).

What is the proper format for the Phone or SMS values?

SDS requires Phone and SMS to be in E.164 and + must be included (+1234567890).

Can I export the errors / warnings generated by School Data Sync?

Yes, you can export the list of errors and warnings generated from a run within the SDS UI. For more information monitoring and handling of data errors, warnings, and troubleshooting, see Health and Monitoring

Does Microsoft provide extractor tools for my SIS/SMS data?

Microsoft doesn't build or maintain extractor tools for any SIS/SMS vendor. Many SIS’s/SMS’s have data extraction tools built into the SIS/SMS already. If your SIS/SMS doesn't include an extraction tool, and you need assistance extracting data from your SIS/SMS into our SDS V2.1 CSV file format, contact your SIS/SMS vendor for support.

Why is there a character limitation on email addresses (or username values that are email addresses) in SDS?

Email addresses for all objects in Office 365 must adhere to several RFC standards for internet email addressing. SDS is aligned to the character limitations within each of the core Office 365 services, including SharePoint Online, Exchange Online, and Azure AD.

What special characters aren't supported by School Data Sync?

There are several special characters that aren't supported within School Data Sync when provisioning data to Azure AD. Unsupported special characters found will be replaced with an “_”. To learn about unsupported characters see Restrictions and limitations in OneDrive and SharePoint.

This handling happens automatically.

How many Inbound data flows do I need to create when setting up School Data Sync?

SDS today supports a single SIS/SMS Inbound data flow per tenant.

More than 2 million rows in a csv file – SDS no longer has the limit that SDS (Classic) has with ingesting source data that contains over 2 million rows in a csv file.

Multiple sync methods or source directories - If you have multiple SIS/SMS sources going into the same tenant, then you need to perform some pre-processing before making the data available to SDS. Contact our team.

Single source directory that needs to go to multiple tenants - If you have single SIS/SMS source going into multiple tenants, then you need to perform some pre-processing before making the data available to SDS for import. Contact our team.

Multiple domains for user identity matching – When configuring SDS you must match users from your source directory to users in Azure AD. See User Identity Rules.

Mix of "create new users" and "sync existing users" – As part of managing users between your source data and Azure AD and the mix can be done with a single Microsoft 365 managed users outbound flow.

  • Matching existing users occurs as part of the inbound data from the SIS/SMS and is written to the data lake. Until you create a Microsoft 365 managed users outbound flow, this link isn't pushed to Azure AD.

  • When setting up your Microsoft 365 managed users outbound flow by default, all matched users have the link between the source user and Azure AD written to the users Azure AD object. If you wish to also create unmatched users, you can enable the option and specify the user creation rules.

In SDS (Classic) I also needed to upload empty classes and enrollment CSV files if only synchronizing users. Is this still true?

For SDS v2.1 CSV: If you're only synchronizing users, you only need to provide the organizations (orgs.csv), users (users.csv), and user role and associations to organizations (roles.csv) files. See SDS V2.1 CSV file format.

For SDS v1 CSV: If you're only synchronizing users, you only need to provide the organizations (school.csv) and users (student.csv & teacher.csv) files. See SDS v1 CSV format.

  • Not all fields are supported to bring data into the new SDS experience based on the SDS v1 CSV format. You find them noted in SDS v1 CSV File Format under column Required? as data not supported. Also, Grade values and Course Subject values passed must align to their corresponding List of Values (ENUM) codes.

Can I have extra headers and columns in my CSV files beyond what I intend to sync?

As part of basic validation, when the file names and headers are validated, the files must be formatted correctly.

Each CSV file must contain data for the fields noted 'Yes' under 'Required', the fields noted 'No' are 'Optional' data. The field headers marked as 'Required' must exist in the supplied files or they will not be accepted. The field headers marked as "Optional' aren't required to be present in the files if not passing the corresponding optional data.

See SDS v2.1 CSV file format and SDS v1 CSV format.

If the proper field headers aren't found, the UI won't allow progression through the connect data scenarios based on the CSV format.

If automating the CSV upload, the files will be rejected and not moved over for SDS to pick up for the next run. They'll be removed from the cache after two days.

What happens if I select or upload other files beyond what the CSV format needs?

These files will be ignored and not moved over for SDS to pick up for the next run. They'll be removed from the cache after two days.

In SDS (Classic) I only needed to upload CSV files that have data changes in them. Is this still the same in SDS?

No. You MUST upload all the same files that were provided with prior uploads. If not the data associated with the missing files and their records will be identified 'no longer active' for the current academic year in the next run. (If you need to, you can upload all data files again to process for the next run). The SDS platform architecture supported by the data lake and how longitudinal data is stored is in support of Insights and Analytics capabilities. For more information, see section Determining data awareness and active status in Health and Monitoring.

If we manually remove a user or section, will they reappear when we synchronize again?

Manually updating a class roster will not be overwritten during the following run. SDS makes changes to the roster based on the last run, and not based on manual changes. The only exception is for Reset or Deleting and recreating the Microsoft 365 groups and Teams outbound flow. Aside from those exceptions, here's how SDS treats manual additions and deletions.

Example 1:

  1. A class is synchronized with a teacher and students
  2. The teacher goes to add a co-teacher to the class
  3. The class is later synchronized with no changes to teacher enrollment
  4. (Correct behavior) The co-teacher's membership is unaffected
  5. The class is later synchronized with the co-teacher added
  6. (Correct behavior) the co-teacher's membership is unaffected
  7. The class is later synced w/ the co-teacher removed
  8. (Correct behavior) the co-teacher is removed from the class

Example 2:

  1. A class is synchronized with a teacher and students
  2. The teacher goes to remove a student from the class
  3. The class is later synchronized with no change to the student enrollment
  4. (Correct behavior) The student's non-membership is unaffected
  5. The class is later synchronized with the student removed
  6. (Correct behavior) the student's non-membership is unaffected
  7. The class is later synced w/ the student readded
  8. (Correct behavior) the student is readded to the class

What do Azure AD Connect and SDS do and how can they work together?

Azure Active Directory Connect (Azure AD Connect) syncs on-premises Azure AD users, groups, and objects to Azure AD in Office 365.

SDS synchronizes other student and teacher attributes from the Student Information System (SIS) with existing users already synchronized and created by Azure AD Connect. Adding student and teacher attributes evolves the identity and enables apps to provide richer user experiences based on these distinguishable attributes and education personas. SDS allows you to automatically create class groups and Teams within Teams for Education, OneNote Class Notebooks, and class rostering for third party application integration. Azure AD Connect and SDS will never conflict, as SDS will not synchronize or overwrite any attribute managed by Azure AD Connect. SDS also provides the option to create new users. If you don't want to synchronize and create them with Azure AD Connect from your on-premises AD you can use SDS to synchronize and create them directly from your SIS/SMS data.

Azure AD Connect and SDS

I'm not seeing my data for the next academic year. Is there anything I need to do?

When it's time to transition to the next academic school year you need to start the transition process, including creating a new inbound flow configured for the next academic school year. These steps are similar to when you initially on boarded and created your prior inbound flow.

The process doesn't require you to re-create new Microsoft 365 outbound flows. The existing outbound flow(s) and configurations will persist after the steps have been completed for your new inbound flow for the next academic school year. The next run will start processing the new academic year data beginning with inbound flow and continue processing the data with your existing outbound flows.

For more information, see Academic Year Transition.

In SDS (Classic) I currently use the SDS V1 file format, can I still use it with SDS?

Ability for IT admins to provide SDS v1 CSV file format to upload to the new SDS experience natively. This feature also allows the ability for a tenant that is using SDS just for user management scenarios, and not class groups, you only need to provide the organizations (school.csv) and users (student.csv & teacher.csv) files.

Transitioning to the new SDS experience should only be performed as part of back-to-school or academic year transition events and shouldn't be done within an active school year to avoid potential user disruptions.

  • Not all fields are supported to bring data into the new SDS experience based on the v1 format. You find the unsupported fields noted in SDS v1 CSV File Format under column Required? as data not supported. Also, Grade values and Course Subject values passed must align to their corresponding List of Values (ENUM) codes.
  • Customers using other SDS (Classic) formats need to move to supported formats:
    • SDS UK CSV -> SDS v1 CSV or SDS v2.1 CSV
    • SDS v2 CSV -> SDS v2.1 CSV
    • Clever CSV -> SDS v1 CSV or SDS v2.1 CSV
    • Apple School Manager (ASM) -> SDS v2.1 CSV
      • For Apple School Manager, you only need to provide orgs.csv, with a single organization record, users.csv, and roles.csv required fields and corresponding data.

In SDS (Classic) I currently use the SDS v1 file format, UK file format, and/or Clever file format. I want to move to SDS with SDS v2.1 CSV. What should I do?

Switching between formats should only be performed as part of back-to-school or academic year transition events and shouldn't be done within an active school year to avoid potential user disruptions.

User matching for existing users is done as part of the inbound flow and not immediately written to the Azure AD user object, follow the steps for Data Ingestion with SDS v2.1 CSV

In SDS (Classic) I currently use the SDS V2 file format. I want to move to SDS with SDS V2.1 CSV. What should I do?

To transition from V2 CSV to V2.1 CSV, tenants should provide data in the SDS V2.1 CSV file format

Next, follow the steps for Data Ingestion with SDS v2.1 CSV.

Continue with any top actions as needed based on the scenarios you need SDS to support.

Does SDS support ability to reuse a class between academic sessions or academic years?

  • There's support but there are limitations and caveats.
  • For the subscription there can only be a single active Academic Year configured and associated with the active Inbound flow.
    • Data in the Education Data Lake for Roster data is segmented with at least an Academic Year association based on the Admin configured Academic Year for the active Inbound flow.
      • Academic Session data, for example, Fall Semester / Spring Semester, is linked to the Admin configured Academic Year for the active Inbound flow.
        • Supplied Academic Session data and dates, from the source data, is parented with the defined Academic Year. Manipulation of Academic Session dates outside of the bounds of the active Academic Year could have unsupported downstream effects.
  • If the source system supplies the same sourcedId value for a Class between Academic Sessions and / or Academic Years, SDS reuses the same Azure AD Group, as long as the Classes Azure AD Group hasn't been archived / processed to mark Classes expired with Group / Section cleanup options.
    • When SDS creates and manages the Class / Azure AD Group, based on the supplied Class record from the source, the Azure AD Group attribute is stamped with the SIS ID value that matches the sourcedId value for the corresponding Class record. This allows SDS to link the Class to reuse the same Azure AD Group.
    • As Class Group Memberships / Enrollments are processed during the same SDS Academic Year, changes are processed as data is supplied.
      • If any manual changes are made to memberships that are not present / managed by the source data, those memberships are retained only in the Azure AD Group membership data and not be reflected in the Education Data Lake for Roster data.
    • If the Classes Azure AD groups are archived / process to mark the Classes as expired, with the Group / Section clean up options, SDS recognizes the Class as being new and create a new Class / Azure AD Group in the subsequent run.
  • As Class Group Memberships / Enrollments are processed when transitioning to a new SDS Academic Year, any manual changes made, and weren't recognized in the data coming from the source system data, won't be recognized in the Education Data Lake for Roster data and reflects what is defined in the source system. The manual membership changes are retained in the Azure AD Group membership data.
    • It's always recommended that the source system contains all the membership changes and are processed through SDS rather than relying on performing them manually.
  • If the need is to have the Class Group Memberships / Enrollments managed from the source system, and not persisted from one Academic Year to the next, for a Class / Azure AD Group, the source data must stop providing the enrollments records. The Class Group Memberships / Enrollments must not exist in the supplied source data to be processed before the SDS Inbound flow expires for the active Academic Year.
    • If not the Admin needs to manually remove the records from the Azure AD Group.

I handle the provisioning of users and classes without SDS (manually, through Graph API, or using another solution), and I would like to synchronize my organizational data with Insights for Education Leaders dashboards. What should I do?

Your tenant needs to use SDS to supply the organizational data.

If going with CSV files, you don't need to provide data for classes and enrollments.

When configuring your inbound flow, you need to define how the application should match your users between the user data provided and what is in your Azure AD. See User Identity Rules.

When the user match runs, SDS finds the match and establish it based on the defined rule.

After the run, you'll need to understand your data health and correct any errors and warnings as needed. See Health and Monitoring.

When satisfied with your data health you need to Enable Sync to Insights for Education Leaders.

I use SDS V2.1 CSV format provision part of the users and classes of my tenant through SDS, but I would like to see data in Education Insights experiences for the entire organization. What should I do?

If your tenant has any users in its active directory that are using supported Microsoft apps for providing activity data to power Education Insights but aren't in the data uploaded to SDS, you won't be able to link those users and their activity in the Education Insights experiences.

If your tenant wants to link users to activity in the Education Insights experiences, you need to add the data to be sent through SDS. The tenant doesn't need to add specific class and enrollment information but must expand the data records it sends for organizations (orgs.csv), users (users.csv), and user role and associations to organizations (roles.csv).

The tenant can add the data to an existing extract to be processed with the next upload.

Some of my students and staff are enrolled in two academic units. How do I make sure that their data is reflected in both units, especially for Education Insights experiences?

With SDS V2.1 CSV format and OneRoster API, a user can be associated with more than one organization unit (for example, students who are registered in a special program at another facility). For SDS V2.1 CSV format, the tenant needs to provide two lines in the roles.csv file for that student – one for each organization association.

SDS V2.1 CSV format also supports users that have different roles at different organizations. Provide two lines in the roles.csv for the user, organization, and role combination and mark one of the roles as isPrimary = true.

In SDS (Classic) Settings page there are toggle options to manage Insights data. Where are these managed now?

In SDS (Classic) there were two toggle options available, under the Manage Education Insights section. One for Collect activity data for Insights and the other Allow Advanced Inferences. Both toggles are being moved from managed as part of SDS to the Insights Admin experience. Until the move is complete and the settings are retired from the SDS (Classic) settings page, if you manage the settings in either location, the last change made from either location will be used.

How is the isPrimary value processed with Inbound data?

  • For SDS V2.1 CSV

    • isPrimary indicates if this role is the primary (true) or secondary (false) role for that organization. Default value is false.

    • If passing multiple roles, for the same user and same org, there should be one ‘active’ primary role for the organization.

    • If isPrimary is set to true on multiple records (same user, multiple roles, same organization).

      • Write the data as received but identify the records as warnings.

      • If tenant doesn't like the handling, they need to provide the desired isPrimary role / record from the source data.

    • If multiple roles are passed for the same organization, and isPrimary isn't set to ‘true’ for one of the roles.

      • Write the data as received but identify the records as warnings.

      • If tenant doesn't like the handling, they need to provide the desired isPrimary role / record from the source data.

  • OneRoster v1.1

    • Specification only allows passing one role per user.
    • They can pass in the same role for a user to multiple organizations.
    • Since one role per organization then records are marked as isPrimary ‘True’.

How are user identity match rules determined when multiple roles and/or organizations are associated to a user?

When a user has multiple roles, the following rules are used to determine what staff or student match rules should be used between the user record and the Azure AD user object.

  • If isPrimary is set for all student roles, even if association to a staff role exists, the match is made based on the student role.
  • If isPrimary is set for any staff role, even if association to a student role exists, the match is made based on the staff role.
  • If isPrimary is set for both staff and student role, the match is made based on the staff role.
  • If isPrimary isn't set for any roles, especially with a mix for both staff and student roles, the match is based on the staff role.

If the user is also associated with multiple organizations, the following is also used to determine the value when writing the role to the Azure AD user object.

Why am I seeing errors and/or warnings raised in SDS from data runs?

For more information See Health and Monitoring.

What is the default List of Values supported?

For more information, see Default List of Values.

I was using SDS for Insights, can I still use SDS (Classic) for provisioning?

Existing SDS for Insights / Insights Premium customers, actively using the "Ingest from my SDS (Classic) Sync Profiles" in SDS to link data for insights, have been updated to the Insights for Education Leaders sync.

During your academic year transition, you can still continue to select the option "Ingest from my SDS (Classic) Sync Profiles." See Academic Year Transition.

This data source option won't be available for new inbound flows starting late 2023. We recommend SDS (Classic) customers plan for transition to SDS.

I was using SDS for Insights, but I want to transition from SDS (Classic) to SDS to manage my Microsoft 365 user and groups. Is this possible?

Yes. See Transition from SDS (Classic). The following is a brief overview of the steps:

The Education Data Lake Export is showing 'Critical error' for the status. What should I do?

If the Education Data Lake Export status card shows 'Critical error' for the status, this means there's a problem when processing data. If it's been more than 24 hours and you continue to see the same status, contact support.

I created my Microsoft 365 groups and Teams flow right after my Microsoft 356 user flow, but the status shows 'Skipped' on the groups flow. What does this mean?

If you see a status of Skipped, it means that the flow wasn't included in the current or last Run. The scenario is seen when there's an active run and a request was made to create a new outbound flow. The new outbound flow will be included in the next run.

In SDS (Classic) I could delay student access until a date specified when creating the Sync Profile. I do not see that option in SDS. How do we now delay student access to their classes?

In SDS (Classic) some administrators choose to delay student membership access to Class Teams when first setting up to a future date. School Data Sync and Microsoft Teams for Education has improved this process to empower educators to control the timing when students have access to Class Teams. When using SDS to automate the creation of the Class Team, educators have early access to the Class Teams. When the educator is ready, they can select Activate to allow students and other group members access.

I see a prompt for 'Ready for next year?' but it shows all of my Top Actions have been completed. Can I create a new Microsoft 365 or Insights outbound flow?

Any remaining Top Actions will appear or setup buttons will be enabled after completing the steps to get ready for the next year. Select Start transition to begin when you're ready with your new data for the next Academic year.

After I onboarded to SDS and created a Microsoft 365 user flow, why are my SDS (Classic) sync profiles stopped?

Sync Profiles are stopped to prevent issues where both SDS and SDS (Classic) are managing a tenants Microsoft 365 users, groups and class Teams. Not doing so would result in creating data conflicts or overwriting data. The system detects if there are active SDS (Classic) sync profiles and will stop them from running. Additionally the ability to interact with the SDS (Classic) sync profiles is prevented. The only interactions that are allowed are to select and view and SDS (Classic) sync profile and delete.

I have onboarded SDS and want to delete my SDS (Classic) sync profiles, but I see portions of the interface don't allow for any interactions. How do I delete them?

There are two ways to delete your SDS (Classic) profiles. This process doesn't interact with SDS or the data that SDS is managing in any way.

  • After navigating to SDS (Classic), you can navigate to each sync profile individually by using the navigation on the left. Select a sync profile and a dialog box is presented with an option to delete the sync profile. Select the option and follow the remaining prompts to complete the process. Repeat the step for each of your SDS (Classic) sync profiles.

  • After navigating to SDS (Classic), you can navigate to the Settings page. A dialog box is presented with an option to delete all SDS (Classic) sync profiles. Select the option and follow the remaining prompts to complete the process.

I onboarded SDS and created Microsoft 365 outbound flows, but I'm not happy with the current capabilities supported and I'm not able to interact with SDS (Classic). How can I go back to SDS (Classic) to provision and manage my user and group data?

First, make sure to work with your deployment manager or support if there are any concerns. If you still want to go back, for more information, see Reverting back to SDS (Classic).

How can I use SDS Security Groups to create a Conditional Access Policy to block students from using third party apps?

Important

SDS (Classic) TRANSITION CUSTOMERS: Based on the group splits selected, SDS may not bind to (Classic) Security Groups and associate memberships. You may need to review and apply any configuration settings based on your previous (Classic) Security Groups to the new ones created by SDS. The check and updates will need to happen after the first run has completed for the Security Groups flow as part of your transition steps.

Go the Azure Active Directory Portal Conditional Access Policy Editor and create a conditional access policy:

  • For Assignments->Users and Groups, select the security group you created in Step B4.

    Screenshot showing assignments for Users and Groups.

  • For Assignments->Cloud Apps, include “All cloud apps” and exclude “Microsoft applications”

    First screenshot showing assignments for Cloud Apps to include “All cloud apps”.

    Second screenshot showing assignments for Cloud Apps to exclude “Microsoft applications”.

  • For Access Controls-> Conditions, select “Block Access”

  • Select Enable Policy

  • Select Save

You may customize this conditional access policy to allow the group of students to also use specific third-party applications. Go to Assignments -> Cloud Apps -> Exclude Select the third party applications you want to allow, then save the policy.

How can I use SDS Administrative Units to allow teachers password administration role to reset student passwords?

Important

SDS (Classic) TRANSITION CUSTOMERS: Based on the group splits selected, SDS may not bind to (Classic) Administrative Units and associate memberships. You may need to review and apply any configuration settings based on your previous (Classic) Administrative Units to the new ones created by SDS. The check and updates will need to happen after the first run has completed for the Administrative Units flow as part of your transition steps.

First enable the Administrative Units provisioning with the option selected for Organizations + Role Groups. During the subsequent run, SDS creates Administrative Units by Organization + Role Group combination. The option allows the ability for permitted teachers to perform delegated IT administration for students of school administrative unit.

After the subsequent sync, an option is added for every teacher account within SDS people pages. Go to SDS | People and search for the desired staff member.

Screenshot showing password administrative role disabled.

Turn on the option for Password administration role. This allows teachers the ability to reset student passwords that are associated with their same organization.

Screenshot showing password administrative role enabled.

When the Password administrative role has been assigned note that there could be a delay after enabling and when the change completes updating policy configurations.

Refer to the "How to reset a Students Password" page for instructions for staff after they have access to the limited Admin panel.