What's New With School Data Sync

  • Article

Welcome to the School Data Sync (SDS) “What’s New” page. We're continuously improving the service with new features and updates that make SDS easy to set up and use. Bookmark this page so you can stay on top of the latest improvements.

Note

We're excited to announce that the new School Data Sync (SDS) experience is Generally Available.

Important

Customers preparing for upcoming Southern Hemisphere Back to School (January 2024) can choose to transition to School Data Sync or continue with SDS (Classic). See SDS Roadmap for features or scenarios pending support.

Warning

Environments, with enrollments for more than 50k users, where provisioning tasks (writing users, groups, and memberships to Microsoft Entra ID) may take more than 24 hours to process is currently not recommended. Monitor SDS Roadmap and "What's New" page for Large Organization Support for when this is lifted.

Warning

Environments that need to directly connect data from multiple SISs into a single subscription is currently not supported. If this configuration is needed, before support is added, then pre-processing, outside of SDS, will need to occur to combine the data before submitting the data through one of the supported CSV formats. Username values in the CSV will need to include the fully qualified domain value, e.g. username@domain, for user targeted-domain identity mapping. Monitor SDS Roadmap and "What's New" page for Single Subscription - Multi-Academy Trust Support for when this is lifted.

April 22, 2024

  • Automate SDS CSV Upload with Power Automate: Ability for IT admins to automate SDS CSV uploads from extracts from their SIS / SMS. Instead of manually uploading data changes from source data a Power Automate template is available to assist with uploading those data changes to SDS.

April 4, 2024

  • Graph API Sync Management: SDS expanded the new industryData Graph API to support managed data use scenarios:

    • Manage Users
    • Manage Class Groups
    • Manage Administrative Units
    • Manage Security Groups

March 29, 2024

  • Additional Entra ID Extension Property Sync: Ability to support passing:

    • Class Code to Group Object SectionNumber education extension property.
    • User Number to User Object StudentNumber education extension property.

March 12, 2024

  • Organization Filters for Managed Data to Microsoft 365: SDS is designed to allow IT admins to ingest all their Student Information System (SIS) / Student Management System (SMS) data within the supported schema, and then pick and choose what they want to do with it. They could choose to enable any or all the available Manage data provisioning types (manage users, class groups, security groups, administrative units).
    • Now with Organization Filters for Managed Data to provision data to Microsoft 365, you can keep the default to Include all organizations or you may select Filter by organizations to only provision a subset of the users or classes ingested from the connected data.
    • SDS adds the ability for you to select which organizations you provision forward into Microsoft 365, while ignoring other users and classes within other organizations. You can also create multiple Managed data configurations to apply different settings to different organizations. One example is a separate managed data configuration for Elementary vs Middle / High School.

December 15, 2023

  • Additional Entra ID Extension Property Sync: Ability to support passing:

    • Organization SourcedId to Group Object SchoolId education extension property.

November 15, 2023

SDS has various enhancements and improvements over SDS (Classic):

  • Decoupled data ingestion: Data ingestion is decoupled from all Microsoft 365 directory provisioning tasks. The updated architecture provides more flexibility and control over using SDS with synced SIS/SMS data for various small, targeted purposes instead of forcing admins to deploy SDS and several of its capabilities in an all-or-nothing approach.
  • Faster syncs and fewer errors: Mapping users with Microsoft Entra ID now occurs during data ingestion against a locally cached copy of Microsoft Entra ID, allowing for faster processing times, more efficient filtering out of unmatched users for provisioning enrollment updates, and dramatically reducing the volume of errors generated. For more information, see User identity rules.
  • Scale: Extra-large organizations no longer need to split up single data sources into separate CSV extracts and configure multiple Sync Profiles. Now, SDS uses a single connect data configuration to bring in SIS/SMS data from a single source.
  • Health, monitoring, statistics, and reporting: Improved visibility into the health of your data with Run health and Flow health, including institution statistics, for awareness and calls to action.
  • Modern UI: A new user interface for managing educational institution data and is aligned with the other Microsoft 365 admin center experiences.

SDS (Classic) has an invitation message and link on the homepage, which allows you to navigate to the new experience. This milestone also completes the bridge, which first started with SDS for Insights, which is now rebrand as the new School Data Sync.

The SDS team will continue to add new features over time. Stay tuned for more updates!

General Availability Improvements

Based on feedback from public preview SDS has added improvements and updates supporting general availability.

  • Home Dashboard - A simplified and focused experience to improve readability and reduce scrolling to see important information and calls to action.

    • Consolidated Run health and Academic Year into single card
    • Consolidated Matched users insights and Microsoft 365 User insights cards
    • Moved Update Source Data button from Home Dashboard to Sync | Configuration | Connect data tab
    • Reduced Microsoft 365 User Management, Group and Class Management, and IT Group Management top action cards into single Manage data action card
    • Dropped Enable Sync with Insights Premium and Enable Education Data Lake Export top action cards and moved enablement and management to Sync | Configuration | Insights & analytics tab
    • Added polling to automatically refresh page every 2 minutes

  • Manage Data - Improved and simplified the ability to enable and manage multiple Microsoft 365 provisioning options. Manage data wizard reduces four wizards into a single, simplified experience reducing onboarding and edit configuration timing.

  • Sync | Configuration - Single location to view and manage configurations enabled by a customer for Connect data, Manage data to Microsoft 365, and Insights & analytics.

    • Connect data contains an overview of Academic year configuration and connected SIS / SMS configuration and supports Edit and Upload CSV data scenarios
    • Manage data contains overview of Microsoft 365 provisioning type configuration selections and supports Edit, Reset, and Delete scenarios
    • Insights & analytics supports enablement and management of Sync with Insights for Education Leaders and Education Data Lake Export
    • Renames Data left navigation to Sync
    • Data | Health is now Sync | Health
    • Retires Data | Flow Health
    • Moves custom List of Values management from Data left navigation to Settings

  • CSV Data Validation during UI Upload - New ability when uploading CSV files, SDS runs an initial data validation from the files uploaded through the UI.

    • The uploaded data passes through SDS data matching and validation rules, identifying the good required and provided optional data, and flagging invalid data.
    • Not all of the validation rules are included, only the ones most encountered.
    • This experience isn't blocking during the onboarding or csv upload experiences.
    • If data is flagged, you have two options: 1) either exit and return later after fixing the data or 2) proceed to the next step to continue setting up your data connection and later supplying the corrected data with subsequent uploads.

Available since Public Preview

SDS contains many of the same top features as SDS (Classic), such as the ability to synchronize users and create class groups and teams.

  • SDS v2.1 CSV Format: SDS has evolved the SDS CSV file schema for both K12 and Higher Education, and includes new data elements, like broader user roles, user demographics and user program flags. The new data elements are used for both SDS provisioning, insights, and analytics scenarios. The specification is inspired by OneRoster standards and includes several innovative features. For more information on the full format, see SDS v2.1 CSV format.

    • Only requires 3 CSV files for user management only scenarios: orgs.csv, users.csv, and roles.csv
    • Only requires 5 CSV files for user and class management scenarios: orgs.csv, users.csv, roles.csv, classes.csv, and enrollments.csv
    • 29 Organization / Primary User roles for K12 & Higher Education (including students and teachers)
    • 20 Organization types for K12 & Higher Education with support for Organizational Hierarchy
    • Support for user to multiple organization associations
    • Grade Level association for Educators and Students
    • 27 Class Enrollment roles for K12 & Higher Education, including scenarios where users may be an Educator (owner) in one class and a student (member) in another.

  • SDS v1 CSV Format: IT Admins can provide SDS (Classic) v1 CSV format with SDS to synchronize user and group data elements.

  • More data from OneRoster API: IT Admins can synchronize data elements previously only supported in CSV files like parents and guardians, demographics, and user flags data (support is based on providers that have completed changes to support providing the extra data). For more information on SIS/SMS OneRoster API and provider data support, see OneRoster Provider Overview.

  • Custom List of Values: SDS has made some more changes to its schema, but it's likely that different institutions have different enumerations. SDS now allows admins to create custom reference definitions for key enumerations like organization Grade Level, Academic Subjects, and others. This allows for simplified exports from SIS / SMS with data elements. This will also ensure Insights and Analytics solutions enabled reflect the naming conventions for these attributes present within the institution, instead of being forced to use the default values provided within the schema. For more information on expanding enumerations, see managing List of Values.

  • Manage Microsoft 365 Users: A core feature is the ability to map SIS/SMS users with your Microsoft Entra users. If using Microsoft Entra Connect, or another provisioning process for end users, the default setting is just to match existing users between the systems based on your configuration for User Identity Rules. SDS can also be configured to create and license staff and student users for those not automating users to their cloud Microsoft Entra ID. SDS also includes configuration of default passwords that follow Microsoft Entra Password Protection that meets scoring calculations to be accepted. For more information as part of managing users with Microsoft Entra ID, see Manage data.

  • Manage Microsoft 365 Groups and Class Teams: Another core feature is the ability to create groups and class Team and sync enrollments from SIS/SMS data. An improvement for Class Teams and delayed student access, now the educator can prepare the class early, and once ready, they can Activate the class to let students gain access to the class. Administrators can choose not to automatically create class teams, for educator-initiated scenarios, by having them be based off a Microsoft Entra Class Group that the educator is an owner of. Additionally, admins can define the group Owner and Member roles beyond just the teacher as Owner and student as Member, based on the 27 class enrollment roles now supporting K12 & Higher Education. For more information, see Manage data.

  • Security Group Sync: A key feature is the ability to create security groups and sync security group memberships. Security group sync helps IT admins setup and configures group-based management in Microsoft Entra ID, Teams, and other core Microsoft 365 services, which utilize security groups for management of settings and policies. Security groups can be used for Device Management (Intune for Education, mobile device management), Application Management (conditional access policies, Teams policies) and Identity Management (Group Based Licensing, Self-Service Password Reset policies). For more information, see Manage data.

  • Administrative Unit Sync: Another key feature is the ability to create Administrative Units (AUs) and manage their memberships. AUs define an organization's structure for delegated management. For example, each school in a district can be an AU. AUs now support Teams, Microsoft Groups, and SharePoint sites, along with users and Microsoft Entra groups. AUs provide the ability to scope delegated admins to just a subset of the broader directory for local help-desk roles, support teacher password reset, support Information Barrier policy scenarios, and Dynamic Groups. For more information, see Manage data.

  • Education Data Lake Export: Education Data Lake Export allows Administrators to securely access the EDU Data Lake from their Microsoft subscription boundary by generating a daily snapshot of the EDU Data Lake data and copying to their Azure subscription and Data Lake. With Education Insights data in their Azure Data Lake, organizations can then combine it with other data sources such as Learning Management Systems or Assessment data. They can then use Microsoft Fabric, Microsoft Synapse, Power BI, Azure Machine Learning, and / or Artificial Intelligence to create their own customized analytics and reports. Learn more at enabling Education Data Lake Export.

    • To receive the copy from the export, you need an Azure subscription. If you don't have an Azure subscription, you can set up a free subscription here, or check the the current list of Azure offers.
    • You need the role assignment of "Owner" on the preferred subscription you're using and the subscription is selected as default.