Sync health and monitoring

Home dashboard

During data processing, School Data Sync (SDS) validates data by bringing in good data to the data lake and flagging bad data. At the end of each sync run, statistics are generated to assist with health and monitoring.

To determine the health of the data, is passed through data matching and validation rules to help safeguard good required and optional data only going into the data lake. Data that doesn't pass validation is identified as errors or warnings and isn't sent to the data lake.

If there were no errors or warnings found, the result of the run is Completed. The home dashboard will state, "No Data Errors or Warnings Found" and "We did not encounter any data errors or warnings during your last run. Keep up the great work!"

If there are errors and warnings found, the result of the run is Completed with errors.

• Error is raised when a value on a record for required data didn't pass a data matching and or validation rule; the record was removed and not sent to the data lake.

If there are only warnings found, the result of the run is Completed with warnings.

• Warning is raised when a value on a record for optional data didn't pass a data matching and or validation rule; the value was removed but the record was sent to the data lake.

The Home Dashboard informs that ‘We found some issues with your data’ and encourage you to Investigate Sync health. For more information on the results of the Sync run, select Investigate sync health.

Sync health

Sync health is a tool to help you better understand the health of your data that is synced and to assist with data investigations.

Changes to the data occur based on data changes from the most recent sync run.

A sync run is the sequencing of both connect and manage data configurations.

Sync health also provides a historical representation of the last 14 runs.

• Run start: Time stamp when overall run started
• Run end: Time stamp when overall run ended
• Run status: Overall status across all data flows
  • Running: Actively executing
  • Completed: Completed without any errors or warnings
  • Completed with Errors: Completed but errors were found
  • Completed with Warnings: Completed but only warnings were found
  • Failed: Run canceled by system or customer

To investigate the flagged data, you can select Download report to retrieve a copy of the report.

The report is formatted as follows:

Column Name	Description	Example
Rule	Representation of Validation Rule	RequiredDataViolated
ExternalIdentifier	External / Sourced ID of the related entity from the source system	7B2C8A0B33-F7E5-460B-993A-7257165430F7
Severity	Severity representation of the flagged data	Validation Error, Validation Warning
EntityCode	Representation of area related to the error	Person, Organization, Enrollment
FriendlyMessage	Based on Error and Warning Messages	A required value for {record} is missing in field name: {field} from source [{fileName}/{apiEndPoint}] to create the record.
FlowName	Name of the Inbound or Outbound flow	OneRoster API Inbound
SourceSystemName	Name of the Source where the record came from	Contoso SIS
Year	Academic Year	2023
Time	Data / Time when the record was identified during processing, UTC.	2023-08-21T02:53:00Z

• For more information on data matching and validation rules, see Validation Rules and Descriptions.
• For more information on the default List of Values supported, see Default List of Values.
• familyName, givenName, and email are required for users that have contact / guardian roles.
• Expect phone and sms to be in E.164 and + must be included. (Example: +1234567890)

Tip

To manually upload changes after your first run to update your source data see Update Source Data with SDS v2.1 CSV.

Warning

The maximum number of uploads with a corresponding run is six times per calendar day. After that, if you upload, it will only be run up to two more times based on SDS automated run cadence of every 12 hours.

Prioritizing errors for remediation

When troubleshooting errors, we recommend you prioritize validation errors with the most instances for the same entity code ahead of troubleshooting any other errors.

Validation errors may result in numerous subsequent errors for each instance within the same data run. Often, remediating these validation errors first remediate many roster-related errors that are found during validation.

Sync health view details

To see statistics from the run, you can select View details to open a fly-out to see Run detail. The statistics from the run are shown on the Overview tab.

To better understand what stage the Errors and Warnings may have been raised from, select Stages tab. Information is broken into the following Stages, depending on what Manage data configurations have been enabled.

• Institution data

• Microsoft 365 Users

• Microsoft 365 Groups

• Microsoft IT Groups

• Latest status

  • Completed: Completed without any errors or warnings
  • Completed with errors: Completed but errors were found
  • Completed with warnings: Completed but only warnings were found
  • Failed: Run canceled by system or customer

Note

For more information, see Institutional and Run Statistics on statistics found on Run details.

How SDS determines data being present and sets active status

Note

For more information about Academic Year configuration, see Academic Year handling

A user's association to a session (for example, Academic Year) is based on their role, tied to an organization.

A user's association to a class is based on their role tied to an enrollment, which also includes a link to a session.

Based on the connected data, these rules are used to determine awareness of the record and its session status in the data store.

• The data reflects when a new record is presented for the first time.
  • SDS sets the first seen date (time) and last modified date (time) as current, and if appropriate, it marks record as "is active in session" to be true.
• The data reflects when the same record is present in the subsequent run.
  • SDS preserves the first seen date (time) value, set the last modified date (time) to current, and leave "is active in session" to be true.
• The data reflects when the same record isn't present in a subsequent run.
  • SDS preserves the first seen date (time) and the last modified date (time) values and if appropriate, mark the record as "is active in session" to false.
    • Exceptions happen when organizations, people (users) and session records persist over time and aren't inactivated.
    • There are rolling updates for "inactivated". For example, if a user record isn't present, the system preserves the existing first seen date (time) and last modified date (time) values. The system sets "is active in session" to false for the users' org/role and enrollment records.

Note

For more information about data handling, see Validate, Store and Data Health on SDS Overview.