Sync health and monitoring

Article
06/14/2024

Home dashboard

During data processing, School Data Sync (SDS) validates data by bringing in good data to the cache data and flagging bad data. At the end of each sync run, statistics are generated to assist with health and monitoring.

To determine the health of the data, is passed through data matching and validation rules to help safeguard good required and optional data only going into the cache. Data that doesn't pass validation is identified as errors or warnings and isn't sent to the cache.

If there were no errors or warnings found, the result of the run is Completed. The home dashboard will state, "No Data Errors or Warnings Found" and "We did not encounter any data errors or warnings during your last run. Keep up the great work!"

If there are errors and warnings found, the result of the run is Completed with errors.

Error is raised when a value on a record for required data didn't pass a data matching and or validation rule; the record was removed and not sent to the cache.

If there are only warnings found, the result of the run is Completed with warnings.

Warning is raised when a value on a record for optional data didn't pass a data matching and or validation rule; the value was removed but the record was sent to the cache.

The Home Dashboard informs that ‘We found some issues with your data’ and encourage you to Investigate Sync health. For more information on the results of the Sync run, select Investigate sync health.

Sync health

Sync health is a tool to help you better understand the health of your data that is synced and to assist with data investigations.

Changes to the data occur based on data changes from the most recent sync run.

A sync run is the sequencing of both connect and manage data configurations.

Sync health also provides a historical representation of the last 14 runs.

Run start: Time stamp when overall run started
Run end: Time stamp when overall run ended
Run status: Overall status across all data flows
- Running: Actively executing
- Completed: Completed without any errors or warnings
- Completed with Errors: Completed but errors were found
- Completed with Warnings: Completed but only warnings were found
- Failed: Run canceled by system or customer

To investigate the flagged data, you can select Download report to retrieve a copy of the report.

The report is formatted as follows:

Column Name	Description	Example
Rule	Representation of Validation Rule	RequiredDataViolated
ExternalIdentifier	External / Sourced ID of the related entity from the source system	7B2C8A0B33-F7E5-460B-993A-7257165430F7
Severity	Severity representation of the flagged data	Validation Error, Validation Warning
EntityCode	Representation of area related to the error	Person, Organization, Enrollment
FriendlyMessage	Based on Error and Warning Messages	A required value for {record} is missing in field name: {field} from source [{fileName}/{apiEndPoint}] to create the record.
FlowName	Name of the Inbound or Outbound flow	OneRoster API Inbound
SourceSystemName	Name of the Source where the record came from	Contoso SIS
Time	Data / Time when the record was identified during processing, UTC.	2023-08-21T02:53:00Z

For more information on data matching and validation rules, see Validation Rules and Descriptions.
For more information on the default List of Values supported, see Default List of Values.
familyName, givenName, and email are required for users that have contact / guardian roles.
Expect phone and sms to be in E.164 and + must be included. (Example: +1234567890)

Tip

To manually upload changes after your first run to update your source data see Manual CSV Upload with School Data Sync.

Warning

The maximum number of uploads with a corresponding run is six times per calendar day. After that, if you upload, it will only be run up to two more times based on SDS automated run cadence of every 12 hours.

Prioritizing errors for remediation

When troubleshooting errors, we recommend you prioritize validation errors with the most instances for the same entity code ahead of troubleshooting any other errors.

Validation errors may result in numerous subsequent errors for each instance within the same data run. Often, remediating these validation errors first remediate many roster-related errors that are found during validation.

Sync health view details

To see statistics from the run, you can select View details to open a fly-out to see Run detail. The statistics from the run are shown on the Overview tab.

To better understand what stage the Errors and Warnings may have been raised from, select Stages tab. Information is broken into the following Stages, depending on what Manage data configurations have been enabled.

Institution data
Microsoft 365 Users
Microsoft 365 Groups
Microsoft IT Groups
Latest status
- Completed: Completed without any errors or warnings
- Completed with errors: Completed but errors were found
- Completed with warnings: Completed but only warnings were found
- Failed: Run canceled by system or customer

Note

For more information, see Institutional and Run Statistics on statistics found on Run details.

How SDS determines data being present and sets active status

A user's association to a organization is based on their role, tied to an organization.

A user's association to a class is based on their role tied to an enrollment.

Based on the connected data, these rules are used to determine awareness of the record and its session status in the data store.

The data reflects when a new record is presented for the first time.
- SDS sets the first seen date (time) and last modified date (time) as current.
The data reflects when the same record is present in the subsequent run.
- SDS preserves the first seen date (time) value, set the last modified date (time) to current.
The data reflects when the same record isn't present in a subsequent run.
- SDS preserves the first seen date (time) and the last modified date (time) values.
  - Exceptions happen when organizations, people (users) and session records persist over time and aren't inactivated.
  - There are rolling updates for "inactivated". For example, if a user record isn't present, the system preserves the existing first seen date (time) and last modified date (time) values.