Internal system error using Purview data scan

Question

When using data scan in Azure Purview I got an internal system error, with SQL Server and Fabric. It was working before. Stopped working on Nov 21.

Answer 1

Hi,

All working now! The problem was solved without any change from our side. I believe it was a problem in Azure.

BR,

Paulo

Answer 2

Hi @Paulo Valente

If your Purview data scans suddenly started failing with “Internal system error” around Nov 21, something likely changed in your setup—maybe an update, a permission tweak, or a network rule. These errors usually pop up when Purview can’t connect properly or doesn’t have the right access.

Start with the Basics

• Check what changed recently Did you update your Integration Runtime (IR), rotate credentials, or modify firewall rules? Even small changes can break scans.
• Look at your IR health
  1. If you’re using Self-hosted IR, make sure it’s running and updated to the latest version. Also check memory—big scans can fail if the machine is low on resources.
  2. For Azure IR, confirm the region and whether Managed VNet is enabled (needed for private endpoints).

Permissions Matter

• For Microsoft Fabric, Purview’s managed identity needs Contributor access on the Fabric workspace. Missing this is a common cause of internal errors.
• For SQL Server or Azure SQL, make sure Purview’s identity has the right read permissions and firewall access. If you use MSI, double-check RBAC roles like Storage Blob Data Reader.

Network & Connectivity

Run the Test Connection in Purview. If it passes, great—but remember, scans might still fail if private endpoints aren’t configured correctly. Enable Managed VNet and Private Link if needed.

Fabric-Specific Checks

• If you’re scanning Lakehouse tables, confirm they’re on Delta Lake 2.4 with Spark 3.4. Older versions can trigger errors.
• When setting up table lookup rules, make sure reference assets belong to the same governance domain.

Narrow It Down

Try scanning a smaller scope first—just one schema or table. If that works, it expands gradually. This helps isolate problem areas.

Check the Logs

Go to Purview Studio → Insights → Scan History. Look for patterns:

• Did failures spike after Nov 21?
• Any extra error codes besides “internal system error”?
• Which scan rule set was used?

Checklist

• Update and restart your IR (Azure or SHIR).
• Re-check permissions for Purview MSI on Fabric and SQL.
• Enable Managed VNet if you’re using private endpoints.
• Reduce scan scope and retry.
• Validate Fabric Lakehouse versions and reference assets.
• If it still fails, export logs and raise a support ticket with Microsoft.

Preventing Future Issues

• Keep IR updated and sized properly.
• Maintain a simple change log for credentials and permissions.
• Use Purview Insights regularly to catch anomalies early.

Answer 3

If you are encountering an internal system error while using the data scan in Azure Purview, especially after it was previously functioning, there are several troubleshooting steps you can take:

1. Check for Credential Changes: Ensure that the credentials to your resource have not changed or been rotated. If they have, you will need to update your scan with the correct credentials.
2. Azure Policy Restrictions: Verify if there is an Azure Policy that is preventing updates to your resources, which might affect the scan functionality.
3. Self-hosted Integration Runtime: If you are using a self-hosted integration runtime, confirm that it is up to date and properly connected to your network.
4. Network Settings: If you are using private endpoints or virtual networks, check your network settings, particularly the Network Security Group (NSG) rules, to ensure they allow the necessary traffic.
5. Version Compatibility: Ensure that you are using the correct version of the self-hosted integration runtime. If you are running an older version, consider upgrading to the latest version.

If the issue persists after checking these areas, you may need to contact Microsoft support for further assistance.

---

References:

• Troubleshoot your scans and connections in the Microsoft Purview Data Map
• Common errors and messages