Edit

Share via

Delete completed system jobs and process log to comply with retention policy

Managing completed system jobs and the process log is crucial for maintaining data privacy compliance and ensuring optimal system performance in Dataverse.

The System Job AsyncOperation table stores system jobs. The actual database table name is AsyncOperationBase. System jobs represent asynchronous operations, including registered workflows, plug-ins, and background operations such as bulk deletion, bulk import, and roll-up operations.

When an asynchronous workflow starts in your Dataverse organization, a record is created in the AsyncOperation table to track its progress. More records are created in the Process Log (WorkflowLog) table to keep logs of the workflow execution. The actual database table name for these logs is WorkflowLogBase. Business process flows also store logs for stage transitions and actions in the WorkflowLog table.

Every time a system job completes, its outcome is logged as a system job in a AsyncOperation record. Completed system jobs are records that capture the results of these various automated operations in Dataverse. Over time, these records accumulate and, if unmanaged, can impact system storage and performance.

If your organization relies heavily on workflows or business process flows, the associated tables can grow significantly over time, potentially leading to performance degradation and excessive database storage consumption.

Types of completed system jobs and their statuses

System jobs

  • Delete plug-in trace log records
  • Delete completed system jobs
  • Delete completed process sessions for Sync workflows

Completed system jobs are categorized into three main types based on their completion status:

  • Succeeded: Jobs that have been completed successfully without errors.
  • Failed: Jobs that have encountered errors and haven't completed as intended.
  • Canceled: Jobs that were stopped before completion, either by user intervention or by system constraints.

Deletion service to automatically delete completed system jobs

By default, the system provides a deletion service to automatically delete system jobs that completed successfully, failed, or were canceled. The default is set in the OrganizationSettingsEditor tool and can be changed using the tool or from the Environment Settings page.

Update retention policy using OrganizationSettingsEditor

To check this setting:

  1. Install the OrganizationSettingsEditor tool.
  2. Add and edit the EnableSystemJobCleanup.
  3. Set the EnableSystemJobCleanup to 'true' (if this isn't already set to true).

Set retention policy

There are three parameters to allow different retention periods to be set.

Update retention policy using Environment Settings page (Preview)

[This section is prerelease documentation and is subject to change.]

Important

  • This is a preview feature.
  • Preview features aren’t meant for production use and might have restricted functionality. These features are subject to supplemental terms of use, and are available before an official release so that customers can get early access and provide feedback.
  1. Sign in to the Power Platform admin center.
  2. Select Manage in the navigation pane.
  3. In the Manage pane, select Environments. Then select an environment.
  4. Select Settings.
  5. Select Audit and logs > Set retention policy for system job deletions (Preview).
  6. Select the checkbox for Allow deletion service to delete system jobs.
  7. Select or enter a period in days for each type of system jobs, including Successful jobs, Failed jobs, and Cancelled jobs.
  8. Select Save.

Note

This deletion service deletes the system jobs. Deleted system jobs no longer show up in the All System jobs or the All Bulk Deletion System jobs lists. There's a recurring job called Delete completed system jobs. This job is suspended when the deletion service is on.

Bulk deletion jobs

Important

This job is in a suspended state when the deletion service is used.

All environments are configured with an out-of-the-box bulk deletion job to delete successfully completed workflow system jobs older than 30 days. Customers can configure other bulk deletion jobs to delete AsyncOperation records. We recommend setting up a job to delete completed system jobs—regardless of type or outcome—that are older than 30 days. This job helps prevent the AsyncOperation table from accumulating excess records.

All environments include a built-in bulk deletion job that removes successfully completed workflow system jobs older than 30 days. Customers can create more bulk deletion jobs to delete AsyncOperation records. We recommend configuring a job that deletes any completed system job—regardless of type or result—older than 30 days to prevent unnecessary buildup in the AsyncOperation table.

You can use the bulk deletion system jobs to delete unneeded records from both the AsyncOperation and WorkflowLog tables.

View the bulk deletion system jobs

The bulk record deletion jobs show the history of system jobs that were run and the results.

Note

This deletion service deletes the system jobs and deleted bulk deletion system jobs no longer show up in the All System jobs or the All Bulk Deletion System jobs lists.

  1. Sign in to the Power Platform admin center.
  2. In the navigation pane, select Manage.
  3. In the Manage pane, select Environments.
  4. On the Environments page, select an environment.
  5. In the command bar, select Settings.
  6. Expand Data management, then select Bulk deletion.

From the Bulk Deletion grid, use the view selector to show the completed, in-progress, pending, and recurring bulk deletion system jobs.

The Recurring Bulk Deletion System Jobs view shows the out-of-the-box job definitions for the recurring bulk deletion system jobs.

When you open one of these system job records, you can see the query uses to identify which records to delete and the schedule the job runs on. For these out-of-the-box system jobs, you can't modify the query used by the system job, but you can modify the schedule the job runs on.

If you switch the view to display jobs that are scheduled, in progress, or completed, you can cancel, resume, or postpone the job using the options available in the Actions menu when viewing the record.

Create your own bulk deletion jobs

If the Deletion service or the out-of-the-box system bulk deletion jobs don't meet your organization's needs, you can create your own bulk deletion job.

First, turn off the Deletion Service by completing the following steps.

  1. Add and edit the EnableSystemJobCleanup.
  2. Set the EnableSystemJobCleanup to 'false'.

From the Bulk Deletion grid, select New on the command bar to define a query for the records you want to delete. Use the preview feature to review the records the query returns and verify that it captures the intended data set.

To clean up workflow records from the AsyncOperation table, in the Look for entity dropdown, select the System Jobs table; in the Use saved view dropdown, select [new] to create your own query. You can only delete completed workflows. Workflows waiting to run or currently in progress can't (and shouldn't be) deleted by your system job.

Add the following conditions to your query:

  • System Job Type Equals Workflow (Optional)- target workflow records.
  • Status Equals Completed - only completed workflows are valid to complete.
  • [Optional] Filter on the Status Reason Equals Succeeded/Failed/Canceled - valid for completed status.
  • [Optional] Filter on Completed On field to only delete older workflows.
  • [Optional] Any more filters you want to apply.

Next you set the frequency of the bulk deletion job to run at. You can create a one-time bulk deletion job or define a schedule to allow your job to run at set intervals.

Priority-based, bulk delete jobs for asyncoperation

To unblock customers that have significant asyncoperation size, we introduced a priority-based bulk delete feature for asyncoperation cleanup. This feature is available by default for all organizations. Here are some key points to note:

  • Bulk delete priority applies only to newly created, nonrecurring bulk delete jobs for the AsyncOperation entity.
  • To maximize this feature's effectiveness and optimize job performance, create jobs with varying filter conditions—such as different date ranges, system job types, status codes, or other criteria.
  • Organizations can define up to five priority jobs. Any more jobs are created with default priority.
  • Regardless of the number of jobs for other entities in the queue, one AsyncOperation job can always be created with priority.
  • There's no change with respect to creation steps for bulk, delete jobs.

Plug-in trace logs

Plug-in can be used for asynchronous operations. An example is to create a task upon account creation. When you're testing the plug-in, you can set PluginTraceLogSetting to all or exceptions. You can view the plug-in trace logs in the Power Platform Environment Settings app.

To automate the cleanup of Plug-in trace logs, Dataverse provides a recurring job that deletes records older than one day.

  • This job runs every day, ensuring that the system doesn't retain unnecessary Plug-in trace logs indefinitely.
  • The deletion process helps free up storage space and keeps the system performing efficiently.

Although you can delete Plug-in trace logs daily, it's recommended not to allow this option in your production environment, as it may affect system performance and storage. Instead, ensure that the PluginTraceLogSetting is set to off for the production environment. This best practice ensures that the plug-in trace log table doesn't grow uncontrollably and cause the database to reach its capacity limit.

Process sessions for sync workflows

Process session (dialog session) is created when a user runs or executes a dialog process. It contains the logs about the run, such as the user who ran it, time the process started, and the actions performed.

To automate the cleanup of process sessions, Dataverse provides a recurring job that deletes records older than one day.

  • This job runs every day, ensuring that the system doesn't retain unnecessary process sessions indefinitely.
  • The deletion process helps free up storage space and keeps the system performing efficiently.

Best practices for designing workflows

After removing unnecessary records from your workflow tables, consider refining your workflow design to help slow future table growth

For asynchronous workflows, we recommend enabling Automatically delete completed workflow jobs (to save disk space) in the workflow editor. This setting allows the system to automatically delete logs for successful completed jobs, conserving storage space. Logs from failed workflow executions are always retained for troubleshooting purpose.

Set Workflow Job Retention to automatically delete completed workflow jobs.

For synchronous workflows, we recommend enabling Keep logs for workflow jobs that encountered errors in the workflow editor. This setting ensures that logs from failed executions are retained for troubleshooting, while logs from successful run are automatically deleted to save space.

Set Workflow Log Retention to keep logs for workflow jobs that encountered errors.

AsyncOperation file capacity usage

Historically, the entire async operation context used to be serialized directly in the AsyncOperation table Data property, leading to quick growth in size if there are asynchronous plug-in registrations on large records (such as attachment). Since early 2021, the data portion of async operations is moved to file store. As a result, async operation data size is partially counted in database capacity and partially in file capacity. This shift helps reduce the costs—since file capacity is charged at a lower rate than database capacity—and improves overall performance by enabling more efficient queries against the AsyncOperation table.

Note

Files related to AsyncOperation records can't be deleted directly. To reclaim file capacity, review the retention and cleanup policies for AsyncOperation records described in this article. When AsyncOperation records are deleted, their corresponding files get deleted as well.

For more information on Dataverse storage model and reporting, see New Microsoft Dataverse storage capacity.

Related content

Reduce database storage
Remove a large amount of specific, targeted data with bulk deletion
Microsoft Dataverse real-time workflows
Classic Dataverse background workflows
Automatically delete completed background workflow jobs
Asynchronous service

  • Last updated on