Asynchronous service
The asynchronous service executes long-running operations independent of the main Microsoft Dataverse core operation. Executing long-running operations this way results in improved overall system performance and improved scalability. The asynchronous service is a managed first-in, first-out (FIFO) queue for the execution of asynchronous registered plug-ins, workflows, and operations such as bulk mail, bulk import, and campaign activity propagation. These operations are registered with the asynchronous service and executed periodically when the service processes its queue.
After an event occurs and any synchronous extensions are processed, Dataverse serializes the context for any asynchronous extensions and saves it in the System Job (AsyncOperation) table. The system job defines and tracks the execution of the asynchronous operation. As resources become available, Dataverse processes system jobs and executes the operations they define. Dataverse processes any data operations defined in the extension again in the event execution pipeline, but this time as a synchronous operation.
System jobs are evaluated as a queue using the CreatedOn date. If there are no conditions to defer execution, they're executed as soon as resources are available. Execution isn't always performed in the order set by the CreatedOn
date because different types of operations require different resources.
A system job can be dependent on another system job so that it will begin only after the other system job completes. The DependencyToken column value establishes this dependency when a system job is created. If the DependencyToken
value is null, the system job has no dependencies. Dependent system jobs have the same DependencyToken
value and are executed in the order they were created. If a system job is postponed, all subsequent dependent system jobs continue to wait until the postponed system job executes.
Note
This dependency system cannot be used by plug-ins registered to run asynchronously because the system jobs for them are created by the system.
You can perform the following operations to manage system jobs using the AsyncOperation Table.
- Retrieve system jobs
- Delete system jobs
- Manage system job states
- Postpone system jobs
Note
Creating system jobs with code is not supported. Although the AsyncOperation
table supports several writeable columns and create operations, only the following columns are supported for update:
You can view system jobs in the application by navigating to Settings > System > System Jobs and you can also search them using Advanced find in model-driven apps.
Using code, you can retrieve system jobs like any other table. The following table lists selected columns that are important in understanding system jobs:
Column | Description |
---|---|
AsyncOperationId |
Unique identifier of the system job. |
CompletedOn |
Date and time when the system job was completed. |
CreatedBy |
Unique identifier of the user who created the system job. |
CreatedOn |
Date and time when the system job was created. |
Data |
Unstructured data associated with the system job. |
DependencyToken |
Execution of all operations with the same dependency token is serialized. Learn about execution order and dependencies |
Depth |
Number of SDK calls made since the first call. |
ErrorCode |
Error code returned from a canceled system job. |
ExecutionTimeSpan |
Time that the system job took to execute. |
FriendlyMessage |
Message provided by the system job. |
IsWaitingForEvent |
Indicates that the system job is waiting for an event. |
Message |
Message related to the system job. |
MessageName |
Name of the message that started this system job. |
ModifiedBy |
Unique identifier of the user who last modified the system job. |
ModifiedOn |
Date and time when the system job was last modified. |
Name |
Name of the system job. |
OperationType |
Type of the system job. Learn about operation types |
OwnerId |
Unique identifier of the user or team who owns the system job. |
OwningBusinessUnit |
Unique identifier of the business unit that owns the system job. |
OwningExtensionId |
Unique identifier of the owning extension with which the system job is associated. |
OwningTeam |
Unique identifier of the team who owns the record. |
OwningUser |
Unique identifier of the user who owns the record. |
PostponeUntil |
Indicates whether the system job should run only after the specified date and time. Learn to postpone system jobs |
PrimaryEntityType |
Type of table with which the system job is primarily associated. |
RecurrencePattern |
Pattern of the system job's recurrence. Learn about recurrence start times and patterns |
RecurrenceStartTime |
Starting time in UTC for the recurrence pattern. Learn about recurrence start times and patterns |
RegardingObjectId |
Unique identifier of the object with which the system job is associated. |
RetryCount |
Number of times to retry the system job. |
Sequence |
Order in which operations were submitted. |
StartedOn |
Date and time when the system job was started. |
StateCode |
Status of the system job. Learn to manage system job states |
StatusCode |
Reason for the status of the system job. Learn to manage system job states |
UTCConversionTimeZoneCode |
Time zone code that was in use when the record was created. |
WorkflowStageName |
Name of a workflow stage. |
You can use the following examples to retrieve System Job data.
Use the following Web API Query to retrieve the columns in the table above. [Learn how to query data using the Web API](webapi/query/overview.md
GET <organization URL>/api/data/v9.2/asyncoperations?$top=1000
&$select=
asyncoperationid,
completedon,
createdon,
data,
dependencytoken,
depth,
errorcode,
executiontimespan,
friendlymessage,
iswaitingforevent,
message,
messagename,
modifiedon,
name,
operationtype,
_ownerid_value,
postponeuntil,
primaryentitytype,
recurrencepattern,
recurrencestarttime,
_regardingobjectid_value,
retrycount,
sequence,
startedon,
statecode,
utcconversiontimezonecode,
workflowstagename
&$expand=
createdby($select=fullname),
modifiedby($select=fullname),
owningbusinessunit($select=name),
owningextensionid($select=name),
owningteam($select=name),
owninguser($select=fullname)
Note
With the Web API there is a single-valued navigation property for each table that supports system jobs. The name of this navigation property follows the pattern regardingobjectid_<table logical name>
.
The OperationType column describes categories of system jobs. Dataverse initiates many of these types to perform maintenance tasks.
Note
You cannot perform cancel, pause, or resume operations on system jobs generated by the platform.
Some of the types of these platform generated jobs are included in the following table:
OperationType Value | OperationType Label |
---|---|
9 | SQM Data Collection |
16 | Collect Organization Statistics |
18 | Calculate Organization Storage Size |
19 | Collect Organization Database Statistics |
20 | Collection Organization Size Statistics |
22 | Calculate Organization Maximum Storage Size |
24 | Update Statistic Intervals |
25 | Organization Full Text Catalog Index |
27 | Update Contract States |
31 | Storage Limit Notification |
For a complete list, see OperationType Choices/Options
Recurring system jobs require information about when they should start and how often to recur. These values are stored in the AsyncOperation
table RecurrenceStartTime
and RecurrencePattern
columns.
Because you can't create AsyncOperation
records directly with code, you need to interpret these values when you query the data. These property values are set indirectly by using messages that create new system jobs. The BulkDelete
and BulkDeleteDuplicates
messages both include parameters or properties in the corresponding Web API actions or SDK for .NET request classes. More information: BulkDetectDuplicatesRequest Class, BulkDetectDuplicates Action, BulkDeleteRequest Class, and BulkDelete Action
The RecurrenceStartTime
is simply a datetime value to indicate when the system job should start. If it isn't set, the system job was expected to start immediately.
The RecurrencePattern
column stores information about how frequently recurring system jobs occur. The platform sometimes sets this value when a new asyncoperation record is created. You can set this value to change the pattern.
The values for this column use parts of the RFC2445 Internet standard (Internet Calendaring and Scheduling Core Object Specification).
The following table provides from examples:
Recurrence pattern | Frequency of job execution |
---|---|
FREQ=MONTHLY; |
Once a month |
FREQ=WEEKLY; |
Once a week |
FREQ=DAILY; |
Once a day |
FREQ=DAILY;INTERVAL=3; |
Every three days |
FREQ=HOURLY; |
Once an hour |
If an INTERVAL
value isn't set, it's interpreted as 1
.
Use the queries in this section to help diagnose problems.
Use this query to understand the distribution and frequency of different types of jobs. The results might tell you which jobs are causing a problem.
This query doesn't order by count
descending. You may want to use FetchXml with Web API instead. Query data using FetchXml
GET [Organization URI]/api/data/v9.2/asyncoperations?$apply=groupby((statecode,statuscode,operationtype),aggregate($count as count))
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
Prefer: odata.include-annotations="OData.Community.Display.V1.FormattedValue"
Use this query to extract a count of all jobs within the AsyncOperation
table that are in a Suspended state. This query helps you to:
- Understand the volume and nature of waiting jobs.
- Identify where the hold-ups are occurring.
- Make informed decisions about how to address them to improve system performance and throughput.
This query doesn't order by count
descending. You may want to use FetchXml with Web API instead. Query data using FetchXml
GET [Organization URI]/api/data/v9.2/asyncoperations?$apply=filter((statecode eq 1))/groupby((statecode,statuscode,operationtype),aggregate($count as count))
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
Prefer: odata.include-annotations="OData.Community.Display.V1.FormattedValue"
Here's what to look for in the results:
Quantify Suspended Jobs: The query specifically targets jobs that are Suspended (
statecode
=1
). The results give you a count of all such jobs, categorized by theirstatuscode
andoperationtype
.Operation Type Breakdown: The count of Suspended jobs grouped
operationtype
, showing you the types of operations are most commonly in a Suspended state.Identifying Potential Bottlenecks: A high count of Suspended jobs for extended periods of time might be due to resource limitations, dependencies on other processes, or system misconfigurations.
Capacity and Resource Management: If certain jobs are consistently in the Suspended state, it could indicate that the system lacks the necessary resources to process these jobs efficiently.
System Health Check: The jobs in a Suspended state serve as a health indicator. A healthy system should ideally have minimal jobs in a Suspended state or at least show a quick turnover from Suspended to active processing.
Workflow Efficiency: The results can shed light on workflow efficiency. If a particular
operationtype
has a high count of Suspended jobs, it might indicate inefficiencies or the need for optimization within that workflow.
This query provides a detailed breakdown of workflow-related jobs, filtered by an operationtype
value for workflows. Use the query results to gain a comprehensive view of workflow jobs, manage system resources more effectively, and ensure that workflows are running smoothly and efficiently.
GET [Organization URI]/api/data/v9.2/asyncoperations?$apply=filter((operationtype eq 10))/groupby((statecode,statuscode,operationtype),aggregate($count as count))
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
Prefer: odata.include-annotations="OData.Community.Display.V1.FormattedValue"
Here's what to look for in the results:
Workflow-Specific Jobs: The
operationtype
=10
filter limits the results to workflow jobs. You can see how many jobs are related to workflows and their current statuses. You can also apply this example query with other types of operations. Learn more about Operation TypesJob State Distribution: The
statecode
tells you the current state of these workflow jobs, such as whether they're Ready, Suspended, Locked, or Completed.Status Code Analysis: The
statuscode
can give specific insights into the reason behind a job's current state. For example, it could indicate if a job is Waiting for Resources, Waiting, In Progress, Pausing, Cancelling, Succeeded, Failed, or Cancelled.Count of Each Category: The total number of jobs for each
statecode
andstatuscode
combination. This helps in identifying the most common outcomes of workflow operations.Identifying Common Outcomes: With the results ordered by count in descending order, you can identify the most frequent outcomes or bottlenecks in workflow processing.
Troubleshooting and Optimization: High counts in Failed status or Suspended states can highlight areas where workflows might be failing or getting stuck, signaling a need for troubleshooting or process optimization.
Performance Metrics: Understanding which workflows are most common and how they're distributed across different states can help in assessing the performance and reliability of the workflow management system.
Capacity Planning: Consistently high number of In Progress or Waiting workflows could suggest that more resources are needed to handle the load, or that there's a need to optimize the workflow execution environment.
Workflow Management: The query results can guide administrators on managing workflows more effectively, such as deciding which workflows to prioritize or identifying workflows that can be optimized or deactivated/disabled.
System Health Check: The overall results can serve as a health check for your workflow system, indicating whether the system is performing optimally or if there are areas that require attention.
Use this query to retrieve a detailed analysis of jobs from the AsyncOperation
table that are in a specific state of readiness but are pending execution due to unavailability of system resources. This query can identify factors contributing to slowness and making decisions for efficiency and better handling of backlog.
GET [Organization URI]/api/data/v9.2/asyncoperations?$apply=filter((statecode eq 0 and statuscode eq 0))/groupby((statecode,statuscode,operationtype),aggregate($count as count))
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
Prefer: odata.include-annotations="OData.Community.Display.V1.FormattedValue"
Here's what to look for in the results:
Filter for Ready Jobs Waiting for Resources: Limiting results where
statecode
=0
andstatuscode
=0
filters for jobs that are in a Ready and Waiting For Resources. This combination indicates jobs that are queued up and prepared to run but are on hold.Optimizing Job Scheduling: Identifying patterns in job readiness and wait times can inform improvements in job scheduling, possibly leading to a more balanced distribution of system load.
Identifying Underlying Issues: In some cases, jobs waiting for resources might not be solely a resource issue but could be indicative of underlying problems such as deadlocks or inefficient resource locking mechanisms.
Depending on the size of the Data column of the AsyncOperation
table, the data in that column may be saved in file storage. The DataBlobId column has a value when the row uses file storage. To save space, you might want to identify and delete these records. Use the following queries to discover these records
Use this query to count the number of records in the AsyncOperation
table where the datablobid
column isn't null.
GET [Organization URI]/api/data/v9.2/asyncoperations?$apply=filter((datablobid ne null))/aggregate($count as FileStorageCount)
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
Here's what to look for in the results:
Data Storage Implications: You may want to delete the records in using file storage to save space so this is helpful to know. Large numbers might suggest significant space used by these blobs, which can be important for database size managemement.
System Performance Considerations: If the
FileStorageCount
is unexpectedly high, you may want to take further action such as Bulk delete and clean up.
Use this query to count the number of records in the AsyncOperation
table where the datablobid
field is NULL
.
GET [Organization URI]/api/data/v9.2/asyncoperations?$apply=filter((datablobid eq null))/aggregate($count as DBCount)
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
Here's what to look for in the results:
Understanding Async Operations not in Blob Storage: The
DBCount
result will indicate the volume of async operations that do not have associated data blobs. This shows us the storage status when not accounting for the blobs.Identifying inefficiencies Unless intended, high count here may suggest the need to schedule cleanup and run bulk delete. Low count in blob storage and high count here would attribute this as the primary volume contributor.
The results of this query shows the job types, the name of the jobs, and the number of times this job exists on the table consuming file storage. Use this to identify the specific job names that have the greatest impact on file consumption and create a bulk delete job for records having that name.
This will enable the identification of the specific job names that have the greatest impact on file consumption. As a result, the customer can initiate a bulk delete process for that particular job by targeting its name.
This query doesn't order by the jobs
column decending. You may want to use FetchXml with Web API instead. Query data using FetchXml
GET [Organization URI]/api/data/v9.2/asyncoperations?$apply=filter((datablobid ne null))/groupby((operationtype,name,friendlymessage),aggregate($count as jobs))
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
Prefer: odata.include-annotations="OData.Community.Display.V1.FormattedValue"
Use this query to get total file size and record count for system jobs by state, status and owning extension.
This example uses the encoded FetchXml to send the query using Web API. Query data using FetchXml
GET [Organization URI]/api/data/v9.2/asyncoperations?fetchXml=%3Cfetch%20aggregate%3D%27true%27%3E%20%3Centity%20name%3D%27asyncoperation%27%3E%20%3Cattribute%20name%3D%27owningextensionid%27%20alias%3D%27owningextension%27%20groupby%3D%27true%27%20%2F%3E%20%3Cattribute%20name%3D%27statecode%27%20alias%3D%27statecode%27%20groupby%3D%27true%27%20%2F%3E%20%3Cattribute%20name%3D%27statuscode%27%20alias%3D%27statuscode%27%20groupby%3D%27true%27%20%2F%3E%20%3Cattribute%20name%3D%27operationtype%27%20alias%3D%27operationtype%27%20groupby%3D%27true%27%20%2F%3E%20%3Clink-entity%20name%3D%27fileattachment%27%20to%3D%27datablobid%27%20from%3D%27fileattachmentid%27%20alias%3D%27fileattachment%27%20link-type%3D%27inner%27%3E%20%3Cattribute%20name%3D%27filesizeinbytes%27%20alias%3D%27TotalSize%27%20aggregate%3D%27sum%27%20%2F%3E%20%3Cattribute%20name%3D%27filesizeinbytes%27%20alias%3D%27RecordCount%27%20aggregate%3D%27count%27%20%2F%3E%20%3Cfilter%3E%20%3Ccondition%20attribute%3D%27objectidtypecode%27%20operator%3D%27eq%27%20value%3D%274700%27%20%2F%3E%20%3C%2Ffilter%3E%20%3Corder%20alias%3D%27TotalSize%27%20descending%3D%27true%27%20%2F%3E%20%3C%2Flink-entity%3E%20%3C%2Fentity%3E%20%3C%2Ffetch%3E
Accept: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
Prefer: odata.include-annotations="OData.Community.Display.V1.FormattedValue"
Here's what to look for in the results:
- Record Count:
RecordCount
tells you how many records are being returned for each grouping of system job records. This will give you an idea of the volume of asynchronous operations being performed and which types are most common. - Total File Size:
TotalSize
tells you the amount of data these operations are handling. This can help you identify if there are any unusually large files that could be affecting system performance. - Grouping by Owning Entities: The query groups results by
owningextensionid
,owningextensionidname
,statecode
,statuscode
, andoperationtype
. Look at these groupings to pinpoint which extensions are generating the most activity and if there are specific operation types that are predominant. - Operation States and Statuses: The inclusion of
statecode
andstatuscode
in the grouping will help you determine the current state and status of these operations, such as which ones are pending, in progress, or completed. - Ordering by TotalSize: Since the results are ordered by
TotalSize
in descending order, pay attention to the top results as they will highlight the operations that are consuming the most storage. This could be important for identifying potential areas for optimization or cleanup.
You can delete system jobs in the application or in code just like any other table if you have the necessary privileges to do so.
Note
When registering asynchronous plug-ins, there is an option to automatically delete successful operations. It is recommended that you use it. Learn to write a plug-in
The common practice is to create a recurring bulk deletion job that deletes successful jobs. Learn how to remove a large amount of specific, targeted data with bulk deletion
The status of the system job changes multiple times until the operation is completed. The following are the StateCode
and StatusCode
options that represent the available state and status reason values:
StateCode Value | StateCode Label | StatusCode Value | StatusCode Label |
---|---|---|---|
0 |
Ready | 0 |
Waiting For Resources |
1 |
Suspended | 10 |
Waiting |
2 |
Locked | 20 |
In Progress |
2 |
Locked | 21 |
Pausing |
2 |
Locked | 22 |
Canceling |
3 |
Completed | 30 |
Succeeded |
3 |
Completed | 31 |
Failed |
3 |
Completed | 32 |
Canceled |
You can change the status of system jobs in the application by navigating to Settings > System > System Jobs and using commands available in the More Actions menu.
Note
Any action you can perform via this UI can also be performed using code. You cannot perform cancel, pause, or resume operations on system jobs generated by the platform. Learn about Operation types
Together with options to manage views, the following options to manage system jobs are available:
Option | Description |
---|---|
Delete | Using the command. Deletes a system job |
Bulk Delete | Using the More Actions menu. Opens a wizard to define conditions and create a new Bulk Deletion system job to delete the matching system jobs. |
Cancel | Using the More Actions menu. Cancels the system job. |
Resume | Using the More Actions menu. Resumes a paused system job. |
Postpone | Using the More Actions menu. Reschedules a system job |
Pause | Using the More Actions menu. Pauses a system job. |
Whether the requested operation occurs depends on the state of the system job. For example, you can't pause a job that is completed or hasn't started yet. The following table describes the conditions for each change and what happen when they're selected.
Option | Valid StateCode values | Change |
---|---|---|
Delete | any | System Job is deleted |
Cancel | 0 (Ready) 1 (Suspended) 2 (Locked) |
StateCode changed to 3 (Completed) and StatusCode changed to 32 (Cancelled) |
Resume | 1 (Suspended) |
StateCode changed to 0 (Ready) |
Postpone | 0 (Ready) 2 (Locked) |
Postpone Job dialog prompts user for datetime value to postpone the system job. Learn to Postpone system jobs |
Pause | 2 (Locked) |
StateCode changed to 1 (Suspended) |
The PostPoneUntil
column contains a datetime value when the system job changes state from 1
(Suspended) to 0
(Ready). The PostPoneUntil
, StateCode
, and StatusCode
columns are the only AsyncOperation
table columns supported for update.