Retrieve and detect changes to metadata
The classes in the Microsoft.Xrm.Sdk.Metadata.Query namespace and the RetrieveMetadataChangesRequest and RetrieveMetadataChangesResponse classes let you build efficient metadata queries and capture changes to metadata as they occur over time.
All code examples referenced in this document are found in Sample: Query Metadata and Detect Changes.
The technical article Query Metadata Using JavaScript provides a JavaScript library to use the objects and messages in client-sided code.
Strategies for using metadata
Metadata lets you create applications that adapt as the Dynamics 365 Customer Engagement (on-premises) data model changes. Metadata is important for the following types of application:
UI for client applications
Integration tools that have to map Dynamics 365 Customer Engagement (on-premises) data to external systems
Development tools
Using the classes in the Microsoft.Xrm.Sdk.Metadata.Query namespace you can implement designs that will exist somewhere between a lightweight query and a persistent metadata cache.
Lightweight query
An example of a lightweight query is when you have a custom web resource UI that provides a select control to display the current options in a Dynamics 365 Customer Engagement Option Set (Picklist) attribute. You do not want to hard-code these options because you would have to update that code if the available options are ever changed. Instead you can construct a query to just retrieve those options values and labels from the metadata.
You do not have to cache this data because you can use the Microsoft.Xrm.Sdk.Metadata.Query classes to retrieve this data directly from the Dynamics 365 Customer Engagement application cache.
Persistent metadata cache
When you have an application that must be able to work while disconnected from the Dynamics 365 Server, or that is sensitive to limited network bandwidth between the client and the server, such as a mobile application, you will want to implement a persistent metadata cache.
With a persistent metadata cache your application will have to query all the necessary metadata the first time it connects. Then you will save that data in the application. The next time the application connects to the server you can retrieve just the difference since your last query, which should be much less data to transmit, and then merge the changes into your metadata cache when your application is loading.
How frequently you should poll for metadata changes depends on the expected volatility of metadata for your application and how long your application will remaining running. There is no event available that you can use to detect when metadata changes occur. There is a limit to the number of days that deleted metadata changes are saved and a request for changes that occurs beyond that limit will require a full re-initialization of the metadata cache. For more information see Deleted metadata expiration.
When there are no changes the query should respond quickly and there will be no data to transmit back. However, if there are changes, especially if there are deleted metadata items that have to be removed from your cache, you can expect that the request may take some additional time to finish. More information: Performance When Retrieving Deleted Metadata
Retrieve only the metadata you need
Metadata is frequently retrieved or synchronized when an application starts and can affect the time the application takes to load. This is particularly true for mobile applications retrieving metadata for the first time. Retrieving only the metadata you need is very important to create an application that performs well.
The EntityQueryExpression class provides a structure consistent with the QueryExpression class you use to create complex queries to retrieve entity data. Unlike the RetrieveAllEntitiesRequest, RetrieveEntityRequest, RetrieveAttributeRequest, or RetrieveRelationshipRequest classes, the RetrieveMetadataChangesRequest contains a Query
parameter that accepts an EntityQueryExpression instance that you can use to specify specific criteria for the data to return in addition to which properties you want. You can use RetrieveMetadataChangesRequest to return the full set of metadata that you get using the RetrieveAllEntitiesRequest, or just a label for a specific attribute.
Specify your filter criteria
The EntityQueryExpression.Criteria property accepts a MetadataFilterExpression that contains a collection of MetadataConditionExpression objects that allow for defining conditions for filtering entity properties based on their value. These conditions use a MetadataConditionOperator that allows for the following operators:
MetadataConditionOperator.Equals
MetadataConditionOperator.NotEquals
MetadataConditionOperator.GreaterThan
MetadataConditionOperator.LessThan
The MetadataFilterExpression also includes a LogicalOperator to represent whether to apply
And
orOr
logic when you evaluate the conditions.Not all properties can be used as filter criteria. Only properties that represent simple data types, enumerations, BooleanManagedProperty or AttributeRequiredLevelManagedProperty types can be used in a MetadataFilterExpression. When a BooleanManagedProperty or AttributeRequiredLevelManagedProperty is specified, only the
Value
property is evaluated.The following table lists EntityMetadata properties that cannot be used in a MetadataFilterExpression:
The following example shows a MetadataFilterExpression that will return a set of non-intersect, user-owned entities not included in a list of entities to exclude:
// An array SchemaName values for non-intersect, user-owned entities that should not be returned.
String[] excludedEntities = {
"WorkflowLog",
"Template",
"CustomerOpportunityRole",
"Import",
"UserQueryVisualization",
"UserEntityInstanceData",
"ImportLog",
"RecurrenceRule",
"QuoteClose",
"UserForm",
"SharePointDocumentLocation",
"Queue",
"DuplicateRule",
"OpportunityClose",
"Workflow",
"RecurringAppointmentMaster",
"CustomerRelationship",
"Annotation",
"SharePointSite",
"ImportData",
"ImportFile",
"OrderClose",
"Contract",
"BulkOperation",
"CampaignResponse",
"Connection",
"Report",
"CampaignActivity",
"UserEntityUISettings",
"IncidentResolution",
"GoalRollupQuery",
"MailMergeTemplate",
"Campaign",
"PostFollow",
"ImportMap",
"Goal",
"AsyncOperation",
"ProcessSession",
"UserQuery",
"ActivityPointer",
"List",
"ServiceAppointment"};
//A filter expression to limit entities returned to non-intersect, user-owned entities not found in the list of excluded entities.
MetadataFilterExpression EntityFilter = new MetadataFilterExpression(LogicalOperator.And);
EntityFilter.Conditions.Add(new MetadataConditionExpression("IsIntersect", MetadataConditionOperator.Equals, false));
EntityFilter.Conditions.Add(new MetadataConditionExpression("OwnershipType", MetadataConditionOperator.Equals, OwnershipTypes.UserOwned));
EntityFilter.Conditions.Add(new MetadataConditionExpression("SchemaName", MetadataConditionOperator.NotIn, excludedEntities));
MetadataConditionExpression isVisibileInMobileTrue = new MetadataConditionExpression("IsVisibleInMobile", MetadataConditionOperator.Equals, true);
EntityFilter.Conditions.Add(isVisibileInMobileTrue);
Specify the properties you want
The Properties property accepts a MetadataPropertiesExpression. You can set MetadataPropertiesExpression.AllProperties to true
if you want to return all the properties or you can provide a collection of strings to the MetadataPropertiesExpression.PropertyNames to define which properties you want to include in the results.
The strongly typed objects returned will include all properties, but only those that you request will have data. All other properties will be null, with the following few exceptions: every item of metadata will include the MetadataId ,LogicalName
and HasChanged values if they exist for that item. You do not have to specify them in the Properties you request.
If you are not using managed code and are actually parsing the responseXML
returned from the XMLHttpRequest you will get elements for each property but only those you request will contain data. The following XML shows the contact entity metadata xml that will be returned when IsVisibleInMobile
is the only property requested.
<a:EntityMetadata>
<c:MetadataId>608861bc-50a4-4c5f-a02c-21fe1943e2cf</c:MetadataId>
<c:HasChanged i:nil="true"/>
<c:ActivityTypeMask i:nil="true"/>
<c:Attributes i:nil="true"/>
<c:AutoRouteToOwnerQueue i:nil="true"/>
<c:CanBeInManyToMany i:nil="true"/>
<c:CanBePrimaryEntityInRelationship i:nil="true"/>
<c:CanBeRelatedEntityInRelationship i:nil="true"/>
<c:CanCreateAttributes i:nil="true"/>
<c:CanCreateCharts i:nil="true"/>
<c:CanCreateForms i:nil="true"/>
<c:CanCreateViews i:nil="true"/>
<c:CanModifyAdditionalSettings i:nil="true"/>
<c:CanTriggerWorkflow i:nil="true"/>
<c:Description i:nil="true"/>
<c:DisplayCollectionName i:nil="true"/>
<c:DisplayName i:nil="true"/>
<c:IconLargeName i:nil="true"/>
<c:IconMediumName i:nil="true"/>
<c:IconSmallName i:nil="true"/>
<c:IsActivity i:nil="true"/>
<c:IsActivityParty i:nil="true"/>
<c:IsAuditEnabled i:nil="true"/>
<c:IsAvailableOffline i:nil="true"/>
<c:IsChildEntity i:nil="true"/>
<c:IsConnectionsEnabled i:nil="true"/>
<c:IsCustomEntity i:nil="true"/>
<c:IsCustomizable i:nil="true"/>
<c:IsDocumentManagementEnabled i:nil="true"/>
<c:IsDuplicateDetectionEnabled i:nil="true"/>
<c:IsEnabledForCharts i:nil="true"/>
<c:IsImportable i:nil="true"/>
<c:IsIntersect i:nil="true"/>
<c:IsMailMergeEnabled i:nil="true"/>
<c:IsManaged i:nil="true"/>
<c:IsMappable i:nil="true"/>
<c:IsReadingPaneEnabled i:nil="true"/>
<c:IsRenameable i:nil="true"/>
<c:IsValidForAdvancedFind i:nil="true"/>
<c:IsValidForQueue i:nil="true"/>
<c:IsVisibleInMobile>
<a:CanBeChanged>false</a:CanBeChanged>
<a:ManagedPropertyLogicalName>canmodifymobilevisibility</a:ManagedPropertyLogicalName>
<a:Value>false</a:Value>
</c:IsVisibleInMobile>
<c:LogicalName>contact</c:LogicalName>
<c:ManyToManyRelationships i:nil="true"/>
<c:ManyToOneRelationships i:nil="true"/>
<c:ObjectTypeCode i:nil="true"/>
<c:OneToManyRelationships i:nil="true"/>
<c:OwnershipType i:nil="true"/>
<c:PrimaryIdAttribute i:nil="true"/>
<c:PrimaryNameAttribute i:nil="true"/>
<c:Privileges i:nil="true"/>
<c:RecurrenceBaseEntityLogicalName i:nil="true"/>
<c:ReportViewName i:nil="true"/>
<c:SchemaName i:nil="true"/>
</a:EntityMetadata>
In a future release further efficiencies may be achieved by not returning elements with null values for properties that were not requested. If you write code to parse this XML you should expect that the XML returned for the same query could be reduced to only the following XML.
<a:EntityMetadata>
<c:MetadataId>608861bc-50a4-4c5f-a02c-21fe1943e2cf</c:MetadataId>
<c:IsVisibleInMobile>
<a:CanBeChanged>false</a:CanBeChanged>
<a:ManagedPropertyLogicalName>canmodifymobilevisibility</a:ManagedPropertyLogicalName>
<a:Value>false</a:Value>
</c:IsVisibleInMobile>
<c:LogicalName>contact</c:LogicalName>
</a:EntityMetadata>
Metadata is returned in a hierarchical structure just as it is using the RetrieveAllEntitiesRequest. To access a specific attribute or relationship you must create a query that returns the entity they are part of. If you want to retrieve data about a specific attribute, you must include the EntityMetadata.Attributes property in your EntityQueryExpression.Properties. For the entity relationships to be returned, you must include one or more of the following EntityMetadata properties: ManyToManyRelationships, ManyToOneRelationships, or OneToManyRelationships.
The following example will return the Attributes
property for requested entities:
//A properties expression to limit the properties to be included with entities
MetadataPropertiesExpression EntityProperties = new MetadataPropertiesExpression()
{
AllProperties = false
};
EntityProperties.PropertyNames.AddRange(new string[] { "Attributes" });
Retrieve attribute metadata
The EntityQueryExpression.AttributeQuery property accepts an AttributeQueryExpression that defines Criteria and Properties for attributes to be returned for the entities that match the EntityQueryExpressionCriteria and Properties.
The following table lists AttributeMetadata properties that cannot be used in a MetadataFilterExpression
The following example will limit Attributes returned to only those that have an OptionSet
and will only return the OptionSet and AttributeType properties for those attributes:
//A condition expresson to return optionset attributes
MetadataConditionExpression[] optionsetAttributeTypes = new MetadataConditionExpression[] {
new MetadataConditionExpression("AttributeType", MetadataConditionOperator.Equals, AttributeTypeCode.Picklist),
new MetadataConditionExpression("AttributeType", MetadataConditionOperator.Equals, AttributeTypeCode.State),
new MetadataConditionExpression("AttributeType", MetadataConditionOperator.Equals, AttributeTypeCode.Status),
new MetadataConditionExpression("AttributeType", MetadataConditionOperator.Equals, AttributeTypeCode.Boolean)
};
//A filter expression to apply the optionsetAttributeTypes condition expression
MetadataFilterExpression AttributeFilter = new MetadataFilterExpression(LogicalOperator.Or);
AttributeFilter.Conditions.AddRange(optionsetAttributeTypes);
//A Properties expression to limit the properties to be included with attributes
MetadataPropertiesExpression AttributeProperties = new MetadataPropertiesExpression() { AllProperties = false };
AttributeProperties.PropertyNames.Add("OptionSet");
AttributeProperties.PropertyNames.Add("AttributeType");
Retrieve relationship metadata
The EntityQueryExpression.RelationshipQuery property accepts a RelationshipQueryExpression to specify the entity relationship Criteria and Properties you want for the entities that match the EntityQueryExpressionCriteria and Properties.
Use the RelationshipType property in your criteria to specify whether you want to return ManyToMany Relationships or OneToMany Relationships.
The following table lists Relationship metadata properties that cannot be used in a MetadataFilterExpression:
- OneToManyRelationshipMetadata.AssociatedMenuConfiguration
- OneToManyRelationshipMetadata.CascadeConfiguration
- ManyToManyRelationshipMetadata.Entity1AssociatedMenuConfiguration
-ManyToManyRelationshipMetadata.Entity2AssociatedMenuConfiguration
Retrieve labels
Finally, the EntityQueryExpression.LabelQuery property accepts a LabelQueryExpression that lets you specify one or more integer LCID
values for to determine which localized labels to return. Valid locale ID values can be found at Locale ID (LCID) Chart. If an organization has many language packs installed the labels for all languages will be returned unless you specify a LabelQuery.
The following example defines a LabelQueryExpression that will limit labels to only those representing the users preferred language.
private Guid _userId;
private int _languageCode;
_userId = ((WhoAmIResponse)_service.Execute(new WhoAmIRequest())).UserId;
_languageCode = RetrieveUserUILanguageCode(_userId);
protected int RetrieveUserUILanguageCode(Guid userId)
{
QueryExpression userSettingsQuery = new QueryExpression("usersettings");
userSettingsQuery.ColumnSet.AddColumns("uilanguageid", "systemuserid");
userSettingsQuery.Criteria.AddCondition("systemuserid", ConditionOperator.Equal, userId);
EntityCollection userSettings = _service.RetrieveMultiple(userSettingsQuery);
if (userSettings.Entities.Count > 0)
{
return (int)userSettings.Entities[0]["uilanguageid"];
}
return 0;
}
//A label query expression to limit the labels returned to only those for the user's preferred language
LabelQueryExpression labelQuery = new LabelQueryExpression();
labelQuery.FilterLanguages.Add(_languageCode);
Retrieve new or changed metadata
The RetrieveMetadataChangesResponse class returns a strongly typed EntityMetadataCollection that contains the requested data. The RetrieveMetadataChangesResponse class also provides a ServerVersionStamp value that you can pass to the RetrieveMetadataChangesRequest.ClientVersionStamp property in later requests. When a value is included for the ClientVersionStamp property, only data that matches the EntityQueryExpression and has changed since the ClientVersionStamp
was retrieved will be returned. The only exception to this is when your EntityQueryExpression.Properties includes EntityMetadata.Privileges. Privileges will always be returned regardless of the ClientVersionStamp. This way your application can determine whether any important changes have occurred that you care about since you last queried the metadata. You can then merge any new or changed metadata into your persistent metadata cache so that your application will be able to avoid the performance issues with downloading metadata you may not need.
The HasChanged property provides a way to detect which child elements in a metadata item have changed. Because all the metadata is returned as part of the containing metadata item, when the label of a OptionMetadata has changed, the containing EntityMetadata, AttributeMetadata, and OptionSetMetadata properties are returned. However, the HasChanged property will be false for those containing metadata items. Only the OptionMetadata.HasChanged property will be true.
The following example makes an initial request by defining an EntityQueryExpression and executing a request with a ClientVersionStamp set to null.
//An entity query expression to combine the filter expressions and property expressions for the query.
EntityQueryExpression entityQueryExpression = new EntityQueryExpression()
{
Criteria = EntityFilter,
Properties = EntityProperties,
AttributeQuery = new AttributeQueryExpression()
{
Criteria = AttributeFilter,
Properties = AttributeProperties
},
LabelQuery = labelQuery
};
//Retrieve the metadata for the query without a ClientVersionStamp
RetrieveMetadataChangesResponse initialRequest = getMetadataChanges(entityQueryExpression, null, DeletedMetadataFilters.OptionSet);
protected RetrieveMetadataChangesResponse getMetadataChanges(
EntityQueryExpression entityQueryExpression,
String clientVersionStamp,
DeletedMetadataFilters deletedMetadataFilter)
{
RetrieveMetadataChangesRequest retrieveMetadataChangesRequest = new RetrieveMetadataChangesRequest()
{
Query = entityQueryExpression,
ClientVersionStamp = clientVersionStamp,
DeletedMetadataFilters = deletedMetadataFilter
};
return (RetrieveMetadataChangesResponse)_service.Execute(retrieveMetadataChangesRequest);
}
Retrieve information about deleted metadata
The RetrieveMetadataChangesResponse.DeletedMetadata property will return a DeletedMetadataCollection when the ClientVersionStamp and DeletedMetadataFilters properties are set on the RetrieveMetadataChangesRequest. The DeletedMetadataCollection contains the MetadataId values of any EntityMetadata, AttributeMetadata or RelationshipMetadataBase objects that have been deleted from the system in a time limit. For more information, see Deleted metadata expiration.
Use the DeletedMetadataFilters enumeration with the RetrieveMetadataChangesRequest.DeletedMetadataFilters to limit the information to only those types of metadata you are interested in. The DeletedMetadataFilters enumeration provides the following options:
DeletedMetadataFilters.Entity (Default)
DeletedMetadataFilters.Attribute
DeletedMetadataFilters.Relationship
DeletedMetadataFilters.Label
DeletedMetadataFilters.OptionSet
You will also use DeletedMetadataFilters enumeration as a key to the RetrieveMetadataChangesResponse.DeletedMetadata to filter the
GUID
values found in the RetrieveMetadataChangesResponse.DeletedMetadata property.When you design a metadata cache you will want to use the MetadataId for each item so that you can identify deleted metadata items and remove them.
Deleted metadata expiration
Any metadata items that are deleted are tracked for a limited period of time specified by the Organization.ExpireSubscriptionsInDays
value. By default this value is 90 days. If the RetrieveMetadataChangesRequest.ClientVersionStamp value indicates that the last metadata query was from before the expiration date, the service will throw an ExpiredVersionStamp
error (0x80044352), When you are retrieving data to refresh and existing metadata cache you should always try to catch this error and be prepared to re-initialize your metadata cache with the results from a second request passed without a ClientVersionStamp. The ExpiredVersionStamp
error is also thrown when changes on the server, such as changes to the ExpireSubscriptionsInDays
value, affect accurately tracking the deleted metadata.
The following example passes a ClientVersionStamp and catches the ExpiredVersionStamp
. If the error is caught the cache is reinitialized by passing in a new request with the ClientVersionStamp set to null
.
protected String updateOptionLabelList(EntityQueryExpression entityQueryExpression, String clientVersionStamp)
{
//Retrieve metadata changes and add them to the cache
RetrieveMetadataChangesResponse updateResponse;
try
{
updateResponse = getMetadataChanges(entityQueryExpression, clientVersionStamp, DeletedMetadataFilters.OptionSet);
addOptionLabelsToCache(updateResponse.EntityMetadata, true);
removeOptionLabelsFromCache(updateResponse.DeletedMetadata, true);
}
catch (FaultException<Microsoft.Xrm.Sdk.OrganizationServiceFault> ex)
{
// Check for ErrorCodes.ExpiredVersionStamp (0x80044352)
// Will occur when the timestamp exceeds the Organization.ExpireSubscriptionsInDays value, which is 90 by default.
if (ex.Detail.ErrorCode == unchecked((int)0x80044352))
{
//reinitialize cache
_optionLabelList.Clear();
updateResponse = getMetadataChanges(entityQueryExpression, null, DeletedMetadataFilters.OptionSet);
//Add them to the cache and display the changes
addOptionLabelsToCache(updateResponse.EntityMetadata, true);
}
else
{
throw ex;
}
}
return updateResponse.ServerVersionStamp;
}
Performance when retrieving deleted metadata
When a metadata item is deleted it is saved in the database and not in the Dynamics 365 Customer Engagement metadata cache. Although the deleted metadata is limited to just the MetadataId and the type of metadata item, accessing the database is an operation that will require more server resources than just querying for changes.
See also
Write Applications and Server Extensions
Offline Use of the Dynamics 365 Customer Engagement Services
Sample: Query Metadata and Detect Changes
Extend the Metadata Model for Dynamics 365 Customer Engagement
Customize Entity Metadata
Customize Entity Attribute Metadata
Customize Entity Relationship Metadata