Cache Schema data
Some applications benefit from maintaining a persistent cache of schema definitions for a Dataverse organization. For example:
- Applications that need to work while disconnected from the Dataverse server
- Applications that are sensitive to limited network bandwidth, such as mobile applications
Dataverse has a lot of schema definition data, and few applications must track all of it. Limiting the amount of data to cache is essential.
The RetrieveMetadataChanges
message provides two capabilities:
- Query: Compose a single query to retrieve just the schema data you need. Composing queries is covered in Query Schema Definitions.
- Cache Management: If you cache the schema definition data with your app, you can use
RetrieveMetadataChanges
to efficiently retrieve only the changes since your last query. Use the information about these changes to add or remove items in your cache. Caching schema data might result in a significant improvement time for your application to start, and is the focus of this article.
After you define a query for the schema data you want to cache, you can use other capabilities of the RetrieveMetadataChanges
message to create and maintain a cache of Dataverse schema data.
Create a cache
As described in Query Schema Definitions, the first step is to create a query that defines the kind of schema information you're interested in. Then use the RetrieveMetadataChanges
message to execute that query and cache the schema definitions returned in your application. You should find a way to persist this data that is appropriate for the needs of your app. You might save it as a file that your application reads when it starts. You also need to save data about the last time you ran the query.
Structure your cache in a way that makes sense for your application. To manage removal of deleted items, it's important that your cache uses the MetadataId
as a unique identifier. More information Remove deleted items.
Detect changes
There's no event available to detect when schema definition changes occur. The needs of your application should drive refreshing the cache. People usually retrieve changes when the application starts. But you might poll the system regularly if you want to detect changes over time. Regardless of your strategy, it's important to keep track of the time you sent the previous request.
The RetrieveMetadataChangesResponse.ServerVersionStamp
property contains information about the point in time the request to RetrieveMetadataChanges
occurred. You must take the ServerVersionStamp
value from the previous response and use it as the value for the RetrieveMetadataChangesRequest.ClientVersionStamp
when you send it again using the same query.
When you include the ClientVersionStamp
property in the request, the RetrieveMetadataChangesResponse.EntityMetadata
property returned contains only the changed or added schema data since the previous request. You can add them to your cache.
If you also include the DeletedMetadataFilters
parameter, any schema items deleted since the previous request are included in the RetrieveMetadataChangesResponse.DeletedMetadata
property. You can remove them from your cache.
This result should be smaller and faster than executing the original query again.
Manage expired cache
Information about changes is stored for 90 days by default. This value is stored in the Organization.ExpireSubscriptionsInDays property. If you send a request with a ClientVersionStamp
value that is older than the setting value, Dataverse returns an ExpiredVersionStamp
error (0x80044352
). You should be prepared to handle this specific error and reinitialize your cache when it occurs.
Note
You should be prepared to manage this error even if you think that your value will always be less than 90 days. This error will be thrown when any changes on the server affect accurately tracking deleted schema data. For example, changing the Organization.ExpireSubscriptionsInDays
property will invalidate all previous VersionStamp
values. Some of these changes may not be caused by actions you take, but could be triggered by system maintenance.
Add changed items
Just as when you perform your initial query to initialize your cache, the RetrieveMetadataChangesResponse.EntityMetadata
property returned for subsequent requests using RetrieveMetadataChangesRequest.ClientVersionStamp
includes the full hierarchy of changed items.
Each item in the hierarchy has a nullable boolean HasChanged
value. When this value is false, it means the current item hasn't changed, but something below it in the hierarchy has changed. When the HasChanged
value is true, it means the current item in the hierarchy has changed.
Note
If you request EntityMetadata.Privileges
in your query the privileges will always be returned, whether they have changed or not. Privileges typically don't change.
Tracking Options
One area that might not seem intuitive is how options for choice columns are tracked. This area is a good example to understand the HasChanged
property.
When a new option is added to a choice column, the following data in the hierarchy will be returned with the RetrieveMetadataChangesResponse.EntityMetadata
property:
- EntityMetadataCollection
- EntityMetadata: The table definition.
- EntityMetadata.Attributes[]
- EnumAttributeMetadata: The base class for Choice columns.
- EnumAttributeMetadata.OptionSet
- OptionSetMetadata:
HasChanged
is true.- OptionSetMetadata.Options: All options are returned.
- OptionMetadata
- OptionMetadata.Color: Only new option has value, if set.
- OptionMetadata.Label: Only new option has value.
- OptionMetadata.Value: Always has value.
- OptionMetadata.HasChanged: Value is null.
- OptionMetadata
- OptionSetMetadata.Options: All options are returned.
- OptionSetMetadata:
- EnumAttributeMetadata.OptionSet
- EnumAttributeMetadata: The base class for Choice columns.
- EntityMetadata.Attributes[]
- EntityMetadata: The table definition.
You know that the EntityMetadata
and EnumAttributeMetadata
haven't changed because the HasChanged
property is false. Only the OptionSetMetadata.HasChanged
property is true. All the current valid options are returned. The OptionMetadata.HasChanged
property for all the options, including the new one, are null.
Only the newly created option includes data for the Label
property. The OptionMetadata.HasChanged
property is true only when one of it's properties (like Color
or Label
) changes.
Remove deleted items
When you include both ClientVersionStamp
and DeletedMetadataFilters
parameters for a RetrieveMetadataChanges
message, the RetrieveMetadataChangesResponse.DeletedMetadata
property contains data about any deleted items. This property is a DeletedMetadataCollection
that contains a set of Keys
and Values
. The keys are DeletedMetadataFilters
enum values and you can use these values to access a subset the deleted values.
Note
The DeletedMetadataFilters
parameter is optional and there is some performance impact when you include it because the data is retrieved from the database directly rather than from the internal cache. Use it only when you must detect deleted items and set the filter value to the minimum amount of data you need to have returned.
The deleted values are returned as a collection of Guid
values. The values for deleted items aren't filtered based on your query. There are many Guid
values that aren't in your cache, but the Guid
values for your cached item are included. You must look for any matching Guid
values and remove those items. Disregard any values that aren't in your cache.
Note
The definition of the Web API DeletedMetadataFilters EnumType is slightly different from the SDK DeletedMetadataFilters Enum.
The Web API DeletedMetadataFilters EnumType doesn't have the Entity
member. You must use Default
instead.
Tracking Deleted Options
Notice that the DeletedMetadataFilters
enum contains a member for OptionSet
, but not option. If any options are removed from a choice column, you won't find a reference to a specific option that was deleted. You need to go to your cached options and compare them to the current options returned for that OptionSet
.
See also
Web API Query schema definitions and detect changes Sample (C#)
SDK for .NET Query schema definitions and detect changes Sample (C#)
Query Schema Definitions