SharePoint Server 2007 and Records Management: Part 4 of 4

This is the fourth and final post in a series on Records Management using SharePoint Server 2007:

1. Introduction to Records Management, and Definition of Terms

2. Records Management using Standard SharePoint Features

3. Using the DoD 5015.2 Records Management Resource Kit

4. Commonly Requested Enhancements and Features (this post)

In the previous posts, we examined the basics of Electronic Records Management (ERM) and how standard SharePoint Server 2007 features, and the DoD 5015 Records Management Resource Kit can help meet many of the requirements that organisations have regarding Records Management.

This post examines some of the most-requested features from organisations wishing to use the SharePoint/DoD Resource Kit functionality to implement an ERM strategy. These features represent custom add-on functionality that can be developed to enhance the existing features.

Commonly Requested Enhancements and Features

The DoD Resource Kit extends the Records Management functionality of SharePoint Server 2007, but it has been designed specifically to meet the required ways of working specified by the DoD 5015.2 standard. There are many other national and local standards in use around the world which specify different ways of working, for which the Resource Kit may not be suitable. Through working with customers in the UK, our team in Microsoft Consulting Services have found that the following features are often requested.

Document Management

Although what happens when a document arrives at the records centre is critical, often what happens before documents are sent to the records centre may be even more important to the success of a records management solution. This is especially true of SharePoint administrative tasks such as designing and implementing content types. It is important to understand fully the requirements of the document management solution and how it integrates with the records management system.

Distributed content type management

One of the features that is most requested by customers in a records management environment is a distributed Content Type management system. Content types are used to control the definition and structure of content and are especially important in a records management vein, where content types control all processes from routing to retention and expiry.

SharePoint Server 2007 allows for complex structures to be built up around Content Types, but the management of content types is by default restricted in scope to individual site collections. If you want to make a set of content types available across multiple site collections, you must package the content types as a ‘Feature’ and then deploy the Feature across multiple sites – usually considered a ‘development task’ rather than something typical business users would untertake. Furthermore, you will probably want to make changes to your content types over time, meaning that you must also develop a custom means of distributing changes to content types across multiple site collections. This means that, on the records management side, the use of multiple records centers would require careful manual synchronization of content types; on the document management side, careful design must go into the structure of site collections and sites to reduce the impact this has, and inevitably there are organisation-wide content types that must be synchronized across disparate site collections.

It would be desirable to allow content types to be defined and maintained in a single, “master” site collection, and for them to be automatically and periodically distributed to child, or “destination” site collections.

A custom solution can be developed to allow for a single defined metadata repository to be created and maintained and destination sites to automatically retrieve updates when they are made on the master site collection.

Distributed library management

Implementing a records management solution often forces an organisation to re-think how they classify content in collaboration and document management areas. It is difficult to carefully structure and automate a records management system when the source documents are completely unstructured and have no formal metadata system.

Similar to the concept of centrally managing content types and then “distributing” those content types to recipient ‘subscriber’ site collections, a common requirement is that similar functionality be implemented to manage and associate consistent sets of metadata with document libraries. This creates the concept of a “Library Type”, where library settings such as versioning and auditing, and default content types and folders, can be set and maintained on a library template, which is then distributed to targeted destination sites.

For example, you could define a custom “Project Admin” library type. All libraries using our imagined Project Admin type would have versioning turned on, and would include a pre-defined set of ten default document content types with templates – because these settings and content types are defined in the library type. This library can be distributed to every project site within the organisation, and when subsequent updates are made to the library settings or default content types and templates, the updates are re-distributed to project sites.

Note that the concept described above requires a custom solution to implement.

Records Management

Custom automated routing

The routing provided by the default SharePoint records management functionality is based on matching the names of content types of arriving records to locations within a simplistic flat file plan; and the DoD pack supplies no automated routing besides the default functionality. This is fine for small organisations, but generally the volume of records and routing is too large to allow for rudimentary or manual routing.

In general the only sustainable, maintainable system that can be implemented is a rules-based routing system, which examines the content and its metadata before deciding on a location in the file plan and retention and disposition instructions. Such systems must generally be maintained by non-technical users, and must be flexible enough to represent the full gamut of possible properties and locations that the file plan dictates.

This is closely dependant on a single, centrally managed metadata repository, which is shared amongst document management sites and records management centers.

The router must usually allow for the automatic creation of folders if they don’t exist, and for the allocation of fixed ID’s to the folders, either randomly following a fixed structure or based on a pre-defined pattern or file plan numbering system.

Automated Metadata Management

Along with the routing and classification of records, organisations often require records of different types to have additional metadata fields populated and maintained on them. These fields can be populated based on existing properties of the incoming document, the location of the originating document, or properties inferred from a rules definition somewhere. Examples of such properties include the document’s project, business purpose, template information such as customer information, etc.

This type of property management can be handled either on the document management system, where metadata of documents is augmented when they are created and edited based on factors such as where they are located and their purpose and structure; or as they arrive at the records centre and are processed to determine their structure, content and purpose.

Usually included in this type of requirement is a scheme for centrally managing and administering a metadata scheme, whereby organisation-wide metadata structures can be defined and maintained. These structures are often hierarchical “tags” that can be applied to items, forming classification systems for the content and structure of items, documents and records.

Folder-level retention and disposition

Although with various configuration options, both the standard SharePoint Server 2007 features and the DoD pack allow for some level of folder-level retention and disposition, it is often complex to maintain. The ideal strategy is to control the retention and disposition of records based on Content Type, and if the number of ‘classes’ of records defined are small enough, then this is a good strategy to use. However, in organisations that require many classes of content, this can lead to difficulties in maintaining and structuring the content types, and in some cases the only pragmatic solution is to have the ability to define these structures on a folder level.

Default SharePoint Server 2007 functionality does not allow for sequential disposition steps, and the DoD pack mostly enforces this structure on the category, or document library, level. It is often required that folder-level (or at the very least content-type-level) dispositions can be defined.

Customisable export

Included in the DoD pack (as one of the disposition instructions) is the instruction to “export” records (Transfer), either before deletion or as a single instruction. However, the structure that the records are exported in is a SharePoint content-management based structure, which is useful only in the context of restoring those files at a later date to a SharePoint-based system.

The ability to export part, or all, of the file plan in a definable standard format (such as a ‘national archives’-mandated format) is an extremely useful function. This can be done either manually (by selecting to export a part of the file plan) or as a result of a set of records expiring. In the case of government or financial organisations this feature is often required to demonstrate compliance.

Scalability

SharePoint Server 2007 itself is very scalable, and can scale up and out to allow for large amounts of users and data. Certain restrictions on the functionality within the SharePoint Records Center, and the DoD pack, limit this potential however. This is due to many of the features being designed to work well within a single SharePoint site collection, i.e. a single Records Center site. For small organisations or those with a low record throughput, this is ideal. However, for systems declaring thousands of records a month, the recommended maximum storage capacity of a single Records Center site collection (or content database) may be reached relatively quickly; so in those circumstances the system requires modifications in order to scale to the required volumes.

Generally, catering for scale involves designing for multiple records centers to handle the volumes of data. This requires modifying the built-in functionality of SharePoint and the Records Center so that they are multi-centre aware.

In addition, modifications to the design are necessary to avoid software boundaries in any part of the functionality, and to make administration and maintenance tasks manageable when dealing with thousands or potentially millions of items such as records, workflows, events, etc.

Often the system can handle the growth of content in terms of documents and records, but corresponding system entries such as list entries and user population will start to reach the recommended software boundaries. Each case where this is likely to occur must be examined and planned for.

Customized workflows

The combined system of document and records management often involves several workflow processes, from managing of documents to declaration of records to disposition of records. It is very seldom that the out-of-the-box functionality will match the individual nuances of any organisation, and customized workflows are almost always implemented as part of this type of solution.

Reporting

Some organisations have legislative requirements for reporting on usage and adherence to process for their records management solution. For example, government organisations may be required to prove that a certain record was destroyed correctly according to their disposition policies, for FOI (Freedom Of Information) purposes. It is important to review the standard reporting features provided with SharePoint to determine whether they will include enough information to satisfy the requirements. Often a custom reporting framework will be required to extend the amount of detail captured into the standard reports.

Conclusion

Implementing a records management solution for any organisation can be a complex and difficult undertaking. The technology can form only a small part in a project that involves everything from analyzing how people work with their documents, all the way through to how legislation may influence the way records are maintained and destroyed.

However, the technology is often the keystone decision factor in these projects. Cost, complexity, ease of implementation, support availability – they all play an important part in deciding on the design and implementation of a technological solution.

Microsoft Office SharePoint Server provides excellent tools for collaboration and document management, and already has a large footprint in many organisations, serving as a platform for collaboration, document management, search, publishing, and custom development. It has a very distinct role to play in records management implementation as well.

Records management within SharePoint can be implemented up to the level of complexity that is required. It offers great basic-level records management capabilities out-of-the-box, well-suited to meeting the typical records management requirements for small organisations who may be implementing records management for the first time, and organisations where legislation does not heavily influence the management of records.

The DoD 5015 Resource Kit provides additional functionality, which can make an implementation compliant with the DoD 5015.2 standards and cover a large proportion of requirements with records management. The pack is fantastic in extending the built-in functionality with features that are fuller in records management functionality, allowing larger organisations or those more mature in records management more flexibility and possibilities, and easing some of the administrative overhead of records management.

However, as with any technology, the features and limitations must be well understood so that informed design decisions can be made. A proof-of-concept or pilot implementation would be advisable to model the ways of working using this solution. Microsoft Consulting Services have built experience in developing custom records management solutions to the requirements described in this article and can be engaged to assist with similar projects.

The DoD 5015 Resource Kit itself serves to show that SharePoint Server 2007 is an extensible platform, with powerful API’s and a wealth of help documentation and example solutions available in the public domain. If required, it is possible to build on this platform to adapt SharePoint to meet many different standards or ways of working. This allows solutions built using SharePoint Server 2007 to meet the stringent and complex requirements of records management in most large enterprises.

This post was written by Peter Reid and modified by James Kemp and Graham Tyler:

Peter Reid SharePoint Consultant Microsoft Consulting Services UK Click here to see my bio

Comments

• Anonymous
  October 27, 2009
  This is a comprehensive cover of the topic. I learned a lot from the 4 articles posted. Thank you!