Unable to perform Discovery search from EAC getting the below error

"An unknown error occurred on the search server. Please contact your administrator for assistance. The message from the search server is 'Could not locate Client Access Virtual Directory object with InternalUrl stamped for service. Last server checked: E2k13SRV"

Cause: Whenever discovery search is performed for the first time using EAC in Exchange 2013, it will check for the following 6 internal URLS in the background and it will cache it locally

Found URL: https://e15-cas.contoso.com/owa

Found URL: https://e15-cas.contoso.com/ecp

Found URL: https://e15-cas.contoso.com/EWS/Exchange.asmx

Found URL: https://e15-cas.contoso.com/Microsoft-Server-ActiveSync

Found URL: https://e15-cas.contoso.com/OAB

Found URL: https://e15-cas.contoso.com/mapi

[GlobalServiceUrls.DiscoverUrlsIfNeeded] Successfully found all needed VDirs

Resolution:

We need to make sure all the required virtual directories are present and internal URLs are published on those VDirs. In my case mapi virtual directory was missing and it failed while fetching the internal url of mapi virtual directory on the E15 CAS server.

Once Mapi virtual directory was created with InternalUrl, error  went away.

1. Event-based assistants These respond to events as they occur. For example, the resource booking assistant monitors requests to book rooms and lets users know whether the room is unavailable.

2. Throttle-based assistants These operate all the time to process data and are sub­ject to throttling to ensure that their workload does not affect the responsiveness of a server. The Managed Folder Assistant is the best known of these assistants.                        

Other-known assistants are the following:

1. Calendar assistant Checks mailboxes for incoming meeting requests and processes the requests according to mailbox settings

2. Resource booking assistant Processes meeting requests for room and equipment mailboxes

3. Scheduling assistant Scans attendee calendars for suitable meeting slots and sug­gests them to meeting organizers

4. Junk email options assistant Copies the safelist data from user mailboxes to their Active Directory accounts so the anti-spam agents can use it

5. Managed folder assistant Processes retention policies for managed folders and applies retention policy tags to items in mailboxes

Exchange uses a work cycle model to define how its assistants perform work while remain­ing active in the background to accomplish goals over a set period. For example, you might define that the Managed Folder Assistant must process every mailbox on a server at least once in 24 hours. Given the constraints determined in a work cycle, the assistant will work out how much work it has to do over the period and create an internal schedule to do the work on a phased basis so that a smooth and predictable demand is generated on the server rather than the peaks created by the older model. The assistant monitors system conditions on an ongoing basis to make any required adjustments to perform well. If the server is under high demand for a period due to user activity, the assistant can back off its work and then speed up when server load drops.

The Get-MailboxServer and Set-MailboxServer cmdlets retrieve and set work cycle informa­tion for assistant processes. First, find out what the current work cycles are for the assistants that run on a multirole server:

Get-MailboxServer –Identity ExServer2 | Format-List *WorkCycle*

The majority of the assistant processes process the data for which they are responsible at least once daily because the work cycle period is set to one day. In effect, the goal for the assistant is to process all relevant data for all mailboxes on the server in a 24-hour period, assuming that system resources allow it and the assistants are not throttled back to allow more important processes to proceed. The checkpoint for an assistant tells it how often it should check for new objects that need to be processed, such as looking to see whether new mailboxes have been added to the databases on a server. In most cases, the check­point is equivalent to the work cycle period. It makes sense to have the same work cycle settings applied to all Mailbox servers in the organization. This is easily done by running the Get-MailboxServer command to fetch a collection of all Mailbox servers and then using Set-MailboxServer to set the properties.

After you apply a retention policy to a mailbox, you can either wait for the next scheduled run of the MFA or start it manually so that the new policy is applied immediately. When the MFA runs, it performs the following tasks:

1. It applies the tags specified in retention policies to the mailboxes covered by these policies and stamps the items in the various folders covered by the policies with the appropriate tag name and expiration date.

2. If a policy defines a retention or expiry period for items, it stamps a Messaging Appli­cation Programming Interface (MAPI) property (ElcMoveDate) on the items, indicat­ing the date and time from which the retention period will start. A future run of the MFA can then use this date and time to calculate when to delete an item or mark it as expired.

3. It locates items in folders that are past their expiration date and takes whatever action is defined in the policy (delete, move to Archive).

    

The MFA works on a work cycle basis: it assesses the expected workload in terms of the number of mailboxes it has to process and then spreads out its processing across the com­plete window. For example, if 600 mailboxes are to be processed over three hours, the MFA creates its own internal schedule to process 200 mailboxes per hour or roughly three mailboxes per minute. In addition, a checkpoint is defined for the work cycle, at which time the MFA looks for new mailboxes that should be added to its list for processing. The default values for the work cycle and checkpoint are both 1 day, so the MFA attempts to process every mailbox in its list daily and checks for new mailboxes daily. The MFA logs event 9017

when it begins to process mailboxes, but the more important event to look for is 9025, logged when the MFA has had to skip a mailbox for some reason, such as when the mail­box is being moved to another database.

Overall, the work cycle mechanism makes more effective use of server resources in an easy and relaxed manner throughout the day and doesn’t create potential spikes in demand.

You might want to run the MFA immediately, perhaps to apply a policy to a group of users for the first time. Forcing immediate execution for a selected mailbox is useful when you start to apply policies to mailboxes and want to gauge the effect of the policy by examin­ing the contents of a known mailbox. This might be easier than asking users what hap­pened to items in their mailboxes. To force processing for a selected mailbox, specify its name with the –Identity parameter:

Start-ManagedFolderAssistant –Identity 'John'

Note: We should avoid running Start-managedFolderAssistant against bulk of users during the production hours, which may lead to some performance issues on the mailbox server.

The time required for the MFA to complete its run depends on the number of mailboxes and the number of items to which it has to apply retention policies. A run on a small server that hosts a few hundred mailboxes will complete in a couple of minutes unless the mail­boxes hold thousands of items. However, processing 6,000 mailboxes, each of which holds an average of 25,000 items, could take several hours, especially if the server is loaded with other tasks or the policies cause a heavy I/O load because many items are permanently removed or moved from primary to archive mailboxes. You should monitor the first runs of the MFA on a server to gauge the scope of the activity and how long a normal run takes to complete.

After the MFA has applied a new policy to a mailbox, the next time the user connects to the mailbox with a client that supports retention policies, user will see that retention tags are shown on items and the retention policy options are visible. Another important point that you should understand is that if you apply a retention policy that contains a default policy tag, the MFA stamps the default tag on every item in the mailbox. This action forces Outlook to download the complete contents of the mailbox the next time the client connects and synchronizes with Exchange. Such a massive synchronization has the poten­tial to flood a network and keep clients fully occupied for a long time. Including a default archive tag in a policy does not have the same effect because the MFA does not stamp every item with this tag.

User notified with retention policies

The first evidence users see that their mailbox has been assigned a retention policy is when retention information appears when they look at messages. This information is based on the tag stamped on an item by the MFA. Thirty days or so before an item’s retention period expires, Outlook and Outlook Web App begin to inform users that they might want to take action to preserve the item; otherwise, the MFA will process it again and delete it or move it into the archive, depending on the action required by policy. These warnings are visible when a message is opened or shown in the message preview shows how Outlook Web App advises that a message has five days before it expires as the result of the retention policy tag placed on the Inbox. The user now has the choice either to take action or to let the message expire, in which case the MFA will process whatever action is defined in the tag.

Outlook Web App display warning that an item will expire after certain specific deadline

Users have two options. First, they can move the item to a different folder and so remove it from the influence of the retention policy tag that applies to Inbox items. After it is moved, the item is governed by the default policy tag defined in the retention policy that applies to the mailbox, if one exists, or by an explicit tag that is applied to the folder and therefore inherited by all items that are added to the folder. If neither of these conditions exists, the item is left untagged and is therefore not subject to processing by the MFA

The second option is to apply a personal tag to the item. Users can choose from any of the personal tags defined in the retention policy applied to their mailbox by right-clicking an item and then selecting the personal tag to apply.

You won’t see the user interface for retention policies unless a policy is applied to your mailbox and the MFA has processed the mailbox to apply the retention policy. As part of this process, the MFA creates a hidden folder-associated item (FAI) in the user mailbox that clients use to populate the retention tag picker. If the policy is subsequently updated with a new retention or archive tag, the new tag will not be visible to clients until after the MFA next processes the mailbox.

After a personal tag has been applied to an item, the item is no longer subject to the provisions of either the folder policy or the default policy because an explicit tag always takes precedence over a tag placed on a folder. The personal tag also remains with the item if it is moved to another folder or into the personal archive. If users want to impose a different retention policy on the item, they must replace the existing tag with a new personal tag.

Hidden working of MFA

When the MFA first processes a mailbox, it creates a hidden FAI item of type IPM.Configuration.MRM in the Associated Contents table of the Inbox folder. The item stores MRM configuration in XML format in a PR_ROAMING_XMLSTREAM property. (PR = property; Roaming means that the information is available wherever the client connects, and XMLSTREAM indicates the type.) The MFA then updates the item any time a change occurs in the retention policy assigned to the mailbox, such as when a new personal tag is added to the policy. If a new retention policy is assigned, the MFA updates the item with details of that policy.

Exchange uses FAIs as a means to hold configuration and other data it needs to store in a mailbox but does not want to reveal to users when they run clients such as Outlook. The item the MFA creates holds details of the retention policy that has been assigned to the mailbox, including details of the retention tags the client needs to display in its user interface. If the item does not exist in the mailbox, a client remains unaware that a reten­tion policy is in force. This is why the MFA has to process a mailbox before the client user interface is populated with details of the policy. In addition to the tags provisioned through the retention policy, the item also holds information about any personal tags the user has selected for his own use through Outlook Web App Options, you cannot see this information with a normal client, but you can with the MFCMAPI utility by opening a mailbox that has been assigned a retention policy, opening the Inbox folder, and then opening the associated contents table before finally identify­ing the MRM item shows the details of the MRM configuration item as exposed by MFCMAPI, and you can clearly see details of the retention tags specified in the policy in the text box paste it in notepad and review the tags information.

The MFA uses a number of MAPI properties for mailbox folders and items during its pro­cessing. These properties are:

1. PR_START_DATE_ETC For a folder, this property holds the GUID of the retention tag that administers a folder. Every tag is assigned a GUID (stored logically in the GUID property and visible with EMS). This GUID is stamped into this property so the MFA knows which retention tag, action, and period applies to the folder. For an item, the property holds a composite value containing the default retention period plus the start time for the retention period. By adding the retention period to the start time, the MFA determines the expiry date.

2. PR_RETENTION_PERIOD This is the number of days items should be retained in a folder. When a personal tag is applied to an item, this property is set. However, if the property does not exist, the item (or subfolder) inherits the retention period from its parent folder.

3. PR_RETENTION_DATE This is the calculated date when an item’s retention period expires. Clients display this information to users. When clients work in online mode (such as in Outlook Web App), Exchange takes care of calculating this value, other­wise clients that work in offline mode perform the calculation.PR_RETENTION_FLAGS For a folder, this flag indicates whether the retention tag is inherited from the parent folder. If the user sets an explicit tag on a folder, the value is nonzero.

4. PR_POLICY_TAG This exists only for items and contains a binary encoded value pointing to the policy tag that governs the item.

5. PR_ARCHIVE_TAG This exists only for items and points to the archive tag that gov­erns an item.

6. PR_ARCHIVE_PERIOD This exists only when an item has been stamped with an explicit archive tag to contain the number of days in the archive retention period.

7. PR_ARCHIVE_DATE This contains the date when an item will be archived.

   You can view these properties through MFCMAPI shows the value of the PR_RETENTION_DATE property for an item.

   

   Viewing the PR_RETENTION_DATE property for an item

   Retention date calculation

   Date calculation is a very important part of the work the MFA does when it processes items. The MFA has to understand the date that should be used to calculate the age of the item (when the item first appears in the mailbox) and either the date when the item will expire or the date when the MFA has to take the action the retention policy requires (deleting the item or moving it into the archive).

   Given the nature of email, most items enter a mailbox when they are delivered in the Inbox or are sent and stored in Sent Items. Items often stay in these folders for much of their lifetime and, if they do stay, will probably come under the control of either a folder tag (if defined for the folder) or a default tag (that applies to untagged items in the mailbox). The exception, of course, occurs when a user explicitly applies a personal tag to an item in the Inbox or Sent Items.

   When the MFA runs, it examines items in the mailbox and determines what processing is required. Assume that an Inbox item exists that was delivered on 1 April 2013 and that a folder tag that requires items to be deleted and allow recovery after 30 days is applied to the Inbox. MFA stamps this item with a start date of 1 April 2013 by updating the PR_START_DATE_ETC property. It then calculates the expiration date by adding 30 days to the start date; 1 May 2013 is the result. This date is then stamped on the item by updating the PR_RETENTION_DATE property. On or after 1 May 2013, the MFA returns to the item and discovers that its retention period has expired. (If you view the item by using Outlook before the MFA returns, you’ll see the item marked as Expired.) When the MFA processes an expired item, it takes the retention action defined in the tag. In this case, it moves the item into the Recoverable Items folder, where it will remain until the retention period for the mailbox expires, at which time the MFA will remove the item permanently from the database.

   This is the simplest kind of processing the MFA is called on to perform, but it happens for a large percentage of items because many users leave messages in the Inbox and Sent Items folders. People who let items accumulate in these folders are often called pilers because they create piles of messages and then rely on client search facilities to locate specific items when required.

   Broadly speaking, the other kind of user behavior is represented by the filers, or people who move items from the Inbox and Sent Items into more appropriate folders in which the items form collections that represent important categories of work (or play) as viewed by the user. Or indeed the user attempts to keep the Inbox and other folders under some form of control by deleting unwanted items on a regular basis.

Managed Availability polls and analyzes hundreds of health metrics in Exchange 2013.  If something is found to be wrong, most of the time it will be fixed automatically.  But of course sometimes Managed Availability won’t be able to fix on its own.  In those cases, Managed Availability will escalate the issue to an administrator by means of event logging, and perhaps alerting if System Center Operations Manager is used with Exchange 2013. When an administrator needs to get involved and investigate the issue, they can begin by using the Get-HealthReport and Get-ServerHealth cmdlets.

I recently encountered an issue in Exchange 2013 Sp1 wherein server health and health report was showing “unknown” and all the server components state are showing “inactive”. Though health manager service was up and running it was still not able to pull the server health report.

The moment I try to bring the server component in active state it throws the Watson error

[PS] C:\Windows\system32>Set-ServerComponentState -State active –component ServerWideOffline –requestor functional

cmdlet Set-ServerComponentState at command pipeline position 1

Supply values for the following parameters:

Component: ServerWideOffline

Requester: functional

WARNING: An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object.

Object reference not set to an instance of an object.

    + CategoryInfo          : NotSpecified: (:) [Set-ServerComponentState], NullReferenceException

    + FullyQualifiedErrorId : System.NullReferenceException,Microsoft.Exchange.Management.SystemConfigurationTasks.Set

   ServerComponentState

    + PSComputerName        : HEM-E2013-01.extatic.net

While troubleshooting this problem found out that due to some reason Exchange 2013 SP1 setup previously failed while performing the action "BuildToBuildUpgrade" because of that health manager service was not able to pull the reports and were showing as unknown.

We checked the regedit and found watermark key and Buildtobuildupgrade key

 We took backup of  registry and deleted watermark key and  Buildtobuildupgrade key from MBX and HUB transport role

-- Ran command setup /prepareAD /IAcceptExchangeServerLicenseTerms

-- We then started setup /m:upgrade /IAcceptExchangeServerLicenseTerms

Upgrade completed successfully, post which search health report was showing healthy and all the server components where showing active. Hope the above post may help many Exchange folks facing the similar problem.

Cheers

-Anil

Cause

Before initiating the reseed process source server will try to establish a connection to the remote powershell HTTP protocol using WS-Management

Resolution

Remove the winhttp proxy

Command to see if there is a proxy: netsh winhttp show proxy

Command to remove the proxy: netsh winhttp reset proxy

More Information

Before initiating the reseed process source server will try to establish a connection to the remote host on HTTP protocol using WS-Management, which was failing because all the Http traffic was going through the proxy server.

Windows PowerShell Remote Commands

Host applications can use a PowerShell object to run commands that use WS-Management–based Windows PowerShell remoting to create a temporary or persistent connection to the remote computer. This includes establishing an interactive session with a single remote computer.

PowerShell remoting uses the HTTP protocol, it is affected by HTTP proxy settings. In enterprises that have proxy servers, users cannot access a Windows PowerShell remote computer directly.

Update-MailboxDatabaseCopy -Identity "EDB15\DAGMember" -DeleteExistingFiles

“Processing data from remote server failed with the following error message: The connection to the specified remote host was refused. Verify that the WS-Management service is running on the remote host and configured to listen for requests  on the correct port and HTTP URL. For more information, see the about_Remote_Troubleshooting Help topic.    + CategoryInfo          : OperationStopped: (System.Manageme...pressionSyncJob:PSInvokeExpressionSyncJob) [], PSRe   motingTransportException    + FullyQualifiedErrorId : JobFailure”

https://support.microsoft.com/kb/2027064

https://technet.microsoft.com/en-us/library/hh847850.aspx

- Anil

When creating or configuring an RPT, you can select one of the following actions such as 'Delete and Allow recovery', 'Permanently delete', 'Mark as Past Retention Limit' and 'Move to Archive'  to specify what retention action should be taken when a mailbox item reaches its retention age:

Move to Archive   This action moves a message to the user's archive mailbox. Tags that have this action applied are known as archive tags. Messages are moved to a folder in the archive mailbox that has the same name as the source folder in the user's primary mailbox. This allows users to easily locate messages in their archive mailbox. The Move to Archive action is available only for DPTs and personal tags. You can't create an RPT with the Move to Archive action. If the mailbox user doesn't have an archive mailbox, no action is taken.

RPT tag only works with 'Delete and Allow recovery' & 'Permanently delete' unless we are creating RPT for 'Recoverable Items' folder. 'Recoverable Items' is a hidden folder in the Non-IPM sub-tree. It contains the Deletions, Versions, Purges, and Audits sub-folders. Retention tags for this folder move items from the Recoverable Items folder in the user's primary mailbox to the Recoverable Items folder in the user's archive mailbox. You can assign only the Move To Archive retention action to tags for this folder.

If we try to create a Retention Policy tag with folder type ‘Inbox’ or any other default folders with ‘Move To Archive’ action. It won’t work and it is expected to fail.

But once the RPT is created with any other action like ‘Delete and Allow Recovery” or “Permanently delete”, then we can later modify the retention action with ‘Move to Archive’ action as mentioned in the below screenshots.

Though we could be able to set the ‘Move to Archive’ action on created RPT tag, but it won’t work since ‘move to archive’ action with RPT tag is not recommended, such misconfiguration of RPT tag may lead to other issues like archive database growth and archive mailbox dumpster bloating issues.

- Anil

Followed the below steps for assigning the permission to user “E15-User1” for creating distribution groups from EMS (Exchange management shell) and EAC (Exchange admin center)

Created a new child management role associated with “Distribution Groups” parent role

New-ManagementRole “DistributionGroupManagement” –Parent “Distribution Groups”

New-RoleGroup “Manage Distribution Groups” –Role ‘DistributionGroupManagement’  -Member  E15-User1

If we login to EAC using “E15-User1” credential and then  try to create a new distribution group, Organizational unit option is grayed out as can be seen in the print screen here.

We can see the Organizational Unit parameter is available in the newly created child role.

We are able to create the new distribution groups using the Exchange management shell, but if we login to EAC using “E15-User1” credential and then an Organizational unit option is not available.

Thought of using another management role “Security Group Creation and Membership” for the same purpose  

New-ManagementRole “ManagementSecurityGroups” –parent “Security Group Creation and Membership”

Get-ManagementRoleEntry “ManageSecurityGroups\*”

I can see the New-DistributionGroup command is inherited to ManageSecurityGroups from parent “Security Group Creation and Membership”

Organizational Unit parameter is available in newly created ‘ManageSecurityGroups’ child role

Set-RoleGroup “Manage Distribution Groups” –Role ‘ManageSecurityGroups

It still doesn’t work from EAC, but I can create distribution group using the Exchange management shell.

Resolution:

For fixing the problem  followed the below steps

The moment I add both the child roles ‘DistributionGroupManagement’   and ‘ManageSecurityGroups’ in the same role group “Manage Distribution Groups” organizational unit option is available in EAC.

Set-RoleGroup “Manage Distribution Groups” –Role “ManageSecurityGroups”, “DistributionGroupManagement”

It seems that we have to duplicate the permissions to make it work from EAC (Exchange admin center)

I have recently encounter this problem and It seems to be the known issue in E15 SP1. Till the time we have official fix for this issue, we can go with the workaround of duplicating the permissions.

- Anil