Understanding the Exchange Mailbox Folder Assistant

Mailbox Folder Assistant One of Many Background Assistants Used by Exchange

Exchange Server and Exchange Online include several mailbox assistants (background jobs) to perform automated processing for mailbox data. The two basic kinds are:

• Event-based assistants These respond to events as they occur. For example, the resource booking assistant monitors requests to book rooms and lets users know whether the room is unavailable.
• Throttle-based assistants These run on an ongoing basis to process data and are subject to throttling to ensure that their workload does not affect server responsiveness.

Some examples of the different assistants include:

• Calendar assistant: Checks mailboxes for incoming meeting requests and processes the requests according to mailbox settings.
• Resource booking assistant: Processes meeting requests for room and equipment mailboxes.
• Scheduling assistant: Scans attendee calendars for suitable meeting slots and suggests them to the meeting organizer.

The focus of this article is the Managed Folder Assistant (MFA), a good example of a throttle-based assistant. This process applies Exchange mailbox retention policies (and in the cloud, Microsoft 365 retention policies) to Exchange mailbox items. Another Microsoft 365 background assistant handles retention processing for other workloads. The MFA originally appeared in Exchange 2010 and is present in every version since, including Exchange Online. In all cases, the MFA does the same thing – evaluate retention policies and make sure that their settings apply to mailboxes.

Managing the MFA Work Cycle

Exchange uses a work cycle model to define how its assistants process data over a set period. The work cycles are predefined and cannot be altered for Exchange Online, but you can for Exchange Server. For instance, you could define that the Managed Folder Assistant must process every mailbox on a server at least once in 24 hours. Given the constraints determined in a work cycle, the assistant will work out how much work it has to do over the period and create an internal schedule to do the work on a phased basis so that a smooth and predictable demand is generated on the server rather than the peaks created by the older model. The assistant monitors system conditions on an ongoing basis to make any required adjustments to perform well. If the server is under high demand for a period due to user activity, the assistant can back off its work and then speed up when the server load drops.

For Exchange Server, the Get-MailboxServer cmdlet retrieves work cycle information for assistant processes. Here’s how to discover the current work cycles for the assistants that run on a multirole server:

Get-MailboxServer –Identity ExServer2 | Format-List *WorkCycle*

Most Exchange Server assistants process the data for which they are responsible at least once daily because the default work cycle period is set to one day. In effect, the goal for the assistant is to process all relevant data for all mailboxes on the server in a 24-hour period, assuming that system resources allow it and the assistants are not throttled back to allow more important processes to proceed. The checkpoint for an assistant tells it how often it should check for new objects that need to be processed, such as looking to see whether new mailboxes have been added to the databases on a server. In most cases, the check­point is equivalent to the work cycle period.

It makes sense to have the same work cycle settings applied to all Mailbox servers in the organization. This is easily done by running the Get-MailboxServer command to fetch a collection of all Mailbox servers. For Exchange Server versions than Exchange 2016, use Set-MailboxServer to update work cycle schedules. For Exchange 2016 and later, use the New-SettingOverride cmdlet as described in Microsoft’s documentation.

When the MFA starts to process mailboxes it assesses the expected workload in terms of the number of mailboxes to process and then spreads out its processing across the work cycle window. For example, if 600 mailboxes are to be processed over three hours, the MFA creates its own internal schedule to process 200 mailboxes per hour or roughly three mailboxes per minute. In addition, a checkpoint is defined for the work cycle, at which time the MFA looks for new mailboxes that should be added to its list for processing. The default values for the work cycle and checkpoint are both 1 day, so the MFA attempts to process every mailbox in its list daily and checks for new mailboxes daily. For Exchange Server, MFA logs event 9017 when it begins to process mailboxes, but the more important event to look for is 9025, logged when the MFA has had to skip a mailbox for some reason, such as when the mailbox is being moved to another database.

Overall, the work cycle mechanism makes more effective use of server resources in an easy and relaxed manner throughout the day and doesn’t create potential spikes in demand.

How the MFA Applies Retention Policies

After an administrator includes a mailbox within the scope of a mailbox retention or Microsoft 365 retention policy, the MFA will process the mailbox during its next workcycle. The sole caveat is that the mailbox must hold at least 10 MB of content. That used to be quite a big amount of data and is intended to show that the mailbox is in active use. Today, a single message could be larger than 10 MB, but the same test applies.

For Exchange Server, MFA takes the folder retention tags defined in retention policies to mailboxes and stamps mailbox items in the target folders with the appropriate tag and expiration date. For Exchange Online, MFA combines retention tags from mailbox retention policies and Microsoft 365 retention policies. However, Microsoft 365 retention policies operate on a mailbox basis and don’t support folder tags. Microsoft would like cloud customers to move to Microsoft 365 retention policies and the lack of folder tags and default archive tags are two challenges to overcome.

If a retention policy or retention tag defines an expiry period for items, MFA stamps a Messaging Application Programming Interface (MAPI) property (ElcMoveDate) on items, indicating the date and time when the retention period starts. A future run of the MFA can then use this date and time to calculate when to delete an item or mark it as expired.

MFA then finds items in folders that are past their expiration date and takes whatever action is defined in the tag or retention policy (permanently delete, delete and allow recovery, or archive).

You might want to run the MFA immediately, perhaps to apply a policy to a group of users for the first time. Forcing immediate execution for a selected mailbox is useful when you start to apply policies to mailboxes and want to gauge the effect of the policy by examining the contents of a known mailbox. This might be easier than asking users what happened to items in their mailboxes (especially if you’ve made a mistake with the policy and just removed half the items from the mailbox). To force processing for a selected mailbox, run the Start-ManagedFolderAssistant cmdlet and specify the mailbox name with the –Identity parameter:

Start-ManagedFolderAssistant –Identity 'Akers, Kim'

Don’t expect results immediately. Exchange will schedule MFA to process the target mailbox when resources allow.

Normal Processing Done by the Managed Folder Assistant

The time required for MFA to complete a run depends on the number of mailboxes and the number of items that it processes. A run on a small server hosting a few hundred mailboxes will complete in a couple of minutes unless the mailboxes hold tens of thousands of items. However, processing 7,000 mailboxes, each of which holds an average of 20,000 items, could take several hours, especially if the server is loaded with other tasks or the policies cause a heavy I/O load because many items are permanently removed or moved from primary to archive mailboxes. You should monitor the first runs of the MFA on a server to gauge the scope of the activity and how long a normal run takes to complete. Equipped with this information, you can quickly assess whether future runs are progressing as expected.

To check details of MFA processing for a mailbox, use the Export-MailboxDiagnosticsLogs cmdlet. Here’s an example:

$Log = Export-MailboxDiagnosticLogs -Identity James.Ryan -ExtendedProperties
$xml = [xml]($Log.MailboxLog)
$xml.Properties.MailboxTable.Property | Where-Object {$_.Name -like "ELC*"}

Name                                    Value
----                                    -----
ElcAssistantLock                        1
ElcLastRunTotalProcessingTime           240374
ElcLastRunSubAssistantProcessingTime    206663
ElcLastRunUpdatedFolderCount            69
ElcLastRunTaggedFolderCount             0
ElcLastRunUpdatedItemCount              329
ElcLastRunTaggedWithArchiveItemCount    0
ElcLastRunTaggedWithExpiryItemCount     271
ElcLastRunDeletedFromRootItemCount      2
ElcLastRunDeletedFromDumpsterItemCount  0
ElcLastRunArchivedFromRootItemCount     43
ElcLastRunArchivedFromDumpsterItemCount 0
ELCLastSuccessTimestamp                 03/11/2023 09:08:40
ElcLastRunSkippedNoTagItemCount         0
ElcLastRunSkippedWithTagItemCount       0
ElcLastRunSkippedNotExcludedItemCount   0
ElcFaiSaveStatus                        SaveSucceeded
ElcFaiDeleteStatus                      DeleteNotAttempted

Table 1 lists the most interesting properties reported by the diagnostic logs.

This article explains how to examine ELC properties for mailboxes to discover the last time they were processed by MFA.

For Exchange Online, you don’t need to worry about scheduling MFA because Exchange Online takes care of this kind of background processing. The goal for Exchange Online is for MFA to process mailboxes at least once weekly. In reality, MFA usually processes mailboxes at least a couple of times weekly.

After the MFA applies a retention policy to a mailbox, the next time the user connects to the mailbox with a client (like Outlook) that supports retention policies, they see retention tags shown on items. Retention-enabled clients also make it possible for users to interact with retention policy settings.

User Interaction with Retention Policies

The first evidence users see that their mailbox has been assigned a retention policy is when retention information is visible for messages. This information is based on the tag stamped on an item by the MFA. Retention information is visible when a message is opened. Figure 1 shows how OWA displays the name of the assigned retention tag and when the item expires. Upon expiry, MFA runs the action defined in the retention tag to remove or archive the item. To keep the item, users can apply a different retention tag.

Users don’t see information about retention policies unless their mailbox is subject to one or more retention policies and after MFA processes the mailbox. As part of this process, the MFA creates a hidden folder-associated item (FAI) to hold data used to populate the retention tag picker. In the cloud, the picker includes Exchange personal retention tags and Microsoft 365 retention labels published to the mailbox through Microsoft 365 retention policies. If the retention policies assigned to the mailbox are subsequently updated with a new retention or archive tag, the new tag remains invisible to clients until after the MFA next processes the mailbox.

After an item receives a personal tag, the item is no longer subject to the provisions of either the folder tag (if it exists) or the default tag (if included in the policy) because a personal tag always takes precedence. The personal tag remains with the item if it moves to another folder or into the user’s archive mailbox. If users want to impose different retention settings on the item, they must replace the tag with a new personal tag.

The Hidden Working of the Managed Folder Assistant

When MFA first processes a mailbox, it creates a hidden FAI item of message class IPM.Configuration.MRM in the Associated Contents table of the Inbox folder. The item stores the MRM configuration in XML format in the PR_ROAMING_XMLSTREAM property. You cannot see this information with a normal client, but you can with the MFCMAPI utility by opening a mailbox that has a retention policy, opening the Inbox folder, and then opening the associated contents table before finally finding the MRM item. Figure 2 shows the details of the FAI as exposed by the MFCMAPI utility, and you can clearly see details of some retention tags.

Exchange uses FAIs as a means to hold configuration and other data it needs to store in a mailbox but does not want to reveal to users when they run clients such as Outlook. The item the MFA creates holds details of the retention policy that has been assigned to the mailbox, including details of the retention tags the client should display in its user interface. MFA updates the FAI any time a change occurs in the retention policy assigned to the mailbox, such as when a new personal tag is added to the policy. If a new retention policy is assigned, the MFA updates the item with details of that policy.

If the FAI does not exist in the mailbox, clients remain unaware that a retention policy is in force. This is why the MFA has to process a mailbox before client user interfaces populate the details of the policy. In addition to the tags provisioned through retention policies, the item also holds information about any personal tags the user has selected for his own use through OWA Options.

Mailbox Properties Used in Retention Processing

The MFA uses several MAPI properties for mailbox folders and items during its processing. These properties are:

• PR_START_DATE_ETC: If present for a folder, this property holds the GUID of the retention tag governing the folder. Every tag is assigned a GUID (stored logically in the GUID property and visible with PowerShell). This GUID is stamped into this property so the MFA knows which retention tag, action, and period applies to the folder. For an item, the property holds a composite value containing the default retention period plus the start time for the retention period. By adding the retention period to the start time, the MFA determines the expiry date.
• PR_RETENTION_PERIOD: This is the number of days to retain an item. When a personal tag is applied to an item, this property is set. However, if the property does not exist, the item (or subfolder) inherits the retention period from its parent folder or the default delete tag.
• PR_RETENTION_DATE: This is the calculated date when an item’s retention period expires. Clients display this information to users. When clients work in online mode (such as in OWA), Exchange calculates this value, otherwise, the client performs the calculation.
• PR_RETENTION_FLAGS: For a folder, this flag indicates whether the retention tag is inherited from the parent folder. If the user sets an explicit tag on a folder, the value is nonzero.
• PR_POLICY_TAG: This exists only for items and contains a binary encoded value pointing to the retention tag that governs the item.
• PR_ARCHIVE_TAG: This exists only for items and points to the archive tag that governs an item.
• PR_ARCHIVE_PERIOD: This exists only when an item has been stamped with an explicit archive tag to contain the number of days in the archive retention period.
• PR_ARCHIVE_DATE: This contains the date when an item will be archived.

You can view these properties for mailbox items using the MFCMAPI utility (works for both Exchange Server and Exchange Online). Figure 3 shows the value of the PR_RETENTION_DATE property for a mailbox item.

Retention Date Calculation

Date calculation is a very important part of the work the MFA does when it processes items. The MFA has to understand the date that should be used to calculate the age of the item (when the item first appears in the mailbox) and either the date when the item will expire or the date when the MFA has to take the action the retention policy requires (deleting the item or moving it into the archive).

Given the nature of email, most items enter a mailbox when they are delivered to the Inbox or are sent and stored in Sent Items. Items often stay in these folders for much of their lifetime and, if they do stay, will probably come under the control of either a folder tag (if defined for the folder) or a default tag (that applies to untagged items in the mailbox). The exception, of course, occurs when a user explicitly applies a personal tag to an item in the Inbox or Sent Items.

When the MFA runs, it examines items in the mailbox and determines what processing is required. Assume that an Inbox item exists that was delivered on 1 April 2023 and that a folder tag that requires items to be deleted and allow recovery after 30 days is applied to the Inbox. MFA stamps this item with a start date of 1 April 2023 by updating the PR_START_DATE_ETC property. It then calculates the expiration date by adding 30 days to the start date; 1 May 2023 is the result. This date is then stamped on the item by updating the PR_RETENTION_DATE property. On or after 1 May 2023, the MFA returns to the item and discovers that its retention period has expired. When the MFA processes an expired item, it takes the retention action defined in the tag. In this case, it moves the item into the Recoverable Items folder, where it will remain until the retention period for the mailbox expires, at which time the MFA will remove the item permanently from the database.

This is the simplest kind of retention date processing the MFA performs, but it happens for a large percentage of items because many users leave messages in the Inbox and Sent Items folders. People who let items accumulate in these folders are often called pilers because they create piles of messages and then rely on client search facilities to locate specific items when required.

Broadly speaking, the other kind of user behavior is represented by the filers, or people who move items from the Inbox and Sent Items into more appropriate folders in which the items form collections that represent important categories of work (or play) as viewed by the user. Or indeed the user attempts to keep the Inbox and other folders under some form of control by deleting unwanted items on a regular basis.

Mailbox Retention Processing is Very Logical

Above all, the method used by the MFA to apply retention to mailbox items is very logical. The retention period and retention action come from the retention tag; the MFA simply does what it’s told to do and deletes items or moves them to the archive when the retention period lapses.