Avoid High-End Licenses with DIY Adaptive Retention Policies

In my last article covering adaptive scopes for Microsoft 365 retention policies, I covered how to use Azure AD custom attributes to identify the set of accounts for retention processing. Apart from taking too long to evaluate and report the set of accounts found by the scope query, adaptive scopes work well and are a good way to make sure that Microsoft Purview retention processing applies to specific sets of accounts automatically without ongoing updates by an administrator.

The downside of adaptive scopes is their requirement for Office 365 E5 or Microsoft 365 E5 compliance licenses. Microsoft very much wants to maximize the amount of revenue generated from its cloud user base, perhaps as a result of slowing Office 365 user growth. The upshot is that any feature that includes automatic processing, especially in the compliance area, usually requires high-end licenses.

Static Retention Policies

Microsoft 365 retention policies with static scopes are available to Office 365 E3 tenants. A static scope means that administrators are responsible for managing the set of locations processed by a policy. A location can be a mailbox, site, group, team, or OneDrive account.

Sometimes, no management is needed, as in the case when a retention policy applies to all locations of a certain type (like all mailboxes or all sites). In other circumstances, making sure that the set of static locations specified in a policy is accurate can take considerable effort.

PowerShell and Azure Automation Deliver Updates for Adaptive Retention Policies

The thought occurred to me that it should be possible to build a form of do-it-yourself adaptive retention processing with a combination of a PowerShell script and Azure Automation. The combination replicates what adaptive scopes do without the need for high-end licenses. In this scenario, the PowerShell script:

  • Identifies the set of locations for the retention policy to process. Adaptive scopes query Azure AD to find this information. Our script can do the same.
  • Checks the locations already specified in the retention policy to calculate locations that should be removed and those to be added.
  • Updates the retention policy with the new locations.

Azure Automation runs the script on a scheduled basis to detect and apply changes to the retention policy. The recent addition of support for managed identities by V3.0 of the Exchange Online management module makes this process easier, even if the cmdlets that run against the compliance endpoint still require credentials to connect (Microsoft is working to fix this problem and a fix might be available when you read this article).

An Example Implementation

To prove that the principal works, I wrote an Azure Automation runbook (a version of a script to run in the Azure Automation environment). Adaptive scopes can process users and groups or SharePoint Online Sites. For this demonstration, the script processes user accounts by updating the Exchange Online mailboxes and OneDrive for Business accounts owned by the user accounts found by the query.

To begin, I created a new static retention policy with a static scope and defined one location for mailboxes and OneDrive accounts (Figure 1). Although this might seem like a strange step, it’s important to do this and not target the policy at all available locations, A policy with All shown for a location looks for and processes all locations. For instance, if you choose to process All mailboxes, the policy processes all user, shared, and inactive mailboxes.

A static retention policy with just one Exchange and one OneDrive location
Figure 1: A static retention policy with just one Exchange and one OneDrive location

Microsoft Purview treats retention policies differently when they process all available locations for a workload (like all mailboxes). The Set-RetentionCompliancePolicy cmdlet used by the script can only update locations if the policy is set up to process selected locations. In other words, the Set-RetentionCompliancePolicy cmdlet can’t switch a policy between all-in and selective processing.

Writing the Script

With a target policy to work against, we can go ahead and find some mailboxes and OneDrive accounts to process. The script does the following:

  • Connects to the Microsoft Graph PowerShell SDK with a managed identity to run the Get-MgOrganization cmdlet to find the default domain for the tenant. We need this to calculate the URLs for the target OneDrive for Business accounts.
  • Connect to Exchange Online with a managed identity.
  • Connect to the compliance endpoint using account credentials stored in Azure Key Vault. The need to use stored credentials will no longer be necessary when Microsoft upgrades the compliance cmdlets to use managed identities (or fixes certificate-based authentication).
  • Use the same account credentials to connect to the SharePoint Online management endpoint.
  • Find the set of target mailboxes. This example uses the same custom attribute to mark mailboxes as in the previous article.
  • Find the set of OneDrive for Business accounts in the tenant and store the owner name and URL in a hash table.
  • Loop through the set of mailboxes to check the user principal name for each mailbox against the OneDrive hash table. If a match is found, we know that the OneDrive account belongs to the mailbox owner, so we store it in a list.
  • Retrieve the set of current locations in the policy using the Get-RetentionCompliancePolicy cmdlet.
  • Check what mailboxes are not in the current location set and create a list of mailboxes to add to the policy. Do the same for OneDrive accounts.
  • Check if any mailboxes and OneDrive accounts are within the scope of the policy but shouldn’t be. If we find any, add them to a list of locations to remove from the policy.
  • Call the Set-RetentionCompliancePolicy cmdlet to add locations that should be in the policy and remove those that should not.
  • Call the Get-RetentionCompliancePolicy cmdlet to report the set of locations now covered by the policy.

I’ve never seen the code Microsoft uses to implement active scopes for retention policies, but I’m certain that much the same kind of processing occurs to figure out the set of locations that a policy should cover. I’m sure that the Microsoft code is more elegant and precise than mine (and not written in PowerShell), but the point is that the code works and does the same job as an adaptive scope does.

Be aware that a compliance policy supports a maximum of 1,000 individual mailboxes. More importantly, the limit is lower (100) for SharePoint Online sites and OneDrive for Business accounts. Testing and error handling for scripts used in production need to handle these limits. To get around the limits, you can create multiple policies with the same retention settings targeted at different sets of mailboxes, sites, and accounts.

Cybersecurity Risk Management for Active Directory

Discover how to prevent and recover from AD attacks through these Cybersecurity Risk Management Solutions.

Learn More!

Running the Script

You can download the script from GitHub and test it for yourself. The script runs in Azure Automation. Some changes to the authentication section of the code will make it run interactively.

Figure 2 shows the results of the script after running in the test pane of an Azure Automation session. The output shows object identifiers for the mailboxes to add and remove, while URLs are used for OneDrive for Business accounts. These are the values used to identify locations internally. If you wanted to, it would be easy to change the outputs to show the names of the accounts. However, as the script runs as a scheduled Azure Automation task, no one will probably look at the output unless something goes wrong.

Testing the script to update retention policy locations in Azure Automation
Figure 2: Testing the script to update retention policy locations in Azure Automation

Figure 3 shows the properties of the retention policy after the script runs. Where previously there was just one location noted for both Exchange Online and OneDrive for Business, now the policy covers 24 mailboxes and 23 accounts. The discrepancy between the two numbers is due to a new account that hasn’t yet signed into OneDrive.

The updated retention policy with locations added for processing
Figure 3: The updated retention policy with locations added for processing

The same principal explained here could extend to create an adaptive retention policy for Microsoft 365 Groups or SharePoint Online sites. It’s just a matter of PowerShell programming.

DIY is Not Always Possible

Recreating a Microsoft 365 feature with DIY code isn’t always possible. In this instance, it’s reasonably straightforward to create a custom form of adaptive retention processing. The combination of PowerShell and Azure Automation effectively replicates what Microsoft delivers in adaptive scopes.

About the Author

Tony Redmond

Tony Redmond has written thousands of articles about Microsoft technology since 1996. He is the lead author for the Office 365 for IT Pros eBook, the only book covering Office 365 that is updated monthly to keep pace with change in the cloud. Apart from contributing to Practical365.com, Tony also writes at Office365itpros.com to support the development of the eBook. He has been a Microsoft MVP since 2004.

Comments

  1. Joe

    Tony – Could you offer your expertise on how to account for a large environment that needs a policy applied to thousands of mailboxes? We developed our own script to dynamically create/assign retention policies in blocks of 1000 mailboxes but always looking for a “better way”! Thanks

    1. Avatar photo
      Tony Redmond

      This sounds like it’s a good use case for consideration for adaptive scopes. You could have one policy then instead of lots of individual policies broken down into blocks of 1,000 mailboxes. Cost is an issue (E5 licenses) but maybe it can be justified by the saving in maintenance for all those policies?

  2. Erica Toelle

    Hey Tony – could you please add a mention for the 1,000 user/100 site include limit on static policies? Also, it’s a character limit, not a hard limit, so keep that in mind for error handling. Thank you!

  3. Albert

    Really nice Tony!

Leave a Reply