Moving from Static to Dynamic Locations

Microsoft 365 retention policies allow organizations to control how information is retained and removed for multiple workloads. Until recently, retention policies have been static. In other words, once you create a policy, the target locations for the policy remain unaltered until an administrator updates the policy to add or remove locations (mailbox, sites, and groups). For example, a new executive joins the company, and an administrator amends the retention policy applying to executives to add their mailbox and OneDrive for Business account.

Flagged for availability in August 2021, it’s taken Microsoft a little longer to make adaptive scopes for retention policies available in public preview. Now that the roadblocks are removed, tenants with Office 365 E5 licenses, or Microsoft 365 E5 compliance licenses, should see adaptive scopes available in the Information governance section of the Microsoft 365 compliance center. General availability will follow in due course, probably in early 2022.

What’s an Adaptive Scope?

An adaptive scope is a filter used to identify a subset of target locations. This isn’t breakthrough technology because Microsoft 365 already uses filters in features like Exchange Online dynamic distribution lists or dynamic Microsoft 365 groups (and teams). What’s different is that you can associate one or more adaptive scopes with a retention policy. When this happens, Microsoft 365 resolves the filters contained in adaptive scopes against Azure AD to identify the set of target locations for the policy and, critically, does so on an ongoing basis to include or exclude locations which meet or do not match the scope conditions. In other words, adaptive scopes are automatic location management for retention policies. They’re well suited for dealing with applying retention policies in scenarios like:

  • All Azure AD accounts belonging to a specific department or located in a certain country.
  • All SharePoint Online sites of a particular type.
  • All Microsoft 365 groups holding product information.

In fact, anywhere you can identify a user, group, or site with a custom property, you can use an adaptive scope to find and apply a retention policy to that location.

In this article we’ll discuss how to use adaptive scopes with retention policies for Users and Groups. Adaptive scopes for retention policies targeted at SharePoint Online and OneDrive for Business use the same fundamental approach but have their own quirks, so I cover them in a separate article.

Adaptive Scope Requirements

Before we go into the details of how adaptive scopes work, two important points need to be understood. First, you can’t update an existing retention policy which uses static locations to use an adaptive scope. Instead, you create a new retention policy with an adaptive scope which finds the same set of locations and use it to replace the old static policy. To ensure a smooth switchover, the old policy should remain in place until the new policy is active for all target locations.

The second point is that although static retention policies are covered by Office 365 E3, adaptive scopes require Office 365 E5 or Microsoft 365 E5 licenses. Or rather, any account which benefits from an adaptive scope requires an E5 license. This is to be expected because the licensing follows Microsoft’s normal approach whereby any automatic management within Microsoft 365 requires E5 licenses.

Scope Types

Adaptive scopes are managed through the Information governance section of the Microsoft 365 compliance center (or Records management if you’re already using that solution) and come in three types:

  • Users: Applies to mailboxes, OneDrive for Business accounts, and the compliance records captured in Exchange Online for Teams chats conversations and Yammer user messages (when Yammer networks are in Microsoft 365 native mode). This type of scope doesn’t filter for Teams and Yammer messages explicitly. They are processed along with other user mailbox content.
  • Sites: SharePoint Online sites and OneDrive for Business accounts (when identified by URL, name, or a refinable string).
  • Microsoft 365 Groups: This type of scope covers the Teams channel conversations and Yammer community messages captured as compliance records in group mailboxes.

Although you can’t mix location types (like sites and mailboxes) within an adaptive scope, you can include multiple adaptive scopes within a single retention policy to ensure that the same retention settings apply across multiple workloads. You can also include multiple adaptive scopes of the same type (for instance, three adaptive scopes for users) in a single retention policy.

Obviously, the mixing and matching of adaptive scopes within retention policies allows great flexibility. And because Microsoft 365 evaluates the filters against Azure AD on an ongoing basis, the limits which exist for locations included in non-org-wide static retention policies (like 100 SharePoint Online sites or 1,000 mailboxes) don’t exist. An adaptive scope could cover 500,000 users or 100,000 sites.

Creating a User Adaptive Scope

Because it works somewhat a dynamic Azure AD group, the user adaptive scope type is the easiest to create and understand. In the Information governance section of the Microsoft 365 compliance center, choose Adaptive scopes (preview) and create a new scope. Give the scope a name and description, and then move to define the scope type. At this point, you choose one from Users, Sites, and Microsoft 365 Groups. Choose Users.

The compliance center opens a query builder where you can define the properties and values used in the filter (Figure 1).

Creating a user adaptive scope to find Azure AD accounts
Figure 1: Creating a user adaptive scope to find Azure AD accounts

The query builder is straightforward (simple) and doesn’t need much description. The query builder supports four operators:

  • is equal to.
  • is not equal to.
  • starts with.
  • not starts with.

Remember that the query is against Azure AD, so not all mailbox properties can be included in a query. You can use the properties shown in Figure 1 (the fifteen single-value custom attributes are usable; the five multi-value custom attributes are not). If you want to check against multiple values for the same property, use multiple or tests (Figure 1 features a check for several variations of job title).

The advanced query builder is no more than a field to input a query in OPATH syntax. For example, to find users in France who belong to the IT department and job title is “Architect” and have “IC” in custom attribute 4, the query is:

Department -eq “IT” -and CountryOrRegion -eq “France” -and Title -eq “Architect” -and CustomAttribute4 -eq “IC”

Advanced queries can use a wider set of operators (such as like and notlike string comparisons). For instance, this query finds users in any department beginning with Sales whose job title contains Manager:

Department -like 'Sales*' -and Title -like '*Manager*'

User queries users can also find mailboxes to process based on the mailbox type or state. An example of a mailbox type query is to find shared or resource mailboxes. This query creates an adaptive scope to process shared mailboxes based on the IsShared flag used to mark these mailboxes. Remember that these mailboxes will need Office 365 E5 licenses if processed by retention policies based on an adaptive scope.

IsShared -eq 'True'

Inactive mailboxes are an example of a mailbox state. Mailboxes reach this state if they are within the scope of a retention hold or include items with retention labels when deleted. The query to find inactive mailboxes for an adaptive scope uses the IsInactiveMailbox flag.

IsInActiveMailbox -eq 'True'

Although advanced queries are more flexible, outside mailbox states and types, the Azure AD account properties used in queries remain limited to the set presented in the query builder. As an example, I tested with an OPATH query used in a dynamic group to find accounts licensed with Office 365 E5 (below). Azure AD can resolve the query to calculate group membership. But it doesn’t work for an adaptive scope:

user.assignedPlans -any (assignedPlan.servicePlanId -eq "2f442157-a11c-46b9-ae5b-6e39ff4e5849" -and assignedPlan.capabilityStatus -eq "Enabled")

Validating Scope Results

Microsoft already has a bunch of processes used for retention processing. According to sources, a new process resolved the queries for adaptive scopes to determine what locations the scope will find. The process runs daily but might need several runs to find every possible location. the query builder doesn’t resolve queries or have any way to show the expected results, so there’s no immediate way to prove that a scope will return the expected set of accounts, sites, or groups. Instead, you must wait before the background job processes the scope and makes its results available. According to Microsoft, this could take up to five days (it’s usually faster for users and groups), which makes the deployment of adaptive scopes quite frustrating at times. You don’t have to wait to see the set of calculated locations for a scope as you can use a scope immediately after creation in a retention policy. However, it’s obviously better to have confidence that a scope finds the right accounts before using it in a retention policy.

To see the set of target locations calculated for a scope, select the scope, and then Scope details in the fly-out pane (Figure 2). If you see “No data available,” it either means that the query didn’t find anything, or a background process has not resolved the query yet.

Checking the set of locations found by a user adaptive scope
Figure 2: Checking the set of locations found by a user adaptive scope

It’s also true that you might see incomplete results for a query. I have checked scope results after a day or so and found a couple of mailboxes and checked again after another day to find the full set I expected. Microsoft 365 is a very distributed environment spread across hundreds of thousands of servers, so it’s understandable that processing can take some time. Nevertheless, given that it’s possible for Azure AD to calculate dynamic group membership much quicker, it’s puzzling why adaptive scope results are so slow.

Once a set of scope results appear, you can compare the set of locations returned for a user adaptive scope by running the Get-Recipient cmdlet with the same query. For instance, the equivalent Get-Recipient command for the OPATH query:

CustomAttribute4 -eq “IC” -or Title -eq “Consultant” -or Title -eq “Architect” -or Title -eq “Senior Consultant”

is:

Get-Recipient -RecipientTypeDetails UserMailbox -Filter {CustomAttribute4 -eq "ic" -or (Title -eq "Architect" -or Title -eq "Consultant" -or Title -eq "Senior Consultant")} | ft displayname, customattribute4, title

DisplayName                 CustomAttribute4 Title
-----------                 ---------------- -----
Marc Vilas                  IC               Architect
Brian Weakliam (Operations) IC               Architect
Eoin Redmond (Ireland)                       Architect
Oisin Johnston              IC               Consultant
Ben James                                    Architect

If both Get-Recipient and the scope details agree (as they do based on the list in Figure 2), you know that the scope is working as expected.

Creating a Group Adaptive Scope

The process to create a group adaptive scope is like creating a group adaptive scope with the only difference being that a smaller set of properties is available to build the query. Groups don’t generally have assigned departments, cities, countries, and so on. In effect, although you can filter by display name or description, the custom properties will probably be the basis for most filters used in group adaptive scopes. Figure 3 shows the set of properties available.

Creating an adaptive scope for Microsoft 365 Groups
Figure 3: Creating an adaptive scope for Microsoft 365 Groups

Although you can’t see the filter criteria in Figure 3, I chose to look for groups with “France” set in CustomAttribute15. The equivalent Get-Recipient command is:

Get-Recipient -RecipientTypeDetails GroupMailbox -Filter {CustomAttribute15 -eq "France"} | Format-Table DIsplayName

DisplayName
-----------
Project Aloha (Sophia-Antipolis)
Nice Cote d'Azur
Tango (Caen)
French Food Lovers

Once again, we can check the groups the adaptive scope finds by using scope details.

Viewing the set of Microsoft 365 Groups found by an adaptive scope
Figure 4: Viewing the set of Microsoft 365 Groups found by an adaptive scope

Using Adaptive Scopes in Retention Policies

Using adaptive scopes to select target locations for retention policies is simple. Remember, you can’t upgrade a retention policy from static to adaptive, so you must create new policies to use adaptive scopes.

When you create a new retention policy, you can choose static or adaptive. The next step is to select at least one adaptive scope to use. You can combine multiple scopes in one retention policy, such as in the example where you want the same retention policy to apply to employees in multiple European countries.

In Figure 5, the retention policy has two adaptive scopes. When you select adaptive scopes, Microsoft 365 calculates what the valid types of location are. In this case, both are user adaptive scopes, so the target locations are those valid for this scope type.

Using Adaptive Scopes with Microsoft 365 Retention Policies for Users and Groups
Figure 5: Selecting adaptive scopes for a retention policy

There’s no difference in selecting retention settings (for example, delete all items after five years). All that’s changed is the dynamic nature of location targeting for the retention policy. When you save the new policy, Microsoft 365 publishes its details to the workloads, which take charge of enforcing the retention settings.

Scoping for the Enterprise

Adaptive scopes solve a problem for organizations where locations subject to retention change often. Enterprises with large and distributed employee populations will probably benefit most, and as those are the organizations most likely to have E5 licenses, they’ll be able to enjoy this new benefit with extra cost. For smaller organizations or those still unconvinced that the features included in E5 are worth the money, adaptive scopes are unlikely to be the point that makes them upgrade because other ways (manual or using PowerShell) exist to update retention policies.

I’d like to see more flexibility in the query builder and applying queries to generate adaptive scope results is too slow (as is the GUI in general). Admittingly, this is a V1.0 release, and hopefully Microsoft will smoothen the way adaptive scopes work over time.

About the Author

Tony Redmond

Tony Redmond has written thousands of articles about Microsoft technology since 1996. He is the lead author for the Office 365 for IT Pros eBook, the only book covering Office 365 that is updated monthly to keep pace with change in the cloud. Apart from contributing to Practical365.com, Tony also writes at Office365itpros.com to support the development of the eBook. He has been a Microsoft MVP since 2004.

Comments

  1. Matt

    I just can’t seem to get my scope to work ):

    I’m trying to target all users with @domain.com email address, but it won’t pull any data.

    Back to the drawing board.

    1. Avatar photo
      Tony Redmond

      I think that getting users on the basis of email addresses is difficult. Certainly, the checks against email addresses for dynamic distribution lists have always been a nightmare. According to an Exchange engineer, some extra indexing needs to be done to make email addresses a viable option. If you want to use email addresses, write some PowerShell to insert the email domain into one of the custom attributes and create the adaptive scope against that attribute.

      1. Matt

        I am not strong in powershell, I know some basic commands.

        So maybe I will try to target the “Department” instead. After creating a scope, should I start seeing results pretty soon, or can it take a while?

        Thanks!

        Matt

        1. Avatar photo
          Tony Redmond

          It can take a few days because the population depends on background processes.

  2. Paul

    Any insight on the backend “workload” process Microsoft utilizes for applying the Adaptive Scope Retention policy to users? We have a Adaptive Scope working with hundred of users and the Retention Policy won’t apply, even after the 7 day wait period. I got nothing from Microsoft on how admins could check the Retention Policy and what it is doing in the background to ensure it will apply. No errors or anything that would make me want to “retry” the distribution and kick another 7 day wait in process. Odd

    Thanks!

    1. Avatar photo
      Tony Redmond

      It’s a backend process that runs on its own schedule. Essentially, it reads Azure AD and figures out what accounts come within a scope and applies the policy to those accounts. It can take time for the retention workload to recognize that new users are available for a policy. I’d give it another week and if nothing happens, escalate with Microsoft. Although support can’t do anything to kick off the job, they can ask for an engineer to investigate.

      1. Paul

        Thought as much when it comes to the hidden back-end processes! Thanks so much Tony and love your work!

  3. Doug Maxfield

    New to using retention policies in Compliance (Purview). Have worked with retention policies in Exchange. Using retention policies in Purview, is there a limit to the number of users that can be added to a Retention Policy if a Static Policy is created?

  4. Andy

    Great write up. I am wondering if you can advise me with using adaptive scopes for leavers of an organisation. I am trying to set a retention policy for leavers that keeps all of their emails for a few months and then deletes them. I think adaptive scopes is the way to go, just not sure how.

    1. Avatar photo
      Tony Redmond

      I wouldn’t use adaptive scopes for this kind of scenario. Instead, create a retention policy to keep all items for however long you want, and assign the policy to leavers. When they leave, delete their account. Their mailbox will remain available in an inactive state for as long as the hold imposed by the retention policy lasts.

      If you want to use an adaptive scope, you’ll probably end up using a custom attribute (or set the department value to be something like “leaver”), but their mailboxes will be online and available (unless you hide them), and you’ll be consuming licenses because the accounts are still active.

Leave a Reply