Moving from Static to Dynamic Locations
Microsoft 365 retention policies allow organizations to control how information is retained and removed for multiple workloads. Until recently, retention policies have been static. In other words, once you create a policy, the target locations for the policy remain unaltered until an administrator updates the policy to add or remove locations (mailbox, sites, and groups). For example, a new executive joins the company, and an administrator amends the retention policy applying to executives to add their mailbox and OneDrive for Business account.
Flagged for availability in August 2021, it’s taken Microsoft a little longer to make adaptive scopes for retention policies available in public preview. Now that the roadblocks are removed, tenants with Office 365 E5 licenses, or Microsoft 365 E5 compliance licenses, should see adaptive scopes available in the Information governance section of the Microsoft 365 compliance center. General availability will follow in due course, probably in early 2022.
What’s an Adaptive Scope?
An adaptive scope is a filter used to identify a subset of target locations. This isn’t breakthrough technology because Microsoft 365 already uses filters in features like Exchange Online dynamic distribution lists or dynamic Microsoft 365 groups (and teams). What’s different is that you can associate one or more adaptive scopes with a retention policy. When this happens, Microsoft 365 resolves the filters contained in adaptive scopes against Azure AD to identify the set of target locations for the policy and, critically, does so on an ongoing basis to include or exclude locations which meet or do not match the scope conditions. In other words, adaptive scopes are automatic location management for retention policies. They’re well suited for dealing with applying retention policies in scenarios like:
- All Azure AD accounts belonging to a specific department or located in a certain country.
- All SharePoint Online sites of a particular type.
- All Microsoft 365 groups holding product information.
In fact, anywhere you can identify a user, group, or site with a custom property, you can use an adaptive scope to find and apply a retention policy to that location.
In this article we’ll discuss how to use adaptive scopes with retention policies for Users and Groups. Adaptive scopes for retention policies targeted at SharePoint Online and OneDrive for Business use the same fundamental approach but have their own quirks, so I cover them in a separate article.
Adaptive Scope Requirements
Before we go into the details of how adaptive scopes work, two important points need to be understood. First, you can’t update an existing retention policy which uses static locations to use an adaptive scope. Instead, you create a new retention policy with an adaptive scope which finds the same set of locations and use it to replace the old static policy. To ensure a smooth switchover, the old policy should remain in place until the new policy is active for all target locations.
The second point is that although static retention policies are covered by Office 365 E3, adaptive scopes require Office 365 E5 or Microsoft 365 E5 licenses. Or rather, any account which benefits from an adaptive scope requires an E5 license. This is to be expected because the licensing follows Microsoft’s normal approach whereby any automatic management within Microsoft 365 requires E5 licenses.
Scope Types
Adaptive scopes are managed through the Information governance section of the Microsoft 365 compliance center (or Records management if you’re already using that solution) and come in three types:
- Users: Applies to mailboxes, OneDrive for Business accounts, and the compliance records captured in Exchange Online for Teams chats conversations and Yammer user messages (when Yammer networks are in Microsoft 365 native mode). This type of scope doesn’t filter for Teams and Yammer messages explicitly. They are processed along with other user mailbox content.
- Sites: SharePoint Online sites and OneDrive for Business accounts (when identified by URL, name, or a refinable string).
- Microsoft 365 Groups: This type of scope covers the Teams channel conversations and Yammer community messages captured as compliance records in group mailboxes.
Although you can’t mix location types (like sites and mailboxes) within an adaptive scope, you can include multiple adaptive scopes within a single retention policy to ensure that the same retention settings apply across multiple workloads. You can also include multiple adaptive scopes of the same type (for instance, three adaptive scopes for users) in a single retention policy.
Obviously, the mixing and matching of adaptive scopes within retention policies allows great flexibility. And because Microsoft 365 evaluates the filters against Azure AD on an ongoing basis, the limits which exist for locations included in non-org-wide static retention policies (like 100 SharePoint Online sites or 1,000 mailboxes) don’t exist. An adaptive scope could cover 500,000 users or 100,000 sites.
Creating a User Adaptive Scope
Because it works somewhat a dynamic Azure AD group, the user adaptive scope type is the easiest to create and understand. In the Information governance section of the Microsoft 365 compliance center, choose Adaptive scopes (preview) and create a new scope. Give the scope a name and description, and then move to define the scope type. At this point, you choose one from Users, Sites, and Microsoft 365 Groups. Choose Users.
The compliance center opens a query builder where you can define the properties and values used in the filter (Figure 1).
The query builder is straightforward (simple) and doesn’t need much description. The query builder supports four operators:
- is equal to.
- is not equal to.
- starts with.
- not starts with.
Remember that the query is against Azure AD, so not all mailbox properties can be included in a query. You can use the properties shown in Figure 1 (the fifteen single-value custom attributes are usable; the five multi-value custom attributes are not). If you want to check against multiple values for the same property, use multiple or tests (Figure 1 features a check for several variations of job title).
The advanced query builder is no more than a field to input a query in OPATH syntax. For example, to find users in France who belong to the IT department and job title is “Architect” and have “IC” in custom attribute 4, the query is:
Department -eq “IT” -and CountryOrRegion -eq “France” -and Title -eq “Architect” -and CustomAttribute4 -eq “IC”
Advanced queries can use a wider set of operators (such as like and notlike string comparisons). For instance, this query finds users in any department beginning with Sales whose job title contains Manager:
Department -like 'Sales*' -and Title -like '*Manager*'
User queries users can also find mailboxes to process based on the mailbox type or state. An example of a mailbox type query is to find shared or resource mailboxes. This query creates an adaptive scope to process shared mailboxes based on the IsShared flag used to mark these mailboxes. Remember that these mailboxes will need Office 365 E5 licenses if processed by retention policies based on an adaptive scope.
IsShared -eq 'True'
Inactive mailboxes are an example of a mailbox state. Mailboxes reach this state if they are within the scope of a retention hold or include items with retention labels when deleted. The query to find inactive mailboxes for an adaptive scope uses the IsInactiveMailbox flag.
IsInActiveMailbox -eq 'True'
Although advanced queries are more flexible, outside mailbox states and types, the Azure AD account properties used in queries remain limited to the set presented in the query builder. As an example, I tested with an OPATH query used in a dynamic group to find accounts licensed with Office 365 E5 (below). Azure AD can resolve the query to calculate group membership. But it doesn’t work for an adaptive scope:
user.assignedPlans -any (assignedPlan.servicePlanId -eq "2f442157-a11c-46b9-ae5b-6e39ff4e5849" -and assignedPlan.capabilityStatus -eq "Enabled")
Validating Scope Results
Microsoft already has a bunch of processes used for retention processing. According to sources, a new process resolved the queries for adaptive scopes to determine what locations the scope will find. The process runs daily but might need several runs to find every possible location. the query builder doesn’t resolve queries or have any way to show the expected results, so there’s no immediate way to prove that a scope will return the expected set of accounts, sites, or groups. Instead, you must wait before the background job processes the scope and makes its results available. According to Microsoft, this could take up to five days (it’s usually faster for users and groups), which makes the deployment of adaptive scopes quite frustrating at times. You don’t have to wait to see the set of calculated locations for a scope as you can use a scope immediately after creation in a retention policy. However, it’s obviously better to have confidence that a scope finds the right accounts before using it in a retention policy.
To see the set of target locations calculated for a scope, select the scope, and then Scope details in the fly-out pane (Figure 2). If you see “No data available,” it either means that the query didn’t find anything, or a background process has not resolved the query yet.
It’s also true that you might see incomplete results for a query. I have checked scope results after a day or so and found a couple of mailboxes and checked again after another day to find the full set I expected. Microsoft 365 is a very distributed environment spread across hundreds of thousands of servers, so it’s understandable that processing can take some time. Nevertheless, given that it’s possible for Azure AD to calculate dynamic group membership much quicker, it’s puzzling why adaptive scope results are so slow.
Once a set of scope results appear, you can compare the set of locations returned for a user adaptive scope by running the Get-Recipient cmdlet with the same query. For instance, the equivalent Get-Recipient command for the OPATH query:
CustomAttribute4 -eq “IC” -or Title -eq “Consultant” -or Title -eq “Architect” -or Title -eq “Senior Consultant”
is:
Get-Recipient -RecipientTypeDetails UserMailbox -Filter {CustomAttribute4 -eq "ic" -or (Title -eq "Architect" -or Title -eq "Consultant" -or Title -eq "Senior Consultant")} | ft displayname, customattribute4, title
DisplayName CustomAttribute4 Title
----------- ---------------- -----
Marc Vilas IC Architect
Brian Weakliam (Operations) IC Architect
Eoin Redmond (Ireland) Architect
Oisin Johnston IC Consultant
Ben James Architect
If both Get-Recipient and the scope details agree (as they do based on the list in Figure 2), you know that the scope is working as expected.
Creating a Group Adaptive Scope
The process to create a group adaptive scope is like creating a group adaptive scope with the only difference being that a smaller set of properties is available to build the query. Groups don’t generally have assigned departments, cities, countries, and so on. In effect, although you can filter by display name or description, the custom properties will probably be the basis for most filters used in group adaptive scopes. Figure 3 shows the set of properties available.
Although you can’t see the filter criteria in Figure 3, I chose to look for groups with “France” set in CustomAttribute15. The equivalent Get-Recipient command is:
Get-Recipient -RecipientTypeDetails GroupMailbox -Filter {CustomAttribute15 -eq "France"} | Format-Table DIsplayName
DisplayName
-----------
Project Aloha (Sophia-Antipolis)
Nice Cote d'Azur
Tango (Caen)
French Food Lovers
Once again, we can check the groups the adaptive scope finds by using scope details.
Using Adaptive Scopes in Retention Policies
Using adaptive scopes to select target locations for retention policies is simple. Remember, you can’t upgrade a retention policy from static to adaptive, so you must create new policies to use adaptive scopes.
When you create a new retention policy, you can choose static or adaptive. The next step is to select at least one adaptive scope to use. You can combine multiple scopes in one retention policy, such as in the example where you want the same retention policy to apply to employees in multiple European countries.
In Figure 5, the retention policy has two adaptive scopes. When you select adaptive scopes, Microsoft 365 calculates what the valid types of location are. In this case, both are user adaptive scopes, so the target locations are those valid for this scope type.
There’s no difference in selecting retention settings (for example, delete all items after five years). All that’s changed is the dynamic nature of location targeting for the retention policy. When you save the new policy, Microsoft 365 publishes its details to the workloads, which take charge of enforcing the retention settings.
Scoping for the Enterprise
Adaptive scopes solve a problem for organizations where locations subject to retention change often. Enterprises with large and distributed employee populations will probably benefit most, and as those are the organizations most likely to have E5 licenses, they’ll be able to enjoy this new benefit with extra cost. For smaller organizations or those still unconvinced that the features included in E5 are worth the money, adaptive scopes are unlikely to be the point that makes them upgrade because other ways (manual or using PowerShell) exist to update retention policies.
I’d like to see more flexibility in the query builder and applying queries to generate adaptive scope results is too slow (as is the GUI in general). Admittingly, this is a V1.0 release, and hopefully Microsoft will smoothen the way adaptive scopes work over time.
The Real Person!
The Real Person!
Good afternoon, I need to have the AdaptiveScope updated with PowerShell and with all the best will in the world I am not getting the correct Conditions set.
Below is an example, just to test, but this is running into an error all the time:
//
# Ensure the necessary module is imported
#Import-Module ExchangeOnlineManagement
# Connect to Exchange Online
#Connect-ExchangeOnline
# Connect to the Security & Compliance Center
# Connect-IPPSSession
# Define the simple conditions correctly
$simpleConditions = @(
@{
“Conditions” = @(
@{
“Value” = “US”;
“Operator” = “Equals”;
“Name” = “CountryOrRegion”
},
@{
“Value” = “Canada”;
“Operator” = “Equals”;
“Name” = “CountryOrRegion”
}
);
“Conjunction” = “Or”
},
@{
“Value” = “Finance”;
“Operator” = “Equals”;
“Name” = “Department”
}
)
# Ensure $simpleConditions is an array and its elements are hashtables
if ($simpleConditions -is [System.Array] -and $simpleConditions[0] -is [System.Collections.Hashtable]) {
Write-Host “Conditions are correctly defined as an array of hashtables.”
} else {
Write-Host “Conditions are not correctly defined. Please check the format.”
}
# Define the filter conditions
$simpleFilterConditions = @{
“Conditions” = $simpleConditions;
“Conjunction” = “And”
}
# Debugging output to check the structure of $simpleFilterConditions
Write-Host “Simple Filter Conditions Structure:”
$simpleFilterConditions | Format-List
# Check if the New-AdaptiveScope cmdlet is available
if (Get-Command -Name New-AdaptiveScope -ErrorAction SilentlyContinue) {
# Create the adaptive scope
New-AdaptiveScope -Name “Test Scope” -LocationType User -FilterConditions $simpleFilterConditions
} else {
Write-Host “The New-AdaptiveScope cmdlet is not available in this session.”
}
//
The error is:
//
Write-ErrorMessage : |Microsoft.Exchange.Management.UnifiedPolicy.InvalidFilterConditionsException|Invalid filter conditions. Context: at depth 1. Error: Unexpected value type for key ‘Conditions’. Expected type: System.Object[].
At C:\Users\pvli\AppData\Local\Temp\tmpEXO_2bb42rbt.lg0\tmpEXO_2bb42rbt.lg0.psm1:1205 char:13
+ Write-ErrorMessage $ErrorObject
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (Microsoft.Offic…ewAdaptiveScope:String) [New-AdaptiveScope], InvalidFilterConditionsException
+ FullyQualifiedErrorId : [TimeStamp=Fri, 20 Sep 2024 13:30:34 GMT],Write-ErrorMessage//
We have added a number of checks to get this correct, but I am out of options. Could you assist or guide me to someone who is really understanding the correct filter conditions.
I think there’s a problem with the cmdlet. Not even the example in https://learn.microsoft.com/en-us/powershell/module/exchange/new-adaptivescope?view=exchange-ps works.
I’ve reported the issue to Microsoft. You can help by creating a support incident. The more pressure that’s put on the engineering group, the sooner this will be fixed.
Hi Tony,
Great article. When creating adaptive scopes based on a group, can it be a security group or must it be a Microsoft 365? not sure we want to create M365 groups for this as it will create a corresponding SharePoint site. Hoping we can use security groups as these have some of the same attributes such as “display name”
Cheers.
Freddie.
Nope. Adaptive scopes only support Microsoft 365 Groups. https://learn.microsoft.com/purview/purview-adaptive-scopes?WT.mc_id=M365-MVP-9501#configure-adaptive-scopes
The Real Person!
The Real Person!
Tony, so based on testing anf this article you cannot target a group with a query? In the case i am looking at is using to create scope with member of, if I use m365 group i cant select location if my target is Onedrive
Maybe I need to wait more time. On indexing , how do we check , i been told by MS that for many oitems it may be indexing issue?
If you want to access OneDrive, use the users adaptive scope.
The Real Person!
The Real Person!
Thanks so users in adaptive scope with query ? Type
User
Query summary
MemberofGroup -eq ‘OneDriveAdaptiveScopeTest’
As I understand it, you want to have an adaptive scope for the OneDrive accounts of users who are members of a group.
One way to solve the problem is to use an Exchange custom attribute (adaptive scopes support CustomAttribute1 through CustomAttribute15) as the basis for the adaptive scope. You would need to make sure that the attribute is set to a specific value for each member of the group. You could create a dynamic group based on the attributes or use PowerShell to look for user accounts with the attribute set, and then updating the group membership so that it includes everyone with the attribute.
I wrote the solution up in https://office365itpros.com/2024/10/15/adaptive-scope-group-members/
The Real Person!
The Real Person!
Thanks, I did get it to Pull my accunt based on a custom attribute defined in the Adaptive Scope. What I am not understanding is the role of the group since in Purview it does the allow the targeting via group?
The group makes it easy to check that the adaptive scope targets the correct accounts. If the two memberships match up, you know everything is working. And you can use the group for other administrative purposes, such as the membership of a team to share documents. But you don’t need to use the group if you don’t want to.
The Real Person!
The Real Person!
Hello, I have been told that can use an administrative unit to target Onedrive locations, i tried and 0 luck, waited weeks. Was targeting onedrive using an adaptive scope using adminstrative unit. MS engineer via case said this would work, do you know if this is possible? https://learn.microsoft.com/en-us/purview/purview-admin-units
Hi Tony,
Great article. When creating adaptive scopes based on a group, can it be a security group or must it be a Microsoft 365? not sure we want to create M365 groups for this as it will create a corresponding SharePoint site. Hoping we can use security groups as these have some of the same attributes such as “display name”.
Cheers.
Freddie.
Hi Tony,
Thanks for your effort.
One question, I couldn’t find anything to query adaptive scope members with PowerShell or Graph API.
Do you know if there is a way other then the Compliance Portal to get a list of actual members?
Nope. There’s no public API to retrieve the scope members.
Hi Tony! Great content! We are in need of creating adaptive scope DR policies for SharePoint alone. What would be the license requirement for this.
Office 365 E5 or above.
Excellent article and thanks for sharing the details. I am trying to get the user details using below adaptive queries, but the results are not displayed. please provide your comment.
Scope 1: MemberOfGroup -ne ‘<>’
Scope 2: MemberOfGroup -ne ‘<>’
How long have you waited? It can take days before Microsoft 365 processes an adaptive scope to resolve results.
is there any option to pull users using user principal domain?
No. Not unless you store the domain in one of the 15 custom attributes available for user accounts, which is what I would do.
I just can’t seem to get my scope to work ):
I’m trying to target all users with @domain.com email address, but it won’t pull any data.
Back to the drawing board.
I think that getting users on the basis of email addresses is difficult. Certainly, the checks against email addresses for dynamic distribution lists have always been a nightmare. According to an Exchange engineer, some extra indexing needs to be done to make email addresses a viable option. If you want to use email addresses, write some PowerShell to insert the email domain into one of the custom attributes and create the adaptive scope against that attribute.
I am not strong in powershell, I know some basic commands.
So maybe I will try to target the “Department” instead. After creating a scope, should I start seeing results pretty soon, or can it take a while?
Thanks!
Matt
It can take a few days because the population depends on background processes.
Any insight on the backend “workload” process Microsoft utilizes for applying the Adaptive Scope Retention policy to users? We have a Adaptive Scope working with hundred of users and the Retention Policy won’t apply, even after the 7 day wait period. I got nothing from Microsoft on how admins could check the Retention Policy and what it is doing in the background to ensure it will apply. No errors or anything that would make me want to “retry” the distribution and kick another 7 day wait in process. Odd
Thanks!
It’s a backend process that runs on its own schedule. Essentially, it reads Azure AD and figures out what accounts come within a scope and applies the policy to those accounts. It can take time for the retention workload to recognize that new users are available for a policy. I’d give it another week and if nothing happens, escalate with Microsoft. Although support can’t do anything to kick off the job, they can ask for an engineer to investigate.
Thought as much when it comes to the hidden back-end processes! Thanks so much Tony and love your work!
New to using retention policies in Compliance (Purview). Have worked with retention policies in Exchange. Using retention policies in Purview, is there a limit to the number of users that can be added to a Retention Policy if a Static Policy is created?
Yes. The limit is 1,000. See https://docs.microsoft.com/en-us/microsoft-365/compliance/retention-limits?view=o365-worldwide
Great write up. I am wondering if you can advise me with using adaptive scopes for leavers of an organisation. I am trying to set a retention policy for leavers that keeps all of their emails for a few months and then deletes them. I think adaptive scopes is the way to go, just not sure how.
I wouldn’t use adaptive scopes for this kind of scenario. Instead, create a retention policy to keep all items for however long you want, and assign the policy to leavers. When they leave, delete their account. Their mailbox will remain available in an inactive state for as long as the hold imposed by the retention policy lasts.
If you want to use an adaptive scope, you’ll probably end up using a custom attribute (or set the department value to be something like “leaver”), but their mailboxes will be online and available (unless you hide them), and you’ll be consuming licenses because the accounts are still active.
I have an adaptive scope that lists inactive mailboxes that will have retained content in them due to the mailboxes containing messages with retention labels applied to them when the associated account was deleted. The scope includes IsInactiveMailbox -eq ‘True’. Since the license was removed when the associated AD account was deleted, does that mean that the mailboxes won’t need a license when they subsequently appear in the adaptive scope? (I’ve tested that they do appear in the scope) I want to associate a retention policy to the scope that will delete non-held content – eg, delete after 60 days instead of the default delete after 3 years. That way non-record content can be removed right after the account is deleted instead of waiting for the default retention policy to remove them.
Adaptive scopes only find objects, they don’t have anything to do with licenses. If a deleted mailbox shows up in an adaptive scope, it will remain there until the last hold is released on the mailbox. Adding a new retention policy to remove ‘normal’ content won’t really speed things up as the mailbox will remain in place until the last hold lapses. It will, however, remove any non-tagged content, which is what I think you want to do.