Decrypting Conditional Access Complexity
Conditional Access is a powerful tool granting an easy way to bolster the security of an Office 365 tenant. The flexibility of Conditional Access means it can fit most organizational and security requirements easily. However, as with most things in technology, with flexibility there often comes complexity.
I see many tenants that have grown over time and as organizations grow and change (for example, adopting a hybrid working model), Conditional Access is updated to reflect the changing requirements. Unfortunately, as these changes are made, old policies, groups and assignments are not always tidied up. After a while, Conditional Access loses the flexibility it previously had because it is harder to predict the impact a change will have when there are a mess of policies that target different groups or apps.
There are tools available to help admins understand their Conditional Access policies better such as the Conditional Access Insights and Reporting Dashboard and the Conditional Access What-If tool. They are both fantastic tools, but they are somewhat limited in picking apart the detail of complex Conditional Access policy combinations.
To generate the information needed to decrypt Conditional Access policies in a practical manner, I created a PowerShell script (available on GitHub) to document not just Conditional Access policy settings, but also detail who is impacted by each policy and why.
Read more: Avoid needing this script by Planning Azure AD Conditional Access Policies appropriately
How does the conditional access assessment work?
The assessment script outputs an Excel Workbook with three tabs. The first tab (Figure 1), titled “Conditional Access by Column”, shows the detail of each Conditional Access policy and the settings for each. It also translates any object ID references to show the real names for objects such as users, groups, apps and roles. This is useful to get an idea of what each policy does.
![Performing a Conditional Access Assessment with PowerShell](https://s40823.pcdn.co/wp-content/uploads/2022/06/Fig1-1024x353.png)
The second tab, “Conditional Access by Row” shows the Conditional Access data but with the columns and rows swapped. While it’s the same data, this format is useful when filtering for all policies with a specific condition for example.
The third tab, “User policies”, shows all users who are impacted (or exempted) by policies and why. Here, all users included in a policy are listed regardless of if they are included by nested group membership, role assignment or because a policy is assigned to “all users”. There are nine columns in this tab, each providing valuable detail:
- User – Contains the User Principal Name of the user
- User Excluded – If set to “Yes”, the user is excluded from this policy
- Apps – The apps that this policy targets
- Policy – The name of the policy
- Policy State – The state of the policy (i.e. enabled, disabled, report-only)
- Inherited from Group – If the policy is inherited because of group membership, this column shows the group name and ID (including nested group structures separated by ‘;’). If a user is impacted by a policy because of more than one group, there are additional line items added for each inclusion of that user / policy.
- Exclusion inherited from Group – If the user is excluded from a policy due to group membership, this column shows the group name and ID (including nested group structures separated by ‘;’). Because any exclusion overrides all inclusions, each line item for a specific user / policy will include all exclusions (Separated by “&”).
- Inherited from Role Assignment – If the user is assigned a policy due to role assignments, this column details which role the user is assigned that includes them in the policy.
- Exclusion inherited from Role Assignment – If the user is excluded from a policy due to role assignment, this column details which role(s) the user is a member of that are excluding them from the policy (separated by “&”)
For example, Figure 2 shows the policies assigned to the admin user. This output shows the policies the user is assigned and how they have been assigned. We also see that the user is excluded from many of the policies.
![Conditional Access](https://s40823.pcdn.co/wp-content/uploads/2022/06/Fig2-1024x222.png)
Figure 3 shows that for many of these policies, the user is a direct exclusion. However, for two of the policies, the user is also a member of groups that are excluded. We also see the user is excluded from three policies due to role membership.
![Conditional Access](https://s40823.pcdn.co/wp-content/uploads/2022/06/Fig3-1024x234.png)
Top 10 Security Events to Monitor in Azure Active Directory and Office 365
Discover how native auditing tools can help — and how to overcome their shortcomings.
How to use the report
While the report provides a lot of detail, it needs to be sorted to provide value. The “Conditional Access” tab is useful as a reference for the individual settings of each policy but the “User policies” tab provides the most value.
Looking at an example, in my tenant there is a policy named “CA007: Require multi-factor authentication for risky sign-ins” which enforces MFA for users with medium or high sign-in risk. This policy is assigned to all users but does not seem to be working for some. To identify the issue and the full list of users impacted, I filter the “Policy” column for the policy I want to check and then filter for anywhere the user is excluded. This shows me that there are 13 users excluded from this policy and most of them have been excluded because they are members of a nested group structure (Figure 4).
![Conditional Access](https://s40823.pcdn.co/wp-content/uploads/2022/06/Fig4-1024x162.png)
Another example, I want to delete the Azure AD group “NestedCAGroup”. By filtering on the “Exclusion inherited from Group”, I see that there are 10 users excluded from CA007 (Figure 5) because of this group and of those 10 users, for 9 of them this is the only reason they are excluded from the policy (with one user being directly excluded). If I delete this group now, these users will no longer be excluded.
![Conditional Access](https://s40823.pcdn.co/wp-content/uploads/2022/06/Fig5-1024x129.png)
There are many ways to filter the information depending on what you want to know. These are just two examples of how you can quickly identify potential issues with Conditional Access.
Preparing the script
An Azure AD App Registration is needed to run the script. The script by default uses delegated permissions but also supports application permissions with both Certificate-based Authentication and Client Secret. I have created a preparation script to set up the App Registration and Certificate. For more detail on the preparation script, check out the similar example in this Office 365 Migration Assessment script.
The script also needs access to c:\temp on the local machine to output the report. This report will be an Excel file in the format ConditionalAccessAssessment-<Date Time>.xlsx. Also required are the ImportExcel and MSAL.PS PowerShell Modules installed on the local machine – both are available from the PowerShell Gallery and installed using the following cmdlets:
Install-Module ImportExcel Install-Module MSAL.PS
The preparation script (Prepare-ConditionalAccessDetailReport.ps1) is available in the same folder on GitHub and requires the AzureAD (or AzureADPreview) module to be installed. The output of the script (shown in Figure 6) is the Tenant ID, Client ID, and the syntax required to run the assessment script. By default this prepares an app registration with delegated permissions however the parameters –UseClientSecret and -UseCertificate will provision application permissions and a client secret or certificate respectively.
![Performing a Conditional Access Assessment with PowerShell](https://s40823.pcdn.co/wp-content/uploads/2022/06/Fig6.png)
Manually setting up the app registration is also possible with the following permissions required:
- RoleManagement.Read.All
- Application.Read.All
- Group.Read.All
- Policy.Read.All
- Policy.Read.ConditionalAccess
- User.Read.All
Running the report
- The Perform-ConditionalAccessDetailReport.ps1 script is run using the output from the preparation script. The script runs relatively quickly (about 5 minutes for my 3,000-user tenant with some complex policies present) and updates on progress as it goes. Due to the nature of nested groups, the script will warn about circular nesting (as shown in Figure 7) and only process each group once per policy. When running with delegated permissions you will be prompted to sign in to retrieve an access token. This requires an account with a minimum of the Global Reader role assigned.
![Performing a Conditional Access Assessment with PowerShell](https://s40823.pcdn.co/wp-content/uploads/2022/06/Fig7.png)
As the script uses the Microsoft Graph API to request information, I have also added the switch -ShowGraphCalls (Shown in Figure 8) which outputs the URI of each call to the MS Graph to the screen. I recommend using this flag to get an idea of where the information is coming from and perhaps leveraging this information to create your own scripts.
![Performing a Conditional Access Assessment with PowerShell](https://s40823.pcdn.co/wp-content/uploads/2022/06/Fig8.png)
Sharing and learning from scripts
As I mentioned in my article on the Office 365 Migration Assessment script, there are always new ways to look at problems and new perspectives are always welcome. This script gets a lot of useful information from the Microsoft Graph which provides endless possibilities for reporting and automating tasks.
If you aren’t familiar with the MS Graph API, I recommend using the -ShowGraphCalls switch to start learning and work towards building scripts that address your own requirements. Of course, once they are ready – share them online to help others too!
Top 10 Security Events to Monitor in Azure Active Directory and Office 365
Discover how native auditing tools can help — and how to overcome their shortcomings.
The Real Person!
The Real Person!
Hi Sean,
Thank you for the script.
Is there way to get the below info-
All sign-ins (interactive, non-interactive, all, etc), whether or not they would’ve failed for a particular CA on Report-Only.
– This is marked via “Report-Only; Failure”.
– Need a two week to one month snapshot of all of these failures to review:
– – Who failed?
– – What failed (product, sign-in, etc)?
– – Origin IP
Your help is appreciated.
Security has a problem granting group.read.all scope because with application permission type this can allow reading of all group conversations. Would the script work with just groupmember.read.all scope instead ?
It worked perfectly here. Congratulations.
it is possible to see if a policy is not accessed in X days or months?
Hi Sean,
Thanks for sharing the scripts. I’ve been battling the Insights and Reporting for my report only CA’s.
Am I correct in saying that if a report-only CA is applied to a group, then the results for it’s application to a user in the group isn’t reflected correctly?
I’m noticing that for enabled CA’s for a group are applying correctly to users; but my Report-Only ones are not, which is making it hard to judge what the impact would be prior to enablement.
The same behaviour appears to be occurring with the ‘What If’ tool.
Gotcha – makes sense. Thanks 🙂
Any chance the script can be built to run from the current user context if the needed permissions are in place? As external consultant doing Azure AD security reviews we are only provided a Global Reader permission, and usually not allowed to install or register any apps or service principals.
Graph API (including the new Graph SDK) module all require an app registration to he created. The permissions granted to the app are limited to “read” permissions so beyond creating and consenting to the app reg there’s really no need to have anything more than Global Reader.
You could always modify it to use the Azure AD module but Graph is definitely the way forward for automation in Azure AD / O365, particularly if you want to future proof your code.
Hi Sean, Very nice script. I definitely intend to “borrow” some of the graph API functions as I’ve been struggling with that.
I found one typo in your script that’s causing it to misreport the included locations. You just have it spelled as “includLocation” missing an e in a few places. Search/replace should fix it.
Thanks and keep up the great work.
Good spot Craig! I’ll update the code. Thanks!