Performing a Conditional Access Assessment with PowerShell

Decrypting Conditional Access Complexity

Conditional Access is a powerful tool granting an easy way to bolster the security of an Office 365 tenant. The flexibility of Conditional Access means it can fit most organizational and security requirements easily. However, as with most things in technology, with flexibility there often comes complexity.

I see many tenants that have grown over time and as organizations grow and change (for example, adopting a hybrid working model), Conditional Access is updated to reflect the changing requirements. Unfortunately, as these changes are made, old policies, groups and assignments are not always tidied up. After a while, Conditional Access loses the flexibility it previously had because it is harder to predict the impact a change will have when there are a mess of policies that target different groups or apps.

There are tools available to help admins understand their Conditional Access policies better such as the Conditional Access Insights and Reporting Dashboard and the Conditional Access What-If tool. They are both fantastic tools, but they are somewhat limited in picking apart the detail of complex Conditional Access policy combinations.

To generate the information needed to decrypt Conditional Access policies in a practical manner, I created a PowerShell script (available on GitHub) to document not just Conditional Access policy settings, but also detail who is impacted by each policy and why.

Read more: Avoid needing this script by Planning Azure AD Conditional Access Policies appropriately

How does the conditional access assessment work?

The assessment script outputs an Excel Workbook with three tabs. The first tab (Figure 1), titled “Conditional Access by Column”, shows the detail of each Conditional Access policy and the settings for each. It also translates any object ID references to show the real names for objects such as users, groups, apps and roles. This is useful to get an idea of what each policy does.

The second tab, “Conditional Access by Row” shows the Conditional Access data but with the columns and rows swapped. While it’s the same data, this format is useful when filtering for all policies with a specific condition for example.

The third tab, “User policies”, shows all users who are impacted (or exempted) by policies and why. Here, all users included in a policy are listed regardless of if they are included by nested group membership, role assignment or because a policy is assigned to “all users”. There are nine columns in this tab, each providing valuable detail:

• User – Contains the User Principal Name of the user
• User Excluded – If set to “Yes”, the user is excluded from this policy
• Apps – The apps that this policy targets
• Policy – The name of the policy
• Policy State – The state of the policy (i.e. enabled, disabled, report-only)
• Inherited from Group – If the policy is inherited because of group membership, this column shows the group name and ID (including nested group structures separated by ‘;’). If a user is impacted by a policy because of more than one group, there are additional line items added for each inclusion of that user / policy.
• Exclusion inherited from Group – If the user is excluded from a policy due to group membership, this column shows the group name and ID (including nested group structures separated by ‘;’). Because any exclusion overrides all inclusions, each line item for a specific user / policy will include all exclusions (Separated by “&”).
• Inherited from Role Assignment – If the user is assigned a policy due to role assignments, this column details which role the user is assigned that includes them in the policy.
• Exclusion inherited from Role Assignment – If the user is excluded from a policy due to role assignment, this column details which role(s) the user is a member of that are excluding them from the policy (separated by “&”)

For example, Figure 2 shows the policies assigned to the admin user. This output shows the policies the user is assigned and how they have been assigned. We also see that the user is excluded from many of the policies.

Figure 3 shows that for many of these policies, the user is a direct exclusion. However, for two of the policies, the user is also a member of groups that are excluded. We also see the user is excluded from three policies due to role membership.

How to use the report

While the report provides a lot of detail, it needs to be sorted to provide value. The “Conditional Access” tab is useful as a reference for the individual settings of each policy but the “User policies” tab provides the most value.

Looking at an example, in my tenant there is a policy named “CA007: Require multi-factor authentication for risky sign-ins” which enforces MFA for users with medium or high sign-in risk. This policy is assigned to all users but does not seem to be working for some. To identify the issue and the full list of users impacted, I filter the “Policy” column for the policy I want to check and then filter for anywhere the user is excluded. This shows me that there are 13 users excluded from this policy and most of them have been excluded because they are members of a nested group structure (Figure 4).

Another example, I want to delete the Azure AD group “NestedCAGroup”. By filtering on the “Exclusion inherited from Group”, I see that there are 10 users excluded from CA007 (Figure 5) because of this group and of those 10 users, for 9 of them this is the only reason they are excluded from the policy (with one user being directly excluded). If I delete this group now, these users will no longer be excluded.

There are many ways to filter the information depending on what you want to know. These are just two examples of how you can quickly identify potential issues with Conditional Access.

Preparing the script

An Azure AD App Registration is needed to run the script. The script by default uses delegated permissions but also supports application permissions with both Certificate-based Authentication and Client Secret. I have created a preparation script to set up the App Registration and Certificate. For more detail on the preparation script, check out the similar example in this Office 365 Migration Assessment script.

The script also needs access to c:\temp on the local machine to output the report. This report will be an Excel file in the format ConditionalAccessAssessment-<Date Time>.xlsx. Also required are the ImportExcel and MSAL.PS PowerShell Modules installed on the local machine – both are available from the PowerShell Gallery and installed using the following cmdlets:

Install-Module ImportExcel
Install-Module MSAL.PS

The preparation script (Prepare-ConditionalAccessDetailReport.ps1) is available in the same folder on GitHub and requires the AzureAD (or AzureADPreview) module to be installed. The output of the script (shown in Figure 6) is the Tenant ID, Client ID, and the syntax required to run the assessment script. By default this prepares an app registration with delegated permissions however the parameters –UseClientSecret and -UseCertificate will provision application permissions and a client secret or certificate respectively.

Manually setting up the app registration is also possible with the following permissions required:

• RoleManagement.Read.All
• Application.Read.All
• Group.Read.All
• Policy.Read.All
• Policy.Read.ConditionalAccess
• User.Read.All

Read more: This script supports using Certificate-based Authentication for unattended scenarios, to find out more about the benefits of using certificates over Client Secrets, check out this article on why using app secrets in production is a bad idea

Running the report

• The Perform-ConditionalAccessDetailReport.ps1 script is run using the output from the preparation script. The script runs relatively quickly (about 5 minutes for my 3,000-user tenant with some complex policies present) and updates on progress as it goes. Due to the nature of nested groups, the script will warn about circular nesting (as shown in Figure 7) and only process each group once per policy. When running with delegated permissions you will be prompted to sign in to retrieve an access token. This requires an account with a minimum of the Global Reader role assigned.

As the script uses the Microsoft Graph API to request information, I have also added the switch -ShowGraphCalls (Shown in Figure 8) which outputs the URI of each call to the MS Graph to the screen. I recommend using this flag to get an idea of where the information is coming from and perhaps leveraging this information to create your own scripts.

Sharing and learning from scripts

As I mentioned in my article on the Office 365 Migration Assessment script, there are always new ways to look at problems and new perspectives are always welcome. This script gets a lot of useful information from the Microsoft Graph which provides endless possibilities for reporting and automating tasks.

If you aren’t familiar with the MS Graph API, I recommend using the -ShowGraphCalls switch to start learning and work towards building scripts that address your own requirements. Of course, once they are ready – share them online to help others too!