Decrypting Conditional Access Complexity

Conditional Access is a powerful tool granting an easy way to bolster the security of an Office 365 tenant. The flexibility of Conditional Access means it can fit most organizational and security requirements easily. However, as with most things in technology, with flexibility there often comes complexity.

I see many tenants that have grown over time and as organizations grow and change (for example, adopting a hybrid working model), Conditional Access is updated to reflect the changing requirements. Unfortunately, as these changes are made, old policies, groups and assignments are not always tidied up. After a while, Conditional Access loses the flexibility it previously had because it is harder to predict the impact a change will have when there are a mess of policies that target different groups or apps.

There are tools available to help admins understand their Conditional Access policies better such as the Conditional Access Insights and Reporting Dashboard and the Conditional Access What-If tool. They are both fantastic tools, but they are somewhat limited in picking apart the detail of complex Conditional Access policy combinations.

To generate the information needed to decrypt Conditional Access policies in a practical manner, I created a PowerShell script (available on GitHub) to document not just Conditional Access policy settings, but also detail who is impacted by each policy and why.

Read more: Avoid needing this script by Planning Azure AD Conditional Access Policies appropriately

How does the conditional access assessment work?

The assessment script outputs an Excel Workbook with three tabs. The first tab (Figure 1), titled “Conditional Access by Column”, shows the detail of each Conditional Access policy and the settings for each. It also translates any object ID references to show the real names for objects such as users, groups, apps and roles. This is useful to get an idea of what each policy does.

Performing a Conditional Access Assessment with PowerShell
Figure 1: The Conditional Access tabs shows all Conditional Access policies and their settings

The second tab, “Conditional Access by Row” shows the Conditional Access data but with the columns and rows swapped. While it’s the same data, this format is useful when filtering for all policies with a specific condition for example.

The third tab, “User policies”, shows all users who are impacted (or exempted) by policies and why. Here, all users included in a policy are listed regardless of if they are included by nested group membership, role assignment or because a policy is assigned to “all users”. There are nine columns in this tab, each providing valuable detail:

  • User – Contains the User Principal Name of the user
  • User Excluded – If set to “Yes”, the user is excluded from this policy
  • Apps – The apps that this policy targets
  • Policy – The name of the policy
  • Policy State – The state of the policy (i.e. enabled, disabled, report-only)
  • Inherited from Group – If the policy is inherited because of group membership, this column shows the group name and ID (including nested group structures separated by ‘;’). If a user is impacted by a policy because of more than one group, there are additional line items added for each inclusion of that user / policy.
  • Exclusion inherited from Group – If the user is excluded from a policy due to group membership, this column shows the group name and ID (including nested group structures separated by ‘;’). Because any exclusion overrides all inclusions, each line item for a specific user / policy will include all exclusions (Separated by “&”).
  • Inherited from Role Assignment – If the user is assigned a policy due to role assignments, this column details which role the user is assigned that includes them in the policy.
  • Exclusion inherited from Role Assignment – If the user is excluded from a policy due to role assignment, this column details which role(s) the user is a member of that are excluding them from the policy (separated by “&”)

For example, Figure 2 shows the policies assigned to the admin user. This output shows the policies the user is assigned and how they have been assigned. We also see that the user is excluded from many of the policies.

Conditional Access
Figure 2: Sample output of “User policies” tab

Figure 3 shows that for many of these policies, the user is a direct exclusion. However, for two of the policies, the user is also a member of groups that are excluded. We also see the user is excluded from three policies due to role membership.

Conditional Access
Figure 3: Details on exclusions are also captured in the “User policies” tab

Top 10 Security Events to Monitor in Azure Active Directory and Office 365

Discover how native auditing tools can help — and how to overcome their shortcomings.

How to use the report

While the report provides a lot of detail, it needs to be sorted to provide value. The “Conditional Access” tab is useful as a reference for the individual settings of each policy but the “User policies” tab provides the most value.

Looking at an example, in my tenant there is a policy named “CA007: Require multi-factor authentication for risky sign-ins” which enforces MFA for users with medium or high sign-in risk. This policy is assigned to all users but does not seem to be working for some. To identify the issue and the full list of users impacted, I filter the “Policy” column for the policy I want to check and then filter for anywhere the user is excluded. This shows me that there are 13 users excluded from this policy and most of them have been excluded because they are members of a nested group structure (Figure 4).

Conditional Access
Figure 4: Verify user policy exclusions by filtering the Excel file

Another example, I want to delete the Azure AD group “NestedCAGroup”. By filtering on the “Exclusion inherited from Group”, I see that there are 10 users excluded from CA007 (Figure 5) because of this group and of those 10 users, for 9 of them this is the only reason they are excluded from the policy (with one user being directly excluded). If I delete this group now, these users will no longer be excluded.

Conditional Access
Figure 5: Filter for specific groups to see which policies they impact

There are many ways to filter the information depending on what you want to know. These are just two examples of how you can quickly identify potential issues with Conditional Access.

Preparing the script

An Azure AD App Registration is needed to run the script. The script by default uses delegated permissions but also supports application permissions with both Certificate-based Authentication and Client Secret. I have created a preparation script to set up the App Registration and Certificate. For more detail on the preparation script, check out the similar example in this Office 365 Migration Assessment script.

The script also needs access to c:\temp on the local machine to output the report. This report will be an Excel file in the format ConditionalAccessAssessment-<Date Time>.xlsx. Also required are the ImportExcel and MSAL.PS PowerShell Modules installed on the local machine – both are available from the PowerShell Gallery and installed using the following cmdlets:

Install-Module ImportExcel
Install-Module MSAL.PS

The preparation script (Prepare-ConditionalAccessDetailReport.ps1) is available in the same folder on GitHub and requires the AzureAD (or AzureADPreview) module to be installed. The output of the script (shown in Figure 6) is the Tenant ID, Client ID, and the syntax required to run the assessment script. By default this prepares an app registration with delegated permissions however the parameters UseClientSecret and -UseCertificate will provision application permissions and a client secret or certificate respectively.

Performing a Conditional Access Assessment with PowerShell
Figure 6: The preparation script creates the app registration in the tenant and output the details required

Manually setting up the app registration is also possible with the following permissions required:

  • RoleManagement.Read.All
  • Application.Read.All
  • Group.Read.All
  • Policy.Read.All
  • Policy.Read.ConditionalAccess
  • User.Read.All

Read more: This script supports using Certificate-based Authentication for unattended scenarios, to find out more about the benefits of using certificates over Client Secrets, check out this article on why using app secrets in production is a bad idea

Running the report

  • The Perform-ConditionalAccessDetailReport.ps1 script is run using the output from the preparation script. The script runs relatively quickly (about 5 minutes for my 3,000-user tenant with some complex policies present) and updates on progress as it goes. Due to the nature of nested groups, the script will warn about circular nesting (as shown in Figure 7) and only process each group once per policy. When running with delegated permissions you will be prompted to sign in to retrieve an access token. This requires an account with a minimum of the Global Reader role assigned.
Performing a Conditional Access Assessment with PowerShell
Figure 7: If the script detects circular nesting it will warn and only process each group once per policy

As the script uses the Microsoft Graph API to request information, I have also added the switch -ShowGraphCalls (Shown in Figure 8) which outputs the URI of each call to the MS Graph to the screen. I recommend using this flag to get an idea of where the information is coming from and perhaps leveraging this information to create your own scripts.

Performing a Conditional Access Assessment with PowerShell
Figure 8: Using the -ShowGraphCalls switch will output all Microsoft Graph URIs to the screen as the script progresses

Sharing and learning from scripts

As I mentioned in my article on the Office 365 Migration Assessment script, there are always new ways to look at problems and new perspectives are always welcome. This script gets a lot of useful information from the Microsoft Graph which provides endless possibilities for reporting and automating tasks.

If you aren’t familiar with the MS Graph API, I recommend using the -ShowGraphCalls switch to start learning and work towards building scripts that address your own requirements. Of course, once they are ready – share them online to help others too!

Top 10 Security Events to Monitor in Azure Active Directory and Office 365

Discover how native auditing tools can help — and how to overcome their shortcomings.

About the Author

Sean McAvinue

Sean McAvinue is a Microsoft MVP in Office Development and has been working with Microsoft Technologies for more than 10 years. As Modern Workplace Practice Lead at Ergo Group, he helps customers with planning, deploying and maximizing the many benefits of Microsoft 365 with a focus on security and automation. With a passion for creative problem solving, he enjoys developing solutions for business requirements by leveraging new technologies or by extending the built-in functionality with automation. Blogs frequently at https://seanmcavinue.net and loves sharing and collaborating with the community. To reach out to Sean, you can find him on Twitter at @sean_mcavinue

Comments

  1. John

    Security has a problem granting group.read.all scope because with application permission type this can allow reading of all group conversations. Would the script work with just groupmember.read.all scope instead ?

  2. Leandro

    It worked perfectly here. Congratulations.
    it is possible to see if a policy is not accessed in X days or months?

  3. Tom

    Hi Sean,
    Thanks for sharing the scripts. I’ve been battling the Insights and Reporting for my report only CA’s.
    Am I correct in saying that if a report-only CA is applied to a group, then the results for it’s application to a user in the group isn’t reflected correctly?
    I’m noticing that for enabled CA’s for a group are applying correctly to users; but my Report-Only ones are not, which is making it hard to judge what the impact would be prior to enablement.
    The same behaviour appears to be occurring with the ‘What If’ tool.

  4. Casper

    Gotcha – makes sense. Thanks 🙂

  5. Casper

    Any chance the script can be built to run from the current user context if the needed permissions are in place? As external consultant doing Azure AD security reviews we are only provided a Global Reader permission, and usually not allowed to install or register any apps or service principals.

    1. Sean McAvinue

      Graph API (including the new Graph SDK) module all require an app registration to he created. The permissions granted to the app are limited to “read” permissions so beyond creating and consenting to the app reg there’s really no need to have anything more than Global Reader.

      You could always modify it to use the Azure AD module but Graph is definitely the way forward for automation in Azure AD / O365, particularly if you want to future proof your code.

  6. craig ohler

    Hi Sean, Very nice script. I definitely intend to “borrow” some of the graph API functions as I’ve been struggling with that.

    I found one typo in your script that’s causing it to misreport the included locations. You just have it spelled as “includLocation” missing an e in a few places. Search/replace should fix it.

    Thanks and keep up the great work.

    1. Sean McAvinue

      Good spot Craig! I’ll update the code. Thanks!

Leave a Reply