Setting up and maintaining Conditional Access can be challenging, especially for administrators who aren’t deeply familiar with this security component in Microsoft Entra. As a Microsoft security consultant, I see, on a weekly basis, organizations struggle with implementing effective Conditional Access policies.
A real-life example I encountered during the last week of April 2025: while onboarding a new customer active in the financial sector to our Security Operations Center (SOC), I conducted a brief security audit to ensure they met the minimum security standard required for the onboarding. The customer’s administrators believed that all users were required to approve a Multi-Factor Authentication prompt with the Microsoft Authenticator app at sign-in. However, the audit revealed a critical security risk: internal non-privileged users were excluded from MFA due to a misconfiguration in the setup of Conditional Access. Without this audit, the issue might have gone unnoticed, leaving the organization with a false sense of security and an invalid DORA compliance attestation.
To help administrators establish a strong Conditional Access baseline, there are several tools and sources publicly available. Paul Robichaux already covered the most popular ones in this Practical 365 blog post.
Simulate Conditional Access Scenarios
As tools do exist to help implement a Conditional Access setup, there are also tools available to validate the effectiveness of Conditional Access policies. While Microsoft’s What If tool exists, it has notable limitations: it doesn’t support saving or running multiple simulations, doesn’t allow repeated testing, does not really provide strategic and technical insights that map to compliance controls, nor does it offer clear outcomes of the simulations.
Desired State is the Way to Go
A Conditional Access setup is essentially a technical translation of the organizational access policies that the business has defined and approved. These organizational access policies specify when MFA is required, if company-managed devices are required, what locations are allowed for accessing certain applications, etc. For IT to make sure that they comply with these access policies, they should be continuously checking if the Conditional Access setup still meets the business-defined access policies.
The most effective approach? Desired state testing.
Desired state testing compares the actual outcome of a sign-in attempt to the expected (desired) outcome. In the context of Conditional Access, this means simulating different sign-in scenarios and checking whether the results match the intended access policy. If they do, the configuration is compliant. If not, the desired state is not (yet) achieved. This might be due to a misconfiguration or policy drift in Conditional Access.
In practice, this involves writing tests for every possible access case. For example: verify if MFA is applied when needed, verify if a company device is required, verify if a password reset is forced on high user risk, verify if a sign-in is blocked outside a trusted location, etc.
Of course, it is quite impossible to write simulations for all possible access scenarios (for all users, all applications, all IP addresses, etc.). Simulations should only be created for relevant resources that are used in the Conditional Access policies.
How to check for Conditional Access Desired State?
In February 2022, I developed the Conditional Access Simulator as a closed-source paid web service, enabling organizations to simulate and validate their access policies across various scenarios and predict the impact of Conditional Access policy changes. As this tool gained high interest, it also inspired others.
In 2024, the open-source tool Maester was released, offering similar functionality. Merill Fernando, Fabian Bader, and Thomas Naunheim form the core development team of Maester, a Microsoft Security test automation framework. However, at the time of writing this blog post, over 70 different people have contributed to the project. An advantage of Maester is that an administrator can add their own tests, written in PowerShell, to their version of the Maester framework.
Say Yes to Automation
Covering all relevant access scenarios in a tenant requires a large number of simulations. Manually creating these tests for Conditional Access is not only time-consuming, but it also leaves gaps in forgetting certain access cases or misconfiguring a desired state.
Although I’m very proud of creating the first Conditional Access-simulating tool that allows for repeated testing and desired state, I’m also a big fan of open-source and community-driven solutions. That’s why I decided to build a plug-in for Maester to help automate the creation of Conditional Access tests across all relevant scenarios in a tenant.
The Conditional Access Validator
The goal of this plug-in is to make it easier for administrators to create desired state tests for their Conditional Access. These desired state tests often start from the current Conditional Access policies.
The Conditional Access Validator is an open-source PowerShell tool that automatically generates Maester tests for Conditional Access (Figure 1). Maester will simulate these tests on the Microsoft Graph What-If endpoint.
The Conditional Access Validator creates an HTML report with the following views:
Maester Test Code
The ‘Maester code’ view (Figure 1) shows the generated raw Maester tests. These tests are in the Pester format. Pester is the PowerShell test framework that Maester uses. This ‘Maester code’ view includes a button to copy the generated Maester code or save the code directly in a local Maester project.
Simulation List
The ‘List’ view (Figure 2) shows all generated tests in a readable and collapsible view. This makes it easier for technical and nontechnical people to review both the generated tests and the current Conditional Access setup.
Test Flow Chart
The ‘Flow Chart’ view (Figure 3) shows a visual representation of each simulation, and thus also a flow chart of the current Conditional Access setup. This is not a representation of the Conditional Access policies, but rather an overview of all relevant sign-in scenarios and their applied action. This flow chart can be useful to get an overview of how the Conditional Access policies behave or troubleshoot issues with policies.
User Impact Matrix
The ‘User Impact Matrix’ view already existed in the Conditional Access Blueprint framework, but is now integrated in the Conditional Access Validator. The User Impact Matrix provides an overview of which users are included or excluded for each Conditional Access policy. Because the output can be exported as a CSV, it allows for easy filtering. By filtering the columns of the CSV file, an administrator can answer questions like:
- What user accounts are excluded from MFA?
- What service accounts can only access our environment from IP XXX.XXX.XXX.XXX?
- What user accounts are not blocked from accessing Azure Management applications?
- What internal users can use old legacy authentication methods?
Persona Report
The main approach of the Conditional Access Blueprint is to use a static set of Conditional Access policies and only Entra groups as needed. These Entra groups represent personas (Figure 4), which are types of user accounts of a tenant. A persona can be an individual user or a group of users. Examples of personas are internal administrators, developer users, and Teams meeting room devices. With the concept of nested Entra groups, the personas form a hierarchy.
The ‘Persona report’ view (Figure 5) visually shows the included and excluded Entra groups of each Conditional Access policy. This allows for easy reviewing of the Conditional Access persona setup.
Decision Tree
To help administrators decide what Conditional Access controls to apply on each persona, there is the Conditional Access Persona Decision Tree. This decision tree is integrated into the Conditional Access Validator (Figure 6).
The concept of automatically generating Maester tests for Conditional Access, showing Conditional Access policies in a flow chart, and generating an overview of assigned policies per user are truly unique. Administrators can use these views to get insights into their Conditional Access policies, how they behave, and what misconfigurations might be in place.
An example web report of the Conditional Access Validator can be found here.
Getting the Conditional Access Validator Up and Running
To install the Conditional Access Validator, clone the GitHub Repository:
git clone https://github.com/jasperbaes/Conditional-Access-Validator.git
Navigate to the cloned project:
cd Conditional-Access-Validator
Install the official PowerShell module to authenticate and use the Microsoft Graph API:
Install-Module Microsoft.Graph
To let the Conditional Access Validator authenticate to the Microsoft Graph API, there are two options:
With an Entra user account
To connect with an Entra user account, execute the following command in the terminal to open a sign-in window. The required permissions for this user account are ‘Policy.Read.All’, ‘Directory.Read.All’, and ‘Application.Read.All’.
Connect-MgGraph
With an Entra Service Principal
An Entra Service Principal can be used to authenticate to the Microsoft Graph API without user credentials or Multi-Factor Authentication. This Service Principal needs the ‘Policy.Read.All’, ‘Directory.Read.All’ and ‘Application.Read.All’ permission. To connect with an Entra Service Principal, create the file settings.json in the root of the project and set the Service Principal credentials with the following JSON object:
{
"tenantID": "",
"clientID": "",
"clientSecret": ""
}
Information on how to get these credentials can be found in this Microsoft article.
Now, run the following command to start the script:
.\run.ps1
Once the script is finished (Figure 7), an HTML report will automatically open in the default browser.
Having Conditional Access policies that are in ‘report-only’ mode in the HTML report can simulate the impact of these policies without enabling them. To include Conditional Access policies that are in ‘report-only’ mode, use the following command:
.\run.ps1 -IncludeReportOnly
Conclusion
To validate the effectiveness of Entra Conditional Access, testing the desired state seems like the way to go. As this approach is not natively supported in the Entra Portal, the community has kicked in with tools like Maester. To help administrators with creating the desired state tests for Conditional Access, the Conditional Access Validator is a PowerShell tool that automatically generates Maester tests, based on the current state of the configured Conditional Access policies.
