Monitor Risk on an Ongoing Basis or Prepare for Surprises
The Microsoft Entra ID Protection features included with Microsoft Entra ID P2 subscriptions are a very powerful way to combat identity-based attacks. ID Protection allows admins to use the risk status (as determined by Microsoft) generated in real-time for sign-ins and through behavioral analysis by Microsoft Entra to automatically block sign-ins or force password resets if connections exceed a configurable risk threshold.
Blocking sign-ins automatically is great for tenants with the right licenses but doesn’t help everybody. Putting aside any conversations about gating security features behind top-tier licensing SKUs, all is not lost for organizations without premium subscriptions. While premium licensing is required for some of the enhanced detail and automated remediation features, the risky users, sign-ins, and risk detection reports are available to tenants with Entra ID P1 or Entra ID Free (Entra ID Free tenants do not get risk detection reports) in a limited capacity as detailed in Microsoft documentation.
Risk events in Microsoft Entra can often be overlooked by busy administrators, particularly in smaller environments where there may not be a dedicated security operations team responsible for monitoring risk. As with many things in Microsoft 365, a little automation can go a long way to bridge some of these gaps. In this article, I explain how to create an Azure Automation runbook (code available on GitHub) to report risk detections in a tenant and email the report to administrators daily.
Getting Started
The first requirement is an Azure Automation account. Azure Automation is a great way to manage automated reporting such as monitoring Unified Audit log events and performing repetitive tasks such as provisioning Microsoft Teams based on templates.
To keep things simple, the script uses a managed identity to connect to the Microsoft Graph SDK. As the script uses a managed identity, the Automation account should be associated with the tenant Entra ID directory the script will run against.
Next, the service principal of the automation account needs the following Graph permissions to run the script:
- “IdentityRiskyUser.Read.All” – This permission is required to retrieve the Risky User data.
- “IdentityRiskEvent.Read.All” – This permission is required to retrieve the Risk Event data.
- “IdentityRiskyServicePrincipal.Read.All” – This permission is required to retrieve the Risky Service Principal data.
- “Mail.Send” – This permission is required to send out the report via email at the end of the script
Assigning the permissions required to the service principal is done using the Microsoft Graph PowerShell SDK using the steps described in the article about Using Power Automate and Azure Automation to Manage the Lifecycle of SharePoint Sites.
This code shows how to assign the permissions by:
- Defining the required permissions in an array.
- Getting the service principal of the Microsoft Graph app.
- Getting the identifier of the Automation account service principal.
- Getting the Role IDs for each of the required permissions.
- Looping through each role and assigning the role to the Automation account service principal
All you need to do is replace the name of the automation account (aa-EntraRiskReport’ in my example).
##List Required Permissions
$Permissions = @(
"IdentityRiskyUser.Read.All"
"IdentityRiskEvent.Read.All"
"IdentityRiskyServicePrincipal.Read.All"
"Mail.Send"
)
##Get Graph Service Principal
$GraphApp = Get-MgServicePrincipal -Filter "AppId eq '00000003-0000-0000-c000-000000000000'"
##Get Automation Account Service Principal ID
$MIID = (Get-MgServicePrincipal -Filter "displayName eq 'aa-EntraRiskReport'").id
##Get Graph Role IDs
[array]$Roles = $GraphApp.AppRoles | Where-Object {$Permissions -contains $_.Value}
##Assign each permission
foreach($role in $roles){
$AppRoleAssignment = @{
"PrincipalId" = $MIID
"ResourceId" = $GraphApp.Id
"AppRoleId" = $Role.Id
}
# Assign the Graph permission
New-MgServicePrincipalAppRoleAssignment -ServicePrincipalId $MIID -BodyParameter $AppRoleAssignment
}
Next, the following Microsoft Graph PowerShell SDK modules must be installed as resources for the Azure automation account:
- Microsoft.Graph.Authentication.
- Microsoft.Graph.Identity.SignIns.
- Microsoft.Graph.Users.Actions.
The Microsoft Graph SDK modules receive regular updates so it’s important to ensure they are up to date. For more information on updating the modules in Azure Automation, check out this article on how to keep your Microsoft Graph PowerShell Modules for Azure Automation updated.
Adding the Code
The script for this task is available on GitHub. At a high level, the script performs the following tasks:
- Declares variables for the sender address, recipient address, and subject of the report email.
- Connects to the Microsoft Graph using the PowerShell SDK.
- Gathers the risk detections, risky users, and risky service principals from the tenant, only including the entries that have an “at risk” status.
- Parses the data into collections containing the information that should be contained in the report.
- Converts the data to HTML format for the report.
- Adds some basic HTML formatting to the output.
- Adds the HTML report to the body of a new mail along with the subject and recipient parameters.
- Sends the report using the Send-MgUserMail cmdlet specifying the sender address variable.
To run in your environment, the sender and recipient variables on lines 20 and 21 of the script should be updated with valid sender and recipient addresses from your tenant.
Review the Report
The report includes three tables, Risk Detections, Risky Users and Risky Service Principals. The Risk Detections table, shown in Figure 1, lists the risk events that have occurred in the tenant that are at the Risk State “at Risk.” This means an admin hasn’t updated the status of the triggering event, which usually implies that it hasn’t been reviewed yet.
The second and third tables contain the list of users (Figure 2) and service principals where the Risk State is “at Risk.” Generally, if a user or service principal has a risk status listed, it will have one or more risk detections contributing to that status. The Risk Detections table can be used to provide context to identities listed in the user and service principal tables.
When an administrator actions a Risk within the Microsoft Entra admin center, the Risk State is updated, and the risk no longer appears in the report.
Schedule the Report
A key benefit of running scripts like this in Azure Automation is the scheduling functionality. To run script every morning, create a daily schedule in the automation account like the one shown in Figure 3.
Next, from the runbook, link the schedule to the runbook (Figure 4) to schedule the script to run each morning.
Simple, but Effective
The Automation account will run the script based on the schedule and deliver the report each morning via email. It’s important to not just read the report but to make sure that you action the risk status in your environment and update the status of risks as they are mitigated. While this information is available from the Entra admin center, delivering it in a daily report is a good way to ensure the information is visible to the people who need it easily. There’s nothing particularly complicated in the script but it does what it needs to by connecting to the tenant, gathering data, formatting it, and then sending it out via email. I’m sure there are improvements that can be made, particularly on the formatting of the report, but I’ll leave that to someone with a better eye for design than me.
TEC Talk: Active Directory Modernization: A Journey to a Secure and Flexible Identity Infrastructure
Join Julian Stephan and Joe Sharmer’s Free Webinar on Feb. 1st @ 11 AM EST.
Hi, Thanks for this, how do I do this with reporting if a user has reached 80% of their mailbox storage?
This won’t help with mailbox sizes. Your options there are to enable archives with a retention policy (provided you have appropriate licensing) or two delete emails.
Hello Sean
Thanks for the helpful post. Everything is working perfectly. Apparently Microsoft has tweaked the licensing again. My tenant has Entra ID P1 licensed. Nevertheless I get the error message “[AccessDenied] : Your tenant is not licensed for this feature. Please upgrade your subscription to access it.”
I am having some issues and could use some help. When running the runbook, I am getting a long error. Copied below. What can I do to get this solution working?
System.Management.Automation.MethodInvocationException: Exception calling “ShouldContinue” with “2” argument(s): “A command that prompts the user failed because the host program or the command type does not support user interaction. The host was attempting to request confirmation with the following message: PowerShellGet requires NuGet provider version ‘2.8.5.201’ or newer to interact with NuGet-based repositories. The NuGet provider must be available in ‘C:\Program Files\PackageManagement\ProviderAssemblies’ or ‘C:\Users\ContainerUser\AppData\Local\PackageManagement\ProviderAssemblies’. You can also install the NuGet provider by running ‘Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force’. Do you want PowerShellGet to install and import the NuGet provider now?”
—> System.Management.Automation.Host.HostException: A command that prompts the user failed because the host program or the command type does not support user interaction. The host was attempting to request confirmation with the following message: PowerShellGet requires NuGet provider version ‘2.8.5.201’ or newer to interact with NuGet-based repositories. The NuGet provider must be available in ‘C:\Program Files\PackageManagement\ProviderAssemblies’ or ‘C:\Users\ContainerUser\AppData\Local\PackageManagement\ProviderAssemblies’. You can also install the NuGet provider by running ‘Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force’. Do you want PowerShellGet to install and import the NuGet provider now?
at System.Management.Automation.Internal.Host.InternalHostUserInterface.ThrowPromptNotInteractive(String promptMessage)
at System.Management.Automation.Internal.Host.InternalHostUserInterface.PromptForChoice(String caption, String message, Collection`1 choices, Int32 defaultChoice)
at System.Management.Automation.MshCommandRuntime.InquireHelper(String inquireMessage, String inquireCaption, Boolean allowYesToAll, Boolean allowNoToAll, Boolean replaceNoWithHalt, Boolean hasSecurityImpact)
at System.Management.Automation.MshCommandRuntime.DoShouldContinue(String query, String caption, Boolean hasSecurityImpact, Boolean supportsToAllOptions, Boolean& yesToAll, Boolean& noToAll)
at System.Management.Automation.MshCommandRuntime.ShouldContinue(String query, String caption)
at System.Management.Automation.Cmdlet.ShouldContinue(String query, String caption)
at CallSite.Target(Closure , CallSite , PSScriptCmdlet , Object , Object )
— End of inner exception stack trace —
at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception exception, Type typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo)
at CallSite.Target(Closure , CallSite , PSScriptCmdlet , Object , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
at System.Management.Automation.Interpreter.DynamicInstruction`4.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
NuGet provider is required to interact with NuGet-based repositories. Please ensure that ‘2.8.5.201’ or newer version of NuGet provider is installed.
The specified module ‘Microsoft.Graph’ was not loaded because no valid module file was found in any module directory.
Unhandled Exception – Message:’The type initializer for ‘Microsoft.PackageManagement.Internal.Utility.Plugin.DynamicType’ threw an exception.’ Name:’TypeInitializationException’ Stack Trace:’ at Microsoft.PackageManagement.Internal.Utility.Plugin.DynamicType.Create(Type tInterface, OrderedDictionary`2 instanceMethods, List`2 delegateMethods, List`1 stubMethods, List`2 usedInstances)
at Microsoft.PackageManagement.Internal.Utility.Plugin.DynamicInterface.CreateProxy(Type tInterface, Object[] instances)
at Microsoft.PackageManagement.Internal.Utility.Plugin.DynamicInterface.DynamicCast(Type tInterface, Object[] instances)
at Microsoft.PackageManagement.Internal.Utility.Plugin.DynamicInterface.DynamicCast[TInterface](Object[] instances)
at Microsoft.PackageManagement.Internal.Utility.Plugin.DynamicInterfaceExtensions.As[TInterface](Object instance)
at Microsoft.PowerShell.PackageManagement.Cmdlets.CmdletBase.get_PackageManagementHost()
at Microsoft.PowerShell.PackageManagement.Cmdlets.CmdletBase.SelectProviders(String[] names)
at Microsoft.PowerShell.PackageManagement.Cmdlets.CmdletWithProvider.get_SelectedProviders()
at Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackageProvider.get_SelectedProviders()
at Microsoft.PowerShell.PackageManagement.Cmdlets.CmdletWithProvider.b__23_0()
at Microsoft.PackageManagement.Internal.Utility.Extensions.DictionaryExtensions.GetOrAdd[TKey,TValue](IDictionary`2 dictionary, TKey key, Func`1 valueFunction)
at Microsoft.PackageManagement.Internal.Utility.Extensions.Singleton`1.GetOrAdd(Func`1 newInstance, Object primaryKey, Object[] keys)
at Microsoft.PackageManagement.Internal.Utility.Extensions.SingletonExtensions.GetOrAdd[TResult](Object primaryKey, Func`1 newInstance, Object[] keys)
at Microsoft.PowerShell.PackageManagement.Cmdlets.CmdletWithProvider.get_CachedSelectedProviders()
at Microsoft.PowerShell.PackageManagement.Cmdlets.CmdletWithProvider.GenerateDynamicParameters()
at Microsoft.PowerShell.PackageManagement.Cmdlets.AsyncCmdlet.c__DisplayClass83_0.b__0()’
The specified module ‘NuGet’ was not loaded because no valid module file was found in any module directory.
System.Management.Automation.CommandNotFoundException: The term ‘Connect-MgGraph’ is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandTypes commandTypes, SearchResolutionOptions searchResolutionOptions, CommandOrigin commandOrigin, ExecutionContext context)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandOrigin commandOrigin, ExecutionContext context)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandOrigin commandOrigin)
at System.Management.Automation.CommandDiscovery.LookupCommandProcessor(String commandName, CommandOrigin commandOrigin, Nullable`1 useLocalScope)
at System.Management.Automation.ExecutionContext.CreateCommand(String command, Boolean dotSource)
at System.Management.Automation.PipelineOps.AddCommand(PipelineProcessor pipe, CommandParameterInternal[] commandElements, CommandBaseAst commandBaseAst, CommandRedirection[] redirections, ExecutionContext context)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
System.Management.Automation.CommandNotFoundException: The ‘Get-MgRiskDetection’ command was found in the module ‘Microsoft.Graph.Identity.SignIns’, but the module could not be loaded. For more information, run ‘Import-Module Microsoft.Graph.Identity.SignIns’.
at System.Management.Automation.CommandDiscovery.TryModuleAutoDiscovery(String commandName, ExecutionContext context, String originalCommandName, CommandOrigin commandOrigin, SearchResolutionOptions searchResolutionOptions, CommandTypes commandTypes, Exception& lastError)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandTypes commandTypes, SearchResolutionOptions searchResolutionOptions, CommandOrigin commandOrigin, ExecutionContext context)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandOrigin commandOrigin, ExecutionContext context)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandOrigin commandOrigin)
at System.Management.Automation.CommandDiscovery.LookupCommandProcessor(String commandName, CommandOrigin commandOrigin, Nullable`1 useLocalScope)
at System.Management.Automation.ExecutionContext.CreateCommand(String command, Boolean dotSource)
at System.Management.Automation.PipelineOps.AddCommand(PipelineProcessor pipe, CommandParameterInternal[] commandElements, CommandBaseAst commandBaseAst, CommandRedirection[] redirections, ExecutionContext context)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
System.Management.Automation.CommandNotFoundException: The ‘Get-MgRiskyUser’ command was found in the module ‘Microsoft.Graph.Identity.SignIns’, but the module could not be loaded. For more information, run ‘Import-Module Microsoft.Graph.Identity.SignIns’.
at System.Management.Automation.CommandDiscovery.TryModuleAutoDiscovery(String commandName, ExecutionContext context, String originalCommandName, CommandOrigin commandOrigin, SearchResolutionOptions searchResolutionOptions, CommandTypes commandTypes, Exception& lastError)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandTypes commandTypes, SearchResolutionOptions searchResolutionOptions, CommandOrigin commandOrigin, ExecutionContext context)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandOrigin commandOrigin, ExecutionContext context)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandOrigin commandOrigin)
at System.Management.Automation.CommandDiscovery.LookupCommandProcessor(String commandName, CommandOrigin commandOrigin, Nullable`1 useLocalScope)
at System.Management.Automation.ExecutionContext.CreateCommand(String command, Boolean dotSource)
at System.Management.Automation.PipelineOps.AddCommand(PipelineProcessor pipe, CommandParameterInternal[] commandElements, CommandBaseAst commandBaseAst, CommandRedirection[] redirections, ExecutionContext context)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
System.Management.Automation.CommandNotFoundException: The ‘Get-MgRiskyServicePrincipal’ command was found in the module ‘Microsoft.Graph.Identity.SignIns’, but the module could not be loaded. For more information, run ‘Import-Module Microsoft.Graph.Identity.SignIns’.
at System.Management.Automation.CommandDiscovery.TryModuleAutoDiscovery(String commandName, ExecutionContext context, String originalCommandName, CommandOrigin commandOrigin, SearchResolutionOptions searchResolutionOptions, CommandTypes commandTypes, Exception& lastError)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandTypes commandTypes, SearchResolutionOptions searchResolutionOptions, CommandOrigin commandOrigin, ExecutionContext context)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandOrigin commandOrigin, ExecutionContext context)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandOrigin commandOrigin)
at System.Management.Automation.CommandDiscovery.LookupCommandProcessor(String commandName, CommandOrigin commandOrigin, Nullable`1 useLocalScope)
at System.Management.Automation.ExecutionContext.CreateCommand(String command, Boolean dotSource)
at System.Management.Automation.PipelineOps.AddCommand(PipelineProcessor pipe, CommandParameterInternal[] commandElements, CommandBaseAst commandBaseAst, CommandRedirection[] redirections, ExecutionContext context)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
System.Management.Automation.CommandNotFoundException: The ‘Send-MgUserMail’ command was found in the module ‘Microsoft.Graph.Users.Actions’, but the module could not be loaded. For more information, run ‘Import-Module Microsoft.Graph.Users.Actions’.
at System.Management.Automation.CommandDiscovery.TryModuleAutoDiscovery(String commandName, ExecutionContext context, String originalCommandName, CommandOrigin commandOrigin, SearchResolutionOptions searchResolutionOptions, CommandTypes commandTypes, Exception& lastError)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandTypes commandTypes, SearchResolutionOptions searchResolutionOptions, CommandOrigin commandOrigin, ExecutionContext context)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandOrigin commandOrigin, ExecutionContext context)
at System.Management.Automation.CommandDiscovery.LookupCommandInfo(String commandName, CommandOrigin commandOrigin)
at System.Management.Automation.CommandDiscovery.LookupCommandProcessor(String commandName, CommandOrigin commandOrigin, Nullable`1 useLocalScope)
at System.Management.Automation.ExecutionContext.CreateCommand(String command, Boolean dotSource)
at System.Management.Automation.PipelineOps.AddCommand(PipelineProcessor pipe, CommandParameterInternal[] commandElements, CommandBaseAst commandBaseAst, CommandRedirection[] redirections, ExecutionContext context)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
This seems like the modules are not installed correctly in the automation account.
Thank you – got it fixed. Great job!