Use Azure Automation and PowerShell to Track Critical App Audit Events
I hear a lot about the need to monitor Entra ID apps these days. Attackers can sneak in their own apps with permissions to exfiltrate or remove data, and software vendors use apps to make their products available to Microsoft 365 tenant users. Most of the time, a vendor app is just fine. Sometimes, it’s more doubtful, such as the enterprise app used by ChatGPT to allow users to upload files from SharePoint Online and OneDrive for Business for processing by its AI models. For the purpose of this article, we define an enterprise app to be an app owned by another Entra ID tenant. An app registration (or registered app) is an app created in your tenant. I use the generic “apps” term to cover both types.
Various tools are available to monitor apps present in a tenant. The app governance solution in Microsoft Defender for Cloud Apps is an option if you have Microsoft Defender for Office 365 Plan 1 or 2 licenses. You could also export audit data to Microsoft Sentinel and use Kusto Query Language (KQL) queries to check for anomalous situations. Other commercial offerings are available to monitor app usage in a tenant, or you can create your own monitoring solution with PowerShell, which is what I cover here.
The solution outlined in this article is not fully developed. Instead, it’s a working example of how to interrogate the audit log to find and analyze important app management events. You can add to the functionality to make the script do what you consider is critical.
Purview Audit and App Events
Microsoft Purview Audit captures a multitude of audit events for different actions performed by administrators, users, and background processes against different objects, including apps. The Entra audit log is a rich source of information that flows into the Purview unified audit log. Reporting the assignment of high-priority permissions to apps is an example of using the Entra audit log. Reporting consents for permissions granted to apps is an example of using audit data originating from Entra and surfaced in the unified audit log. The point is that the unified audit log ingests events from the Entra audit log and retains that data for 180 days (audit standard) or 365 days (audit premium) rather than the 30-day retention used by Entra.
Some of the example PowerShell scripts available on the internet don’t attempt to interpret the AuditData payload in audit events and are happy to report the fact that such and such an event happened at this time. Although Microsoft makes the task harder by inconsistencies in how important data is stored in audit events, it is possible to extract critical information such as permissions granted to an app. It just takes extra effort and some knowledge of what the data in the AuditData payload might contain.
Using the AuditLog Query Graph API
Many scripts that interrogate the audit log use the Search-UnifiedAuditLog cmdlet. The best thing about the cmdlet is that it performs a synchronous search. The worst thing is that you must wait for the synchronous search to finish before a script can perform other tasks.
I don’t use Search-UnifiedAuditLog in this case. Instead, the script uses the AuditLogQuery Graph API to find relevant audit events. Periodic checking for potential problems is best done with background jobs, and an Azure Automation runbook is the most secure and robust method for running scheduled tasks. You might notice that I use the beta version of the API. That’s because Microsoft withdrew the production version to fix some issues. By the time you read this, the production (V1.0) version of the AuditLogQuery API might be available.
The required components for the solution are:
- An Azure Automation account. The account is represented in Entra ID by a service principal.
- Graph permissions are assigned to the automation account’s service principal to allow access to tenant data. The permissions are:
- AuditLogsQuery.Read.All (access the Purview audit log).Application.Read.All (read information about apps and service principals).
- DelegatedPermissionGrant.Read.All (read information about delegated permission grants).
- Microsoft Graph PowerShell SDK modules are loaded into the automation account to allow the account to run cmdlets. The modules are:
- Microsoft.Graph.Authentication (authenticate connect to the Graph with a managed identity).Microsoft.Graph.Users.Actions (run the Send-MgUserMail cmdlet to send a message containing details of critical app events).Microsoft.Graph.Applications (run the Get-MgApplication and Get-MgServicePrincipal cmdlets to fetch details of apps and service principals).
- Microsoft.Graph.Identity.SignIns (run the Get-MgOauth2PermissionGrant cmdlet to fetch details of delegated permission grants).
Details of how to assign the Graph permissions to the service principal of the automation account and how to load modules into the account are described here.
Scripting to Find and Analyze App Management Audit Events
The script is straightforward and does the following. After authenticating, the code checks to verify whether it is running in an interactive session. If so, delegated permissions are used and the email sent at the end comes from the signed-in user. If not, the code assumes that it runs in Azure Automation and uses a predefined email address as the mail sender. The Mail.Send permission allows the script to send as any mailbox in the tenant. You can use RBAC for Applications to restrict access to sensitive mailboxes.
The script creates and submits an audit query job to find events over the last 90 days for the actions listed below. These are only some of the app management events captured in the audit log, but they are ones involving the assignment of applications and delegated permissions, so they are worth reviewing. Feel free to add more events to the query and analysis (the full stop at the end of the event names is part of the event name):
- Add app role assignment to service principal.
- Add application.
- Add delegated permission grant.
- Update application – Certificates and secrets management (with a trailing space!)
- Consent to application.
After the audit job finishes, the script puts the audit events found by the search into an array. It then runs Get-MgServicePrincipal to find all service principals and populates a hash table to make this data faster to retrieve when analyzing audit records. The script also fetches roles (permissions) defined by the Microsoft Graph. Again, this is to make subsequent lookups faster.
The script analyzes each audit record to extract relevant data from the AuditData payload. I’ve done my best to understand the content of audit event properties that are sometimes formatted in a peculiar manner. Feel free to correct me if I’ve got something wrong. Lookups are performed to result in the GUID identifiers for service principals and apps to make this information more accessible to humans. The information extracted from the audit records is stored in a PowerShell list. Figure 1 shows some of the data collected in the list.
After creating the report, the script checks for any apps in the list that have high-priority permissions. These are permissions that hackers commonly exploit as they explore or remove data from tenants. If any apps with high-priority permissions are detected, their details are included in the output report.
Finally, the script creates an HTML report and CSV file from the report data. These files are emailed to the email address defined in the script. The email address can be for any mail-enabled object, such as a user, group, distribution list, or even a mail-enabled public folder. The Send-MgUserMail cmdlet sends the message with the HTML report as the message body and the CSV file as an attachment (Figure 2).
The full script is available from GitHub.
Interactive First, then Azure Automation
I always recommend developing code to run interactively first. It’s much easier to find and fix bugs in an interactive Microsoft Graph session than to try to find out where things might be going wrong when code executes on a headless server, which is what happens with Azure Automation runbooks. Once the script works, you can move it to Azure Automation, test it there (Figure 3), and make any final tweaks, like adding the runbook to a schedule so that it runs weekly, monthly, or at whatever interval you deem appropriate.
The point is that there’s nothing too difficult in creating and executing runbooks to process Microsoft 365 data. In this scenario, the hardest challenge is to decipher the contents of the audit event payloads. This becomes easier after a while when you start to understand how the developers defined their audit events. Then again, audit events have a nasty habit of using odd data formats, so be prepared to be persistent when extracting the valuable information contained in the app management events.
