Practical Graph: Using the Microsoft AuditLog Query Graph API (Preview)

The Unified Audit Log is a Terrific Source of Information

Since its introduction in 2016, the unified audit log has been a great source of information for Microsoft 365 tenant administrators. Depending on the workloads running in a tenant, the audit log can ingest over 1,600 different events. Tenants with Purview Audit (standard) licenses (Office 365 E3) can access audit events for 180 days while those with Purview Audit (premium) licenses can access events for 365 days. Despite some funky formatting of information in some audit events generated by certain workloads, there’s no doubt about the usefulness of being able to query the audit log to discover what happened when inside a tenant.

According to Microsoft 365 roadmap item 117587, the API will be generally available in May 2024.

The AuditLog Query Graph API

Until now, the only programmatic method to query the audit log has been to run the Search-UnifiedAuditLog cmdlet. This method works well, even if Microsoft made unexpected changes to the way the cmdlet works in mid-2023. So far, no public explanation has appeared to justify the change. I guess finding unannounced changes that impact customer scripts is part of the joy of working in the cloud.

Microsoft introduced a new search feature in the Purview Compliance portal in 2022. Instead of performing an interactive search, the portal creates search queries that the service processes in the background. The AuditLog Query Graph API (aka, the Microsoft Purview Audit Search API) is a preview feature to create and run search queries and retrieve search results in a manner like that used by Purview. Several steps are needed to execute a search:

• Create the search query by setting the values of the search parameters.
• Submit a search query (job).
• When the search query completes, retrieve the search results (audit records).

This all sounds very easy but it’s more complicated than running an audit search with the Search-UnifiedAuditLog cmdlet. Let’s examine the AuditLog Query API in a little more detail.

Creating an Audit Search Query

I used the Microsoft Graph PowerShell SDK to test the AuditLog Query API. The first thing to note is that I used the AuditLogsQuery.Read.All permission to access the audit log. This permission allows access to all audit data. Other permissions are available to grant limited access to audit data, such as only allowing access to Exchange Online audit events. You’ll also need to sign in with an administrator account to use a delegated permission. Jobs that run as background processes such as an Azure Automation runbook use application permissions and don’t need to sign in as an administrator.

This code snipper shows how to create a hash table with parameters for a new search query and how to post the query to the audit log queries endpoint to create a new job.

$Uri = "https://graph.microsoft.com/beta/security/auditLog/queries"
$SearchName = ("Audit Search {0}" -f (Get-Date -format 'dd-MMM-yyyy HH:mm'))

$SearchParameters = @{
    "displayName"           = $SearchName
    "filterStartDateTime"   = $StartDateSearch
    "filterEndDateTime"     = $EndDateSearch
    "operationFilters"      = $Operations
}

$SearchQuery = Invoke-MgGraphRequest -Method POST -Uri $Uri -Body $SearchParameters
$SearchId = $SearchQuery.Id

The $Operations variable contains a comma-separated list of audit events to search for. The start and end date for the search are passed in sortable date format with a Z suffix. Here’s an example of what the hash table looks like for a query that looks for two SharePoint Online operations from midnight on February 29, 2024, to midnight on March 2, 2024:

Name                           Value
----                           -----
filterEndDateTime              2024-03-02T00:00:00Z
displayname                    Audit Search 04-Mar-2024 05:29
filterStartarDateTime          2024-02-29T00:00:00Z
operationsFilters              {fileuploaded, filemodified}

Some like to fetch all audit records for a period because they want to load the audit data into an SIEM for long-term retention and query purposes. With the Search-UnifiedAuditLog cmdlet, the way to fetch every available audit record is to not specify any operations.

To fetch every audit record for a specific period, don’t include the operationsFilter in the parameters used for an Auditlog query. The parameters will then just be a display name, start date and time, and end date and time. Alternatively, you can include a recordTypeFilters filter to define the set of audit record types that you’re interested in.

Running an Audit Search Query

It can take up to ten minutes for the search query to complete. Eventually, the search finishes, and you can fetch whatever audit records the query finds. This snippet shows how to fetch the first page of audit records.

$Uri = ("https://graph.microsoft.com/beta/security/auditLog/queries/{0}/records" -f $SearchId)
[array]$SearchRecords = Invoke-MgGraphRequest -Uri $Uri -Method GET

The API returns 150 records in a page. If there are fewer records than 150 to fetch, you can proceed to analyze the returned records. However, if more data is available, your code must use the nextlink URIs returned when additional data is available to fetch the next page until no more are available. Forcing code to fetch data in limited chunks is a common process called pagination that is used by Graph APIs to restrict the amount of data retrieved at one time. Here’s the code to paginate and fetch the available audit records for a query:

# Paginate to fetch all available audit records
$NextLink = $SearchRecords.'@Odata.NextLink'
While ($null -ne $NextLink) {
    $SearchRecords = $null
    [array]$SearchRecords = Invoke-MgGraphRequest -Uri $NextLink -Method GET 
    $AuditRecords += $SearchRecords.value
    Write-Host ("{0} audit records fetched so far..." -f $AuditRecords.count)
    $NextLink = $SearchRecords.'@odata.NextLink' 
}

Pagination works well with Graph APIs. The preview version of the AuditLog Query API seems to be quite delicate and some of my attempts to use nextlink pointers resulted in a “500 Internal Server Error,” which blocked attempts to retrieve audit records.

The Microsoft Graph PowerShell SDK makes it easier for developers by including an All parameter for many cmdlets to fetch all available data. Cmdlets to manage audit log queries exist in the Microsoft Graph PowerShell SDK (V2.15). For instance, to retrieve the audit records found by a query, you can run the Get-MgBetaSecurityAuditLogQueryRecord cmdlet and specify the All parameter:

[array]$AuditRecords = Get-MgBetaSecurityAuditLogQueryRecord -AuditLogQueryId $SearchId -All

At first glance, the cmdlet seems to be a great way to fetch all audit records returned by an audit log search. The cmdlet does return all the records, but no matter what I did, the content of the AuditData property was always Microsoft.Graph.Beta.PowerShell.Models.MicrosoftGraphSecurityAuditData. Without the content of the AuditData property to analyze, the value of the retrieved audit records is much diminished. The upshot is that I continued working with Graph queries. This issue is the kind of thing that will likely be resolved as Microsoft brings the API to general availability.

Analyzing the Audit Data

Audit records have two parts. The first is constant across all workloads and contains information like a timestamp, operation name, and user or system account responsible for the action. The second, stored in the AuditData property, is where the information about what happened is found. Each workload controls the information stored in AuditData and considerable variance exists between workloads. For this reason, parsing the content of audit records takes more effort and bespoke coding than should be necessary.

The good news is that the AuditLog Query API returns the AuditData property as a hash table that’s easy to access. The bad news is that the same variance in content exists across workloads (I didn’t expect anything else). In other words, for each kind of audit event, you must figure out what the AuditData property holds and what the data means in the same manner as with the results generated by the Search-UnifiedAuditLog cmdlet.

Figure 1 shows some of the information from audit records fetched from a query. I have no idea what object has an identifier of eba15bfd-c28e-4433-a20e-0278888c5825. I can’t find a user account or service principal with this value. It’s a blessed mystery.

You can download the script I used for testing from GitHub.

Focusing on the Ease of Audit Event Extraction

It’s always important to remember when software is in a preview stage. The code is there for people to test, and it will improve before it reaches general availability. Given the amount of audit data generated by Microsoft 365 operations, it’s understandable that Microsoft would want to move away from synchronous interactive audit search operations of the type run by the Search-UnifiedAuditLog cmdlet to something more attuned to service operations that can run in the background. That’s the way that the Purview compliance portal works now, and the existence of the AuditLog Query API is another sign that Microsoft would like administrators to interrogate audit events in this mode.

Microsoft’s problem is that the ease of working with a single PowerShell cmdlet compared against the complexity of setting up asynchronous audit searches to run in the background and then fetching the results is such that administrators are likely to eschew the Graph API approach. Everyone likes an easy life, and Search-UnifiedAuditLog remains the easiest way to retrieve data from the Microsoft 365 audit log.