Microsoft Changed the Way the Search-UnifiedAuditLog Cmdlet Works Without Telling Anyone

Returning Limited Search Results Might Catch You Out

I’m a big fan of using the Search-UnifiedAuditLog cmdlet to interrogate Microsoft 365 audit data to discover what happens in a Microsoft 365 tenant. Examples of the usefulness of audit data are monitoring for events that might indicate that an attacker is trying to sneak into a tenant or checking for the addition of specific user accounts to the membership of teams. As Paul Robichaux has pointed out, the audit log isn’t perfect, but it’s still a great place to go looking when you want to find out who did what.

Having used the Search-UnifiedAuditLog cmdlet to interrogate the audit log since its introduction in 2015, I think I know most of the peccadillos of the audit log. However, recently I noticed that audit log searches didn’t work the way that I expected. Where searches used to return hundreds of events, now they max out at 120 events.

Checking Audit Events for a Day

For example, to discover new events or to understand how the different workloads capture audit events, I regularly run a search to find all events generated in my tenant for a day. This is how I discovered that Microsoft changed the name of the event captured when users delete SharePoint Online or OneDrive for Business files from FileDeleted to FileRecycled. Given that deleted files travel through a two-stage recycle bin, a logic exists to justify the rename. The problem is that Microsoft never told anyone when they renamed the event, which means that any script looking for these events promptly returned zero.

The kind of search I use is:

[array]$Records = Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-1) -EndDate (Get-Date) -Formatted -ResultSize 5000
$Records | Group-Object Operations -NoElement | Sort-Object Count -Descending | Format-Table Name, Count -AutoSize 

Name                                Count
----                                -----
FileAccessed                           24
MoveToDeletedItems                     24
ListViewed                              9
MailItemsAccessed                       9
AccessedAggregates                      8
Search                                  8

Depending on activity levels, the unified audit log for my tenant generates a few thousand audit events daily. Now, I get 120. Always 120. Never more than 120 audit events for a command that I have used for years. This is despite the command specifying that the search can return up to 5,000 records by passing that value in the ResultSize parameter. Microsoft’s documentation for the Search-UnifiedAuditLog cmdlet says:

“The ResultSize parameter specifies the maximum number of results to return. The default value is 100, maximum is 5,000.”

I know that the search returns a subset of the available records by checking the timestamp for the first and last records in the returned set. As you can see, the events span just over a three-hour period and not the full day requested.

$Records.count
120
$Records[0].CreationDate
Wednesday 23 August 2023 19:39:09
$Records[119].CreationDate
Wednesday 23 August 2023 16:21:21

Some other folks have run into the problem (here’s an example), but I’m surprised that no more fuss has resulted. This might be because people haven’t noticed that their audit searches aren’t returning full results.

Use the SessionCommand Parameter

As it turns out, there’s a simple solution. MVP Vasil Michev points out that if you add the SessionCommand ReturnLargeSet parameter to the search, you get a better result:

[array]$Records = Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-1) -EndDate (Get-Date) -Formatted -ResultSize 5000 -SessionCommand ReturnLargeSet
$Records | Group-Object Operations -NoElement | Sort-Object Count -Descending | Format-Table Name, Count -AutoSize 

Name                                   Count
----                                   -----
MailItemsAccessed                        380
PlanRead                                 165
UserLoggedIn                             125
MoveToDeletedItems                       117
TaskListRead                             112
TaskModified                             107
Search                                    74
FileAccessed                              63
TaskUpdated                               33

The search now returns many more events and covers the anticipated period:

$Records.count
1526
$Records[0].CreationDate
Wednesday 23 August 2023 19:37:21
$Records[1525].CreationDate
Tuesday 22 August 2023 20:20:23

So what’s the problem? A couple come to mind:

• Microsoft has published no guidance to say that generic audit log searches return limited results.
• Microsoft’s documentation doesn’t recommend the use of the SessionCommand parameter in searches. Some of the examples show the use of SessionCommand ReturnLargeSet, but I can’t find a specific directive to use the parameter in the kind of searches that I’ve conducted since 2015. Apart from referencing SessionCommand in the context of managing large amounts of audit data, Microsoft’s page describing Best Practices for using Search-UnifiedAuditLog doesn’t address the topic either.

In a nutshell, it seems that Microsoft has changed the way the Search-UnifiedAuditLog cmdlet works without saying anything. The net effect is that scripts that perform audit log searches might suddenly return limited results that their users don’t notice.

Unsorted and Latency

Microsoft’s documentation for the Search-UnifiedAuditLog cmdlet says:

“ReturnLargeSet: This value causes the cmdlet to return unsorted data. By using paging, you can access a maximum of 50,000 results. This is the recommended value if an ordered result is not required and has been optimized for search latency.”

Unsorted and latency are two important words in this text. Because the retrieved data is unsorted, it’s important to sort the records to get an accurate timeline for the audit events. Look at the difference in the timestamps for the first and last records in the examples shown above when we sort the data.

You could argue that the difference is small, but accuracy is important in compliance operations, and assembling actions into an exact timeline is part of achieving accuracy.

Latency is important because Microsoft is aware that the Search-UnifiedAuditLog cmdlet has a checkered history of reliability since its introduction. Some of the issues are due to timeouts when the cmdlet becomes unresponsive. Other issues include the return of duplicated events or failure to find available events.

It’s common to find that organizations routinely strip audit events from the log to ingest into an external repository. Sometimes this is for long-term (multi-year) retention, sometimes it’s because the external repository offers superior search and analysis features for the audit data. No matter what the reason is, the fact is that organizations run scripts to strip and reuse audit data. The scripts they use perform catch-all searches to find every audit event generated in a tenant. And the scripts include a lot of error handling to make sure that they deal with cmdlet errors.

A New Audit Log Search

The “classic” audit log search feature in the Microsoft Purview Compliance portal uses the Search-UnifiedAuditLog cmdlet and suffers from some of the same issues. Microsoft believes that returning search data synchronously is part of the problem, which led to the introduction of a new audit log search in mid-2022. Background jobs run the searches and the results appear when the search completes.

Microsoft believes that the new audit search is a more intelligent and predictable way to perform audit searches, especially because Microsoft 365 workloads generate more data than ever before. In addition, on July 19, 2023, Microsoft doubled the retention period for audit events for accounts with Office 365 E3 licenses from 90 to 180 days. Accounts with Office 365 E5 licenses retain audit events for 365 days. Welcome as the doubling of the retention period is, it does add to the amount of data that an audit log search must process.

Asynchronous searches take longer. The Search-UnifiedAuditLog command described above took 37 seconds. The equivalent new search took 4 minutes and 15 seconds (Figure 1). The extra time is needed to set up and run the search.

Like any IT function, the time required to perform a task can vary depending on the amount of data and the service load at the time. My experience is that asynchronous searches can be quite slow, which can be frustrating when investigating a problem by looking for what’s available in audit data. On the upside, the new audit search allows up to ten filtered searches to run concurrently (per administrator account) and the data retrieved by each search can be exported for slicing and dicing to find the necessary data.

Microsoft Should Fix Search-UnifiedAuditLog

Although I don’t know the reason, I speculate that Microsoft might limit the results returned by Search-UnifiedAuditLog to:

• Conserve resources.
• Convince customers to run focused searches rather than catch-all scans.
• Encourage customers to move to the new audit log search.

Two big problems exist if this is what’s happening. First, the total lack of communication. Apart from the announcement about the extended audit log retention period, Microsoft has been quiet on the topic of audit log searches. Second, no cmdlet is available to perform asynchronous audit log searches. Organizations can’t replace the Search-UnifiedAuditLog cmdlet even if they wanted to.

The sad thing is that an obvious solution exists. Microsoft should fix the Search-UnifiedAuditLog cmdlet to make it more robust and reliable, and even add the ability to conduct asynchronous searches. We’ve learned a lot about running audit log searches since 2015. It would be a pity to discard that collective experience and all the customer scripts that have been written over the last eight years.