Returning Limited Search Results Might Catch You Out

I’m a big fan of using the Search-UnifiedAuditLog cmdlet to interrogate Microsoft 365 audit data to discover what happens in a Microsoft 365 tenant. Examples of the usefulness of audit data are monitoring for events that might indicate that an attacker is trying to sneak into a tenant or checking for the addition of specific user accounts to the membership of teams. As Paul Robichaux has pointed out, the audit log isn’t perfect, but it’s still a great place to go looking when you want to find out who did what.

Having used the Search-UnifiedAuditLog cmdlet to interrogate the audit log since its introduction in 2015, I think I know most of the peccadillos of the audit log. However, recently I noticed that audit log searches didn’t work the way that I expected. Where searches used to return hundreds of events, now they max out at 120 events.

Update: On November 29, Microsoft finally issued a service health update (MO694194) to update customers that using the Search-UnifiedAuditLog cmdlet might not return all expected results. To date, Microsoft has not said if they will fix the problem. The only advice offered is to use the SessionCommand parameter as discussed below.

Checking Audit Events for a Day

For example, to discover new events or to understand how the different workloads capture audit events, I regularly run a search to find all events generated in my tenant for a day. This is how I discovered that Microsoft changed the name of the event captured when users delete SharePoint Online or OneDrive for Business files from FileDeleted to FileRecycled. Given that deleted files travel through a two-stage recycle bin, a logic exists to justify the rename. The problem is that Microsoft never told anyone when they renamed the event, which means that any script looking for these events promptly returned zero.

The kind of search I use is:

[array]$Records = Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-1) -EndDate (Get-Date) -Formatted -ResultSize 5000
$Records | Group-Object Operations -NoElement | Sort-Object Count -Descending | Format-Table Name, Count -AutoSize 

Name                                Count
----                                -----
FileAccessed                           24
MoveToDeletedItems                     24
ListViewed                              9
MailItemsAccessed                       9
AccessedAggregates                      8
Search                                  8

Depending on activity levels, the unified audit log for my tenant generates a few thousand audit events daily. Now, I get 120. Always 120. Never more than 120 audit events for a command that I have used for years. This is despite the command specifying that the search can return up to 5,000 records by passing that value in the ResultSize parameter. Microsoft’s documentation for the Search-UnifiedAuditLog cmdlet says:

“The ResultSize parameter specifies the maximum number of results to return. The default value is 100, maximum is 5,000.”

I know that the search returns a subset of the available records by checking the timestamp for the first and last records in the returned set. As you can see, the events span just over a three-hour period and not the full day requested.

$Records.count
120
$Records[0].CreationDate
Wednesday 23 August 2023 19:39:09
$Records[119].CreationDate
Wednesday 23 August 2023 16:21:21

Some other folks have run into the problem (here’s an example), but I’m surprised that no more fuss has resulted. This might be because people haven’t noticed that their audit searches aren’t returning full results.

Microsoft Platform Migration Planning and Consolidation

Simplify migration planning, overcome migration challenges, and finish projects faster while minimizing the costs, risks and disruptions to users.

Use the SessionCommand Parameter

As it turns out, there’s a simple solution. MVP Vasil Michev points out that if you add the SessionCommand ReturnLargeSet parameter to the search, you get a better result:

[array]$Records = Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-1) -EndDate (Get-Date) -Formatted -ResultSize 5000 -SessionCommand ReturnLargeSet
$Records | Group-Object Operations -NoElement | Sort-Object Count -Descending | Format-Table Name, Count -AutoSize 

Name                                   Count
----                                   -----
MailItemsAccessed                        380
PlanRead                                 165
UserLoggedIn                             125
MoveToDeletedItems                       117
TaskListRead                             112
TaskModified                             107
Search                                    74
FileAccessed                              63
TaskUpdated                               33

The search now returns many more events and covers the anticipated period:

$Records.count
1526
$Records[0].CreationDate
Wednesday 23 August 2023 19:37:21
$Records[1525].CreationDate
Tuesday 22 August 2023 20:20:23

So what’s the problem? A couple come to mind:

  • Microsoft has published no guidance to say that generic audit log searches return limited results.
  • Microsoft’s documentation doesn’t recommend the use of the SessionCommand parameter in searches. Some of the examples show the use of SessionCommand ReturnLargeSet, but I can’t find a specific directive to use the parameter in the kind of searches that I’ve conducted since 2015. Apart from referencing SessionCommand in the context of managing large amounts of audit data, Microsoft’s page describing Best Practices for using Search-UnifiedAuditLog doesn’t address the topic either.

In a nutshell, it seems that Microsoft has changed the way the Search-UnifiedAuditLog cmdlet works without saying anything. The net effect is that scripts that perform audit log searches might suddenly return limited results that their users don’t notice.

Unsorted and Latency

Microsoft’s documentation for the Search-UnifiedAuditLog cmdlet says:

“ReturnLargeSet: This value causes the cmdlet to return unsorted data. By using paging, you can access a maximum of 50,000 results. This is the recommended value if an ordered result is not required and has been optimized for search latency.”

Unsorted and latency are two important words in this text. Because the retrieved data is unsorted, it’s important to sort the records to get an accurate timeline for the audit events. Look at the difference in the timestamps for the first and last records in the examples shown above when we sort the data.

You could argue that the difference is small, but accuracy is important in compliance operations, and assembling actions into an exact timeline is part of achieving accuracy.

Latency is important because Microsoft is aware that the Search-UnifiedAuditLog cmdlet has a checkered history of reliability since its introduction. Some of the issues are due to timeouts when the cmdlet becomes unresponsive. Other issues include the return of duplicated events or failure to find available events.

It’s common to find that organizations routinely strip audit events from the log to ingest into an external repository. Sometimes this is for long-term (multi-year) retention, sometimes it’s because the external repository offers superior search and analysis features for the audit data. No matter what the reason is, the fact is that organizations run scripts to strip and reuse audit data. The scripts they use perform catch-all searches to find every audit event generated in a tenant. And the scripts include a lot of error handling to make sure that they deal with cmdlet errors.

A New Audit Log Search

The “classic” audit log search feature in the Microsoft Purview Compliance portal uses the Search-UnifiedAuditLog cmdlet and suffers from some of the same issues. Microsoft believes that returning search data synchronously is part of the problem, which led to the introduction of a new audit log search in mid-2022. Background jobs run the searches and the results appear when the search completes.

Microsoft believes that the new audit search is a more intelligent and predictable way to perform audit searches, especially because Microsoft 365 workloads generate more data than ever before. In addition, on July 19, 2023, Microsoft doubled the retention period for audit events for accounts with Office 365 E3 licenses from 90 to 180 days. Accounts with Office 365 E5 licenses retain audit events for 365 days. Welcome as the doubling of the retention period is, it does add to the amount of data that an audit log search must process.

Asynchronous searches take longer. The Search-UnifiedAuditLog command described above took 37 seconds. The equivalent new search took 4 minutes and 15 seconds (Figure 1). The extra time is needed to set up and run the search.

The new audit log search takes longer than the classic audit log search does
Figure 1: The new audit log search takes longer than the classic audit log search does

Like any IT function, the time required to perform a task can vary depending on the amount of data and the service load at the time. My experience is that asynchronous searches can be quite slow, which can be frustrating when investigating a problem by looking for what’s available in audit data. On the upside, the new audit search allows up to ten filtered searches to run concurrently (per administrator account) and the data retrieved by each search can be exported for slicing and dicing to find the necessary data.

Microsoft Should Fix Search-UnifiedAuditLog

Although I don’t know the reason, I speculate that Microsoft might limit the results returned by Search-UnifiedAuditLog to:

  • Conserve resources.
  • Convince customers to run focused searches rather than catch-all scans.
  • Encourage customers to move to the new audit log search.

Two big problems exist if this is what’s happening. First, the total lack of communication. Apart from the announcement about the extended audit log retention period, Microsoft has been quiet on the topic of audit log searches. Second, no cmdlet is available to perform asynchronous audit log searches. Organizations can’t replace the Search-UnifiedAuditLog cmdlet even if they wanted to.

The sad thing is that an obvious solution exists. Microsoft should fix the Search-UnifiedAuditLog cmdlet to make it more robust and reliable, and even add the ability to conduct asynchronous searches. We’ve learned a lot about running audit log searches since 2015. It would be a pity to discard that collective experience and all the customer scripts that have been written over the last eight years.

About the Author

Tony Redmond

Tony Redmond has written thousands of articles about Microsoft technology since 1996. He is the lead author for the Office 365 for IT Pros eBook, the only book covering Office 365 that is updated monthly to keep pace with change in the cloud. Apart from contributing to Practical365.com, Tony also writes at Office365itpros.com to support the development of the eBook. He has been a Microsoft MVP since 2004.

Comments

  1. Tomas Valenta

    Very good article. Thanks very much, Tony.
    Would you have some time to talk about the Audit Logs with me? I’m working (for a client) on a script that will rely on Microsoft Premium Audit licensing (over 365-day audit retention), and I would love to share the experience.
    Reach out if you can, and in any case, have a great day.

  2. Eric

    Thank you so much for posting this. We store our historical usage of Power BI and were concerned about the dropoff in usage but now it all makes sense. Our estimate was that this change by Microsoft must have occurred mid-May of this year. Anyways, we were able to recreate some history based on your tip though our tenant is only retaining 90 days of audit activity for all users, most are E3. So if Microsoft changed it to 180 days it must not be an automatic change.

    1. Avatar photo
      Tony Redmond

      Do us all a favor and log a support call with Microsoft to protest the loss of data. It’s the only way that pressure is put on the development group to change their practices.

  3. Joseph

    Now on to a curiosity. I am always keen to let the tools and powershell “tell me more” about their use by using tricks like

    Get-Module -Name ExchangeOnlineManagement | Select-Object -ExpandProperty ExportedCmdlets | Format-List

    Now I expected to find Search-UnifiedAuditLog among the results of that command, but of course, it’s not there.

    I am feeling foolish yet again. I wanted to try to begin understanding WHY Search-UnifiedAuditLog is provided by ExchangeOnlineManagement, because in fact: that alone makes no sense to me AT ALL.

    It feels like the entire scripting space is just a huge hodge-podge to me. I am shocked that they get away with this and there’s no general hue and cry from the user space.

    I guess it is what it is. I leave feedback almost as often as I’m asked when perusing the “documentation” at M$ HQ, and believe me: it’s blunt and unforgiving.

    Thank you once again for being a voice of reason!

  4. Nuno Mota

    I’ve always been frustrated with the Search-UnifiedAuditLog cmdlet… This is such an important cmdlet for so many organisations, and yet, it is a nightmare to use. A lot of admins run searches without realising they are missing results, which can be very dangerous.

    I’ve been using “-SessionCommand ReturnLargeSet -ResultSize 5000” for ages now in order to return “all” results. However, I also have to:
    1) search in sets of 1h using -StartDate and -EndDate (given the large number of users and, therefore, log entries in my environment);
    2) generate a new SessionId for each of these 1h searches;
    3) and then use SessionId and SessionCommand to repeatedly run the search cmdlet (for the same 1h time frame) until I get zero results returned…

    Am I retrieving all audit log entries using this method? I believe so, but who knows!

    1. Avatar photo
      Tony Redmond

      There are a bunch of methods used to retrieve audit data, but that sounds like a reasonable approach for fetching large quantities of audit data for storage elsewhere.

  5. Martin

    Thanks very much for sharing Toni. That’s exactly the challenge, I’m facing right now.
    The Compliance Department wanted to get some proof, that our implemented Retention Policies work and therefore, I wanted to get proof of “deletion by Retention Policies”.

    Do you have any idea, what kind of events mark “deleted by retention”.
    Or is it just a “FileDeleted” (new: “FileRecycled”) Entry, but executed by the System Account…?

    And to the end: is Microsoft aware, that we stuggle with these “non communicated” changes in the Unified Audit Log Searches…? Do you have anyone contacted at Microsoft regarding the challenge, we’re all facing out here…?

    Thanks and best regards,
    Martin

    1. Avatar photo
      Tony Redmond

      The events should be FileRecycled, but I would need to check. This should be easy to verify by running a search when you know that some items have been removed by retention policies (check the sites to see when documents are removed) and then examining the content of the audit events.

      And I have been talking to Microsoft about this issue. The development team that deals with audit events is very aware of my feelings on the topic and know exactly why I published this article to highlight the problem.

Leave a Reply