Returning Limited Search Results Might Catch You Out
I’m a big fan of using the Search-UnifiedAuditLog cmdlet to interrogate Microsoft 365 audit data to discover what happens in a Microsoft 365 tenant. Examples of the usefulness of audit data are monitoring for events that might indicate that an attacker is trying to sneak into a tenant or checking for the addition of specific user accounts to the membership of teams. As Paul Robichaux has pointed out, the audit log isn’t perfect, but it’s still a great place to go looking when you want to find out who did what.
Having used the Search-UnifiedAuditLog cmdlet to interrogate the audit log since its introduction in 2015, I think I know most of the peccadillos of the audit log. However, recently I noticed that audit log searches didn’t work the way that I expected. Where searches used to return hundreds of events, now they max out at 120 events.
Update: On November 29, Microsoft finally issued a service health update (MO694194) to update customers that using the Search-UnifiedAuditLog cmdlet might not return all expected results. To date, Microsoft has not said if they will fix the problem. The only advice offered is to use the SessionCommand parameter as discussed below.
Checking Audit Events for a Day
For example, to discover new events or to understand how the different workloads capture audit events, I regularly run a search to find all events generated in my tenant for a day. This is how I discovered that Microsoft changed the name of the event captured when users delete SharePoint Online or OneDrive for Business files from FileDeleted to FileRecycled. Given that deleted files travel through a two-stage recycle bin, a logic exists to justify the rename. The problem is that Microsoft never told anyone when they renamed the event, which means that any script looking for these events promptly returned zero.
The kind of search I use is:
[array]$Records = Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-1) -EndDate (Get-Date) -Formatted -ResultSize 5000 $Records | Group-Object Operations -NoElement | Sort-Object Count -Descending | Format-Table Name, Count -AutoSize Name Count ---- ----- FileAccessed 24 MoveToDeletedItems 24 ListViewed 9 MailItemsAccessed 9 AccessedAggregates 8 Search 8
Depending on activity levels, the unified audit log for my tenant generates a few thousand audit events daily. Now, I get 120. Always 120. Never more than 120 audit events for a command that I have used for years. This is despite the command specifying that the search can return up to 5,000 records by passing that value in the ResultSize parameter. Microsoft’s documentation for the Search-UnifiedAuditLog cmdlet says:
“The ResultSize parameter specifies the maximum number of results to return. The default value is 100, maximum is 5,000.”
I know that the search returns a subset of the available records by checking the timestamp for the first and last records in the returned set. As you can see, the events span just over a three-hour period and not the full day requested.
$Records.count 120 $Records[0].CreationDate Wednesday 23 August 2023 19:39:09 $Records[119].CreationDate Wednesday 23 August 2023 16:21:21
Some other folks have run into the problem (here’s an example), but I’m surprised that no more fuss has resulted. This might be because people haven’t noticed that their audit searches aren’t returning full results.
Microsoft Platform Migration Planning and Consolidation
Simplify migration planning, overcome migration challenges, and finish projects faster while minimizing the costs, risks and disruptions to users.
Use the SessionCommand Parameter
As it turns out, there’s a simple solution. MVP Vasil Michev points out that if you add the SessionCommand ReturnLargeSet parameter to the search, you get a better result:
[array]$Records = Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-1) -EndDate (Get-Date) -Formatted -ResultSize 5000 -SessionCommand ReturnLargeSet $Records | Group-Object Operations -NoElement | Sort-Object Count -Descending | Format-Table Name, Count -AutoSize Name Count ---- ----- MailItemsAccessed 380 PlanRead 165 UserLoggedIn 125 MoveToDeletedItems 117 TaskListRead 112 TaskModified 107 Search 74 FileAccessed 63 TaskUpdated 33
The search now returns many more events and covers the anticipated period:
$Records.count 1526 $Records[0].CreationDate Wednesday 23 August 2023 19:37:21 $Records[1525].CreationDate Tuesday 22 August 2023 20:20:23
So what’s the problem? A couple come to mind:
- Microsoft has published no guidance to say that generic audit log searches return limited results.
- Microsoft’s documentation doesn’t recommend the use of the SessionCommand parameter in searches. Some of the examples show the use of SessionCommand ReturnLargeSet, but I can’t find a specific directive to use the parameter in the kind of searches that I’ve conducted since 2015. Apart from referencing SessionCommand in the context of managing large amounts of audit data, Microsoft’s page describing Best Practices for using Search-UnifiedAuditLog doesn’t address the topic either.
In a nutshell, it seems that Microsoft has changed the way the Search-UnifiedAuditLog cmdlet works without saying anything. The net effect is that scripts that perform audit log searches might suddenly return limited results that their users don’t notice.
Unsorted and Latency
Microsoft’s documentation for the Search-UnifiedAuditLog cmdlet says:
“ReturnLargeSet: This value causes the cmdlet to return unsorted data. By using paging, you can access a maximum of 50,000 results. This is the recommended value if an ordered result is not required and has been optimized for search latency.”
Unsorted and latency are two important words in this text. Because the retrieved data is unsorted, it’s important to sort the records to get an accurate timeline for the audit events. Look at the difference in the timestamps for the first and last records in the examples shown above when we sort the data.
You could argue that the difference is small, but accuracy is important in compliance operations, and assembling actions into an exact timeline is part of achieving accuracy.
Latency is important because Microsoft is aware that the Search-UnifiedAuditLog cmdlet has a checkered history of reliability since its introduction. Some of the issues are due to timeouts when the cmdlet becomes unresponsive. Other issues include the return of duplicated events or failure to find available events.
It’s common to find that organizations routinely strip audit events from the log to ingest into an external repository. Sometimes this is for long-term (multi-year) retention, sometimes it’s because the external repository offers superior search and analysis features for the audit data. No matter what the reason is, the fact is that organizations run scripts to strip and reuse audit data. The scripts they use perform catch-all searches to find every audit event generated in a tenant. And the scripts include a lot of error handling to make sure that they deal with cmdlet errors.
A New Audit Log Search
The “classic” audit log search feature in the Microsoft Purview Compliance portal uses the Search-UnifiedAuditLog cmdlet and suffers from some of the same issues. Microsoft believes that returning search data synchronously is part of the problem, which led to the introduction of a new audit log search in mid-2022. Background jobs run the searches and the results appear when the search completes.
Microsoft believes that the new audit search is a more intelligent and predictable way to perform audit searches, especially because Microsoft 365 workloads generate more data than ever before. In addition, on July 19, 2023, Microsoft doubled the retention period for audit events for accounts with Office 365 E3 licenses from 90 to 180 days. Accounts with Office 365 E5 licenses retain audit events for 365 days. Welcome as the doubling of the retention period is, it does add to the amount of data that an audit log search must process.
Asynchronous searches take longer. The Search-UnifiedAuditLog command described above took 37 seconds. The equivalent new search took 4 minutes and 15 seconds (Figure 1). The extra time is needed to set up and run the search.
Like any IT function, the time required to perform a task can vary depending on the amount of data and the service load at the time. My experience is that asynchronous searches can be quite slow, which can be frustrating when investigating a problem by looking for what’s available in audit data. On the upside, the new audit search allows up to ten filtered searches to run concurrently (per administrator account) and the data retrieved by each search can be exported for slicing and dicing to find the necessary data.
Microsoft Should Fix Search-UnifiedAuditLog
Although I don’t know the reason, I speculate that Microsoft might limit the results returned by Search-UnifiedAuditLog to:
- Conserve resources.
- Convince customers to run focused searches rather than catch-all scans.
- Encourage customers to move to the new audit log search.
Two big problems exist if this is what’s happening. First, the total lack of communication. Apart from the announcement about the extended audit log retention period, Microsoft has been quiet on the topic of audit log searches. Second, no cmdlet is available to perform asynchronous audit log searches. Organizations can’t replace the Search-UnifiedAuditLog cmdlet even if they wanted to.
The sad thing is that an obvious solution exists. Microsoft should fix the Search-UnifiedAuditLog cmdlet to make it more robust and reliable, and even add the ability to conduct asynchronous searches. We’ve learned a lot about running audit log searches since 2015. It would be a pity to discard that collective experience and all the customer scripts that have been written over the last eight years.
I mentioned this on other post, just saw this and thought was good to share it.
Here is my approach to solve this problem, I had something alike and wanted to share it with you, there was a lot of chatter on one specific parameter rendering the 5000 limit useless, within the 24 hours that is, so I created a 4 hour iteration ignoring the bogus parameter, hopes it helps you.
# ignore command “set-whatever” over 1000 hits every x hours
$global:day = (Get-Date)
# set start date at midnight
$hours = $global:day.TimeOfDay.TotalMinutes
$startdate = $global:day.AddMinutes(- $hours)
# set end date at midnight
$enddate = $startdate.AddHours(24)
#
$logsearch=@()
# iterate every x hours ignoring bogus operation
$increment=4
for($i=0; $i -le (24-$increment); $i=$i+$increment) { $i
$logsearch += Search-UnifiedAuditLog -StartDate $startdate.AddHours($i) -EndDate $startdate.AddHours(4+$i) -RecordType -SessionCommand ReturnLargeSet -resultsize 5000|? Operations -NotMatch “set-whatever”
}
#
you could add a group of operation restrictions by using “-in” operator if you have a bunch.
#
With Microsoft going away from Search-MailboxAuditLog at the end of the month I have been trying to get the same results from something else including the Search-UnifiedAuditLog and the GUI in Content Search. I might be missing something but I cannot get the same data out and cannot get any data for Shared Mailboxes. I have been looking for some articles to see if I have been missing something or finding alternatives but so far have not found any. Has anyone found a way to do this?
The search I am used to doing would be something like this:
Search-MailboxAuditLog -Identity sharedmailbox@domain.com -ShowDetails -StartDate 04/18/2024 -EndDate 04/20/2024 | export-csv c:\temp\SharedMailbox_04-22-24.csv -NoTypeInformation
I think the new one would be something like this:
Search-UnifiedAuditLog -RecordType exchangeitem -Identity sharedmailbox@domain.com -StartDate 04/18/2024 -EndDate 04/20/2024 | export-csv c:\temp\SharedMailbox_04-22-24.csv -NoTypeInformation
Microsoft seem to do this all the time where they deprecate something but do not have a replacement. Unless I am missing something this is a huge loss to track issues especially for Shared Mailbox where multiple people are doing things and they seem to generate questions on how something happened.
If someone has a way to get the same information I would certainly appreciate any information or a clue on what I need to do.
Have you tried running the Search-UnifiedAuditLog cmdlet with the SessionCommand ReturnLargeSet
For example:
Search-UnifiedAuditLog -RecordType exchangeitem -UserIds sharedmailbox@domain.com -StartDate 04/18/2024 -EndDate 04/20/2024 -SessionCommand ReturnLargeSet | export-csv c:\temp\SharedMailbox_04-22-24.csv -NoTypeInformation
Note that SessionCommand can return duplicate records. You can remove these by sorting on Identity or by using a HighCompleteNess search. See https://office365itpros.com/2024/03/26/high-completeness-audit-log/ for details. Or you can try the new AuditLogQuery Graph API (see https://practical365.com/audit-log-query-api/). The Graph API is how the Purview Audit Search runs its searches.
Hi. There is no -identity in Search-UnifiedAuditLog. At least not in my cmdlet.
UserIds…
Microsoft provided now Beta Graph cmdlets for AuditLogs.
New-MgBetaSecurityAuditLogQuery
Here is a good summary: https://www.invictus-ir.com/news/using-microsoft-graph-for-unified-audit-log-acquisitions
I used their module as a base for my own script.
Following Graph permision can be used: AuditLogsQuery.Read.All
There are also permissions for each service available, for example AuditLogsQuery-SharePoint.Read.All
We have an in-depth article covering the use of the new AuditQuery API coming soon. There are some funnies that we have found that we’re currently reviewing with Microsoft…
Very good article. Thanks very much, Tony.
Would you have some time to talk about the Audit Logs with me? I’m working (for a client) on a script that will rely on Microsoft Premium Audit licensing (over 365-day audit retention), and I would love to share the experience.
Reach out if you can, and in any case, have a great day.
Thank you so much for posting this. We store our historical usage of Power BI and were concerned about the dropoff in usage but now it all makes sense. Our estimate was that this change by Microsoft must have occurred mid-May of this year. Anyways, we were able to recreate some history based on your tip though our tenant is only retaining 90 days of audit activity for all users, most are E3. So if Microsoft changed it to 180 days it must not be an automatic change.
Do us all a favor and log a support call with Microsoft to protest the loss of data. It’s the only way that pressure is put on the development group to change their practices.
Now on to a curiosity. I am always keen to let the tools and powershell “tell me more” about their use by using tricks like
Get-Module -Name ExchangeOnlineManagement | Select-Object -ExpandProperty ExportedCmdlets | Format-List
Now I expected to find Search-UnifiedAuditLog among the results of that command, but of course, it’s not there.
I am feeling foolish yet again. I wanted to try to begin understanding WHY Search-UnifiedAuditLog is provided by ExchangeOnlineManagement, because in fact: that alone makes no sense to me AT ALL.
It feels like the entire scripting space is just a huge hodge-podge to me. I am shocked that they get away with this and there’s no general hue and cry from the user space.
I guess it is what it is. I leave feedback almost as often as I’m asked when perusing the “documentation” at M$ HQ, and believe me: it’s blunt and unforgiving.
Thank you once again for being a voice of reason!
I’ve always been frustrated with the Search-UnifiedAuditLog cmdlet… This is such an important cmdlet for so many organisations, and yet, it is a nightmare to use. A lot of admins run searches without realising they are missing results, which can be very dangerous.
I’ve been using “-SessionCommand ReturnLargeSet -ResultSize 5000” for ages now in order to return “all” results. However, I also have to:
1) search in sets of 1h using -StartDate and -EndDate (given the large number of users and, therefore, log entries in my environment);
2) generate a new SessionId for each of these 1h searches;
3) and then use SessionId and SessionCommand to repeatedly run the search cmdlet (for the same 1h time frame) until I get zero results returned…
Am I retrieving all audit log entries using this method? I believe so, but who knows!
There are a bunch of methods used to retrieve audit data, but that sounds like a reasonable approach for fetching large quantities of audit data for storage elsewhere.
Thanks very much for sharing Toni. That’s exactly the challenge, I’m facing right now.
The Compliance Department wanted to get some proof, that our implemented Retention Policies work and therefore, I wanted to get proof of “deletion by Retention Policies”.
Do you have any idea, what kind of events mark “deleted by retention”.
Or is it just a “FileDeleted” (new: “FileRecycled”) Entry, but executed by the System Account…?
And to the end: is Microsoft aware, that we stuggle with these “non communicated” changes in the Unified Audit Log Searches…? Do you have anyone contacted at Microsoft regarding the challenge, we’re all facing out here…?
Thanks and best regards,
Martin
The events should be FileRecycled, but I would need to check. This should be easy to verify by running a search when you know that some items have been removed by retention policies (check the sites to see when documents are removed) and then examining the content of the audit events.
And I have been talking to Microsoft about this issue. The development team that deals with audit events is very aware of my feelings on the topic and know exactly why I published this article to highlight the problem.