Why Graph Throttling Doesn’t Seem to Affect Graph SDK Cmdlets
A recent discussion about handling throttled Graph requests caused a question about why throttling doesn’t appear to happen when Microsoft Graph PowerShell SDK cmdlets run requests. Well, that assumption is incorrect. Graph SDK cmdlets are throttled, but in a different manner. Let’s discuss.
The Retry Handler
The Microsoft Graph service uses throttling to control the demands made on the service by requests issued by client applications. Microsoft Graph PowerShell SDK cmdlets, including Invoke-MgGraphRequest, use the Graph SDK retry handler to deal with rate limiting or transient situations where their requests result in a 429 status code response (or “too many requests”). In this situation, the Graph service has throttled the client application to prevent its requests flooding the system and impacting other applications.
Because different throttling limits apply to different services, the problem can surface more often in some scenarios when using certain APIs. For instance, because the Devices and Intune APIs implement relatively low thresholds, requests using these APIs can experience throttling when handling large numbers of objects.
Combining multiple endpoints in a request can also provoke throttling. For example, the request shown below accesses the User endpoint to retrieve the first three properties selected in the request, and the signInactivity resource to fetch the sign-in activity for the account. It’s a common scenario to want to report details of user accounts and their sign-in history, and in this case both endpoints apply the same threshold for the identity platform.
The identity platform threshold is more complicated than other workloads because it uses different values for transactions and applies different thresholds based on tenant sizes. In any case, this extract from a script (see below for full code) is a request that is almost guaranteed to provoke throttling when executed in a loop over more than 30 accounts (or the same account multiple times):
$Uri = ("https://graph.microsoft.com/v1.0/users/{0}?`$select=id,displayName,userPrincipalName,signInActivity" -f $User.Id)
$Data = Invoke-MgGraphRequest -Uri $Uri -Method Get
Usually, the Graph begins to throttle this transaction at between 22 and 25 requests. The number of transactions that run without throttling depends on the client application activity observed by the service and the endpoints accessed. In this case, if the lookup for sign-in activity is removed, requests made against the User endpoint are not normally throttled until after a significantly higher number of requests, if at all. The reason why is explained by the much reduced cost of the single-endpoint request compared to its dual-endpoint counterpart.
Attempting to Avoid Graph Throttling
When a 429 condition happens, the retry handler backs off and retries the transaction after a short period. If the retry is unsuccessful, the handler increases the wait interval and retries again. This mechanism is designed to allow the throttle condition to ease and normal processing to resume. From a user’s perspective, they might report that the script freezes and becomes unresponsive before processing some more data. This cycle of freezing and resuming is normal, and it happens because internally the retry handler has stepped in to manage the throttled request while honoring its settings in conjunction with the information in the Retry-After response header returned by the Graph service.
The Retry-After response header contains a recommended time for how long the retry handler should wait before making a new request. If you see scripts experiencing freeze and resume behavior, adding a small delay with the Start-Sleep cmdlet after processing each object or every few objects can be a good way to avoid throttling. Expanding on the code featured above to provoke throttling while fetching user account data, here’s an illustration of incorporating a pause after processing sets of five objects:
[array]$Users = Get-MgUser -All -filter "usertype eq 'Member' and accountEnabled eq true" `
-Property "id,displayName"
[int]$Pause = 2500
[int]$i=0
$Report = [System.Collections.Generic.List[Object]]::new()
ForEach ($User in $Users) {
$i++
Write-Host ("Checking user {0} {1}" -f $i, $User.DisplayName)
$Uri = ("https://graph.microsoft.com/v1.0/users/{0}?`$select=id,displayName,userPrincipalName,signInActivity" -f $User.Id)
Try {
$Data = Invoke-MgGraphRequest -Uri $Uri -Method GET -ResponseHeadersVariable $Response -ErrorAction Stop
If ($Data) {
$LastSignIn = $null
$LastSignIn = $Data.signInActivity.lastSignInDateTime
If ($null -ne $LastSignIn) {
$LastSignIn = Get-Date $LastSignIn -Format 'dd-MMM-yyyy HH:mm'
} Else {
$LastSignIn = "Never"
}
$ReportLine = [PSCustomObject][Ordered]@{
DisplayName = $Data.displayName
UserPrincipalName = $Data.userPrincipalName
LastSignIn = $LastSignIn
}
$Report.Add($ReportLine)
} Else {
Write-Host "No data found for user" $User.DisplayName
}
} Catch {
Write-Host "Error getting user" $User.DisplayName
Write-Host $_.Exception.Message
Continue
}
If ($i % 5 -eq 0 -and $i -ne $Users.count) {
Write-Host "Processed $i users, pausing for $Pause milliseconds..."; Start-Sleep -Milliseconds $Pause
}
}
[Note: I’d never write code like this that first creates a collection of user objects to process with a Graph request and then uses a Graph request for each object. All the information used by the script can be fetched with the first request, and that’s the right way to proceed.]
Pausing is no guarantee of avoiding throttling. As pointed out previously, service-specific thresholds are in force, and it all depends on the type and number of requests being made to the Graph service. The best way of avoiding throttling is to fetch all the objects to be processed in as few requests as possible and only make further requests when necessary.
In the case of the script above, a longer pause might be necessary, and if that doesn’t work and excessive demand provokes the Graph service to issue a 429 response, something like the following happens:
Checking user 52 Jack Smith (IT Coordinator) Error getting user Jack Smith (IT Coordinator) Response status code does not indicate success: TooManyRequests ().
The user object in question (Jack Smith) is not processed because although the script handles the error condition, it does not pause and retry the request. That’s simple enough code to implement but some retry code must be included to make sure that the script processes all objects.
Retry Handler Settings
The Get-MgRequestContext cmdlet reports the settings for the retry handler in the current session. These values apply to all Graph SDK cmdlets:
Get-MgRequestContext ClientTimeout RetryDelay MaxRetry RetriesTimeLimit ------------- ---------- -------- ---------------- 00:05:00 3 3 00:00:00
The values shown above are the default settings to control the HTTP client timeout (five seconds), the time in seconds between retries (maximum 180 seconds), the number of retries that Graph SDK cmdlets attempt (maximum 10), and the maximum time in seconds for retry requests. The value shown means the Graph uses the default of sixty seconds. If the Graph instructs the retry handler to wait for a longer interval than its retry delay setting, it complies with that request. Otherwise, the setting is used, even if it is longer than the value contained in the retry-after header.
You can change the settings for the retry handler by running the Set-MgRequestContext cmdlet before running Invoke-MgGraphRequest or any script involving Graph SDK cmdlets. For example, this command changes the number of retries from 3 to 10 and increases the time between retries to 5 seconds:
Set-MgRequestContext -MaxRetry 10 -RetryDelay 5
It’s usual to experiment with different combinations of settings before determining the best combination for a script. Be careful about making too dramatic a change. Instead, make small incremental changes until you find the sweet spot for your script. Remember that making changes to the retry handler settings won’t increase the speed of your code and doesn’t remove the need for code to handle the effects of throttling. But adjusting the settings might make sure that objects can be processed, even if the Graph imposes a little throttling.
Throttling Applies to All
The Microsoft Graph PowerShell SDK cmdlets (and the associated Microsoft Entra cmdlets) aren’t immune to throttling. Limits and thresholds still apply but are invisible most of the time. While not recommended, if you want to explore the limits and prove that they exist, you can force the Graph service to throttle the requests made by Graph SDK cmdlets. However, an easy life is preferable, which is why it’s usually best to use the automatic token refresh, pagination, and retry handler built into the Graph SDK cmdlets instead of running Graph requests with the Invoke-RestMethod or Invoke-WebRequest cmdlets.
