Practical Graph: All About Pagination and Fetching Data

Fetching Graph Data One Page at a Time

To maintain performance and conserve resources, Graph API requests return a limited set of data when you make calls to fetch information about Groups, Teams, user accounts, and other types of objects. The set fetched by the Graph is represented as a page of data that can fit a certain number of objects. The Graph developers choose a page size which is a compromise between fetching all available objects and performance. The usual default page for Graph API requests holds 100 objects but individual APIs might return more, depending on the complexity and size of each object. As we’ll discuss later, it’s possible to increase the number of objects fetched in a single operation with the upper limit being up to 999 objects.

Pagination is the process of retrieving multiple pages to fetch the full set of objects that match a request. For instance, a query to find all licensed user accounts in a tenant might find 2,000 objects. The initial response to the query contains the first page with 100 objects. To fetch the remaining 1,900 objects, the code must continue to fetch pages until all the objects are retrieved. Fetching a succession of pages to retrieve all available data is known as pagination.

Apart from stopping developers from making performance-intensive requests like attempting to fetch 25,000 user accounts at one time, pagination also supports recovery. If a problem occurs when fetching data, the code can resume by fetching the next available page instead of restarting at the beginning.

Using the NextLink URL to Fetch the Next Page

Microsoft documentation says “When more than one query request is required to retrieve all the results, Microsoft Graph returns an @odata.nextLink property in the response that contains a URL to the next page of results.” In other words, to fetch the next page of results, scripts can use a GET request with the nextLink URL. The URL contains a skip token to tell the Graph where it should commence to retrieve objects from.

Therefore, when a script fetches Graph data, the code must check for the presence of a nextLink to see if an additional page is available and continue doing so until the query returns a null nextLink. Here’s an example that fetches all the teams in a tenant and stores details of the teams for further processing. The code:

• Fetches the first page of data using the Invoke-RestMethod cmdlet. Notice the use of the escape (backtick) character when forming the Uri because a $ sign precedes the Graph filter qualifier.
• Details of the retrieved teams are inserted into a PowerShell hashtable. A hashtable can only contain two properties (key and value), so it’s suitable for storing the group identifier (GUID) and the display name for each team. If you want to store more information, create and populate an array or use a generic list instead.
• Examines the nextLink and if it is not null, uses a GET request using the URL in the nextLink to fetch the next page of data. As the code fetches each page, it updates the data in the hashtable.
• Continues to fetch pages until the nextLink is null. After processing the last page, the hashtable contains details of all teams.

$TeamsHash = @{}
$Headers = @{Authorization = "Bearer $token"}
$ContentType = "application/json"
$uri = "https://graph.microsoft.com/V1.0/groups?`$filter=resourceProvisioningOptions/any(x:x eq 'Team')"
$Teams = Invoke-RestMethod -Method GET -Uri $Uri -ContentType $ContentType -Headers $Headers
$Teams.Value.ForEach( {
   $TeamsHash.Add($_.Id, $_.DisplayName) } )
   $NextLink = $Teams.'@Odata.NextLink'
      While ($NextLink -ne $Null) {
      $Teams = Invoke-WebRequest -Method GET -Uri $NextLink -ContentType $ContentType -Headers $Headers | ConvertFrom-Json
      $Teams.Value.ForEach( {
         $TeamsHash.Add($_.Id, $_.DisplayName) } )
   $NextLink = $Teams.'@odata.NextLink' 
}

The same technique works with other Graph endpoints, including those used to retrieve documents from a SharePoint library, or sign-in information for user accounts. This article includes a useful PowerShell function to retrieve information from the Graph using pagination to fetch all available data.

Microsoft Graph SDK Cmdlets and Automatic Pagination

Graph SDK cmdlets run Graph API requests to fetch, add, update, and remove data. Because the cmdlets are underpinned by Graph API requests, they are subject to the rules of pagination. However, some of the most popular Microsoft Graph PowerShell SDK cmdlets such as Get-MgUser and Get-MgGroup support an All parameter to instruct the Graph to return all available pages. For instance, to find the set of Teams with the Get-Team cmdlet, the code is:

[array]$Teams = Get-Team -All

Behind the scenes, the Graph API request made by the cmdlet performs pagination to fetch all the pages holding information about teams automatically. Not having to write code to deal with pagination is one of the reasons why it is easier to use an SDK cmdlet than the equivalent Graph API request.

Paginating with Invoke-MgGraphRequest

Scripts can use the Invoke-MgGraphRequest cmdlet to run Graph API requests. In this case, the script must deal with pagination. Take this example of fetching audit log records.

$Uri = ("https://graph.microsoft.com/beta/security/auditLog/queries/{0}/records?`$Top=200" -f $SearchId)
[array]$SearchRecords = Invoke-MgGraphRequest -Uri $Uri -Method GET

After running the query, the results are in the variable defined for the output. The presence of the ‘@odata.nextLink’ and the URL to fetch the next page of objects is obvious. The top parameter tells the Graph that the page should include 200 objects. This is a larger value than the default (150). Often developers increase the default size to reduce the number of calls needed to fetch all objects. It’s also possible to reduce the page size to retrieve a smaller number of objects when you only want to retrieve a few objects. The maximum number of objects is usually 999, but you should check to establish the maximum (and minimum) page size supported by the APIs used by the script.

$SearchRecords | Format-Table Name, Value -Autosize

Name            Value
----            -----
value           {0065e0de-752c-468e-96d4-08dc3a40d3c6, 07cf0fb2-397c-4095-f824-08dc3a4516e0…
@odata.nextLink https://graph.microsoft.com/beta/security/auditLog/queries/5416a32f-c538-4966-9906-476c0972af5f/records?$Top=20…
@odata.count    200
@odata.context  https://graph.microsoft.com/beta/$metadata#security/auditLog/queries('5416a32f-c538-4966-...

The code to paginate to fetch all pages is straightforward and is much the same as the Teams example described above:

$NextLink = $SearchRecords.'@odata.NextLink'
While ($null -ne $NextLink) {
    $SearchRecords = $null
    [array]$SearchRecords = Invoke-MgGraphRequest -Uri $NextLink -Method GET 
    $AuditRecords += $SearchRecords.value
    Write-Host ("{0} audit records fetched so far..." -f $AuditRecords.count)
    $NextLink = $SearchRecords.'@odata.NextLink' 
}

The nextLink URL contains all the query properties (like Top or Filter) used in the original query. The exception is the ConsistencyLevel header used to retrieve Entra ID objects. If the query is complex (for instance, a check against the set of licenses held by user accounts), you must include the parameter in each request using the nextLink URL.

Pagination is the Norm

If you’re new to working with Graph API requests and are accustomed to being able to fetch all available data with a cmdlet, it can take a little while to realize that pagination is a fact of Graph life, especially if no one tells you what’s happening. Once you understand that the Graph uses pagination and you go through the process once or twice, the logic of the approach becomes obvious and pagination becomes second nature.