Copilot Usage Data Isn’t Sufficiently Granular, but the allInterActionHistory API Might Help
Several sessions at the recent European SharePoint Conference (ESPC) in Stockholm discussed the issue of how to retrieve solid data about Copilot usage in Microsoft 365. As noted by multiple speakers, the simple fact is that the usage data reported by Microsoft is not good enough.
The usage data, available via a Graph API and surfaced in the reports section of the Microsoft 365 admin center, shows Copilot activity across several apps. However, the usefulness of the data is handicapped by the fact that a user is considered active in an app like Copilot for Outlook if they have a single interaction daily. The same lack of granular insight is present for other applications where a single interaction counts the same as sustained work requiring many steps.
Another problem is that automatic interactions, like the summaries created by Copilot for Word when users open documents, generate signals that don’t reflect real user activity.
Because of the deficiencies in Copilot usage data, organizations that have adopted Microsoft 365 Copilot are looking elsewhere for insights. For instance, Ståle Hansen advocates using Viva Insights to review and understand how people are interacting with Copilot while Tatu Seppälä gave an interesting session explaining how to extract information for Copilot interactions from the audit log before visualising the data in Power BI. And of course, I’ve been active in this area by investigating whether compliance records can reveal more about how people interact with Copilot.
The allInteractionHistory Graph API
This brings me neatly to the Graph allInteractionHistory API, which proclaims that it can access “all Microsoft 365 Copilot interaction data, including user prompts to Copilot and Copilot responses. This API captures the user intent, the resources accessed by Copilot, and the response to the user for Microsoft 365 apps such as Teams, Word, and Outlook.” While I can’t be certain, the chances are that the API accesses the same datastore that’s used by Viva Insights to derive its reports.
To access the data, I created an Entra ID app, added consent for the permission, and uploaded an X.509 certificate. I then ran the Connect-MgGraph cmdlet to connect to the Graph SDK in app-only mode as follows:
Connect-MgGraph -AppId $AppId -TenantId $TenantId -CertificateThumbprint $Thumbprint -NoWelcome
To run a query to find Copilot interactions, you need to know the user identifier of the account and the time range to search for interactions. This information allows you to construct a URI for the request, which will look something like this:
https://graph.microsoft.com/beta/copilot/users/ce0e26f8-da88-4efa-90ad-d16df1d9500d/interactionHistory/getAllEnterpriseInteractions?$filter=createdDateTime gt 2024-11-04T00:00:00Z and createdDateTime lt 2024-12-05T00:00:00Z
You can refine the query further to focus in on specific Copilot apps. For example, to only find the interactions with Microsoft 365 Chat (BizChat), the query is:
https://graph.microsoft.com/beta/copilot/users/ ce0e26f8-da88-4efa-90ad-d16df1d9500d/interactionHistory/getAllEnterpriseInteractions?$filter=createdDateTime gt 2024-11-05T00:00:00Z and createdDateTime lt 2024-12-06T00:00:00Z and appclass eq 'IPM.SkypeTeams.Message.Copilot.BizChat'
Points to Consider About Using the allInteractionHistory API
Running a query to find Copilot interactions is like any other Graph request. Your code needs to handle pagination to fetch all available data. Some other points of note include:
- The new API is in beta and only supports application permissions. Before an app can access interaction data, it must have consent to use the AiEnterpriseInteraction.Read.All permission. Depending on what you do, the app might need other permissions, like User.Read.All to access license information.
- The account being queried must have a Microsoft 365 Copilot license. If they don’t, the query fails. This applies even if a user previously held a Copilot license and some interactions exist. I have no idea why this check exists because it adds no value. Many organizations switch expensive Copilot licenses between accounts by removing licenses from accounts that can’t or don’t use Copilot and reassigning the licenses to people who can exploit AI assistance. Insisting that accounts have Copilot licenses before their interactions can be reported is just silly.
- The API is very slow to retrieve data. It’s a beta API, so problems are expected, but being able to fetch 10 records at a time means that it can take a long time to retrieve all the information for a user (several thousand interactions have existed for my account over the last three months).
- No Microsoft Graph PowerShell SDK cmdlet is available yet. It takes time before the SDK generation process learns about the metadata for a new API to allow it to generate cmdlets. I expect a cmdlet to be available in a month or so.
After fetching all data, creating a report of what’s found is a matter of interpreting the content of each interaction to extract details like the Copilot app, whether the interaction is a user prompt or an AI response, and associated information like a document used by Copilot in its response. The information is similar to the data available from compliance records, albeit packaged in different forms. Figure 1 shows the kind of report that can be created. In this instance, I’ve used the ImportExcel module to output an Excel worksheet. The script creates a CSV output file if the module isn’t available.
I looked for automatic interactions to understand the influence of these events over-reporting. Here’s what I found in a 30-day analysis of my Copilot interactions:
Copilot interactions for Tony Redmond betweeen 05-Nov-2024 and 06-Dec-2024 Name Count ---- ----- Tony Redmond 526 Copilot in Word 181 Microsoft 365 Chat 172 Outlook 167 Copilot in Teams 1 338 of the 1047 interactions are automatic (32.28%) 526 of the interactions are user prompts (50.24%) 17.48% of the interactions are Copilot responses to user prompts
The high number of automatic interactions is striking.
You can download my script from GitHub.
Good Start, Lots to Do
It’s good to have an API that can return more granular information about Copilot interactions than is available from the usage reports API or audit records. Let’s hope that Microsoft improves the performance and removes the silly check for a Copilot license and that the API reaches production status in the near future.
