Microsoft’s Major Focus on Agents
After writing about the Graph allInteractionHistory API to report Copilot usage for an account, I was asked if an API to report agent usage is available. As you know, Microsoft is all over artificial intelligence at present, and agents are a way to customize Microsoft 365 Copilot. SharePoint custom agents are an example of customizing Microsoft 365 Copilot. The most recent development is a big push for Copilot agents with the January 15, 2025 “Copilot for All” announcement,
In this case, the reader wanted programmatic access to the agent usage data for Microsoft 365 Copilot that’s available in the usage reports section of the Microsoft 365 admin center. Microsoft 365 generates usage reports from a data warehouse that holds usage information gathered from workloads. Access to generate the reports is via the usage reports Graph API. Recently, Microsoft added a usage reports API for Microsoft 365 Copilot activity. However, the reports obtained through the usage reports API are limited and don’t contain anything about agent usage. Initial examination of the compliance records for Copilot interactions stored in user mailboxes revealed no further clues.
No Copilot Agents Mean No Usage Data
Although I checked out several tenants to establish if any trace of agent usage data could be found using the Graph API, I wanted to run some tests in my own tenant to confirm the findings. The usage data revealed no agent activity, so it was time to create an agent (or two) to do some testing. Cue some perusing of the Microsoft documentation to determine just how easy it is to create a declarative agent.
A declarative agent is a customized version of Microsoft 365 Copilot tailored for a specific scenario. Customization is accomplished by providing instructions (to tell the agent what to do), actions, and knowledge (like web sites and SharePoint sites for Copilot to find information). Once created, users can interact with declarative agents through apps like Microsoft 365 Chat (BizChat) and Teams.
This YouTube video by Microsoft’s Vesa Juvonen is helpful. It covers the basics of creating a declarative agent using the Teams Toolkit in Visual Studio Code. I followed much the same path as shared by MVP Paul Bullock about his experience of building an agent to help write blog posts. Given that I write a lot of articles, an agent that reasoned over articles published on selected web sites seemed like a good place to start. To be able to create and test a declarative agent with the Teams Toolkit, you’ll need an account with a Microsoft 365 Copilot license.
Creating a Declarative Agent with the Teams Toolkit
The combination of Visual Studio Code and the Teams Toolkit makes it easy to create a declarative agent (editing an existing agent is another matter). After the Teams Toolkit creates the necessary files for the agent, you can edit the files to change the behavior of the agent. The three important files that are changed are:
- The agent configuration file: Add details of the capabilities the agent has. Knowledge sources are an important capability. The declarative agent schema supports up to four targeted web sites for an agent together with SharePoint or OneDrive locations as knowledge locations (custom SharePoint agents are another example of targeted queries).
- The manifest: Update the agent name, publisher name, version number, and description.
- The instructions file: Write up to 8,000 characters in natural language to tell the agent what it’s expected to do.
After making the necessary changes, you can provision the agent to test it with BizChat (Figure 1). Provisioning sideloads the agent into BizChat to allow the developer to test the agent.
At first glance, things looked pretty good, and the agent seemed to work as expected. But then I noticed that the agent wasn’t using knowledge from https://office365itpros.com, one of the web search sites declared in the capabilities section of the declarativeAgentdev.json configuration file. This was odd. The site is indexed, and its 1,700-plus articles are available for anyone to find. On the upside, the agent could find and use articles from practical365.com.
Some probing found that Google search works for both sites. However, Copilot uses Bing for its web searches and Bing couldn’t find anything from office365itpros.com because it had not indexed the site. The solution is to use the Bing Webmaster tools to ensure that the site is indexed and available in Bing searches. After importing the site from the Google search console and waiting for Bing to index the site, everything started to work. As a side effect, DuckDuckGo also started to index and search the site properly.
Publishing the Declarative Agent
The next challenge is how to get the agent to other users. Provisioning limits access to the agent to the author. Publication makes the agent available to other users through the integrated apps section of the Microsoft 365 admin center or the Teams admin center. The option to publish the agent appears in the lifecycle section of the Teams Toolkit (Figure 2).
Publishing an agent creates a ZIP file called appPackage.dev.zip in the build directory of the folder where you store the agent files. To deploy the agent to users within your tenant, go to the Integrated Apps section (under Settings) in the Microsoft 365 admin center and use the Upload custom apps option to upload the ZIP file. The upload process extracts and processes the app manifest from the ZIP file and if everything is OK, you can go ahead and deploy it to some or all users in the tenant (Figure 3).
Publishing the Declarative Agent to Teams
An obvious advantage of the Teams Toolkit in Visual Studio Code integration is that it takes care of the publication of the agent as a Teams app. To make sure that the process of approving the app in the Teams admin center works, you must make sure that the Shared Copilots Teams app is unblocked.
After the app is published from Visual Studio Code, you can go to the Manage apps section of the Teams admin center and make the app available to users. Due to caching, it takes some time before a newly-published app is available in the Teams admin center, but once it does, you can publish it to make the app available in the tenant app store. Users will then have to wait for up to four hours before they can install and use the app. The documentation makes it seem that everything happens much faster, but you should plan for delays to allow apps to slowly make their way through the Microsoft 365 plumbing.
Creating Declarative Agents with Copilot Studio and BizChat
Although most of my time focused on using the Teams Toolkit approach, I also looked at creating agents through Copilot Studio and Microsoft 365 Chat.
Copilot Studio is described as “a graphical, low-code tool for both creating an agent—including building automation with Power Automate—and extending a Microsoft 365 Copilot with your own enterprise data and scenarios.” Copilot Studio covers much more than declarative agents and is the tool of choice for creating complex agents. It was straightforward to develop a declarative agent in Copilot Studio (Figure 4). As a shortcut, I reused the same instructions and knowledge sources from the Teams Toolkit agent.
You can test the agent inside Copilot Studio and once it’s ready, publish the agent to “channels” like Teams (not channels in Teams, just Teams as a channel for publication). Here the availability options for the app (Figure 5) include sharing the agent with a link that can be sent via email or posted in a Teams message or submitting the app for review in the Teams admin center.
The app submission process works smoothly and the same process in the Teams admin center is needed to publish the app to users.
Creating an Agent in Microsoft 365 Chat
The last method I tried was to create an agent inside Microsoft 365 Chat (Figure 6). This option launches a simplified version of Copilot Studio that can be much the same in terms of configuring a chat agent to reason over specific knowledge sources using the same instructions. One obstacle is that I couldn’t find any way to configure specific external web sites as sources of knowledge despite this capability being available for agents created using Copilot Studio or the Teams Toolkit.
Publication of agents created in Microsoft 365 Chat is by sharing a link to the agent, which looks something like this:
https://m365.cloud.microsoft/apps/?templatedAppId=9e2962b8-7eae-4124-86f3-1799090725e1&templateInstanceId=8f269e3e-9a42-43dd-a0ad-42aa41d6ed27&environment=Default-a662313f-14fc-43a2-9a7a-d2e27f4f3478&source=embedded-builder
Anyone receiving the link can install the agent into the set of agents available in Microsoft 365 Copilot and use it there (if they have a Microsoft 365 Copilot license). It’s a very personal way of sharing an agent with others.
No Better Answer for Agent Usage Data
After trying out the Copilot agents created through the Teams Toolkit, Copilot Studio, and Microsoft 365 chat, I found that interactions with custom declarative agents running in the context of an app, like Microsoft 365 chat, are captured like any other interaction with that app, just like “normal” human requests and AI responses. However, agents created with Copilot Studio and run within Teams did log compliance records that can be identified as belonging to a custom agent. At the top of the MFCMAPI listing shown in Figure 7, you can see that I prompted a custom agent (shown with its app id) and received a response, this time using the agent name.
I didn’t see any data reported by the allInteractionHistory API for the interaction shown in Figure 7, so some inconsistency exists about the capture of data for custom agents. In terms of the Microsoft 365 admin center, I could see some raw numbers for usage by “agents built by your org,” but no details. Let’s hope that Microsoft updates the usage reports API to expose more information for agent activity.
The other thing I learned is that creating, provisioning, and deploying a declarative agent requires more time than you might imagine. Even discounting the time spent figuring out why Bing ignored the office365itpros.com site, I spent more hours than I care to think about playing with agent configurations to understand out how things work. If you get into the agent business, expect to dictate some serious effort to figure out how best to build and manage custom agents for your tenant.
We’re still in the early days of agent development and management and things will probably get easier over time. I keep on saying that about Microsoft 365 topics. One day, I shall be right.
