Microsoft Teams Private Chat Migration Challenges Explained

Teams Migration Hurdles

We see a lot of tenant to tenant (T2T) migrations with several thousand users – often tens of thousands of users. These large T2T migrations face challenges when it comes to Microsoft Teams private chat migrations. The primary challenges are throughput performance and content fidelity. This post outlines the difficulties in migrating Teams private chats so you can prepare your project plan accordingly.

The idea for this post originated from the very active live Q&A in the recent TEC Talk: Lessons Learned in Merger and Acquisition Microsoft 365 Tenant Consolidation Projects. Below is the original question and answer, of which we’ll dive into the details more in this post.

Attendee: The largest challenge I have been facing is migrating 1:1 chats from one Teams tenant to another. Is there a clean and concise way to accomplish this? 

Answer: We agree, this is perhaps the largest challenge right now.  Each chat, and reply, needs a API call to preserve all the chat metadata.  The best advice is to try to limit the amount of chat that is being migrated.  Some tools let you only migrate the previous 2 weeks in the format with metadata, and migrate the rest to a text file.  Reading 1:1 chat often isn’t the issue, writing is.  This allows for the most recent work data to migrate over quickly, but the other items go into a format that is quicker to migrate.

Why is Throughput Performance for Microsoft Teams Private Chat Migration Slow?

Many more private chat messages exist today than 2+ years ago because of the incredible growth in Teams during the pandemic. Some large organizations with tens of thousands of users may have millions of private chat messages to move in a Teams migration. This may take months to complete. The challenge in an Office 365 tenant to tenant migration is that there are more private chat messages to migrate than most organizations ever imagine.

But three problems exist – not one – in a chat migration project!

Problem 1: Counting Private Chats

The first problem is to discover all the private chats. This discovery process can be quick or very time-consuming based on the chosen solution.

One solution to counting private chat messages is to call the getTeamsUserActivityCounts method in the Graph API to return the number of messages. However, it can only return the number of messages in a specified period (D7, D30, D90, and D180). It does not return the total number of private chats or messages. These results can be used if you are willing to extrapolate the data and use an estimation.

A second solution is to use the getAllMessages method in the Graph API. Below are the Export API queries to get all messages. This discovery process is very slow to run; but provides an accurate count at the run time. For some organizations, this discovery process must start running weeks before they plan to start the chat migration.

https://graph.microsoft.com/v1.0/users/{user A Id/chats/getAllMessages
https://graph.microsoft.com/v1.0/users/{user B Id/chats/getAllMessages

If user A has a conversation with user B, first call receives all messages from user A + user B messages, and second call receives all messages from user A + user B messages. So, this creates doubled traffic, if there is user C, third call creates even more traffic. And on the migration side it’s necessary to control duplicated messages so as not to migrate duplicated ones. Also, it’s necessary to request all the messages to check that all the chats are discovered even before the migration.

The deduplication effort is what provides a more accurate count for the chat migration. It’s hard to predict what the difference in count will be; but it could reduce the count by 50% or more.

foreach(user in allTenantUsers)
{
	savedMessages = getSavedMessages()
	discoveredChats = getDiscoveredChats()
	messages = SendRequest(https://graph.microsoft.com/v1.0/users/{user A Id/chats/getAllMessages)
	foreach(message in messages)
	{
		if(message is not in savedMessages)
		{
			saveMessage(message)
			chatId = message.ChatId
			if(chatId is not in discoveredChats)
			{
				saveChat(message.Id, chatId)
			}
		}
	}
}

Notes:

1. The SendRequest method is just pseudo code to send an HTTP Request.
2. You can define which users are in allTenantUsers. You only want to include mailbox-enabled users

A third solution is to use the Get-ExoMailboxFolderStatistics cmdlet in the Exchange Online PowerShell V2 module. You can use the cmdlet to extract the count of items in the TeamsMessagesData folder. The TeamsMessagesData folder contains the compliance messages for private chat messages. This solution is considerably faster than the second solution; but not as precise because there is no way to identify and remove duplicate messages. Thus, the count of messages may be higher than the number of messages that will be migrated. An example of how to use this method is documented on the Office 365 for IT Pros site.

Problem 2: Timing the move of private chats

The second problem is determining the best time to migrate the private chat messages. Most organizations want to finish the chat migration at a predetermined cutover point. This ensures that users receive the latest messages in the source tenant and the messages are added to the target tenant in the correct order. But the timing of this is difficult to achieve in a Teams migration. And there are options to consider:

1. Write the private chat messages to a folder in Outlook in the target tenant. However, writing these messages to the Outlook folder will not display the same messages in the Teams client. Thus, the messages are not available in for reading or searching in the Teams client. There is a lack of continuity with this solution that the users would have to accept.
2. Migrate all the private chat messages from source to the target so that they appear in Teams. This provides the most complete expected user experience. If possible, merge the messages in a private chat into a smaller number of messages. This will write faster since there are technically fewer messages to write to the target tenant.
3. Migrate the most recent messages and leave the remainder behind in the source tenant. This provides a partial user experience because not all the messages are migrated.
4. Migrate the most recent messages and write the remaining messages to an HTML file. The HTML file is stored in the Microsoft Teams Chat Files folder in the OneDrive of the user who initiated the original private chat and direct permissions assigned to the other users in the private chat. This solution also delivers a partial user experience, but it is better since all messages are available to the users. Users can open the HTML file to search for and read messages. The challenge for users is that they must search for messages in two places:
   1. In Teams chat
   2. in HTML files containing the archived chats
5. Write all the private chat messages to an HTML file. This also provides a partial user experience, but the user cannot access their messages directly in the Teams client (unless the HTML files are added later to a private chat. In addition, the user must search for messages in the HTML files containing the archived chats.

Note: The HTML file is stored in the user’s OneDrive and direct permissions granted to the other users in the private chat.

So which of the options would best suit your need to meet the migration deadline and meet your users’ expectations?

Here is some sample code to create a private chat in the target tenant and write messages:

discoveredChats = getDiscoveredChats()
foreach(chat in discoveredChats)
{
	CreateChat(POST https://graph.microsoft.com/v1.0/chats)
	
	savedMessages = getSavedMessages(chat.Id)
	foreach(message in savedMessages)
	{
		if(optionArchive)
		{
			saveToHtmlFile(message)
		}else
		{
			CreateMessage(POST /v1.0/chats/{chat ID}/messages/{message ID})
		}
	}
	if(optionArchive)
	{
		UploadHtmlFileToChat(chat.Id)
	}
	foreach(member in chat.Members)
	{
		addMember(POST https://graph.microsoft.com/v1.0/chats/{chat ID}/members)
	}
}

Notes: 1. The CreateChat and CreateMessage methods are just pseudo code to send an HTTP Request.

Problem 3: Throughput performance

The third problem is throughput performance. There are two Graph APIs available for reading and one available for writing. The APIs do not support batch processing. Batch processing would greatly improve throughput. That is, batch processing would allow for reading and writing of multiple messages in one process rather than reading and writing each message in one process. This is because Microsoft also applies throttling to the number of messages being processed. I will provide details on the limits in the next section.

For information on SharePoint throttling, please review my previous article on SharePoint Online Content Migrations which gives a high level overview of the problem that is also applicable here.

Why A Problem Exists With 100% Content Fidelity?

There are two APIs (Export and Graph) available to read private chat message content from the source tenant:

The SharePoint Migration Export (Asynchronous Metadata Read) API https://docs.microsoft.com/en-us/sharepoint/dev/apis/export-amr-api.

• Forces multiple reads of messages depending on how many chat participants there are
• Allows for incremental migration
• Does not support batch processing
• About the same fidelity of content as the Microsoft Graph API
• Faster for reading. See Microsoft Teams service limits using Teams Export API.

• Can find the private chats (and the chat ID) and read the chat messages.

The Microsoft Graph API https://docs.microsoft.com/en-us/graph/use-the-api

• Allows for incremental migration
• Does not support batch processing
• Slower for reading. See Microsoft Teams service limits using Graph API

• Cannot find the private chats (or the chat ID); nor find the messages in the private chats. Information about private chats is not available.

Thus, only the Export API can be used to find the private chats and read the messages.

Only the Microsoft Graph API can write private chat message content to the target tenant:

• Cannot impersonate the original owner of the chat message when writing the messages. This means that users don’t see themselves as the creator of the migrated private chat thread or the messages in the thread. The owner is the user account used to migrate the private chat messages.
• The write speed matches the read speed of the Microsoft Graph API (which cannot be used for chats and chat messages); but it is much slower than Export API’s read speed (which is used for chats and chat messages). See Microsoft Teams service limits using Graph API

Of course, there are other issues to consider with respect to chat message migrations:

• Message notifications. At this time, the notifications sent to users cannot be suppressed via an API method when writing private chat messages to the target tenant. This includes @mentions of users within the private chat messages. The result is that users can receive a huge number of notifications in Teams. The receipt of these notifications is a common complaint of users during the private chat message migration. Our advice is to ask users to disable notifications in the settings of the Teams client in their target tenant (Figure 1). Click on Custom to configure the individual notifications and turn all settings to off for the duration of the migration. However, this may not be possible if users cannot access the target tenant yet.

• Setting the author or owner on the message. This is not possible to do with the Graph API. The Import API supports setting the author or owner on messages, but only for channel messages. Figure 2 shows chat messages in a source tenant. The owner is clearly visible. Figure 3 shows messages after migration. The owner information is altered significantly because it uses the Migration account.

The migration account cannot impersonate the owners of the chat messages when it creates the migrated messages. Instead, the username and date/time are appended to the message when created in the target tenant.

• Skipping selected messages (e.g. bots). Yes, this would reduce the number of messages for private chats. We are not seeing extensive use of bots in private chats yet.
• Including guest (external) users during the migration of private chat messages. There are two scenarios to consider here:
  • Private chat threads started by the guest (external) user. The private chat message started by the external user cannot be read because the source message thread was created in the external user’s tenant. That is, access to the guest user’s tenant must be granted to read the source message. It may be possible to create a new private chat message thread in the target tenant; but start with a different owner of the first message. There is more research required here to determine what is the best solution.

Note: If these private chat threads are not migrated; the messages are lost to the migrated users.

• Private chat threads started by a user in the source tenant that include a guest (external) user. These messages can be read in the source tenant. The migration task can include or exclude any user in the thread – including guest (external) users. Thus, guest (external) users can be excluded from migrated private chat messages. They would not receive notifications or links to the new private chat. They would keep the original private chat – although it would be effectively useless.
• Merging messages allows for faster writing of messages on the target tenant. As mentioned earlier, there is a difference in requests per second (RPS) between reading with the Export API (200 RPS) and writing with the Graph API (20 RPS). Rather than allowing the writing task become a bottleneck in migrating private chats,  the source messages can be merged for faster writing on the target.

Figures 4 and 5 compare the effect of Not Merged vs Merged for private chat messages. Figure 5 shows the effect of merging a group of messages into one message to increase throughput. Writing individual messages will take considerably longer without this solution. Unfortunately, users often raise concerns about the merging of their messages. They prefer the same user experience on source and target. However, this preference must be weighed against the slowness of private chat migrations.

We did some performance testing to compare results of not merging to merging. We create a private chat with several thousand messages and measured the throughput performance of the first 1000 messages.

No other chat migrations were run in parallel for this test.

Simple small messages were used for the testing. More complex messages contain embedded images and file attachments – which take longer to process.

No throttling was experienced in the source tenant. A very large private chat migration (i.e., multiple chat migrations running in parallel) will likely experience throttling and reach an upper-performance limit for reading on the source tenant.

This test result showed that chat migrations using merged messages is about 8 times faster. This is the result when using the Export API for reading and Graph API for writing. As noted earlier, the Export API can read (REQUEST) messages at 200 requests per second (RPS) and the Graph API can write (POST) messages at 20 RPS.

Conclusion

In conclusion, we reviewed the performance issues for discovering and migrating private chats. We also reviewed the available APIs along with their capabilities. Then I reviewed some additional issues that impact user expectations and migration performance. I reviewed how merging messages can improve migration performance, but the migrated messages may raise concerns from users. I hope this review will help with your private chat migration project.

At Quest, my team overcame the problems posed by Microsoft Teams private chat migrations and I explain how in this blog post – Teams Chat Migration: How we got around the challenges of private chat.