Office 365 Supervision Improvements

The Supervision Review feature has been available for a while in Office 365, even its “V2” version is two years old by now. Recently, Microsoft announced a refresh of the feature, introducing multiple improvements in both the setup and review processes. Some of these improvements have already been rolled out in your tenants, so in this article I will talk you through some of the updates. But first, here’s a quick refresher.

A quick refresher on Supervision

In a nutshell, the Supervision Review feature gives you the tools to monitor some, or all, of your employees’ communications. The reasons why you would want to do this vary from internal regulations to strict compliance requirements such as the FINRA rule 3110.

To configure Supervision in your tenant, you must create at least one Supervision policy, specifying the types of communication you want to monitor, which users or groups will fall under the scope of the policy and who will be responsible for conducting the review. Policy configuration is performed via the Security and Compliance Center (SCC), under the Supervision tab. Behind the scenes, a policy object and a rule associated with it are created and then pushed to the corresponding workloads. Once the policy has been successfully published, copies of the communication items will be stored in a special mailbox.

To check whether your employees are communicating in accordance with the regulatory requirements, the reviewers access the data stored in the supervision mailbox and upon inspecting each item, take an appropriate action. For additional information on configuring and using Supervision refer to the official documentation here.

New Supervision Review Policies

Now we’re up to speed on Supervision Review, let’s find out what’s new in the process of configuring Supervision policies. One major improvement is the ability to configure more granular conditions and exceptions, both in terms of the users that will be covered by the policy, as well as the actual items captured. The UI now features no less than 16 different conditions, in addition to the “directionality” (inbound, outbound or internal) controls, as shown on the screenshot below:

Microsoft listened to their customer feedback prior to this release and have introduced some new, useful capabilities. For example, the ability to scope supervision based on: the domain, the retention label applied to an item, or based on both pre-defined or custom sensitive information types. This makes it much easier for reviewers to focus on the items that matter most. Notably however, the UI could use some improvements when it comes to using the custom sensitive information types.

Another new improvement, currently only available in private preview, targets scenarios with offensive language use and seeks to solve any profanity issues by leveraging AI and ML models. The feature is called Intelligent filters and is far more advanced than the policies you can configure by leveraging the “Message contains any of these words” condition.

If you’re interested in participating in this preview, you can do so by sending an email to supervisionolpreview@service.microsoft.com and describe the scenario you’re trying to solve.

Lastly, and probably most important, Supervision policies now support messages exchanged in Microsoft Teams channels and private chats. By default, this is an enabled featured, however you do have the option to turn it on or off on a per-user basis. A screenshot is demonstrated below of how you can do so.

The local copy of Teams messages from the Team Chats folder in the user’s mailbox is used to capture those items. This is the reason why you can see a “msg” file type, and why they all have the subject as “IM”. An example item is shown in the next section. It is important to note that latency can be expected, unlike email communications which are captured in near real-time, Teams items are being processed once every 24 hours.

Improved reviewer experience

The other area Microsoft has focused on is the reviewer experience. Until recently, reviewers were only able to examine and act upon captured items via the Supervisory review add-in in Outlook or OWA. Apart from the fact that the new OWA version doesn’t support the Supervisory Review add-in yet, the OWA experience was spot on, the Supervisory Review mailbox is added automatically and all relevant content and actions are present.  

The experience in Outlook still has a lot of room for improvement. The main issue resides in the special recipient type used by the Supervisory Review mailboxes (SupervisoryReviewPolicyMailbox) and that the mailboxes are hidden from the GAL by default. Because of this, Outlook is not able to display these mailboxes. This combined with the fact that only folder-level permissions are granted to the reviewer, and Microsoft’s initial guidance instructions were incorrect, makes this a frustrating experience in Outlook for Admins. Microsoft have since corrected the erroneous instructions, however the process still remains tedious because it requires admin intervention to get the identity of the Supervisory mailbox and to update its properties and permissions.

To address the issues with the review experience, Microsoft have now released an Integrated review dashboard. This allows you to review items directly from the SCC UI (or the new Microsoft 365 Compliance Center). To access the new experience, select any of the Supervision policies you’ve created and press Open. You will then be taken to the Supervision dashboard where you will get a quick summary for the given policy on the Home page. On each panel, you can see the number of items pending review or already resolved, the list of users and groups under the scope of the policy, which workloads does the policy cover and who are the corresponding reviewers:

Under the Review tab, you will receive a list of all items that are still pending review. The items can be filtered out by tag: pending, compliant, non-compliant, and questionable, but you’ll notice that no other filtering options are exposed here. So this part of the experience appears to be inferior when using the Supervisory review add-in for OWA or Outlook, both of which allow you to perform searches against items stored in the corresponding Supervisory Review mailbox.

You can sort items by Type, Subject, Sender and Date. Selecting an item allows you to preview its content, including metadata. And, if additional details are needed, such as full header information, you can use the Download button to get a full copy of the item. You can use the navigation buttons to switch to the next or previous items, and of course this can also be done by selecting the corresponding item from the item list.

In the Review pane, you can tag an item and add a comment. After pressing Save, the item will be moved to the corresponding Compliant, Non-Compliant or Questionable filter view. Alternatively, you can directly Resolve the item by pressing the corresponding button on top of the Review UI. It’s important to note that there is a change in behavior compared to the experience in OWA and Outlook – once an item has been marked as resolved here, no actions can be performed on it.

The biggest improvement in the review experience is the ability to perform bulk actions by using the checkboxes to select some or all items, then pressing the corresponding button. In contrast, the experience in OWA or Outlook requires you to first select an item, press the add-in button to perform an action, then rinse and repeat for each item individually. Available bulk actions in the new experience include: tagging items, adding comments on items, resolving items.

The last part of the new review experience is the Resolved items tab, where you can see a list of all the items marked as resolved. You are again presented with the option to preview the message content, including metadata, and download a full copy if needed. In addition, a history of any actions performed against the item is also shown, which can also be found under the review tab.

As previously mentioned, the Resolve action is final in the new UI, meaning once an item is marked as resolved it cannot be tagged or commented on. These actions are still available when using the add-in-based review experience in OWA or Outlook. Another difference with the new experience is that any “view” actions are not being displayed in the History pane.

Auditing and reporting

The last set of improvements are focused on reporting and auditing the Supervisory review functionality. Like other sections in the SCC, basic insights are displayed in small widgets on top of the Supervision page. However, in my experience, the information provided isn’t always reliable, fresh data insights appear to take a few days to complete and for some reason it chooses to ignore the 2-year-old policy I have in my tenant. Since you can get those details via the Review experience we covered above, this is not a major issue.

The more important data is exposed via the Supervision report, which you can access by clicking the Supervision widget or directly using this link. The report gives you a daily view of the number of items at each stage of the review process, and unlike most of the reports we can generate in Office 365, it allows you to export few years’ worth of data. If you’re not fan of the UI, you can also use PowerShell to get some of the data via the corresponding Get-Supervisory*Report cmdlets.

On the auditing side of things, actions related to management of Supervision policies now flow to the Unified Audit Log in Office 365. Below is an example on how to query the Unified Audit log for actions:

Actions performed by reviewers do not flow into the unified audit log but can be examined via the reports or via the Get-SupervisoryReviewActivity cmdlet.

Conclusion

In this article, we did a quick review of some new and exciting additions to the Supervision feature in Office 365. The most significant of these additions is arguably the added support for capturing Teams communications, although numerous other improvements have been made to management of Supervision policies, reporting and auditing. On the reviewer side of things, some of the pains of the add-in-based review experience have been addressed by introducing a new UI in the SCC, allowing you to perform the review directly from the browser. That is, if you have the necessary permissions to access the corresponding section of the SCC.

As some organizations might not feel inclined to grant reviewers certain permissions, using the Supervisory Review add-in to work with the review items in either OWA or Outlook remains a viable option. Unfortunately, no improvements have been made for the latter scenario, and accessing Supervisory Review mailboxes in Outlook remains a multi-step process, with many of the steps requiring admin intervention.