Practical Graph: Using GitHub Copilot for Microsoft 365 PowerShell Development

Is AI Assistance Worth $10/Month for PowerShell Development?

As some of you might have noticed, I am fond of writing PowerShell to automate Microsoft 365 operations. Over the last few years, I’ve accumulated quite a few scripts in the Office 365 for IT Pros GitHub repository. I am not a professional developer and writing code has always been a hit-and-miss affair, especially when learning about new cmdlets, like the many thousand in the Microsoft Graph PowerShell SDK. Some months ago, Andy Schneider reported good results using GitHub Copilot to upgrade scripts to use the Microsoft Graph PowerShell SDK. Given that I use the SDK a lot, everything pointed to a tool well worth trying, which is what I’ve been doing for the last four months.

At $10/month for a GitHub Copilot personal license, the investment is a no-brainer. GitHub says that predictive text helps developers complete tasks 55% faster. In other words, the AI should help me save 30 minutes for a script that normally takes an hour to write. This should release time for coffee drinking. I use the Copilot integration with Visual Studio Code, which works very well.

Copilot Basics

GitHub Copilot follows the same approach as taken by other Microsoft Copilots. Users interact through applications (like Visual Studio Code) to input prompts. The prompts tell GitHub Copilot what to do. Because we’re dealing with code, the prompts cover things like creating a function or writing a script or program. GitHub Copilot processes the prompt against its Large Language Model (LLMs).

GitHub trained its LLMs using natural language text and code stored in public GitHub repositories and other publicly-available sources. This is important because it means that GitHub Copilot excels when it can work off a substantial base of known code. For instance, the prompt shown in Figure 1 asks GitHub Copilot to create a PowerShell function to format a size in bytes and output the size in KB, MB, or GB.

Similar functions have likely been created many times before (like the one in the script to create a report about files in a OneDrive for Business account), so GitHub Copilot has no difficulty processing the prompt to generate the Convert-BytesToSize function:

function Convert-BytesToSize {
    param (
        [Parameter(Mandatory = $true)]
        [long]$Bytes
    )

    $Kilobytes = $Bytes / 1KB
    $Megabytes = $Bytes / 1MB
    $Gigabytes = $Bytes / 1GB

    if ($Gigabytes -ge 1) {
        return "{0:N2} GB" -f $Gigabytes
    }
    elseif ($Megabytes -ge 1) {
        return "{0:N2} MB" -f $Megabytes
    }
    elseif ($Kilobytes -ge 1) {
        return "{0:N2} KB" -f $Kilobytes
    }
    else {
        return "{0:N2} bytes" -f $Bytes
    }
}

Being able to create functions like this in a couple of seconds is an impressive display of what AI can do when it has been trained to do the job. As we’ll see, it is less impressive when asked to cope with APIs where fewer examples are available to train its LLMs. This is especially true for some of the more recent Microsoft Graph APIs.

For instance, Microsoft recently updated the Planner Graph API to support application permissions. There aren’t many examples of using the Planner Graph API published. Practical365.com has some articles and example scripts, like synchronizing tasks from the Microsoft 365 admin center and an incomplete task analysis. However, although the scripts are stored in GitHub, they and other examples might not have been present when GitHub built its LLMs.

After the LLM processes the prompt, GitHub Copilot returns its results to the application. The user can accept or delete the code and try again until the generated code is satisfactory.

There’s no doubt that GitHub Copilot takes care of some mundane tasks very well, like inserting comments into code. It also learns quickly to code like the signed-in user. For instance, I use Write-Host commands in a particular way that GitHub Copilot is all too happy to replicate.

But copying someone’s style or filling in comments isn’t magic. GitHub Copilot serves developers writing in many languages on many platforms. PowerShell is just one of those languages and Microsoft 365 PowerShell is one of the platforms. Let’s look at using GitHub Copilot with the kind of PowerShell you’d expect to work with to automate Microsoft 365.

GitHub Copilot and Microsoft 365

I decided to ask GitHub Copilot to write a function to report the last signed in date for an Entra ID account. Figure 2 shows the prompt and the function generated by GitHub Copilot.

On the surface, the code looks impressive. But it’s not because it uses the Get User Activities Graph API, which is designed to retrieve activity details within apps. In short, this function just won’t work.

Successful use of all generative AI tools depends on the completeness of the prompt submitted by the user for processing. The AI hadn’t failed: I just wasn’t precise enough for th AI to figure out what I wanted. I therefore tried again with this prompt:

Write a function to return the last successful sign in date for an Entra ID user account based on the Entra ID signin logs

Figure 3 shows the result. Again, the code looks good, but again, it doesn’t work.

The problem this time is that GitHub Copilot had a little hallucination and created a cmdlet or function called Get-EntraIDSignInLogs that doesn’t exist. Perhaps it exists somewhere in a GitHub repository, but that’s no good to me.

At this point, you might think that I am ultra-critical about the ability of GitHub Copilot to generate code. I’m not. The code has structure and indicates the flow necessary to return the desired result, so it is helpful. The problem might be that Copilot doesn’t have enough information in its large language models to come up with either of two solutions that this human considered:

• Using the last successful signed-in date recorded for Entra ID user accounts.
• Using Entra ID sign-in logs to find the last successful sign-in event for a user. Programmatic access to sign-in logs requires Entra ID P1 licenses.

Taking the structure generated by GitHub Copilot, I could drop in some code to interrogate the Entra ID sign-in logs and get the job done. Having the structure already there saved some time. Maybe not enough for a coffee, but certainly some minutes.

The hand-crafted human code uses the Get-MgAuditLogSignIn cmdlet to find the last three sign-in events for a user and then extracts the first successful sign-in from the set to return the timestamp for the event.

function Get-LastSuccessfulSignInDate {
    param (
        [Parameter(Mandatory = $true)]
        [string]$UserId
    )
    $StartDate = Get-Date (Get-Date).AddDays(-14) -format 'yyyy-MM-dd'
    $EndDate = (Get-Date -format 'yyyy-MM-dd')
    
    # Retrieve the first three sign-in logs for the specified user account
    [array]$SignInRecords = Get-MgAuditLogSignIn -Filter "createdDateTime gt $StartDate and createdDateTime lt $EndDate and UserId eq '$UserId'" -Top 3 -Sort "createdDateTime DESC"

    # Make sure that we have successful
    $LastSignInRecord = $SignInRecords | Where-Object {$_.Status.errorcode -eq 0} | Select-Object -First 1

    # Return the date of the last successful sign-in
    if ($LastSignInRecord) {
        return $LastSignInRecord.CreatedDateTime
    } else {
        return $null
    }
}

Exchange Online Examples

Exchange PowerShell has been around since late 2006, so there are lots of Exchange scripts stashed in GitHub repositories. I asked GitHub Copilot to “Find all Exchange Online user mailboxes with a primary email address belonging to a specific domain.” GitHub Copilot responded with:

$domain = "example.com"
$mailboxes = Get-EXOMailbox -Filter "EmailAddresses -like '*@$domain'"

foreach ($mailbox in $mailboxes) {
    # Do something with each mailbox
    Write-Output $mailbox.UserPrincipalName
} 

The code works if you update the $domain variable with a domain name. However, it’s poor code because it doesn’t filter for user mailboxes as requested and filters against the EmailAddresses property instead of the Primary SMTP address. In any case, it’s certainly something to work with.

Exchange Online administrators have written scripts to report mailbox sizes for years. I recently covered the topic, so it seemed like a good test for GitHub Copilot. Here’s the prompt I used:

Write a PowerShell script to find all Exchange Online user mailboxes and report the number of items and total item size (in MB) for each mailbox

GitHub Copilot generated this script:

# Connect to Exchange Online
Connect-ExchangeOnline

# Get all user mailboxes
$mailboxes = Get-Mailbox -ResultSize Unlimited

# Iterate through each mailbox
foreach ($mailbox in $mailboxes) {
    $mailboxName = $mailbox.DisplayName
    $itemCount = (Get-MailboxStatistics -Identity $mailboxName).ItemCount
    $totalItemSize = (Get-MailboxStatistics -Identity $mailboxName).TotalItemSize.Value.ToMB()

    Write-Host "Mailbox: $mailboxName"
    Write-Host "Number of Items: $itemCount"
    Write-Host "Total Item Size: $totalItemSize MB"
    Write-Host "------------------------"
}

It’s a very basic script that looks as if it might do the job but doesn’t. The Get-MailboxStatistics cmdlet doesn’t have a ToMB method, so the script fails for every mailbox with this error: “Method invocation failed because [Deserialized.Microsoft.Exchange.Data.ByteQuantifiedSize] does not contain a method named ‘ToMB’.”

The best that can be said about the generated script is that it could be a starting point for someone to attack a problem. But given that so many scripts have been written to solve this problem, I was disappointed that GitHub Copilot didn’t produce a better result.

Overall, GitHub Copilot is Worth The Cost

Like any generative AI, you cannot assume that whatever the AI creates is accurate and useful. As we’ve seen, the code generated by GitHub Copilot doesn’t work all the time, and when it does, the code probably needs to be tweaked to achieve the desired result.

This might be something specific to dealing with PowerShell cmdlets from Microsoft 365 modules because I don’t see the same problems with general-purpose PowerShell. My success ratio in generating good code might improve with better prompts too.

The best advice I can give is that you should try GitHub Copilot if you’re not already using it. I suspect that if I was a professional developer I would find more value to extract from GitHub Copilot. For now, I am happy with the assistance delivered by the AI and consider it well worth $10 per month. As time passes, the ability of GitHub Copilot to generate good PowerShell code for Microsoft 365 will likely improve and then the price will seem like a real bargain.