Home » Exchange Server » Exchange Recovery: Failed DAG Member in Exchange Server 2010

Exchange Recovery: Failed DAG Member in Exchange Server 2010

In this tutorial I will demonstrate a recovery scenario for a failed Mailbox server that is a member of an Exchange 2010 Database Availability Group.  In this scenario the DAG has two members, EX1 and EX2.  EX2 has suffered a serious hardware failure and needs to be recovered.

With server EX2 down the each mailbox database in the DAG has switched over to EX1 and shows the following status information.

The Exchange recovery process begins by reinstalling Windows Server 2008 R2 on the new server.

Installing Windows Server 2008 R2
Installing Windows Server 2008 R2

Because this Exchange recovery is for a member of an Exchange 2010 DAG the server must be installed with the Enterprise edition of Windows Server 2008 R2.

Exchange 2010 DAG members require the Enterprise edition of Windows Server
Exchange 2010 DAG members require the Enterprise edition of Windows Server

After Windows Server 2008 R2 is finished installing log on to the server and complete the following tasks:

  • Configure the Timezone settings
  • Configure the Automatic Update settings
  • Configure the server with the same TCP/IP configuration as the previous server
  • Configure the server with the same name as the previous server (in this case EX2)
  • Join the server to the Active Directory domain

The next step is to install the Exchange 2010 pre-requisites for the Mailbox server role.  From an elevated PowerShell prompt run the following commands.

After the server has restarted we also need to install the Exchange Server 2010 SP1 hotfixes for Windows Server 2008 R2. These updates require another restart of the server.

Before installing Exchange Server 2010 on the server being recovered we first need to remove it from the DAG. On another Exchange 2010 server open the Exchange Management Shell and run the following commands.

First, determine which mailbox databases the server was hosting a copy of, the activation preferences, and any replay lag that was configured. In this example server EX2 hosted copies of Mailbox Database 01 and Mailbox Database 02.

Next, remove the failed server from each of the mailbox databases that it held a copy of.

Warnings will appear because the failed Exchange server EX2 can't be communicated with, however the change can be confirmed by repeating the earlier command.

Next, remove the failed server from the Database Availability Group. Run the following command in the Exchange Management Shell.

Note: in some DAG topologies this action will fail with an error “A quorum of cluster nodes was not present to form a cluster”. If that error occurs use the solution in this article – Unable to Remove Failed Server from DAG Membership in Exchange Server 2010

When you are ready to proceed with the Exchange 2010 install open a command prompt and run the following command from the directory that has the Exchange setup files located within.

When setup has complete and the server has been rebooted, add the recovered server back in to the Database Availability Group.

Then, taking note of any replay or truncation lag times, and activation preferences that were earlier identified, re-add the mailbox database copies to the recovered server. This process can take a long time depending on the size of the mailbox databases that need to be reseeded.

You can now verify that the databases have the same settings that were identified earlier.

The failed DAG member has now been recovered and the Exchange 2010 Database Availability Group is back to normal operation.

Paul is a Microsoft MVP for Office Servers and Services. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server. Paul is a co-author of Office 365 for IT Pros and several other books, and is also a Pluralsight author.
Category: Exchange Server

18 comments

  1. Nithyanandham.s says:

    It’s really an good article ,thanks a lot …….because it helped me a lot while at the time of recovering the failed server in dag ….. Keep posting

  2. Paolo says:

    thanks a lot … one of my DAG members suddenly failed … got to recover it and this article was a lifesaver.
    BTW Keep up with the good work …this site rocks

  3. Matthias Koller says:

    Hi all

    This is an excellent article, thanks to Paul.
    But there are some unclear steps for me, because we have a diffent dessign:
    – All servers are Exchange 2010
    – 3 Mailboxservers (all members of one DAG): installed on VmWare ESX
    – 2 servers with the CAS and HUB trasport roll on it (both are members of one Cas arrey): installed on VmWare ESX
    So as you can see, all our Exchange servers are installed on VmWare ESX. And this is it why the recovery of a DAG member would be different.
    I know that MS disadvises DAG on VmWare. But it is our dessign now, which I am not able to change for now.

    The firest steps for recovery are logic.
    – …
    – Remove the failed server from each DB.
    – Remove the fialed server from the DAG.

    Is there a practicable way to process a recover from a VmWare snapshot or do I have to rebuild the whole server first?
    The snapshot recovery is easy but then how to remove the DB copies from that recovered server completley?

    Any answer is highly appreciate. Sine I could not find any relayable ansers on this in the Internet.

    Kind regards

    • Snapshots are not supported. You should not take a snapshot or recover from snapshot for Exchange servers.

      Whether your servers are virtualized or not makes no difference to the recovery process for a completely failed DAG member except that you could deploy the new VM from template rather than manually reinstall the OS I guess.

  4. Anand says:

    Hi Paul,

    My server is with SP1. When we have to install SP1 on servers? I am recovering two DAG members only with Mailbox role. I have separate HUB/CAS server configured.

    Thanks,

  5. Edith says:

    Hi Paul:

    Maybe this is an obvious question, but there is something that is not enough clear for me. I’m recovering a failed DAG member, I did exactly what the article says, with only a little difference, I still have the old files in the DB and log volumes. I’m facing some problems adding the copy of the databases in the recovered server, so, my question is: Do I need to delete the old files, I just moved them to another volume in the same server)? … Sorry for my english, I’m still learning 🙂

    • Yes you need to re-add the database copies to the recovered server. As far as the DAG is concerned those copies were removed when that DAG member was removed. The existing files on your volumes will cause it to fail to add the new copy, and should be removed/moved out of the way first.

  6. Andy says:

    Hi Paul,

    Great article.
    Question….When left with one surviving DAG member, how would you remove this server from DAG safely and ensure all databases get mounted on it so that it becomes a stand alone mailbox server?

  7. Binary says:

    Hi Paul,

    I had to demonstrate Disaster recovery of Exchange 2010 in my company in cold disaster site. I have performed Point-In-time recovery to recover all exchange servers one HUB/CAS server and two Mailbox Server in DAG. All went fine I was able to access my blank mailbox. I have faced couple of issues which is not mentioned in above steps or may be it is not required if you are recovering one DAG member. Please correct me if I am wrong.

    1. After recovering server from scratch you are missing permission on your Admin account/service account on local server. You have to add your admin account/service account and other Exchange groups to appropriate local group before you go further in re-configuring your exchange server.for example Exchange Server, Exchange Server Services groups were missing from local groups.
    2. If you have installed your exchange server in customized folder like on D: drive instead of using native installation folder, this recovery will not select customized folder in this case you will have mismatch in standard if you are following any in your company.
    3. You have to have all drives added to new server what was there on your old mailbox server before you run setup /m:recoverserver switch otherwise setup will fail.

    I have faced all these issues in my recovery procedure, please let me know if all these are practical issue one can face during recovery?
    Also let me know if this is the right solution for cold disaster site solution?

  8. Cheerd says:

    I have 2 srv in one dag
    One failed and tried to recover but I can’t …. i saw the databases aren’t mounted …tried to mount but still not possible
    … one db ls service down and the other db in failed copy states
    So how do remove the srv including the db from the server

  9. Ali says:

    Hi Paul, its really an informative post from you. I am going through Exchange 2010 disaster but my scenario is little different from the above scenarios. I have two DAG servers and one of my DAG server’s database volume has gone as it was on SAN storage and due to disaster on SAN we are also unable to recover it. The server is there but the volume containing database has gone. Now please suggest from below two options for what option I should go for or suggest if you think there is some better option.
    1) Do I need to go through the whole above process of rebuilding the failed DAG again.
    2) To work on re provisioning a new volume and start making the databases on it again. (If yes then please let me know the steps)

  10. Ali says:

    Hi Paul,

    Need some further assistance from you.. Since we have to procure new storage and it will take two or three weeks, currently storage of my current active dag is just bleeding out very fast I have delete some logs but its not enough and I have very less space left. I will be grateful if you can guide me the fix for this situation. Below are the steps that I already have taken but its not serving the purpose.

    >Deleted old database logs.
    >Deleted unnecessary mailboxes to have some white space.
    >Archive the mail boxes with bigger volumes to have white space.

Leave a Reply

Your email address will not be published. Required fields are marked *