When a mailbox database copy has failed in an Exchange Server 2010 Database Availability Group (DAG) it may be necessary to reseed the mailbox server with the failed database copy.

Exchange 2010 DAG with a Failed Database Copy
Exchange 2010 DAG with a Failed Database Copy

To reseed the database copy launch the Exchange Management Shell on the server that is in a failed state.

First we need to suspend replication for the mailbox database copy on this server.  Use the following command, specifying the mailbox database in the format “<mailbox database name><server name>”.

[PS] C:\>Suspend-MailboxDatabaseCopy -Identity "Mailbox Database 01EX2"

Confirm
Are you sure you want to perform this action?
Suspending mailbox database copy "Mailbox Database 01" on server "EX2".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y

The mailbox database copy status will now change from “Failed” to “Failed and Suspended”.

Exchange 2010 Mailbox Database in Failed and Suspended State
Exchange 2010 Mailbox Database in Failed and Suspended State

Next we reseed the database with a new copy by issuing the following command.

[PS] C:>Update-MailboxDatabaseCopy -Identity "Mailbox Database 01EX2" -DeleteExistingFiles

The duration of the seeding process will depend on the size of the database and the speed of the network. When the update has completed it will automatically resume replication for the database copy. If you want to prevent automatic resume of replication use this command instead.

[PS] C:>Update-MailboxDatabaseCopy -Identity "Mailbox Database 01EX2" -DeleteExistingFiles -ManualResume

About the Author

Paul Cunningham

Paul is a former Microsoft MVP for Office Apps and Services. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server. Paul no longer writes for Practical365.com.

Comments

  1. Aussupport

    Hi Paul,

    Update-MailboxDatabaseCopy -Identity “AAP-DB2\Srv1” -DeleteExistingFiles

    A source-side operation failed. Error There isn’t enough free disk space to perform this operation.. [Database: AAP-DB2
    , Server: Srv1.domain.com]

    So how do i delete or make free space?

    As

  2. Omar

    in Ex2010 , when we delate Logs of one database in server1 in DAG; how to synch again ? thanks

  3. Derik Geddings

    Still a super relevant issue, I just resized some storage to gain space back after moving to new smaller datastores and had to reseed in order to do this but was getting this error. Thanks for the post.

  4. Pavan

    When I am reseeding database copy of A server from passive copy of B server. Passive server copystatus shows “seedingsource”. Exchange 2010 dag

    Can somebody help why it is showing like this.?

    Thanks in advance.

  5. Ahmed

    Hi Paul,

    I’d really thank you for useful info. you always publish and share.
    i have a small question we have a MBX 2013 with CU12 box with crashed “Ex replication service” suddenly occurred .
    we tried to remove the CU12 and re-install again with also remove the server from the DAG and add again, nothing worked out.
    the only solution helped out is to re-install the exchange again .
    do you have any clue what might be the root cause of this issue ?

    Thank you very much in advance.

  6. SAM

    I am getting issue in Exchange 2016 where lag database is having high replay queue length and secondly the content indexing status shows auto suspended. PLease see below screenshot for reference.

    DB1FR4012\MS-FR2-EXM-002 Healthy 0 14 2/14/2017 11:48:10 AM Healthy
    DB1FR2005\MS-FR2-EXM-002 Healthy 0 3422 2/14/2017 11:47:46 AM AutoSuspended
    DB1FR2006\MS-FR2-EXM-002 Healthy 0 10544 2/14/2017 11:48:17 AM AutoSuspended
    DB1FR2007\MS-FR2-EXM-002 Healthy 0 10706 2/14/2017 11:48:08 AM AutoSuspended
    DB1FR2008\MS-FR2-EXM-002 Healthy 0 10291 2/14/2017 11:48:13 AM AutoSuspended
    DB1FR2009\MS-FR2-EXM-002 Healthy 0 10913 2/14/2017 11:48:18 AM AutoSuspended
    DB1FR2010\MS-FR2-EXM-002 Healthy 0 10409 2/14/2017 11:48:17 AM AutoSuspended
    DB1FR2011\MS-FR2-EXM-002 Healthy 0 12619 2/14/2017 11:48:11 AM AutoSuspended
    DB1FR2012\MS-FR2-EXM-002 Healthy 0 2150 2/14/2017 11:48:16 AM AutoSuspended

    On other server we do not have any issues for lag copy, replay queue length is normal and content indexing is healthy.

      1. SAM

        Yes

        1. Avatar photo
  7. Rawdon Smith

    Hi Information in event log just prior to error …

    The active copy of database ‘I1-Database’ on this server is starting a seeding operation for the passive copy of the database on server SERVER2.

    Information Store (6744) I1-Database: An internal copy (for seeding or analysis purposes) is starting. The streaming ESE backup APIs are being used for the transfer method.

    Information Store (6744) I1-Database: Starting the copy or backup of the file I:ExchSrvMailboxI1-DatabaseI1-Database.edb (size 402594 Mb).

    Information Store (6744) Data transfer method is shared memory (1024 KB).

    Information Store (6744) I1-Database: Ending the internal copy (for seeding or analysis purposes) of the file I:ExchSrvMailboxI1-DatabaseI1-Database.edb. Not all data in the file has been read (read 4194304 bytes out of 422151028736 bytes).

    followed by

    Information Store (6744) I1-Database: The internal database copy (for seeding or analysis purposes) has been stopped because it was halted by the client or because the connection with the client failed.

    1. Avatar photo
      1. Rawdon Smith

        Hi

        Able to resolve by stopping and disabling third party services on the server in issue (ie Snap related services for NetApp and TSM) – we also stoppeddisabled the DPM services.

        Restarted server.

        Reseeded 4 databases using EMC overnight (DBs were all approx. 350 GB in size).

        Databases back to Mounted and Healthy state.

  8. Rawdon Smith

    Hi. Getting an issue on a RESEED exercise in an Exchange2010 DAG environment. I have 4 servers in a DAG with 4 databases on each. Two mounted DB servers and 2 Servers with copies on. One server is healthy and the other has its,4 DBs in FailedandSuspended state.

    Error: A source-side operation failed. Error An error occurred while performing the seed operation. Error: The requested resource is in use. Database….. Server.

    Exchange Management Shell command attempted:
    Update-MailboxDatabaseCopy -Identity DBServer -DeleteExistingFiles.

    Log files are mounting up.

    Am I safer to move mailboxes to another server with mounted and healthy databases and rebuild this exchange server?

      1. Rawdon Smith

        Hi Paul

        I think the issue may be backup process related – we use TSM for long term retention and DPM for weekday usage. Looking at TSM there were no sessions for the DAG or the server in the last day (as I would expect) and I thought DPM was rescheduled to avoid the reseed window but there may have been a process running. Presumably DPM would have a detrimental affect on a reseed process too?

        1. Avatar photo

          Seeding/reseeding relies on VSS. If a backup is running that is using VSS, then a seed/reseed operation can’t be performed at the same time.

          It doesn’t impact ongoing replication, only seeding/reseeding.

          1. Rawdon Smith

            Hi – have checked backups and that is not the issue – we are now getting …

            Information Store (6744) “Database”: The internal database copy (for seeding or analysis purposes) has been stopped because it was halted by the client or because the connection with the client failed.

  9. Jeff Flogerzi

    Thanks I love this site. Not being an exchange admin but being forced to tend to a simple 2 node 2010 install, this website is concise and to the point. Keep it up 🙂

  10. Sk. Mohammad Ali

    exchange 2010 database copy status failed and suspended the problem arise everyday. I update database copy everyday from EMC. Please give me a solution.

  11. Tran Phu

    Dear Paul,

    I followed these steps to fix my exchange 2010 SP3 (Window server 2008 R2 SP1) after I installed Firmware Package Version: 23.33.0-0043. However I still got that error as below:

    I used command fsutil fsinfo ntfsinfo d: , the Bytes Per Physical Sector : 512

    But I used the command eseutil /ml E0000386999.log to check the file log. It showed the message: Env Log Sec size: 4096 (actual @ log gen 512). I do not know how to fix this problem.

    Did I get a mistake here.
    Hope everybody can help me to this problem.

  12. James Lux

    Interesting. Updated from 2010 SP2 R8 to 2010 SP3 R12 last week and got this error now in eventlog and EMS only. EMC showed replication was OK and without errors.
    EMS displayed an communication error only for one of four replicated DB’s.
    Paused the Replication for that DB via EMC and re-enabled few seconds later. Sync was finished in seconds. So I guess replication was working before but some minor problem existed that resulted in the error. Thanks for your posting here!

    1. James Lux

      Well, after some hours, the DB Replication failed again for that database (only in EMS again). So I started a DB Copy with “-deleteExistingFiles” option. Takes some time with a 1TB Database…. Hope, it will run stable afterwards.

      1. James Lux

        This DB runs stable now. But an other DB repl went bad. This time with the error

        Event ID 2153
        The log copier was unable to communicate with server ‘ABC-2’. The copy of database ‘ABCDB2’ is in a disconnected state. The communication error was: Communication was terminated by server ‘ABC-2’: Data could not be read because the communication channel was closed. The copier will automatically retry after a short delay.

        In fact, it did not retry after a short delay. The Replication i fact war well but this error was still shown in management shell only.

        Found an other interesting blog that identified the MTU size of the dedicated Replication LAN-Interface as the potential problem.
        netsh interface ipv4 show subinterfaces
        shows that the Exchange LAN and Replication Interface have different MTU-sizes.
        Set bot Interfaces to MTU 1300 and paused/reenabled replication.

        Replication interface in is a Cross-Over LAN Link as the servers are in the same site.

        Hoping it will not run into error again.

        1. James Lux

          Problem is still happening at random… No idea what should be wrong – espeacially as it was working troublefree with SP2 RP8

          1. James Lux

            So, I disabled TCP Offloading and RSS on the replication NIC but after some troublefree days the error reoccoured. In eventlog I see that the replica went offline but got online and healthy shortly afterwards. But this change of state is not synced to the management shell output – opposite to the management console where everything is listed as healthy.

            There is nothing listed to the connection of the NIC’s anywhere.
            And it’s only happening for one of four databases. If there would really be a outage on the line, it would affect all DB’s not only one.

            I will ignore this from now on.

            In eventlog in Microsoft/Exchange/HighAvailability/Debug I see:

            10.03.2016 06:57:55 146 ReplayState ‘Disconnected’ for database ‘DB1(GUID-String)’ on node ‘ABC-2’ changed from ‘False’ to ‘True’

            10.03.2016 06:57:55 146 ReplayState ‘CopyStatusEnum’ for database ‘DB1(GUID-String)’ on node ‘ABC-2’ changed from ‘Healthy’ to ‘DisconnectedAndHealthy’

            10.03.2016 06:58:20 146 ReplayState ‘Disconnected’ for database ‘DB1(GUID-String)’ on node ‘ABC-2’ changed from ‘True’ to ‘False’

            10.03.2016 06:58:20 146 ReplayState ‘CopyStatusEnum’ for database ‘DB1(GUID-String)’ on node ‘ABC-2’ changed from ‘DisconnectedAndHealthy’ to ‘Healthy’

            In eventlog in Application I see:

            10.03.2016 06:57:55 2153 The log copier was unable to communicate with server ‘ABC-1’. The copy of database ‘DB1ABC-2’ is in a disconnected state. The communication error was: Communication was terminated by server ‘ABC-1’: Data could not be read because the communication channel was closed. The copier will automatically retry after a short delay.

            **sorry for all these comments but I hope, it may help someone**

  13. Tran Phu

    Hi Paul, hope you can help me to solve this problem:

    Database redundancy health check failed.
    Database copy: Mailbox Database 11
    Redundancy count: 1

    Error: Passive copy ‘Mailbox Database 11CSVCMBR2’ is not in a good state. Status: Failed.

    Name Status RealCopyQueu InspectorQue ReplayQueue CIState
    e ue
    —- —— ———— ———— ———– ——-
    Mailbox Datab Mounted 0 0 0 Healthy
    ase 11CSVCMB
    R1
    Mailbox Datab Failed 1447 0 1 Healthy
    ase 11CSVCMB
    R2

    ===============
    Full Status
    ===============

    Identity : Mailbox Database 11CSVCMBR1
    Name : Mailbox Database 11CSVCMBR1
    DatabaseName : Mailbox Database 11
    Status : Mounted
    MailboxServer : CSVCMBR1
    ActiveDatabaseCopy : csvcmbr1
    ActivationSuspended : False
    ActionInitiator : Unknown
    ErrorMessage :
    ErrorEventId :
    ExtendedErrorInfo :
    SuspendComment :
    SinglePageRestore : 0
    ContentIndexState : Healthy
    ContentIndexErrorMessage :
    CopyQueueLength : 0
    ReplayQueueLength : 0
    LatestAvailableLogTime :
    LastCopyNotificationedLogTime :
    LastCopiedLogTime :
    LastInspectedLogTime :
    LastReplayedLogTime :
    LastLogGenerated : 0
    LastLogCopyNotified : 0
    LastLogCopied : 0
    LastLogInspected : 0
    LastLogReplayed : 0
    LogsReplayedSinceInstanceStart : 0
    LogsCopiedSinceInstanceStart : 0
    LatestFullBackupTime : 4/15/2015 1:01:09 PM
    LatestIncrementalBackupTime :
    LatestDifferentialBackupTime :
    LatestCopyBackupTime : 12/2/2013 2:26:37 PM
    SnapshotBackup : True
    SnapshotLatestFullBackup : True
    SnapshotLatestIncrementalBackup :
    SnapshotLatestDifferentialBackup :
    SnapshotLatestCopyBackup : True
    LogReplayQueueIncreasing : False
    LogCopyQueueIncreasing : False
    OutstandingDumpsterRequests : {}
    OutgoingConnections :
    IncomingLogCopyingNetwork :
    SeedingNetwork :
    ActiveCopy : True

    Identity : Mailbox Database 11CSVCMBR2
    Name : Mailbox Database 11CSVCMBR2
    DatabaseName : Mailbox Database 11
    Status : Failed
    MailboxServer : CSVCMBR2
    ActiveDatabaseCopy : csvcmbr1
    ActivationSuspended : False
    ActionInitiator : Administrator
    ErrorMessage : The Microsoft Exchange Replication service e
    ncountered an error while inspecting the log
    s and database for Mailbox Database 11CSVCM
    BR2 on startup. Error: File check failed : L
    ogfile ‘D:MicrosoftExchange ServerV14Mai
    lboxMailbox Database 11E00.log’ is generat
    ion 2788053; however, the expected generatio
    n is 2788055.

    ErrorEventId : 2070
    ExtendedErrorInfo :
    SuspendComment :
    SinglePageRestore : 0
    ContentIndexState : Healthy
    ContentIndexErrorMessage :
    CopyQueueLength : 1447
    ReplayQueueLength : 1
    LatestAvailableLogTime : 2/21/2016 9:57:22 PM
    LastCopyNotificationedLogTime : 2/21/2016 9:57:22 PM
    LastCopiedLogTime : 2/21/2016 9:57:22 PM
    LastInspectedLogTime : 2/21/2016 9:57:22 PM
    LastReplayedLogTime : 2/21/2016 9:53:30 PM
    LastLogGenerated : 2789501
    LastLogCopyNotified : 2788054
    LastLogCopied : 2788054
    LastLogInspected : 2788054
    LastLogReplayed : 2788053
    LogsReplayedSinceInstanceStart : 0
    LogsCopiedSinceInstanceStart : 0
    LatestFullBackupTime : 2/21/2016 1:00:39 PM
    LatestIncrementalBackupTime :
    LatestDifferentialBackupTime :
    LatestCopyBackupTime : 12/2/2013 2:26:37 PM
    SnapshotBackup : True
    SnapshotLatestFullBackup : True
    SnapshotLatestIncrementalBackup :
    SnapshotLatestDifferentialBackup :
    SnapshotLatestCopyBackup : True
    LogReplayQueueIncreasing : False
    LogCopyQueueIncreasing : False
    OutstandingDumpsterRequests : {}
    OutgoingConnections :
    IncomingLogCopyingNetwork :
    SeedingNetwork :
    ActiveCopy : False

    I also got one problem after check it at event views with event ID 2070:

    The Microsoft Exchange Replication service encountered an error while inspecting the logs and database for Mailbox Database 21CSVCMBR2 on startup. Error: File check failed : Logfile ‘D:MicrosoftExchange ServerV14MailboxMailbox Database 21E00.log’ is generation 3521494; however, the expected generation is 3521523.

    And this with event ID 124:
    At ‘2/22/2016 10:43:58 AM’ the Exchange Information Store Database ‘Mailbox Database 21’ copy on this server encountered an error. Consult the event log on the server for other “ExchangeStoreDb” or “msexchangeRepl” events that may identify the specific failure. The Replication Service will automatically attempt to retry.

    Thanks a lot from your help.

      1. Tran Phu

        Hi Paul,

        It is really work, thank you so much!

      2. Tran Phu

        Hi Paul,

        I just think the mailbox is ok, but after I restart the server today, it showed failed message as following:
        The log copier was unable to continue processing for database ‘Mailbox Database 21CSVCMBR1’ because an error occured on the target server: Continuous replication – block mode has been terminated. Error: the log file sector size does not match the current volume’s sector size (-546) [HResult: 0x80131500]. The copier will automatically retry after a short delay.

        This was happened after I upgraded firmware the new version in Megaraid (LSI). Is there any way to solve it. Thank you!

  14. Ritesh

    Hey Paul,

    I am trying to reseed the DB but getting the error:
    A source-side operation failed. Error An err
    or occurred while performing the seed operat
    ion. Error: An error occurred while processi
    ng a request on server ‘MAIL***1’. Error:
    Couldn’t open backup file handle for databas
    e ‘Mailbox Database 2’ to server ‘MAIL***1
    ‘. Hresult: 0x50d. Error: A database backup
    is already in progress. Please verify that n
    o other seeding or incremental reseeding ope
    rations are started for this database, and t
    hen try the operation again by rerunning the
    Update-MailboxDatabaseCopy cmdlet..

  15. David

    I have questions on user impact. If a database copy has failed, is there any impact to the users that are conducting email business based on the first email server DAG member) and its mailbox database?

  16. Chaitu

    Hello Paul,

    Your articles are precise enough to easily understand! Appreciate the effort.

    We are going to have to rebuild one of the servers in a 2 Member DAG. I was planning on leveraging the RecoverServer switch for this with reference from the article https://technet.microsoft.com/en-us/library/dd638206(v=exchg.141).aspx.

    However, I have a question, as per the article above, it is recommended by Microsoft to remove any Database Copies on the server that is going to be recovered (in my case rebuilt). I do that before OS recovery and once the server is up and running, when I add back the database copies on the rebuilt server, is it ok to leave the Database and Log Files (before recovery) in the directories as they are? Will the seeding from active copy happen with the delta only?

    Cheers,
    Chaitu

      1. Chaitu

        Hello Paul,

        Thanks for the quick reply, just curious to know what would be the implications if I leave them in the same drive? I have ran this in a lab, it seem to sync from where it left off, but small databases in the lab, no particular way to figure out if it copied over the delta or the whole database altogether.

        Cheers,
        Chaitu

  17. TheK

    Thx Paul for ExchangeServerPro articles. Saved my day many times, you rock dude. Thanks for sharing your knowledge.

  18. Craig SCCM Noob

    Thank you very much for your help 🙂

  19. DaveZ

    Hi Paul, I have used this procedure many times in the past with great success. Thank you for that.

    This time I ran into an error:

    “A source-side operation failed. Error An error occurred while performing the seed operation. Error: The process cannot access the file ‘D:
    ExchangeDBDB11DB11.edb’ because it is being used by another process…”

    Not sure what this means, or how to remedy it.

    Thank you in advance!

      1. DaveZ

        Thank you Paul for your reply. Turns out I rebooted and then reseeded per your article. Worked perfectly.

        Thank you again for your article.

        Dave

  20. Garrett Michael Hayes

    I’m hoping you can help me understand a problem I’m having with a reseed. I’m trying to reseed a data store on one server (EX1) following a disk issue (target LUN was inadvertently disconnected briefly).

    The status on EX2 shows an error on one log file (says the file is corrupt). Checking the directory, it appears the file is actually not present. An attempt to reseed failed after several hours because of that.

    Is there a switch or setting to tell Exchange to bypass a missing log?

  21. Mark Adams

    Hi Paul,

    I have a DAG active and passive, with a separate replication network (172.x.x.x). If I reseed a failed DB will the reseed using the replication network rather than the data (10.x.x.x) network? The replication network is on a separate switch with no access to the data network.

    Thanks,

      1. Mark Adams

        Thanks Paul, one last question, I’m trying to reseed via the EMC by right clicking on the db, and selecting “Resume Database Copy” is starts then fails. The event log shows its missing a log file, but I’ve checked the log folder and the missing log is there, in the active log folder and the passive log folder?

        Thanks,

      2. Barton

        Hello Paul,

        I will need to take down the passive copy of my two-member DAG very soon for at least a good number of hours. I remember when I did this the last time, the queue got so big all efforts to reseed the passive copy failed. If worse comes to worse, should I simply delete the passive copy in ECM, delete the EBD database and logs on the passive server and then make a new copy in ECM? Would that work?

        Also keep in mind as long as the replication between the two Exchange servers are being held up, I cannot run a full backup and flush the transaction logs. That can also be a problem. Is there a way to stop all replication just so that I can run the backup and flush transaction logs?

        1. Avatar photo

          Suspending replication on the DB copy will still cause the copy queue to build up.

          If you know it’s going to be down for a long time and that it will impact your disk capacity and/or backups, I suppose you could just remove that database copy and re-add it later. Understand the risks of this, you’re reducing your copies of that database by 1…

          Otherwise, if a database copy won’t resume or is otherwise failed you only need to reseed it using the steps shown in the article above. No need to manually delete files or remove/re-add the DB copy.

  22. Ed Kummel

    Paul,
    Thanks for your continues support and information to the Exchange community. I rely heavily on the things you post.
    I have one question regarding reseeding the database.
    Recently, one of the drives in my JBOD DAS on one of my servers threw a predictive failure. The database never showed failed.
    The vendor replaced the drive and the new drive is now working fine.
    During all this, the mailbox databases affected by this JBOD DAS never showed “Failed”.
    Do we still need to reseed the database even though the database never failed in the first place?

    1. Avatar photo
  23. Gary

    Thank You, the PowerShell command resolved our problems as EMC wasn’t working.

  24. Dennis Hartmann

    Paul, I reseeded after the active copy got corrupted due to the logs filling up the logs HD. The DB failed over to the passive but the now the new active DB throws me this error: “The active copy of MBDB02 has a missing or corrupted log file (E06002526D1.log). To keep this copy healthy, replication will restart at generation 2434770. A full backup must be performed before an incremental backup is possible.” I’m currently performing a full backup but in the meantime, the CQL is rising steadily. Any idea what’s happening?

  25. Ndukaa

    Thanks Paul!

  26. Alexander

    Hi Paul,

    every time I move one of my copies,the copy status stuck at initializing and I cant reseed. I used your method but I get the following error:

    WARNING: The Microsoft Exchange Replication service has not responded to the suspend request yet. The service may not
    be running. Press CTRL+C to stop waiting for the service to respond. You can check to see if it is suspended by using
    the Get-MailboxDatabaseCopyStatus cmdlet in the Exchange Management Shell.

    1. Damon

      Alexander, ever get an answer to your issue with the DB showing as Iitializing? how did you resolve it?

  27. AKhil

    Paul,

    so can be Microsoft remove exchange on-premises in future .

    personally do you really thing that every customer will go for cloud or customer will demand for on-premises ?
    because as i aware every is going with hybrid not full cloud . what do you say?

    1. Avatar photo

      I can’t predict the future and I’m not going to just guess what Microsoft plans to do.

      Try to avoid guessing what “every customer” is going to do or is doing. There’s a lot of different customers around the world and they all have their own special needs and circumstances.

      I expect some percentage of customers will go full cloud, some will go hybrid, and some will always stay on-premises. Beyond that there is nothing I can predict.

  28. AKhil

    Hi Paul,

    I have some query which is mention below.

    1) will on premises exchange will be finished from industry
    2 ) is Microsoft planning to introduce next version of exchange for on premises or they will only consider on online exchange.
    3) is Microsoft planning to flush on- primises

    I known these are stupid query but for being exchange person we should be aware before some disaster , so that we can start leaning new technology.

    As i know maximum customers are deploying hybrid not full cloud that indicates that no one is prepare for full cloud, but we are scare.

    Please reply my query

    1. Avatar photo

      As far as I am aware Microsoft has publicly stated that they plan to deliver another version of Exchange for on-premises customers. We can probably expect to see a preview of that later next year and perhaps it will ship in 2016.

      I would not wait before learning new technologies.

  29. Jesheem

    Hey thanks for the tutorial.
    I am facing a situation wherein my Mailbox Database is in a healthy state. But the event viewer throws up the error “A source-side operation failed……”
    I did try the “Update-MailboxDatabaseCopy -Identity DB1MBX1 -CatalogOnly” command. But it still throws up the same error.

    Should I go ahead with the method mentioned here?

  30. Adam Fowler

    Great instructions. Although I didn’t use the -manualresume switch, it didn’t automatically resume replication. I manually resumed and it worked perfectly.

  31. Jay

    Hi – Your solutions worked fine but after multiple attempts as for some reason it was failing because of Win Rm service going down. Thanks a lot for your blog. keep writing 🙂

    on side note, do you know is there any way i can replace existing FILE witness server with new server ?
    we have two Exchange server running DAG and one file witness server (server 2003) and file server is failing for past few months and i was thinking to replace with latest 2008R2 File server.

    Thanks again

    1. Avatar photo

      You can change the FSW any time using Set-DatabaseAvailabilityGroup. You just need to set up the permissions first (ie add Exchange Trusted Subsystem to local admins group on the new FSW) and also be aware that in Failover Cluster management tool the old UNC path will still appear in the name of the FSW resource.

  32. Trra

    Hello Experts,

    My reseed has been running for days! 1 TB DB over wan link.. Can I run backup while reseed in progress?

    Thanks
    Tarra

    1. Avatar photo
  33. Philip Kitheka

    Hi Paul,
    I am facing a problem with DAG in exchange 2010 sp1 environment.

    I have 2 Mailbox servers having a couple of mailbox databases (active/passive). i have been trying the reseeding operation on DTDB1 database without success. below is the error. the db size is 201.6GB. kindly assist.

    A source-side operation failed. Error An error occurred while performing the seed operation. Error: An error occurred while processing a request on server ‘EXCH-DRS-SVR02’. Error: An I/O error occurred while attempting to access file ‘I:Program FilesExchangeHomeMailboxDTDB1DTDB1.edb’ on source server EXCH-DRS-SVR02. Error: SendSeedingDataInternal: PerformDatabaseRead fails with code 0xFFFFFC18. Expected 65536 bytes read 0.

    Copy Status: Failed and Suspended
    Content index state: failed
    Copy queue length (logs): 4718233
    Replay queue length (logs): 0

    Please suggest.

    Thanks in advance

  34. Imran

    Paul Paul Paul Paul Your the MAN…! Blesses you saved my day
    cheers

    regards,
    imran from afghanistan

  35. senthil

    What is the purpose of : MailboxDatabaseReseed.ps1 ?
    Is it doing the same job ?

  36. Anees Ur Rehman

    Hi Paul,

    I am facing a problem with DAG

    I have 2 Mailbox servers in San Francisco and DAG is working fine on both servers.
    Now I have to implement DAG on a Server which is in New York. Everything is ok. Servers are accessible. I am able to ping the DAG from every server. There is vpn tunnel between sf and ny zone.

    But when I started the replication it gives the following error from ecp
    Passive Failed and Suspended
    Copy queue length: 8067
    Content index state: Suspended

    Then I started to reseed from EMS, I got following error

    The seeding operation failed. Error: An error occurred while performing the seed operation. Error: Communication was terminated by server ‘SanFrMailBoX1’: Data could not be read because the communication channel was closed. [Database: SFDataBase, Server: newyorkmbx.xyz.com]
    + CategoryInfo : InvalidOperation: (:) [Update-MailboxDatabaseCopy], SeedInProgressException
    + FullyQualifiedErrorId : [Server=newyorkmbx,RequestId=cb11f9f1-4157-4e3a-a056-f35cf61db5ba,TimeStamp=2/7/2014 6:36:50 AM] 6079FA2A,Microsoft.Exchange.Management.SystemConfigurationTasks.UpdateDatabaseCopy
    + PSComputerName : newyorkmbx.xyz.com

    Please suggest.

    Thanks in advance

    1. Sumit

      Hi Anees,

      Facing same error for DR copy across the WAN link. Please share if you have managed to identify the cause if the error while seeding.

      Thanks,
      Sumit

      1. Anees Ur Rehman

        Hi Sumit,

        Disable/Uninstall any antivirus. Your problem will be resolved.

        Regards,

        Anees

        1. Sumit

          Thanks Anees for your response. I will check and update the result.

          Regards,
          Sumit

  37. Ihsan

    Hi there
    Please check the “Cluster Service” if it’s not started, this solution will not work.
    In my case, this was the problem why i couldn’t reseed the failed database copy.

  38. Gaurav K. Passi

    Hi Paul,

    Once again, great and very helpful article!!!

    I’m running into a problem while seeding one of the databases to the passive server.

    We’ve 2 node multisite exchange 2010 dag. We’ve a dedicated 100 mbps connection between the sites.
    We’ve 4 mailbox dabases which are off different sizes. 2 small databases replicated properly and are online. 3rd database was of 75 gb in size and went into ‘Failed & Suspended’ mode after creating a copy. I was able to fix it using this very helpful article.

    My 4th database is of 195 gb and even after running this command, it’s still going into ‘Failed & Suspended’ mode. I’ve reseeded it twice but it doesn’t go into the ‘Healthy’ status even though the seeding completes without any issues.

    Can you think of any reason why this would happen. Can you please suggest a solution to fix this issue?

    Thanks in advance 🙂

    1. Gaurav K. Passi

      Sorry, forget to mention that the ‘Copy Queue Length’ is about 642,000 and it didn’t go down after the full backup of DAG.

      My ping response to the other site is 28ms.

      I’ve ran the command to test the replication and it don’t show any problems:

      [PS] C:Windowssystem32>Test-ReplicationHealth

      Server Check Result Error
      —— —– —— —–
      EXC01 ClusterService Passed
      EXC01 ReplayService Passed
      EXC01 ActiveManager Passed
      EXC01 TasksRpcListener Passed
      EXC01 TcpListener Passed
      EXC01 ServerLocatorService Passed
      EXC01 DagMembersUp Passed
      EXC01 ClusterNetwork Passed
      EXC01 QuorumGroup Passed
      EXC01 FileShareQuorum Passed

        1. Gaurav K. Passi

          Hi Paul!

          Thanks for getting back so quickly 🙂

          I went through the event logs and it shows that the seeding process completed properly:

          Information event id: 4135, MSExchangeRepl:
          Seeding operation from the active copy of database ‘New York’ to the passive copy of the database on server EXC02 is complete.

          Then there is this event log following earlier one:

          Error Event ID: 4113, MSExchangeRepl:
          Database redundancy health check failed.
          Database copy: New York
          Redundancy count: 1

          The details of the event logs says:

          Error: Passive copy ‘New YorkEXC02’ is not in a good state. Status: FailedAndSuspended.

          ErrorMessage: The required log file 621253 for New YorkEXC02 is missing on the active copy. If you removed the log file, please replace it. If the log file is lost, the database copy will need to be reseeded using Update-MailboxDatabaseCopy.

          I reseeded the database again and it still went into ‘Failed & Suspended’ mode.

          How can I have my database copy fully replicated and working without dismounting the original database and manually copying it over as a file to the passive server?

        2. Avatar photo

          I’ve seen a similar issue in the past, and the solution was to run a full backup of the database in question then try and reseed again. The full backup updates the log checkpoint and truncates the logs and should therefore resolve the missing log file issue.

        3. Gaurav K. Passi

          Thanks!!! I’ll try it and will update you on the status 🙂

        4. Gaurav K. Passi

          It worked this time. I just completed the reseeding of database copy and it’s healthy!!!

          Thanks alot for your help Paul 🙂

          One last thing. Are there any advanages or disadvantages in enabling ‘Circular Logging’ on the databases now that they’re fully replicated?

          The reason that I’m asking this is because we do ‘Full Exchange’ backups every night and if the circular logging is enabled then the changes are already been committed to the databases and it’s being replicated to the passive server also. Is it still worth to have the circular logging turned off.

        5. Avatar photo
  39. bilal

    Aoa dear,
    please tell me one question my exchange 2010 is properly configured all rules are set, all user are using in 2 database . first DB1 is 750 users and 2nd DB1 is 750 …. one DB1 are working good but the anpther DB are show error in webmail did you have any good solution.

  40. SocalAdmin

    Not sure exactly which process is triggering this, but every attempt to reseed a large database was getting interrupted hourly with a communications error. I’ve attempted from both active and passive copies of the database, as well as putting the seeding source into maintenance mode first. Nothing helped and this was the error:

    A source-side operation failed. Error An error occurred while performing the seed operation. Error: Communication was terminated by server ‘MB1’: Data could not be read because the communication channel was closed..

    Smaller .edb files that could copy in less than an hour succeeded every time, but the larger ones that took over an hour would fail 100% of the time. Resuming was not an option when this error happened.

    Based on the event logs, I was able to figure out the timing coincided with Microsoft Exchange VSS writer attempting to do an hourly backup. Event 4088 complains it cannot backup database because the copy is in the ‘SeedingSource’ state, followed by event 2045 VSS Witer failed with error 80070015 when checking volume dependencies.

    This only happened for the seeding database, all other databases backed up fine. However, as soon as this occurred the seeding was fatally interrupted and event 4104 triggered on the source server “Seeding from the passive copy for database to the passive copy on server X was cancelled … because the copy configuration changed on the seeding source’.

    It is definitely related to logs being replayed. However, this is supposed to be suspended based on the description of Event 4102 (“Passive copy of database on this server is starting a seed … Log replay ill be suspended for this copy and will be automatically resumed when seeding is complete.”) However, I have circular logging enabled, so this is my guess, as we run no third-party software in this environment but YMMV. I’m not in a position to turn that off at this time to confirm.

    The fix for us was to disable the Microsoft Software Shadow Copy Provider and Volume Shadow Copy services while seeding the logs. When the backup/replay routine was called, VSS would output errors 13 and 8193 to the event log (both related to being unable to start due to service disabled) instead of doing the above. We let two cycles of the log replay get skipped until the seeding completed (2h:30m in our environment) and then switched both services back to manual and let everything catch up naturally.

    This may not work for you in a snapshot-heavy/VSS-required installation or when available bandwidth will take you days to seed – so it would be interesting to know a more elegant way of handling this issue.

  41. RMuhammd

    I have a cross site DAG all is well in each site in terms of copies, test-replicationhealth. When i go to add copy in other site i am getting source-side communication errors. socket operation was attempted and unreachable on DAG nic IP for the source server.

    any help would be great! thanks

  42. mis

    How to make the service replication running? the copy status in EX1 shows disconnected & resynchronizing.

    Failure mesage : A serve-side administrative operation has failed. The microsoft exchange replcation service may not be running on server EX01.___. com

    Specific RPC erro message : Error 0x6d9 (There are no more endpoints available from the endpoint mapper) From cli_GetCopyStatusEx2

  43. chander

    Your Article is SuperB, got me out of a jam. Same issue, Failed Status.
    Fyi for anyone interested, while running the command to Re-Seed I rec. this error:

    Error: A database backup is already in progress, Please verify that no other seeding or incremental reseeding operations are started for this database, and then try the operation again by rerunning the Update-MailboxDabaseCopy cmdlet…

    There was a backup by commvault running at the time of this reseeding, re-ran this command during the day and Voila this Article worked wonders.

  44. Andrew Roe

    I tried this process and I get the following error:

    A source-side operation failed. Error There isn’t enough free disk space to perform this operation.. [Database: DB9, Server: server1.acmecorp.com]

    it seems like it is not deleting the existing files first? would there be an issue with removing some files manually then re-trying the command or should I blow away the whole copy and create a new copy from scratch (750GB database)

    1. Avatar photo
  45. Alain

    Hello,
    Thanks you for your post. I Still have one question. Is it ok to execute the command if I only have 30 Gb of free space for a 200Gb database?

    Do you know if the
    Update-MailboxDatabaseCopy -DeleteExistingFiles

    first delete the problematic database and after reconstruct them.

    Thanks you very much

    Alain

  46. Guido

    Paul, just a dumb question, it is the same to run it from the active or passive node? Or should be on a specific one? If you could tell me why i will be appreciated! thanks!

  47. david sia

    Paul..

    This proces did not work for me….

    even after the Reseeding I still get the “failed suspended”

    is there any harm in going into EMC and remove the “failed suspended” passive copy from the server..

    will this impact the Active copy of the DB

    thanks

    Dave

  48. James Branch

    Awesome… this saved my day! Thanks Paul

    James

  49. Richard Cunningham

    Hi Paul, just investigating some weirdness which has happened in our exchange env, would the loss of FSW cause a failed & suspended status on our DBs?

    Basically, ETS had disappeared from the local admins group where the FSW was hosted, which caused the cluster to not be able to contact it… this coincided with our DBs going into a failed state…

    1. Avatar photo

      If quorum was lost then your databases will go offline. The loss of the FSW alone normally wouldn’t cause that, but I don’t know the full detail of your DAG.

      I do hear from people who have an undetected FSW issue, and find that their databases go offline when they’re doing patching/rebooting of the mailbox servers in the DAG.

  50. Jack Schweigel

    We have to move all our Mbx Db on VMware from RDMs to VMDKs – 4 servers with 4 DB each = 16 DB. One DB active per server. Is there a variation of this process we could use? Something like – Add a new VMDK disk for each DB, incorporate into DAG, replicate, remove old (RDM) replicate DB and disks? Would need some specifics on that or a similar process.

  51. David Fletcher

    Thanks for your help. This is a great site that gives real solutions to problems that I am having.

    This a little weird. Why does the update database copy work when I do it from the EMS but from the EMC, it always goes back into failed state? Do you do everything from the EMS?

    1. Avatar photo
      1. Dave Purscell

        David is correct. I just did another reseed this weekend. When tried from the EMC it failed (after 6+ hours) due to a single missing log file. I ran the update copy again from the EMS and it went perfectly. Perhaps some parameter which is not being sent from the EMC.

  52. Hein Traag

    Just wanted to pop in and say THANK YOU! We recently moved to a DAG environment and after a copy action two databases failed. Was able to get them going again with this how-to.

    Thanks again!

  53. Dave Purscell

    Is there a way to pause the reseed process, so that we can continue it later during off hours.

    Based on progress measurements, it appears the reseed will take another 32 hours to complete. We are performing it on the weekend with minimal staff, but we are consuming 100% of the VPN bandwidth.

  54. Dave Purscell

    Any way to limit bandwidth consumption during the reseed? Last time I did this I took down the VPN for several days. I really don’t care how long it takes to reseed if it can do so in the background.

  55. ZEFel

    hi,

    can u tell me how to bring back PAM role of my 4-node cluster – after some problem with network i have operational cluster but cmdlet Get-DatabaseAvailabilityGroup showing empty PAM and OS

    –MC

    1. Avatar photo
  56. Martin

    Saved my day! 🙂

Leave a Reply