When a mailbox database copy has failed in an Exchange Server 2010 Database Availability Group (DAG) it may be necessary to reseed the mailbox server with the failed database copy.
To reseed the database copy launch the Exchange Management Shell on the server that is in a failed state.
First we need to suspend replication for the mailbox database copy on this server. Use the following command, specifying the mailbox database in the format “<mailbox database name><server name>”.
[PS] C:\>Suspend-MailboxDatabaseCopy -Identity "Mailbox Database 01EX2" Confirm Are you sure you want to perform this action? Suspending mailbox database copy "Mailbox Database 01" on server "EX2". [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): y
The mailbox database copy status will now change from “Failed” to “Failed and Suspended”.
Next we reseed the database with a new copy by issuing the following command.
[PS] C:>Update-MailboxDatabaseCopy -Identity "Mailbox Database 01EX2" -DeleteExistingFiles
The duration of the seeding process will depend on the size of the database and the speed of the network. When the update has completed it will automatically resume replication for the database copy. If you want to prevent automatic resume of replication use this command instead.
[PS] C:>Update-MailboxDatabaseCopy -Identity "Mailbox Database 01EX2" -DeleteExistingFiles -ManualResume
Hi Paul,
Update-MailboxDatabaseCopy -Identity “AAP-DB2\Srv1” -DeleteExistingFiles
A source-side operation failed. Error There isn’t enough free disk space to perform this operation.. [Database: AAP-DB2
, Server: Srv1.domain.com]
So how do i delete or make free space?
As
in Ex2010 , when we delate Logs of one database in server1 in DAG; how to synch again ? thanks
Still a super relevant issue, I just resized some storage to gain space back after moving to new smaller datastores and had to reseed in order to do this but was getting this error. Thanks for the post.
When I am reseeding database copy of A server from passive copy of B server. Passive server copystatus shows “seedingsource”. Exchange 2010 dag
Can somebody help why it is showing like this.?
Thanks in advance.
Hi Paul,
I’d really thank you for useful info. you always publish and share.
i have a small question we have a MBX 2013 with CU12 box with crashed “Ex replication service” suddenly occurred .
we tried to remove the CU12 and re-install again with also remove the server from the DAG and add again, nothing worked out.
the only solution helped out is to re-install the exchange again .
do you have any clue what might be the root cause of this issue ?
Thank you very much in advance.
I am getting issue in Exchange 2016 where lag database is having high replay queue length and secondly the content indexing status shows auto suspended. PLease see below screenshot for reference.
DB1FR4012\MS-FR2-EXM-002 Healthy 0 14 2/14/2017 11:48:10 AM Healthy
DB1FR2005\MS-FR2-EXM-002 Healthy 0 3422 2/14/2017 11:47:46 AM AutoSuspended
DB1FR2006\MS-FR2-EXM-002 Healthy 0 10544 2/14/2017 11:48:17 AM AutoSuspended
DB1FR2007\MS-FR2-EXM-002 Healthy 0 10706 2/14/2017 11:48:08 AM AutoSuspended
DB1FR2008\MS-FR2-EXM-002 Healthy 0 10291 2/14/2017 11:48:13 AM AutoSuspended
DB1FR2009\MS-FR2-EXM-002 Healthy 0 10913 2/14/2017 11:48:18 AM AutoSuspended
DB1FR2010\MS-FR2-EXM-002 Healthy 0 10409 2/14/2017 11:48:17 AM AutoSuspended
DB1FR2011\MS-FR2-EXM-002 Healthy 0 12619 2/14/2017 11:48:11 AM AutoSuspended
DB1FR2012\MS-FR2-EXM-002 Healthy 0 2150 2/14/2017 11:48:16 AM AutoSuspended
On other server we do not have any issues for lag copy, replay queue length is normal and content indexing is healthy.
Is this 2016 CU4?
Yes
Can you take a look at this article and see if it explains what is happening in your case.
https://blogs.technet.microsoft.com/exchange/2016/03/03/lagged-database-copy-enhancements-in-exchange-server-2016-cu1/
It actually appears to be a known configuration change in 2016 CU4 according to this: https://blogs.technet.microsoft.com/jcoiffin/2017/03/03/contentindexstate-autosuspended-for-lagged-copies/
Pingback: Error “Passive copy of Mailbox Database is not in a good state. Status: FailedAndSuspended” | Site Title
Hi Information in event log just prior to error …
The active copy of database ‘I1-Database’ on this server is starting a seeding operation for the passive copy of the database on server SERVER2.
Information Store (6744) I1-Database: An internal copy (for seeding or analysis purposes) is starting. The streaming ESE backup APIs are being used for the transfer method.
Information Store (6744) I1-Database: Starting the copy or backup of the file I:ExchSrvMailboxI1-DatabaseI1-Database.edb (size 402594 Mb).
Information Store (6744) Data transfer method is shared memory (1024 KB).
Information Store (6744) I1-Database: Ending the internal copy (for seeding or analysis purposes) of the file I:ExchSrvMailboxI1-DatabaseI1-Database.edb. Not all data in the file has been read (read 4194304 bytes out of 422151028736 bytes).
followed by
Information Store (6744) I1-Database: The internal database copy (for seeding or analysis purposes) has been stopped because it was halted by the client or because the connection with the client failed.
A little bit of Googling for those error messages led me to this article:
https://blogs.technet.microsoft.com/timmcmic/2012/12/04/exchange-2010-adding-database-copies-to-databases-with-a-single-copy-results-in-copy-status-failed-after-seeding-is-completed/
Hi
Able to resolve by stopping and disabling third party services on the server in issue (ie Snap related services for NetApp and TSM) – we also stoppeddisabled the DPM services.
Restarted server.
Reseeded 4 databases using EMC overnight (DBs were all approx. 350 GB in size).
Databases back to Mounted and Healthy state.
A lot of the Google search results for your error messages mentioned NetApp and Snapmanager issues.
Hi. Getting an issue on a RESEED exercise in an Exchange2010 DAG environment. I have 4 servers in a DAG with 4 databases on each. Two mounted DB servers and 2 Servers with copies on. One server is healthy and the other has its,4 DBs in FailedandSuspended state.
Error: A source-side operation failed. Error An error occurred while performing the seed operation. Error: The requested resource is in use. Database….. Server.
Exchange Management Shell command attempted:
Update-MailboxDatabaseCopy -Identity DBServer -DeleteExistingFiles.
Log files are mounting up.
Am I safer to move mailboxes to another server with mounted and healthy databases and rebuild this exchange server?
Seems like a big effort for potentially a small issue. Why not investigate further?
Hi Paul
I think the issue may be backup process related – we use TSM for long term retention and DPM for weekday usage. Looking at TSM there were no sessions for the DAG or the server in the last day (as I would expect) and I thought DPM was rescheduled to avoid the reseed window but there may have been a process running. Presumably DPM would have a detrimental affect on a reseed process too?
Seeding/reseeding relies on VSS. If a backup is running that is using VSS, then a seed/reseed operation can’t be performed at the same time.
It doesn’t impact ongoing replication, only seeding/reseeding.
Hi – have checked backups and that is not the issue – we are now getting …
Information Store (6744) “Database”: The internal database copy (for seeding or analysis purposes) has been stopped because it was halted by the client or because the connection with the client failed.
Thanks I love this site. Not being an exchange admin but being forced to tend to a simple 2 node 2010 install, this website is concise and to the point. Keep it up 🙂
exchange 2010 database copy status failed and suspended the problem arise everyday. I update database copy everyday from EMC. Please give me a solution.
Dear Paul,
I followed these steps to fix my exchange 2010 SP3 (Window server 2008 R2 SP1) after I installed Firmware Package Version: 23.33.0-0043. However I still got that error as below:
I used command fsutil fsinfo ntfsinfo d: , the Bytes Per Physical Sector : 512
But I used the command eseutil /ml E0000386999.log to check the file log. It showed the message: Env Log Sec size: 4096 (actual @ log gen 512). I do not know how to fix this problem.
Did I get a mistake here.
Hope everybody can help me to this problem.
Interesting. Updated from 2010 SP2 R8 to 2010 SP3 R12 last week and got this error now in eventlog and EMS only. EMC showed replication was OK and without errors.
EMS displayed an communication error only for one of four replicated DB’s.
Paused the Replication for that DB via EMC and re-enabled few seconds later. Sync was finished in seconds. So I guess replication was working before but some minor problem existed that resulted in the error. Thanks for your posting here!
Well, after some hours, the DB Replication failed again for that database (only in EMS again). So I started a DB Copy with “-deleteExistingFiles” option. Takes some time with a 1TB Database…. Hope, it will run stable afterwards.
This DB runs stable now. But an other DB repl went bad. This time with the error
Event ID 2153
The log copier was unable to communicate with server ‘ABC-2’. The copy of database ‘ABCDB2’ is in a disconnected state. The communication error was: Communication was terminated by server ‘ABC-2’: Data could not be read because the communication channel was closed. The copier will automatically retry after a short delay.
In fact, it did not retry after a short delay. The Replication i fact war well but this error was still shown in management shell only.
Found an other interesting blog that identified the MTU size of the dedicated Replication LAN-Interface as the potential problem.
netsh interface ipv4 show subinterfaces
shows that the Exchange LAN and Replication Interface have different MTU-sizes.
Set bot Interfaces to MTU 1300 and paused/reenabled replication.
Replication interface in is a Cross-Over LAN Link as the servers are in the same site.
Hoping it will not run into error again.
Problem is still happening at random… No idea what should be wrong – espeacially as it was working troublefree with SP2 RP8
So, I disabled TCP Offloading and RSS on the replication NIC but after some troublefree days the error reoccoured. In eventlog I see that the replica went offline but got online and healthy shortly afterwards. But this change of state is not synced to the management shell output – opposite to the management console where everything is listed as healthy.
There is nothing listed to the connection of the NIC’s anywhere.
And it’s only happening for one of four databases. If there would really be a outage on the line, it would affect all DB’s not only one.
I will ignore this from now on.
In eventlog in Microsoft/Exchange/HighAvailability/Debug I see:
10.03.2016 06:57:55 146 ReplayState ‘Disconnected’ for database ‘DB1(GUID-String)’ on node ‘ABC-2’ changed from ‘False’ to ‘True’
10.03.2016 06:57:55 146 ReplayState ‘CopyStatusEnum’ for database ‘DB1(GUID-String)’ on node ‘ABC-2’ changed from ‘Healthy’ to ‘DisconnectedAndHealthy’
10.03.2016 06:58:20 146 ReplayState ‘Disconnected’ for database ‘DB1(GUID-String)’ on node ‘ABC-2’ changed from ‘True’ to ‘False’
10.03.2016 06:58:20 146 ReplayState ‘CopyStatusEnum’ for database ‘DB1(GUID-String)’ on node ‘ABC-2’ changed from ‘DisconnectedAndHealthy’ to ‘Healthy’
In eventlog in Application I see:
10.03.2016 06:57:55 2153 The log copier was unable to communicate with server ‘ABC-1’. The copy of database ‘DB1ABC-2’ is in a disconnected state. The communication error was: Communication was terminated by server ‘ABC-1’: Data could not be read because the communication channel was closed. The copier will automatically retry after a short delay.
**sorry for all these comments but I hope, it may help someone**
Hi Paul, hope you can help me to solve this problem:
Database redundancy health check failed.
Database copy: Mailbox Database 11
Redundancy count: 1
Error: Passive copy ‘Mailbox Database 11CSVCMBR2’ is not in a good state. Status: Failed.
Name Status RealCopyQueu InspectorQue ReplayQueue CIState
e ue
—- —— ———— ———— ———– ——-
Mailbox Datab Mounted 0 0 0 Healthy
ase 11CSVCMB
R1
Mailbox Datab Failed 1447 0 1 Healthy
ase 11CSVCMB
R2
===============
Full Status
===============
Identity : Mailbox Database 11CSVCMBR1
Name : Mailbox Database 11CSVCMBR1
DatabaseName : Mailbox Database 11
Status : Mounted
MailboxServer : CSVCMBR1
ActiveDatabaseCopy : csvcmbr1
ActivationSuspended : False
ActionInitiator : Unknown
ErrorMessage :
ErrorEventId :
ExtendedErrorInfo :
SuspendComment :
SinglePageRestore : 0
ContentIndexState : Healthy
ContentIndexErrorMessage :
CopyQueueLength : 0
ReplayQueueLength : 0
LatestAvailableLogTime :
LastCopyNotificationedLogTime :
LastCopiedLogTime :
LastInspectedLogTime :
LastReplayedLogTime :
LastLogGenerated : 0
LastLogCopyNotified : 0
LastLogCopied : 0
LastLogInspected : 0
LastLogReplayed : 0
LogsReplayedSinceInstanceStart : 0
LogsCopiedSinceInstanceStart : 0
LatestFullBackupTime : 4/15/2015 1:01:09 PM
LatestIncrementalBackupTime :
LatestDifferentialBackupTime :
LatestCopyBackupTime : 12/2/2013 2:26:37 PM
SnapshotBackup : True
SnapshotLatestFullBackup : True
SnapshotLatestIncrementalBackup :
SnapshotLatestDifferentialBackup :
SnapshotLatestCopyBackup : True
LogReplayQueueIncreasing : False
LogCopyQueueIncreasing : False
OutstandingDumpsterRequests : {}
OutgoingConnections :
IncomingLogCopyingNetwork :
SeedingNetwork :
ActiveCopy : True
Identity : Mailbox Database 11CSVCMBR2
Name : Mailbox Database 11CSVCMBR2
DatabaseName : Mailbox Database 11
Status : Failed
MailboxServer : CSVCMBR2
ActiveDatabaseCopy : csvcmbr1
ActivationSuspended : False
ActionInitiator : Administrator
ErrorMessage : The Microsoft Exchange Replication service e
ncountered an error while inspecting the log
s and database for Mailbox Database 11CSVCM
BR2 on startup. Error: File check failed : L
ogfile ‘D:MicrosoftExchange ServerV14Mai
lboxMailbox Database 11E00.log’ is generat
ion 2788053; however, the expected generatio
n is 2788055.
ErrorEventId : 2070
ExtendedErrorInfo :
SuspendComment :
SinglePageRestore : 0
ContentIndexState : Healthy
ContentIndexErrorMessage :
CopyQueueLength : 1447
ReplayQueueLength : 1
LatestAvailableLogTime : 2/21/2016 9:57:22 PM
LastCopyNotificationedLogTime : 2/21/2016 9:57:22 PM
LastCopiedLogTime : 2/21/2016 9:57:22 PM
LastInspectedLogTime : 2/21/2016 9:57:22 PM
LastReplayedLogTime : 2/21/2016 9:53:30 PM
LastLogGenerated : 2789501
LastLogCopyNotified : 2788054
LastLogCopied : 2788054
LastLogInspected : 2788054
LastLogReplayed : 2788053
LogsReplayedSinceInstanceStart : 0
LogsCopiedSinceInstanceStart : 0
LatestFullBackupTime : 2/21/2016 1:00:39 PM
LatestIncrementalBackupTime :
LatestDifferentialBackupTime :
LatestCopyBackupTime : 12/2/2013 2:26:37 PM
SnapshotBackup : True
SnapshotLatestFullBackup : True
SnapshotLatestIncrementalBackup :
SnapshotLatestDifferentialBackup :
SnapshotLatestCopyBackup : True
LogReplayQueueIncreasing : False
LogCopyQueueIncreasing : False
OutstandingDumpsterRequests : {}
OutgoingConnections :
IncomingLogCopyingNetwork :
SeedingNetwork :
ActiveCopy : False
I also got one problem after check it at event views with event ID 2070:
The Microsoft Exchange Replication service encountered an error while inspecting the logs and database for Mailbox Database 21CSVCMBR2 on startup. Error: File check failed : Logfile ‘D:MicrosoftExchange ServerV14MailboxMailbox Database 21E00.log’ is generation 3521494; however, the expected generation is 3521523.
And this with event ID 124:
At ‘2/22/2016 10:43:58 AM’ the Exchange Information Store Database ‘Mailbox Database 21’ copy on this server encountered an error. Consult the event log on the server for other “ExchangeStoreDb” or “msexchangeRepl” events that may identify the specific failure. The Replication Service will automatically attempt to retry.
Thanks a lot from your help.
You should reseed the failed database copy.
Hi Paul,
It is really work, thank you so much!
Hi Paul,
I just think the mailbox is ok, but after I restart the server today, it showed failed message as following:
The log copier was unable to continue processing for database ‘Mailbox Database 21CSVCMBR1’ because an error occured on the target server: Continuous replication – block mode has been terminated. Error: the log file sector size does not match the current volume’s sector size (-546) [HResult: 0x80131500]. The copier will automatically retry after a short delay.
This was happened after I upgraded firmware the new version in Megaraid (LSI). Is there any way to solve it. Thank you!
Hey Paul,
I am trying to reseed the DB but getting the error:
A source-side operation failed. Error An err
or occurred while performing the seed operat
ion. Error: An error occurred while processi
ng a request on server ‘MAIL***1’. Error:
Couldn’t open backup file handle for databas
e ‘Mailbox Database 2’ to server ‘MAIL***1
‘. Hresult: 0x50d. Error: A database backup
is already in progress. Please verify that n
o other seeding or incremental reseeding ope
rations are started for this database, and t
hen try the operation again by rerunning the
Update-MailboxDatabaseCopy cmdlet..
You can’t reseed a database while another reseed is running, or while a backup is running.
I have questions on user impact. If a database copy has failed, is there any impact to the users that are conducting email business based on the first email server DAG member) and its mailbox database?
If there’s still an active and healthy copy for them to connect to then no. Other than they no longer have HA.
Hello Paul,
Your articles are precise enough to easily understand! Appreciate the effort.
We are going to have to rebuild one of the servers in a 2 Member DAG. I was planning on leveraging the RecoverServer switch for this with reference from the article https://technet.microsoft.com/en-us/library/dd638206(v=exchg.141).aspx.
However, I have a question, as per the article above, it is recommended by Microsoft to remove any Database Copies on the server that is going to be recovered (in my case rebuilt). I do that before OS recovery and once the server is up and running, when I add back the database copies on the rebuilt server, is it ok to leave the Database and Log Files (before recovery) in the directories as they are? Will the seeding from active copy happen with the delta only?
Cheers,
Chaitu
When you add the database copies back again you’ll need to delete any existing files first.
Hello Paul,
Thanks for the quick reply, just curious to know what would be the implications if I leave them in the same drive? I have ran this in a lab, it seem to sync from where it left off, but small databases in the lab, no particular way to figure out if it copied over the delta or the whole database altogether.
Cheers,
Chaitu
My advice is to play it safe and reseed a fresh copy.
Thx Paul for ExchangeServerPro articles. Saved my day many times, you rock dude. Thanks for sharing your knowledge.
Thank you very much for your help 🙂
Hi Paul, I have used this procedure many times in the past with great success. Thank you for that.
This time I ran into an error:
“A source-side operation failed. Error An error occurred while performing the seed operation. Error: The process cannot access the file ‘D:
ExchangeDBDB11DB11.edb’ because it is being used by another process…”
Not sure what this means, or how to remedy it.
Thank you in advance!
Check your antivirus exclusions.
https://technet.microsoft.com/en-us/library/bb332342%28v=exchg.150%29.aspx
Thank you Paul for your reply. Turns out I rebooted and then reseeded per your article. Worked perfectly.
Thank you again for your article.
Dave
I’m hoping you can help me understand a problem I’m having with a reseed. I’m trying to reseed a data store on one server (EX1) following a disk issue (target LUN was inadvertently disconnected briefly).
The status on EX2 shows an error on one log file (says the file is corrupt). Checking the directory, it appears the file is actually not present. An attempt to reseed failed after several hours because of that.
Is there a switch or setting to tell Exchange to bypass a missing log?
Hi Paul,
I have a DAG active and passive, with a separate replication network (172.x.x.x). If I reseed a failed DB will the reseed using the replication network rather than the data (10.x.x.x) network? The replication network is on a separate switch with no access to the data network.
Thanks,
If replication is disabled on the MAPI network, then yes.
Thanks Paul, one last question, I’m trying to reseed via the EMC by right clicking on the db, and selecting “Resume Database Copy” is starts then fails. The event log shows its missing a log file, but I’ve checked the log folder and the missing log is there, in the active log folder and the passive log folder?
Thanks,
Resume- is not the same as reseeding. If you want to reseed the passive copy follow the steps in the article above.
Hello Paul,
I will need to take down the passive copy of my two-member DAG very soon for at least a good number of hours. I remember when I did this the last time, the queue got so big all efforts to reseed the passive copy failed. If worse comes to worse, should I simply delete the passive copy in ECM, delete the EBD database and logs on the passive server and then make a new copy in ECM? Would that work?
Also keep in mind as long as the replication between the two Exchange servers are being held up, I cannot run a full backup and flush the transaction logs. That can also be a problem. Is there a way to stop all replication just so that I can run the backup and flush transaction logs?
Suspending replication on the DB copy will still cause the copy queue to build up.
If you know it’s going to be down for a long time and that it will impact your disk capacity and/or backups, I suppose you could just remove that database copy and re-add it later. Understand the risks of this, you’re reducing your copies of that database by 1…
Otherwise, if a database copy won’t resume or is otherwise failed you only need to reseed it using the steps shown in the article above. No need to manually delete files or remove/re-add the DB copy.
Paul,
Thanks for your continues support and information to the Exchange community. I rely heavily on the things you post.
I have one question regarding reseeding the database.
Recently, one of the drives in my JBOD DAS on one of my servers threw a predictive failure. The database never showed failed.
The vendor replaced the drive and the new drive is now working fine.
During all this, the mailbox databases affected by this JBOD DAS never showed “Failed”.
Do we still need to reseed the database even though the database never failed in the first place?
Was it a JBOD volume hosting the database copy? Ie, the only disk that had that database copy on it? If so, I’m surprised it isn’t failed. Maybe it was actually a RAID volume?
Thank You, the PowerShell command resolved our problems as EMC wasn’t working.
Paul, I reseeded after the active copy got corrupted due to the logs filling up the logs HD. The DB failed over to the passive but the now the new active DB throws me this error: “The active copy of MBDB02 has a missing or corrupted log file (E06002526D1.log). To keep this copy healthy, replication will restart at generation 2434770. A full backup must be performed before an incremental backup is possible.” I’m currently performing a full backup but in the meantime, the CQL is rising steadily. Any idea what’s happening?
Pingback: How To Pst A Mailbox In Exchange 2010 - ORG.org
Thanks Paul!
Hi Paul,
every time I move one of my copies,the copy status stuck at initializing and I cant reseed. I used your method but I get the following error:
WARNING: The Microsoft Exchange Replication service has not responded to the suspend request yet. The service may not
be running. Press CTRL+C to stop waiting for the service to respond. You can check to see if it is suspended by using
the Get-MailboxDatabaseCopyStatus cmdlet in the Exchange Management Shell.
Alexander, ever get an answer to your issue with the DB showing as Iitializing? how did you resolve it?
Paul,
so can be Microsoft remove exchange on-premises in future .
personally do you really thing that every customer will go for cloud or customer will demand for on-premises ?
because as i aware every is going with hybrid not full cloud . what do you say?
I can’t predict the future and I’m not going to just guess what Microsoft plans to do.
Try to avoid guessing what “every customer” is going to do or is doing. There’s a lot of different customers around the world and they all have their own special needs and circumstances.
I expect some percentage of customers will go full cloud, some will go hybrid, and some will always stay on-premises. Beyond that there is nothing I can predict.
Hi Paul,
I have some query which is mention below.
1) will on premises exchange will be finished from industry
2 ) is Microsoft planning to introduce next version of exchange for on premises or they will only consider on online exchange.
3) is Microsoft planning to flush on- primises
I known these are stupid query but for being exchange person we should be aware before some disaster , so that we can start leaning new technology.
As i know maximum customers are deploying hybrid not full cloud that indicates that no one is prepare for full cloud, but we are scare.
Please reply my query
As far as I am aware Microsoft has publicly stated that they plan to deliver another version of Exchange for on-premises customers. We can probably expect to see a preview of that later next year and perhaps it will ship in 2016.
I would not wait before learning new technologies.
Hey thanks for the tutorial.
I am facing a situation wherein my Mailbox Database is in a healthy state. But the event viewer throws up the error “A source-side operation failed……”
I did try the “Update-MailboxDatabaseCopy -Identity DB1MBX1 -CatalogOnly” command. But it still throws up the same error.
Should I go ahead with the method mentioned here?
Great instructions. Although I didn’t use the -manualresume switch, it didn’t automatically resume replication. I manually resumed and it worked perfectly.
Hi – Your solutions worked fine but after multiple attempts as for some reason it was failing because of Win Rm service going down. Thanks a lot for your blog. keep writing 🙂
on side note, do you know is there any way i can replace existing FILE witness server with new server ?
we have two Exchange server running DAG and one file witness server (server 2003) and file server is failing for past few months and i was thinking to replace with latest 2008R2 File server.
Thanks again
You can change the FSW any time using Set-DatabaseAvailabilityGroup. You just need to set up the permissions first (ie add Exchange Trusted Subsystem to local admins group on the new FSW) and also be aware that in Failover Cluster management tool the old UNC path will still appear in the name of the FSW resource.
Hello Experts,
My reseed has been running for days! 1 TB DB over wan link.. Can I run backup while reseed in progress?
Thanks
Tarra
Nope, reseed and VSS backups can’t run at the same time.
1TB database size is a risk if you’re having to reseed across a WAN. I would prefer multiple, smaller databases in that situation.
Hi Paul,
I am facing a problem with DAG in exchange 2010 sp1 environment.
I have 2 Mailbox servers having a couple of mailbox databases (active/passive). i have been trying the reseeding operation on DTDB1 database without success. below is the error. the db size is 201.6GB. kindly assist.
A source-side operation failed. Error An error occurred while performing the seed operation. Error: An error occurred while processing a request on server ‘EXCH-DRS-SVR02’. Error: An I/O error occurred while attempting to access file ‘I:Program FilesExchangeHomeMailboxDTDB1DTDB1.edb’ on source server EXCH-DRS-SVR02. Error: SendSeedingDataInternal: PerformDatabaseRead fails with code 0xFFFFFC18. Expected 65536 bytes read 0.
Copy Status: Failed and Suspended
Content index state: failed
Copy queue length (logs): 4718233
Replay queue length (logs): 0
Please suggest.
Thanks in advance
Paul Paul Paul Paul Your the MAN…! Blesses you saved my day
cheers
regards,
imran from afghanistan
What is the purpose of : MailboxDatabaseReseed.ps1 ?
Is it doing the same job ?
Hi Paul,
I am facing a problem with DAG
I have 2 Mailbox servers in San Francisco and DAG is working fine on both servers.
Now I have to implement DAG on a Server which is in New York. Everything is ok. Servers are accessible. I am able to ping the DAG from every server. There is vpn tunnel between sf and ny zone.
But when I started the replication it gives the following error from ecp
Passive Failed and Suspended
Copy queue length: 8067
Content index state: Suspended
Then I started to reseed from EMS, I got following error
The seeding operation failed. Error: An error occurred while performing the seed operation. Error: Communication was terminated by server ‘SanFrMailBoX1’: Data could not be read because the communication channel was closed. [Database: SFDataBase, Server: newyorkmbx.xyz.com]
+ CategoryInfo : InvalidOperation: (:) [Update-MailboxDatabaseCopy], SeedInProgressException
+ FullyQualifiedErrorId : [Server=newyorkmbx,RequestId=cb11f9f1-4157-4e3a-a056-f35cf61db5ba,TimeStamp=2/7/2014 6:36:50 AM] 6079FA2A,Microsoft.Exchange.Management.SystemConfigurationTasks.UpdateDatabaseCopy
+ PSComputerName : newyorkmbx.xyz.com
Please suggest.
Thanks in advance
Hi Anees,
Facing same error for DR copy across the WAN link. Please share if you have managed to identify the cause if the error while seeding.
Thanks,
Sumit
Hi Sumit,
Disable/Uninstall any antivirus. Your problem will be resolved.
Regards,
Anees
Thanks Anees for your response. I will check and update the result.
Regards,
Sumit
Hi there
Please check the “Cluster Service” if it’s not started, this solution will not work.
In my case, this was the problem why i couldn’t reseed the failed database copy.
Hi Paul,
Once again, great and very helpful article!!!
I’m running into a problem while seeding one of the databases to the passive server.
We’ve 2 node multisite exchange 2010 dag. We’ve a dedicated 100 mbps connection between the sites.
We’ve 4 mailbox dabases which are off different sizes. 2 small databases replicated properly and are online. 3rd database was of 75 gb in size and went into ‘Failed & Suspended’ mode after creating a copy. I was able to fix it using this very helpful article.
My 4th database is of 195 gb and even after running this command, it’s still going into ‘Failed & Suspended’ mode. I’ve reseeded it twice but it doesn’t go into the ‘Healthy’ status even though the seeding completes without any issues.
Can you think of any reason why this would happen. Can you please suggest a solution to fix this issue?
Thanks in advance 🙂
Sorry, forget to mention that the ‘Copy Queue Length’ is about 642,000 and it didn’t go down after the full backup of DAG.
My ping response to the other site is 28ms.
I’ve ran the command to test the replication and it don’t show any problems:
[PS] C:Windowssystem32>Test-ReplicationHealth
Server Check Result Error
—— —– —— —–
EXC01 ClusterService Passed
EXC01 ReplayService Passed
EXC01 ActiveManager Passed
EXC01 TasksRpcListener Passed
EXC01 TcpListener Passed
EXC01 ServerLocatorService Passed
EXC01 DagMembersUp Passed
EXC01 ClusterNetwork Passed
EXC01 QuorumGroup Passed
EXC01 FileShareQuorum Passed
Check your event logs. There should be some more info being logged when the DB copy status changes.
Hi Paul!
Thanks for getting back so quickly 🙂
I went through the event logs and it shows that the seeding process completed properly:
Information event id: 4135, MSExchangeRepl:
Seeding operation from the active copy of database ‘New York’ to the passive copy of the database on server EXC02 is complete.
Then there is this event log following earlier one:
Error Event ID: 4113, MSExchangeRepl:
Database redundancy health check failed.
Database copy: New York
Redundancy count: 1
The details of the event logs says:
Error: Passive copy ‘New YorkEXC02’ is not in a good state. Status: FailedAndSuspended.
ErrorMessage: The required log file 621253 for New YorkEXC02 is missing on the active copy. If you removed the log file, please replace it. If the log file is lost, the database copy will need to be reseeded using Update-MailboxDatabaseCopy.
I reseeded the database again and it still went into ‘Failed & Suspended’ mode.
How can I have my database copy fully replicated and working without dismounting the original database and manually copying it over as a file to the passive server?
I’ve seen a similar issue in the past, and the solution was to run a full backup of the database in question then try and reseed again. The full backup updates the log checkpoint and truncates the logs and should therefore resolve the missing log file issue.
Thanks!!! I’ll try it and will update you on the status 🙂
It worked this time. I just completed the reseeding of database copy and it’s healthy!!!
Thanks alot for your help Paul 🙂
One last thing. Are there any advanages or disadvantages in enabling ‘Circular Logging’ on the databases now that they’re fully replicated?
The reason that I’m asking this is because we do ‘Full Exchange’ backups every night and if the circular logging is enabled then the changes are already been committed to the databases and it’s being replicated to the passive server also. Is it still worth to have the circular logging turned off.
For a DAG circular logging works differently to a standalone mailbox server/database.
Some reading:
http://blogs.technet.com/b/scottschnoll/archive/2011/06/27/circular-logging-and-mailbox-database-copies.aspx
Aoa dear,
please tell me one question my exchange 2010 is properly configured all rules are set, all user are using in 2 database . first DB1 is 750 users and 2nd DB1 is 750 …. one DB1 are working good but the anpther DB are show error in webmail did you have any good solution.
Not sure exactly which process is triggering this, but every attempt to reseed a large database was getting interrupted hourly with a communications error. I’ve attempted from both active and passive copies of the database, as well as putting the seeding source into maintenance mode first. Nothing helped and this was the error:
A source-side operation failed. Error An error occurred while performing the seed operation. Error: Communication was terminated by server ‘MB1’: Data could not be read because the communication channel was closed..
Smaller .edb files that could copy in less than an hour succeeded every time, but the larger ones that took over an hour would fail 100% of the time. Resuming was not an option when this error happened.
Based on the event logs, I was able to figure out the timing coincided with Microsoft Exchange VSS writer attempting to do an hourly backup. Event 4088 complains it cannot backup database because the copy is in the ‘SeedingSource’ state, followed by event 2045 VSS Witer failed with error 80070015 when checking volume dependencies.
This only happened for the seeding database, all other databases backed up fine. However, as soon as this occurred the seeding was fatally interrupted and event 4104 triggered on the source server “Seeding from the passive copy for database to the passive copy on server X was cancelled … because the copy configuration changed on the seeding source’.
It is definitely related to logs being replayed. However, this is supposed to be suspended based on the description of Event 4102 (“Passive copy of database on this server is starting a seed … Log replay ill be suspended for this copy and will be automatically resumed when seeding is complete.”) However, I have circular logging enabled, so this is my guess, as we run no third-party software in this environment but YMMV. I’m not in a position to turn that off at this time to confirm.
The fix for us was to disable the Microsoft Software Shadow Copy Provider and Volume Shadow Copy services while seeding the logs. When the backup/replay routine was called, VSS would output errors 13 and 8193 to the event log (both related to being unable to start due to service disabled) instead of doing the above. We let two cycles of the log replay get skipped until the seeding completed (2h:30m in our environment) and then switched both services back to manual and let everything catch up naturally.
This may not work for you in a snapshot-heavy/VSS-required installation or when available bandwidth will take you days to seed – so it would be interesting to know a more elegant way of handling this issue.
I have a cross site DAG all is well in each site in terms of copies, test-replicationhealth. When i go to add copy in other site i am getting source-side communication errors. socket operation was attempted and unreachable on DAG nic IP for the source server.
any help would be great! thanks
How to make the service replication running? the copy status in EX1 shows disconnected & resynchronizing.
Failure mesage : A serve-side administrative operation has failed. The microsoft exchange replcation service may not be running on server EX01.___. com
Specific RPC erro message : Error 0x6d9 (There are no more endpoints available from the endpoint mapper) From cli_GetCopyStatusEx2
Log on to the server. Open Services.msc. FInd the service and start it.
Your Article is SuperB, got me out of a jam. Same issue, Failed Status.
Fyi for anyone interested, while running the command to Re-Seed I rec. this error:
Error: A database backup is already in progress, Please verify that no other seeding or incremental reseeding operations are started for this database, and then try the operation again by rerunning the Update-MailboxDabaseCopy cmdlet…
There was a backup by commvault running at the time of this reseeding, re-ran this command during the day and Voila this Article worked wonders.
I tried this process and I get the following error:
A source-side operation failed. Error There isn’t enough free disk space to perform this operation.. [Database: DB9, Server: server1.acmecorp.com]
it seems like it is not deleting the existing files first? would there be an issue with removing some files manually then re-trying the command or should I blow away the whole copy and create a new copy from scratch (750GB database)
Reseed includes the database file, content index files, and transaction logs. How much free space do you have on your DB and log drives for the source and destination servers?
Hello,
Thanks you for your post. I Still have one question. Is it ok to execute the command if I only have 30 Gb of free space for a 200Gb database?
Do you know if the
Update-MailboxDatabaseCopy -DeleteExistingFiles
first delete the problematic database and after reconstruct them.
Thanks you very much
Alain
The existing files are deleted first before the reseed process begins.
Paul, just a dumb question, it is the same to run it from the active or passive node? Or should be on a specific one? If you could tell me why i will be appreciated! thanks!
I tend to run the reseed command from the passive node that I’m trying to reseed.
Paul..
This proces did not work for me….
even after the Reseeding I still get the “failed suspended”
is there any harm in going into EMC and remove the “failed suspended” passive copy from the server..
will this impact the Active copy of the DB
thanks
Dave
Awesome… this saved my day! Thanks Paul
James
Hi Paul, just investigating some weirdness which has happened in our exchange env, would the loss of FSW cause a failed & suspended status on our DBs?
Basically, ETS had disappeared from the local admins group where the FSW was hosted, which caused the cluster to not be able to contact it… this coincided with our DBs going into a failed state…
If quorum was lost then your databases will go offline. The loss of the FSW alone normally wouldn’t cause that, but I don’t know the full detail of your DAG.
I do hear from people who have an undetected FSW issue, and find that their databases go offline when they’re doing patching/rebooting of the mailbox servers in the DAG.
You sir are a lifesaver, panic/crisis averted!
We have to move all our Mbx Db on VMware from RDMs to VMDKs – 4 servers with 4 DB each = 16 DB. One DB active per server. Is there a variation of this process we could use? Something like – Add a new VMDK disk for each DB, incorporate into DAG, replicate, remove old (RDM) replicate DB and disks? Would need some specifics on that or a similar process.
Pingback: Exchange Server 2010 – Failover & Maintenance | f i c i l i t y .net
Thanks for your help. This is a great site that gives real solutions to problems that I am having.
This a little weird. Why does the update database copy work when I do it from the EMS but from the EMC, it always goes back into failed state? Do you do everything from the EMS?
I do most things from the shell. But the console gives you pretty much all the options you need for running a reseed, so I’m not sure why its failing for you. The event log should tell you more.
David is correct. I just did another reseed this weekend. When tried from the EMC it failed (after 6+ hours) due to a single missing log file. I ran the update copy again from the EMS and it went perfectly. Perhaps some parameter which is not being sent from the EMC.
Pingback: Confluence: Exchange 2010
Just wanted to pop in and say THANK YOU! We recently moved to a DAG environment and after a copy action two databases failed. Was able to get them going again with this how-to.
Thanks again!
Is there a way to pause the reseed process, so that we can continue it later during off hours.
Based on progress measurements, it appears the reseed will take another 32 hours to complete. We are performing it on the weekend with minimal staff, but we are consuming 100% of the VPN bandwidth.
No, I don’t know of a way to pause it. Bandwidth consumption issues could be solved by either dedicated replication networks, or perhaps QoS.
I’m planning to performing an Offline Reseed later today.
http://blogs.technet.com/b/timmcmic/archive/2011/07/12/exchange-2010-using-vss-to-perform-an-online-offline-database-seed.aspx
Any way to limit bandwidth consumption during the reseed? Last time I did this I took down the VPN for several days. I really don’t care how long it takes to reseed if it can do so in the background.
hi,
can u tell me how to bring back PAM role of my 4-node cluster – after some problem with network i have operational cluster but cmdlet Get-DatabaseAvailabilityGroup showing empty PAM and OS
–MC
Hi ZEFel,
To see the PAM and Operational Servers fields populated you need to include the -status parameter with Get-DatabaseAvailabilityGroup
ie
Get-DatabaseAvailabilityGroup -Status | fl
Saved my day! 🙂