This article is an excerpt from the Exchange Server Troubleshooting Companion.
During normal Exchange database operations, it’s certainly not unusual for a database process to terminate unexpectedly. A termination might be due to losing access to backend storage, file system corruption, or server-wide power loss. Many Exchange support problems involve corrupt databases which will not mount, and therefore prevent users from accessing their mailbox data. The following issues can prevent an Exchange database from mounting:
- The Microsoft Exchange Information Store Service is unable to start
- Missing Exchange database files
- Database is in “Dirty Shutdown” state
- Not enough free disk space on database or log file volume
- Loss of access to underlying storage
- While database is in Clean Shutdown state, significant logical corruption exists causing the database to dismount when an attempt is made to read the data
In situations where the Information Store service will not start, the most common issues involve communications between Exchange and Active Directory or some interaction involving a file used by Exchange and anti-virus software. I recommend using event viewer to verify Active Directory communications as well as verifying there are no service dependencies not met (such as the Microsoft Exchange Active Directory Topology Service not being started). With anti-virus software, verify all exclusions have been properly configured. It is common to discover that an anti-virus product either locks the database files or blocks necessary RPC communications. Anti-virus is also a common reason why database files become corrupt as a result of multiple processes attempting to access the files simultaneously. It is also common to see that an anti-virus product quarantines database files or logs, a step that might break the sequence of the log stream, making recovery from backup challenging, and possibly breaking DAG replication.
Dirty Shutdown
The words “Dirty Shutdown” are fairly self-explanatory; a database is down and not in a healthy state. The words can still cause anxiety for Exchange Administrators who recall spending many hours while frantically attempting to get a database to mount and restore functionality. Over ten years of support, I have probably worked on well over a hundred Dirty Shutdown support cases. But to be honest, a decade worth of experience is not required to troubleshoot a Dirty Shutdown event accurately, especially since the proper course of actions can easily fit into a decision tree. The diagram below provides a high-level flow chart for navigating common scenarios where a database will not mount.
First, let’s define what happens in a Dirty Shutdown. When Exchange dismounts a database, the Information Store ensures that all transactions (dirty database pages) in cache are committed to the database (.edb) file. When this is allowed to happen, the database is said to be in “Clean Shutdown” and requires no transaction logs to mount because all transactions have already been replayed into the database. The diagram below demonstrates how the ESEUTIL utility validates that a database is in Clean Shutdown state.
However, if something happens to cause the database to terminate abruptly, it is probable to result in a Dirty Shutdown state. Whether a Dirty Shutdown happens and which transaction logs may be required to recover the database is dependent entirely upon which transactions were being processed at the time of the failure. The diagram below displays a database in Dirty Shutdown, requiring log 3 (E0000000003.log) to mount.
The DB1 database shown in the diagram above requires transaction log E0000000003.log because the transactions contained within it were either partially committed to the .edb file or are simply required to bring the database to a consistent state. It is hard to say exactly why certain logs are required without a deep programmatic understanding of JET Databases, but nonetheless the log is required and must be available before the database can be mounted.
Learn more about solving Exchange database problems in the Exchange Server Troubleshooting Companion.
Hi Paul,
I am facing an issue with my Exchange Servers, the setup is on Exchange Servers 2013 with Active Directory on Windows Server 2008 R2. I have two exchange servers in a DAG. And on Active Database Copy Status is “Dismounted” and “Content Index State” is showing “Failed”.
In the meanwhile on Secondary Server Passive Database copy status showing “Service Down” and “Content Index State” showing “Unknown”.
Both the copies are corrupted. And i run the eseutil with /P and /D switch to recover the Mailbox database on active server. Recovery was successful but with /D switch is stop at 30% due to insufficient storage so i abandon the defragmentaion.
But when i run eseutil /mh DB.edb to check its status then it showed Clean Shutdown.
Now what i have to do next that database will mount and come online for operations.
Waiting for your reply
Hi,
I’m not an Exchange admin and only understood some of your note, but I do have a related question about Exchange Online Service (EOS). We’re about to put well over 100,000 accounts into EOS that are currently hosted in house. I’m concerned about the lack of backup specifically to protect against corruption. I know EOS has redundant copies and since we’re going to use legal hold there’s additional protection, but I’m not comfortable that we have adequate data protection against human error, virus, or other legit corruption events.
Is my concern unfounded ? Should we trust Microsoft’s EOS completely ?
This question isn’t really relevant to this blog post. Sounds like you’ve got specific business requirements that you want to be sure are met with Exchange Online. I can’t tell you here whether you should trust or not trust EXO to meet those requirements. There are solutions for your concerns, but whether they meet your needs satisfactorily is a matter for you to decide. I’d suggest that if you haven’t got the in house experience to make this assessment that you hire a consultant to help you.
book purchassed.
I very rarely purchase technical books, finding that the majority of information is already either out there on the internet or in my head already. However since becoming a subscriber going back to the bootcamp days I have found myself more and more visiting practical365.com
Its easy to read format makes it a valuable resource to any exchange server adminsistrator at any level – helpdesk to senior messaging. The purchase of the book was a no-brainer.
Thanks to Paul and the rest of those who contributed, looking forward to what is probably another quality publication.
Thanks for the feedback!
Hey Andrew & Paul,
Can you create a blank required transaction log such as E0000000003.log and mount the DB? Just wondering how 3-3 (0x3-0x3) converts to E0000000003.log i’m sure the prod farm may not convert this easily.
Cheers
Josh
The first database on a server will have a log sequence starting with E00. The second database will have a sequence starting with E01 & so on. Each sequence will start at that, then progress upwards using hexadecimal. So the first log would be E0000000001.log followed by E0000000002.log all the way to E00FFFFFFFF.log
To answer your other question, an ESE database will only accept properly formatted transaction log files so what you suggest will not work. There are some tricks you can pull (like if the last log is missing, renaming the preceding log to E00.log, & using the /a parameter) but unfortunately that is not one of them
Thanks Andrew, makes sense. I’ll do some testing in Dev and confirm the results.
Cheers
Yep, that work flow is right from what I’m seing
MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-515)
Error Number JET Error Error Description
Error -501 (0xfffffe0b) JET_errLogFileCorrupt Log File is Corrupt
Error -514 (0xfffffdfe) JET_errBadLogVersion Log file generated with different Exchange Server or edition
Error -515 (0xfffffdfd) JET_errInvalidLogSequence Any log file from the sequence is missing
Error -533 (0xfffffdeb) JET_errCheckpointCorrupt Checkpoint file is deleted or corrupt
Yep. You can run an “eseutil /ml E00” (with E00 being the log sequence of that particular database) to validate all logs are present and not corrupted. So in this case, it views it as corrupted.
Trying to order your book and it will not accept any credit card numbers.
Hi Frank, your bank might be blocking the transaction if it’s outside of your normal buying patterns. You can try choosing PayPal instead and using your CC that way (doesn’t require a PayPal account), or contact your bank to get the block removed.
Excellent write
Ho Andrew,
Is it still so that if you repair a db, you are in unsupported state according to MS? Are you also doing a write up how to fix a db with the required logs and other possible (dirty) shutdowns fixes?
Correct. Please see the updated policy below. This is only an excerpt from the “Exchange Server Troubleshooting Companion” eBook Paul and I recently released. Within this same chapter, I do discuss this support policy as well as how to recover from the scenario you describe.
https://blogs.technet.microsoft.com/exchange/2015/05/01/new-support-policy-for-repaired-exchange-databases/