Exchange database repair – The eseutil /p process
Recently, a reader of the website came to me with a question- “You explained in detail on how to perform various operations to recover data from an exchange database “which is great”. But, what exactly repair (eseutil /p) processes do?
Exchange Database Recovery – Using eseutil commands: https://msexchangeguru.com/2009/07/12/exchange-database-recovery-using-eseutil-commands/
Well first I thought of just answering his questions and then got a second thought on why not write an article with what I know about ESEUTIL and the repair process? So, here we go.
Believe me guys, if you are an Exchange pro, you got to spend some time reading an understanding this article.
ESEUTIL, ESEUTIL and ESEUTIL!!!
I interviewed a candidate recently, “What is eseutil” he started off by explaining the eseutil commands and different switches. Well, that aint my question… My question was, “What is eseutil”?
Well, my answer would be “ESEUTIL.exe or the Exchange Storage Engine Utility is an executable file residing in the bin directory which is used to perform various operations like recovery, repair, integrity check on an Exchange database while it is offline”.
I would like to stress on “while it is offline” because a database will be locked by the store.exe process when it is mounted/online. Therefore, to run eseutil on a database, the database should be taken offline. Eseutil can only be used on one database at a time.
The file’s which backs eseutil.exe to perform various operations are Ese.dll, jcb.dll, and Exchmem.dll. These files reside in the bin directory too. This means that if you need to run eseutil on a database all you need is these 4 files and it will work fine even on an XP machine.
The Repair process
Question: When do we end up having to repair an exchange database?
Answer: Your database is in a dirty shutdown state and the required log files are missing.
To get an overview about transaction log files and the process in which they commit to the database, refer:
The Operative synopsis of Exchange database and ESE: https://msexchangeguru.com/2011/02/26/ese/
When you database is in a “dirty shutdown”, it will look like this when you dump the database using the eseutil /mh command:
So, long story short à your database is missing some log files and there is no way we can create in manually. At this point, if you don’t have a backup, you need to repair the database to get it to a “clean shutdown” state so that you can mount it and retrieve data.
Repairing databases involves the following three stages, in this order:
########################################################################
Run Eseutil in /P mode to perform a database page-level and table-level repair.
Run Eseutil in /D mode to fully rebuild indexes and defragment the database. eseutil /d does is removes empty pages in the database file and rebuilds its indices
Run Isinteg only on the database to repair the database at the application level and fix the logical corruption of the database.
########################################################################
Before proceeding, look at the figure below:
I mentioned before that the exchange database structure is called “B-tree structure”. The B-tree structure consists of pages, rows, tables etc. The repair process could discards rows from tables or even entire tables based on how “bad shaped” the database is right now. After completing the ESE-level repairs, it is necessary to perform an application-level repair to correct problems that may now exist at the application level because of missing data. Use Isinteg or “Information store integrity checker” to perform this application-level analysis and repair on databases. The following example illustrates how the repair mode in Eseutil works.
For example, a table in the database stores messages for all mailboxes. A separate table is used for each user’s Inbox folder. Suppose that a message is lost when using Eseutil to repair the message table. Eseutil does not correlate the message with the reference to it in each Inbox folder because Eseutil does not have the information about the cross-table schema of the application. Isinteg is needed to compare the repaired message table with each Inbox to remove a lost message from the Inbox folder.
Eseutil looks at each Exchange database page and table and ensures consistency and integrity within each table. Isinteg repairs a database at the application level and ensures the integrity of the relationships between tables.
So, what did you understand?
Let’s study the figure together.
Fig1:
This shows the current state the database is in right now. You can see there are several corrupted pages and link broken between rows and tables.
Fig2:
This shows the state of the database after running the repair process (eseutil /p). As you can see, the process got rid of the corrupt pages, rows from tables or an entire table and filled the spaces with empty space called White spaces.
Fig3:
Here we ran eseutil /d, the offline defragmentation process to reduce the physical size of the database and fully rebuild indexes. This process removes empty pages (white spaces) in the database file and rebuilds its indices.
Fig4:
If you look at the figure you can see I have mentioned “Physical numbering” and “Logical numbering”. It is pretty self explanatory. When there is a 5MB JPEG image stored in a users inbox, it is not stored as a “5MB file inside the database”. Rather, this file is spilt and allocated spaces within the database and there are pointers to the location of the file. Well, did you ever have an issue where you opened an email message and entire body was filled with “garbled” images? Yeah… these pointers would have got misplaced somehow.
When the offline defrag process ran, it got rid of the white spaces in the database and full rebuild the indices. This leaved the database in a situation where 3 should point to 3, but 3 is now pointing to 7. Look at the figure closely one more time. I am talking about the Physical number” and “Logical number”. ISINTEG basically fixes this.
So now you know what happens when an exchange database is “repaired”. The process is lot time consuming and runs @ 4-5 GB/hr. Again, if you have proper backup in place you will never end up doing a repair of you exchange database.
Ratish Nair
MVP Exchange
Team@ MSExchangeGuru
Keywords: Exchange database repair, eseutil /p, what does repair on database do, Repairing exchange database, eseutil for exchange database
March 2nd, 2011 at 10:13 am
Seriously awesome man. I think Microsoft is paying you Lol
Tim
March 4th, 2011 at 5:05 pm
This is great. Thanks for taking the time to lay this out for us.
March 6th, 2011 at 10:34 pm
It is vry helpful… Thanks a lot
March 11th, 2011 at 11:20 am
Brilliant
March 13th, 2011 at 10:04 pm
[…] Entenda como funciona e como usar aqui: https://msexchangeguru.com/2011/03/02/databse-repair/ […]
March 29th, 2011 at 12:22 am
It is very Helpful…Thankyou.
March 29th, 2011 at 10:46 pm
This is the type of explanation MS should be displaying in kb’s. People with experience might find easy to tackle these complicated process by reading kb’s, not all.
Thanks Guru and team.
March 30th, 2011 at 2:15 am
I love your blog! It was a very nice read.. keep up the good work you are very talented!…
April 4th, 2011 at 2:41 am
I always wondered who it worked and you have explained it beautifully. Thanks mate. Good work
April 12th, 2011 at 2:57 pm
Great man…simply superb…
May 24th, 2011 at 3:55 pm
All the articles are being posted in this msexchangeguru websits are awesome and very much helpful to understand every nook and corner part of exchange domain . I was expecting this sort of technical explanation everywhere and finally I got it … Thanks for your great effort and time ..
June 18th, 2011 at 2:26 am
thanks , its a very good article
July 13th, 2011 at 8:18 am
Firstly: Thanks for this really useful explanation!!
But one remark though – one will need esoxal.dll additionally to the 4 files mentioned above in the article to run eseutil on another machine than the exchange server. It’s also located in the …\exchsrvr\bin directory.
September 15th, 2011 at 8:47 pm
Gud one..
September 23rd, 2011 at 4:39 am
Thanks for explaining important things to us.
February 6th, 2012 at 5:22 am
Thanks a lot for this step by step helpful guide to solve the Dirty shutdown problem. Although I got a repaired database, I am experiencing some important data loss issue. Is there any way to prevent this data loss problem.
I have a backup copy but it is 3 months old.
Please help !
February 16th, 2012 at 10:30 pm
@Lenin,
You are experiencing this data loss issue because of eseutil /p repair process. Actually, this command line utility removes the corrupt database pages ( As Ratish Nair has also mentioned in above post) during the repair process to get the database into a consistent state or “clean shutdown”.
If you have a copy of offline database, then you may use a third party application like Stellar Phoenix Exchange server recovery software. I would suggest to try demo version which is free of cost to test the efficiency of the tool. You can download trial version from here : http://exchange.stellarservertools.com/
Let me know if it works for you !
Luis Bob
February 17th, 2012 at 8:53 am
I would agree Stellar Exchange recovery tool is the best in market…
March 9th, 2012 at 11:07 am
Wondering if anyone has any input on a situation I had. I work for a small company with 60 Exchange users. The EDB file itself grew from 20GB in December, to 45GB. Well above the expected growth rate base on the previous year’s growth.
On the 45GB DB, whitespace was reported at about 4GB, the mailboxes themselves totaled about 20GB. So there’s roughly 21GB in unexplained data in the EDB. We migrated all mailboxes to a new DB. After all mailboxes were removed, the whitespace was reported at 44GB or almost all available space within the DB. Ran eseutil /d and DB dropped to about 1GB in size.
The new DB size we migrated mailboxes to is roughly 24GB, with maybe 200MB in whitespace, so pretty clean.
My question is, do you have any idea what could have been taking up the extra 20 to 25GB space in the old DB prior to moving the mailboxes out? Indexes possibly?
We played with cleaning out recoverableitems and got back about 1-2GB. There are no litigation holds, or odd policies retain deleted items, in place.
Any input is appreciated.
March 9th, 2012 at 11:17 am
@JoeC
Well – first of all good job that you are in the right direction.
Before checking event 1221 to know the whitespace, you need to ensure online defrag is completing on the DB…
Event series 700 shows you that…
If you really need to know what consuming extra space on the database, you gotta pass it to MS but the question is – Why to take that much pain when you have already taken care of the issue… 🙂
March 9th, 2012 at 11:34 am
Thanks for a quick response! It’s one of those things where people are asking me ‘why the growth?’ and ‘what was taking up all of that space?’ and ‘why?’. Sometimes fixing the problem is not enough…
I do have event ID 700’s, however upon further investigation, it doesn’t appear that it was running against the mailbox database itself. It’s running against C:\…\Exchange Server\V14\TransportRoles\data\Queue\mail.que’ and ‘D:\…\GFI\MailEssentials\Quarantine\quarantine.edb’ (spam filter)
I inherited this Exchange server, but I assume it was a pretty vanilla install…
March 9th, 2012 at 11:37 am
I’ll add, on the properties of the DB itself, I do have ‘Enable background database maintenance (24 x 7 ESE scanning)’ enabled. Which I thought was the proper way to ensure that took place…
April 29th, 2012 at 3:32 pm
Pretty! This was an extremely wonderful post. Thanks
for providing this information.
August 3rd, 2012 at 6:08 pm
[…] Exchange database repair – The eseutil /p process: https://msexchangeguru.com/2011/03/02/databse-re… […]
August 22nd, 2012 at 8:48 pm
Thank you very much for providing these steps………
September 20th, 2012 at 11:06 am
You have explained it in very simple and easy to understand for every one.
Please cover the SCC,SCR,LCR and CCR with DAIG 2010.
Thanks
Nashim Khan
November 15th, 2012 at 2:59 pm
I did all the step but when i try to do the ISInteg and confirme the database that appears offline i received the followinr error:
Isinteg cannot initiate verification process. Please review the log file for more information.
any Help?
February 26th, 2013 at 1:27 am
Very good presentation on ESEUTIL vs ISINTEG
March 1st, 2013 at 1:51 pm
I’ve followed first 2 steps eseutil /p and eseutil /d and got that damn database priv1.edb clear shutdown state. I must confess that I’ve managed to format before the partition that has had that .edb database. Well all good and well I’ve restaurated .edb with get data back and put it in it’s place. Eseutil /md showed me that was a dirty shutdown and I’ve spend cca 8 h fixing tha table and indexes wint swithces /p an /d. Although I’ve stopped all exchanges services inclusive IIS I didn’t manage to start step 3 command isinteg /fix so then I’ve tried to mount the database it returned me with hideous horror that same error : c 1041724.
So do you think you can help me?!
March 2nd, 2013 at 12:55 pm
Isinteg doesn’t functioned on my database. I coldn’t make it run. Isinteg “e:>program files\exchsrv\ pivr1.edb” doesn’t return any result – just automated help… I’m in a state of disperation, confused. Please help me. The other command worked and brought the database in clean sutdown but I couldn’t mount it because isinteg isn’t running. Help please.
March 2nd, 2013 at 6:23 pm
Isinteg doesn’t functioned on my database. I coldn’t make it run. Isinteg /fix “e:>program files\exchsrv\ pivr1.edb” doesn’t return any result – just automated help… I’m in a state of disperation, confused. Please help me. The other command worked and brought the database in clean sutdown but I couldn’t mount it because isinteg isn’t running. Help please.
April 11th, 2013 at 11:56 am
Thanks Ratish for putting the useful information about ESEUTIL tools. It’ll help the users to run the utility correctly and fix the Exchange database files. Recently, we’re also faced the database dirty shutdown problem which corrupted our database and hanged us from using any email in our inbox. Then, with the help of a third party program, we successfully rebuild and mount the database to MS Exchange server. here is the link of that tool: http://www.serversdatarecovery.com/exchange.html
Although eseutil /p repair command is a useful inbuilt utility but always does not work accurately and gives the data less than our expectation. In such cases, when inbuilt utilities do not work, it is good to go with a third party program.
Note: I would suggest to make backup of your databases regularly !
Jack
July 9th, 2013 at 5:23 pm
What a explaination.. Thanks a lot. Really appreaciate your help..
August 26th, 2013 at 1:59 am
Thanks for sharing a valuable information. For performing such task user needs some technical knowledge and sufficient time. When i performing the search on Google found Many companies offering Exchange data repair software.
Some company offering Free Demo version also. By the use of these tools you perform this task with few clicks. From there one such tool is here: http://www.undeletepcfiles.com/exchange-edb-recovery-tool.html
I hope this tool would help you visitors.
Thanks
November 23rd, 2013 at 1:40 am
Its really helpful. First time i could clear my concept regarding database repair.
February 10th, 2014 at 7:16 am
Seriously well explained.
Perhaps you can add a link to a site which explains the switches..
May 26th, 2014 at 4:02 pm
Great ƿost.
November 13th, 2014 at 3:31 pm
Simple and more clarity. Thanks Rathish
April 4th, 2015 at 4:51 pm
I created a new database, now i want my old database logs to be replayed on new database using the othr database logs, can i achive this using /r?
May 7th, 2015 at 3:45 am
No it is not possible. There will be signature mismatch.
August 13th, 2015 at 4:40 am
Seriously awesome man… this command was brilliant.!
March 17th, 2016 at 12:43 pm
Thank you for this great article
September 2nd, 2016 at 10:41 am
Awesome article. I came across a corrupt mailbox database for the first time for one of my clients this week, and my team did not do a very good job of helping me understand the repair process. This sheds a great deal of light on it, and helps me become much more knowledgeable. Thanks for sharing this great info.