MSExchangeGuru.com

Have you ever imagined the background processes involved in storing your email message in an exchange database? The process involved in the creation of transaction log files? How does “The famous Store.exe”- Microsoft Exchange Information Store service function and what is its significance? Keep reading….

If you thought and concluded that “It’s no cake-walk”, take a look at this article. This is not an explanation through a developer’s eyes, but I have tried to make it simple and yet “effective”.

So, what’s this ESE anyways and why do I care it exists?

Microsoft Exchange Information store uses ESE which is a DLL file to store records and indexes in an exchange database. ESE ties up the store.exe process to the actual *.edb exchange database.

Exchange database use a data structure called the B-tree database structure. Data inside the database is stored in B-trees, and the B-trees are divided into pages. To understand more about B-tree structure, take a look at this article.

Before we proceed further, you need to know what the below terminologies mean:

1. Transactions
   
2. Concept of ACID
   
3. Background process by which data (email) reaches an exchange database
   
4. Transaction log files
   
5. Checkpoint file
   
6. Mailbox database cache
   
7. Page Size
   

Transactions

A transaction is a series of operations that are treated as atomic and MUST BE completed sequentially. In a database system, a transaction might consist of one or more data-manipulation statements and queries, each reading and/or writing information in the database. This means that if there is a start point, there should be an end point. This implies consistency and integrity of data as highly important.

All transactions end with a COMMIT operation, which means that if any of the operation in a transaction fails, the COMMIT operation won’t be a success and the change/transaction will not be written on the database.

Sorry if I confused you. Let’s take an example – A transaction can be defined as the background set of operation involved when moving a message from the Inbox to a sub folder. An end user doesn’t care what happens in the background, but as an exchange admin you need to know about it (at least if you care). Here, the message moved is deleted from one folder and added to another folder and also the folder properties are updated with this information. As we defined above, if a failure occurs we won’t see duplicate copies of the message or missing copy – WHY because – The transaction never completed. In no case can a partial transaction be committed to the database since that would leave the database in an inconsistent state.

Important – If you thought in this case the client is Outlook and the transaction is carried over from Outlook to the database, well surprise. You’re wrong!!! Read carefully:

Remember having a time when you had a user move a 15MB email from one folder to another and your servers “MSExchangeISà RPC Requests” counter peaked high, stayed there for a while or maybe took the store down? What happened here was, it was a normal operation but the server couldn’t handle it because of various reasons like performance issues, huge database size etc.

Whenever Outlook contacts the server to retrieve data (Send/Receive button in Outlook), it makes an RPC call which can be seen from Performance monitor:

Start > Run > Perfmon

As you see in the figure:

RPC requests –> Clients are requesting for data

RPC Operations/sec –> Server is acknowledging the client request.

Remember: If you see the RPC requests spiking high more that RPC operations, pull up your socks and start working. Your server might go down any moment.

Here Outlook client will make contact with store.exe on the server. Store.exe is the one responsible to initiate the transaction to the database.

Outlook —> Store.exe —>Target database. This makes store.exe the real client when it comes to a transaction.

Concept of ACID

I remember my trainer asked me once, “You are booking a movie/train/flight ticket online. Don’t you think there is a possibility someone book the same seat you selected?” It made me think for a while.

Answer is No. Any transaction follows the ACID concept.

ACID means Atomic, Consistent, Isolated, Durable

Atomic Either all the operations occur or none of them occur.

Consistent The database is transformed from one correct state to another.

Isolated Changes are not visible until they are committed.

Durable Committed transactions are preserved in the database even if the system fails.

Let’s now relate how ESE tackles in case of different set of stages at which a transaction goes through.

While a transaction is going on and the system crashed, ESE does automatic recovery at start and rolls back any uncommitted transactions.

If ESE fails before a transaction is committed, the entire transaction is rolled back, and it is as if the transaction never occurred.

If ESE crashes after the transaction is committed, the entire transaction is persisted, and the changes are visible to clients.

Question – Is this ESE/ACID concept restricted to exchange database?

NO… Active Directory database (NTDS.dit), SQL etc all share the same concept…The database file has no idea that it is storing emails, but just chunks of data!!!

Background process by which data (email) reaches an exchange database:

This section describes the process involved in a transaction from client to the database.

Say you sent an email. Once the email reaches the destination exchange server, before changes are made to the database file:

• Microsoft Exchange writes the changes to a transaction log file.
  
• After the change is safely logged to the log, it will then be written to the database file.
  

These changes might be available to end users just after the changes are secured to the transaction log, but before the changes are written to the database file. Physically writing changes to the database file is a low-priority task during a normal operation.

Let’s say the exchange server crashed for some reason and the database got terminated because the Store.exe program got crashed too. The cached changes aren’t lost just because the memory cache was destroyed.

This is the condition in which the database enters a “Dirty shutdown” state.

So now you got the server back up and running and the store.exe is started. Now, Exchange will start scanning the log files, and check if there are any logs left which the database missed and writes those changes to the database file. This is called replaying log files and the process of writing missing information to the database is called COMMITTING log file. The database will know which was the last successfully committed log file and will only replay those logs which it finds missing.

Now, how will the database know which was the last committed log file. To know this, you will need to get familiar with Exx.log and Exx.chk files.

Transaction log files:

Exchange follows a mechanism where it creates 5MB log files (in E2K3) and 1MB log files in (E2K7/E2010) rather than creating one big huge file which is hard to be maintained in the memory. This series of log files are known as transaction log files in Exchange. When a log file is full, it closes itself and assumes itself with a sequential number. Another log file will be created automatically called Exx.log. Again, once this is filled with data, the same mechanism continues. The very first log file created in your organization will beExx00000001.log.

If Exx is E00, it means that this log file belongs to the databases/databases in the First Storage group.

If Exx is E09, it means this log file belongs to the databases/databases in the Tenth Storage group.

Note: If Exx is E0A, it means this log file belongs to the databases/databases in the eleventh Storage group.

So, the naming convention is not exactly like 1, 2, 3 etc., so make a note of this.

Checkpoint file:

Back to the question: How will the database know which was the last committed log file?

There is a file known as the checkpoint file (Enn.chk)which remembers the last successfully committed log file to the database so that it does not have to scan through all the logs to see who was the last committed one and where to resume. This will take more time in the recovery process. Again, the checkpoint file follows the same naming convention as transaction log files.

Mailbox database Cache:

This is a must know for all exchange admins.

This simply means the amount of transactions exchange store in its memory (Memory available in the server in the form of Transaction log files) before flushing it to the database. This used to be 20MB with exchange 2003 and got increased to 100MB in Exchange 2010. To make it more clear, Exchange 2003 transaction log file is 5MB in size, so exchange store’s 4 log files in its memory before committing the same to the database. With Exchange 2010, now 100MB = 100 log files will be stored in memory before committing the same to the database.

Didn’t get it? Read the portion “Background process by which data (email) reaches an exchange database” and read this again. This is a very critical piece of information for all Exchange fellas.

Page size

One of the prime reasons Microsoft came up with “only” 64bit architecture for Exchange 2007 and 2010 is to take maximum advantage of memory. Now with mailbox database cache increased from 20MB to 100MB, the amount of packets exchange could be able to store on the disk was inevitable. This is where Page Size comes into play. The Page Size difference with various versions of exchange as follows:

Exchange 2003 – 4KB

Exchange 2007 – 8KB

Exchange 2010 – 32KB

The Page Size is the minimum unit size needed for performing read and write operations to the database. Performing operations in memory is quicker to reading from the disk. Thus ESE reduces the IOPS by accelerating the page size to 32KB.

Reading from the disk is slower than performing operations in memory; therefore, by increasing the page size to 32 KB, ESE reduces IOPS, which increases performance by caching the larger page size in memory.

Long story short –> With Exchange 2010 running on a 64bit architecture with Mailbox database cache of 100MB and page size of 32KB, yes it does give an “overall IOPS reduction of 90% compared to Exchange 2003 and 70% compared to Exchange 2007″.

So far, that’s all you need to uncover the interpretation of ESE and Exchange database. I hope you could find it relatable and informative. Do give me your feedback on the article and your ideas/inputs to make it to more useful are indeed appreciated.

Ratish Nair
MVP Exchange
Team@ MSExchangeGuru

Keywords: Exchange database architecture explained, how logs files are committed to an exchange database, Exchange and ESE

Posted February 26th, 2011 under Database Management, Exchange 2010. RSS 2.0 feed. Leave a response, or trackback.