Exchange 2010 Site Disaster Recovery on a Dime! Part 3: Backup, Restore, Recovery

I began this series by explaining how to build a low cost site or datacenter disaster recovery solution with the new Database Availability Group (DAG) feature in Exchange 2010. Next, I covered the process of failing over to your other site in case of a disaster. Naturally, I hope you never experience a disaster in your Exchange environment, but if you do this article will ensure you are prepared.

In Part 3, I will walk you through the steps of performing a Backup, Restore, and finally a Recovery. While it is important to know how to do a proper backup, it is equally if not more important to be able to use it in case disaster strikes.

The Backup Process
Performing a backup of Exchange 2010 databases is not that difficult; just make sure that your backup software uses VSS technology since the traditional streaming backup API is not available in Exchange 2010.

The built-in Windows Server Backup has this capability, but it lacks many other functions that a real backup solution has. Therefore, I tend to rank Windows Server Backup as a poor man’s backup software. See my earlier post about Windows Server Backup and Exchange 2007 for more details. While the article is written for Exchange 2007, it is also applicable to Exchange 2010.

Other vendors are working on their backup software to ensure its compatibility with Exchange 2010 and some already have it working. An example is Microsoft’s Data Protection Manager 2010.

No matter which backup software you use, the steps for doing a backup are essentially the same. Backup software communicates with VSS, which in turn communicates with the Exchange Write that is installed during the Exchange installation. During this process, only changed blocks on disk will be transferred to the backup software, which is responsible for saving and storing data for later retrieval. By transferring only changed blocks on disk the backup time is decreased and so is the number of bytes on the wire.

In the example we have been using for this series, we have one server running in the primary datacenter and another server in the Disaster & Recovery (DR) datacenter.
The question arises: ‘Where do I do the backup -- on one or both servers?’ The answer is ‘It depends.’ (Don’t you love this answer!)

Your options are:
1. Do the Exchange database backup on one server and Exchange database copy of the other servers
2. OR backup only one server, but which one?

For the Exchange Admin who has been around Exchange awhile, the question about purging transaction log files always comes up. The beauty of the DAG design in Exchange 2010 can be seen during this process: when doing backup of a database in a database availability group (DAG), it will automatically purge the corresponding transaction log files on all replicas of that database. The server running the database where the backup is performed communicates to the other servers having a database replicated, telling them that a backup has been done and that it is now time to purge transaction log files. Which files to purge depends on several factors, such as checkpoint, replay lag time and truncation lag time. Thus, you should not expect them all to be purged with a normal full backup. With this in mind, make sure to size your transaction log LUN correctly if using replay lag time and truncation log time.

The Restore Process
The process of restoring a database located in a DAG is pretty much the same as doing it on a mailbox database that is not a member of DAG replication. The decision you must make is whether to use the lagged copy of the database or to perform a traditional restore.

How can we take advantage of the lagged database copy?
Lagged database copies can be used for recovering a logical corruption in a mailbox or mailbox database, or recovering individual mails or folders within a mailbox. The recovery process is simple, but you must consider the replaylagtime settings carefully so that you can discover a problem in time to use the lagged database copy before transaction log files are replayed into the lagged database.

Components needed for the recovery include a ‘recovery mailbox database’. The first step is to create a recovery mailbox database:

New-MailboxDatabase -Name RecoveryDB -Verbose -Recovery –EdbFilePath E:\Recovery\RecoveryDB.edb –LogFolderPath E:\Recovery -Server FQDNofServerInRecoverySite

This will create a recovery mailbox database with paths set to E:\Recovery

Next step is to get a file copy of the mailbox database you want to extract data from into the E:\Recovery folder. You could use a regular restore from your backup, but it’s often faster to make a copy of the lagged database. You may use the amount of transaction log files that suits your purpose.

Before doing a file copy, it is best to pause the replication with the Suspend-MailboxDatabaseCopy command:

Suspend-MailboxDatabaseCopy 'database name\FQDNofServerInDRSite'-SuspendComment "Recover data from database" -Verbose

We use VSS to do a shadow copy of the database we want to extract data from. The Syntax for vssadmin.exe command line tool is “vssadmin create shadow /For=”

As you can see, you can only do a VSS shadow copy for full volume, meaning the volume is either a disk such as D:\ or a mountpoint. You probably have database files and transaction log files on separate disks, so you must create shadow copies for both disks.

vssadmin create shadow /For=D:
vssadmin create shadow /For=G:

Pay attention to the result you get:
Shadow Copy Volume Name:
\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2

In previous versions of Windows, you could simply do a copy from the strange path above to your recovery folder, but this seems to be have either broken or been taken away in Windows Server 2008 R2. This is how it used to look:

copy “\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\path_to_your_edb_file” E:\RecoveryDB

But I discovered another way of doing it, with explorerer.exe:
Right click on your C drive, select Properties and Previous Version Tab. Here you should see the newly created shadow copy. Select it and click open. A new window opens in which you can drill down to wherever your Exchange database files are located and simply do a file copy of the edb file and corresponding transaction log files to the E:\Recovery folder. Which transaction log files you need to copy depends on how far forward you want to replay information into the database. Simply check the file stamps on the files. Warming: In real life, this file copy will take a long time!

The Recovery Process

Now we can do a recovery of the database in E:\Recovery folder. Start by deleting the checkpoint file “xxx.chk”. Next use eseutil from an elevated command prompt in the E:\Recovery folder:

eseutil /r xxx /a

Where xxx is the transaction log file prefix, such as E00. Output will look something similar to this:

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.00
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating RECOVERY mode...
Logfile base name: e00
Log files:
System files:
Performing soft recovery...
Restore Status (% complete)
0 10 20 30 40 50 60 70 80 90 100
----------------------------------------
..............................

This could take a long time depending on how many transaction log files you have to roll into the database file. The speed of rolling transaction log files is approximately 2 log files per second.

Next step is to rename the edb file to recoverydb.edb since that was the name chosen when we created the recovery database.

If everything has gone well, we can simply mount our recoverydb:

Mount-MailboxDatabase RecoveryDB

To see what mailboxes there is in the RecoveryDB, use:

Get-MailboxStatistics -Database RecoveryDB

To extract data from the Recovery database, use the Restore-Mailbox command:

Restore-Mailbox -RecoveryDatabase RecoveryDB -TargetFolder Recovery -Identity 'target mailbox' -RecoveryMailbox 'mailbox to get data from' -BadItemLimit 999 –Verbose

You could use several more parameters with Restore-Mailbox such as -ExcludeFolders, -SenderKeyWords, -AttachmentFilenames, -ContentKeywords, -AllContentKeywords, and many more. See the documentation on TechNet for full syntax of Restore-Mailbox.

When restore-mailbox command is finished, you will see a folder structure inside the ‘target mailbox’ named ‘Recovery’ with the extracted data beneath.

Now it’s Time to Clean Up

Now that you have managed to extract data from the lagged copy, you must start cleaning up (‘But Mom!’) First delete the shadow copy; otherwise, it will eventually fill up the shadow storage disk space.

Start by listing your current shadow copies with:

vssadmin list shadows

Look for the shadow you made before. (For example, timestamp is good to use.) Then delete the shadow copy with:

vssadmin delete shadows /Shadow=ShadowId

Where shadowID looks like {XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX}

When you have finished, use the Recovery database. Simply delete it with:

Remove-MailboxDatabase RecoveryDB

And delete files in the E:\RecoveryDB folder.

Don’t forget to un-pause the replication of transaction log files with Resume-MailboxDatabaseCopy:

Resume-MailboxDatabaseCopy -Identity 'Mailbox Database 2036433681\FQDNofServerInDRSite' –ReplicationOnly

-ReplicationOnly is there to stop ActiveManager from accidentally activating the database DR site.

What if you encounter a corrupted file and need to recover the complete server but also want to go back a few hours in time? (and your DeLorean is all out of plutonium…)
You can always use your regular backup, but you could also use the lagged copy. Using the lagged copy in this scenario is even simpler than described above. Suspend replication, delete checkpoint file and as many transaction log files as you need to “go back in time.” Then select the amount of time and use eseutil /r to replay the transaction log files left on disk.

Next step is to do a switchover to the recovery site and server. Please see part 2 of this series for more detail: Exchange 2010 Site Disaster Recovery on a dime! Part2: Navigating the Failover process

Another approach would be to use a dial-tone database together with a recovery database. I will save this discussion for a future article.

Exchange 2007 Service Pack 2 backup feature

It has been a lot of disappointment of the missing feature of a native backup solution in Exchange 2007 when running on Windows Server 2008. It has been so much roar out there that Microsoft decided to do something about it. With Exchange 2007 Service Pack 2 the native backup of Exchange capability is back.

It will not be a separate software or anything that is visible in a GUI somewhere, it will just be an added feature of Windows Server Backup since Windows Server 2008 don’t use NTBackup anymore.

The Windows Server Backup is aimed towards small or even medium size organizations and also as test and troubleshooting tool, so don’t expect to much of it.

How to do a backup of Exchange with Windows Server backup?
First you must install Windows Server Backup. This can be done from servermanager, add feature and select Windows Server Backup feature or from command prompt with “ServerManagerCMD.exe -i Backup Backup-Tools”. There is no reboot required and after the installation.
Start Windows Server Backup on your Exchange server. You must do the backup locally, there is no over the network backup capability. A shortcut is created during installation in Administrative Tools or you can run “wbadmin.msc”

After Windows Server Backup has started you can select to create a scheduled backup schema or simply do a single backup. Click the “Backup Once…” link in the Actions pane, The Backup Once Wizard starts. Select “Different Options” and then click “Next >”, Select either “Full Server” or “Custom”, if Custom is selected you are prompted with options of which disk to backup.
There is no option of only doing backup of Exchange databases or only selecting some folders. The granularity is the complete disk and nothing smaller.
Select the disks that you know contain Exchange files. You will not see that Exchange databases and transaction log files are selected for backup, you must know where your Exchange files are located. Click “Next >”.
Next option is to select the destination for the backup. If you select Local drives you must select a destination drive that are not included in the backup. The other option is a Remote shared folder, then you have to enter a shared folder on another server on your network. Backup will be created in a subfolder called WindowsImageBackup with the Access Control of either Inherit or Do not inherit (equals, you specify an account that are granted permission to the backup files).
If the destination location already contain a backup it will be overwritten.
Next selection is to either do a VSS copy backup or a VSS full backup, there is no option of the old traditional streaming or incremental backup.
Copy backup will simply copy the selected disk or disks to the destination, the full backup will do the same but also purge the application log files . This is the Exchange transaction log files and this is the option you should select if you’re not using any other backup software on your Exchange database files.

Backup will be placed in a folder called WindowsImageBackup and consist of a couple of XML files and the selected disks stored as VHD files. Virtual Hard Disc files are pretty cool, it is the native file format for hard disc with Microsoft virtualization technologies and you can copy or transport those files around and mount them on other computers for examination or whatever reason you might have.

Doing recovery of Exchange databases with Windows Server Backup.
When doing a recovery the database in question must be dismounted first. In a real world scenario it is already since you probably lost a hard drive or you did a disaster recovery installation of the server and its now time for recover the databases. Databases will be dismounted if needed by the Windows Server Backup.

Start the recover wizard by clicking “Recover …” in the actions pane in Windows Server Backup.
A recovery of Exchange database and transaction log files can either go to the original location and replace the original files or be destined to the Recovery storage Group (RSG) or even to a different server.
For this exercise select the local server and then select a time when you did the backup you want to recover, next select applications since it is the Exchange application data we want to recover and then select Exchange. If you click on the “View Details” button you will see databases that will be recovered, you cannot select individual databases to be recovered, all databases will be recovered. To have a more granular option; you must create multiple backups with different databases in each backup and for this to work each database and transaction log file must be on separate disk since the smallest option of granularity is the complete disk. The checkbox “Do not perform a roll-forward recovery of the application databases” means that Exchange will not roll forward transaction log files into the the recovered database. Next option is to recover to the original location or another location. Select original location. Recover to a different server or different location will be explained in another article.
Click recover on the Confirmation page if you is satisfied with you selections and to start the recover of you Exchange databases.
If you did not have dismounted the databases before you started the recovery they will be dismounted by Windows Server Backup. Permission to dismount databases is required for this to work as expected.
If everything goes well, database and transaction log files will be read from backup and written back to disk and finally transaction log files will be replayed into the database

Conclusions:
Does it work? The answer is, it depends. It will work if being used for doing backups and doing restore in a disaster scenario. It will not behave as expected when doing restore to a different server or different location, such as the Recovery Storage Group.

Windows Server Backup with Exchange plug-in don’t have granularity when doing backup or when doing a recovery, it will recover whatever there is on the backup. It’s all or nothing type of backup and restore.

Windows Server Backup can be configured to do scheduled backup from the GUI, but you can also use Windows Server Backup from the command line. This gives you more flexibility to make your own schedules with different options. I am thinking destinations here since each destination can only hold one backup, so if doing multiple backups to the same destination, only the last will be preserved.

Command line reference for Windows Server Backup.
http://go.microsoft.com/fwlink/?LinkId=93131

How to use the command line and to do a restore to the Recovery Storage Group is a story for another day.

Why do you backup Exchange databases?

The biggest challenge with Exchange backups facing administrators today is the time factor. This factor comprises both the time necessary to perform the backup and the time necessary to complete the restore. Users want to have more data in their mailboxes; but in order to adhere to the SLA commitment, quotas must be set on users’ mailboxes. The end result? Users get frustrated from having to constantly manage their mailboxes in order to maintain a pre-set quota limit.

This situation is most noticeable if you use the old traditional streaming backup API that has been in Exchange since the beginning and is still there today. Since there has not been any active development, however, the recommended backup method is to leverage VSS (Windows Volume Shadow Copy Service). http://msexchangeteam.com/archive/2008/08/25/449684.aspx

With streaming backup API, you transfer every bit and byte of Exchange database to the backup software while performing a full backup. VSS operates a little differently. Instead of streaming the database file, it reads changed blocks from disk and then hands it over to the backup software. This procedure saves a lot of time when doing differential or incremental backups. Depending on how you configure the backup software to handle VSS, it can perform tasks such as incremental backups while keeping track of which blocks belong where. From that, an administrator can build up a complete backup in case he or she needs to do a restore. The benefit is that small backups are always done--only the restore is large and takes a long time.

When doing backups with streaming API, it automatically performs a consistency check by reading every byte of the database file. With VSS, however, you must tell the backup software to do the consistency check, otherwise you risk backing up a bad database file.

An option is to allow your backup software to integrate with Exchange and permit the storage to leverage functionality found on the storage to handle copies of LUN’s. In this case, the backup will only take a few seconds as far as the Exchange server sees it. You should still complete the time-consuming task of transferring the files located on the backup LUN to tape or other disk. This solution is dependant upon whether or not your storage is configured with this functionality and whether or not your backup software can do the integration.

No matter what backup method you use, all the backup data must be stored somewhere.

What if there was a way to have all your Exchange data spread out to multiple copies that automatically were kept in sync? With Exchange 2007 CCR together with SCR, you could make this happen. CCR is built upon a Window’s failover cluster, and data is replicated between the two nodes automatically. This produces two copies of Exchange data, and by introducing an SCR you can also introduce a third copy.

What happens if you need more? Simply add additional SCR copies. SCR are techniques that do not replay the copied transaction log files at once; they have a “lag” before doing the replay into the Exchange database. Lag time can be configured so that you can run a CCR with instant replaying of transaction log files to have two copies of Exchange data always updated. Then, you can have one SCR copy lagging behind a couple of hours and additional SCR copies lagging behind a couple of days.

http://msexchangeteam.com/archive/2006/08/09/428642.aspx http://msexchangeteam.com/archive/2007/07/19/446454.aspx

A common scenario is a user who has accidently deleted something from his or her mailbox. This delete instruction has replicated to the two CCR nodes, but the delete instruction may not have been replayed to one of your SCR copies. In this case, you can still recover the deleted mail from your SCR copy by stopping replication and then replaying the transaction log to the SCR database. Finally, mount the database and retrieve the data.

Another cumbersome task is managing the huge amount of data in Exchange. This data causes a lot of IO to the disk subsystem where Exchange has its databases and transaction log files. This is another reason to keep the databases small.

With Exchange 2007 the IO drop by 70% so you can allow the databases to grow without spending a fortune on the storage system. http://msexchangeteam.com/archive/2008/07/10/449188.aspx

So what about the idea that you run Exchange 2007 in a CCR and SCR configuration to be sure that you have multiple copies of Exchange data around? You allow users to have large mailboxes and perhaps you switch your storage solution from an expensive SAN, both money-wise and technology-wise (i.e. specially- trained personnel, fiber channel technology etc.), to a cheaper solution like DAS or iSCSI.

The benefits of this switch would include cheaper storage, meaning that users will no longer have to delete items from their mailboxes and move them into a PST file. With large mailboxes, the need for a restore is minimized and, therefore, also the need for doing backups.

But there still can be a need for doing backup and restore if your databases and disk fails. Should this happen, you have multiple copies of Exchange data already. Plus, it is a very simple process to make the other node of the cluster start acting as your Exchange server with a healthy set of data--this is in most cases done automatically. You can also manually activate the SCR copy with its own set of data.

All this information pertains to Exchange 2007, but Exchange 2010 will solve this problem with its new cluster technique, Database Availability Group or DAG for short. http://technet.microsoft.com/en-us/library/dd633496(EXCHG.140).aspx

So the question is why do you do backup of your Exchange data? Some may say that they need to store a copy of all data offsite. Although this is a good and valid reason, it will also be solved by having the SCR copy located somewhere else. It could even be on the other side of the world!
I am not saying that running without backup will suite everyone, but chances are you will discover that backups are not as necessary as you initially thought.