Retention Tags Fiasco - Part II

In my last post I talked about having to eat humble pie by going to the Powers-That-Be and admit to a mistake. A two week default retention tag was applied to a group of mailboxes and lots of mail was deleted.

As it turned out we are having to restore all the databases containing affected mailboxes. Many of those databases held secondary (archive) mailboxes. 94 mailbox databases in all I believe...

When we were restoring the primary mailboxes the command Restore-Mailbox worked flawlessly. It was speedy. It was accurate. Lovely.

Then we got down to recovering the data in the Archive mailbox, and things started to get sketchy.
Restore-Mailbox command does not understand archive mailboxes. Before I did this I called Microsoft Support with a simple question.

"I don't see a -Archive option for Restore-Mailbox, so just how does it does work with archive mailboxes?"
"Ummm, I see what you mean. Very possible it just knows the mailboxes in the databases are archive mailbox and will handle accordingly."
"So, very possible, or you're certain?"
"Yes, I'm certain."

Well he might have been certain, but when 9PM rolled around and he was watching his T.V., I was finding out the hard way he had no clue what he was talking about.

But fortunately for me, this guy did :
http://eightwone.com/2011/01/07/restoring-personal-archives/

So -- for instance -- I have a mailbox database, called Arch01. In the mailbox database there are only Archive Mailboxes.

So to get a reference to all the mailboxes that have archive mailboxes in that database:
(We only have 1600 users with archive databases.)

$ArchMBX = Get-mailbox -archive -Resultsize 2000 |
           ?{$_.ArchiveDataBase -eq 'Arch01'}

Then use the New-MailboxRestoreRequest command:

$ArchMBX | %{
   New-MailboxRestoreRequest -SourceDatabase Arch01Recovery `
   -SourceStoreMailbox $_.ArchiveGUID -TargetMailbox $_.Identity `
   -TargetisArchive -BatchName Arch01 -Suspend `
}

I am doing this on 16 very large databases and 16 small ones. The New-MailboxRestoreRequest command can be very slow. Not sure if the amount of data in some of these mailboxes (many are over 8G and a few are upwards into the 20G range) or if there is something else going on here. Some mailboxes were taking up to 18 hours to complete.

Seemed to me the Mailbox Restore Request was bogging down and getting slower and slower and moving less and less data. I did have about 450 requests queued up, but I've had more than that in some mail migrations. I experimented with restarting the Mailbox Replication Service on all the CAS servers in this AD Site. For me, every 4 hours seemed to be the magic time frame. I would just suspend all the jobs, then restart that service on all the CAS servers and resume the jobs.

Another odd thing about the New-MailboxRestore command is it was crashing store.exe on some occasions, like corruption in a mailbox. Initially, when a New-MailboxRestoreRequest failed, I'd set the -BadItem limit to something very high and resume it. Now I just make a note of that mailbox and move on. In my mind there is only a 25% chance they were hit anyway. Why sweat it? If they need to be restore, we would restore the mailbox database again, but from a different time frame.

Finally, I found that throwing a lot of recoveries at a mailbox database seems to over tax the log file space as well as indexing. We use circular logging but the data was being pumped in too fast. As item were being recovered, there were added to the indexing queue and filling it up, too.

So now I start all my new request with -Suspended to be on the safe side. I also choked the Mailbox Replication Service to only allow 5 mailbox restores max per target database. This seemed to help on those log file directories and eased the indexing some.