Mailbox Move error:Couldn't switch the mailbox into Sync Source mode

I've been working on a project to import PST files for quite some time. Some users we've helped have had many PST files and some had very large files.

One of our Archive Mailbox databases became very large for our tastes, so we decided it was time to trim all these databases down to 100G each from 380G each. Our main issue with the size was a place to restore. The time it takes to restore the database wasn't a problem, these were archived messages and the SLA was a full day. The problem was the 400G chunk of free space needed to do the restore. That was hard to find.

So as we moved archive mailboxes, I came across this error from time to time. The first time I saw this error, we were dealing with a user's mailbox that had over 22,000 folders. So I just exported her archive mailbox into several pieces and then imported it into another database. I just assumed that was a one time issue, and just moved on to my next task.

Then it happened again. This time is was on an archive mailbox 35G in size. I started searching on the error and found very little. Here's the error:

==============================================================

FailureCode                      : -2146233088

FailureType                      : SourceMailboxAlreadyBeingMovedTransientException

FailureSide                      : Source

Message                          : Error: Couldn't switch the mailbox into Sync Source mode.

                                   This could be because of one of the following reasons:

                                     Another administrator is currently moving the mailbox.

                                     The mailbox is locked.

                                     The Microsoft Exchange Mailbox Replication service (MRS) doesn't have the correct

                                   permissions.

                                     Network errors are preventing MRS from cleanly closing its session with the Mailbo

                                   x server. If this is the case, MRS may continue to encounter this error for up to 2

                                   hours - this duration is controlled by the TCP KeepAlive settings on the Mailbox ser

                                   ver.

                                   Wait for the mailbox to be released before attempting to move this mailbox again.

FailureTimestamp                 : 9/15/2012 6:24:00 PM

FailureContext                   : --------

                                   Operation: IMailbox.SetInTransitStatus

                                   OperationSide: Source

                                   Archive ()

                                   Status: SyncSource

==============================================================

I finally called Microsoft Support who said "This is a very common issue. Here's what you need to do... Change the TCP KeepAliveTime for the servers involved in the move. Source, Target, and CAS."

http://technet.microsoft.com/en-us/library/dd349797(v=WS.10).aspx

It turned out that I had to do that so much, I wrote a simple script to make the changes for me easily.

Get the code here: http://poshcode.org/3808

Enterprise Wide PST Import - The Functions

This is a Part 12 in a series of posts about my experience tackling the migration of PST files.

The first post in the series is here.

(This is the last post in this series.)

The PST Utility Module is really just a collection of functions. Here's a list of the functions and short description of how they are used.

Get the module here.

Queue Processing:

These queue processing functions are used to do the work of Importing PST into Archive Mailboxes as well as maintenance of the queues.

Add-PSTIMportQueue
Given a user name, find their PST files, set up the job queues for the PST files, and notify the user and admin with an initial report. A more detailed explanation is here.

Process-PSTImportQueue
Loop through each job and take appropriate action until the job goes through all the stages of: Copied,  Imported, Cleaned up, and Notified. A more detailed explanation is here.

Set-PSTImportQueue
Set different attributes on a user’s jobs, or a single job. A more detailed explanation is here.

Remove-PSTImportQueue
Remove a user or a job from the queue.

Report-PSTDiscovery
Finds all PST files for All Mailbox Users and tracks their usage – weekly run

Report-PSTOverallSummary
Generates a daily detailed report in an HTML file – also sends a summary report to the Admin and Boss.

Admin Functions:

These functions are meant for Admins to run. They can help the Admin decide how to process a user's PST files.

Check-ForPST
Same as the Get-PSTFileList above except input is geared to an Admin and not a Job Object

Get-ArchiveUserInfo

A quick report about a user, do they have an Archive mailbox already, GPO, etc

Get-ImportStatus
A quick way to check on import queues and suspend and restart, etc

Move-PSTs
This checks for and moves a users PST file from the Home Share to their local PC

New-OnlineArchiveByUser
This adds the Archive Mailbox to a user, setting all the defaults. For people without PST files to import

Optimize-PSTImportQueue
Sort Users by number of PST files, so the most number of users get done quickly

Helper Functions

Add-ToGPO

Add a user to a group we use to control Disallow PST Growth

Adjust-Quota
Sets quota to default for Online PST users

CC
Counts collections 0, 1, many

ConvertTo-ComputerName

Returns the computername for an IP

ConvertTo-IP
Returns the IP for a given Comptername

Clean-OnlinePSTGPO
In our world, there is a distinct set of rules for the GPO, we check them here. Add people who are missing and remove people who should not be there.

Copy-PSTusingBITS
Uses BITS to copy the PST file to the share for processing. From PC’s and far away sites

Copy-PSTtoImportShare
Uses copy-item to copy the PST file to the share for processing. Used for local AD site files

Format-JobStatus
Returns the job status based on user or overall

Get-ISODayOfWeek
Returns the day of the week for a given date

Get-FreeDiskSpace
Returns the free space of a given computer drive – used when moving PST files back to a local computer.

Get-ClientAccessUserLog
Returns raw info on a user extracted from RPC logs on CAS servers

Get-MRSServer
Returns the MRS Server to use for this move

Get-OutlookEXEVersion
Returns the version of Outlook installed on a computer by looking at the EXE

Get-SingleMailboxObject
Returns a mailbox object, if results are anything other than a single entry, it returns nothing

Get-PSTFileList
Return the PST file collection from Home Shares or PC or some directory

Get-ArchDB
Returns the best candidate mailbox database for the Import. By smallest mailbox database size.

isMemberof
Tests the membership of a group

Import-PSTFromShare
Starts the import process using the Job Object for settings

Lock-PSTIQ
Creates a zero length file to signal processing is happening

Move-PSTToLocal
This moves a PST file from a Home Share to the local PC – if all test conditions are true

New-ArchPSTFileLogObject
Creates a new object for logging the PST file activity of Users with Archive Mailboxes

New-ArchPSTIndex
Finds or creates a new Index entry for a PST file. Used to search the PST related Filelog Objects

New-PSTFileLogObject
Creates a new object for Logging all PST files for all users

New-PSTFileIndex
Finds or creates a new index entry for a PST file object for All Users

New-PSTJobObject
Creates the Job object used in PST import Jobs

New-PSTReportObject
Creates object used in PST Reporting

New-OnlineArchiveByJob
Gives the user an Archive Mailbox, using the Job Object passed. Only used in Add-PstImportQueue

New-TargetPSTDirectory
Creates a new directory in the share using the User’s log in name

Reset-PSTBackupFileNames
Resets all Backup filenames to Now() using the Format ‘<filename>-yyyyMMddHHmmss.<ext.>’

Send-NotificationInitialReport
When add-pst is run, and discovers PST files to process, a summary is sent to the user.

Send-NotificationFinalReport
When processing is finished, the results are sent to the user.

Send-NotificationPSTRemoval
When reports are run, a message is sent to the user about still connected PST files (14 day cycle)

Test-OutlookClientVersion
A way to control what client version are accepted for import

Test-PSTIQLock
Test if the Queue is locked, returns true/false

Unlock-PSTIQ
Removes the lock – when the processing pass is done

Introduction: The Beginings

Part 1: Script Requirements

Part 2: Add-PSTImportQueue

Part 3: Process-PSTImportQueue

Part 4: Some Tools we added

Part 5: Set-PSTImportQueue

Part 6: About PST Capture

Part 7: More PST Import Tools

Part 8: Using RoboCopy

Part 9: Morning Status Report

Part 10: Using BITS Transfer

Part 11: Get the script / Set up

Part 12: The Functions

Enterprise Wide PST Import - Get the script / Set up

This is a Part 11 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in the series is here.

I put these scripts together over time and having to go back and look at all the moving parts makes me realize how much time went into this.

Get the module here.

So follow these few simple steps and you're ready to import PST files!

First you'll need the permissions:

Granting the permission to do PST file Import and Exports is explained here. Just assign yourself the role.

As for finding and copying PST files, well you might need a little more umph. I have an account that has Domain Admin privilages. That account is for Admin duties only. (I have separate 'normal' account I use for everything else.) My admin account also gives me access to all the computers where PST files may be hiding, since Domain Admins is in the Local Administrators group on each computer.

You could use a service account which has all these privilages and run these functions using that account. I tend to not like using service accounts because you lose accountability.

Create the Share:

Exchange PST file import and export cmdlets require a share. Find a server that can handle the amount of PST files you plan to import and create a share. Grant the "Exchange Trusted SubSystem" group full access to this share. We named our share "PSTImports."  It's why all the folder structuture begins there.

We were lucky enough to have a utility server lying around that has 1TB of space on one of the drives. We use the server for reporting and performance history. And there was plenty of free space to handle importing 800G of PST files in one pass. We never got that high, and we even added pieces into the script that examines the free space of our server to choke PST copies in case we ran short of space. You should adjust this setting for your server. It is $Script:FreeSpaceLimit and is set in MB.

You'll want to create some other directories within this share:

• ^Completed
• ZZSavedQueueLogs

Other directories are created during the import for each user being processed to hold their PST files. After the Import, the PST files are replaced with results of the Import in CSV, then moved to the ^Completed directory for historical reasons. It's named with the leading ^ for sorting purposes. It will always be at the top ;)

ZZSavedQueueLogs is just a backup queue files. A backup is made each time you run the functions. I can't tell you how many times having those backed up has saved me some extra work.
I check this directory -- when I think about it -- and kill any older stuff.

Add PSTUtils As A Module and Optionally add to profiles:

Log on the server with the account that has Local Administrator rights and create this directory:
C:\Windows\System32\WindowsPowerShell\v1.0\Modules\PSTUtility
And copy the PSTUtility module there.

When you load up powershell then "get-module -listavailable", you should see the PSTUtility as a module you can import. Just use the command "import-module PSTUtility"

Add this to your $profile so you have it available each time you log in.
I use this command all the time: "notepad $profile" make my changes and then save. Restart powershell to load the new profile changes and you are ready to go.

Edit the Module: look over and change the $Script: level variables to suit your particular environment.

Configure Initial IP/Computer searches: (Optional)

When we get a request for a user's PST files to be imported, we usually just get a name. I'd love to have the computer name so I can determine if the Outlook Version on that computer will indeed support the Archive Mailbox feature. It's not always supplied. So we ask. "I don't know. He took his latop with him and he is gone for a few days." Let's just say, it's not always easy to get. Even if you ak the users and tell them how to find it, its a hit and miss proposition.

One can find a user's log in information in the RPC logs of your CAS servers. But theres a lot of information in those logs ... lots!  So I wrote a small utility that grabs what I call 'Connect Logs' - this is an subset of the RCP logs where OUTLOOK.EXE does a status of "Connect."  This entry holds the Version of Outlook connecting, legacyExchangeDN, the Mode (Cached or Classic), and the IP address. Looking thru this subset of logs vs the full logs is a time saver when looking up 50 people and trying to find the information on them.

Of course, users don't always 'connect' every day. They leave Outlook running for days and days. So their connect log may be there but it was last Thursday. I tried various combinations of how to search for this data and how far back to look etc.

I finaly just decided to get 30 days worth of these connect entries and use those for all my reporting and PST file imports. I have a script that runs every night at 2AM and gathers this subset of logs. It's "Get-CASConnectLogEntries" And you can copy that piece out, or add the 'import-module PSTUtils" to the profile of whatever is running the script. I have it running seperate in a batch file I have for overnight reports.

The directory location I keep these is in a different place where I keep all my reporting information. You can of course modify the variable and keep it where you like. I use these connect files for other reports too. There's lots of information in those logs.

I should add, running Get-CASConnectLogEntries at 2 AM is optional. If you know the computername each time you do Add-PSTImportQueue, you can bypass setting all this up. When you know the computer name, you can query the computer for all the information you need.

That's all there is to setting up everything.

Once you have these pieces in place, the hard part starts. Finding people to migrate to the Archive Mailbox.

Next, I'll be posting a list of all the Functions and how I use them.



Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions

Report-RecipientCounts - And Newborn Kittens

Yesterday some lady sent me an email -- at work -- and she needed to know if I wanted some of her newborn kittens. At first, I'm like: "Duh! As if!"

I don't know this woman, or care too. I just shrugged it off.

Then I heard my fellow admins cringe or moan one after the other the same email. Then someone piped up, "How many people did this go to?"

Crap! I should have seen this coming and just as I'm finishing up my trace message commands, into my cube walks one of the "Powers-That-Be" wanna bes.

"How can this happen? I though we had this locked down so no one could send a message to everyone in the company!" Crap! She sent it to everyone? No wonder my command was taking longer than I had expected. When my report came back, she had sent it 3 times, not just once. The first 2 times she tried some groups and found out she did have the permissions needed to send to the larger groups.

But new born kittens are important and the news must get out! She pulled up the address book, clicked the 1st person, then <ctl><a> and had everyone selected and put them on the To line.

This really happened. (Well, not exactly like this but very close.)

Protection from the Kitten Email Flood

You protect yourself with Default Recipient count maximums:
(Get-TransportConfig).MaxRecipientEnvelopeLimit - We had ours set to 4000.

In the Olden Days (Exchange 2003) the way recipients were counted is different than today in Exchange 2010. Back in 2003, the groups were all expanded before there was a count of users. But there were issues with doing the expansion before the count, I gather -- something about expanding too many groups and taking forever to finally deliver. I can't remember all of them. So the Microsoft Exchange Boys changed the way the count of users was made. Distribution Groups were now just counted as one. So if I sent a message to myself and a group containing 50 people, the recipient count would be 2.

Our 4000 setting was left over from those bygone days of 2003. 4000 was just fine for our world of 25,000 users. Many people sent to groups and those recipients ultimately numbered high in the 1000's but never as high as 4000. The setting was just ported over to 2007 and then to 2010. We didn't have any reason to take notice.
Until now.

So Make A Report Already!

So the burning question of the day went from "How did this happen?" to "Can we set the Max to 20 recipients?" -- well that's a bad thing too. People must work. In my book, you slap the hand of the person who did the bad deed and move on. I convienced the Power-Wanna-Be that we had to change the number, true, but we need to know what a good number is.

Then they asked "What is normally the highest number of recipients sent to each day? Each week?"
I had to say, "I do not know ... But I can find out."

So I wrote this script which keeps a running tab of recipient counts.
You can get it here.

Enterprise Wide PST Import - Using BITS Transfer

This is a Part 10 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in the series is here.

Times Change
Our original goal was to remove all PST files from our user's Home shares and into their Exchange 2010 Archive Mailbox. This has expanded to include all PST files everywhere now. I understand the legal discovery benefit. There is also a Help Desk benefit because users are always calling in about this PST file or that PST file. It's just better not to have PST files at all. So the Powers-That-Be modified our goal to include all PST files no matter where they might be hiding.


The New Challenge
When asked, we were importing PST files for people who were not at HQ as a courtesy. This was limited to a small group of people. While migrating those users, I hit the barrier of bandwidth. Some locations just don't have that much to spare.

Starting off, I just tried to do a conventional OS copy, which didn't work so well. There were several cases where the network team was called in because something "clogging up the WAN and it was very slow."

I tried RoboCopy and posted about using RoboCopy here for pulling the PST files back across the WAN line. Mainly because I knew it had the feature to pause and allow other traffic to move. Seemed like the prefect solution. But there were issues on my end. Like using a conventional OS copy, RoboCopy was always running in the current session of Powershell, or some spawned Powershell session. If you forgot that and exited the session, or logged off the server, the copy stopped.

BITS version
I saw a post on Powershell.com (here) about using BITS transfer and started to investigate. In a few tests BITS performed as advertised. The WAN did not suffer -- the network and server teams had already configured BITS on all PC and servers to operate in a "friendly to the WAN" way. I was just hopping on their bandwagon.

So BITS is my answer for the file transfer. If I rebooted my utility server, the job was still there. If the user rebooted their PC the job was still there.

There are still issues; The PST files are in use nearly all the time, The PC goes to sleep, Users carry their laptop home. Many are all normal problems that can't be scripted. ;)

PST files being "In Use"
We either wait until the user goes home for the day and tell them to shut down Outlook and pray they do. Or we talk them into disconnecting the PST files. If the users want the PST files imported, they usually are very willing to help.

Disconnecting PST files
Users are not to savvy about disconnecting PST files. In many cases we have to help. Usually when we get to this stage, there are many reasons we are helping them. They just don't ever get this kind of experience and don't have a clue. Someone set up PST files for them and they just use them as folders.

PC going to sleep
This requires some hands on. We can send the instructions to the user if they are a local admin on their PC, but many are not. We talk with the Local Technician and ask if they can help us by turning off that feature for a period of time. Or we just log into the machine to do that remotely.
(Will this be my next Powershell script to turn off power save mode and turning it back on, with the PC not having Powershell installed? I should look into that...)


Status Update
This project is moving along smoothly and we've finished almost 50% of the PST on Home Shares. I think I am about done posting about this project, most of what is happening now is just routine. Soon I'll be posting the Module which has all the code. I hope to post that in the next two weeks to a month.



Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions

Retention Tags Fiasco - Part II

In my last post I talked about having to eat humble pie by going to the Powers-That-Be and admit to a mistake. A two week default retention tag was applied to a group of mailboxes and lots of mail was deleted.

As it turned out we are having to restore all the databases containing affected mailboxes. Many of those databases held secondary (archive) mailboxes. 94 mailbox databases in all I believe...

When we were restoring the primary mailboxes the command Restore-Mailbox worked flawlessly. It was speedy. It was accurate. Lovely.

Then we got down to recovering the data in the Archive mailbox, and things started to get sketchy.
Restore-Mailbox command does not understand archive mailboxes. Before I did this I called Microsoft Support with a simple question.

"I don't see a -Archive option for Restore-Mailbox, so just how does it does work with archive mailboxes?"
"Ummm, I see what you mean. Very possible it just knows the mailboxes in the databases are archive mailbox and will handle accordingly."
"So, very possible, or you're certain?"
"Yes, I'm certain."

Well he might have been certain, but when 9PM rolled around and he was watching his T.V., I was finding out the hard way he had no clue what he was talking about.

But fortunately for me, this guy did :
http://eightwone.com/2011/01/07/restoring-personal-archives/

So -- for instance -- I have a mailbox database, called Arch01. In the mailbox database there are only Archive Mailboxes.

So to get a reference to all the mailboxes that have archive mailboxes in that database:
(We only have 1600 users with archive databases.)

$ArchMBX = Get-mailbox -archive -Resultsize 2000 |
           ?{$_.ArchiveDataBase -eq 'Arch01'}

Then use the New-MailboxRestoreRequest command:

$ArchMBX | %{
   New-MailboxRestoreRequest -SourceDatabase Arch01Recovery `
   -SourceStoreMailbox $_.ArchiveGUID -TargetMailbox $_.Identity `
   -TargetisArchive -BatchName Arch01 -Suspend `
}

I am doing this on 16 very large databases and 16 small ones. The New-MailboxRestoreRequest command can be very slow. Not sure if the amount of data in some of these mailboxes (many are over 8G and a few are upwards into the 20G range) or if there is something else going on here. Some mailboxes were taking up to 18 hours to complete.

Seemed to me the Mailbox Restore Request was bogging down and getting slower and slower and moving less and less data. I did have about 450 requests queued up, but I've had more than that in some mail migrations. I experimented with restarting the Mailbox Replication Service on all the CAS servers in this AD Site. For me, every 4 hours seemed to be the magic time frame. I would just suspend all the jobs, then restart that service on all the CAS servers and resume the jobs.

Another odd thing about the New-MailboxRestore command is it was crashing store.exe on some occasions, like corruption in a mailbox. Initially, when a New-MailboxRestoreRequest failed, I'd set the -BadItem limit to something very high and resume it. Now I just make a note of that mailbox and move on. In my mind there is only a 25% chance they were hit anyway. Why sweat it? If they need to be restore, we would restore the mailbox database again, but from a different time frame.

Finally, I found that throwing a lot of recoveries at a mailbox database seems to over tax the log file space as well as indexing. We use circular logging but the data was being pumped in too fast. As item were being recovered, there were added to the indexing queue and filling it up, too.

So now I start all my new request with -Suspended to be on the safe side. I also choked the Mailbox Replication Service to only allow 5 mailbox restores max per target database. This seemed to help on those log file directories and eased the indexing some.

Retention Tags Fiasco

An Innocent Beginning
A user requested to have a Personal 'Delete' Retention Tag for 14 days. I created one and added it to the policy that is applied to about 1600 users.  A personal folders tag is meant for folders created by the user and can be applied to any folder. We also have a "Default -All other folders 14 Day Delete" which deletes anything over 14 days old in any normal folder without a policy applied. Essentially, the default policy.

I hate to admit this, but I did a pretty dumb thing. I chose the wrong one and applied it to the Policy for the 1600+ users.

So the person who requested this new policy calls me and says, "Hey, this was supposed to just delete stuff in my single folder but it's deleting everything older than 14 days!" Of course I think he is the typical crazy user. Then I look at my mail and notice that some old stuff I need is gone. I looked at the policy applied to me and lo and behold, there it was, big a life. "14 day delete (2 weeks)" stamped all over my mailbox folders. This is where I panicked.

I went to all 32 mailbox servers and stopped the service for the "Exchange Mailbox Assistants." I prayed that this did not get too many people... (Yes, I know I can run a command to do this remotely, but our servers are not set up for that as yet. I just never got around to it. I'll get to that as soon as I finish ...)

Then I had to humbly take myself to the boss and say, I screwed the pooch. Not only did I not have a Change Management ticket, but I made the change during the day. Two major infractions. I guess I was thinking this was only a minor maintenance thing, or I was not capable of screwing up. Probably a lot of both. So I had added this policy in, a very simple thing for sure, but it caused a very big problem.

The Powers That Be wanted to know:

1. Who did this effect?
2. How can we get their mail back?
3. How do we erase the Tag?

A call went out to Microsoft. I wanted to find the quickest way to reverse what I had done.

Who did this effect?
We bumped the logging level up to expert so when we started the Mailbox Assistants back up again we might find some indicators as to which mailboxes had been touched. Then we started looking through the event logs to see if there was any indication of mailbox identities already there...

Event 9001 - the service is starting on a database
Event 9017 - Mailbox Assistant starting work, there are 600 mailboxes in here, etc -- but no information of which ones we're working on.
Event 9018 - there was a failure for a mailbox. Not saying which or why it failed. Just a count.
9021 special request started
9022 special request finished
9025 a mailbox was skipped - here is the GUID - seems many of these were disconnected mailboxes
9037 there was an error on something
9112 - list of stale mailboxes with decayed watermarks - not sure what this is yet

Bottom line: There is no way to tell who this affected. You have to look at the mailboxes themselves. But there were 1600 of them. We can't open each one. So we had to bite the bullet and send a message to all the 1600 users saying "uh, you may not have been hit ... but, uh, look at a message. If you see <this> you might be one of the people we want to help. Oh, and you can also tell because you may or may not have lost all you mail older than 14 days." -- Boy, that's gonna hurt.

How Do We Get Their Mail Back?
Restoring from backup is the only way. Unless you can live with the solution of recovering deleted items into a newly created folder. We gave that to users as a temporary solution, but did not expect them to want to refile all that data. So, again.. restore from backup.  To top it all off we don't have that much room left on any servers.

These people are scattered all over the place with primary mailboxes in one database and secondary mailboxes in others. There are 72 database involved. 16 are housing secondary mailboxes with a size of 325G or more. There is hardly a place anywhere to restore those databases. Luckily there were two servers with enough space to handle two of these databases each. The other 56 databases we got over two nights, putting one database on each server, where we could. (Did you remember that you can only have one Recovery Database mounted on a server at one time? I had to learn that one again!)

How Do We Erase The Tag?
I was afraid this was going to be the hardest part, and it turned out to be the easiest.
"If thy policy offends thee, pluck it out!"

So we set the Retention Policy on all mailboxes to $Null
Then we removed the offending Policy Tag from the Retention Policy
Then we deleted the Policy Tag
Then we deleted the Retention Policy
Finally, Restart the Mailbox Assistant Service
Since there is no Policy Tag to reference, it removes the Tag from the message. One could have just as easily applied a new policy and it would have overwritten any existing Tags. We never tested that option though. We removed all the Policy Tags and started the Mailbox assistant service on the server with my mailbox. All the scary tags disappeared.

Fixing The Wrong:
First I wanted all the mailboxes in the database that had a secondary Archive mailbox. I sorted them because I wanted some kind of easy indicator of how far along the process had gotten. The alphabet was good enough for me.

# find only the people with Archive Mailboxes
$MBX = Get-Mailboxdatabase DB01 | get-mailbox -archive | Sort DisplayName

Next it was simply piping the results to the restore command:

$MBX | Restore-Mailbox -RecoveryDatabase 'DB01Recovery' -MaxThreads 25


Lesson Learned
Don't be too lazy: I even thought about creating a test Retention Policy and applying it to the requestor but I never did. I remember talking myself out of it -- "It a person tag, just do it and move on to the next task on that long list of things to do today."

You're never too busy to do something right the first time: I am reminded of this yet again. A few years from now, I'll get in a hurry again and Murphy's Law will hit me over the head. It can actualy save you time to take the time to do it right. This screwing the pooch cost me many hours. Let's not forget the backup team, who had to jump through hoops to get me all these restores.

Ongoing Indexing Issues on DAG02

See the update to this problem here.

We've been having issues with indexing on one DAG and we thought we had it fixed when we went to Service Pack 1, Rollup 6. This indexing issue doesn't happen very often and I have to learn everything all over again. Putting all my notes here, gives me one place to refresh my view of history.

Before Rollup6 we would see these issues...
People would call and say "I click search and it just spins ... nothing is ever returned."

Do a "Get-MailboxDatabaseCopyStatus DB01" and it reports all is well.
Odd that the Get-MailboxDatabaseCopyStatus would say the Content Index is fine when it obviously isn't. You could failover to another database copy and have the same issue. Reports good, but search doesn't return anything.

(During this, we also found that some clients would search and return results, but when you click on one of the items, you'd get an error message: "Could not display item." This turned out to be a client issue and an update to a newer version cleared that up. But it sent us on a wild goose chase for a bit.)

To fix the "index-don't-work-but-reports-good" issue, we would log onto the server where the passive database copy lived, then:

1. Suspend the database copy
2. Stop both indexing services: "Microsoft Search (Exchange)" Service and "Microsoft Exchange Search Indexer" Service
3. Remove the catalog for that database
4. Start both Indexing services
5. Resume the database copy

(This link shows a better way to reset these indexes. Although it's essentially the same thing, it's done via a script already written for you.
http://blogs.msdn.com/b/pepeedu/archive/2010/12/14/exchange-2010-database-copy-on-server-has-content-index-catalog-files-in-the-following-state-failed.aspx)

This generated a "Crawl" of the database to rebuild the index from scratch. After the crawl is compete you can activate that copy of the database and then update the Content Index on the now passive database. using:

Update-MailboxDatabaseCopy DB01\MBX02 -CatalogOnly -Force

I still need to know when this was failing, and not by a customer calling and saying "I ain't getting nuthin."

Test-ExchangeSearch to the rescue -- at least this was something real. The test adds a message to the System mailbox and then checks to see if it is available through the Search Indexer. An actual honest to goodness test!

Get-Mailboxdatabase -Server MBX02 | Test-ExchangeSearch

This command gets us the results we want. They tell us if the Indexing is crap (Result = -1) or how long it takes to retrieve the results. Sometimes this will return a MAPI Error, but you run the test again and it returns a value just fine. Maybe by testing it you woke it up?

After Rollup6
We never found out why those indexes were getting corrupt and why only on that DAG. We have 7 other DAGs that don't have that issue. But after Rollup6, we never saw those indexing issues again.
No, not those issues. We found new ones. But not so terrible ones. At least the Get-MailboxDatabaseCopyStatus reports correctly now.

And in this lastest episode we started seeing a Content Indexing error on the active copy of the database. Update-DatabaseCopy -CatalogOnly doesn't work on the Active copy of a database. And since we are under strict orders to not change anything without Change Management documentation and waiting 14 days for approval, we could not move the database to fix this content index.

I tried to restart the Microsoft Search (Exchange) Service, which tries to restart the Microsoft Exchange Search Indexer Service and the Indexer service hung. I could restart the indexing duo on any other servers in the DAG. It was very clear this database and this server were in a deadlock battle with no relief in sight. In fact this proved it:

sl $exscripts
.\TroubleShoot-CI.ps1 -Database DB01

So I tried to restart the Microsoft Search (Exchange) Service again and the Microsoft Exchange Search Indexer hung while stopping again. I loaded up Task Manger and killed the Exchange Search process, which did immediately restart because the Process ID changed. It finally dawned on me that the Microsoft Search (Exchange) Service had never actually restarted, so I found that process and killed it.

Everything started to work again. So to clear a deadlock, at least in this case, was to kill the Microsoft Search (Exchange) Service process.



This Explains the Troubleshooters:
http://blogs.technet.com/b/exchange/archive/2011/01/18/3411844.aspx

Resetting the Content Indexer to force a new crawl:
http://blogs.msdn.com/b/pepeedu/archive/2010/12/14/exchange-2010-database-copy-on-server-has-content-index-catalog-files-in-the-following-state-failed.aspx

Enterprise Wide PST Import - Morning Status Report

This is Part 9 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in the series is here.


This is the report I get each morning. The information is derived from several sources.

First, we took a snapshot of our starting point in this project. That was done on Oct 9th, 2011. That's why all the calculation is done from that date.

We run a report every Sunday. It looks for all users, if they have a HomeDirectory defined, and do they have any PST files there? We log each PST file, so we know when there are new ones, if one is removed.

It's just a simple CSV file.

And every day, we get a list of all the people that have an Archive Mailbox and check each one again. Has that file been removed?, is the PST file in use?, etc. The user also gets a reminder message of PST files still attached to Outlook and instructions on how they should be detaching those PST files.  The user gets a message every 14 days from the day they were migrated.

We run another process manually on Wednesdays that looks to see how long it's been since a PST file was last accessed.  If it is older than 30 days, it tries to move the PST file to the user's local hard drive. Assuming there is space to hold it, of course.

New Hires are put in a GPO that "Disallows PST Growth."  Archive Mailbox users are put in that same GPO. We are calling these people "Controlled Users" because we can controll their PST file habit.

In the gap between New Hires and Archive Mailbox users, are the "Uncontrolled Users." Users not in the GPO and free to add messages and create new PST files.

The ultimate goal of this project is the removal of the PST files from the HomeDiectory.

What this report means:

· We have removed 4.3TB of PST data off Home shares

· Uncontrolled users (Not Online PST Users and Not New Hires) added 1TB of data to new PST files since Oct 2011

· Uncontrolled users have added, to their existing PST files, 2TB of data

· Leaving us a net removal of 1.3TB

· Uncontrolled Users added 55 PST files just last week.

· The 42% is: NumberOfPSTFilesRemoved / NumberOfPSTFilesOnOct92011

	Percent Complete		42 %
	People with Archive MBX		1,585
	All Users with PST files on HomeShares
		9-Oct-11	24-May-2012	Diff
	Number of PSTs	23,516	18,739	4,777
	Size on Shares	12,396,679	11,068,508	1,328,171
	Users with PSTs on H:	3,004	2,506	498
		Since Begining	Last Week
	New PSTs on Shares	2,293	55
	Size New PSTs	1,063,297	29,538
	Users w/Archive Mailbox: PST files on HomeShares
		Discovered	Processed	Removed
	Number of PST Files	14,248	8,810	9,916
	Size on Shares	6,992,846	3,495,739	4,346,864

Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions