Monday, December 3, 2012

Mailbox Move error:Couldn't switch the mailbox into Sync Source mode

I've been working on a project to import PST files for quite some time. Some users we've helped have had many PST files and some had very large files.

One of our Archive Mailbox databases became very large for our tastes, so we decided it was time to trim all these databases down to 100G each from 380G each. Our main issue with the size was a place to restore. The time it takes to restore the database wasn't a problem, these were archived messages and the SLA was a full day. The problem was the 400G chunk of free space needed to do the restore. That was hard to find.

So as we moved archive mailboxes, I came across this error from time to time. The first time I saw this error, we were dealing with a user's mailbox that had over 22,000 folders. So I just exported her archive mailbox into several pieces and then imported it into another database. I just assumed that was a one time issue, and just moved on to my next task.

Then it happened again. This time is was on an archive mailbox 35G in size. I started searching on the error and found very little. Here's the error:

==============================================================
FailureCode                      : -2146233088
FailureType                      : SourceMailboxAlreadyBeingMovedTransientException
FailureSide                      : Source
Message                          : Error: Couldn't switch the mailbox into Sync Source mode.
                                   This could be because of one of the following reasons:
                                     Another administrator is currently moving the mailbox.
                                     The mailbox is locked.
                                     The Microsoft Exchange Mailbox Replication service (MRS) doesn't have the correct
                                   permissions.
                                     Network errors are preventing MRS from cleanly closing its session with the Mailbo
                                   x server. If this is the case, MRS may continue to encounter this error for up to 2
                                   hours - this duration is controlled by the TCP KeepAlive settings on the Mailbox ser
                                   ver.
                                   Wait for the mailbox to be released before attempting to move this mailbox again.
FailureTimestamp                 : 9/15/2012 6:24:00 PM
FailureContext                   : --------
                                   Operation: IMailbox.SetInTransitStatus
                                   OperationSide: Source
                                   Archive ()
                                   Status: SyncSource
==============================================================

I finally called Microsoft Support who said "This is a very common issue. Here's what you need to do... Change the TCP KeepAliveTime for the servers involved in the move. Source, Target, and CAS."
It turned out that I had to do that so much, I wrote a simple script to make the changes for me easily.

Get the code here: http://poshcode.org/3808


Enterprise Wide PST Import - The Functions

This is a Part 12 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
(This is the last post in this series.)


The PST Utility Module is really just a collection of functions. Here's a list of the functions and short description of how they are used.

Get the module here.

Queue Processing:

These queue processing functions are used to do the work of Importing PST into Archive Mailboxes as well as maintenance of the queues.

Add-PSTIMportQueue
Given a user name, find their PST files, set up the job queues for the PST files, and notify the user and admin with an initial report. A more detailed explanation is here.

Process-PSTImportQueue

Loop through each job and take appropriate action until the job goes through all the stages of: Copied,  Imported, Cleaned up, and Notified. A more detailed explanation is here.

Set-PSTImportQueue

Set different attributes on a user’s jobs, or a single job. A more detailed explanation is here.

Remove-PSTImportQueue
Remove a user or a job from the queue.

Report-PSTDiscovery
Finds all PST files for All Mailbox Users and tracks their usage – weekly run

Report-PSTOverallSummary
Generates a daily detailed report in an HTML file – also sends a summary report to the Admin and Boss.


Admin Functions:

These functions are meant for Admins to run. They can help the Admin decide how to process a user's PST files.


Check-ForPST
Same as the Get-PSTFileList above except input is geared to an Admin and not a Job Object


Get-ArchiveUserInfo
A quick report about a user, do they have an Archive mailbox already, GPO, etc

Get-ImportStatus
A quick way to check on import queues and suspend and restart, etc

Move-PSTs
This checks for and moves a users PST file from the Home Share to their local PC

New-OnlineArchiveByUser
This adds the Archive Mailbox to a user, setting all the defaults. For people without PST files to import

Optimize-PSTImportQueue
Sort Users by number of PST files, so the most number of users get done quickly


Helper Functions

Add-ToGPO
Add a user to a group we use to control Disallow PST Growth

Adjust-Quota
Sets quota to default for Online PST users

CC
Counts collections 0, 1, many

ConvertTo-ComputerName
Returns the computername for an IP

ConvertTo-IP
Returns the IP for a given Comptername

Clean-OnlinePSTGPO
In our world, there is a distinct set of rules for the GPO, we check them here. Add people who are missing and remove people who should not be there.

Copy-PSTusingBITS
Uses BITS to copy the PST file to the share for processing. From PC’s and far away sites

Copy-PSTtoImportShare
Uses copy-item to copy the PST file to the share for processing. Used for local AD site files

Format-JobStatus
Returns the job status based on user or overall

Get-ISODayOfWeek
Returns the day of the week for a given date

Get-FreeDiskSpace
Returns the free space of a given computer drive – used when moving PST files back to a local computer.

Get-ClientAccessUserLog
Returns raw info on a user extracted from RPC logs on CAS servers

Get-MRSServer
Returns the MRS Server to use for this move

Get-OutlookEXEVersion
Returns the version of Outlook installed on a computer by looking at the EXE

Get-SingleMailboxObject
Returns a mailbox object, if results are anything other than a single entry, it returns nothing

Get-PSTFileList
Return the PST file collection from Home Shares or PC or some directory

Get-ArchDB
Returns the best candidate mailbox database for the Import. By smallest mailbox database size.

isMemberof
Tests the membership of a group

Import-PSTFromShare
Starts the import process using the Job Object for settings

Lock-PSTIQ
Creates a zero length file to signal processing is happening

Move-PSTToLocal
This moves a PST file from a Home Share to the local PC – if all test conditions are true

New-ArchPSTFileLogObject
Creates a new object for logging the PST file activity of Users with Archive Mailboxes

New-ArchPSTIndex
Finds or creates a new Index entry for a PST file. Used to search the PST related Filelog Objects

New-PSTFileLogObject
Creates a new object for Logging all PST files for all users

New-PSTFileIndex
Finds or creates a new index entry for a PST file object for All Users

New-PSTJobObject
Creates the Job object used in PST import Jobs

New-PSTReportObject
Creates object used in PST Reporting

New-OnlineArchiveByJob
Gives the user an Archive Mailbox, using the Job Object passed. Only used in Add-PstImportQueue

New-TargetPSTDirectory
Creates a new directory in the share using the User’s log in name

Reset-PSTBackupFileNames
Resets all Backup filenames to Now() using the Format ‘<filename>-yyyyMMddHHmmss.<ext.>’

Send-NotificationInitialReport
When add-pst is run, and discovers PST files to process, a summary is sent to the user.

Send-NotificationFinalReport
When processing is finished, the results are sent to the user.

Send-NotificationPSTRemoval
When reports are run, a message is sent to the user about still connected PST files (14 day cycle)

Test-OutlookClientVersion
A way to control what client version are accepted for import

Test-PSTIQLock
Test if the Queue is locked, returns true/false

Unlock-PSTIQ
Removes the lock – when the processing pass is done










Enterprise Wide PST Import - Get the script / Set up

This is a Part 11 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in the series is here.

I put these scripts together over time and having to go back and look at all the moving parts makes me realize how much time went into this.

Get the module here.

So follow these few simple steps and you're ready to import PST files!


First you'll need the permissions:

Granting the permission to do PST file Import and Exports is explained here. Just assign yourself the role.

As for finding and copying PST files, well you might need a little more umph. I have an account that has Domain Admin privilages. That account is for Admin duties only. (I have separate 'normal' account I use for everything else.) My admin account also gives me access to all the computers where PST files may be hiding, since Domain Admins is in the Local Administrators group on each computer.

You could use a service account which has all these privilages and run these functions using that account. I tend to not like using service accounts because you lose accountability.

Create the Share:

Exchange PST file import and export cmdlets require a share. Find a server that can handle the amount of PST files you plan to import and create a share. Grant the "Exchange Trusted SubSystem" group full access to this share. We named our share "PSTImports."  It's why all the folder structuture begins there.

We were lucky enough to have a utility server lying around that has 1TB of space on one of the drives. We use the server for reporting and performance history. And there was plenty of free space to handle importing 800G of PST files in one pass. We never got that high, and we even added pieces into the script that examines the free space of our server to choke PST copies in case we ran short of space. You should adjust this setting for your server. It is $Script:FreeSpaceLimit and is set in MB.

You'll want to create some other directories within this share:
  • ^Completed
  • ZZSavedQueueLogs
Other directories are created during the import for each user being processed to hold their PST files. After the Import, the PST files are replaced with results of the Import in CSV, then moved to the ^Completed directory for historical reasons. It's named with the leading ^ for sorting purposes. It will always be at the top ;)

ZZSavedQueueLogs is just a backup queue files. A backup is made each time you run the functions. I can't tell you how many times having those backed up has saved me some extra work.
I check this directory -- when I think about it -- and kill any older stuff.

Add PSTUtils As A Module and Optionally add to profiles:

Log on the server with the account that has Local Administrator rights and create this directory:
C:\Windows\System32\WindowsPowerShell\v1.0\Modules\PSTUtility
And copy the PSTUtility module there.

When you load up powershell then "get-module -listavailable", you should see the PSTUtility as a module you can import. Just use the command "import-module PSTUtility"

Add this to your $profile so you have it available each time you log in.
I use this command all the time: "notepad $profile" make my changes and then save. Restart powershell to load the new profile changes and you are ready to go.

Edit the Module: look over and change the $Script: level variables to suit your particular environment.

Configure Initial IP/Computer searches: (Optional)

When we get a request for a user's PST files to be imported, we usually just get a name. I'd love to have the computer name so I can determine if the Outlook Version on that computer will indeed support the Archive Mailbox feature. It's not always supplied. So we ask. "I don't know. He took his latop with him and he is gone for a few days." Let's just say, it's not always easy to get. Even if you ak the users and tell them how to find it, its a hit and miss proposition.

One can find a user's log in information in the RPC logs of your CAS servers. But theres a lot of information in those logs ... lots!  So I wrote a small utility that grabs what I call 'Connect Logs' - this is an subset of the RCP logs where OUTLOOK.EXE does a status of "Connect."  This entry holds the Version of Outlook connecting, legacyExchangeDN, the Mode (Cached or Classic), and the IP address. Looking thru this subset of logs vs the full logs is a time saver when looking up 50 people and trying to find the information on them.

Of course, users don't always 'connect' every day. They leave Outlook running for days and days. So their connect log may be there but it was last Thursday. I tried various combinations of how to search for this data and how far back to look etc.

I finaly just decided to get 30 days worth of these connect entries and use those for all my reporting and PST file imports. I have a script that runs every night at 2AM and gathers this subset of logs. It's "Get-CASConnectLogEntries" And you can copy that piece out, or add the 'import-module PSTUtils" to the profile of whatever is running the script. I have it running seperate in a batch file I have for overnight reports.

The directory location I keep these is in a different place where I keep all my reporting information. You can of course modify the variable and keep it where you like. I use these connect files for other reports too. There's lots of information in those logs.

I should add, running Get-CASConnectLogEntries at 2 AM is optional. If you know the computername each time you do Add-PSTImportQueue, you can bypass setting all this up. When you know the computer name, you can query the computer for all the information you need.

That's all there is to setting up everything.

Once you have these pieces in place, the hard part starts. Finding people to migrate to the Archive Mailbox.

Next, I'll be posting a list of all the Functions and how I use them.



Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions