Enterprise Wide PST Import - Morning Status Report

This is Part 9 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in the series is here.


This is the report I get each morning. The information is derived from several sources.

First, we took a snapshot of our starting point in this project. That was done on Oct 9th, 2011. That's why all the calculation is done from that date.

We run a report every Sunday. It looks for all users, if they have a HomeDirectory defined, and do they have any PST files there? We log each PST file, so we know when there are new ones, if one is removed.

It's just a simple CSV file.

And every day, we get a list of all the people that have an Archive Mailbox and check each one again. Has that file been removed?, is the PST file in use?, etc. The user also gets a reminder message of PST files still attached to Outlook and instructions on how they should be detaching those PST files.  The user gets a message every 14 days from the day they were migrated.

We run another process manually on Wednesdays that looks to see how long it's been since a PST file was last accessed.  If it is older than 30 days, it tries to move the PST file to the user's local hard drive. Assuming there is space to hold it, of course.

New Hires are put in a GPO that "Disallows PST Growth."  Archive Mailbox users are put in that same GPO. We are calling these people "Controlled Users" because we can controll their PST file habit.

In the gap between New Hires and Archive Mailbox users, are the "Uncontrolled Users." Users not in the GPO and free to add messages and create new PST files.

The ultimate goal of this project is the removal of the PST files from the HomeDiectory.

What this report means:

· We have removed 4.3TB of PST data off Home shares

· Uncontrolled users (Not Online PST Users and Not New Hires) added 1TB of data to new PST files since Oct 2011

· Uncontrolled users have added, to their existing PST files, 2TB of data

· Leaving us a net removal of 1.3TB

· Uncontrolled Users added 55 PST files just last week.

· The 42% is: NumberOfPSTFilesRemoved / NumberOfPSTFilesOnOct92011

	Percent Complete		42 %
	People with Archive MBX		1,585
	All Users with PST files on HomeShares
		9-Oct-11	24-May-2012	Diff
	Number of PSTs	23,516	18,739	4,777
	Size on Shares	12,396,679	11,068,508	1,328,171
	Users with PSTs on H:	3,004	2,506	498
		Since Begining	Last Week
	New PSTs on Shares	2,293	55
	Size New PSTs	1,063,297	29,538
	Users w/Archive Mailbox: PST files on HomeShares
		Discovered	Processed	Removed
	Number of PST Files	14,248	8,810	9,916
	Size on Shares	6,992,846	3,495,739	4,346,864

Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions

Enterprise Wide PST Import - Using RoboCopy

This is Part 8 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in this series is here.

Our users are world-wide and even though we are concentrating on our Headquarters, many other users are seeing the benefit of importing their PST files. Especially those who travel a lot. Nearly everyone of these heavy travelers carry an external hard drive with their PST files. Now they can get those messages via OWA and they can discard the extra weight of the external hard drive.

We started getting many request for Archive mailbox from all over the world. The challenge is to copy the PST file to our staging area so it can be processed by the server.

Alas, Copy-Item seems to be really slow when working over a WAN line. And many of our pipes can get saturated. I started experimenting with Robocopy, and I am not entirely happy with the results, but it's better than copy-item.

I created a small stand alone script (RoboCopy-Item) that does the very simple  copy of a file. I am experimenting with the settings like /IPG:300 /Z, etc -- trying to find the best overall throughput for our environment. Fast, but not choking the WAN. I am still experimenting there.

In the Add-PSTImportQueue function we now look to see if a user is in HQ or outside somewhere. If they are outside HQ then we mark the job with a status of RoboCopy.

Then we use: Robocopy-PSTImportQueue which looks at the import queue and starts background jobs for each job with a RoboCopy status. I do this on a by user, by location selection. We don't want to have 50 jobs running for 30 users in 1 location. I keep it at 1 user per 1 location at any given time.

Then we use a Get-RoboCopyjob <jobnumber> that just gets the 1st 10 lines and that last 20 lines of a job, so you can quckly see the status of the job.

We toyed with the idea of incorporating this into the overall script so we could just run it, but too many things can go wrong and the powershell window with all the jobs running can get closed ot the server rebooted, etc.

We want to find a better way to do this.

Still bigger files -- over 1.5G -- are taking forever. We are using BITS to control the amount of traffic that can be allowed, so we have to go modify the Registry and restart BITS service. But copies from computers with that setting fills up the pipe to that location.

I'll update as we search for the better way.



Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions

More PST Import Utils - Get-ImportStatus & Lock-File

This is Part 7 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in this series is here.

Quicker Stats
We found the MailboxImport queue needs a little tender care from time to time. We always would run this command to get a quick understanding of what was going on in the queue:

Get-MailboxImportRequest | Get-MailboxImportRequestStatistics

Sometimes we needed :

Get-MailboxImportRequest -Status Failed | Resume-MailboxImportRequest

I really got tried of typing all that out all the time, so I created a short function for me.

Function Get-ImportStatus (){
      #---------------------------------------------------------
      # a helper function to display mailbox import info
      # option to show only a subset -- by batchname
      #      sometimes the list is just long
      # option to restart failed jobs --
      #      sometimes jobs failed because the service crashed on a bad PST file
      #      or too many jobs for one mailbox
      # option to restart suspended jobs
      #      the script can suspend jobs it thinks may be causing issues
      # option to suspend all jobs
      #      a single PST can crash the MB rep service and all jobs start over
      #      you can't tell exactly which is the culprit so suspend all jobs
      #      and investigate

      # left the -confirm off on purpose
     
      param (
             $Batch=$null,
              [switch]$RestartFailed,
              [switch]$RestartSuspended,
              [switch]$SuspendAll
       )
      If($Batch) {
              Get-MailboxImportRequest -BatchName $Batch |
              Get-MailboxImportRequestStatistics
       }
      ElseIf($RestartFailed.IsPresent) {
              Get-MailboxImportRequest -Status Failed |
              Resume-MailboxImportRequest
       }
      ElseIf($RestartSuspended.IsPresent) {
              Get-MailboxImportRequest -Status Suspended |
              Resume-MailboxImportRequest
       }
      ElseIf($SuspendAll.IsPresent) {
              Get-MailboxImportRequest |
              Suspend-MailboxImportRequest
       }
      Else {
              Get-MailboxImportRequest | Sort Name |
              Get-MailboxImportRequestStatistics
       }
     
}

Stepping all over each other
You now how it is, everyone gets busy and stops checking with others about what's going on and crap happens. The way we were handling the queue files became an issue. If two people ran the script, the last one to write was the winner. I noticed this when I tried to schedule a task and the queue was trashed. (Lots of manual fixing up there.) And there were a few other small disasters, too.

I remembered and old way to make sure one process did not step on the other, create a "lock" file when you started your work and then delete the "lock" file when you done.
This wasn't exactly elegant, but it works just fine. We're just doing this with a zero lenth file.

Simply check for the existance of the file (Test-PSTIQLock) and if false, lock the file (Lock-PSTIQ) process the queue and then remove the lock (Unlock-PSTIQ)



Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions