Enterprise Wide PST Import -- Set-PSTImportQueue

This is Part 5 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in the series is here.

As we progressed with importing PST files and processing more and more users, we found we were having to Repair PST files or mark them to be skipped manually. Some users were saying - please wait, don't process me yet, wait for 2 days.
This was becoming a pain more than anything, so I sat down and decided to do something about it.

Right about this same time, I had decided we had too many little scripts scattered about everywhere and I wanted them all in one place easily accessible. And all of these scripts shared some functions. So I created a Module and started to migrate all the scripts there. Well it's not a real Module, but more like a repository for all the functions and scripts we used with the PST migrations.
Then I added two new functions: Set-PSTImportQueue and Remove-PSTImportQueue
(I've only used remove once, when a user decided to not have their PST imported.)

Set-PSTImportQueue is just a way for me to change settings on a Import Job without loading up Excel and making mistakes. Here are the options:

• -DisplayName -- The person we are working on. This allows you to work on all the jobs associated with this name. You use it with the other options to change settings, like -JobStatus, etc.
• -JobName -- Isolate this update to a particular job
• -JobStatus -- Change the JobStatus -- reset back to New, etc. Sometimes it useful to change this status to something the script doesn't recognize, just to skip this job, or set of jobs.
• -IP -- This is the computer Name, sometimes you may not have known it during the add, this is just a way to add the computername to the jobs. We use this entry later when moving PST files to the local PC
• -OrgUNCName  -- There are cases where the user moved the file, and rather than doing a new discovery, just change the location.
• -MRServer  -- In our 2 AD site world, having the wrong MR server setting can make the jobs just sit in the queue. We have dedicated CAS servers for this process, one in each site. If the Archive database is in Site 1 and an MRServer in site 2 get chosen by mistake, or the database moved, you need to reset the job status and change this to a MRServer in Site 1.
• -SkipReason -- A Place to log why a PST file was skipped, "Age, Size, Backup, Sharepoint List, Corrupt, Missing, etc" This shows up on the Final Report.
• -ClientVer -- A place to log the client version, mainly for records and reports.
• -ClientVerOK -- The is true or false. By default Process-PSTImportQueue will not process jobs that have ClinetVerOK set to false. It just skips them. Sometimes setting this to false on all jobs for a user allows you to skip this user for now.
• -ProcessFileOff -- As it sounds, changes the ProcessJob to $false -- skipped jobs are set to false. You might want to set the SkipReason at the same time.
• -ProcessFileOn -- As it sounds, changes the ProcessJob to $true
• -CompleteQueueFile -- The PSTCompleteQueue file needs maintenance from time to time, so this setting allows you to work on that file. The default is the PSTImportQueue

Remove-PSTImportQueue just takes two options:

• -DisplayName -- This will remove all jobs with this users name.
• -JobName -- This will remove this particular Job

Introduction: The Beginings

Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions

Enterprise Wide PST Import -- Some Tools we added

This is Part 4 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in the series is here.

Tools To Help Do the Job 

When this project started we were mandated to import over 23,000 PST files into mailboxes and get the PST files off the home shares. The important part here is "get the space cleared up off the home shares."

So each week we are given a list of names of people moved to Windows 7 OS and Office 2010. The list can have 20 names, or 120 names. So we process them. In many cases, there can be upwards to 50 PST files per person. One user had over 500. (I know! I had to double check.)

So the queue can be fat with over 1500 items at times, we found that sometimes the processing of the queue was very slow, mostly due to one person having a lot of files and most of them big.

Optimized For Speed

We needed to optimize the queue. We talked about just sorting buy the size of the file, and then decided that would make some people with only one file wait too long to be finished. We wanted to finish as many people as possible in the night. So all the people with just one PST file needed to be moved to the front of the line.

Pipe the queue to group by Displayname, sort by count, then recreate the queue from the smallest to the largest. Quick and easy. This usually gets 80 - 90 % of the users done over night. We start about 7PM.

Move the PST files off.

We really struggled to get a good handle on this. People are pretty freaky about their PST files. In our final report, we ask them to disconnect the PST from their client and then move them locally. But a large majority can't do that without help, or don't care. Many never read the message at all.

We had to figure out how to get the PST files off the Home shares and do this without freaking out the user.

Finally we agreed on a place to put the files. In the users "my documents" directory on their C drive. It's not the most perfect place, but it is somewhat secure. At least secure enough from the common user. We had to know the PC name, and what OS it was, just to find the correct place quickly. We also needed to know that the user is in our HQ building. a small percentage of users with Archive mailboxes are outside our HQ office. We wanted to skip those.

We agreed the PST files needed to be disconnected for 30 days. This was long enough, everyone thought, for the user to forget they had them. If a PST file has a LastWrite Timestamp of "right now" -- that PST is most likely open and connected to Outlook.

So:
If you have PST files on your home share,
And you are in the HQ office,
And those PST last write timestamp is 30 days old.
We move the file for you to ..\Documents\Outlook Files\
If there is a file there with that name already, we just add a random number to the name and keep going.

Grading Our Progress

Since moving the PSTs off the Home shares is the real ultimate goal, we started to keep track of all the PST files, creation dates and removal dates.

As of this date, we had over 23,000 files to move, and we moved about 6,000. Not too bad. Not fantastic, but good.  I'll post some info on our reports later on.

But there is more work to do...

I am still constantly opening the PST Import Queue in a spreadsheet and modifying it and saving it again. I am human and make many mistakes. I must figure out how to do that easier...
Next time: "Set-PSTImportQueue"


Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions