In my last post I talked about having to eat humble pie by going to the Powers-That-Be and admit to a mistake. A two week default retention tag was applied to a group of mailboxes and lots of mail was deleted.
As it turned out we are having to restore all the databases containing affected mailboxes. Many of those databases held secondary (archive) mailboxes. 94 mailbox databases in all I believe...
When we were restoring the primary mailboxes the command Restore-Mailbox worked flawlessly. It was speedy. It was accurate. Lovely.
Then we got down to recovering the data in the Archive mailbox, and things started to get sketchy.
Restore-Mailbox command does not understand archive mailboxes. Before I did this I called Microsoft Support with a simple question.
"I don't see a -Archive option for Restore-Mailbox, so just how does it does work with archive mailboxes?"
"Ummm, I see what you mean. Very possible it just knows the mailboxes in the databases are archive mailbox and will handle accordingly."
"So, very possible, or you're certain?"
"Yes, I'm certain."
Well he might have been certain, but when 9PM rolled around and he was watching his T.V., I was finding out the hard way he had no clue what he was talking about.
But fortunately for me, this guy did :
http://eightwone.com/2011/01/07/restoring-personal-archives/
So -- for instance -- I have a mailbox database, called Arch01. In the mailbox database there are only Archive Mailboxes.
So to get a reference to all the mailboxes that have archive mailboxes in that database:
(We only have 1600 users with archive databases.)
$ArchMBX = Get-mailbox -archive -Resultsize 2000 |
?{$_.ArchiveDataBase -eq 'Arch01'}
Then use the New-MailboxRestoreRequest command:
$ArchMBX | %{
New-MailboxRestoreRequest -SourceDatabase Arch01Recovery `
-SourceStoreMailbox $_.ArchiveGUID -TargetMailbox $_.Identity `
-TargetisArchive -BatchName Arch01 -Suspend `
}
I am doing this on 16 very large databases and 16 small ones. The New-MailboxRestoreRequest command can be very slow. Not sure if the amount of data in some of these mailboxes (many are over 8G and a few are upwards into the 20G range) or if there is something else going on here. Some mailboxes were taking up to 18 hours to complete.
Seemed to me the Mailbox Restore Request was bogging down and getting slower and slower and moving less and less data. I did have about 450 requests queued up, but I've had more than that in some mail migrations. I experimented with restarting the Mailbox Replication Service on all the CAS servers in this AD Site. For me, every 4 hours seemed to be the magic time frame. I would just suspend all the jobs, then restart that service on all the CAS servers and resume the jobs.
Another odd thing about the New-MailboxRestore command is it was crashing store.exe on some occasions, like corruption in a mailbox. Initially, when a New-MailboxRestoreRequest failed, I'd set the -BadItem limit to something very high and resume it. Now I just make a note of that mailbox and move on. In my mind there is only a 25% chance they were hit anyway. Why sweat it? If they need to be restore, we would restore the mailbox database again, but from a different time frame.
Finally, I found that throwing a lot of recoveries at a mailbox database seems to over tax the log file space as well as indexing. We use circular logging but the data was being pumped in too fast. As item were being recovered, there were added to the indexing queue and filling it up, too.
So now I start all my new request with -Suspended to be on the safe side. I also choked the Mailbox Replication Service to only allow 5 mailbox restores max per target database. This seemed to help on those log file directories and eased the indexing some.
Monday, July 16, 2012
Monday, July 2, 2012
Retention Tags Fiasco
An Innocent Beginning
A user requested to have a Personal 'Delete' Retention Tag for 14 days. I created one and added it to the policy that is applied to about 1600 users. A personal folders tag is meant for folders created by the user and can be applied to any folder. We also have a "Default -All other folders 14 Day Delete" which deletes anything over 14 days old in any normal folder without a policy applied. Essentially, the default policy.
I hate to admit this, but I did a pretty dumb thing. I chose the wrong one and applied it to the Policy for the 1600+ users.
So the person who requested this new policy calls me and says, "Hey, this was supposed to just delete stuff in my single folder but it's deleting everything older than 14 days!" Of course I think he is the typical crazy user. Then I look at my mail and notice that some old stuff I need is gone. I looked at the policy applied to me and lo and behold, there it was, big a life. "14 day delete (2 weeks)" stamped all over my mailbox folders. This is where I panicked.
I went to all 32 mailbox servers and stopped the service for the "Exchange Mailbox Assistants." I prayed that this did not get too many people... (Yes, I know I can run a command to do this remotely, but our servers are not set up for that as yet. I just never got around to it. I'll get to that as soon as I finish ...)
Then I had to humbly take myself to the boss and say, I screwed the pooch. Not only did I not have a Change Management ticket, but I made the change during the day. Two major infractions. I guess I was thinking this was only a minor maintenance thing, or I was not capable of screwing up. Probably a lot of both. So I had added this policy in, a very simple thing for sure, but it caused a very big problem.
The Powers That Be wanted to know:
Who did this effect?
We bumped the logging level up to expert so when we started the Mailbox Assistants back up again we might find some indicators as to which mailboxes had been touched. Then we started looking through the event logs to see if there was any indication of mailbox identities already there...
Event 9001 - the service is starting on a database
Event 9017 - Mailbox Assistant starting work, there are 600 mailboxes in here, etc -- but no information of which ones we're working on.
Event 9018 - there was a failure for a mailbox. Not saying which or why it failed. Just a count.
9021 special request started
9022 special request finished
9025 a mailbox was skipped - here is the GUID - seems many of these were disconnected mailboxes
9037 there was an error on something
9112 - list of stale mailboxes with decayed watermarks - not sure what this is yet
Bottom line: There is no way to tell who this affected. You have to look at the mailboxes themselves. But there were 1600 of them. We can't open each one. So we had to bite the bullet and send a message to all the 1600 users saying "uh, you may not have been hit ... but, uh, look at a message. If you see <this> you might be one of the people we want to help. Oh, and you can also tell because you may or may not have lost all you mail older than 14 days." -- Boy, that's gonna hurt.
How Do We Get Their Mail Back?
Restoring from backup is the only way. Unless you can live with the solution of recovering deleted items into a newly created folder. We gave that to users as a temporary solution, but did not expect them to want to refile all that data. So, again.. restore from backup. To top it all off we don't have that much room left on any servers.
These people are scattered all over the place with primary mailboxes in one database and secondary mailboxes in others. There are 72 database involved. 16 are housing secondary mailboxes with a size of 325G or more. There is hardly a place anywhere to restore those databases. Luckily there were two servers with enough space to handle two of these databases each. The other 56 databases we got over two nights, putting one database on each server, where we could. (Did you remember that you can only have one Recovery Database mounted on a server at one time? I had to learn that one again!)
How Do We Erase The Tag?
I was afraid this was going to be the hardest part, and it turned out to be the easiest.
"If thy policy offends thee, pluck it out!"
So we set the Retention Policy on all mailboxes to $Null
Then we removed the offending Policy Tag from the Retention Policy
Then we deleted the Policy Tag
Then we deleted the Retention Policy
Finally, Restart the Mailbox Assistant Service
Since there is no Policy Tag to reference, it removes the Tag from the message. One could have just as easily applied a new policy and it would have overwritten any existing Tags. We never tested that option though. We removed all the Policy Tags and started the Mailbox assistant service on the server with my mailbox. All the scary tags disappeared.
Fixing The Wrong:
First I wanted all the mailboxes in the database that had a secondary Archive mailbox. I sorted them because I wanted some kind of easy indicator of how far along the process had gotten. The alphabet was good enough for me.
# find only the people with Archive Mailboxes
$MBX = Get-Mailboxdatabase DB01 | get-mailbox -archive | Sort DisplayName
Next it was simply piping the results to the restore command:
$MBX | Restore-Mailbox -RecoveryDatabase 'DB01Recovery' -MaxThreads 25
Lesson Learned
Don't be too lazy: I even thought about creating a test Retention Policy and applying it to the requestor but I never did. I remember talking myself out of it -- "It a person tag, just do it and move on to the next task on that long list of things to do today."
You're never too busy to do something right the first time: I am reminded of this yet again. A few years from now, I'll get in a hurry again and Murphy's Law will hit me over the head. It can actualy save you time to take the time to do it right. This screwing the pooch cost me many hours. Let's not forget the backup team, who had to jump through hoops to get me all these restores.
A user requested to have a Personal 'Delete' Retention Tag for 14 days. I created one and added it to the policy that is applied to about 1600 users. A personal folders tag is meant for folders created by the user and can be applied to any folder. We also have a "Default -All other folders 14 Day Delete" which deletes anything over 14 days old in any normal folder without a policy applied. Essentially, the default policy.
I hate to admit this, but I did a pretty dumb thing. I chose the wrong one and applied it to the Policy for the 1600+ users.
So the person who requested this new policy calls me and says, "Hey, this was supposed to just delete stuff in my single folder but it's deleting everything older than 14 days!" Of course I think he is the typical crazy user. Then I look at my mail and notice that some old stuff I need is gone. I looked at the policy applied to me and lo and behold, there it was, big a life. "14 day delete (2 weeks)" stamped all over my mailbox folders. This is where I panicked.
I went to all 32 mailbox servers and stopped the service for the "Exchange Mailbox Assistants." I prayed that this did not get too many people... (Yes, I know I can run a command to do this remotely, but our servers are not set up for that as yet. I just never got around to it. I'll get to that as soon as I finish ...)
Then I had to humbly take myself to the boss and say, I screwed the pooch. Not only did I not have a Change Management ticket, but I made the change during the day. Two major infractions. I guess I was thinking this was only a minor maintenance thing, or I was not capable of screwing up. Probably a lot of both. So I had added this policy in, a very simple thing for sure, but it caused a very big problem.
The Powers That Be wanted to know:
- Who did this effect?
- How can we get their mail back?
- How do we erase the Tag?
Who did this effect?
We bumped the logging level up to expert so when we started the Mailbox Assistants back up again we might find some indicators as to which mailboxes had been touched. Then we started looking through the event logs to see if there was any indication of mailbox identities already there...
Event 9001 - the service is starting on a database
Event 9017 - Mailbox Assistant starting work, there are 600 mailboxes in here, etc -- but no information of which ones we're working on.
Event 9018 - there was a failure for a mailbox. Not saying which or why it failed. Just a count.
9021 special request started
9022 special request finished
9025 a mailbox was skipped - here is the GUID - seems many of these were disconnected mailboxes
9037 there was an error on something
9112 - list of stale mailboxes with decayed watermarks - not sure what this is yet
Bottom line: There is no way to tell who this affected. You have to look at the mailboxes themselves. But there were 1600 of them. We can't open each one. So we had to bite the bullet and send a message to all the 1600 users saying "uh, you may not have been hit ... but, uh, look at a message. If you see <this> you might be one of the people we want to help. Oh, and you can also tell because you may or may not have lost all you mail older than 14 days." -- Boy, that's gonna hurt.
How Do We Get Their Mail Back?
Restoring from backup is the only way. Unless you can live with the solution of recovering deleted items into a newly created folder. We gave that to users as a temporary solution, but did not expect them to want to refile all that data. So, again.. restore from backup. To top it all off we don't have that much room left on any servers.
These people are scattered all over the place with primary mailboxes in one database and secondary mailboxes in others. There are 72 database involved. 16 are housing secondary mailboxes with a size of 325G or more. There is hardly a place anywhere to restore those databases. Luckily there were two servers with enough space to handle two of these databases each. The other 56 databases we got over two nights, putting one database on each server, where we could. (Did you remember that you can only have one Recovery Database mounted on a server at one time? I had to learn that one again!)
How Do We Erase The Tag?
I was afraid this was going to be the hardest part, and it turned out to be the easiest.
"If thy policy offends thee, pluck it out!"
So we set the Retention Policy on all mailboxes to $Null
Then we removed the offending Policy Tag from the Retention Policy
Then we deleted the Policy Tag
Then we deleted the Retention Policy
Finally, Restart the Mailbox Assistant Service
Since there is no Policy Tag to reference, it removes the Tag from the message. One could have just as easily applied a new policy and it would have overwritten any existing Tags. We never tested that option though. We removed all the Policy Tags and started the Mailbox assistant service on the server with my mailbox. All the scary tags disappeared.
Fixing The Wrong:
First I wanted all the mailboxes in the database that had a secondary Archive mailbox. I sorted them because I wanted some kind of easy indicator of how far along the process had gotten. The alphabet was good enough for me.
# find only the people with Archive Mailboxes
$MBX = Get-Mailboxdatabase DB01 | get-mailbox -archive | Sort DisplayName
Next it was simply piping the results to the restore command:
$MBX | Restore-Mailbox -RecoveryDatabase 'DB01Recovery' -MaxThreads 25
Lesson Learned
Don't be too lazy: I even thought about creating a test Retention Policy and applying it to the requestor but I never did. I remember talking myself out of it -- "It a person tag, just do it and move on to the next task on that long list of things to do today."
You're never too busy to do something right the first time: I am reminded of this yet again. A few years from now, I'll get in a hurry again and Murphy's Law will hit me over the head. It can actualy save you time to take the time to do it right. This screwing the pooch cost me many hours. Let's not forget the backup team, who had to jump through hoops to get me all these restores.
Monday, June 4, 2012
Ongoing Indexing Issues on DAG02
See the update to this problem here.
We've been having issues with indexing on one DAG and we thought we had it fixed when we went to Service Pack 1, Rollup 6. This indexing issue doesn't happen very often and I have to learn everything all over again. Putting all my notes here, gives me one place to refresh my view of history.
Before Rollup6 we would see these issues...
People would call and say "I click search and it just spins ... nothing is ever returned."
Do a "Get-MailboxDatabaseCopyStatus DB01" and it reports all is well.
Odd that the Get-MailboxDatabaseCopyStatus would say the Content Index is fine when it obviously isn't. You could failover to another database copy and have the same issue. Reports good, but search doesn't return anything.
(During this, we also found that some clients would search and return results, but when you click on one of the items, you'd get an error message: "Could not display item." This turned out to be a client issue and an update to a newer version cleared that up. But it sent us on a wild goose chase for a bit.)
To fix the "index-don't-work-but-reports-good" issue, we would log onto the server where the passive database copy lived, then:
http://blogs.msdn.com/b/pepeedu/archive/2010/12/14/exchange-2010-database-copy-on-server-has-content-index-catalog-files-in-the-following-state-failed.aspx)
This generated a "Crawl" of the database to rebuild the index from scratch. After the crawl is compete you can activate that copy of the database and then update the Content Index on the now passive database. using:
Update-MailboxDatabaseCopy DB01\MBX02 -CatalogOnly -Force
I still need to know when this was failing, and not by a customer calling and saying "I ain't getting nuthin."
Test-ExchangeSearch to the rescue -- at least this was something real. The test adds a message to the System mailbox and then checks to see if it is available through the Search Indexer. An actual honest to goodness test!
Get-Mailboxdatabase -Server MBX02 | Test-ExchangeSearch
This command gets us the results we want. They tell us if the Indexing is crap (Result = -1) or how long it takes to retrieve the results. Sometimes this will return a MAPI Error, but you run the test again and it returns a value just fine. Maybe by testing it you woke it up?
After Rollup6
We never found out why those indexes were getting corrupt and why only on that DAG. We have 7 other DAGs that don't have that issue. But after Rollup6, we never saw those indexing issues again.
No, not those issues. We found new ones. But not so terrible ones. At least the Get-MailboxDatabaseCopyStatus reports correctly now.
And in this lastest episode we started seeing a Content Indexing error on the active copy of the database. Update-DatabaseCopy -CatalogOnly doesn't work on the Active copy of a database. And since we are under strict orders to not change anything without Change Management documentation and waiting 14 days for approval, we could not move the database to fix this content index.
I tried to restart the Microsoft Search (Exchange) Service, which tries to restart the Microsoft Exchange Search Indexer Service and the Indexer service hung. I could restart the indexing duo on any other servers in the DAG. It was very clear this database and this server were in a deadlock battle with no relief in sight. In fact this proved it:
sl $exscripts
.\TroubleShoot-CI.ps1 -Database DB01
So I tried to restart the Microsoft Search (Exchange) Service again and the Microsoft Exchange Search Indexer hung while stopping again. I loaded up Task Manger and killed the Exchange Search process, which did immediately restart because the Process ID changed. It finally dawned on me that the Microsoft Search (Exchange) Service had never actually restarted, so I found that process and killed it.
Everything started to work again. So to clear a deadlock, at least in this case, was to kill the Microsoft Search (Exchange) Service process.
This Explains the Troubleshooters:
http://blogs.technet.com/b/exchange/archive/2011/01/18/3411844.aspx
Resetting the Content Indexer to force a new crawl:
http://blogs.msdn.com/b/pepeedu/archive/2010/12/14/exchange-2010-database-copy-on-server-has-content-index-catalog-files-in-the-following-state-failed.aspx
We've been having issues with indexing on one DAG and we thought we had it fixed when we went to Service Pack 1, Rollup 6. This indexing issue doesn't happen very often and I have to learn everything all over again. Putting all my notes here, gives me one place to refresh my view of history.
Before Rollup6 we would see these issues...
People would call and say "I click search and it just spins ... nothing is ever returned."
Do a "Get-MailboxDatabaseCopyStatus DB01" and it reports all is well.
Odd that the Get-MailboxDatabaseCopyStatus would say the Content Index is fine when it obviously isn't. You could failover to another database copy and have the same issue. Reports good, but search doesn't return anything.
(During this, we also found that some clients would search and return results, but when you click on one of the items, you'd get an error message: "Could not display item." This turned out to be a client issue and an update to a newer version cleared that up. But it sent us on a wild goose chase for a bit.)
To fix the "index-don't-work-but-reports-good" issue, we would log onto the server where the passive database copy lived, then:
- Suspend the database copy
- Stop both indexing services: "Microsoft Search (Exchange)" Service and "Microsoft Exchange Search Indexer" Service
- Remove the catalog for that database
- Start both Indexing services
- Resume the database copy
http://blogs.msdn.com/b/pepeedu/archive/2010/12/14/exchange-2010-database-copy-on-server-has-content-index-catalog-files-in-the-following-state-failed.aspx)
This generated a "Crawl" of the database to rebuild the index from scratch. After the crawl is compete you can activate that copy of the database and then update the Content Index on the now passive database. using:
Update-MailboxDatabaseCopy DB01\MBX02 -CatalogOnly -Force
I still need to know when this was failing, and not by a customer calling and saying "I ain't getting nuthin."
Test-ExchangeSearch to the rescue -- at least this was something real. The test adds a message to the System mailbox and then checks to see if it is available through the Search Indexer. An actual honest to goodness test!
Get-Mailboxdatabase -Server MBX02 | Test-ExchangeSearch
This command gets us the results we want. They tell us if the Indexing is crap (Result = -1) or how long it takes to retrieve the results. Sometimes this will return a MAPI Error, but you run the test again and it returns a value just fine. Maybe by testing it you woke it up?
After Rollup6
We never found out why those indexes were getting corrupt and why only on that DAG. We have 7 other DAGs that don't have that issue. But after Rollup6, we never saw those indexing issues again.
No, not those issues. We found new ones. But not so terrible ones. At least the Get-MailboxDatabaseCopyStatus reports correctly now.
And in this lastest episode we started seeing a Content Indexing error on the active copy of the database. Update-DatabaseCopy -CatalogOnly doesn't work on the Active copy of a database. And since we are under strict orders to not change anything without Change Management documentation and waiting 14 days for approval, we could not move the database to fix this content index.
I tried to restart the Microsoft Search (Exchange) Service, which tries to restart the Microsoft Exchange Search Indexer Service and the Indexer service hung. I could restart the indexing duo on any other servers in the DAG. It was very clear this database and this server were in a deadlock battle with no relief in sight. In fact this proved it:
sl $exscripts
.\TroubleShoot-CI.ps1 -Database DB01
So I tried to restart the Microsoft Search (Exchange) Service again and the Microsoft Exchange Search Indexer hung while stopping again. I loaded up Task Manger and killed the Exchange Search process, which did immediately restart because the Process ID changed. It finally dawned on me that the Microsoft Search (Exchange) Service had never actually restarted, so I found that process and killed it.
Everything started to work again. So to clear a deadlock, at least in this case, was to kill the Microsoft Search (Exchange) Service process.
This Explains the Troubleshooters:
http://blogs.technet.com/b/exchange/archive/2011/01/18/3411844.aspx
Resetting the Content Indexer to force a new crawl:
http://blogs.msdn.com/b/pepeedu/archive/2010/12/14/exchange-2010-database-copy-on-server-has-content-index-catalog-files-in-the-following-state-failed.aspx
Thursday, May 24, 2012
Enterprise Wide PST Import - Morning Status Report
This is Part 9 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in the series is here.
This is the report I get each morning. The information is derived from several sources.
The first post in the series is here.
The next post in the series is here.
This is the report I get each morning. The information is derived from several sources.
First, we took a snapshot of our starting point in this project. That was done on Oct 9th, 2011. That's why all the calculation is done from that date.
We run a report every Sunday. It looks for all users, if they have a HomeDirectory defined, and do they have any PST files there? We log each PST file, so we know when there are new ones, if one is removed.
It's just a simple CSV file.
And every day, we get a list of all the people that have an Archive Mailbox and check each one again. Has that file been removed?, is the PST file in use?, etc. The user also gets a reminder message of PST files still attached to Outlook and instructions on how they should be detaching those PST files. The user gets a message every 14 days from the day they were migrated.
We run another process manually on Wednesdays that looks to see how long it's been since a PST file was last accessed. If it is older than 30 days, it tries to move the PST file to the user's local hard drive. Assuming there is space to hold it, of course.
New Hires are put in a GPO that "Disallows PST Growth." Archive Mailbox users are put in that same GPO. We are calling these people "Controlled Users" because we can controll their PST file habit.
In the gap between New Hires and Archive Mailbox users, are the "Uncontrolled Users." Users not in the GPO and free to add messages and create new PST files.
The
ultimate goal of this project is the removal of the PST files from the HomeDiectory.
What
this report means:
· We
have removed 4.3TB of PST data off Home shares
· Uncontrolled
users (Not Online PST Users and Not New Hires) added 1TB of data to new
PST files since Oct 2011
· Uncontrolled
users have added, to their existing PST files, 2TB of data
· Leaving
us a net removal of 1.3TB
· Uncontrolled
Users added 55 PST files just last week.
· The
42% is: NumberOfPSTFilesRemoved /
NumberOfPSTFilesOnOct92011
Percent Complete
|
42 %
|
||||
People with Archive
MBX
|
1,585
|
||||
All Users with PST files on
HomeShares
|
|||||
9-Oct-11
|
24-May-2012
|
Diff
|
|||
Number of PSTs
|
23,516
|
18,739
|
4,777
|
||
Size on Shares
|
12,396,679
|
11,068,508
|
1,328,171
|
||
Users with PSTs on
H:
|
3,004
|
2,506
|
498
|
||
Since Begining
|
Last Week
|
||||
New PSTs on Shares
|
2,293
|
55
|
|||
Size New PSTs
|
1,063,297
|
29,538
|
|||
Users w/Archive Mailbox: PST
files on HomeShares
|
|||||
Discovered
|
Processed
|
Removed
|
|||
Number of PST
Files
|
14,248
|
8,810
|
9,916
|
||
Size on Shares
|
6,992,846
|
3,495,739
|
4,346,864
|
||
Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions
Monday, May 21, 2012
Enterprise Wide PST Import - Using RoboCopy
This is Part 8 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in this series is here.
Our users are world-wide and even though we are concentrating on our Headquarters, many other users are seeing the benefit of importing their PST files. Especially those who travel a lot. Nearly everyone of these heavy travelers carry an external hard drive with their PST files. Now they can get those messages via OWA and they can discard the extra weight of the external hard drive.
We started getting many request for Archive mailbox from all over the world. The challenge is to copy the PST file to our staging area so it can be processed by the server.
Alas, Copy-Item seems to be really slow when working over a WAN line. And many of our pipes can get saturated. I started experimenting with Robocopy, and I am not entirely happy with the results, but it's better than copy-item.
I created a small stand alone script (RoboCopy-Item) that does the very simple copy of a file. I am experimenting with the settings like /IPG:300 /Z, etc -- trying to find the best overall throughput for our environment. Fast, but not choking the WAN. I am still experimenting there.
In the Add-PSTImportQueue function we now look to see if a user is in HQ or outside somewhere. If they are outside HQ then we mark the job with a status of RoboCopy.
Then we use: Robocopy-PSTImportQueue which looks at the import queue and starts background jobs for each job with a RoboCopy status. I do this on a by user, by location selection. We don't want to have 50 jobs running for 30 users in 1 location. I keep it at 1 user per 1 location at any given time.
Then we use a Get-RoboCopyjob <jobnumber> that just gets the 1st 10 lines and that last 20 lines of a job, so you can quckly see the status of the job.
We toyed with the idea of incorporating this into the overall script so we could just run it, but too many things can go wrong and the powershell window with all the jobs running can get closed ot the server rebooted, etc.
We want to find a better way to do this.
Still bigger files -- over 1.5G -- are taking forever. We are using BITS to control the amount of traffic that can be allowed, so we have to go modify the Registry and restart BITS service. But copies from computers with that setting fills up the pipe to that location.
I'll update as we search for the better way.
Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions
The first post in the series is here.
The next post in this series is here.
Our users are world-wide and even though we are concentrating on our Headquarters, many other users are seeing the benefit of importing their PST files. Especially those who travel a lot. Nearly everyone of these heavy travelers carry an external hard drive with their PST files. Now they can get those messages via OWA and they can discard the extra weight of the external hard drive.
We started getting many request for Archive mailbox from all over the world. The challenge is to copy the PST file to our staging area so it can be processed by the server.
Alas, Copy-Item seems to be really slow when working over a WAN line. And many of our pipes can get saturated. I started experimenting with Robocopy, and I am not entirely happy with the results, but it's better than copy-item.
I created a small stand alone script (RoboCopy-Item) that does the very simple copy of a file. I am experimenting with the settings like /IPG:300 /Z, etc -- trying to find the best overall throughput for our environment. Fast, but not choking the WAN. I am still experimenting there.
In the Add-PSTImportQueue function we now look to see if a user is in HQ or outside somewhere. If they are outside HQ then we mark the job with a status of RoboCopy.
Then we use: Robocopy-PSTImportQueue which looks at the import queue and starts background jobs for each job with a RoboCopy status. I do this on a by user, by location selection. We don't want to have 50 jobs running for 30 users in 1 location. I keep it at 1 user per 1 location at any given time.
Then we use a Get-RoboCopyjob <jobnumber> that just gets the 1st 10 lines and that last 20 lines of a job, so you can quckly see the status of the job.
We toyed with the idea of incorporating this into the overall script so we could just run it, but too many things can go wrong and the powershell window with all the jobs running can get closed ot the server rebooted, etc.
We want to find a better way to do this.
Still bigger files -- over 1.5G -- are taking forever. We are using BITS to control the amount of traffic that can be allowed, so we have to go modify the Registry and restart BITS service. But copies from computers with that setting fills up the pipe to that location.
I'll update as we search for the better way.
Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions
Tuesday, May 8, 2012
More PST Import Utils - Get-ImportStatus & Lock-File
This is Part 7 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in this series is here.
Quicker Stats
We found the MailboxImport queue needs a little tender care from time to time. We always would run this command to get a quick understanding of what was going on in the queue:
Get-MailboxImportRequest | Get-MailboxImportRequestStatistics
Sometimes we needed :
Get-MailboxImportRequest -Status Failed | Resume-MailboxImportRequest
I really got tried of typing all that out all the time, so I created a short function for me.
Function Get-ImportStatus (){
#---------------------------------------------------------
# a helper function to display mailbox import info
# option to show only a subset -- by batchname
# sometimes the list is just long
# option to restart failed jobs --
# sometimes jobs failed because the service crashed on a bad PST file
# or too many jobs for one mailbox
# option to restart suspended jobs
# the script can suspend jobs it thinks may be causing issues
# option to suspend all jobs
# a single PST can crash the MB rep service and all jobs start over
# you can't tell exactly which is the culprit so suspend all jobs
# and investigate
# left the -confirm off on purpose
param (
$Batch=$null,
[switch]$RestartFailed,
[switch]$RestartSuspended,
[switch]$SuspendAll
)
If($Batch) {
Get-MailboxImportRequest -BatchName $Batch |
Get-MailboxImportRequestStatistics
}
ElseIf($RestartFailed.IsPresent) {
Get-MailboxImportRequest -Status Failed |
Resume-MailboxImportRequest
}
ElseIf($RestartSuspended.IsPresent) {
Get-MailboxImportRequest -Status Suspended |
Resume-MailboxImportRequest
}
ElseIf($SuspendAll.IsPresent) {
Get-MailboxImportRequest |
Suspend-MailboxImportRequest
}
Else {
Get-MailboxImportRequest | Sort Name |
Get-MailboxImportRequestStatistics
}
}
Stepping all over each other
You now how it is, everyone gets busy and stops checking with others about what's going on and crap happens. The way we were handling the queue files became an issue. If two people ran the script, the last one to write was the winner. I noticed this when I tried to schedule a task and the queue was trashed. (Lots of manual fixing up there.) And there were a few other small disasters, too.
I remembered and old way to make sure one process did not step on the other, create a "lock" file when you started your work and then delete the "lock" file when you done.
This wasn't exactly elegant, but it works just fine. We're just doing this with a zero lenth file.
Simply check for the existance of the file (Test-PSTIQLock) and if false, lock the file (Lock-PSTIQ) process the queue and then remove the lock (Unlock-PSTIQ)
Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions
The first post in the series is here.
The next post in this series is here.
Quicker Stats
We found the MailboxImport queue needs a little tender care from time to time. We always would run this command to get a quick understanding of what was going on in the queue:
Get-MailboxImportRequest | Get-MailboxImportRequestStatistics
Sometimes we needed :
Get-MailboxImportRequest -Status Failed | Resume-MailboxImportRequest
I really got tried of typing all that out all the time, so I created a short function for me.
Function Get-ImportStatus (){
#---------------------------------------------------------
# a helper function to display mailbox import info
# option to show only a subset -- by batchname
# sometimes the list is just long
# option to restart failed jobs --
# sometimes jobs failed because the service crashed on a bad PST file
# or too many jobs for one mailbox
# option to restart suspended jobs
# the script can suspend jobs it thinks may be causing issues
# option to suspend all jobs
# a single PST can crash the MB rep service and all jobs start over
# you can't tell exactly which is the culprit so suspend all jobs
# and investigate
# left the -confirm off on purpose
param (
$Batch=$null,
[switch]$RestartFailed,
[switch]$RestartSuspended,
[switch]$SuspendAll
)
If($Batch) {
Get-MailboxImportRequest -BatchName $Batch |
Get-MailboxImportRequestStatistics
}
ElseIf($RestartFailed.IsPresent) {
Get-MailboxImportRequest -Status Failed |
Resume-MailboxImportRequest
}
ElseIf($RestartSuspended.IsPresent) {
Get-MailboxImportRequest -Status Suspended |
Resume-MailboxImportRequest
}
ElseIf($SuspendAll.IsPresent) {
Get-MailboxImportRequest |
Suspend-MailboxImportRequest
}
Else {
Get-MailboxImportRequest | Sort Name |
Get-MailboxImportRequestStatistics
}
}
Stepping all over each other
You now how it is, everyone gets busy and stops checking with others about what's going on and crap happens. The way we were handling the queue files became an issue. If two people ran the script, the last one to write was the winner. I noticed this when I tried to schedule a task and the queue was trashed. (Lots of manual fixing up there.) And there were a few other small disasters, too.
I remembered and old way to make sure one process did not step on the other, create a "lock" file when you started your work and then delete the "lock" file when you done.
This wasn't exactly elegant, but it works just fine. We're just doing this with a zero lenth file.
Simply check for the existance of the file (Test-PSTIQLock) and if false, lock the file (Lock-PSTIQ) process the queue and then remove the lock (Unlock-PSTIQ)
Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions
Tuesday, April 10, 2012
Enterprise Wide PST Import -- PST Capture
This is Part 6 in a series of posts about my experience tackling the migration of PST files.
The first post in the series is here.
The next post in the series is here.
When the Exchange Team at Microsoft posted about PST capture in July of 2011, I was very excited. That post was really the catalyst that got us started thinking we could really Import PSTs. We started putting our infrastructure together to handle all the PSTs floating out in the wild. The future look bright.
And we waited...
During our waiting period we had our own crisis or two that propelled us into the PST import business. By the time PST Capture was officially out in Jan of 2012, we were already fully functional with our scripts and we saw no compelling reason to change.
Still I saw value. We have many users in the field that don't have Home Shares at Headquarters and they were saving their PST files locally. We needed to get those PST as well, eventually. So I downloaded the PST Capture tool and did some testing.
I was sad to find out that to do discovery on a PC, you had to have an agent installed. We still have that same issue with delivering an EXE to all the PC's in the world. So that was out. I tried it on a few PCs, thinking we could do this from time to time. But I could not get it to work. I didn't go deep into troubleshooting, and just thought it was a firewall issue. That would be a nightmare to get open!
But I did try using the UNC file path and that was working. Until I found out that this did not work on some clients. Seems like it failed on all the clients we needed it to work on. Like Office 2007.
There is no reporting feature with PST Capture, true we could still use the same reports we use now, but we would lose a few statistics, like how many files were processed and skipped.
We stopped testing.
So if you're wondering why we went to all the trouble to write this family of scripts to import PSTs, instead of using PST Capture, now you know the rest of the story.
Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions
The first post in the series is here.
The next post in the series is here.
When the Exchange Team at Microsoft posted about PST capture in July of 2011, I was very excited. That post was really the catalyst that got us started thinking we could really Import PSTs. We started putting our infrastructure together to handle all the PSTs floating out in the wild. The future look bright.
And we waited...
During our waiting period we had our own crisis or two that propelled us into the PST import business. By the time PST Capture was officially out in Jan of 2012, we were already fully functional with our scripts and we saw no compelling reason to change.
Still I saw value. We have many users in the field that don't have Home Shares at Headquarters and they were saving their PST files locally. We needed to get those PST as well, eventually. So I downloaded the PST Capture tool and did some testing.
I was sad to find out that to do discovery on a PC, you had to have an agent installed. We still have that same issue with delivering an EXE to all the PC's in the world. So that was out. I tried it on a few PCs, thinking we could do this from time to time. But I could not get it to work. I didn't go deep into troubleshooting, and just thought it was a firewall issue. That would be a nightmare to get open!
But I did try using the UNC file path and that was working. Until I found out that this did not work on some clients. Seems like it failed on all the clients we needed it to work on. Like Office 2007.
There is no reporting feature with PST Capture, true we could still use the same reports we use now, but we would lose a few statistics, like how many files were processed and skipped.
We stopped testing.
So if you're wondering why we went to all the trouble to write this family of scripts to import PSTs, instead of using PST Capture, now you know the rest of the story.
Introduction: The Beginings
Part 1: Script Requirements
Part 2: Add-PSTImportQueue
Part 3: Process-PSTImportQueue
Part 4: Some Tools we added
Part 5: Set-PSTImportQueue
Part 6: About PST Capture
Part 7: More PST Import Tools
Part 8: Using RoboCopy
Part 9: Morning Status Report
Part 10: Using BITS Transfer
Part 11: Get the script / Set up
Part 12: The Functions
Subscribe to:
Posts (Atom)