Last year we upgraded a big customer's intranet, it was a long project and during most of the time there were 4 to 6 people working on it. I'll stay away from most topics covered in Upgrading to Office SharePoint Server 2007 because they are treated there and I couldn't say anything new. What I want to tell you is how we organized the team, some steps we took to perform the migration and some ideas we had to improve the result and the team work.

It's mainly focused on our migration project, but many things can be applied to any kind of project.

I'll omit many client-specific actions (a very big document modification process), so the result can be applied to any migration process. I also won't enter into design customizations, webpart development and such things.

It's very important to understand that you really should "quickstart" the migration process even if the migration at first is far from perfect. The sooner you have something to play with, the sooner you'll find faults and you can start producing productive things.

If you have any doubt, I’ll love to help you with it, just use the comments :). Remember you can also subscribe to the comments thread.

Scenario

The scenario was a SharePoint 2003 installation with:

• About 30GB of data (which were doubled to 70Gb due to a recategorization process some months before migrating). Over 3000 webs most of them with a single default.aspx, and a customized document library.
• The default area definitions were modified.
• About 20 site collections (the intranet, 10 personal sites, and some team sites).
• Some pages were modified using SPD.
• Over 16000 profiles, 8000 users a day.

Other things (not covered here):

• We have to integrate data the data from >2000 webs into a single document library using folders, keeping permissions, version history and all. About 60Gb. We called this "consolidation".
• We had to "normalize" the 70Gb data, stepping from the typical lookup columns to a custom column types, and change some of the metadata that wasn't properly categorized.
• Some the data (20Gb or so) had to be exported, normalized and imported in another moss farm (the public portal).
• We had to make a complete graphical redesign of the intranet, branding and so.
• Develop a ton of webparts, both existing and new.

As we had new hardware we chose a content database migration.

Create MOSS Software Baseline

The first thing is to create a simple document (an excel for example) in which you specify all of your servers and the typical developer machine and register all the moss related software installed on everyone.

All machines should have the same patch level, updates and so. It is very important as a bug can appear on one machine while not on the other and then it will be a mess!

The initial baseline could be MOSS + SP1 + Infrastructure Update, if you need an specific hotfix, install the proper accumulative update in all machines and annotate it in the baseline.

It is also a good idea to keep all downloaded updates in network share, and if you have both 32 and 64bit machines download them all.

Install the "DDS" (Dirty Development Server)

That's how I call it :). It is a server, distinct from the "regular development" server,

I want it to perform the test migrations without bothering the team members: In our project in the DDSes (1Gb RAM moss + 1.5Gb RAM SQL2005) the migration process took between 9 and 12 hours so, by doing the migration in a different server, nobody was affected.

The third day (literally) of the project we performed the first test migration (of course the results were mostly useless) but it gave us some initial data to play with. So start doing migrations (even unsuccessful) ASAP.

Know your data and customizations

It is crucial to know what customizations have been performed so you should:

• Of course execute prescan.exe. It will give you some info about errors in the DB, customizations made, and unghosted pages.
• Prescan does not detect when you have customized the default are definitions. So if you only look at prescan results your migration can lead to unusable data. The trick to detect those customizations is to have a clean SPS2003 installation (with the same patch level as the real production server) and compare the 60 hive with the production one using Beyond Compare or a folder compare tool. Using this you will get all the changes to the definition xmls, the additional images, and everything they could have installed.

Create a reduced Data Set

Once you know what customizations have been applied, you might need to create a reduced set of data to "debug" your migration:

We had some custom area definitions, so we need to customize the XML files located in %12%\Config\Upgrade so MOSS upgrade can create equivalent webs in MOSS and your data get migrated properly. This files contains mappings between files in SPS2003 and MOSS2007, the features required on each site type and things so.

As the only known way to "debug" that XML is to do the migration itself it is a good idea to have an small subset of the data in the server to do quick migrations and quickly fix that XML.

You can:

• Install the customizations on a clean SPS2003, and create test areas, and list, inserting a few data.
• Take the development environment of the current intranet and delete things until you have a decent but not excessive amount of data.

The idea is to have at least a small amount of data of every list template and site template, so the migration could be performed in your laptop in about and hour or less.

Create target Content-Types

If there are list customizations, you have to create the features for the Content-Types matching those list data. Be sure that not only the name of the fields are the same, but also the internal name of them, because it can lead to some problems.

Remember that the content types must match only the columns referenced in the site definitions, not the columns added though the user interface, those columns are migrated automatically.

That features are the ones that need to be referenced on the %12%\Config\Upgrade XML files.

Every developer works in his own environment

Working on a central server is a thing some people do but I can only see problems. Every developer should have the environment on his PC. As simple as that.

They can have all the data or the reduced data set, but they need to develop on their machines.

The size of the data is not such a problem when using today computers, I had attached a 100Gb database on my laptop and survived. Believe me. What is more, you should have a decent amount of data locally to test with, we had a problem we didn't discover early because the developer only had 50 docs or so in the library and accessing the items collection was not a problem, when we tested that code with 10.000 docs the problem arose...

The development server should be used to integrate everything.

Distribute the Content-Types to Developers

Once you have at least the initial content-types, your developers can start working with them in the webparts, search and things so.

So provide them with the new versions as they become available, but it is important that they (even more if they are not very experienced) start to work with the content-types ASAP.

Refine Migration Process by Iterating

Using the data of the reduced set, and the XML, all you have to do is migrate, check, fix and migrate again. Don't be fooled by the fact that you can open, see or add a new version of the item. A very good test is trying to move the item from one folder to another and then from one library to another, if it moves it is likely to be OK.

If it does not move, you can create a new item and compare the metadata with a migrated one using SharePoint Designer: The migrated will contain the metadata (field names, internal names, guids) from the migration, but the new one will contain the metadata from the Content-Types. If they don't match, modify the Content-Types to match the columns in SPS2003.

This process took us a very long time, but since we had a near real Content-Types by the first weeks everybody could work based on them.

Automatize as much as you can!

Make batch files, stsadm commands, and everything you need to avoid repetitive tasks.

We had a batch that performed the migration, exported, imported, normalized data... It made 85% of the migration itself. Repeatable and not error prone.

You can create scripts that install solutions, activate features, move webs, create webs, upload docs,... Gary Lapointe stsadm commands are a must.

The goal can be that a developer can delete his working web app and create a new one with all the features in less that (let's say..) 20 minutes. It should not be painful or take a lot of time, recreating the working environment in our case was something quite usual.

We also did a lot of stsadm commands to make the migration work easier:

• Re-ghost pages in a branch of webs.
• Configure the layouts of the pages of a branch of webs.
• Configure the themes used by a branch of webs.
• Delete an specific webpart in a branch of webs.
• Export a page webparts and configuration and import in another page (great for designing and configuring a page in the development environment and moving it to production environment).
• Modify the web.config in all servers of the farm by using the SPWebConfigModification class.
• Many more

Since we needed to do a lot of web structure modifications, we made an utility in where you altered the web tree of the site and generated a batch file that performed the site creation, moving, renaming and deleting. So you could do it in your dev environment once, and test it many times in the test migrations.

Of course it is absolutely recommended that you use solutions for the deployment, they get propagated “automagically” to the servers in the farm, so the maintenance gets greatly enhanced.

One tip for multiple frontend farms, install the features by hand, not by a bat (or use a bat with pause statements), the reason is this: when you deploy a solution, the command exits before the solution has been deployed to all the servers in the farm and when if you activate them right away they might not be actually deployed so you might get an error.

There are some utilities in Codeplex to export and import the search settings, managed properties and such configuration things, you should take a look at them: you invest 30 additional minutes now and you save 2 hours and a headache in the real deployment :)

Migrating Profiles

We did not migrate the profiles, we simply re-imported them. Since there was nothing special and all the profile data came from the AD, there was more work in migrating the profiles than in importing them again.

The only "special" thing we did was to change the LDAP query to import only the active profiles instead of importing all the accounts from the AD. There were literally thousands of inactive accounts that shouldn't be imported.

The original query was:

(&(objectCategory=Person)(objectClass=User))

And our query:

(&(objectCategory=person)(objectClass=user)( !(userAccountControl:1.2.840.113556.1.4.803:=2)))

Advance as much work as you can

Since we did not migrate the profiles we installed the farm some days before, we created the webapps, set up the SSP, imported the profiles and QuickLinks, installed the features, did a test migration, create the managed properties, search scopes, etc and run some performance test on it. The older intranet was still being used.

It was a fully functional test migration, after that we deleted the content db using the central administration and re-migrated the real database. All the ssp config, profile importation, quicklinks and so was in place before the migration took place.

That test migration was very useful to get timings of how long would it take to complete the real migration (they were the same) and helped us to synchronize with the team that was migrating the public-facing portal that needed some data from us.

Identify the URL changes due to the migration

It was very specific to us, but helped in the alert migration and user links migration. The main point is that in MOSS all the bucket webs disappear and the site URLs get changed, so a site like "/C11/Product" can become "/Portfolio/Solutions/Product". The bucket webs were that estrange "C-Something" webs that  ruined your URLs :)

Now they are gone, so most of your webs will change the URLs. We wanted to have a matching database to identify SPS2003 sites URL with MOSS2007 URLs.

We did an small utility that parsed the "Upgrade.log" file that the migration process generate and stored the site id (does not change during the migration process), old URL and new URL in a database table.

This “correspondences” table was used when migrating the alerts and the personal links of the users.

Migrating Alerts

We did a custom stsadm command to recreate the alerts, it read the alert data from the Service database in 2003 and created them again in the new places using the data from the correspondences table.

The tables in the 2003 Services db were sub_SubscriptionDetails, sub_UserSubscription, sub_SubscriptionChannels, we read the data from them and create the alerts using the SPWeb.Alerts collection.

To identify the user we matched the emails of the alerts with the emails we got from the user profiles of the SPSite.

One of the points that lead to this approach was the data consolidation we needed to do, but also we weren’t able to attach the servs db as we saw in some blogs, so we got a solution using this approach.

Migrating QuickLinks

When migrating QuickLinks we used a similar approach as in the alerts, we read the data from the QuickLinks table in SPS2003 Profiles DB , and inserted it in MOSS  using the QuickLinks collection of the UserProfile object.

This was very straightforward. The groping of the links keeps just like it was on 2003.

Migrating the Personal Sites

We migrated the few personal sites just like they were, as the number of sites was pretty low we kept them that way in the same web app as the intranet.

If the number were bigger I think we should have moved them to another webapp in another db. The procedure would have been migrating them first and then moving the site collections using stsadm commands.