Update: There is a simple solution to this problem after all. See How to ghost an unghosted content type for more details.
Just about all SharePoint projects require that you create some custom content types and/or add some custom site columns, and this is usually done through a feature that contains the CAML definition for those content types and site columns, as described in the MSDN article Content Type Deployment Using Features. It is also often the case that for some reason these CAML definitions will change during the project. Maybe a certain feature was not specified correctly, the requirements changed, or additional features are added that require new columns to an already existing content type. This is where the first problems start to arise...
If you read the MSDN article about deploying content types you will see the following warning:
"Under no circumstances should you update the content type definition file for a content type after you have installed and activated that content type. Windows SharePoint Services does not track changes made to the content type definition file. Therefore, you have no method for pushing down changes made to site content types to the child content types."
This warning is usually ignored while in development, as clean builds are done regularly where everything is deleted and the solution is deployed to a fresh site collection. The problem usually doesn't show up until after the solution has gone into production and a later update tries to make some changes to these content type definitons. These changes will be pushed to the site content types but the list content types are basically detached copies of the site content types and therefore will not be updated.
Luckily there is a hack solution to this problem, and it comes in the form of a custom STSADM command. This command will push changes to the site content type down to all list content type instances, thus solving the problem once and for all. Or perhaps not? I stumbled upon this command while trying to figure out how to do just such an update, and after trying it out with success in our test environment I decided to go ahead and use the same command in the production environment. Sadly it did not work, and I could not figure out why.
Then I read a post on our internal blog by Jan Inge Dalsbø with a title of something like "Mysteries of SharePoint content types" where he goes on to explain that site content types can actually be unghosted (detached from their CAML definition) if they are edited in the SharePoint UI. Any changes to the CAML definition done after this has happened will not be pushed to the site content type, and you are - pardon my french - properly fucked! The content type will have to be updated through the UI or object model from that point on.
To verify that the content type has in fact been unghosted you need to check the content database. To do this you will have to run a query against it (obviously) and this will actually leave you in an unsupported state! Yep, we all knew you weren't supposed to modify any of the SharePoint databases but as it turns out you're not even allowed to read from it. That being said, here are two before and after screenshots of a content type being unghosted.
Notice that in the second screenshot the definition field has gone from being NULL to containing the content type CAML definiton. The content type has now been detached from its definition, updates will fail, hairs will be pulled, gods will be cursed etc. etc. I haven't dared to poke around in the production database to see if this is actually what happened in my case, but I'm pretty certain it's the same problem.
So where do you go from here? I honestly don't know... Søren Nielsen, the original author of the code used in the STSADM command mentioned earlier, has a hack solution to this problem as well, but it involves modifying the content database. In his blog post Convert “virtual” content types to “physical” he explains the steps needed to reattach the content type to its CAML definition. Keep in mind that this will leave you in an unsupported state!
This was all new to me, and I was a bit shocked to find out that this is how it works. Basically Microsoft has given us a big fat rope to hang ourselves with here. From now on I will definitely tell all site collection administrators to never touch the content types!