This morning I had the (dis)pleasure of upgrading our Exchange 2010 environment to SP3. SP3 has been out for quite a while but I hadn't actually bothered to upgrade the production environment because there were so many issues reported on the TechNet forums with SP3, I figured it would be best to wait a bit and then other things got in the way.
SP3 is a really important upgrade for 2010 because it enables migration to Exchange 2013 and SP3 rollup 3 also fixes the ridiculous issue that OWA only runs in "light" mode under IE11/Windows 8.1.
Without a doubt it was the most painful Exchange Service pack upgrade I have ever done. It involved at least 10 full server reboots, hacking Active Directory with ADSIEdit and manually changing settings on the Exchange virtual directories using the IIS Admin tools. Because they are completely unsupported, I absolutely hate having to resort to these kind of hacks.
I had previously run the upgrade in a test virtual machine environment which went pretty smoothly. There was one extra pre-requisite to install and a couple of custom transport agents broke post-install due to SP3 updating some referenced DLLs, but apart from that it was pretty smooth.
In the actual production environment (which is still a pretty vanilla setup) it was a different story. First the prerequisite required a complete server reboot, which the test machine didn’t. Then the actual install of SP3 failed, some of the components succeeded but it failed on the Hub Transport role, skipping the subsequent roles and leaving the server in a crippled state.
The key part of the error causing the Hub Transport role upgrade to fail was “The virtual directory 'PowerShell' already exists under 'servername/Default Web Site'”. There was a suggestion on the TechNet forums to blow away an entry in Active Directory using ADSIEdit, specifically under Configuration\Services\Microsoft Exchange\Administrative Groups\Administrative Group\Servers\SERVERNAME\Protocols\HTTP. I believe that deleting the Powershell (Default Web Site) and Powershell-Proxy (Default Web Site) entries may have been enough, however I unfortunately followed the advice to delete the owa (Default Web Site) object as well.
This got the service pack through the installer but OWA was completely broken post-install and as the Exchange Management Shell could not see the OWA object anymore (as it had been removed) it was unable to delete it, and trying to create a new OWA virtual directory also failed.
If you do run into this situation with SP3 I’d strongly recommend trying just removing the Powershell and Powershell-Proxy entries using ADSIEdit, then using the Exchange Management Shell to delete the OWA virtual directory properly, to avoid having to hack around with IIS after the upgrade.
As the owa entry had been removed from Active Directory I basically ended up manually deleting the Owa, Public, Exchweb and Exchange virtual directories in IIS. It turned out IIS hadn’t deleted the owa/oma and owa/Calendar virtual directories either, but they were inaccessible via the GUI. You could hack the IIS metabase to get rid of them but I ended up re-creating the owa virtual directory manually to be able to delete the child ones, then also deleting the application pools MSExchangeOWAAppPool and MSExchangeOWACalendarAppPool.
Fortunately, after those changes, the New-OWAVirtualDirectory command was able to run without errors and the owa Active Directory object, as well as the actual virtual directories in IIS, were re-created successfully.
This was an incredibly messy hack and if it hadn’t worked it would have pretty much removed any possibility of getting OWA working, leaving the only course of action to completely reinstall the Client Access Role which I was really hoping to avoid. It seems that the New-OWAVirtualDirectory command isn’t very intelligent, as it doesn’t check whether individual components already existing before trying to re-create them, so there was no way to recover it using supported tools. As I thought I was going to have to completely reinstall the Client Access role to recover the Exchange server, there wasn’t really much to lose by deleting a few more bits from IIS.
I also found that SP3 changed the bindings on the Default Web Site in IIS, in that it added additional HTTP and HTTPS bindings with a host header name of localhost. This also broke the New-OWAVirtualDirectory command until those invalid bindings were removed (we have bindings on the same ports with no host headers so they seemed completely unnecessary, except that they generate SSL errors).
It was a good lesson in not hacking Exchange with ADSIEdit except as a last resort, as mentioned I believe the OWA problem could have been avoided by deleting the OWA virtual directory properly before the SP3 upgrade, instead of following the (Microsoft) recommendation on the TechNet forums to delete it from AD. Of course why SP3 sometimes doesn’t install properly without making such drastic changes in the first place is another matter.