Update Cautiously

2009-11-03

There is not much controversy about the importance of keeping the software on your computer up to date. If you have a computer connected to the Internet, you need to keep things updated to ensure that any vulnerabilities that have been discovered will get patched in a timely manner. You do not want to be the only person on the Internet still dealing with an image handling library vulnerability from 2007, after all.

Where things get controversial is in the area of how you handle software updates. The security experts from large corporate software vendors such as Apple and Microsoft are fairly close to unanimous in their insistence that the best way to keep security vulnerabilities patched is to turn on the automatic software updating mechanism particular to the operating system and let the vendor decide when things should be updated.

Of course, there have been many instances this decade where uncritically accepting any and all software updates could lead to rather severe problems. One of the best remembered examples of this was the deployment of Microsoft Windows XP's first Service Pack. Many system administrators remember the initial deployment of SP1 with a chill up the spine, because it caused software and hardware compatibility problems akin to those one might expect from using an MS Windows upgrade CD to move from Win98 to Win2k or ME.

In fact, I was personally bitten by this problem — not because I installed SP1, but because one of the clients for the consultancy where I worked at the time did so, in direct defiance of the advice we gave that client. The consultancy made close to a thousand dollars in billable hours off that affair, and I got paid my usual rate for the work I did, but the aggravation of solving the problem caused by a client who did not take our advice was for me at least not fully offset by the money.

We told the client "Do not install SP1 until we have had a chance to research the problems people are having with it and determine what, if any, workarounds are necessary to ensure a smooth upgrade." Within a couple of days, the client got a notification from Windows Update informing him that installing Service Pack 1 was very important, and that he should do so right away. He panicked, of course, and did so across his entire network. The effects were smooth except for one computer. He gave us a call and said "I installed SP1 on everything, and one of the computers is broken now."

The natural reaction from the boss, who handled the call, was probably not the best reaction possible. He said "This is why we told you to wait on installing SP1. I'll have my guy go over there and give it a look." The client's reaction was even worse: he panicked again, and tried to uninstall SP1 from his own computer. Being far from the most technically adept person on the planet, what he ended up doing was completely hosing up the computer on his own desk.

As a result, when he called back breathless and fearful to tell us that he tried to remove SP1 in some bizarre, really awful and bad way of doing so that I do not even recall any longer, he broke his computer. The boss sighed and told him we would be there in a few minutes, and we drove out to the client's site together. He worked on the client's computer, and I worked on the receptionist's. Luckily, there weren't any other computers negatively affected by this mess.

The problem with the receptionist's was an interesting one. The hard drive on which WinXP was installed was an SATA drive, which used an SATA controller plugged into a PCI slot. When SP1 was installed, it came with Microsoft's new driver certification checking. The next time the computer was restarted, it got as far as the initial hardware certification check, then noticed that the driver for the SATA controller was not Microsoft certified. MS Windows then unloaded the SATA controller's driver, tried to finish booting, and could no longer read the hard drive because there was no driver for the SATA controller.

Several hours later, the two computers affected by the client's inability to take good advice when it is offered were running smoothly again, at a cost of hundreds of dollars and much frustration on the parts of everyone involved (except, perhaps, the receptionist herself, who got to go home early while we worked).

Other problems for which SP1 was famous included a lot of networking software simply ceasing to work because SP1 deemed it "insecure" and many drivers and applications failing because APIs and ABIs had changed, breaking the software. This was especially common among corporate enterprise networks where a lot of custom software was used to serve specialized business needs.

While SP1 is probably the instance of uncritical application of software updates causing severe problems that is most widely remembered, it is far from the most spectacular. That prize must go to the problems surrounding the SQL Slammer worm.

About four months after people started running afoul of SP1's gotchas, the SQL Slammer worm hit the Internet hard. Nearly 75,000 computers were infected within ten minutes on 25 January 2003, and the only reason it did not spread further than it did is that its denial of service effects were so severe that it took down large networks before the worm could spread through them. The tech world was in a panic and huge chunks of the Internet went "dark", and talk of Internet-killing malware started making the rounds.

Over the ensuing months, there was a constant battle to get as many vulnerable machines as possible updated to eliminate the vulnerability SQL Slammer used to spread. It was revealed that a pair of updates issued by Microsoft months earlier actually addressed this vulnerability, and soon Microsoft and many users of Microsoft software were blaming "lazy" system administrators for failing to keep their systems updated. People surely lost jobs over this. Ironically, Microsoft itself was affected by SQL Slammer, as some of its systems were infected.

It turns out that many of those system administrators who insisted they had kept machines up to date were probably entirely innocent of any wrongdoing or laziness in this case. Certain updates from 2002, if installed in the "wrong" order, would actually result in a security patch important to eliminating the vulnerability SQL Slammer exploited being effectively neutralized. This news came out months after the initial SQL Slammer panic, to relatively little fanfare, but it was an enlightening moment for many system administrators who had previously believed that installing every Microsoft software update right away was the best way to keep security patches up to date.

The lesson to be taken from these examples of disaster caused by vendor-issued software updates is one of caution and testing. Most corporate enterprise networks have some kind of testing policy in place for Microsoft software update deployments these days, sometimes even going so far as to have entire test networks and "beta testers" within the organization whose job it is to suffer any potential ill effects of a software update before it is deployed to the rest of the network. I, for one, never recommend that anyone use Windows Automatic Updates, because you should always know what is getting installed on your computer. The much more recent incident where Microsoft made Firefox vulnerable to .NET vulnerabilities is an excellent example of what happens when you do not know what is being installed on your computer.

No other software vendor has, to my knowledge, achieved quite the scope and impact of these two software update management disasters. Still, any critical software on your computer or within your organization should be patched by software updates with care, and only after at least minimal research into the nature and effects of the update. If it is reasonable to do so, such updates should be tested before deployment to "live" systems, as well — and backups should be kept so that any changes can be undone as needed.

Chad

,

---

Comment

Commenting is closed for this article.

---