BOSTON -- Weeks before the notorious Blaster worm blighted computers everywhere, Bill Arrington and his fellow network administrators at Children's Hospital already knew of the vulnerability that could open the door to his Windows servers and desktops.
"We had all of the information," Arrington said. He and his colleagues, who make up the network operating system (NOS) team in the hospital's information systems division, just figured they'd patch everything during their next stretch of free time.
But the fact that Microsoft sent a special e-mail to the hospital's chief information officer, who in turn communicated the heightened warning to the NOS team, launched everyone into action, Arrington said. "Microsoft doesn't usually [contact the CIO]," he said.
Indeed, when the Blaster, or Lovsan, worm started to circulate, some workstations at Children's Hospital were affected, but the facility's 120 servers were unharmed. Though some of the patches were applied by hand, for the most part, Arrington used a patch manager from Ecora Software Inc., Portsmouth, N.H., and Microsoft's desktop management software, Systems Management Server. The servers were patched, using the Ecora Patch Manager, at night before the worm hit, so they could be rebooted.
When the worm began to spread, it went everywhere, fast, he said. There was never a time when patient care was compromised, and the hospital never went into an alert stage, Arrington said. But network performance faltered, and at one point the firewall shut down. The help desk was flooded with calls from users who couldn't access the network. A separate network team was checking into the firewall, and everyone started putting two and two together.
3,000 desktops to patch
Arrington and a small team of IT professionals worked hard to apply patches, using SMS, to approximately 3,000 workstations throughout the hospital. There were about 200 PCs running Windows XP, so those were easy to patch.
Microsoft usually only makes patches for the current version of an operating system and the previous two service packs. Service Pack 4 was released in late June, so like many other Windows enterprises, Children's Hospital had not yet updated its Windows 2000 machines and was running SP2. To accept the patches, the machines were supposed to at least be running SP3, so before the patches were deployed, the team had to first think about updating its service packs.
Instead, Arrington said, they called Microsoft to ask whether the patches would roll out using SP2. "Microsoft said, 'We haven't tested it, but you should be fine,'" he said. "We took that as a go and ran with it."
Aside from patching the servers and desktops, Children's Hospital also had to purge the worm from its systems. The hospital called its antivirus vendor, McAfee Security, a division of Network Associates Inc., for help. McAfee didn't have a quick solution, so Arrington and his colleagues started playing with the antivirus vendor's Stinger program, a standalone utility that is used to detect and remove viruses and worms.
Code glitch resolved in an hour
The hospital had trouble applying some patches using SMS in conjunction with the Stinger code. It turned out that there was a glitch in Stinger that McAfee was able to rectify within an hour. With the virus-removal utility working and the patches applied, the systems could be cleaned and patched. Almost all of the hospital's 3,000 desktops were inoculated and fortified within two days.
Arrington said that, without patch management and antivirus tools, the whole process would have taken much longer. And having never gone through such an extensive outbreak, he said, the team learned lessons about how to approach this sort of attack if it happens again. "The only reason this took two days was partly because we've never done it before," he said. "If we got a virus like this today, we'd wrap it up in four hours."
In the end, Arrington said, if he were to do anything differently, he would make sure he had the latest service pack installed on each machine. But loading service packs takes time, too. Children's only has SP4 for Windows 2000 installed on about five or six servers today, but in November when the technical team has that stretch of free time, everything will be updated, Arrington said.
FOR MORE INFORMATION:
Best Web Links: Security alerts