There are 5,000 PCs spread throughout dozens of buildings on the 130-acre campus of the Ernest Orlando Lawrence Berkeley National Laboratory.
One network admin is in charge of keeping the security of Windows PCs in each of the lab's 12 main departments up to date. Using Microsoft's Baseline Security Analyzer, Douglas Spindler knows which departments are not up to speed.
"Everybody's behind," said Spindler, Berkeley Lab's Active Directory project rollout coordinator. Some machines, like those used by administrative assistants, go untouched. Can't blame the admins. Microsoft releases security updates constantly, sometimes up to five in one week.
The elite laboratory, where scientists perform research in areas ranging from particle physics to life sciences (and have, over the years, won nine Nobel Prizes), is struggling with the same problem stumping IT pros the world over: how to best deploy security patches for Windows. Today, security is of paramount importance and, for the country's oldest national laboratory, that means plugging "leaks" in the network. Microsoft, in an effort to shore up its complex operating systems and its reputation, has been delivering a steady stream of security patches that is overwhelming IT departments.
"There are too many security holes in Windows, and we need to take care of them," Spindler said in a telephone interview, speaking over the hum in the lab's basketball court-sized server room in the Berkeley hills, above the grounds of the University of California campus. Six air-conditioning units cool 12 rows of servers; Spindler said he was wearing a parka.
David versus Goliath
He and colleagues are testing two software products that will allow admins to remotely and methodically install Windows security patches across the Berkeley Lab network. The first is Software Update Services (SUS), a free application from Microsoft. The second is ZENworks for Desktops 4 from Novell, which plays traffic cop but also can inventory network equipment. Spindler and colleagues began testing the two applications in December and likely will make a decision in February.
One reason Spindler is taking a closer look at SUS and ZENworks is because they allow pros to control the flow of security patches. Lab pros must be able to control patch deployment to avoid, for example, hefty patches clogging network bandwidth, which in turn interrupts the lab's day-to-day work and science experiments.
"A hotfix can hit a snag and delay a few thousand people until someone fixes it," Spindler said.
Not all patches need to be installed, said Karen Christian, founder of In Touch Systems, a consulting firm in San Marcos, Calif. Some Windows patches are more significant than others, and some are not thoroughly tested by Microsoft and can disrupt PCs, she said.
"When people sit down at work, they want to get to work," Christian said. "They don't want to reboot their PC."
Spindler set up SUS on a Windows Server 2003, Release Candidate 2. The server connects to Microsoft's Web site once a day to download the available security patches. Berkeley Lab pros review and approve the patches, and a few dozen users are prompted to begin security updates that typically take less than a minute. Spindler intends to increase the SUS test pool to 50 PCs in February.
Though free, SUS comes with limitations. It only works with Windows 2000 and later versions, can only send out hotfixes -- no service packs or Microsoft Office fixes -- and has no reporting capabilities, Spindler said.
"It is extremely difficult to tell if a machine has been successfully updated," he said.
In addition, Microsoft's SUS instructions are 90 pages long and, at times, are more confusing and esoteric than helpful, he said.
On the other hand, ZENworks, which the lab is testing on 10 PCs, can handle all Windows operating systems back to NT 4.0 and can remotely install security patches without help from users. In addition, ZENworks can inventory software and hardware on the network, a handy feature that would allow the Berkeley Lab pros to track software, software licenses and hardware equipment on the network. However, the rollout of ZENworks would take a month, whereas SUS would take only a few hours.
Though Spindler said ZENworks can "easily replace Microsoft SUS server," he suspects Berkeley Lab will use a combination of the two, with SUS perhaps playing a backup role.
Berkeley Lab must balance the risk of security holes with the cost of plugging them, he said. That ruled out Microsoft's Systems Management Server. It's too expensive, not to mention too complicated, said Spindler, who has viewed demonstrations of the application at Microsoft events.