The stress many systems administrators associate with Microsoft's monthly Patch Tuesday update could ratchet up...
several notches, now that the company had made its patches an all-or-nothing affair.
In a move that's sure to present more challenges to systems administrators, Microsoft has changed its system for patches and moved to a "rollup" model for several operating systems, which include Windows Server 2008 R2, Windows Server 2012 and Windows Server 2012 R2.
Rather than issue a number of updates that can be selectively installed, the Microsoft patch rollup model encompasses a single update that contains multiple fixes. This aligns older operating systems with the "Windows as a Service" cumulative update model in Windows 10 which receives multiple updates as a single package. This new model also includes Windows 7 and Windows 8.1.
Before this change, some administrators would choose which patches to apply, which allowed them to avoid updates that could cause disruptions. But this is no longer a choice with Microsoft's new all-or-nothing approach.
Now, when administrators apply the rollup, if the update doesn't work across the entire IT base then the administators will have to wait until Microsoft issues a fix. "It is mildly concerning, because if there is a zero-day [vulnerability] in there that means that the one [patch] that didn't work could cause [administrators] to leave a security hole open until they can get a version that works well for them," said Wes Miller, research analyst at Directions on Microsoft.
This change is a long time coming, however, and will benefit both Microsoft and the systems administrator because it eliminates the element of uncertainty and the need to account for a wide number of variations, said Peter Christy, director of network research at the 451 Group.
New updates, new schedules and other variations
On Oct. 11 Patch Tuesday, Microsoft will issue several different rollups at different days each month:
- The security-only quality update: A single update with the new security patches for the current month that will only be published to Windows Server Update Services (WSUS). It will be released on the second Tuesday of the month, or what Microsoft calls a B week schedule.
- The security monthly quality rollup: Microsoft also calls this a monthly rollup, which includes the new security fixes for the current month and security fixes from the previous monthly rollups. Microsoft will issue this update to Windows Update, WSUS, and the Windows Update Catalog, which is also on the B week schedule.
- The preview of the monthly quality rollup: Microsoft also calls this the preview rollup. This release contains the new non-security fixes to come in the next monthly rollup and fixes from all previous monthly rollups. Microsoft will release this rollup to WSUS, Windows Update and the Windows Update Catalog. This rollup will come on the third Tuesday of the month, or what Microsoft calls a C week schedule.
- Security and reliability updates will be bundled to the .NET Framework in a separate monthly rollup and a security-only update on Microsoft Update Catalog and Windows Server Update Services every month.
- Fixes for the Internet Explorer version supported in each operating system will come from the security-only and monthly rollups.
Microsoft will release separate updates each month for a variety of reasons, such as out-of-band security fixes and time-zone changes for Daylight Savings Time, according to a blog post by Michael Niehaus, director of product marketing at Microsoft. Many of these will be included in the next monthly rollup, although some will remain separate, such as updates to Microsoft Office, Adobe Flash and Silverlight.
The changes will help Microsoft "improve the reliability and quality of our updates," Niehaus wrote.
Administrators lose some discretion
Adam Fowler, an IT operations manager with a law firm in Australia, said he's seen more problems with Microsoft's security updates in the last year and, as a result, has not applied updates on the day they are released.
That said, he doesn't expect the new cumulative update model to affect his update routine. His current testing process is to wait a few weeks to see if other administrators report problems with the latest patches. Then he uses WSUS to target certain clients, place them in a group and apply the patches, then monitor that group for any issues. If the pilot machines run without trouble, then he applies the patches to the production environment.
"The biggest problem I foresee is that if one of the recommended updates breaks something for us but isn't a widespread problem, we won't ever be able to push out cumulative updates again since they'll always contain that single patch we don't want," Fowler said.
It's unclear if Microsoft plans to release a tool or disclose another way to pull an individual patch. In the Oct. 7 blog post, Niehaus wrote that if a problem occurs with an update, the recommended action is to delay the deployment and contact Microsoft support. Depending on what support suggests, administrators will have several options: roll back the update on affected machines while Microsoft investigates the issue, install other updates known to resolve the issue or work with the publisher (ISV) for the affected application.
Clock starts to tick when Microsoft releases patches
Harjit Dhaliwal, senior systems administrator at University of Vermont, said his patch process starts when he scours numerous online sources -- Twitter, listservs, technical forums -- to see if any updates have disrupted systems that are similar to ones he manages.
"One wrong patch and you bring down the entire infrastructure," Dhaliwal said.
Sometimes an application vendor needs to make an additional fix after an update from Microsoft, which leaves administrators in a technical conundrum if a resolution doesn't arrive until days -- or months -- after Microsoft releases a patch.
Dhaliwal said a recent outage affected hundreds of machines in the school's VMware-based virtual desktop infrastructure (VDI) system after the IT department applied a Microsoft update to his client operating system template. He bounced between VMware and Microsoft support as each blamed the other for the disruption. After the companies passed logs back and forth, VMware admitted it needed to issue an update to its VDI application platform -- but because the update would not be available until the following quarter, Dhaliwal said the school had to remove the patch.
Whether Dhaliwal applies patches immediately or waits until known issues are fixed, Patch Tuesday is a source of stress.
"Patches come out because there is a known issue, a vulnerability or some security issue. If you don't patch, then you're putting yourself at risk because people know about it and they're going to try to do a hack," Dhaliwal said.
451 Group's Christy said he sympathizes with systems administrators who feel responsible to make intelligent decisions on what patches should and shouldn't be installed. "I understand why culturally this kind of change is painful," he said. "But I think if you asked security or software liability experts, they would be pretty uniform in agreement that [the cumulative update] approach would be the way to go."
For more information about the changes to Patch Tuesday, check the TechNet blog here.