True application meltdown story: A headbanging anti-virus upgrade

In the first story in our "application meltdown" series, a network administrator describes an anti-virus upgrade that became a year-long tech nightmare.

This is an adventure in technology, one of those disasters that seems will never end.

SearchWindowsManageability is asking IT pros to share their application meltdown stories, and this is the first in a series based on their responses.

When Daniel Foerst recounted his story, he was waiting for a hardware replacement, the most recent development in an 8-month struggle to have his organization's network and virus protection system coexist.

"We've only been wandering around the virus protection desert for 34 weeks now," Foerst said. "I figure we've got about six more weeks to go."

A replacement was promised within a few days, but he's been waiting for weeks. While pacing the floor, he shared the following tech nightmare. (We will follow his progress with at least one more story.)

SPLENDID SALES PITCH

It was a year ago, the summer of 2001, and Foerst, a Microsoft Exchange/Systems & Network Administrator at The Catholic University of America, was negotiating to renew the license for Network Associates' (NA) anti-virus software.

"Throughout the negotiations NA was bombarding us with the splendor of its new hardware virus protection system called the WebShield e500," he said. "This hardware boasted the ability to scan inbound and outbound traffic at the SMTP, FTP, HTTP, and POP protocol levels.

"It was a good sell," Foerst said ruefully.

After viewing a few Web casts, the university agreed to buy the product for "tens of thousands" of dollars because no other product like it existed, Foerst said. An e500 finally arrived in mid-December 2001.

SIMPLE SET UP

In early January 2002, Foerst completed the simple set up but quickly learned the only way to interface with the e500 was through a secure HTTP interface. This wasn't a big deal, but it caused some problems because he had to connect to the e500 first through a cross-over cable to configure the network adapter to recognize the network.

When the configuration was complete, Foerst verified his network settings with those provided in the instruction manual.

"We changed our DNS settings to route all inbound e-mail to the e500 for virus scanning and then to our Exchange mail system, where we run a second layer of virus protection," he said.

In early January 2002, Foerst put the e500 to the test. He figured he'd see various daily viruses poking through. He underestimated.

"We were amazed by all the features this baby had to help us understand how saturated the e-mail traffic was with viruses," he said.

'THE LOOP' DEBUTS

But things soured shortly. "Within two days of putting it online, our mail slowed to a crawl. Mail was looping between the e500 and our Exchange server. To blame Microsoft for this mishap would be too easy. Everything was groovy before the e500.

"Were we too bold in moving ahead with this product? Should we have done more testing than the simple EICAR.COM test with five to 10 people?" Foerst asked himself. His answer: "Yes."

"But did NA know something that we did not? Yes and no," Foerst said. The product was so new even NA tech support was baffled and relayed issues to the design team for a number of hot fixes.

The system worked for three months, but Foerst spent way too much time with NA tech support.

Adding to Foerst technology woes was a lot of red tape. There were many political battles over the grant number cited in the support contract. Also, NA "mistyped our University name on the contract," he said.

THE LOOP RETURNS

Foerst ran another test, arranging for four people to receive mail on a secondary mail domain. The system worked for several weeks before The Loop returned.

NA instructed him to shut a setting that scanned the outbound connection.

"All I cared about was making the blasted thing work and only run Inbound scans for the time being," Foerst said.

Another several weeks sailed by smoothly, and Foerst expanded the test pool to 50 in-house volunteers, and they didn't report any trouble. He was ready to install NA OS upgrade 2.5 when he was ambushed by The Loop.

NA's response: another hot fix, version 8.

WING AND A PRAYER

"We prayed that some higher power would allow us to transcend the Development Environment and enter the promise land of Production," Foerst said. "So, we upgraded to version 2.5 of the OS, and our reward was a full system failure."

The two-processor system experienced CPU lockups on both processors, a situation one technician told Foerst was "literally impossible."

Foerst ordered a replacement, and after two weeks he learned the appropriate department never received his replacement request. At the time of this writing, he was expecting a replacement system any day.

What's going to happen next? Will this nightmare ever end? Stay tuned.

FOR MORE INFORMATION

Share your application meltdown stories with us. Or submit a "true IT blooper," a tale of a mistake you made or witnessed. The top-rated blooper wins a prize! E-mail Editor@searchWindowsManageability.com

Find out how one company nixed application failures.

Dig deeper on Windows Server and Network Security

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchServerVirtualization

SearchCloudComputing

SearchExchange

SearchSQLServer

SearchWinIT

SearchEnterpriseDesktop

SearchVirtualDesktop

Close