"It isn't enough any more to react to performance problems," says Jerry Rosenberg, co-founder of consultants SRM Associates in Plainview, N.Y. "It's not just your customer service folks complaining that your IT infrastructure has a problem. Now it's really your customer. And your customer isn't going to make a buy decision if you aren't giving him the service he wants. And if he leaves, there are lots of forces, lots of competition,...
to keep him from coming back."
That's a dire scenario, but in today's e-commerce world it's the way the world works. However, there are other ways you can tell that you need performance management for your IT shop.
It may be that your users begin to gripe that everything is too slow. Or it may be that you start to get some read failures on storage systems. Or you notice that you're getting an unusually high number of crashes. Or you find that page-serving times on your Web server are starting to stretch out. Or...
Well, there is any number of reasons that you may need some performance management. And the great thing about this topic is, believe it or not, once you have solved your current computer system performance problem, you'll still need performance management, because Mr. Murphy's famous law is ever operative, and the best way to thwart that law is to be proactive rather than waiting for whatever will inevitably go wrong.
Rosenberg, who's been around the performance management business for some 30 years, working with mainframes, Windows NT, Unix and so forth, says it's not just that we're all living on the Web these days. In the old days, if things slowed down, you just threw more money at the problem, particularly as systems got smaller. With inexpensive computers, it seemed cheaper to buy a newer, bigger computer than to worry about managing what you had. But no more.
"Things have gotten more complex," Rosenberg says. The systems interact to a greater extent. Today, you need to be proactive." Performance management lets you get into that proactive mode.
Thomas Beretvas, president of Beretvas Performance Consultants in Draper, Utah, says performance management is a continuing process in which you perform several steps. First, you have to set some performance goals. Second, you need to gather data on the performance of your system. Third, you then compare the data with the goals, and, finally, you adjust the system so that the performance will more likely meet the goal you have set.
Then, you can do the whole thing over again, because, as we said about Mr. Murphy ...
It's like a feedback loop, a structure in which the output of a system is compared with the input, which is a control signal representing the desired state of the output. The difference between the two signals drives the system being controlled to bring the output to the desired state.
Think of it this way. When you drive a car, you know what the desired state of the system (the car) is. You make adjustments (brake, gas, steering, etc.) to make the difference between the present state of the system and the desired state zero. A red light ahead means the velocity of the car should become zero, so you apply the brake to bring the speed to zero.
It's the same with performance management of computer systems: You observe the state of the system, compare it with the desired state, and make modifications to the system (control it) to make the actual state agree with the desired state.
How do you do that?
That answer could take megabits of data. Beretvas got the basic idea down in just the few words we used to describe the craft, if you will, of performance management in computer systems.
Beretvas is an expert in direct-access storage devices (DASD) working with IBM mainframes. A 28-year veteran of IBM, he concentrated on system performance, with an emphasis on MVS and more specifically on DASD performance on the mainframe.
He says there are numerous ways that indicate you are not getting optimum performance. "But the most obvious," he says, "is that you find that your response times are bad, or that your throughput is bad. Now this may not be because the DASD is bad, but it can be."
But, recall our steps in performance management. You don't know if your response times are bad unless you have some standard against which to compare, so you have to set goals for response times, or throughput, or whatever. Then you have to come up with some way to gather some data to compare against your goals.
That process is an art, Rosenberg says. "It all gets into service-level agreements, he says, because if you don't have some idea of what performance you expect, then when a user says things are too slow, you'll say, 'Well, what's too slow?' and then..."
Here's where performance-management product vendors come in. Most system vendors have some performance monitoring tools. IBM, Beretvas says, has one that runs under MVS (and zOS, for that matter) called Resource Measurement Facility (RMF), available with the OS. Windows 2000 has its Performance Monitor, which lets you gather data for analysis to see what you have going on in your systems.
Tools like those will give you lots of data. All you have to do is look at a log file from a Web server to get an idea of the mountain of data that you will have to deal with once you start collecting it. Faced with this plethora of performance data, how do you make any kind of sense out of it?
"You can be hip deep in data and have lots of numbers," says Ed Wood, a consulting engineer at Candle Corp., in El Segundo, Calif. "But you have to make sense of them somehow." Candle sells performance management tools and also provides consulting services for clients who want to get a handle on their system performance. Its performance management product is called Omegamon.
Clearly, you need to get the data set down to a manageable size, says John DeRosa, a senior software engineer at The Information Systems Manager Inc., in Bethlehem, Pa. Once you know there's a problem, he says, then there are sources of data that will document the problem. But he says now you face the next problem: You have to determine whether you have a real problem now, or if it's coming some time in the future. "The real power of our suite of tools," he says, "is the trending we do. For example, you can ask, 'Six months out, will our network be bogged down? Will we be out of storage space in three months? Will our CPU utilization be way up?' "
DeRosa's suite of tools is called PERFMAN, and it can look into a whole set of parameters for Windows NT-based networks. It gathers RMON data that's available for on the network and then, using statistical methods, it reduces the data to a form that you can use to answer questions about your performance such as we have posed. The hope, moreover, is that the data you gather and the analysis you do will mean that you can forestall a problem that would have come up six months in the future.
Can you gin up a performance management solution of your own? Maybe, but it depends on the capabilities you have in your shop. And, according to Wood, the expertise you need isn't in the IT shops the way it used to be. "A lot of custom applications may touch three or four operating systems for one transaction," he says. "It may talk to a Unix front end, from an NT workstation, and go to a mainframe running DB2 for its data lookups. You may have people that are world class in CICS or MVS but don't understand the other guy's patch."
And Rosenberg notes that knowing how to benchmark, and which counters to check in your management software, is something of an art.
But whether you want to roll your own, or look for some packaged solution, the time is over when you could just get a faster server and bigger disks. Managing things, prioritizing applications, spreading the workload, all with the help of the management system, will help you save money, make you life easier, and, even, maybe keep you in business.