This article originally appeared on SearchSecurity.com.
When an IT environment spans the vast, complex landscape of state government, it's nearly impossible to stop every threat from cyberspace and the physical world.
Take the state of Delware's IT environment, for example, where a network comprised mostly of Windows machines serves the needs of 49,000 employees, in addition to thousands of citizens and a variety of different agencies. It would be bad enough if an online attack against one agency rippled across the state network or if a disaster in one municipality disrupted the flow of digital resources throughout Delaware, but what if the state network were engulfed by a confluence of incidents all at the same time?
As far as state officials at the Delaware Department of Technology and Information (DTI) are concerned, the best way to prepare for chaos is to spring the worst-case scenario on employees on a regular basis by way of disaster drills.
The likely threat
DTI held the first such "tabletop exercise" last October, along with the Delaware State Police, the Federal Bureau of Investigation (FBI) and the Delaware Emergency Management Agency (DEMA). Other drills are happening "all the time" within the central IT department, according to Elayne Starkey, the State of Delaware's CTO.
The ultimate goal is to get everyone thinking about what they'd do in the face of a massive security incident, she said, so they can spring into action if ever faced with a real one.
"While the next big exercise is in October," Starkey said, "I want to do smaller drills with the state police, FBI and DEMA more often and expand it to include drills that are coordinated with municipal governments, regional governments and neighboring state governments,".
Lisa Wragg, the state's disaster recovery coordinator, said the last exercise involved 80 participants from approximately 10 agencies, two school districts, two universities and a private-sector financial institution. The state hired Wayne, Pa.-based SunGard Availability Services -- a disaster continuity procedure specialist -- as a consultant during the planning stages and the actual simulation.
The planners thought of doing a simulation involving a major terrorist attack. In the end, Wragg said, they opted for an exercise based on what the state considers 70-80% of its risk: the insider threat.
"We looked at the kinds of problems that could be caused by malicious insiders," Wragg said, "but we also decided to focus on what you do if a bunch of things happen at once -- a power failure, a massive virus infection and a denial-of-service attack."
Who's in charge?
During the exercise, participants were placed into groups based on their roles and responsibilities, sitting together at large tables.
Each group worked through the exercise's three stages: pre-event preparation, event detection and finally response and recovery. Starkey and Wragg said that the room was constantly buzzing with debate and activity, and that interaction among groups increased as the exercise progressed. The importance of communication between agencies became evident when one group unilaterally decided to shut down the network to deal with the threat at hand, a move that led to confusion among the other groups.
"Coming from a technical agency, I was very surprised by that decision," Wragg said. "But it was quickly flagged as a problem and resolved. It was definitely an 'a-ha' moment that helped illustrate why cybersecurity is so complex."
Starkey said it also raised a key question: When it comes to a cybersecurity incident, who's in charge?
The empty table
Another key moment in the exercise came when participants decided to establish a command center to address the evolving situation. A representative from each group was tapped to participate in the command center.
Wragg said she and her team had earmarked an empty table for a command center and was pleased when participants saw the need to create one. But in hindsight, she said, it became clear that the command center should have been set up a lot more quickly than it was.
In addition, Wragg said the following points were identified during the exercise as areas requiring an improved response:
The best way to work out these kinks, Starkey and Wragg said, is tokeep holding drills large and small and tweaking security policies to take the lessons into account.