Friday, August 27, 2010

Massive Computer Failure In Virginia

Richmond, Va. --

A massive computer failure is crippling Virginia government, knocking out websites, blocking the issuance of driver's licenses, preventing the processing of jobless benefits and delaying welfare payments.

The outage, flaring Wednesday afternoon and expected to disrupt some services through the weekend, is attributed to 228 malfunctioning servers, which supply shared software and applications to clusters of state agency computers.

Twenty-six of more than 80 state agencies were hit by the shutdown, including the office of Gov. Bob McDonnell.

"We're disappointed to have a failure, an outage of this magnitude," Samuel A. Nixon Jr., head of the Virginia Information Technologies Agency, said yesterday. "No matter what you do, it's going to happen on occasion."

The incident is the latest embarrassment for VITA and Northrop Grumman, the company the state hired in 2005 to provide computer and communications services under a $2.3 billion contract -- Virginia's richest-ever privatization deal.

VITA and the firm, whose headquarters was lured to Northern Virginia from California by McDonnell, have quarreled for months over shoddy, expensive service. This past spring, VITA and the company announced a new agreement giving an additional $236 million to Northrop Grumman in return for a pledge of better service.

The Rain family of Lynchburg was hit twice by the computer blackout.

Marc Rain Jr., on his way to Old Dominion University in Norfolk, had lost his driver's license and tried to get it reissued Wednesday at the Department of Motor Vehicles office in Lynchburg, and then again yesterday in Richmond.

"We were dropping him off at college," said Rain's mother, Shelly Rain.

None of DMV's 74 offices could process license applications and may not be able to do so again until Monday, officials said. DMV still is handling other transactions, including vehicle decals and titles, and driving and vehicle records.

With its website inaccessible, thousands of out-of-work Virginians could not file jobless claims with the Virginia Employment Commission.

"Access to our website is down 100 percent," said VEC spokeswoman Joyce Fogg. "So no one can get to our website, not even us."

The Virginia Department of Social Services, which, among other things, manages child-support payments and aid to needy families, reports that the outage is disrupting benefits.

"It appears that some benefit payments will be delayed, but we will know more [today]," said spokeswoman Carla Hill. "We are still assessing and are doing everything we can to get back to normal business processes as quickly as possible."

Nixon, appointed by McDonnell under a new law strengthening gubernatorial control over VITA, said that the shutdown -- apparently the largest for the state since 2007 -- occurred about 3:30 p.m. Wednesday.

Nixon said 228 of 3,600 servers were affected when technicians for EMC, a Northrop Grumman subcontractor, were checking for faulty equipment. Nixon said he believes state computer data are largely intact.

Nixon also said that the interruption was of insufficient magnitude to activate a backup system at a duplicate computer center in Russell County, in Southwest Virginia.

Nixon said it is too early to determine whether Northrop Grumman will be punished financially because of the outage. The latest contract, which extends the company's deal with the state from 10 to 13 years, includes new penalties for poor service.

"It depends on how long the outage remains," Nixon said.

Megan Mitchell, a spokeswoman for Northrop Grumman, said, "Knowledgeable and dedicated staff at the agencies, VITA and Northrop Grumman are working together to respond appropriately to the impacted systems.

She added, "It is our priority to minimize these impacts and restore services as quickly as possible."

However, the incident alarmed legislators already skeptical about the effectiveness of the VITA-Northrop Grumman deal, its rising cost to taxpayers and implications for other privatization ventures.

"It's pretty obvious that Northrop Grumman continues to underperform, and I think it would have been wise for the governor to require quality performance before extending the contract for three years," said Sen. Janet D. Howell, D-Fairfax, a member of the Senate Finance Committee.

"This is a sign that privatization is very complicated and should be entered into with caution."

No comments: