McAfee error triggers massive manual PC clean-up
Posted on | April 22, 2010 | 4 comments
News reports suggest thousands of Windows PCs in large organizations around the globe were thrown into a fit of rebooting yesterday after McAfee distributed a routine antivirus update carrying an egregious error.
Now each one of those computers will have to be manually cleaned. Affected organizations can expect to expend a minimum of 30 minutes of manual labor per PC to get each one back into working order, security experts say.
“There’s no way to automate the process,” says Amrit Williams, CTO of security management system company Big Fix. “It will take however long it takes to touch each single machine. The companies affected by this could be dealing with this for days or weeks.”
In a blog posting late Wednesday, McAfee executive vice president Barry McPherson said “less than one half of one percent of our enterprise accounts globally” were affected. “McAfee teams are working with the highest priority to support impacted customers,” he says.
The incident unfolded after McAfee somehow classified a well-known, legit Windows operating system file, called svchost.exe, as a malicious program. Svchost.exe has long been a crucial part of the Windows operating system. Without it, a PC cannot be networked with other PCs.
Legit files like, svchost.exe, can get intermingled with the tens of thousands of slightly different variants of malicious programs antivirus researchers cull through each day, says Immunet CEO Oliver Friedrichs. “It doesn’t help that some viruses actually masquerade as svchost.exe, leading to confusion and the submission of the legitimate svchost.exe process for analysis,” says Friedrichs.
But quality assurance testing processes are well developed and most of the time prevents antivirus companies from designating legit files as a “false positive,” that ends up quarantined or scrubbed out. “As for why the false positive was not detected during quality assurance, McAfee will have to answer that,” says Friedrichs. “I can definitely sympathize with McAfee. Nobody wants to have this problem while striving to protect people.”
McAfee declined to answer questions, instead directing reporters to McPherson’s post. Here’s what unfolded at about noon Pacific on Wednesday, 21April2010:
As it does several times each week, McAfee sent updated virus signatures to its corporate clients. This is all part of a time-honored cat-and-mouse game in which hackers create slightly different new versions of viruses, thousands of new variants each day. Antivirus companies compete against each other to be the first to detect the latest variants. They then hustle to create fresh virus “signatures,” then push out these protective signatures to corporate customers.
A standard test — running the update on an in-house Windows PC — should have caught the glitch, says Big Fix CTO Williams. He should know. Williams worked at McAfee through 2001 and says he helped develop basic quality assurance tests for signature updates. “It’s very basic testing, not something weird or intricate,” says Williams. “The fact that McAfee didn’t see this as part of normal testing is really shocking.”
Solera Networks, a supplier of network forensics technology, says it helped one large U.S. multi-national company quickly determine that the poisonous update from McAfee threw 50,000 of its PCs into a rebooting frenzy. McAfee advised the company that “remediation time is estimated to be 30 minutes per user, ” says Steve Shillingford, CEO of Solera.
“Estimating $100 per hour, this organization’s lost time alone can be conservatively estimated to cost more than $2.5 million,” says Shillingford. “And that does not factor in lost productivity while users are down.”
Security experts say false positives are impossible to completely eliminate in the frenetic cat-and-mouse world of antivirus protection. McAfee’s gaffe suggests traditional antivirus signature protection may be at its limits, says Ashar Aziz founder and CEO of network security firm FireEye.
“While I’d like to say this is an anomaly, this has happened to several other antivirus vendors and the problem is that antivirus is an antiquated technology that is requiring them to literally process tens of thousands of malware daily,” says Aziz. “What we are seeing is that this technology framework is collapsing under the weight of maintaining a broken signature approach to security.”
by Byron Acohido
Comments
4 Comments »
RSS feed for comments on this post.
I strongly agree with Aziz’s sentiment that the signature approach to security is becoming outdated and needs a radical overhaul. PCs continue to be weighed down by frequent, bloated updates that hog bandwidth and PC resources. Even with all the daily detections pushed down to the desktop, traditional solutions are not foolproof as evidenced by the McAfee false positive error. Users are paying high costs for the traditional antivirus approach — not only in price, but also in (loss of) PC performance. Light, fast cloud-based solutions (like Immunet) that don’t rely on pushing virus signatures to the desktop can detect false positives within minutes, so that the user impact of any potential error is much more limited and can be addressed immediately as opposed to the hour plus per PC it may take to fix one running a traditional non-cloud antivirus solution. What the current situation makes LOUD and clear though (given all the vocal and angry Twitter chatter about the problem from its affected users) is that it really does matter that every Windows PC runs an effective and updated AntiVirus program. The cost of having no solution OR the wrong solution is too high. I would caution anyone uninstalling McAfee to make sure they get some other protection immediately. There are several fast, free alternatives available, including MSE, Immunet, Panda, Prevx, although Immunet is the only one that can run alongside any traditional or next-generation AV product, which makes it a nice companion.
Comment by Denise Terry — 4/22/2010 @ 1:36 am
Who hasn’t been following the issues with Microsoft’s change in Updates through Virtual connections? I find it a little more comforting that through TechNet you can get AES 256 check-sum signatures for updates and verify. Certainly, these “economical” steps toward updates have shown their teeth… and yet it is still in so many’s inventory of services. Cheers to the cloud’s no-fly zone.
What I find endlessly fascinating about the industry direction toward cloud services is the serious problems with cloud architecture barely realized, and the industry’s slow turn to avoid obstacles in favor of statistical fuel use, wasted in making turns. This would be the same as traveling in your car towards a hill when a deer jumps out in your path and you decide not to give-up velocity, betting the deer will keep moving. Good luck with that…
Comment by David Emel — 4/22/2010 @ 3:50 am
Interesting point about how this relates to perhaps unanticipated soft spots in cloud computing.
Comment by Byron — 4/22/2010 @ 8:40 am
I have bad experience about svchost.exe deleted by auto and my PC need re-installed operating system.
Now I not use McAfee anymore
*sorry my english bad
Comment by jens — 11/24/2010 @ 8:20 am