Mrg Flashtest 2011

[Woodrow]

They are back with nastier malware then ever, I am sorry to report that Immunet is getting a beating.

 

http://malwareresearchgroup.com/category/malwareproducttesting/

 

But Immunet is far from alone regarding the fails, I get scared when I see what kind of malware are circulating out there.

 

Is there behavior analyses coming in the cloud soon?

 

I see Emsisoft getting saved by there behavior blocker all the time, Mamutu.

 

I would love to see something similar to this in Immunet.

 

Keep the flag high guys!

 

/W

[dallas7]

...Immunet is getting a beating.

So is just about everything else. In last year's tests, Immunet, with the exception of two (or three) results, was pretty much dependent on BitDefender. But that's so misleading...

 

I wouldn't read too much into their zero-day tests, this current series or last years' results. Their reportage has been consistently sloppy with the rare exception of a full report like the recent PUA Test. How they can be so precise in one and so cursory in another is bewildering. They'll consistently announce some new test or report becoming available in so many hours or days. And it'll arrive weeks later. Or not at all. Like last year's Facebook test.

 

In their forum, the previous postings are in the Flash Tests forum as "MRG Flash Tests, Project 001" while the latest is posted under Official Tests as "MRG Flash Tests 2011, ongoing project." How inconsistent is that?

 

They report results for "PC Tools Antivirus." There is no such app - there's a PC Tools Antivrus Free. In the Project 001 pdf wrap-up it's Spyware Doctor with AV. Or is it?

 

There are several variations of "Prevx" depending on which downloader you choose (prevxcsifree.exe or prevxsafeonline.exe). Both install the same exact free application but with different default settings. When choosing to upgrade there are several payment options that effect protection levels. Which one(s) are they testing? Guess.

 

When will they declare the change from Immunet 2.0 to 3.0? We'll probably never find out.

 

Last year they never indicated when the switch was made from AVG 9 to 2011. If at all.

 

Nor will we ever know which component of a particular suite nabbed the nasty. HIPS? Behavior blocker? Scan? White/blacklist? Firewall? DNS? How about Malwarebytes? Free or paid? What's catching all those nasties? The protection module or an on-demand scan or the IP blocker?

 

In defense of being labeled too critical, the subject is highly technical and pertains to the security of not only our expensive systems but our identities. We're not seeing how well one can copy/paste frame sets in some video editors.

 

How much of that lackadaisical attention to detail is present in the actual testing methodology? I'd venture to say quite a bit so I can't pay any credence to MRG's Flash tests other than with casual observation.

 

For anyone who's interested, I've posted up spreadsheets at

https://docs.google.com/leaf?id=0BxamVvlZYmoyNmZhYTQ0MDEtMmY2OS00MzczLTg2MWEtOTU3Yzc2NDNmYjVj&sort=name&layout=list&num=50

compiling Project 001. I'll post up 2011 as I get around to it. Ignore the highlights and borders in the former (they're for me) but they have a purpose in the 2011 sheet - which should be obvious.

 

Cheers!

[Guest chris mrg]

They are back with nastier malware then ever, I am sorry to report that Immunet is getting a beating.

 

http://malwareresearchgroup.com/category/malwareproducttesting/

 

But Immunet is far from alone regarding the fails, I get scared when I see what kind of malware are circulating out there.

 

Is there behavior analyses coming in the cloud soon?

 

I see Emsisoft getting saved by there behavior blocker all the time, Mamutu.

 

I would love to see something similar to this in Immunet.

 

Keep the flag high guys!

 

/W

[Guest Chris MRG]

So is just about everything else. In last year's tests, Immunet, with the exception of two (or three) results, was pretty much dependent on BitDefender. But that's so misleading...

 

I wouldn't read too much into their zero-day tests, this current series or last years' results. Their reportage has been consistently sloppy with the rare exception of a full report like the recent PUA Test. How they can be so precise in one and so cursory in another is bewildering. They'll consistently announce some new test or report becoming available in so many hours or days. And it'll arrive weeks later. Or not at all. Like last year's Facebook test.

 

In their forum, the previous postings are in the Flash Tests forum as "MRG Flash Tests, Project 001" while the latest is posted under Official Tests as "MRG Flash Tests 2011, ongoing project." How inconsistent is that?

 

They report results for "PC Tools Antivirus." There is no such app - there's a PC Tools Antivrus Free. In the Project 001 pdf wrap-up it's Spyware Doctor with AV. Or is it?

 

There are several variations of "Prevx" depending on which downloader you choose (prevxcsifree.exe or prevxsafeonline.exe). Both install the same exact free application but with different default settings. When choosing to upgrade there are several payment options that effect protection levels. Which one(s) are they testing? Guess.

 

When will they declare the change from Immunet 2.0 to 3.0? We'll probably never find out.

 

Last year they never indicated when the switch was made from AVG 9 to 2011. If at all.

 

Nor will we ever know which component of a particular suite nabbed the nasty. HIPS? Behavior blocker? Scan? White/blacklist? Firewall? DNS? How about Malwarebytes? Free or paid? What's catching all those nasties? The protection module or an on-demand scan or the IP blocker?

 

In defense of being labeled too critical, the subject is highly technical and pertains to the security of not only our expensive systems but our identities. We're not seeing how well one can copy/paste frame sets in some video editors.

 

How much of that lackadaisical attention to detail is present in the actual testing methodology? I'd venture to say quite a bit so I can't pay any credence to MRG's Flash tests other than with casual observation.

 

For anyone who's interested, I've posted up spreadsheets at

https://docs.google.com/leaf?id=0BxamVvlZYmoyNmZhYTQ0MDEtMmY2OS00MzczLTg2MWEtOTU3Yzc2NDNmYjVj&sort=name&layout=list&num=50

compiling Project 001. I'll post up 2011 as I get around to it. Ignore the highlights and borders in the former (they're for me) but they have a purpose in the 2011 sheet - which should be obvious.

 

Cheers!

 

The flash tests are not designed to be detailed. We have made it clear that the latest, registered version of each application is used in every test and that default settings are used.

 

Your comment about Prevx is incorrect. Safe Online and Prevx offer exactly the same default level of protection as full products.

 

You comment on delays in publishing tests, such as the Facebook test etc. The facebook test was delayed because at that time, we started discussing this test and the issue of online banking etc with a TV company. As a result of these discussions, the broadcaster decided to create a programme dedicated to these matters and has invited us to appear to cover some of the issues. We delayed this test as we intend to publish it to coincide with the broadcast of the TV programme later this month.

 

The vast majority of work we do is for our clients and never seen by the public. We have to fit the flash tests and all the other reports on our site around the private work we do for vendors. Clearly, the work for our clients takes priority and so, the tests on our site sometimes get delayed.

 

In terms of the methodology for our official tests (eg, the Online Banking tests, PUA tests etc) I believe ours is amongst the most detailed and valid in the business. Take a look at the testimonials on our site,

[alfred]

As ever, it just goes to show you can never rest when building applications to defend against malware. Frankly, as far as keyloggers go the best way to detect them is with run time analysis. Stay tuned, it's going to be a busy year for us.

 

al

[alfred]

The flash tests are not designed to be detailed. We have made it clear that the latest, registered version of each application is used in every test and that default settings are used.

 

Your comment about Prevx is incorrect. Safe Online and Prevx offer exactly the same default level of protection as full products.

 

You comment on delays in publishing tests, such as the Facebook test etc. The facebook test was delayed because at that time, we started discussing this test and the issue of online banking etc with a TV company. As a result of these discussions, the broadcaster decided to create a programme dedicated to these matters and has invited us to appear to cover some of the issues. We delayed this test as we intend to publish it to coincide with the broadcast of the TV programme later this month.

 

The vast majority of work we do is for our clients and never seen by the public. We have to fit the flash tests and all the other reports on our site around the private work we do for vendors. Clearly, the work for our clients takes priority and so, the tests on our site sometimes get delayed.

 

In terms of the methodology for our official tests (eg, the Online Banking tests, PUA tests etc) I believe ours is amongst the most detailed and valid in the business. Take a look at the testimonials on our site,

 

Hey Chris, can you fire over the samples we missed?

 

Cheers,

al

[Guest chris mrg]

Hi Al,

 

Yes, no problem, we will get these to you. We checked the logs on our server and cant see that you have accessed your account - HW problems have been resolved as far as Im aware and there are 50-100K samples / day there. We want to try different mixes of feeds every day to find one with the smallest overlap with your existing sigs.

 

Email Sveta if you need anything.

 

Cheers,

 

Chris

[alfred]

Hi Al,

 

Yes, no problem, we will get these to you. We checked the logs on our server and cant see that you have accessed your account - HW problems have been resolved as far as Im aware and there are 50-100K samples / day there. We want to try different mixes of feeds every day to find one with the smallest overlap with your existing sigs.

 

Email Sveta if you need anything.

 

Cheers,

 

Chris

 

Thx. Chris.

[Woodrow]

?? Wazzup Chris??

[dallas7]

The flash tests ...snip... on our site,

Thanks for responding. If you re-read my OP, I did offer kudos re your recent PUA report. I've been watching your Flash tests from day one last year and you clearly have not "...made it clear that the latest, registered version of each application is used..." If that were the case, you'd easily present that in your daily reports via a few simple keystrokes, i.e. Immunet Protect Pro 2.0 to Immunet Protect Pro 3.0. While I can try to respect approach "The flash tests are not designed to be detailed," there's a difference between detail and attention to detail. And it's beyond reasoning that your clients (like Immunet) think your readers like the astute members of this forum don't care at all if the expense and expertise associated with a new version are made apparent.

 

As far as "default settings," such things can be affected by user choices made during the install. Granted there are some where just whacking the Continue button to Finish is all the input that's needed, most present the user a choice selection before the process continues. And the convoluted offerings and setups of one particular app (read on) pretty much muddies the waters in your "default settings" pond.

 

Sorry to go off-product here Alfred and gang, but Chris opened the door - I am correct about Prevx. I was a long time customer of theirs since the CSI Scanner days. When Prevx/SOL was released there were major discussions in the other forums regarding the motives behind those two "same but different" installers. I don't have a screenshot of the prevxcsifree.exe install, but I assure you it presents a different view of the SafeOnline Trial (prevxsafeonline.exe) attached here. Each presents different flavors of three licensing levels: removal, realtime protection and de-trialing SOL (though the former is irrelevant to this discussion). In the long run, they kindly refunded my upgrade purchase. Yes, in the end either installer can be made offer the same exact protection but the post-install choices involved toward that end are diverse. That is, there is no default!

 

Well, I'm weary of beating a dead horse, but thanks for confirming my observations. And I continue to stand by them. That's my final say here and I'll continue to watch the Flash tests with only casual interest. Forum members and admins, thank you for your patience.

 

Personally, I am grateful for Immunet and Sourcefire and even more so for the decision to market one branded product. The community does not need another Prevx marketing scheme.

 

Cheers.

[dallas7]

In case anyone is watching these tests unfold, Immunet has for the second day in a row received a Passed without a corresponding Passed by BitDefender, the latter having accumulated an inordinate number of Failed as compared to last year's round of tests. This disturbs the long-term bias where Immunet, with rare exceptions, would pass only if BD passed and would fail if BD failed.

 

Immunet detections are reported as "ClamAV" for yesterday and today is "Cloud." This trend is something to cheer about.

 

I've added the detections data to my spreadsheet in another tab. For the google documents uninitiated, the tabs are links are at the bottom of the sheet which are currently "MRG 2011" and "DETECTIONS." As the document grows, so will the number of tabs (links).

 

My spreadsheets for the 2011 running tally and the complete 2010 compilation are at:

https://docs.google.com/leaf?id=0BxamVvlZYmoyNmZhYTQ0MDEtMmY2OS00MzczLTg2MWEtOTU3Yzc2NDNmYjVj&sort=name&layout=list&num=50

 

Cheers.

[dallas7]

Immunet Plus continues to win in whacking the bad stuff for this series of tests even when BitDefender itself fails. While a mid-pack performance persisted from January (when the tests began), it seems mid-March was a turning point. I have confidence Immunet will continue to excel.

 

http://malwareresearchgroup.com/2011/04/mrg-flash-test-4042011/

http://malwareresearchgroup.com/2011/03/mrg-flash-test-3292011/

 

Until Sveta or Chris release the details of the detections for 3/29 also, it can't be said if Tetra (BitDefender) or Immunet/ClamAV (local or cloud) resulted in a Passed but it's apparent the strengthening of Immunet's non-Tetra detection continues to move forward.

 

For the forty tests so far, Immunet is 50/50 pass/fail, same as Avira, Kaspersky, Eset and (almost) G Data - all of which cost more than Plus. Further, it beats out (and clobbers some of) all the other AVs except for Emsisoft, Norton and Vipre.

 

Good work for sure!

 

Reminder: I'm still maintaining a spreadsheet compiling these MRG test data at

https://docs.google.com/leaf?id=0BxamVvlZYmoyNmZhYTQ0MDEtMmY2OS00MzczLTg2MWEtOTU3Yzc2NDNmYjVj&sort=name&layout=list&num=50

 

Cheers.

[dp_tuku]

Looks like ClamAV engine is doing good too, as Bitdefender has missed some samples while Immunet has caught them.

 

Cheers to the signature dev team. Keep on the good work.

 

 

[dallas7]

Immunet Plus continues to rock and roll...

http://malwareresearchgroup.com/category/malwareproducttesting/

http://malwareresearchgroup.com/malware-tests/flash-test-results/

Immunet is now 56/44 Pass/Fail - overall, equal to or better than most (and much better than some of the biggies).

 

See also my spreadsheet of all results since January:

https://docs.google.com/leaf?id=0BxamVvlZYmoyNmZhYTQ0MDEtMmY2OS00MzczLTg2MWEtOTU3Yzc2NDNmYjVj&sort=name&layout=list&num=50

See the "Detections" tabs for Cloud or BitDefender (Tetra) results.

 

I sure wish Free was represented in this test series, but I fully understand the importance of the emphasis on Plus.

 

Cheers!

[dallas7]

I'm going to re-post this to begin a new thread as Malware Research Group Flash Tests 2011, Recurring Updates and Reports Thread as this one began with a different focus.

 

Since January and after 52 zero-day samples, Immunet Plus is approaching a 60% pass metric and is better than Bitdefender itself at 40%. Excluding Norton, Emsisoft and Vipre, Plus now surpasses all other stand-alone AV suites which are multi-layered batteries including firewalls, proprietary behavior blockers, browser extensions and URL/IP white/black lists.

 

http://malwareresearchgroup.com/category/malwareproducttesting/

http://malwareresearchgroup.com/malware-tests/flash-test-results/

 

I've updated my spreadsheet compilation at

https://docs.google.com/leaf?id=0BxamVvlZYmoyNmZhYTQ0MDEtMmY2OS00MzczLTg2MWEtOTU3Yzc2NDNmYjVj&sort=name&layout=list&num=50

 

Cheers!

 

[sweidre]

I'm going to re-post this to begin a new thread as Malware Research Group Flash Tests 2011, Recurring Updates and Reports Thread as this one began with a different focus.

 

Since January and after 52 zero-day samples, Immunet Plus is approaching a 60% pass metric and is better than Bitdefender itself at 40%. Excluding Norton, Emsisoft and Vipre, Plus now surpasses all other stand-alone AV suites which are multi-layered batteries including firewalls, proprietary behavior blockers, browser extensions and URL/IP white/black lists.

 

http://malwareresear...producttesting/

http://malwareresear...h-test-results/

 

I've updated my spreadsheet compilation at

https://docs.google....out=list&num=50

 

Cheers!

 

I have studied the attachment, that was interesting, but one field of great importance is missing: "False Postives". "False Posives" in quarantine (especially System files) are often more dangerous for the computer system than normal "Malwares"!

Cheers,

sweidre