Thoughts On 'efficacy'

[alfred]

All,

 

Over the next few weeks I'll be posting more about our detection strategy and our general position on the subject of detections and reviewing 'efficacy'. I hope to speak to how we measure it, how I think the public should measure such things and in general how I would like to see you measure it. If I'm lucky I'll coax other Immunet staff to post as well. This is a topic we all feel passionately about and time permitting we'll write on it here.

 

Now, onto our detection efficacy. The Fall months, here at Immunet, are being dedicated to increasing our overall efficacy. To us this means increasing our detection rates on threats which are actively in-field in our Community. This does not mean we are trying to excel at gaming reviewers. Please note, the industry spends far more time focusing on the latter than the former. Our goal is to stamp out threats actively wandering about in our user community and this is not a simple task for a couple of reasons. The first reason is, it's not time spent gaming reviewers. Sound confusing? Let me explain. All of you know where the common websites are for listing websites carrying malware, many of you have access to malware sample sharing sites (private and public both) and more of you share samples amongst yourselves. It's likely not a surprise to you that most AV vendors know this too and they focus on it. They crawl those sites, seed the sharing networks and spend time (a lot of time) building out sharing networks (we also work hard on sharing networks to be fair). All of the files they receive as a bounty of this work are typically fast-lined to a set virus definitions. It means that average youtube reviewers and folks from Wilders get threats which standard AV products crush. Well of course they do. This application of energy takes a lot away from innovation. This is not to say the big guys are not innovating, they obviously do, they have large teams and can generally bifurcate their energy and do both things. Ultimately though, they still concentrate on the review/reviewers because it is what, like it or not, drives sales of their products. Regrettably it also means that you have a tougher slog unless you go this route because it's where the public focuses. Immunet is not large and does not have the luxury of doing both so instead we focus on trying to find new ways to detect malware, from the cloud, and push hard so that our innovation will intersect with the need to satisfy external reviewers as well.

 

Having said all of this, our goal is to double our in-field convictions by November 15, 2010. This begs to question - then, what is the raw number of daily convictions we see so we can have a baseline. The below chart shows our detection rates up to September 1.

 

 

Our rates for August put us somewhere around 10,000 convictions in-field a day. Therefore our goal is be consistently over over 20K (hopefully 25K) by November 15, 2010. What will take us there is SPERO and ETHOS being applied more aggressively as the fall progresses. The chart below actually shows SPERO during a trial where we enabled a highly aggressive version for a week.

 

 

 

As you can see, the results are dramatic. This particular tree though is not in general production but instead a more conservative version rolled instead. Current trees are:

 

W32.SPERO.Startpage

W32.SPERO.SillyFDC

W32.Generic-0922

W32.SPERO.Allapple

 

By December we'll have 10 or 12 of these trees into production. The aggressive tree will also come back into production as well after some tuning, likely by Wednesday of next week.

 

As I mentioned earlier, we also are enabling ETHOS this week into a much more aggressive posture. ETHOS is best described a 'generic detection' it's designed to catch whole families or clusters of threats from specific families. This week I will be moving about 1,000,000 such ETHOS detections into production, tonight in fact I will be firing up 100,000 of them. An ETHOS detection from this grouping would look something like this:

 

W32.ETHOS.SEP.02EE7W

 

SEP denotes the month the detection was enabled, the 6 characters after are the actual first 6 characters of the hash for the detection itself.

 

These are not the only things we're doing, by any means, but hopefully this provides a little more insight into our current approach to increasing detection rates in the IMP Free product.

 

al

[buckslayr]

Al, this is good to see.

[Lord Ami]

Looks interesting. Good read!

[Guest Mature]

Well done you guys!

[christhomas]

I truly agree on the gaming reviews thing: The traditional AV know how to keep their users happy by catching and seeding all the samples

 

Revealing these kind of information to users and more keeps the community mantra alive

[Guest tmania]

Interesting read. It also begs to have the average consumer educated about the value of being protected from the threats that their own cohort sees.

[alfred]

Interesting read. It also begs to have the average consumer educated about the value of being protected from the threats that their own cohort sees.

 

 

I agree completely although I am not sure how to approach that. Most people, rightfully so, simply expect their AV to work as designed. The interest level for nuance is low for average users but early adopters and technical recommenders get it and hopefully they can help educate people.

 

al

[markusg]

ok, you say, large vendors spend time in crawling this sites because of reviews and want to sale. but i think you also want to sale your produkt, so i think you must also watch some public websites, you can create automatic tools and they can download the stuff for you, or perhaps you can also get in contakt with site owners and they send you the stuff.

its ok for me to say, you want not to spend to much time in such lists at this moment, but when you can get this malware so easy you have to take it i think.

i also think vendors sharing samples to, perhaps get in contakt with other vendors perhaps this is usefull.

i also think you need perhaps honeypots and so on, all what helps to detect malware before an user is infected.

and you need good heuristics, this is true :-)

[Lord Ami]

Are there any news? I've seen some Spero detections myself. Looks like detections are improving fast

[alfred]

Are there any news? I've seen some Spero detections myself. Looks like detections are improving fast

 

 

Yep. Things are moving quickly. For today through to Tuesday we are rolling out changes to the SPERO chassis so SPERO detections will be up and down (more down than up) over the next few days as we complete this work. ETHOS is also coming along nicely, we are in the process of moving 3.4 million generics into production for ETHOS. This will result in better download protection (which is what ETHOS is designed for).

 

Best,

Al

[JanTom]

Additionally can:

 

-Malware research users. I have several thousand malware

-Adding the scanner Immunet (VirusTotal, virscan, Jotti)

-malware sample sharing with producers

-WildList

-Honeypot's

-Adding on site Immunet "send malware"

[alfred]

Additionally can:

 

-Malware research users. I have several thousand malware

-Adding the scanner Immunet (VirusTotal, virscan, Jotti)

-malware sample sharing with producers

-WildList

-Honeypot's

-Adding on site Immunet "send malware"

 

Hi Jamtom,

 

To your points:

 

1. Malware research users. I have several thousand malware

 

We are always happy to analyze new malware. Please mail me at alfred@immunet.com and we can set something up.

 

2. Adding the scanner Immunet (VirusTotal, virscan, Jotti)

 

We are currently on the waiting list to get added to VT, I hope it happens this year.

 

3. WildList & Honeypot's

 

We do have access to Wildlist samples, I am frankly skeptical that they have any real value to our community. What they see as wild, we typically do not, at least not in real serious volume. We do make heavy use of honeypots for collection.

 

 

4.Adding on site Immunet "send malware"

 

We do have one, it's at:

 

http://www.immunet.com/contact/index.html

 

It's in the drop down menu, it could be better.

 

al

[Pedersen]

 

4.Adding on site Immunet "send malware"

 

We do have one, it's at:

 

http://www.immunet.com/contact/index.html

 

It's in the drop down menu, it could be better.

 

al

 

 

If necessary, we do offer FTP submission. You can be allowed access by contacting me (pm/mail).