Jump to content


Photo
* * * * * 1 votes

Efficacy Testers Needed


  • Please log in to reply
20 replies to this topic

#1 Alfred

Alfred

    Advanced Member

  • Administrators
  • 401 posts

Posted 31 August 2010 - 04:07 PM

All,

I am in the process of testing some rather aggressive new detection regimes in both our SPERO and ETHOS engines. If you have a VM and are willing to help test for True and False positives, your welcome to come on board. Mail me at alfred@immunet.com.

The caveat here is that this is early in the testing cycle and the goal is to find FP and TP rates both so unpleasant things are likely to happen to the test environment you are using, hence the requirement for a VM is a must.

Best,
Alfred

#2 Guest_Orlando_*

Guest_Orlando_*
  • Guests

Posted 31 August 2010 - 07:36 PM

All,

I am in the process of testing some rather aggressive new detection regimes in both our SPERO and ETHOS engines. If you have a VM and are willing to help test for True and False positives, your welcome to come on board. Mail me at alfred@immunet.com.

The caveat here is that this is early in the testing cycle and the goal is to find FP and TP rates both so unpleasant things are likely to happen to the test environment you are using, hence the requirement for a VM is a must.

Best,
Alfred


You can count on my help. I'm currently playing Immunet with some niche antivirus (now Zillya!). Although the tests are not desired, I will send the problems encountered with other antivirus software (if any). I will send some false positives, etc..

Regards,
Orlando

#3 Alfred

Alfred

    Advanced Member

  • Administrators
  • 401 posts

Posted 31 August 2010 - 08:05 PM

You can count on my help. I'm currently playing Immunet with some niche antivirus (now Zilly!). Although the tests are not desired, I will send the problems encountered with other antivirus software (if any). I will send some false positives, etc..

Regards,
Orlando


Orlando,

Do you have a virtual machine set up?

al

#4 Guest_Orlando_*

Guest_Orlando_*
  • Guests

Posted 31 August 2010 - 08:24 PM

Orlando,

Do you have a virtual machine set up?

al


In my personal computer I haven't the virtual machine, but in the test computer there is the virtual machine to do any testing. Currently, however, has merged the CPU (computer test). The computer with the virtual machine will be operational by Monday.

Regards,
Orlando

#5 Alfred

Alfred

    Advanced Member

  • Administrators
  • 401 posts

Posted 31 August 2010 - 10:13 PM

In my personal computer I haven't the virtual machine, but in the test computer there is the virtual machine to do any testing. Currently, however, has merged the CPU (computer test). The computer with the virtual machine will be operational by Monday.

Regards,
Orlando


OK, Great, just let me know.

#6 Alfred

Alfred

    Advanced Member

  • Administrators
  • 401 posts

Posted 01 September 2010 - 10:33 PM

All,

I am in the process of testing some rather aggressive new detection regimes in both our SPERO and ETHOS engines. If you have a VM and are willing to help test for True and False positives, your welcome to come on board. Mail me at alfred@immunet.com.

The caveat here is that this is early in the testing cycle and the goal is to find FP and TP rates both so unpleasant things are likely to happen to the test environment you are using, hence the requirement for a VM is a must.

Best,
Alfred


So far I am testing the Generic detect tree (lovingly named W32.SPERO.Vacuum.P1) and the results are very promising.

If you are interested in testing - here is a quick cheat sheet:



1) Install new version from http://www.immunet.com/free .

2) Open cmd window as Administrator.

3) Stop the immunet protect service "sc stop immunetprotect"

4) From the same cmd window run ' notepad "c:\Program Files\Immunet
Protect\1.0.22\global.xml" '

5) To the <cloud> node add the following sub-nodes

<server>
b.immunet.com
</server>
<host>
cloud14.immunet.com
</host>

Typically I add them right before the </cloud> tag.

 5b) Find the agent\submission\http_upload_server key and set it's value to cloud14.immunet.com

5c) Save global.xml

6) Next delete the local.xml file from the immunetprotect root directory.

7) Run agent -r from the version directory.

8) Run agent -r from the version directory (Because of a bug)

9) Restart the agent service "sc start immunetprotect"

When you are introducing malware to your system, if it's in bulk you will want to shut off IMP to get it onto the system to be scanned. You can use the sc stop immunetprotect command for this and the sc start command to get it going again.

For this part of the process we are testing a Generic sub-engine and a Family specific engine. The two detection names you are looking to see are:

1. W32.SPERO.Vacuum.P1
2. W32.SPERO.Allaple

Each test should generally be done in this order when testing for SPERO

1. Bring malware set on (with IMP disabled), fire up IMP and set SPERO in the settings to OFF.

2. Scan and note your conviction count.

3. Turn of IMP and issue this command:

c:\ del c:\program files\immunet protect\cache.db

4. Move the same test set back onto the system and fire up IMP again.

5. Enable SPERO and scan. Note the results.

Here is a basic example of what my simple testing looks like:

Test Set 1

1138 pieces of Malware
IMP Without SPERO - 586 Convictions
IMP With SPERO 971 Convictions

As for clean files, this is fairly simple as well. Install your favorite programs and see if we end up with any false detections. If so, please let me know what they are and where I might find them.
  • Graham Perrin likes this

#7 Guest_Orlando_*

Guest_Orlando_*
  • Guests

Posted 02 September 2010 - 07:45 AM

So far I am testing the Generic detect tree (lovingly named W32.SPERO.Vacuum.P1) and the results are very promising.

If you are interested in testing - here is a quick cheat sheet:



1) Install new version from http://www.immunet.com/free .

2) Open cmd window as Administrator.

3) Stop the immunet protect service "sc stop immunetprotect"

4) From the same cmd window run ' notepad "c:\Program Files\Immunet
Protect\1.0.22\global.xml" '

5) To the <cloud> node add the following sub-nodes

<server>
b.immunet.com
</server>
<host>
cloud14.immunet.com
</host>

Typically I add them right before the </cloud> tag.

 5b) Find the agent\submission\http_upload_server key and set it's value to cloud14.immunet.com

5c) Save global.xml

6) Next delete the local.xml file from the immunetprotect root directory.

7) Run agent -r from the version directory.

8) Run agent -r from the version directory (Because of a bug)

9) Restart the agent service "sc start immunetprotect"

When you are introducing malware to your system, if it's in bulk you will want to shut off IMP to get it onto the system to be scanned. You can use the sc stop immunetprotect command for this and the sc start command to get it going again.

For this part of the process we are testing a Generic sub-engine and a Family specific engine. The two detection names you are looking to see are:

1. W32.SPERO.Vacuum.P1
2. W32.SPERO.Allaple

Each test should generally be done in this order when testing for SPERO

1. Bring malware set on (with IMP disabled), fire up IMP and set SPERO in the settings to OFF.

2. Scan and note your conviction count.

3. Turn of IMP and issue this command:

c:\ del c:\program files\immunet protect\cache.db

4. Move the same test set back onto the system and fire up IMP again.

5. Enable SPERO and scan. Note the results.

Here is a basic example of what my simple testing looks like:

Test Set 1

1138 pieces of Malware
IMP Without SPERO - 586 Convictions
IMP With SPERO 971 Convictions

As for clean files, this is fairly simple as well. Install your favorite programs and see if we end up with any false detections. If so, please let me know what they are and where I might find them.


Bad news for me, the CPU will come in a week (due to late shipment), the coputer test will be operational next two weeks. I'll inform you when arrive.

Regards,
Orlando

#8 Chris Thomas

Chris Thomas

    Advanced Member

  • Immunet Insiders
  • PipPipPip
  • 46 posts
  • LocationUnited Arab Emirates

Posted 06 September 2010 - 09:46 AM

I guess, I will test this sometime later when there won't be need for a virtual environment. But it sure reads promising

#9 Shaoran

Shaoran

    Advanced Member

  • Immunet Insiders
  • PipPipPip
  • 35 posts

Posted 08 September 2010 - 10:52 AM

Hi,

For me it was not "c:\Program Files\Immunet Protect\1.0.22\global.xml" but "c:\Program Files\Immunet Protect\2.0.15\global.xml"

So, I try it with 4 malware folders :

2010 : 2903 threats
2009 : 1068 threats
2008 : 297 threats
< 2005 : 5000 threats

without update
2010 : 1292 threats not detected
2009 : 5 threats not detected
2008 : 126 threats not detected
< 2005 : 2520 threats not detected


with update
2010 : 813 threats not detected
2009 : 1 threats not detected
2008 : 124 threats not detected
< 2005 : 2518 threats not detected


So yes, it seems to find more threats, but immunet has still one of the worst score

#10 Alfred

Alfred

    Advanced Member

  • Administrators
  • 401 posts

Posted 08 September 2010 - 05:38 PM

Hi,

For me it was not "c:\Program Files\Immunet Protect\1.0.22\global.xml" but "c:\Program Files\Immunet Protect\2.0.15\global.xml"

So, I try it with 4 malware folders :

2010 : 2903 threats
2009 : 1068 threats
2008 : 297 threats
< 2005 : 5000 threats

without update
2010 : 1292 threats not detected
2009 : 5 threats not detected
2008 : 126 threats not detected
< 2005 : 2520 threats not detected


with update
2010 : 813 threats not detected
2009 : 1 threats not detected
2008 : 124 threats not detected
< 2005 : 2518 threats not detected


So yes, it seems to find more threats, but immunet has still one of the worst score



You need to keep in mind this is a small set of cloud sub engines, we can have dozens. No single one will be a panacea. Also, we will do better on newer malware because we focus on it. Malware from 2008 etc. is not something you should expect our product to detect well, it's also not something people will see in the field with any real frequency. The results on the 2010 and 2009 folders make me very happy.

Thanks a bunch for working with us on this.

al

#11 Shaoran

Shaoran

    Advanced Member

  • Immunet Insiders
  • PipPipPip
  • 35 posts

Posted 08 September 2010 - 05:54 PM

I should precise about the worst score, I was speaking about the first pack (2010) wich is my own pack unlike the others. If you take some free av like avast, antivir, panda cloud or avg, they left something between 100 and 250.

But, for sure, Immunet is still young, it just need to grow :P

#12 Alfred

Alfred

    Advanced Member

  • Administrators
  • 401 posts

Posted 08 September 2010 - 06:23 PM

I should precise about the worst score, I was speaking about the first pack (2010) wich is my own pack unlike the others. If you take some free av like avast, antivir, panda cloud or avg, they left something between 100 and 250.

But, for sure, Immunet is still young, it just need to grow :P



Of course, this is one engine. We are releasing over a dozen in the next two months. This field test is exactly that, just a test. So far the results look fantastic.

al
  • Graham Perrin likes this

#13 buckslayr

buckslayr

    Advanced Member

  • Immunet Insiders
  • PipPipPip
  • 68 posts
  • LocationUSA

Posted 09 September 2010 - 01:15 AM

Of course, this is one engine. We are releasing over a dozen in the next two months. This field test is exactly that, just a test. So far the results look fantastic.

al


Hi Al,
Just a quick question regarding the new engines. At some point in the future with the new engines in place, will there still be a need for the plus version or will the cloud only version give adequate protection as a stand alone?
Immunet Protect 3.0 + AppGuard + ClearCloud DNS

#14 Shaoran

Shaoran

    Advanced Member

  • Immunet Insiders
  • PipPipPip
  • 35 posts

Posted 09 September 2010 - 10:48 AM

Second part of the test, scan of safe files (All my data), there is 5 new false positif, one on BleachBit, one on AdslTV, two on Company of Heroes, one on Dawn of War - Soulstorm (cracked dll only (I hate to insert my cd to play ^^) original file hasn't been detected)


If you want this files, just tell me where upload them :P

#15 Guest_Orlando_*

Guest_Orlando_*
  • Guests

Posted 09 September 2010 - 11:57 AM

Second part of the test, scan of safe files (All my data), there is 5 new false positif, one on BleachBit, one on AdslTV, two on Company of Heroes, one on Dawn of War - Soulstorm (cracked dll only (I hate to insert my cd to play ^^) original file hasn't been detected)


If you want this files, just tell me where upload them :P


You can report false positives in this page: http://www.immunet.c...tact/index.html in the dropdown choose "submit a false positive".

Thanks for the support,
Orlando

#16 Shaoran

Shaoran

    Advanced Member

  • Immunet Insiders
  • PipPipPip
  • 35 posts

Posted 16 September 2010 - 08:20 AM

Hi,

Is this update already added in Immunet without the modifications ?

#17 Alfred

Alfred

    Advanced Member

  • Administrators
  • 401 posts

Posted 16 September 2010 - 10:01 PM

Hi,

Is this update already added in Immunet without the modifications ?



Not sure I understand the question.

#18 Alfred

Alfred

    Advanced Member

  • Administrators
  • 401 posts

Posted 16 September 2010 - 10:03 PM

Second part of the test, scan of safe files (All my data), there is 5 new false positif, one on BleachBit, one on AdslTV, two on Company of Heroes, one on Dawn of War - Soulstorm (cracked dll only (I hate to insert my cd to play ^^) original file hasn't been detected)


If you want this files, just tell me where upload them :P



The easiest way to do this is actually to just roll them back out of quarantine. I get all of those logs and will examine the data there. Sending them (as Orlando suggested) is also quite helpful.

al
  • Graham Perrin likes this

#19 Shaoran

Shaoran

    Advanced Member

  • Immunet Insiders
  • PipPipPip
  • 35 posts

Posted 17 September 2010 - 07:49 AM

Fp have already been fixed. What I means is if you have already sent this update to all Immunet client.
I ask you that because I ran some test recently, and I see that Immunet have a better detection, and there is not any new detection with this test.

#20 Shaoran

Shaoran

    Advanced Member

  • Immunet Insiders
  • PipPipPip
  • 35 posts

Posted 19 September 2010 - 08:48 AM

In fact, I found the answer in another thread, I didn't know that SPERO send detected files to the cloud.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users