Agent.exe Causing System Instability

[Geek4AllSeasons]

Agent.exe has been having hyper-scanning episodes. Every system start-up (actually logon), during software installs and other seemingly random times. CPU usage can be high. A constant 80% to +90% is not uncommon. Adjusting its process priority is blocked.

 

By far the biggest problem is its I/O rate. Constant single to double digit MB's with intermittent bursts well over 100 MB. The highest I've every seen, except for I/O speed benchmarks setting off disk temperature alerts (+130F).

 

It has also blocked windows update by holding locks on system files. The following is an example from the Event Log:

wuauclt (5736) An attempt to delete the file "C:\WINDOWS\SoftwareDistribution\DataStore\Logs\edbtmp.log" failed with system error 5 (0x00000005): "Access is denied. ". The delete file operation will fail with error -1032 (0xfffffbf8).

Unlocker Assistant has reported a lock held by agant.exe

 

Recently agent.exe has been leaking Non-paged Pool memory. It was discovery when my system became unstable. It had been running for several days. Process Explorer reported +90 MB of Non-paged Pool allocated to agent.exe.

 

Restarting the service relieves the problem temporarily. A gui restart times out waiting for the service to shutdown requiring the service to be started manually.

 

Earlier today, agent.exe had +3 MB of NPP allocated. The output from system diagnostics tool run at that time will be emailed to support (+45 MB). At time of this writing the NPP allocation had grown to +9 MB.

 

Contact me if more information is needed. An explicit response/status would be appreciated.

 

david

[ataraxy]

Agent.exe has been having hyper-scanning episodes. Every system start-up (actually logon), during software installs and other seemingly random times. CPU usage can be high. A constant 80% to +90% is not uncommon. Adjusting its process priority is blocked.

 

Based on my experience there isn't much that can be done about this.

 

Have you sent the results of the System Diagnostic Tool? Note that when I used this it saved the output file to my administrator account's desktop so until I logged on as administrator it wasn't clear that anything had happened. Hopefully if enough people report this problem Immunet will be able to figure out what is going on.

[sweidre]

Agent.exe has been having hyper-scanning episodes. Every system start-up (actually logon), during software installs and other seemingly random times. CPU usage can be high. A constant 80% to +90% is not uncommon. Adjusting its process priority is blocked.

By far the biggest problem is its I/O rate. Constant single to double digit MB's with intermittent bursts well over 100 MB. The highest I've every seen, except for I/O speed benchmarks setting off disk temperature alerts (+130F).

It has also blocked windows update by holding locks on system files. The following is an example from the Event Log:

david

Hi Geek4 All Seasons & ataraxy,

Now we see the old serious unsolved problem first noticed by ataraxy in the early April this year: "Agent.exe is constantly writing to disk". (Exists on several old threads!) In addition to this the systemdrive of Geek4AllSeasons is getting too hot. Ataraxy proposes System Diagnostic Tool. I will add another remedy: Install TeamViever in David's PC and in the PC of Orlando. Orlando might then via remote access find the reason! A 3rd possibility is to write an email to support@immunet.com in detail describing the problem!

Cheers,

sweidre

PS. Mods/Admins cannot like ostriches hide their heads in the sand any longer! They must face this annoying problem! It must be solved prior to issuing the enterprise edition of Immunet! DS

[boombastik]

http://forum.immunet.com/index.php?/topic/1164-immunet-302-very-high-cpu-ram-usege/

 

Maybe related to this

[ataraxy]

PS. Mods/Admins cannot like ostriches hide their heads in the sand any longer! They must face this annoying problem! It must be solved prior to issuing the enterprise edition of Immunet! DS

 

I really don't see what the big problem is in tracking down this problem.

 

For me, it started with v3.0.1.6056. The version prior to that worked without problems. Immunet should know what changes they put into v3.0.1.6056, so why not just step through those changes and see when things go wrong. If I knew that Immunet were serious about solving the problem (and that they would provide a very good uninstaller that puts everything back the way it was) I'd be prepared to test the step by step versions. Hopefully others would be prepared to do the same. That seems to be the only way that we'll get to the bottom of this.

[Geek4AllSeasons]

I have a hypothesis which identifies what is causing agent.exe to generate extreme I/O rates for some, but not all users. Fortunately, if proven correct, it's an easy, almost trivial fix. While simple to correct, the problem is a gross misuse of SQLite.

 

Ataraxy's concern was appreciated and his reply very helpful. I had used the System Diagnostic Tool before, but thought it was worth checking out the link he provided. As soon as I learned that log files are SQLite databases, the dreaded "Table Scan" shot to the top of the most likely suspects chart.

 

"history.db" has 1 table, a simple heap, entry sequenced list. The problem is it doesn't have any indices. Most importantly no primary index, which specifies logical table order. If it's a B-Tree or derivative, physical order as well when table pages are contiguous.

 

It appears that SQLite has a pointer in the DB header to the first page of each table. Each page has a link to the next and to the first/ When a row is added it finds the end of the table by following the links on each page.

 

Is any one still awake? I'm putting myself to sleep. :-)

 

The cost of a query is (loosely) measured in DB (database) I/Os Each page access is 1 DB I/O. Table Scan are know to generate combinatorial increases in DB I/O processing a query with 1 or more table joins. In this case the increase is geometric where N = the number of pages allocated to the table.

 

My history.db had 210,402 1 KB pages before tweaking. Each row added to history would produce 210,402 DB I/Os. The magic of B-Trees would create a logarithmic reduction to 3 or 4 DB I/Os to find the table end.

 

After tweaking a page size of 16 KB the number of history.db pages was reduced to 10,928 and database size 174,768 KB. More rows in each page, less slack space. A 94.8% reduction in page count, DB I/Os per row added and an allocated space reduction of 35,634 KB (17%).

 

Indices for historyex.db and Cache.db should be evaluated as well. There maybe significant performance gains possible.

 

Agent.exe started acting up after installing Libre Office. An immediate uninstall/reinstall was the killer.

 

Adding appropriate indices and or adjusting DB page size would require little coding. It should be done tuning production queries.

 

IMO, if proven correct, this is a significant defect. Using the number of users immediately affected, to determine priorities, has limited value.

 

It isn't clear that the non-paged pool and/or system file lock leaks reported in the first post in this thread are database query related. I would guess not. They could cause system problems up to and including lockups and/or crashes without manual intervention. It is possible agent.exe is exhibiting symptoms caused by other software. Prevx and Online Armor are installed. Both have been very stable and unobtrusive.

 

I have invested significant effort, out of a desire and sense of obligation to contribute. Continuing would be a waste of time without an indication of interest from the project team.

 

Other open source projects face similar challenges. Development process support and remote monitoring/diagnostics are areas I have been working toward offering dynamic, proactive and scalable contributions.

 

david

[Guest OrlandoP]

Hi,

 

Please generate a file with the System Diagnostic Tool and send it with these informations to support at immunet.com

 

Orlando

[Geek4AllSeasons]

Hi,

 

Please generate a file with the System Diagnostic Tool and send it with these informations to support at immunet.com

 

Orlando

I ran the System Diagnostic Tool immediately. At 45 MBs it is too large to be emailed. Details to send it to support are being worked out with Pedersen.

 

There are 3 additional SDT files generated, but not submitted in connection with issues reported in other threads, They will be submitted as well. Snapshots taken over time may be useful.

[sweidre]

I ran the System Diagnostic Tool immediately. At 45 MBs it is too large to be emailed. Details to send it to support are being worked out with Pedersen.

Hi,

The huge size 45 MB was expected and cannot of course be emailed! Using P2P is a way but takes long time to transfer! Still, I think, Team Viewer is the best way! But, Pedersen used to fix problems easily, I know! It's very good, that mods & admins have finally realized this problem now!!

Cheers,

sweidre

PS. Go ahead! Do not let go! DS

[Geek4AllSeasons]

Team Viewer is the best way! But, Pedersen used to fix problems easily, I know! It's very good, that mods & admins have finally realized this problem now!!

Cheers,

sweidre

PS. Go ahead! Do not let go! DS

Remote sessions/assistance is built into Windows. Not certain about secure connections, but would be surprised if it wasn't the default and/or an option,

 

I am working out details with Petersen. Worst case the SDT files can be put on a server for support to download.

 

Assuming my hypothesis is proven correct eliminating the immediate DB I/O rate problems/symptoms would be a by product. Implicit policy issues, history retention/archiving and database maintenance procedures for example, require consideration of requirements/options and additional, but not much, more effort to implement.

 

The non-paged pool and system file lock leaks observed are likely separate and will be challenging to resolve. I have a large collection of debugging/diagnostic tools and system utilities available. Some are used to analyze Day 0 malware, I can collect a wide range of data/information if I know what would be helpful.

 

IMO remote sessions are best used for one off, time critical, and the gnarliest situations. Their drawbacks are being reactive and lack of scalability. There's more magic in the "Cloud" than instantaneous distribution of Day 0 threat signatures.

[Guest OrlandoP]

Please upload it on a mediafire.com, for example, and then send an email at support with the link.

 

Orlando

[Geek4AllSeasons]

Please upload it on a mediafire.com, for example, and then send an email at support with the link.

 

Orlando

As requested 4 files have been uploaded to MediaFire and an email sent to support. For each file there is a cross-reference link to a thread describing issues experienced at or near the time the file was created or a short description which is described in a subsequent thread.

 

There appears to be some moderately good news and some not-so-good news.

 

Since tweaking the history.db page, I/O rates have been substantially lower. Process Explorer reports (5 sec,update) total I/O for agent.exe a constant 7.8/9 KB. It jumps to +1 MB to +7 MB when updating history.db for a period of time. During an install or similar system activity the I/O rate would be proportionately and significantly higher.

 

The not-so-good news is in an approximately 24 hour period non-paged pool allocated to agent.exe went from an initially steady 24KB to +2,500 KB. It now is clicking up 1 KB every 10 secs (every other Process Explorer update). The "negative" editorial comment stems from invalidation of a basis to speculate about a possible connection been leaking NPP and abnormally high I/O rates.

 

I run windows on legacy hardware, XP SP3 (fully up to day) on the following platform: P4 2 GHz, 1 GB DDR 266 MHz, PCI 133 MHz.

 

I've encountered what appears to be race condition/un-handled (lost) event problems. The speculation was that very high I/O rates could conceivably cause NPP and/or system file locks to leak. However, to my knowledge I/O rates have not been high (actually rather low) in the past 24 hours. It seems that further investigation will be needed to find the cause.

[dallas7]

Immunet Free v3.0.3.6870

 

This may or may not be strictly related to the issue(s) under discussion here, but it's as good a thread as any...

 

As reported by Task Manager, agent.exe eventually begins utilizing 400,000-600,000 K of VM Size. I close iptray and any attempt to stop the service returns a failure message (using the services mmc or the net stop command).

 

I have a static 2048kb pagefile so a rampant VM size of that magnitude is causing problems. I should note that when agent isn't problematic, the VM Size is about 120,000 K and stopping the service has always been rapid and successful.

 

I see this behavior on two similarly configured XP SP3 32-bit systems though the hardware platforms are significantly dissimilar. It's been annoying enough for me to have stopped running Immunet as of about three weeks ago.

 

I'll give it another shot with the next upgrade... 3.0.4 or 3.1?

 

Cheers.

[Geek4AllSeasons]

Immunet Free v3.0.3.6870

 

This may or may not be strictly related to the issue(s) under discussion here, but it's as good a thread as any...

It possible, maybe likely your issue is the same or closely related. There aren't enough details to be certain.

 

As reported by Task Manager, agent.exe eventually begins utilizing 400,000-600,000 K of VM Size.

IMO the Task Manager doesn't have enough of the right details for a clear understanding of process memory usage, Particularly, the impact on system responsiveness and/or throughput.

 

VM size includes all modules mapped into a processes address space. Many system modules (dlls) that provide services and/or desktop integration are loaded into a process. Security is a biggie. File Unlocker Assistant is another example.

 

Process Explorer, the latest versions in particular, provide the better information. Total Working Set size and Total Private Bytes are good measures process memory consumption. PE breakdown those totals into Private Working Set, Shareable and Shared components.

 

I close iptray and any attempt to stop the service returns a failure message (using the services mmc or the net stop command).

I'm seeing agent.exe is stopping, but doesn't report that back to the issuer of stop control. The GUI waits for 10 to 20 seconds after the process has ended before reporting the failure to respond.

 

I have a static 2048kb pagefile so a rampant VM size of that magnitude is causing problems. I should note that when agent isn't problematic, the VM Size is about 120,000 K and stopping the service has always been rapid and successful.

I agree that 400 MB to 600 MB is a problem, but it may not be the immediate cause of degraded system performance. The availability of system resources to other processes is more significant than the amount of resources used. Simple how much headroom is available.

 

The VM size may have been as large when agent.exe was misbehaving on my system. I didn't pay attention, because agent.exe eating up so much I/O and more importantly Non-Paged Pool, the system was on the verge of crashing. The I/O rates were high enough to overwhelm the system cache driving the disk hard enough to raise its temperature over 130F. The Non-Paged pool is kernel address space locked in to physical memory used by device drives. If it is exhausted the system locks up.

 

I see this behavior on two similarly configured XP SP3 32-bit systems though the hardware platforms are significantly dissimilar,

That in it self is a useful data point. I it were not too inconvenient it would be important to know if non-paged pool was leaking.

 

On my system agent.exe also cased Microsoft Update to fail by locking a file that it wanted to delete.

[dallas7]

IMO the Task Manager doesn't have enough of the right details for a clear understanding of process memory usage...

So what?

 

I agree that 400 MB to 600 MB is a problem...

Well, thank goodness for that.

[Geek4AllSeasons]

I was forced to remove IP from my system after a clean install of 3.03 as requested by support.

 

The first and every boot after installing 3.03 displayed a Blue Screen. The message displayed said windows stopped a driver from corrupting the stack.

 

It's a waste of time:

• attempting to diagnose a continuing stream of problems without the active participation/direction of Immunet support/developers.
• sending SDT files into a black hole
• installing beta quality (at best) software being passed off as releases
• investing time and effort in an attempt to help IP realize its potential when Immunet shows no regard for problems experienced by users.

One can only hope Immunet or SourceFire management recognizes that there are problems and takes corrective action.

[Мира_72]

I did find a file called sm but there was no .exe after it. Could there be another? And how do I tag something with admin rights? Im just not that bright on PCs.

 

thanks so much for you help though

[Guest OrlandoP]

I did find a file called sm but there was no .exe after it. Could there be another? And how do I tag something with admin rights? Im just not that bright on PCs.

 

thanks so much for you help though

 

First of all i suggest you tu read our General Rules, so links that direct you to another site are prohibited.

 

sm.exe if you have got XP may be a file system, anyway I suggest you a complete scan.

 

Orlando