naive questions about thrashing

naive questions about thrashing

[Timothy Miller]

I am running kernel version 2.4.18-26.7.x under Red Hat 7.2.

I wrote a CPU-intensive program which attempts to use over 700 megs of 
RAM on a 512-meg box, therefore it thrashes.

One thing I noticed was that 'top' reported that the kernel ("system") 
was using 68% of the CPU.  (The offending process was getting about 9%.) 
  How much CPU involvement is there in sending I/O requests to the drive 
and waiting on an interrupt?  Maybe I don't understand what's going on, 
but I would expect the CPU involvement in disk I/O to be practically 
NIL, unless it's trying to be really smart about it.  Is it?  Or maybe 
the kernel isn't using DMA... this is a Dell Precision 340.  I'm not 
sure what drive is in it, but I would be surprised if it weren't using DMA.

The next thing has to do with interactivity.  I don't know how much 
interactivity code is in the 2.4 kernel, but as I understand it, it's 
based on a process giving up its timeslice.  Well, if a process is 
causing numerous page faults, is it not, in effect, giving up its 
timeslice?  Is it therefore being scheduled as an interactive process? 
If so, the process scheduler is very mistaken.  In my naive opinion, 
causing a page fault should NOT be an indication of interactivity.  It 
seemed to me that the process was getting too much attention from the 
process scheduler.  The thrashing process was getting run while other 
processes (like X, etc.) were being almost completely starved.

Does causing a page fault get interpreted as interactivity?

Going under the assumption that disk I/O should require almost no CPU 
involvement, it seems to me that memory could be managed so that 
processes which behave themselves should feel no effect from the 
thrashing, while the thrashing process continues to thrash and therefore 
have little opportunity to use the CPU.  This would require that the VM 
not swap out pages which are being used heavily by well-behaved 
processes.  Instead, it seems that the thrashing process is being 
allowed to completely trash everything else running in the system.

Now, I'm not stupid.  Someone's going to tell me that the problem is in 
user space, and that I should fix the user space program.  (I am doing 
that.)  I can certainly understand that perspective, but it seems to me 
that the fairest result of a process misbehaving is that it should only 
punish itself, not every other process in the system.

Re: naive questions about thrashing

[J.A. Magallon]

On 04.30, Timothy Miller wrote:
> I am running kernel version 2.4.18-26.7.x under Red Hat 7.2.
> 
> I wrote a CPU-intensive program which attempts to use over 700 megs of 
> RAM on a 512-meg box, therefore it thrashes.
> 
> One thing I noticed was that 'top' reported that the kernel ("system") 
> was using 68% of the CPU.  (The offending process was getting about 9%.) 
>   How much CPU involvement is there in sending I/O requests to the drive 
> and waiting on an interrupt?  Maybe I don't understand what's going on, 
> but I would expect the CPU involvement in disk I/O to be practically 
> NIL, unless it's trying to be really smart about it.  Is it?  Or maybe 
> the kernel isn't using DMA... this is a Dell Precision 340.  I'm not 
> sure what drive is in it, but I would be surprised if it weren't using DMA.
> 

As I understand it, it is telling you that your programs spends 68% of
its time is kernel space, ie, waiting your pages to come from disk. It
does not mean that the CPU is doing anything, but it is locked by the
kernel.

If you can't afford to buy more memory, recode the thing. So much thrashing
looks like you access your data very randomly. Try to process the data
in a more sequential way, so you just fault after processing a big bunch
of data. With 700Mb of data and a 512Mb box, at least half of your data
fit in memory, so under an ideal sequential access you just would page
300Mb one time...

-- 
J.A. Magallon <jamagallon@able.es>      \                 Software is like sex:
werewolf.able.es                         \           It's better when it's free
Mandrake Linux release 9.2 (Cooker) for i586
Linux 2.4.21-rc1-jam1 (gcc 3.2.2 (Mandrake Linux 9.2 3.2.2-5mdk))

Re: naive questions about thrashing

[Timothy Miller]

J.A. Magallon wrote:
> On 04.30, Timothy Miller wrote:
> 
>>I am running kernel version 2.4.18-26.7.x under Red Hat 7.2.
>>
>>I wrote a CPU-intensive program which attempts to use over 700 megs of 
>>RAM on a 512-meg box, therefore it thrashes.
>>
>>One thing I noticed was that 'top' reported that the kernel ("system") 
>>was using 68% of the CPU.  (The offending process was getting about 9%.) 
>>  How much CPU involvement is there in sending I/O requests to the drive 
>>and waiting on an interrupt?  Maybe I don't understand what's going on, 
>>but I would expect the CPU involvement in disk I/O to be practically 
>>NIL, unless it's trying to be really smart about it.  Is it?  Or maybe 
>>the kernel isn't using DMA... this is a Dell Precision 340.  I'm not 
>>sure what drive is in it, but I would be surprised if it weren't using DMA.
>>
> 
> 
> As I understand it, it is telling you that your programs spends 68% of
> its time is kernel space, ie, waiting your pages to come from disk. It
> does not mean that the CPU is doing anything, but it is locked by the
> kernel.

What would the kernel be locked while waiting on disk I/O?  Shouldn't it 
be running another process?  It's not DOING anything.  The whole idea 
behind a multitasking OS is to overlap the I/O of one process with the 
CPU usage of another whenever possible.  Swapping is an I/O operation.

And for that matter, if every runnable process has pages swapped out so 
that they cannot run, then the CPU should be IDLE.

Am I wrong?

> 
> If you can't afford to buy more memory, recode the thing. So much thrashing
> looks like you access your data very randomly. Try to process the data
> in a more sequential way, so you just fault after processing a big bunch
> of data. With 700Mb of data and a 512Mb box, at least half of your data
> fit in memory, so under an ideal sequential access you just would page
> 300Mb one time...
> 

The process got that large because of a bug in my program.  But a 
side-effect of that was kernel behavior that didn't make sense to me.  I 
decided to ask about it.