Re: Feedback on preemptible kernel patch xfs

Re: Feedback on preemptible kernel patch xfs

[Gerold Jury]

Hi Robert

First the good news.
Even my most ugly ideas where not able to crash your preemtible 2.4.10-pre10-xfs

But, to be sure i repeated everything, neither latencytest-0.42 nor my own tests could
find a difference with or without the preemptible patch.
I do not know if i can expect a lower latency at this stage of development.

A maximum of 15 msec latency with all the stress, i managed to put on the machine
is not that bad anyway.

The CPU is a 1.1GHz Athlon. I forgot to mention this.

I will continue to test the preempt patches.

Do you want me to test anything special ?

Best Regards
Gerold

Re: Feedback on preemptible kernel patch

[Robert Love]

On Sun, 2001-09-09 at 23:24, Daniel Phillips wrote:
> This may not be your fault.  It's a GFP_NOFS recursive allocation - this
> comes either from grow_buffers or ReiserFS, probably the former.  In
> either case, it means we ran completely out of free pages, even though
> the caller is willing to wait.  Hmm.  It smells like a loophole in vm
> scanning.

Hi, Daniel.  If you remember, a few users of the preemption patch
reported instability and/or syslog messages such as:

Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
Sep  9 23:08:02 sjoerd last message repeated 93 times
Sep  9 23:08:02 sjoerd kernel: cation failed (gfp=0x70/1).
Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
Sep  9 23:08:02 sjoerd last message repeated 281 times

It now seems that all of them are indeed using ReiserFS.  There are no
other reported problems with the preemption patch, except from those
users...

I am beginning to muse over the source, looking at when kmalloc is
called with GFP_NOFS in ReiserFS, and then the path that code takes in
the VM source.

I assume the kernel VM code has a hole somewhere, and the request is
falling through?  It should wait, even if no pages were free so, right? 

Where should I begin looking?  How does it relate to ReiserFS?  How is
preemption related?

Thank you very much,

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Daniel Phillips]

On September 15, 2001 09:18 pm, Robert Love wrote:
> On Sun, 2001-09-09 at 23:24, Daniel Phillips wrote:
> > This may not be your fault.  It's a GFP_NOFS recursive allocation - this
> > comes either from grow_buffers or ReiserFS, probably the former.  In
> > either case, it means we ran completely out of free pages, even though
> > the caller is willing to wait.  Hmm.  It smells like a loophole in vm
> > scanning.
> 
> Hi, Daniel.  If you remember, a few users of the preemption patch
> reported instability and/or syslog messages such as:
> 
> Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
> Sep  9 23:08:02 sjoerd last message repeated 93 times
> Sep  9 23:08:02 sjoerd kernel: cation failed (gfp=0x70/1).
> Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
> Sep  9 23:08:02 sjoerd last message repeated 281 times
> 
> It now seems that all of them are indeed using ReiserFS.  There are no
> other reported problems with the preemption patch, except from those
> users...
> 
> I am beginning to muse over the source, looking at when kmalloc is
> called with GFP_NOFS in ReiserFS, and then the path that code takes in
> the VM source.
> 
> I assume the kernel VM code has a hole somewhere, and the request is
> falling through?  It should wait, even if no pages were free so, right? 
> 
> Where should I begin looking?  How does it relate to ReiserFS?  

The only other path that uses NO_FS is grow_buffers.  But the page probably 
got dirtied in generic_file_write, which already put buffers on it.  A NOFS 
allocation could also be triggered by Ext2 (and other filesystems) by having 
lots of dirty mmaps: when page_launder calls page->writepage then the page
won't have buffers on it.  That's probably not what's happening though.

I don't think NOFS is causing the problem by the way, it's just a convenient 
marker to recognize where the allocation is coming from.

What happens is, page_launder calls reseiserfs_writepage which for some 
reason recursively allocates a page (I don't have time to look for the exact 
path - it's probably for the journal - but whoever has the problem can check 
it via show_trace).  We now are in a recursive allocation situation (with 
PF_MEMALLOC) so page_launder doesn't get called and we drop through to get 
the "failed" message.

It's not nice for __alloc_pages to fail back to a caller that's willing to
wait.  See below for one idea about what to do about it.

> How is preemption related?

I'll speculate: page_launder is now yielding to other tasks when it releases
spinlocks to do a writepage.  One of them is likely to come back in and
attempt another allocation while we're at rock bottom.

If that's true then I think we should consider something I've wanted to try:
make callers block on a wait queue in __alloc_pages when memory is really
tight.

Hmm.  We could do that just in this specific case of PF_MEMALLOC+GFP_WAIT.
Semaphores work well for this kind of thing, something like:

	if (!(current->flags & PF_MEMALLOC)) {
		<the existing reclaim/launder logic>
	} else
		if (gfp_mask & __GFP_WAIT) {
			wakeup_kswapd();
			atomic_inc(&memwaiters);
			down(&memwait);
			goto try_again;
		}

Then in kswapd:

        waiters = atomic_read(&memwaiters);
        atomic_sub(waiters, &memwaiters);
	while (waiters--)
		up(&memwaiter);

when we detect that free memory is restored to something reasonable.  This
won't deadlock on memwait because kswapd doesn't use __GFP_WAIT.

We also have to make kswapd count wakeups so we can be sure it doesn't sleep
while somebody is waiting in __alloc_pages.  The only way I know to do this
reliably is with another semaphore:

	void wakeup_kswapd() {
		up(&kswapd_sleep);
	}

and kswapd downs that semaphore instead of doing interruptible_sleep_on_timeout.
Additionally, a timer has to up() the semaphore periodically, to recover the
sleep_on_timeout behaviour.

Sound like overkill?  The alternative is to let GFP_WAIT allocations fail which
forces users like journalling filesystems to busy wait and load up the runqueue.

Sorry I don't have time to code this just now, but I'd like to give this a try
if the problem's still there next week though.  Or if you're in the mood...

--
Daniel

Re: Feedback on preemptible kernel patch

[Daniel Phillips]

On September 16, 2001 03:28 am, Daniel Phillips wrote:
> On September 15, 2001 09:18 pm, Robert Love wrote:
> > On Sun, 2001-09-09 at 23:24, Daniel Phillips wrote:
> > > This may not be your fault.  It's a GFP_NOFS recursive allocation - this
> > > comes either from grow_buffers or ReiserFS, probably the former.  In
> > > either case, it means we ran completely out of free pages, even though
> > > the caller is willing to wait.  Hmm.  It smells like a loophole in vm
> > > scanning.

Oh, wait, I was working off 2.4.9 source, and your user had the problem with
2.4.9-pre4, where we have GFP_NOHIGHIO.  So - reinterpreting the bits - all
those failures are bounce buffer allocations.  People are also getting these
failures without your patch, so relax ;-)

Maybe allowing preemption inside page_launder makes it happen more often.

--
Daniel

Re: Feedback on preemptible kernel patch xfs

[jury gerold]

I used your patch on 2.4.10-pre10-xfs from the SGI cvs tree.
2 files had to be changed
fs/xfs_support/atomic.h and fs/xfs_support/mutex.h
needed a include sched.h

rootfilesystem is ext2 everything else is xfs
athlon optimisation is switched on
chipset is via
the nvidia kernel module for OpenGL acceleration is running
hisax isdn driver for internet access
USB web cam

I have tried heavy filesystem operations (cp -ar x y && rm -rf y)
with a big compile job -j2 and some OpenGL programs together with the 
web cam

on the USB side i had some "_comp parameters have gone AWOL" messages in 
the syslog from the cpia driver
but i remember them from a no preemtion kernel as well

so far everything is stable

i like the idea but i have not made any tests on the latency yet

Regards
Gerold

Robert Love wrote:

>It now seems that all of them are indeed using ReiserFS.  There are no
>other reported problems with the preemption patch, except from those
>users...
>

Re: Feedback on preemptible kernel patch xfs

[Robert Love]

On Tue, 2001-09-18 at 07:05, jury gerold wrote:
> I used your patch on 2.4.10-pre10-xfs from the SGI cvs tree.
> 2 files had to be changed
> fs/xfs_support/atomic.h and fs/xfs_support/mutex.h
> needed a include sched.h

Thank you for reporting this.

I just made a diff against xfs-cvs-20010917 with your changes. 
Obviously I can't merge the changes into the main patch since not
everyone has XFS, but I will make the patch available.

> rootfilesystem is ext2 everything else is xfs
> athlon optimisation is switched on
> chipset is via
> the nvidia kernel module for OpenGL acceleration is running
> hisax isdn driver for internet access
> USB web cam
> 
> I have tried heavy filesystem operations (cp -ar x y && rm -rf y)
> with a big compile job -j2 and some OpenGL programs together with the 
> web cam

Good.  Then we can probably mark XFS as preempt safe (no reason to think
otherwise).  Those file operations were on the XFS partitions, right?

> on the USB side i had some "_comp parameters have gone AWOL" messages in 
> the syslog from the cpia driver
> but i remember them from a no preemtion kernel as well

Yah, probably from mainline kernel...

> so far everything is stable

excellent.

> i like the idea but i have not made any tests on the latency yet

if you do, please post.

thanks for the feedback,

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch xfs

[Robert Love]

On Wed, 2001-09-19 at 21:49, Gerold Jury wrote:
> First the good news.
> Even my most ugly ideas where not able to crash your preemtible
> 2.4.10-pre10-xfs

Good to hear.

> But, to be sure i repeated everything, neither latencytest-0.42 nor
> my own tests could find a difference with or without the preemptible
> patch. I do not know if i can expect a lower latency at this stage of
> development.

I am surprised, you should see a difference, especially with the
latencytest.  Silly question, but you both applied the patch and enabled
the config statement, right?

No, at this stage of development we are seeing greatly reduced latency
times with the patch.  Continued work is going to be on improving
locking mechanisms, but this is something that will come about later and
improve the kernel overall.

> A maximum of 15 msec latency with all the stress, i managed to put on the
> machine is not that bad anyway.

No, 15ms is very good.  I am seeing things 5-10ms here, but much much
higher without preemption. Odd.

> The CPU is a 1.1GHz Athlon. I forgot to mention this.

Oh, Good. we earlier had problems with an Athlon optimized kernel, but
we have solved those problems.

> I will continue to test the preempt patches.

Thank you.

> Do you want me to test anything special ?

I can't think of a benchmark that tests various aspects of a filesystem
(file creation/deletion, directory seeking and listing, etc.) but that
would be great to see if xfs improves with preemption.

You can test raw disk I/O with dbench ftp://samba.org/pub/tridge/dbench/
... try 16 threads (dbench -16).

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch xfs

[Gerold Jury]

On Thursday 20 September 2001 02:56, Robert Love wrote:
> I am surprised, you should see a difference, especially with the
> latencytest.  Silly question, but you both applied the patch and enabled
> the config statement, right?
>
Really, i have checked twice.
The patch could, by the way, write a line to the syslog when enabled.

All the filesystem operations happend on the xfs partitions.
I noticed more equally distributed read/write operations with smaller slices 
during big copy jobs on xfs.
This effect may well come from the preemption patch. I used a spare partition
for the test, so the filesystem was in the same state with both kernels 
during the tests.
Xfs usually delays the write operations and does them in bigger blocks.
The behavior of XFS has changed with the kernel versions towards this 
direction anyway but is clearly different with the preemption patch.

I will redo the latency tests with the standard Xfree86 nvidia driver.
It may give a different picture.
The graphics test and the /proc test have shown the highest latency's.
Both involve the xserver (proc for the xterm).
The other tests have been around 5-6 msec in both cases.

And i will do the dbench test of course.

Gerold

Re: Feedback on preemptible kernel patch xfs

[Robert Love]

On Fri, 2001-09-21 at 08:29, Gerold Jury wrote:
> On Thursday 20 September 2001 02:56, Robert Love wrote:
> > I am surprised, you should see a difference, especially with the
> > latencytest.  Silly question, but you both applied the patch and enabled
> > the config statement, right?
> >
> Really, i have checked twice.
> The patch could, by the way, write a line to the syslog when enabled.

OK, I believe you :)

Yes, but I find all the `NET4.0 loaded!' as crap as it is.  If
CONFIG_PREEMPT is defined, rest assured the code is correct.

> All the filesystem operations happend on the xfs partitions.
> I noticed more equally distributed read/write operations with smaller slices 
> during big copy jobs on xfs.
> This effect may well come from the preemption patch. I used a spare partition
> for the test, so the filesystem was in the same state with both kernels 
> during the tests.
> Xfs usually delays the write operations and does them in bigger blocks.
> The behavior of XFS has changed with the kernel versions towards this 
> direction anyway but is clearly different with the preemption patch.
> 
> I will redo the latency tests with the standard Xfree86 nvidia driver.
> It may give a different picture.
> The graphics test and the /proc test have shown the highest latency's.
> Both involve the xserver (proc for the xterm).
> The other tests have been around 5-6 msec in both cases.
> 
> And i will do the dbench test of course.

Very good. Please let me know.

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Robert Love]

On Fri, 2001-09-14 at 02:40, Arjan Filius wrote:
> Hello Robert,

Hi Arjan,

> I do Athlon/K7 opmimization indeed, and didn't test without.
> Haven't stress-tested it very well yet, but as soon i notice something i
> let you know.

Have you had any oops that were unexplained, after we fixed the other
problems?  I have attached the patch below, you can give it a whirl, but
it is odd you have had no problems.

> Great!

:)

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net


diff -urN linux-2.4.10-pre8/arch/i386/kernel/i387.c linux/arch/i386/kernel/i387.c
--- linux-2.4.10-pre8/arch/i386/kernel/i387.c	Thu Sep 13 19:24:48 2001
+++ linux/arch/i386/kernel/i387.c	Thu Sep 13 20:00:57 2001
@@ -10,6 +10,7 @@
 
 #include <linux/config.h>
 #include <linux/sched.h>
+#include <linux/spinlock.h>
 #include <asm/processor.h>
 #include <asm/i387.h>
 #include <asm/math_emu.h>
@@ -65,6 +66,8 @@
 {
 	struct task_struct *tsk = current;
 
+	ctx_sw_off();
+	
 	if (tsk->flags & PF_USEDFPU) {
 		__save_init_fpu(tsk);
 		return;
diff -urN linux-2.4.10-pre8/include/asm-i386/i387.h linux/include/asm-i386/i387.h
--- linux-2.4.10-pre8/include/asm-i386/i387.h	Thu Sep 13 19:27:28 2001
+++ linux/include/asm-i386/i387.h	Thu Sep 13 20:01:30 2001
@@ -12,6 +12,7 @@
 #define __ASM_I386_I387_H
 
 #include <linux/sched.h>
+#include <linux/spinlock.h>
 #include <asm/processor.h>
 #include <asm/sigcontext.h>
 #include <asm/user.h>
@@ -24,7 +25,7 @@
 extern void restore_fpu( struct task_struct *tsk );
 
 extern void kernel_fpu_begin(void);
-#define kernel_fpu_end() stts()
+#define kernel_fpu_end() stts(); ctx_sw_on()
 
 
 #define unlazy_fpu( tsk ) do { \

Re: Feedback on preemptible kernel patch

[Arjan Filius]

Hi Robert,

On 14 Sep 2001, Robert Love wrote:

> On Fri, 2001-09-14 at 02:40, Arjan Filius wrote:
> > Hello Robert,
>
> Hi Arjan,
>
> > I do Athlon/K7 opmimization indeed, and didn't test without.
> > Haven't stress-tested it very well yet, but as soon i notice something i
> > let you know.
>
> Have you had any oops that were unexplained, after we fixed the other
> problems?
Didn't discover yet, the only "strange" thing is when using <TAB>
autocompleteion in a kterm in kde i get every time a kde crash-bug-report
popup message. however, no kernel messages or whatsoever.


>I have attached the patch below, you can give it a whirl, but
> it is odd you have had no problems.

Well it's a fpu patch, and as far as i know i don't use the fpu that much
at the moment but i'll try that.

in the hope finding the usage of fpu-irq:
sjoerd:/usr/src/linux # cat /proc/interrupts
           CPU0
  0:   13374740          XT-PIC  timer
  1:      14581          XT-PIC  keyboard
  2:          0          XT-PIC  cascade
  9:     238018          XT-PIC  usb-uhci, usb-uhci, EMU10K1
 10:     601500          XT-PIC  ide2, sym53c8xx
 11:      87300          XT-PIC  eth0
 12:     230338          XT-PIC  PS/2 Mouse
 14:     331764          XT-PIC  ide0
 15:      39468          XT-PIC  ide1
NMI:          0
ERR:          0

But /proc/interrupts shows only those irq's which are currently in use, is
there any way to show usage of currenlty unused interrupts?

Greetings,

-- 
Arjan Filius
mailto:iafilius@xs4all.nl

Re: Feedback on preemptible kernel patch

[Erik Mouw]

On Sat, Sep 15, 2001 at 11:44:57AM +0200, Arjan Filius wrote:

Hi Arjan,

> But /proc/interrupts shows only those irq's which are currently in use, is
> there any way to show usage of currenlty unused interrupts?

Yes, /proc/stat. The "intr" line shows the total number of interrupts
and the number of interrupts for each interrupt line.


Erik

-- 
J.A.K. (Erik) Mouw, Information and Communication Theory Group, Department
of Electrical Engineering, Faculty of Information Technology and Systems,
Delft University of Technology, PO BOX 5031,  2600 GA Delft, The Netherlands
Phone: +31-15-2783635  Fax: +31-15-2781843  Email: J.A.K.Mouw@its.tudelft.nl
WWW: http://www-ict.its.tudelft.nl/~erik/

Re: Feedback on preemptible kernel patch

[Robert Love]

On Sat, 2001-09-15 at 05:44, Arjan Filius wrote:
> Hi Robert,

Hi Arjan,

> Didn't discover yet, the only "strange" thing is when using <TAB>
> autocompleteion in a kterm in kde i get every time a kde crash-bug-report
> popup message. however, no kernel messages or whatsoever.

Odd. This does not occur without preemption enabled?

> Well it's a fpu patch, and as far as i know i don't use the fpu that much
> at the moment but i'll try that.

Its a patch for when the kernel uses the FPU, specifically during MMX or
3DNow instructions.  That is why iys for Athlon-optimized kernels,
because the Athlon uses 3DNow instructions for some fast page copies.

When the kernel uses FPU, it does not use SMP locks, since that is a per
CPU condition.  However, preemption can occur in the middle of a CPU
doing that stuff, and that messes stuff up very much.  The Athlon patch
fixes it, it is much needed if your kernel is Athlon-optimized.

Userspace FPU is controlled by the kernel already, so we don't need to
worry about that.  In userspace, the kernel handles everything ... in
kernel land, if we want to go play with something (like 3DNow), we need
to take care to set it up and restore ourselves properly.

> in the hope finding the usage of fpu-irq:
> sjoerd:/usr/src/linux # cat /proc/interrupts
>            CPU0
>   0:   13374740          XT-PIC  timer
>   1:      14581          XT-PIC  keyboard
>   2:          0          XT-PIC  cascade
>   9:     238018          XT-PIC  usb-uhci, usb-uhci, EMU10K1
>  10:     601500          XT-PIC  ide2, sym53c8xx
>  11:      87300          XT-PIC  eth0
>  12:     230338          XT-PIC  PS/2 Mouse
>  14:     331764          XT-PIC  ide0
>  15:      39468          XT-PIC  ide1
> NMI:          0
> ERR:          0
> 
> But /proc/interrupts shows only those irq's which are currently in use, is
> there any way to show usage of currenlty unused interrupts?

cat /proc/stat, see the `intr' line, each field is an increasing
interrupt value.

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Robert Love]

On Thu, 2001-09-13 at 22:47, Dieter Nützel wrote:
> > first, highmem is fixed and the original patch you have from me is good.
> > second, Daniel Phillips gave me some feedback into how to figure out the
> > VM error.  I am working on it, although just the VM potential
> 
> Good to hear.
> 
> > -- ReiserFS may be another problem.
> 
> Can't wait for that.
> 
> > third, you may be experiencing problems with a kernel optimized for
> > Athlon.  this may or may not be related to the current issues with an
> > Athlon-optimized kernel.  Basically, functions in arch/i386/lib/mmx.c
> > seem to need some locking to prevent preemption.  I have a basic patch
> > and we are working on a final one.
> 
> Can you please send this stuff along to me?
> You know I own an Athlon (since yester Athlon II 1 GHz :-) and need some 
> input...

Yes, find the Athlon patch below.  Please let me know if it works.

> Mobo is MSI MS-6167 Rev 1.0B (AMD Irongate C4, yes the very first one)
> 
> Kernel with preempt patch and mmx/3dnow! optimization crash randomly.
> Never had that (without preempt) during the last two years.

Oh, I did not remember you having problems with Athlon.  I try to keep a
list of what problems are being had.

Are there any other configurations where you have problems?

> Thanks,
> 	Dieter

You are welcome; thank you for the feedback.

Before applying, note there are new patches at
http://tech9.net/rml/linux - a patch for 2.4.10-pre9 and a _new_ patch
for 2.4.9-ac10.  These include everything (highmem, etc) except the
Athlon patch.

The problem with Athlon compiled kernels is that MMX/3DNow routines used
in the kernel are not preempt safe (but SMP safe, so I missed them). 
This patch should correct it.


diff -urN linux-2.4.10-pre8/arch/i386/kernel/i387.c linux/arch/i386/kernel/i387.c
--- linux-2.4.10-pre8/arch/i386/kernel/i387.c	Thu Sep 13 19:24:48 2001
+++ linux/arch/i386/kernel/i387.c	Thu Sep 13 20:00:57 2001
@@ -10,6 +10,7 @@
 
 #include <linux/config.h>
 #include <linux/sched.h>
+#include <linux/spinlock.h>
 #include <asm/processor.h>
 #include <asm/i387.h>
 #include <asm/math_emu.h>
@@ -65,6 +66,8 @@
 {
 	struct task_struct *tsk = current;
 
+	ctx_sw_off();
+	
 	if (tsk->flags & PF_USEDFPU) {
 		__save_init_fpu(tsk);
 		return;
diff -urN linux-2.4.10-pre8/include/asm-i386/i387.h linux/include/asm-i386/i387.h
--- linux-2.4.10-pre8/include/asm-i386/i387.h	Thu Sep 13 19:27:28 2001
+++ linux/include/asm-i386/i387.h	Thu Sep 13 20:01:30 2001
@@ -12,6 +12,7 @@
 #define __ASM_I386_I387_H
 
 #include <linux/sched.h>
+#include <linux/spinlock.h>
 #include <asm/processor.h>
 #include <asm/sigcontext.h>
 #include <asm/user.h>
@@ -24,7 +25,7 @@
 extern void restore_fpu( struct task_struct *tsk );
 
 extern void kernel_fpu_begin(void);
-#define kernel_fpu_end() stts()
+#define kernel_fpu_end() stts(); ctx_sw_on()
 
 
 #define unlazy_fpu( tsk ) do { \


-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Dieter Nützel]

[-- Attachment #1: Type: text/plain, Size: 5951 bytes --]

Am Freitag, 14. September 2001 06:35 schrieb Robert Love:
> On Thu, 2001-09-13 at 22:47, Dieter Nützel wrote:

> > > -- ReiserFS may be another problem.
> >
> > Can't wait for that.

Most wanted, now.

> >
> > > third, you may be experiencing problems with a kernel optimized for
> > > Athlon.  this may or may not be related to the current issues with an
> > > Athlon-optimized kernel.  Basically, functions in arch/i386/lib/mmx.c
> > > seem to need some locking to prevent preemption.  I have a basic patch
> > > and we are working on a final one.
> >
> > Can you please send this stuff along to me?
> > You know I own an Athlon (since yester Athlon II 1 GHz :-) and need some
> > input...
>
> Yes, find the Athlon patch below.  Please let me know if it works.

Tried it and it works so far.

It seems to be that kswap put some additional "load" on the disk from time to 
time. Or is it the ReiserFS thing, again?

> > Mobo is MSI MS-6167 Rev 1.0B (AMD Irongate C4, yes the very first one)
> >
> > Kernel with preempt patch and mmx/3dnow! optimization crash randomly.
> > Never had that (without preempt) during the last two years.
>
> Oh, I did not remember you having problems with Athlon.  I try to keep a
> list of what problems are being had.

Have you a copy of my posted ksymoops file?
But the oopses seems to be cured.

> Are there any other configurations where you have problems?

I don't know exactly 'cause kernel hacking is not my main focus.

But have you thought about the MMX/3DNow! stuff in Mesa/OpenGL (XFree86 DRI)?
And what do you think about the XFree86 Xv extentions (video) or the whole 
MPEG2/3/4, Ogg-Vorbis, etc. (multimedia stuff)?

Do all these libraries (progs) need some preempt patches?
That's why I cross posted to the DRI-Devel List (sorry).


> Before applying, note there are new patches at
> http://tech9.net/rml/linux - a patch for 2.4.10-pre9 and a _new_ patch
> for 2.4.9-ac10.  These include everything (highmem, etc) except the
> Athlon patch.

> The problem with Athlon compiled kernels is that MMX/3DNow routines used
> in the kernel are not preempt safe (but SMP safe, so I missed them).
> This patch should correct it.

I understand ;-)
It seems to calm it.

> diff -urN linux-2.4.10-pre8/arch/i386/kernel/i387.c
> linux/arch/i386/kernel/i387.c ---
> linux-2.4.10-pre8/arch/i386/kernel/i387.c	Thu Sep 13 19:24:48 2001 +++
> linux/arch/i386/kernel/i387.c	Thu Sep 13 20:00:57 2001
> @@ -10,6 +10,7 @@
>
>  #include <linux/config.h>
>  #include <linux/sched.h>
> +#include <linux/spinlock.h>
>  #include <asm/processor.h>
>  #include <asm/i387.h>
>  #include <asm/math_emu.h>
> @@ -65,6 +66,8 @@
>  {
>  	struct task_struct *tsk = current;
>
> +	ctx_sw_off();
> +
>  	if (tsk->flags & PF_USEDFPU) {
>  		__save_init_fpu(tsk);
>  		return;
> diff -urN linux-2.4.10-pre8/include/asm-i386/i387.h
> linux/include/asm-i386/i387.h ---
> linux-2.4.10-pre8/include/asm-i386/i387.h	Thu Sep 13 19:27:28 2001 +++
> linux/include/asm-i386/i387.h	Thu Sep 13 20:01:30 2001
> @@ -12,6 +12,7 @@
>  #define __ASM_I386_I387_H
>
>  #include <linux/sched.h>
> +#include <linux/spinlock.h>
>  #include <asm/processor.h>
>  #include <asm/sigcontext.h>
>  #include <asm/user.h>
> @@ -24,7 +25,7 @@
>  extern void restore_fpu( struct task_struct *tsk );
>
>  extern void kernel_fpu_begin(void);
> -#define kernel_fpu_end() stts()
> +#define kernel_fpu_end() stts(); ctx_sw_on()
>
>
>  #define unlazy_fpu( tsk ) do { \

Now, here are my results.

Athlon II 1 GHz (0.18 µm)
MSI MS-6167 Rev 1.0B (Irongate C4)
640 MB PC100-2-2-2 SDRAM
IBM DDYS 18 GB U160 (on AHA-2940UW)
ReiserFS 3.6 on all partitions

dbench-1.1 32 clients

2.4.10-pre9
Throughput 22.8881 MB/sec (NB=28.6102 MB/sec  228.881 MBit/sec)
15.000u 52.710s 3:05.59 36.4%   0+0k 0+0io 911pf+0w
load: 3168

2.4.10-pre9 + patch-rml-2.4.10-pre9-preempt-kernel-1
Throughput 22.7157 MB/sec (NB=28.3946 MB/sec  227.157 MBit/sec)
15.070u 52.730s 3:06.97 36.2%   0+0k 0+0io 911pf+0w
load: 2984


bonnie++

2.4.10-pre
Version 1.92a       ------Sequential Output------ --Sequential Input- 
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
%CP
SunWave1      1248M   117  95 14510  16  6206   6   189  98 27205  16 289.8   
4
Latency               107ms    2546ms     720ms   99241us   75832us    3449ms
Version 1.92a       ------Sequential Create------ --------Random 
Create--------
SunWave1            -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP
                 16  4215  38 +++++ +++ 14227  93  8885  79 +++++ +++  4324  
35
Latency               584ms    8221us   14158us    7681us   14274us     794ms
load: 321


2.4.10-pre9 + patch-rml-2.4.10-pre9-preempt-kernel-1
Version 1.92a       ------Sequential Output------ --Sequential Input- 
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
%CP
SunWave1      1248M    98  96 14716  18  6190   6   181  98 27945  17 291.2   
5
Latency               127ms    4241ms     726ms   97889us   63321us    3071ms
Version 1.92a       ------Sequential Create------ --------Random 
Create--------
SunWave1            -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP
                 16  5781  54 +++++ +++ 14733  95  5899  54 +++++ +++ 11785  
96
Latency               510ms    2160us    4501us    8917us    2130us    6299us
load: 302

Deleting with ReiserFS and the preempt kernel is GREAT!
But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2) or 
plaympeg together with dbench (16, 32). ReiserFS needs some preemption fixes, 
too?

I've attached two small compressed bonnie++ HTML files.

-Dieter

[-- Attachment #2: 2.4.10-pre9-bonnie++.html.bz2 --]
[-- Type: application/x-bzip2, Size: 905 bytes --]

[-- Attachment #3: 2.4.10-pre9-preempt-bonnie++.html.bz2 --]
[-- Type: application/x-bzip2, Size: 900 bytes --]

Re: Feedback on preemptible kernel patch

[Robert Love]

On Sat, 2001-09-15 at 00:25, Dieter Nützel wrote:
> > > >  ReiserFS may be another problem.
> > > Can't wait for that.
> Most wanted, now.

I am working on it, but I am unfamilar with it all.

Are you seeing any specific problems, now?  With the latest preemption
patch on 2.4.10-pre9, do you crash? oops?

The only outstanding issue now is ReiserFS issues.

> Tried it and it works so far.
> 
> It seems to be that kswap put some additional "load" on the disk from time
> to time. Or is it the ReiserFS thing, again?

I don't think its related to ReiserFS.

What sort of activity are you seeing?  How often?  How do you know its
kswapd?

> Have you a copy of my posted ksymoops file?
> But the oopses seems to be cured.

Yah, I just am having a problem sorting  who said what running what and
when :)

I am glad the patch fixed it, the final version of that is going into
the next preemption patch.  Stay tuned.

> I don't know exactly 'cause kernel hacking is not my main focus.
> 
> But have you thought about the MMX/3DNow! stuff in Mesa/OpenGL (XFree86 DRI)?
> And what do you think about the XFree86 Xv extentions (video) or the whole 
> MPEG2/3/4, Ogg-Vorbis, etc. (multimedia stuff)?
> 
> Do all these libraries (progs) need some preempt patches?
> That's why I cross posted to the DRI-Devel List (sorry).

No, these are unrelated.

Sorry for the cross-post, DRI :)

The kernel takes care of saving state for FPU operations for userspace. 
Indeed, it takes care of everything for userspace.  In kernel land, we
don't have that beauty, we have to worry about everything we do and
everything we change, and thus we have this problem.

What exactly happens is another operation is preempting the current
process during an FPU operation, when the CPU is in a different state,
and then everything barfs since it is not as it wants to be.

This is not an issue for userspace.

> I understand ;-)
> It seems to calm it.

Good.

> Now, here are my results.
> <snip>

These results are pretty good.  Throughput seems down 2-3% in many
cases, although latency is greatly improved.  Look at those latency
changes!  From thousands of ms to hundreds of us in bonnie.  Wow.

Even if you don't care about latency (I'm not an audio person or
anything), these changes should be worth it.

> Deleting with ReiserFS and the preempt kernel is GREAT!

Good. I/O latency should be great now, with little change in
throughput...

> But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2) or 
> plaympeg together with dbench (16, 32). ReiserFS needs some preemption
> fixes, too?

You may still get some small hiccups ( < 1 second?) even with the
preemption patch, as kernel locks prevent preemption (the patch can't
guarentee low latency, just preemption outside of the locks).

However, how bad was the hiccups with preemption disabled?  I have heard
reports where it is 3-5sec at times.

As the kernel becomes more scalable (finer-grain locking), preemption
will improve.  Past that, perhaps during 2.5, we can work on some other
things to improve preemption.

> I've attached two small compressed bonnie++ HTML files.

These were neat, thanks.

Thank you for your feedback and support.  Stay current with the kernel
and the preemption patches, and I will try to figure the ReiserFS
crashes out.

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Dieter Nützel]

[-- Attachment #1: Type: text/plain, Size: 5493 bytes --]

Am Samstag, 15. September 2001 07:14 schrieb Robert Love:
> On Sat, 2001-09-15 at 00:25, Dieter Nützel wrote:
> > > > >  ReiserFS may be another problem.
> > > >
> > > > Can't wait for that.
> >
> > Most wanted, now.
>
> I am working on it, but I am unfamilar with it all.
>
> Are you seeing any specific problems, now?  With the latest preemption
> patch on 2.4.10-pre9, do you crash? oops?

No, nothing with 2.4.10-pre9 + patch-rml-2.4.10-pre9-preempt-kernel-1 and 
your MMX/3Dnow! fix.

2.4.10-pre10 + patch-rml-2.4.10-pre10-preempt-kernel-1 seems to be a winner!

See my results below.

> The only outstanding issue now is ReiserFS issues.

Yes, but no crash or oops for me.
"Only" some "stalls" during MPEG/Ogg-Vorbis playback (2-5 sec) :-(

> > It seems to be that kswap put some additional "load" on the disk from
> > time to time. Or is it the ReiserFS thing, again?
>
> I don't think its related to ReiserFS.

I think you are right.

> What sort of activity are you seeing?  How often?  How do you know its
> kswapd?

I saw it with "top" at the first line (but only some few percent).
It was during untarring some mid-sized archives (DRI) which took normally ~10 
sec, but with kswap and 2.4.9-pre9+your patches ~30 sec. Even "sync" needed 
some additional seconds.

Are there some reschedule/context switch (kernel lock release) statements 
missing in ReiserFS?

Is this possible? Chris?

> I am glad the patch fixed it, the final version of that is going into
> the next preemption patch.  Stay tuned.

I am very happy with patch-rml-2.4.10-pre10-preempt-kernel-1.

> These results are pretty good.  Throughput seems down 2-3% in many
> cases, although latency is greatly improved.  Look at those latency
> changes!  From thousands of ms to hundreds of us in bonnie.  Wow.

So look at my latest numbers. This time preempt only, sorry.
If you need 2.4.10-pre10 only, too please ask.

> Even if you don't care about latency (I'm not an audio person or
> anything), these changes should be worth it.

I do. Or better, one of my friend's father will do some digital video editing 
with Linux:-)

> > Deleting with ReiserFS and the preempt kernel is GREAT!
>
> Good. I/O latency should be great now, with little change in
> throughput...

It is.

> > But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2)
> > or plaympeg together with dbench (16, 32). ReiserFS needs some preemption
> > fixes, too?
>
> You may still get some small hiccups ( < 1 second?) even with the
> preemption patch, as kernel locks prevent preemption (the patch can't
> guarentee low latency, just preemption outside of the locks).

Sadly 2-5 seconds at the beginning of dbench and during bonnie++ block 
operations (huge IO pressure, ~20% system, 3-5% user, 116308 kilobytes paged 
out).

> However, how bad was the hiccups with preemption disabled?  I have heard
> reports where it is 3-5sec at times.

Yes, nearly the same.

> As the kernel becomes more scalable (finer-grain locking), preemption
> will improve.  Past that, perhaps during 2.5, we can work on some other
> things to improve preemption.

Is this a ReiserFS only problem? Uninteruptable IO?

> > I've attached two small compressed bonnie++ HTML files.
>
> These were neat, thanks.

One more.

> Thank you for your feedback and support.  Stay current with the kernel
> and the preemption patches,

I will.

> and I will try to figure the ReiserFS crashes out.

No crashes for me only the stalls.

Regards,
	Dieter

2.4.10-pre10 + patch-rml-2.4.10-pre10-preempt-kernel-1

dbench-1.1 32
Throughput 26.2881 MB/sec (NB=32.8601 MB/sec  262.881 MBit/sec)
14.320u 54.140s 2:41.70 42.3%   0+0k 0+0io 911pf+0w
load: 2931
Throughput 27.3814 MB/sec (NB=34.2267 MB/sec  273.814 MBit/sec)
14.040u 55.920s 2:35.29 45.0%   0+0k 0+0io 911pf+0w
load: 2955

dbench 16
Throughput 32.9183 MB/sec (NB=41.1479 MB/sec  329.183 MBit/sec)
6.790u 26.580s 1:05.17 51.2%    0+0k 0+0io 511pf+0w

dbench 8
Throughput 34.8936 MB/sec (NB=43.617 MB/sec  348.936 MBit/sec)
3.300u 12.760s 0:31.27 51.3%    0+0k 0+0io 311pf+0w

dbench 4
Throughput 38.3568 MB/sec (NB=47.946 MB/sec  383.568 MBit/sec)
1.770u 6.160s 0:14.77 53.6%     0+0k 0+0io 211pf+0w
SunWave1>sync

dbench 2
Throughput 53.3066 MB/sec (NB=66.6333 MB/sec  533.066 MBit/sec)
0.940u 2.970s 0:05.95 65.7%     0+0k 0+0io 161pf+0w

dbench 1
Throughput 24.0354 MB/sec (NB=30.0442 MB/sec  240.354 MBit/sec)
0.470u 1.460s 0:06.49 29.7%     0+0k 0+0io 136pf+0w
Throughput 39.1102 MB/sec (NB=48.8878 MB/sec  391.102 MBit/sec)
0.400u 1.530s 0:04.38 44.0%     0+0k 0+0io 136pf+0w
Throughput 44.6889 MB/sec (NB=55.8611 MB/sec  446.889 MBit/sec)
0.500u 1.430s 0:03.96 48.7%     0+0k 0+0io 136pf+0w


Version 1.92a       ------Sequential Output------ --Sequential Input- 
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
%CP
SunWave1      1248M   107  96 16513  21  9004   9   163  98 27844  18 256.9   
3
Latency               115ms    2872ms    1743ms     121ms   64671us    3273ms
Version 1.92a       ------Sequential Create------ --------Random 
Create--------
SunWave1            -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP
                 16  6029  69 +++++ +++ 13252  94  6762  82 +++++ +++ 11342  
94
Latency               338ms   16041us   16309us   24248us     367us   17045us
load: 292

[-- Attachment #2: 2.4.10-pre10.html.bz2 --]
[-- Type: application/x-bzip2, Size: 908 bytes --]

Re: Feedback on preemptible kernel patch

[Daniel Phillips]

On September 18, 2001 06:06 am, Dieter Nützel wrote:
> Am Samstag, 15. September 2001 07:14 schrieb Robert Love:
> > On Sat, 2001-09-15 at 00:25, Dieter Nützel wrote:
> > > It seems to be that kswap put some additional "load" on the disk from
> > > time to time. Or is it the ReiserFS thing, again?
> >
> > I don't think its related to ReiserFS.
> 
> I think you are right.

They were bounce buffers, meaning they can come from any page cache io, and I 
saw at least one report with the same symptoms without the preempt patch.  I 
don't think it's your problem.

--
Daniel

Re: Feedback on preemptible kernel patch

[Roger Larsson]

On Tuesday den 18 September 2001 06:06, Dieter Nützel wrote:
> Yes, but no crash or oops for me.
> "Only" some "stalls" during MPEG/Ogg-Vorbis playback (2-5 sec) :-(

> > > But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2)
> > > or plaympeg together with dbench (16, 32). ReiserFS needs some
> > > preemption fixes, too?
> >
> > You may still get some small hiccups ( < 1 second?) even with the
> > preemption patch, as kernel locks prevent preemption (the patch can't
> > guarentee low latency, just preemption outside of the locks).
>
> Sadly 2-5 seconds at the beginning of dbench and during bonnie++ block
> operations (huge IO pressure, ~20% system, 3-5% user, 116308 kilobytes
> paged out).
>
> > However, how bad was the hiccups with preemption disabled?  I have heard
> > reports where it is 3-5sec at times.
>
> Yes, nearly the same.
>

Do you run with the playback process reniced -N?
It should really run with a low SCHED_FIFO or SCHED_RT policy.
But renicing it with a negative value gives some of the benefits...
(but you need to run as root)
In addition to this the program might need to lock its pages down - the
only thing I can think of that could cause several seconds delay would
be if it has been swapped out...

/RogerL

-- 
Roger Larsson
Skellefteå
Sweden

Re: Feedback on preemptible kernel patch

[Robert Love]

On Tue, 2001-09-18 at 14:18, Roger Larsson wrote:
> Do you run with the playback process reniced -N?
> It should really run with a low SCHED_FIFO or SCHED_RT policy.
> But renicing it with a negative value gives some of the benefits...
> (but you need to run as root)
> In addition to this the program might need to lock its pages down - the
> only thing I can think of that could cause several seconds delay would
> be if it has been swapped out...

Certainly giving it a higher priority should improve results (especially
with preemption), but the application should receive a fair amount of
process attention as it is, as it is TASK_RUNNABLE at all times and the
disk I/O should be routinely preempted.  I am interested how much
renicing it helps, though.

Now, if it has to swap pages, that is a very good point.  I tend to
blame this, or perhaps something with a long held lock (the audio
driver?) for the blips.

Its so hard to tell swap/VM issues now with all the VM work, sadly...:)

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Dieter Nützel]

Am Mittwoch, 19. September 2001 01:31 schrieb Robert Love:
> On Tue, 2001-09-18 at 14:18, Roger Larsson wrote:
> > Do you run with the playback process reniced -N?
> > It should really run with a low SCHED_FIFO or SCHED_RT policy.
> > But renicing it with a negative value gives some of the benefits...
> > (but you need to run as root)
> > In addition to this the program might need to lock its pages down - the
> > only thing I can think of that could cause several seconds delay would
> > be if it has been swapped out...
>
> Certainly giving it a higher priority should improve results (especially
> with preemption), but the application should receive a fair amount of
> process attention as it is, as it is TASK_RUNNABLE at all times and the
> disk I/O should be routinely preempted.  I am interested how much
> renicing it helps, though.

Nearly zero :-)

> Now, if it has to swap pages, that is a very good point.  I tend to
> blame this, or perhaps something with a long held lock (the audio
> driver?) for the blips.

System didn't go into swap during whole test, sorry.

> Its so hard to tell swap/VM issues now with all the VM work, sadly...:)

I point to the second;-)

-Dieter

Re: Feedback on preemptible kernel patch

[Robert Love]

On Tue, 2001-09-18 at 00:06, Dieter Nützel wrote:
> Am Samstag, 15. September 2001 07:14 schrieb Robert Love:
> > Are you seeing any specific problems, now?  With the latest preemption
> > patch on 2.4.10-pre9, do you crash? oops?
> 
> No, nothing with 2.4.10-pre9 + patch-rml-2.4.10-pre9-preempt-kernel-1 and 
> your MMX/3Dnow! fix.
> 
> 2.4.10-pre10 + patch-rml-2.4.10-pre10-preempt-kernel-1 seems to be a winner!
> 
> See my results below.

Excellent.  Note, 2.4.10-pre11 patches are out, but I don't know how
stable it is.  I am not sure I agree with ripping out the VM at this
moment.

Personally, I am using 2.4.9-ac12.  Patches are going up soon.

> > The only outstanding issue now is ReiserFS issues.
> 
> Yes, but no crash or oops for me.
> "Only" some "stalls" during MPEG/Ogg-Vorbis playback (2-5 sec) :-(

The ReiserFS issue may even be a non-issue.  Too much is going on right
now to figure that out.  I am going to keep going through it, though.

If you don't crash, its not an issue for you, at least.

> > > It seems to be that kswap put some additional "load" on the disk from
> > > time to time. Or is it the ReiserFS thing, again?
> >
> > I don't think its related to ReiserFS.
> 
> I think you are right.
> 
> > What sort of activity are you seeing?  How often?  How do you know its
> > kswapd?
> 
> I saw it with "top" at the first line (but only some few percent).
> It was during untarring some mid-sized archives (DRI) which took normally ~10 
> sec, but with kswap and 2.4.9-pre9+your patches ~30 sec. Even "sync" needed 
> some additional seconds.

I don't know what to make of this.  Your's is the first report.

> Are there some reschedule/context switch (kernel lock release) statements 
> missing in ReiserFS?

Actually, if ReiserFS was missing lock statements it would be faster :)
(but then crash, of course)

> Is this possible? Chris?
> 
> > I am glad the patch fixed it, the final version of that is going into
> > the next preemption patch.  Stay tuned.
> 
> I am very happy with patch-rml-2.4.10-pre10-preempt-kernel-1.

I am very glad.  Keep following the patches.

> > These results are pretty good.  Throughput seems down 2-3% in many
> > cases, although latency is greatly improved.  Look at those latency
> > changes!  From thousands of ms to hundreds of us in bonnie.  Wow.
> 
> So look at my latest numbers. This time preempt only, sorry.
> If you need 2.4.10-pre10 only, too please ask.

The numbers look very good, comparing them to your previous posted
results.

Next time you benchmark (for a future kernel, say), I do indeed like
seeing the non-preempt benchmarks so I can gauge things.  I realize its
a pain to compile and boot multiple kernels, though.

> > Even if you don't care about latency (I'm not an audio person or
> > anything), these changes should be worth it.
> 
> I do. Or better, one of my friend's father will do some digital video editing 
> with Linux:-)

Great :)

> > > Deleting with ReiserFS and the preempt kernel is GREAT!
> >
> > Good. I/O latency should be great now, with little change in
> > throughput...
> 
> It is.
> 
> > > But I get some hiccup during noatun (mp3, ogg, etc. player for KDE-2.2)
> > > or plaympeg together with dbench (16, 32). ReiserFS needs some preemption
> > > fixes, too?
> >
> > You may still get some small hiccups ( < 1 second?) even with the
> > preemption patch, as kernel locks prevent preemption (the patch can't
> > guarentee low latency, just preemption outside of the locks).
> 
> Sadly 2-5 seconds at the beginning of dbench and during bonnie++ block 
> operations (huge IO pressure, ~20% system, 3-5% user, 116308 kilobytes paged 
> out).
> 
> > However, how bad was the hiccups with preemption disabled?  I have heard
> > reports where it is 3-5sec at times.
> 
> Yes, nearly the same.

Hm, these we need to figure out.  We need to find what locks are held
too long or perhaps improperly -- stalls that large should not occur.

You don't use ALSA drivers, do you?

> > As the kernel becomes more scalable (finer-grain locking), preemption
> > will improve.  Past that, perhaps during 2.5, we can work on some other
> > things to improve preemption.
> 
> Is this a ReiserFS only problem? Uninteruptable IO?

No, ReiserFS is a good design.  This is in general -- in many places we
hold a very large lock -- ie, lock a whole subsystem from concurrent
access.  What we can do is lock finer and finer items, ie individual
structures, and use read/write and read locks appropriately.

Past that, we can look into replacing the use of SMP spinlocks with
other concurrency primitives for preemption.

> > > I've attached two small compressed bonnie++ HTML files.
> >
> > These were neat, thanks.
> 
> One more.
> 
> > Thank you for your feedback and support.  Stay current with the kernel
> > and the preemption patches,
> 
> I will.
> 
> > and I will try to figure the ReiserFS crashes out.
> 
> No crashes for me only the stalls.

Good.

Again, thanks a bunch for the feedback and benchmarks.  I will keep
looking into ReiserFS, but it may indeed be a non-issue.

You can keep up to date at http://tech9.net/rml/linux ... as I said, new
patches are going up in a moment.

Take care,

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Dieter Nützel]

Robert Love wrote:

> Hi Arjan,
>
> first, highmem is fixed and the original patch you have from me is good.
> second, Daniel Phillips gave me some feedback into how to figure out the
> VM error.  I am working on it, although just the VM potential

Good to hear.

> -- ReiserFS may be another problem.

Can't wait for that.

> third, you may be experiencing problems with a kernel optimized for
> Athlon.  this may or may not be related to the current issues with an
> Athlon-optimized kernel.  Basically, functions in arch/i386/lib/mmx.c
> seem to need some locking to prevent preemption.  I have a basic patch
> and we are working on a final one.

Can you please send this stuff along to me?
You know I own an Athlon (since yester Athlon II 1 GHz :-) and need some 
input...

Mobo is MSI MS-6167 Rev 1.0B (AMD Irongate C4, yes the very first one)

Kernel with preempt patch and mmx/3dnow! optimization crash randomly.
Never had that (without preempt) during the last two years.

Thanks,
	Dieter

Re: Feedback on preemptible kernel patch

[Robert Love]

Arjan, Jordan, and anyone using preemption and highmem:

Can you _please_ test the following patch?  It is my final version of
the highmem patch, and the one I would like to include in the preempt
patch itself.

It should be faster than the patch you have been using before, the locks
in the previous patch were held for the duration that the entire page
remained map.  I don't see any reason for that.

I would appreciate if you could patch your kernel and let me know.  I am
putting 2.4.10-pre8 patches up at http://tech9.net/rml/linux shortly. 
2.4.9-ac10 patches are there, too.

If you test, please reply with: kernel version, do you lock/oops with
highmem enabled but no extra patch (you should), do you lock/oops with
highmem enabled and the original highmem patch (you should not), and do
you lock/oops/something with this new highmem patch (I hope not). 
Obviously enable CONFIG_PREEMPT and CONFIG_HIGHMEM, and test well. 
Please tell me if CONFIG_SMP is enabled (that is another bag of fun...).

Thank you...




--- linux-not-rml/include/asm-i386/highmem.h	Tue Sep 11 17:54:32 2001
+++ linux/include/asm-i386/highmem.h	Tue Sep 11 18:42:13 2001
@@ -88,6 +88,8 @@
 	if (page < highmem_start_page)
 		return page_address(page);
 
+	ctx_sw_off();
+
 	idx = type + KM_TYPE_NR*smp_processor_id();
 	vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
 #if HIGHMEM_DEBUG
@@ -97,6 +99,8 @@
 	set_pte(kmap_pte-idx, mk_pte(page, kmap_prot));
 	__flush_tlb_one(vaddr);
 
+	ctx_sw_on();
+
 	return (void*) vaddr;
 }
 
@@ -106,6 +110,8 @@
 	unsigned long vaddr = (unsigned long) kvaddr;
 	enum fixed_addresses idx = type + KM_TYPE_NR*smp_processor_id();
 
+	ctx_sw_off();
+
 	if (vaddr < FIXADDR_START) // FIXME
 		return;
 
@@ -118,6 +124,8 @@
 	 */
 	pte_clear(kmap_pte-idx);
 	__flush_tlb_one(vaddr);
+
+	ctx_sw_on();
 #endif
 }
 


-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Feedback on preemptible kernel patch

[grue]

I am running 2.4.10-pre4 with the rml-preempt patch.
built and rebooted this on my workstation yesterday when I saw the patch
posted and it's been working great.

I'm running it on a dual P3-550 with 256MB ram with CONFIG_SMP and no
problems whatsoever although it hasn't been worked 'real' hard yet.
(load no higher than 4) ;)

Figured I'd give some positive feedback about the patch. If you want,
Rob, I could run some benchmarks on this against an unpatched kernel, or
if you have some ideas for me to really stress this thing to see if it
breaks, let me know.

--
Gregory Finch

Re: Feedback on preemptible kernel patch

[Robert Love]

On Sat, 2001-09-08 at 01:22, grue@lakesweb.com wrote:
> I am running 2.4.10-pre4 with the rml-preempt patch.
> built and rebooted this on my workstation yesterday when I saw the patch
> posted and it's been working great.

_Very_ glad to hear this.

> I'm running it on a dual P3-550 with 256MB ram with CONFIG_SMP and no
> problems whatsoever although it hasn't been worked 'real' hard yet.
> (load no higher than 4) ;)

I am surprised you have no problems with CONFIG_SMP=y &&
CONFIG_PREEMPT=y.  Promising.

> Figured I'd give some positive feedback about the patch. If you want,
> Rob, I could run some benchmarks on this against an unpatched kernel, or
> if you have some ideas for me to really stress this thing to see if it
> breaks, let me know.

I would love this.  We could use some SMP datapoints badly.

You can run some of the tests made especially for testing latency, like
an audio benchmark.  One such test is at
http://www.gardena.net/benno/linux/latencytest-0.42.tar.gz

Obviously a heavily tasked I/O benchmark is useful, I have used dbench
in the past (ftp://samba.org/pub/tridge/dbench/) (try it with 16
processes or so), but I have been told I should use bonnie.

You can time normal things, too. `time make dep clean bzImage' is always
a favorite :)

Under UP, enabling preemption helps all of this (the recent linuxdevices
article on preemption shows a 30x decrease in latency.).  Both myself
and Nigel have run dbench with good results for -16.  See
http://kpreempt.sourceforge.net/ for some more.

whatever you can... anyhow, thank you for the positive feedback.

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Arjan Filius]

Hello Robert,


I tried 2.4.10-pre4 with patch-rml-2.4.10-pre4-preempt-kernel-1.
But it seems to hit highmem (see below) (i do have 1.5GB ram)
2.4.10-pre4 plain runs just fine.

With the kernel option mem=850M the patched kernel boots an seems to run
fine. However i didn't do any stress testing yet, but i still notice
hickups while playing mp3 files at -10 nice level with mpg123 on a 1.1GHz
Athlon, and removing for example a _large_ file (reiser-on-lvm).

My syslog output with highmem:

Sep  8 18:10:16 sjoerd kernel: kernel BUG at /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95!
Sep  8 18:10:16 sjoerd kernel: invalid operand: 0000
Sep  8 18:10:16 sjoerd kernel: CPU:    0
Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_wp_page+636/1088]
Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 080bdd5c   ecx: f5764260   edx: f4d4c000
Sep  8 18:10:16 sjoerd kernel: esi: c26cca60   edi: ffffffff   ebp: c26ca134   esp: f4d4dec8
Sep  8 18:10:16 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep  8 18:10:16 sjoerd kernel: Process S11dhcpd (pid: 2507, stackpage=f4d4d000)
Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0 0000005f 080bdd5c f5805f00 ffffffff 00000001 c012437d
Sep  8 18:10:16 sjoerd kernel:        f5805f00 f4d49a00 080bdd5c f4c822f4 55d54065 f4d4c000 f4d49a00 f5805f00
Sep  8 18:10:16 sjoerd kernel:        f5805f1c c0111a17 f5805f00 f4d49a00 080bdd5c 00000001 f4d4c000 00000007
Sep  8 18:10:16 sjoerd kernel: Call Trace: [handle_mm_fault+141/224] [do_page_fault+375/1136] [do_page_fault+0/1136] [__mmdrop+58/64] [do_exit+595/640]
Sep  8 18:10:16 sjoerd kernel:    [error_code+52/64]
Sep  8 18:10:16 sjoerd kernel:
Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0 2b 05 ac ba 2a c0 69
Sep  8 18:10:16 sjoerd kernel: MAC unknown INTRUDERS?? (tf) IN=eth0 OUT= MAC= SRC=192.168.0.5 DST=192.168.0.255 LEN=241 TOS=0x02 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=138 DPT=138 LEN=221
Sep  8 18:10:16 sjoerd kernel: MAC unknown INTRUDERS?? (tf) IN=eth0 OUT= MAC= SRC=192.168.0.5 DST=192.168.0.255 LEN=96 TOS=0x02 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=137 DPT=137 LEN=76
Sep  8 18:10:16 sjoerd kernel: kernel BUG at /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95!
Sep  8 18:10:16 sjoerd kernel: invalid operand: 0000
Sep  8 18:10:16 sjoerd kernel: CPU:    0
Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_anonymous_page+130/368]
Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010286
Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 080c501c   ecx: f5764260   edx: f4d4c000
Sep  8 18:10:16 sjoerd kernel: esi: c26c4fec   edi: f5805f00   ebp: f4d497c0   esp: f4d4dea0
Sep  8 18:10:16 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep  8 18:10:16 sjoerd kernel: Process dhcpd (pid: 2508, stackpage=f4d4d000)
Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0 0000005f 080c501c f4d497c0 f5805f00 00000001 c012420f
Sep  8 18:10:16 sjoerd kernel:        f5805f00 f4d497c0 f4c63314 00000001 080c501c 080c501c f5805f00 ffffffff
Sep  8 18:10:16 sjoerd kernel:        00000001 c012434e f5805f00 f4d497c0 080c501c 00000001 f4c63314 f4d4c000
Sep  8 18:10:16 sjoerd kernel: Call Trace: [do_no_page+47/272] [handle_mm_fault+94/224] [do_page_fault+375/1136] [do_page_fault+0/1136] [do_munmap+86/640]
Sep  8 18:10:16 sjoerd kernel:    [fput+116/224] [do_brk+176/368] [sys_brk+187/240] [error_code+52/64]
Sep  8 18:10:16 sjoerd kernel:
Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0 2b 05 ac ba 2a c0 69
Sep  8 18:10:16 sjoerd kernel: kernel BUG at /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95!
Sep  8 18:10:16 sjoerd kernel: invalid operand: 0000
Sep  8 18:10:16 sjoerd kernel: CPU:    0
Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_anonymous_page+130/368]
Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 40017000   ecx: f5735f7c   edx: f4c88000
Sep  8 18:10:16 sjoerd kernel: esi: c26c9298   edi: f5805d80   ebp: f4c945c0   esp: f4c89dc8
Sep  8 18:10:16 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep  8 18:10:16 sjoerd kernel: Process python (pid: 2456, stackpage=f4c89000)
Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0 0000005f 40017000 f4c945c0 f5805d80 00000001 c012420f
Sep  8 18:10:16 sjoerd kernel:        f5805d80 f4c945c0 f4c9c05c 00000001 40017000 40017000 f5805d80 ffffffff
Sep  8 18:10:16 sjoerd kernel:        00000001 c012434e f5805d80 f4c945c0 40017000 00000001 f4c9c05c f4c88000
Sep  8 18:10:16 sjoerd kernel: Call Trace: [do_no_page+47/272] [handle_mm_fault+94/224] [do_page_fault+375/1136] [do_page_fault+0/1136] [block_read_full_page+240/688]
Sep  8 18:10:16 sjoerd kernel:    [error_code+52/64] [file_read_actor+113/224] [do_generic_file_read+505/1344] [generic_file_read+99/128] [file_read_actor+0/224] [sys_read+150/208]
Sep  8 18:10:16 sjoerd kernel:    [system_call+51/56]
Sep  8 18:10:16 sjoerd kernel:
Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0 2b 05 ac ba 2a c0 69
Sep  8 18:10:16 sjoerd kernel: kernel BUG at /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95!
Sep  8 18:10:16 sjoerd kernel: kernel BUG at /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95!
Sep  8 18:10:16 sjoerd kernel: invalid operand: 0000
Sep  8 18:10:16 sjoerd kernel: CPU:    0
Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_wp_page+636/1088]
Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: bffff960   ecx: f5764260   edx: f4ce4000
Sep  8 18:10:16 sjoerd kernel: esi: c26d04d0   edi: ffffffff   ebp: c26ca4a8   esp: f4ce5ec8
Sep  8 18:10:16 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep  8 18:10:16 sjoerd kernel: Process rc (pid: 2514, stackpage=f4ce5000)
Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0 0000005f bffff960 f5805780 ffffffff 00000001 c012437d
Sep  8 18:10:16 sjoerd kernel:        f5805780 f4c54dc0 bffff960 f4ca8ffc 55e30065 f4ce4000 f4c54dc0 f5805780
Sep  8 18:10:16 sjoerd kernel:        f580579c c0111a17 f5805780 f4c54dc0 bffff960 00000001 f4ce4000 00000007
Sep  8 18:10:16 sjoerd kernel: Call Trace: [handle_mm_fault+141/224] [do_page_fault+375/1136] [do_page_fault+0/1136] [__mmdrop+58/64] [do_exit+595/640]
Sep  8 18:10:16 sjoerd kernel:    [error_code+52/64]
Sep  8 18:10:16 sjoerd kernel:
Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0 2b 05 ac ba 2a c0 69
Sep  8 18:10:16 sjoerd kernel: kernel BUG at /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95!
Sep  8 18:10:16 sjoerd kernel: invalid operand: 0000
Sep  8 18:10:16 sjoerd kernel: CPU:    0
Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[filemap_nopage+300/1344]
Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 00000001   ecx: f5764260   edx: f4c3e000
Sep  8 18:10:16 sjoerd kernel: esi: c297ac20   edi: 00000015   ebp: c270df9c   esp: f4c3fb30
Sep  8 18:10:16 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep  8 18:10:16 sjoerd kernel: Process ncpserv (pid: 2513, stackpage=f4c3f000)
Sep  8 18:10:16 sjoerd kernel: Stack: c02110b2 c0211160 0000005f 40016000 f4c54f00 f4c62140 00000001 00000019
Sep  8 18:10:16 sjoerd kernel:        f7af9960 f74f7a24 f74f7980 f4db9c40 c0124252 f4c54f00 40016000 00000001
Sep  8 18:10:16 sjoerd kernel:        400162a8 f4c62140 ffffffff 00000001 c012434e f4c62140 f4c54f00 400162a8
Sep  8 18:10:16 sjoerd kernel: Call Trace: [do_no_page+114/272] [handle_mm_fault+94/224] [do_page_fault+375/1136] [do_page_fault+0/1136] [file_read_actor+177/224]
Sep  8 18:10:16 sjoerd kernel:    [update_atime+68/80] [do_generic_file_read+1333/1344] [do_munmap+86/640] [update_atime+68/80] [error_code+52/64] [clear_user+46/64]
Sep  8 18:10:16 sjoerd kernel:    [padzero+28/32] [load_elf_interp+619/704] [load_elf_binary+1959/2704] [load_elf_binary+0/2704] [nfsd:__insmod_nfsd_O/lib/modules/2.4.10-pre4/kernel/fs/nfsd/nfsd+-13721617/96] [search_binary_handler+152/496]
Sep  8 18:10:16 sjoerd kernel:    [do_execve+380/496] [do_execve+403/496] [sys_execve+47/96] [system_call+51/56]
Sep  8 18:10:16 sjoerd kernel:
Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0 2b 05 ac ba 2a c0 69
Sep  8 18:10:16 sjoerd kernel: LOOUT REJECT TCP IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=356 TOS=0x02 PREC=0x00 TTL=64 ID=32512 PROTO=TCP SPT=32775 DPT=15607 WINDOW=32767 RES=0x00 ACK PSH FIN URGP=0
Sep  8 18:10:16 sjoerd kernel: invalid operand: 0000
Sep  8 18:10:16 sjoerd kernel: CPU:    0
Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_wp_page+636/1088]
Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 080b170c   ecx: f4ce4260   edx: f5946000
Sep  8 18:10:16 sjoerd kernel: esi: c26dec2c   edi: ffffffff   ebp: c26ca2cc   esp: f5947ec8
Sep  8 18:10:16 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep  8 18:10:16 sjoerd kernel: Process rc (pid: 156, stackpage=f5947000)
Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0 0000005f 080b170c f752a080 ffffffff 00000001 c012437d
Sep  8 18:10:16 sjoerd kernel:        f752a080 f75282c0 080b170c f59de2c4 56197065 f5946000 f75282c0 f752a080
Sep  8 18:10:16 sjoerd kernel:        f752a09c c0111a17 f752a080 f75282c0 080b170c 00000001 f5946000 00000007
Sep  8 18:10:16 sjoerd kernel: Call Trace: [handle_mm_fault+141/224] [do_page_fault+375/1136] [do_page_fault+0/1136] [copy_thread+136/160] [do_fork+1619/1792]
Sep  8 18:10:16 sjoerd kernel:    [write_chan+0/544] [sys_fork+20/32] [error_code+52/64]
Sep  8 18:10:16 sjoerd kernel:
Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0 2b 05 ac ba 2a c0 69
Sep  8 18:10:16 sjoerd kernel: kernel BUG at /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95!
Sep  8 18:10:16 sjoerd kernel: invalid operand: 0000
Sep  8 18:10:16 sjoerd kernel: CPU:    0
Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_wp_page+636/1088]
Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 080b04e0   ecx: f5735f7c   edx: c299a000
Sep  8 18:10:16 sjoerd kernel: esi: c2962850   edi: ffffffff   ebp: c292d82c   esp: c299bec8
Sep  8 18:10:16 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep  8 18:10:16 sjoerd kernel: Process init (pid: 1, stackpage=c299b000)
Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0 0000005f 080b04e0 f752a140 ffffffff 00000001 c012437d
Sep  8 18:10:16 sjoerd kernel:        f752a140 f7528180 080b04e0 f751a2c0 5f910065 c299a000 f7528180 f752a140
Sep  8 18:10:16 sjoerd kernel:        f752a15c c0111a17 f752a140 f7528180 080b04e0 00000001 c299a000 00000007
Sep  8 18:10:16 sjoerd kernel: Call Trace: [handle_mm_fault+141/224] [do_page_fault+375/1136] [do_page_fault+0/1136] [copy_thread+136/160] [do_fork+1619/1792]
Sep  8 18:10:16 sjoerd kernel:    [sys_fork+20/32] [error_code+52/64]
Sep  8 18:10:16 sjoerd kernel:
Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0 2b 05 ac ba 2a c0 69

-- 
Arjan Filius
mailto:iafilius@xs4all.nl

Re: Feedback on preemptible kernel patch

[safemode]

On Saturday 08 September 2001 13:33, Arjan Filius wrote:
> Hello Robert,
>
>
> I tried 2.4.10-pre4 with patch-rml-2.4.10-pre4-preempt-kernel-1.
> But it seems to hit highmem (see below) (i do have 1.5GB ram)
> 2.4.10-pre4 plain runs just fine.
>
> With the kernel option mem=850M the patched kernel boots an seems to run
> fine. However i didn't do any stress testing yet, but i still notice
> hickups while playing mp3 files at -10 nice level with mpg123 on a 1.1GHz
> Athlon, and removing for example a _large_ file (reiser-on-lvm).

Have you tried running it without an altered priority level (altered by you 
that is)?  I run freeamp without any special nice level and running it while 
dbench is pushing the system into 25 load or something else is throttling the 
system at 100% cpu and hogging 200MB of ram - it never skips.  With or 
without the patch.  Some programs actually suffer from changing from default 
priority.  i see that with hdparm -t sometimes.  This is on a 850Mhz athlon, 
by the way.  I find mpg123 to be slower than most other players.  Try some of 
them.  If you're console only, freeamp is both a console and gui mp3 player.  
I imagine xmms would be pretty fast too when you turn off any extras.


> My syslog output with highmem:
>
> Sep  8 18:10:16 sjoerd kernel: kernel BUG at
> /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95! Sep  8 18:10:16 sjoerd
> kernel: invalid operand: 0000
> Sep  8 18:10:16 sjoerd kernel: CPU:    0
> Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_wp_page+636/1088]
> Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
> Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 080bdd5c   ecx:
> f5764260   edx: f4d4c000 Sep  8 18:10:16 sjoerd kernel: esi: c26cca60  
> edi: ffffffff   ebp: c26ca134   esp: f4d4dec8 Sep  8 18:10:16 sjoerd
> kernel: ds: 0018   es: 0018   ss: 0018
> Sep  8 18:10:16 sjoerd kernel: Process S11dhcpd (pid: 2507,
> stackpage=f4d4d000) Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0
> 0000005f 080bdd5c f5805f00 ffffffff 00000001 c012437d Sep  8 18:10:16
> sjoerd kernel:        f5805f00 f4d49a00 080bdd5c f4c822f4 55d54065 f4d4c000
> f4d49a00 f5805f00 Sep  8 18:10:16 sjoerd kernel:        f5805f1c c0111a17
> f5805f00 f4d49a00 080bdd5c 00000001 f4d4c000 00000007 Sep  8 18:10:16
> sjoerd kernel: Call Trace: [handle_mm_fault+141/224]
> [do_page_fault+375/1136] [do_page_fault+0/1136] [__mmdrop+58/64]
> [do_exit+595/640] Sep  8 18:10:16 sjoerd kernel:    [error_code+52/64]
> Sep  8 18:10:16 sjoerd kernel:
> Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0
> 2b 05 ac ba 2a c0 69 Sep  8 18:10:16 sjoerd kernel: MAC unknown INTRUDERS??
> (tf) IN=eth0 OUT= MAC= SRC=192.168.0.5 DST=192.168.0.255 LEN=241 TOS=0x02
> PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=138 DPT=138 LEN=221 Sep  8 18:10:16
> sjoerd kernel: MAC unknown INTRUDERS?? (tf) IN=eth0 OUT= MAC=
> SRC=192.168.0.5 DST=192.168.0.255 LEN=96 TOS=0x02 PREC=0x00 TTL=64 ID=0 DF
> PROTO=UDP SPT=137 DPT=137 LEN=76 Sep  8 18:10:16 sjoerd kernel: kernel BUG
> at /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95! Sep  8 18:10:16
> sjoerd kernel: invalid operand: 0000
> Sep  8 18:10:16 sjoerd kernel: CPU:    0
> Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_anonymous_page+130/368]
> Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010286
> Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 080c501c   ecx:
> f5764260   edx: f4d4c000 Sep  8 18:10:16 sjoerd kernel: esi: c26c4fec  
> edi: f5805f00   ebp: f4d497c0   esp: f4d4dea0 Sep  8 18:10:16 sjoerd
> kernel: ds: 0018   es: 0018   ss: 0018
> Sep  8 18:10:16 sjoerd kernel: Process dhcpd (pid: 2508,
> stackpage=f4d4d000) Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0
> 0000005f 080c501c f4d497c0 f5805f00 00000001 c012420f Sep  8 18:10:16
> sjoerd kernel:        f5805f00 f4d497c0 f4c63314 00000001 080c501c 080c501c
> f5805f00 ffffffff Sep  8 18:10:16 sjoerd kernel:        00000001 c012434e
> f5805f00 f4d497c0 080c501c 00000001 f4c63314 f4d4c000 Sep  8 18:10:16
> sjoerd kernel: Call Trace: [do_no_page+47/272] [handle_mm_fault+94/224]
> [do_page_fault+375/1136] [do_page_fault+0/1136] [do_munmap+86/640] Sep  8
> 18:10:16 sjoerd kernel:    [fput+116/224] [do_brk+176/368]
> [sys_brk+187/240] [error_code+52/64] Sep  8 18:10:16 sjoerd kernel:
> Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0
> 2b 05 ac ba 2a c0 69 Sep  8 18:10:16 sjoerd kernel: kernel BUG at
> /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95! Sep  8 18:10:16 sjoerd
> kernel: invalid operand: 0000
> Sep  8 18:10:16 sjoerd kernel: CPU:    0
> Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_anonymous_page+130/368]
> Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
> Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 40017000   ecx:
> f5735f7c   edx: f4c88000 Sep  8 18:10:16 sjoerd kernel: esi: c26c9298  
> edi: f5805d80   ebp: f4c945c0   esp: f4c89dc8 Sep  8 18:10:16 sjoerd
> kernel: ds: 0018   es: 0018   ss: 0018
> Sep  8 18:10:16 sjoerd kernel: Process python (pid: 2456,
> stackpage=f4c89000) Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0
> 0000005f 40017000 f4c945c0 f5805d80 00000001 c012420f Sep  8 18:10:16
> sjoerd kernel:        f5805d80 f4c945c0 f4c9c05c 00000001 40017000 40017000
> f5805d80 ffffffff Sep  8 18:10:16 sjoerd kernel:        00000001 c012434e
> f5805d80 f4c945c0 40017000 00000001 f4c9c05c f4c88000 Sep  8 18:10:16
> sjoerd kernel: Call Trace: [do_no_page+47/272] [handle_mm_fault+94/224]
> [do_page_fault+375/1136] [do_page_fault+0/1136]
> [block_read_full_page+240/688] Sep  8 18:10:16 sjoerd kernel:   
> [error_code+52/64] [file_read_actor+113/224]
> [do_generic_file_read+505/1344] [generic_file_read+99/128]
> [file_read_actor+0/224] [sys_read+150/208] Sep  8 18:10:16 sjoerd kernel:  
>  [system_call+51/56]
> Sep  8 18:10:16 sjoerd kernel:
> Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0
> 2b 05 ac ba 2a c0 69 Sep  8 18:10:16 sjoerd kernel: kernel BUG at
> /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95! Sep  8 18:10:16 sjoerd
> kernel: kernel BUG at /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95!
> Sep  8 18:10:16 sjoerd kernel: invalid operand: 0000
> Sep  8 18:10:16 sjoerd kernel: CPU:    0
> Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_wp_page+636/1088]
> Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
> Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: bffff960   ecx:
> f5764260   edx: f4ce4000 Sep  8 18:10:16 sjoerd kernel: esi: c26d04d0  
> edi: ffffffff   ebp: c26ca4a8   esp: f4ce5ec8 Sep  8 18:10:16 sjoerd
> kernel: ds: 0018   es: 0018   ss: 0018
> Sep  8 18:10:16 sjoerd kernel: Process rc (pid: 2514, stackpage=f4ce5000)
> Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0 0000005f bffff960
> f5805780 ffffffff 00000001 c012437d Sep  8 18:10:16 sjoerd kernel:       
> f5805780 f4c54dc0 bffff960 f4ca8ffc 55e30065 f4ce4000 f4c54dc0 f5805780 Sep
>  8 18:10:16 sjoerd kernel:        f580579c c0111a17 f5805780 f4c54dc0
> bffff960 00000001 f4ce4000 00000007 Sep  8 18:10:16 sjoerd kernel: Call
> Trace: [handle_mm_fault+141/224] [do_page_fault+375/1136]
> [do_page_fault+0/1136] [__mmdrop+58/64] [do_exit+595/640] Sep  8 18:10:16
> sjoerd kernel:    [error_code+52/64]
> Sep  8 18:10:16 sjoerd kernel:
> Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0
> 2b 05 ac ba 2a c0 69 Sep  8 18:10:16 sjoerd kernel: kernel BUG at
> /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95! Sep  8 18:10:16 sjoerd
> kernel: invalid operand: 0000
> Sep  8 18:10:16 sjoerd kernel: CPU:    0
> Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[filemap_nopage+300/1344]
> Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
> Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 00000001   ecx:
> f5764260   edx: f4c3e000 Sep  8 18:10:16 sjoerd kernel: esi: c297ac20  
> edi: 00000015   ebp: c270df9c   esp: f4c3fb30 Sep  8 18:10:16 sjoerd
> kernel: ds: 0018   es: 0018   ss: 0018
> Sep  8 18:10:16 sjoerd kernel: Process ncpserv (pid: 2513,
> stackpage=f4c3f000) Sep  8 18:10:16 sjoerd kernel: Stack: c02110b2 c0211160
> 0000005f 40016000 f4c54f00 f4c62140 00000001 00000019 Sep  8 18:10:16
> sjoerd kernel:        f7af9960 f74f7a24 f74f7980 f4db9c40 c0124252 f4c54f00
> 40016000 00000001 Sep  8 18:10:16 sjoerd kernel:        400162a8 f4c62140
> ffffffff 00000001 c012434e f4c62140 f4c54f00 400162a8 Sep  8 18:10:16
> sjoerd kernel: Call Trace: [do_no_page+114/272] [handle_mm_fault+94/224]
> [do_page_fault+375/1136] [do_page_fault+0/1136] [file_read_actor+177/224]
> Sep  8 18:10:16 sjoerd kernel:    [update_atime+68/80]
> [do_generic_file_read+1333/1344] [do_munmap+86/640] [update_atime+68/80]
> [error_code+52/64] [clear_user+46/64] Sep  8 18:10:16 sjoerd kernel:   
> [padzero+28/32] [load_elf_interp+619/704] [load_elf_binary+1959/2704]
> [load_elf_binary+0/2704]
> [nfsd:__insmod_nfsd_O/lib/modules/2.4.10-pre4/kernel/fs/nfsd/nfsd+-13721617
>/96] [search_binary_handler+152/496] Sep  8 18:10:16 sjoerd kernel:   
> [do_execve+380/496] [do_execve+403/496] [sys_execve+47/96]
> [system_call+51/56] Sep  8 18:10:16 sjoerd kernel:
> Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0
> 2b 05 ac ba 2a c0 69 Sep  8 18:10:16 sjoerd kernel: LOOUT REJECT TCP IN=
> OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=356 TOS=0x02 PREC=0x00 TTL=64
> ID=32512 PROTO=TCP SPT=32775 DPT=15607 WINDOW=32767 RES=0x00 ACK PSH FIN
> URGP=0 Sep  8 18:10:16 sjoerd kernel: invalid operand: 0000
> Sep  8 18:10:16 sjoerd kernel: CPU:    0
> Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_wp_page+636/1088]
> Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
> Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 080b170c   ecx:
> f4ce4260   edx: f5946000 Sep  8 18:10:16 sjoerd kernel: esi: c26dec2c  
> edi: ffffffff   ebp: c26ca2cc   esp: f5947ec8 Sep  8 18:10:16 sjoerd
> kernel: ds: 0018   es: 0018   ss: 0018
> Sep  8 18:10:16 sjoerd kernel: Process rc (pid: 156, stackpage=f5947000)
> Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0 0000005f 080b170c
> f752a080 ffffffff 00000001 c012437d Sep  8 18:10:16 sjoerd kernel:       
> f752a080 f75282c0 080b170c f59de2c4 56197065 f5946000 f75282c0 f752a080 Sep
>  8 18:10:16 sjoerd kernel:        f752a09c c0111a17 f752a080 f75282c0
> 080b170c 00000001 f5946000 00000007 Sep  8 18:10:16 sjoerd kernel: Call
> Trace: [handle_mm_fault+141/224] [do_page_fault+375/1136]
> [do_page_fault+0/1136] [copy_thread+136/160] [do_fork+1619/1792] Sep  8
> 18:10:16 sjoerd kernel:    [write_chan+0/544] [sys_fork+20/32]
> [error_code+52/64] Sep  8 18:10:16 sjoerd kernel:
> Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0
> 2b 05 ac ba 2a c0 69 Sep  8 18:10:16 sjoerd kernel: kernel BUG at
> /usr/src/linux-2.4.10-pre4/include/asm/highmem.h:95! Sep  8 18:10:16 sjoerd
> kernel: invalid operand: 0000
> Sep  8 18:10:16 sjoerd kernel: CPU:    0
> Sep  8 18:10:16 sjoerd kernel: EIP:    0010:[do_wp_page+636/1088]
> Sep  8 18:10:16 sjoerd kernel: EFLAGS: 00010282
> Sep  8 18:10:16 sjoerd kernel: eax: 00000043   ebx: 080b04e0   ecx:
> f5735f7c   edx: c299a000 Sep  8 18:10:16 sjoerd kernel: esi: c2962850  
> edi: ffffffff   ebp: c292d82c   esp: c299bec8 Sep  8 18:10:16 sjoerd
> kernel: ds: 0018   es: 0018   ss: 0018
> Sep  8 18:10:16 sjoerd kernel: Process init (pid: 1, stackpage=c299b000)
> Sep  8 18:10:16 sjoerd kernel: Stack: c0210bd2 c0210cc0 0000005f 080b04e0
> f752a140 ffffffff 00000001 c012437d Sep  8 18:10:16 sjoerd kernel:       
> f752a140 f7528180 080b04e0 f751a2c0 5f910065 c299a000 f7528180 f752a140 Sep
>  8 18:10:16 sjoerd kernel:        f752a15c c0111a17 f752a140 f7528180
> 080b04e0 00000001 c299a000 00000007 Sep  8 18:10:16 sjoerd kernel: Call
> Trace: [handle_mm_fault+141/224] [do_page_fault+375/1136]
> [do_page_fault+0/1136] [copy_thread+136/160] [do_fork+1619/1792] Sep  8
> 18:10:16 sjoerd kernel:    [sys_fork+20/32] [error_code+52/64] Sep  8
> 18:10:16 sjoerd kernel:
> Sep  8 18:10:16 sjoerd kernel: Code: 0f 0b 83 c4 0c 8b 15 e8 2f 2a c0 89 f0
> 2b 05 ac ba 2a c0 69

Re: Feedback on preemptible kernel patch

[Robert Love]

On Sat, 2001-09-08 at 13:33, Arjan Filius wrote:
> I tried 2.4.10-pre4 with patch-rml-2.4.10-pre4-preempt-kernel-1.
> But it seems to hit highmem (see below) (i do have 1.5GB ram)
> 2.4.10-pre4 plain runs just fine.

going through this thread and now looking at the highmem code, it is
clear highmem is not going to be preempt safe.

until Nigel or I (or someone?) can go through it all and add appropriate
locks, its a bomb waiting to blow.

thank you for the feedback...stay tuned.

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Robert Love]

Arjan,

the following patch was written by Manfred Spraul to fix your highmem
bug.  I haven't had a chance to go over it, but I would like it if you
could test it.  It can't hurt.  Patch it on top of the preempt patch and
enable CONFIG_PREEMPT, CONFIG_HIGHMEM, and CONFIG_HIGHMEM_DEBUG.

let me know what happens...any relevant messages, etc. please pass
along. if it does work, id be curious if they are any slowdowns


--- highmem.h.prev      Sun Sep  9 08:59:04 2001
+++ highmem.h   Sun Sep  9 09:00:07 2001
@@ -88,6 +88,7 @@
        if (page < highmem_start_page)
                return page_address(page);

+       ctx_sw_off();
        idx = type + KM_TYPE_NR*smp_processor_id();
        vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
#if HIGHMEM_DEBUG
@@ -119,6 +120,7 @@
        pte_clear(kmap_pte-idx);
        __flush_tlb_one(vaddr);
#endif
+       ctx_sw_on();
}

#endif /* __KERNEL__ */



-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net
-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Arjan Filius]

Hello Robert,


I tried 2.4.10-pre4+preempt+this-patch.
Just booted up, and don't notice anything unusual.

On 9 Sep 2001, Robert Love wrote:

> Arjan,
>
> the following patch was written by Manfred Spraul to fix your highmem
> bug.  I haven't had a chance to go over it, but I would like it if you
> could test it.  It can't hurt.  Patch it on top of the preempt patch and
> enable CONFIG_PREEMPT, CONFIG_HIGHMEM, and CONFIG_HIGHMEM_DEBUG.

I found i do anly have a '#define HIGHMEM_DEBUG 1' in
./include/asm/highmem.h, which is default in 2.4.10-pre4.

>
> let me know what happens...any relevant messages, etc. please pass
> along. if it does work, id be curious if they are any slowdowns

Booting up, X, compiling kernel.. no problems.
For speed, i DO notice other processes seem not to wait on that one
programm which has much disk-access, so the (real) sluggish feeling has
gone. This is however with the preempt patch, and the ctx_sw_ patch below
seems only to affect stability in positive sense.

Can you advice what and how to test performance/latency?
The grafics/statistics on the websites you named are impressive..


Greatings,


>
>
> --- highmem.h.prev      Sun Sep  9 08:59:04 2001
> +++ highmem.h   Sun Sep  9 09:00:07 2001
> @@ -88,6 +88,7 @@
>         if (page < highmem_start_page)
>                 return page_address(page);
>
> +       ctx_sw_off();
>         idx = type + KM_TYPE_NR*smp_processor_id();
>         vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
> #if HIGHMEM_DEBUG
> @@ -119,6 +120,7 @@
>         pte_clear(kmap_pte-idx);
>         __flush_tlb_one(vaddr);
> #endif
> +       ctx_sw_on();
> }
>
> #endif /* __KERNEL__ */
>
>
>
>

-- 
Arjan Filius
mailto:iafilius@xs4all.nl

Re: Feedback on preemptible kernel patch

[Robert Love]

On Sun, 2001-09-09 at 17:07, Arjan Filius wrote:
> I tried 2.4.10-pre4+preempt+this-patch.
> Just booted up, and don't notice anything unusual.

very good so far...

> I found i do anly have a '#define HIGHMEM_DEBUG 1' in
> ./include/asm/highmem.h, which is default in 2.4.10-pre4.

OK, then no problem there.

> Booting up, X, compiling kernel.. no problems.

good...

> For speed, i DO notice other processes seem not to wait on that one
> programm which has much disk-access, so the (real) sluggish feeling has
> gone. This is however with the preempt patch, and the ctx_sw_ patch below
> seems only to affect stability in positive sense.

_GREAT_ ... now, the reason I asked if you notice any new slowdowns is
exactly what you seem to realize: I feared the ctx_sw patch may cause
obvious slowdown.  This could be because the ctw_on/offs are in the
wrong place, and causing much to much locking.

It seems like you notice no problems, and I am happy.

I am glad to hear this news, I am going to take a look at highmem's code
and then integrate a final solution into the preemption patch.

> Can you advice what and how to test performance/latency?
> The grafics/statistics on the websites you named are impressive..

Sure, you can run dbench <ftp://samba.org/pub/tridge/dbench/> try it
with around 16 threads (dbench -16).  You might also want to try playing
an mp3 in the background during this.  Notice it should not have large
skips (one user reporting 3s skipps dropping to 0.5s and 0s).

You can run the audio latency test
<http://www.gardena.net/benno/linux/latencytest-0.42.tar.gz>, although I
heard there are problems compiling it from some other preemption users.

Finally, simply time a kernel compile `time make dep clean bzImage' ...

We can use these for preemption enabled and disabled, highmem enabled
and disabled, etc...

Thank you for your help, 

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Arjan Filius]

Hi,

After my succes report i _do_ noticed something unusual:

I'm not sure it's preempt related, but you wanted feedback :)

Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
Sep  9 23:08:02 sjoerd last message repeated 93 times
Sep  9 23:08:02 sjoerd kernel: cation failed (gfp=0x70/1).
Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
Sep  9 23:08:02 sjoerd last message repeated 281 times

This is at the very moment i make a ppp connection to internet, and
get/set the time with netdate (for the first time after a reboot).
I didn't see this a second time (yet).

Btw this is 2.4.10-pre4+preempt-patch+pacht-below.

Greetings,

On 9 Sep 2001, Robert Love wrote:

> Arjan,
>
> the following patch was written by Manfred Spraul to fix your highmem
> bug.  I haven't had a chance to go over it, but I would like it if you
> could test it.  It can't hurt.  Patch it on top of the preempt patch and
> enable CONFIG_PREEMPT, CONFIG_HIGHMEM, and CONFIG_HIGHMEM_DEBUG.
>
> let me know what happens...any relevant messages, etc. please pass
> along. if it does work, id be curious if they are any slowdowns
>
>
> --- highmem.h.prev      Sun Sep  9 08:59:04 2001
> +++ highmem.h   Sun Sep  9 09:00:07 2001
> @@ -88,6 +88,7 @@
>         if (page < highmem_start_page)
>                 return page_address(page);
>
> +       ctx_sw_off();
>         idx = type + KM_TYPE_NR*smp_processor_id();
>         vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
> #if HIGHMEM_DEBUG
> @@ -119,6 +120,7 @@
>         pte_clear(kmap_pte-idx);
>         __flush_tlb_one(vaddr);
> #endif
> +       ctx_sw_on();
> }
>
> #endif /* __KERNEL__ */
>
>
>
>

-- 
Arjan Filius
mailto:iafilius@xs4all.nl

Re: Feedback on preemptible kernel patch

[Robert Love]

On Sun, 2001-09-09 at 17:23, Arjan Filius wrote:
> After my succes report i _do_ noticed something unusual:
> 
> I'm not sure it's preempt related, but you wanted feedback :)
> 
> Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
> Sep  9 23:08:02 sjoerd last message repeated 93 times
> Sep  9 23:08:02 sjoerd kernel: cation failed (gfp=0x70/1).
> Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
> Sep  9 23:08:02 sjoerd last message repeated 281 times
> 
> This is at the very moment i make a ppp connection to internet, and
> get/set the time with netdate (for the first time after a reboot).
> I didn't see this a second time (yet).
> 

damn, I was exciting we had solved everything :)

actually, I am not confident of what could cause these results.  the
2.4.10-pre is going through another set of changes it should not, and
one of them concerns exactly what you are reporting.

SO, I suggest two options: try pre6.  I don't have patches yet, but I
will diff them soon.  pre5 should apply fairly cleanly, anyhow.

Even better, try 2.4.9-ac10.  It is what I use, and there seems to be
less reported problems.  Plus, Alan is not messing with all the VM work
Linus is playing with right now.  Patches for 2.4.9-ac10 are available.

Both can be had at:
http://tech9.net/rml/linux/

I am curious if you see the error again, and what seems to cause it, but
honestly there is too much work being done in 2.4.10-pre to figure
things out.

Nevertheless, I will look into it -- keep me posted.

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Daniel Phillips]

On September 9, 2001 11:37 pm, Robert Love wrote:
> On Sun, 2001-09-09 at 17:23, Arjan Filius wrote:
> > After my succes report i _do_ noticed something unusual:
> > 
> > I'm not sure it's preempt related, but you wanted feedback :)
> > 
> > Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
> > Sep  9 23:08:02 sjoerd last message repeated 93 times
> > Sep  9 23:08:02 sjoerd kernel: cation failed (gfp=0x70/1).
> > Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
> > Sep  9 23:08:02 sjoerd last message repeated 281 times
> > 
> > This is at the very moment i make a ppp connection to internet, and
> > get/set the time with netdate (for the first time after a reboot).
> > I didn't see this a second time (yet).
> > 
> 
> damn, I was exciting we had solved everything :)
> 
> actually, I am not confident of what could cause these results.  the
> 2.4.10-pre is going through another set of changes it should not, and
> one of them concerns exactly what you are reporting.

This may not be your fault.  It's a GFP_NOFS recursive allocation - this
comes either from grow_buffers or ReiserFS, probably the former.  In
either case, it means we ran completely out of free pages, even though
the caller is willing to wait.  Hmm.  It smells like a loophole in vm
scanning.

--
Daniel

Re: Feedback on preemptible kernel patch

[Jeremy Zawodny]

On Mon, Sep 10, 2001 at 05:24:36AM +0200, Daniel Phillips wrote:
> On September 9, 2001 11:37 pm, Robert Love wrote:
> > On Sun, 2001-09-09 at 17:23, Arjan Filius wrote:
> > > After my succes report i _do_ noticed something unusual:
> > > 
> > > I'm not sure it's preempt related, but you wanted feedback :)
> > > 
> > > Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
> > > Sep  9 23:08:02 sjoerd last message repeated 93 times
> > > Sep  9 23:08:02 sjoerd kernel: cation failed (gfp=0x70/1).
> > > Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
>
> 
> This may not be your fault.  It's a GFP_NOFS recursive allocation -
> this comes either from grow_buffers or ReiserFS, probably the
> former.  In either case, it means we ran completely out of free
> pages, even though the caller is willing to wait.  Hmm.  It smells
> like a loophole in vm scanning.

I've seen that error on a couple 2.4.9 systems at work.  It's
certainly VM related, 'cause it doesn't happen when I disable swap on
them.  I've disabled it for performance reasons (the VM system is a
little retarded in 2.4.x IMHO, so I'm just not letting it swap).

Jeremy
-- 
Jeremy D. Zawodny     |  Perl, Web, MySQL, Linux Magazine, Yahoo!
<Jeremy@Zawodny.com>  |  http://jeremy.zawodny.com/

Re: Feedback on preemptible kernel patch

[Robert Love]

On Sun, 2001-09-09 at 23:24, Daniel Phillips wrote:
> This may not be your fault.  It's a GFP_NOFS recursive allocation - this
> comes either from grow_buffers or ReiserFS, probably the former.  In
> either case, it means we ran completely out of free pages, even though
> the caller is willing to wait.  Hmm.  It smells like a loophole in vm
> scanning.

I am not a VM hacker -- can you tell me where to start? what do you
suspect it is?

If the user stops seeing the error with preemption disabled, is your
theory nulled, or does that just mean the problem is agitated by
preemption?

I don't think Arjan was using ReiserFS, so its from grow_buffers...

I appreciate your help.

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Daniel Phillips]

On September 10, 2001 07:09 am, Robert Love wrote:
> On Sun, 2001-09-09 at 23:24, Daniel Phillips wrote:
> > This may not be your fault.  It's a GFP_NOFS recursive allocation - this
> > comes either from grow_buffers or ReiserFS, probably the former.  In
> > either case, it means we ran completely out of free pages, even though
> > the caller is willing to wait.  Hmm.  It smells like a loophole in vm
> > scanning.
> 
> I am not a VM hacker -- can you tell me where to start? what do you
> suspect it is?
> 
> If the user stops seeing the error with preemption disabled, is your
> theory nulled, or does that just mean the problem is agitated by
> preemption?
> 
> I don't think Arjan was using ReiserFS, so its from grow_buffers...
> 
> I appreciate your help.

The first thing to check is whether memory is really exhausted at the
time the errors are logged (cat /proc/meminfo).  Then you want to see
which paths in __alloc_pages could possibly allow this PF_MEMALLOC +
GFP_WAIT allocation request to drop all the way through without being
serviced.  Sorry, I haven't had time to do that and won't for a few
days.  Even if you triggered it, it is probably a hole in the scan
logic.  We have __GFP_WAIT, so it should wait.

Here's a hint, look very critically at this part of page_alloc.c:

455    /*
456     * Fail in case no progress was made and the
457     * allocation may not be able to block on IO.
458     */
459    return NULL;

--
Daniel

Re: Feedback on preemptible kernel patch

[Arjan Filius]

Hello,


On 10 Sep 2001, Robert Love wrote:

> On Sun, 2001-09-09 at 23:24, Daniel Phillips wrote:
> > This may not be your fault.  It's a GFP_NOFS recursive allocation - this
> > comes either from grow_buffers or ReiserFS, probably the former.  In
> > either case, it means we ran completely out of free pages, even though
> > the caller is willing to wait.  Hmm.  It smells like a loophole in vm
> > scanning.
>
> I am not a VM hacker -- can you tell me where to start? what do you
> suspect it is?
>
> If the user stops seeing the error with preemption disabled, is your
> theory nulled, or does that just mean the problem is agitated by
> preemption?
>
> I don't think Arjan was using ReiserFS, so its from grow_buffers...

Yes I am using reiserfs (for "ages"). better said, reiser on LVM.

Small discription of my system and used setup:
scsi-disk,scsi-cdrom,ide-disk,ide-scsi,ext2,reiser, iptables, ipv6,
acenic-Gbit-ethernet, ramdisk, highmem (1.5GB-ram), Athlon 1.1GHz, Asus
a7v MB (via).


 Greatings,


>
> I appreciate your help.
>
>

-- 
Arjan Filius
mailto:iafilius@xs4all.nl

Re: Feedback on preemptible kernel patch

[Robert Love]

On Mon, 2001-09-10 at 17:29, Arjan Filius wrote:
> Yes I am using reiserfs (for "ages"). better said, reiser on LVM.
> 
> Small discription of my system and used setup:
> scsi-disk,scsi-cdrom,ide-disk,ide-scsi,ext2,reiser, iptables, ipv6,
> acenic-Gbit-ethernet, ramdisk, highmem (1.5GB-ram), Athlon 1.1GHz, Asus
> a7v MB (via).

Hi Arjan,

first, highmem is fixed and the original patch you have from me is good.
second, Daniel Phillips gave me some feedback into how to figure out the
VM error.  I am working on it, although just the VM potential --
ReiserFS may be another problem.

third, you may be experiencing problems with a kernel optimized for
Athlon.  this may or may not be related to the current issues with an
Athlon-optimized kernel.  Basically, functions in arch/i386/lib/mmx.c
seem to need some locking to prevent preemption.  I have a basic patch
and we are working on a final one.

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[george anzinger]

Robert Love wrote:
> 
> On Mon, 2001-09-10 at 17:29, Arjan Filius wrote:
> > Yes I am using reiserfs (for "ages"). better said, reiser on LVM.
> >
> > Small discription of my system and used setup:
> > scsi-disk,scsi-cdrom,ide-disk,ide-scsi,ext2,reiser, iptables, ipv6,
> > acenic-Gbit-ethernet, ramdisk, highmem (1.5GB-ram), Athlon 1.1GHz, Asus
> > a7v MB (via).
> 
> Hi Arjan,
> 
> first, highmem is fixed and the original patch you have from me is good.
> second, Daniel Phillips gave me some feedback into how to figure out the
> VM error.  I am working on it, although just the VM potential --
> ReiserFS may be another problem.
> 
> third, you may be experiencing problems with a kernel optimized for
> Athlon.  this may or may not be related to the current issues with an
> Athlon-optimized kernel.  Basically, functions in arch/i386/lib/mmx.c
> seem to need some locking to prevent preemption.  I have a basic patch
> and we are working on a final one.
> 
Right, the same problem as using floating point in the kernel (mmx uses
the FP regs and they are not saved).  The question is: Just how long do
these routines take?  If it is very long it may be best to just say no. 
One way would be to always pretend that the "in_interrupt" flag is set. 
I think possibly some routines are short and the switch off/ switch on
pair is right, but for the long ones, well the preemption patch is
supposed to make the kernel more preemptable, not less.  Any one have
execution times for these functions?

George

Re: Feedback on preemptible kernel patch

[Robert Love]

On Fri, 2001-09-14 at 03:30, george anzinger wrote:
> Right, the same problem as using floating point in the kernel (mmx uses
> the FP regs and they are not saved).

Right, and I suspect we will find more problems of this type as we go
on.  In fact, the more general case "things that are SMP-safe but not
preempt safe" will be issues, too.  The highmem bug was one of these -
code that was SMP-safe but did not have lock points because it was
per-CPU code.  Preemption ruins all that.

> The question is: Just how long do these routines take?  If it is very long
> it may be best to just say no. One way would be to always pretend that
> the"in_interrupt" flag is set.  I think possibly some routines are
> short and the switch off/ switch on pair is right, but for the long ones,
> well the preemption patch is supposed to make the kernel more preemptable,
> not less.  Any one have execution times for these functions?

Well, its the routines in arch/i386/lib/mmx.c -- and just the ones that
call kernel_begin/end_fpu.  My patch pushes a ctx_sw_off/on pair into
those functions.  Anyhow, if you look, they aren't too long.

However, I agree that we may be destroying our purpose here.  A user of
the patch actually put together a patch that will disable the CONFIG to
use the fast MMX memcpy stuff if preemption was enabled.  He benchmarked
against the two and I can send you those results when I sort through
them.

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: Feedback on preemptible kernel patch

[Arjan Filius]

Hello,

On 9 Sep 2001, Robert Love wrote:

> On Sun, 2001-09-09 at 17:23, Arjan Filius wrote:
> > After my succes report i _do_ noticed something unusual:
> >
> > I'm not sure it's preempt related, but you wanted feedback :)
> >
> > Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
> > Sep  9 23:08:02 sjoerd last message repeated 93 times
> > Sep  9 23:08:02 sjoerd kernel: cation failed (gfp=0x70/1).
> > Sep  9 23:08:02 sjoerd kernel: __alloc_pages: 0-order allocation failed (gfp=0x70/1).
> > Sep  9 23:08:02 sjoerd last message repeated 281 times
> >
> > This is at the very moment i make a ppp connection to internet, and
> > get/set the time with netdate (for the first time after a reboot).
> > I didn't see this a second time (yet).
> >
>
> damn, I was exciting we had solved everything :)
>
> actually, I am not confident of what could cause these results.  the
> 2.4.10-pre is going through another set of changes it should not, and
> one of them concerns exactly what you are reporting.
>
> SO, I suggest two options: try pre6.  I don't have patches yet, but I
> will diff them soon.  pre5 should apply fairly cleanly, anyhow.
>
> Even better, try 2.4.9-ac10.  It is what I use, and there seems to be
> less reported problems.  Plus, Alan is not messing with all the VM work
> Linus is playing with right now.  Patches for 2.4.9-ac10 are available.
>
> Both can be had at:
> http://tech9.net/rml/linux/
>
> I am curious if you see the error again, and what seems to cause it, but
> honestly there is too much work being done in 2.4.10-pre to figure
> things out.
>
> Nevertheless, I will look into it -- keep me posted.
>

It took some time, but here the results.

I do seem to have no problem at all with plain 2.4.10-pre6 and 2.4.9-ac10.

As adviced i tried the 2.4.9-ac10 with the preempt patch (without extra
lock-patch posted on lkm)
Just booted the system and i noticed various (4x) "invalid operand"'s, and
the 4 seconds from my systlog are below.

Sep 11 19:47:31 sjoerd kernel: klogd 1.3-3, log source = /proc/kmsg started.
Sep 11 19:47:31 sjoerd kernel: Inspecting /boot/System.map-2.4.9-ac10
Sep 11 19:47:31 sjoerd kernel: Loaded 13938 symbols from /boot/System.map-2.4.9-ac10.
Sep 11 19:47:31 sjoerd kernel: Symbols match kernel version 2.4.9.
Sep 11 19:47:31 sjoerd kernel: Loaded 584 symbols from 42 modules.
Sep 11 19:47:31 sjoerd kernel: IPv6 v0.8 for NET4.0
Sep 11 19:47:31 sjoerd kernel: IPv6 over IPv4 tunneling driver
Sep 11 19:47:31 sjoerd kernel: divert: not allocating divert_blk for non-ethernet device sit0
Sep 11 19:47:31 sjoerd kernel: divert: allocating divert_blk for eth0
Sep 11 19:47:31 sjoerd kernel: acenic.c: v0.81 04/20/2001  Jes Sorensen, linux-acenic@SunSITE.dk
Sep 11 19:47:31 sjoerd kernel:                             http://home.cern.ch/~jes/gige/acenic.html
Sep 11 19:47:31 sjoerd kernel: eth0: 3Com 3C985 Gigabit Ethernet at 0xdd800000, irq 11
Sep 11 19:47:31 sjoerd kernel:   Tigon II (Rev. 6), Firmware: 12.4.11, MAC: 00:60:08:f6:1d:5b
Sep 11 19:47:31 sjoerd kernel:   PCI cache line size set incorrectly (32 bytes) by BIOS/FW, correcting to 64
Sep 11 19:47:31 sjoerd kernel:   PCI bus width: 32 bits, speed: 33MHz, latency: 64 clks
Sep 11 19:47:31 sjoerd kernel:   Disabling PCI memory write and invalidate
Sep 11 19:47:31 sjoerd kernel: eth0: Firmware up and running
Sep 11 19:47:31 sjoerd kernel: eth0: Optical link UP (Full Duplex, Flow Control: TX RX)
Sep 11 19:47:31 sjoerd kernel: ip_conntrack (8192 buckets, 65536 max)
Sep 11 19:47:31 sjoerd kernel: ip_tables: (c)2000 Netfilter core team
Sep 11 19:47:31 sjoerd kernel: LOIN ICMP rate to high IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=90 TOS=0x02 PREC=0xC0 TTL=255 ID=32406 PROTO=ICMP TYPE=3 CODE=3 [SRC=127.0.0.1 DST=127.0.0.1 LEN=62 TOS=0x02 PREC=0x00 TTL=64 ID=6694 DF PROTO=UDP SPT=32768 DPT=53 LEN=42 ]
Sep 11 19:47:31 sjoerd kernel: LOIN ICMP rate to high IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=102 TOS=0x02 PREC=0xC0 TTL=255 ID=32413 PROTO=ICMP TYPE=3 CODE=3 [SRC=127.0.0.1 DST=127.0.0.1 LEN=74 TOS=0x02 PREC=0x00 TTL=64 ID=6783 DF PROTO=UDP SPT=32769 DPT=53 LEN=54 ]
Sep 11 19:47:31 sjoerd kernel: LOIN ICMP rate to high IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=92 TOS=0x02 PREC=0xC0 TTL=255 ID=32421 PROTO=ICMP TYPE=3 CODE=3 [SRC=127.0.0.1 DST=127.0.0.1 LEN=64 TOS=0x02 PREC=0x00 TTL=64 ID=7284 DF PROTO=UDP SPT=32770 DPT=53 LEN=44 ]
Sep 11 19:47:31 sjoerd kernel: eth0: no IPv6 routers present
Sep 11 19:47:31 sjoerd kernel: LOIN ICMP rate to high IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=102 TOS=0x02 PREC=0xC0 TTL=255 ID=32427 PROTO=ICMP TYPE=3 CODE=3 [SRC=127.0.0.1 DST=127.0.0.1 LEN=74 TOS=0x02 PREC=0x00 TTL=64 ID=7785 DF PROTO=UDP SPT=32770 DPT=53 LEN=54 ]
Sep 11 19:47:31 sjoerd kernel: device eth0 entered promiscuous mode
Sep 11 19:47:31 sjoerd kernel: invalid operand: 0000
Sep 11 19:47:31 sjoerd kernel: CPU:    0
Sep 11 19:47:31 sjoerd kernel: EIP:    0010:[do_anonymous_page+160/304]
Sep 11 19:47:31 sjoerd kernel: EFLAGS: 00010206
Sep 11 19:47:31 sjoerd kernel: eax: f54fc000   ebx: f7771e00   ecx: c023eee8   edx: c0001ff8
Sep 11 19:47:31 sjoerd kernel: esi: c289b37c   edi: f7632c00   ebp: f5519068   esp: f54fddd0
Sep 11 19:47:31 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep 11 19:47:31 sjoerd kernel: Process snort (pid: 2059, stackpage=f54fd000)
Sep 11 19:47:31 sjoerd kernel: Stack: 4001a000 00000000 f7632c00 f7771e00 c01236b1 f7771e00 f7632c00 f5519068
Sep 11 19:47:31 sjoerd kernel:        00000001 4001a000 4001a000 f7771e00 ffffffff 00000001 c012382e f7771e00
Sep 11 19:47:31 sjoerd kernel:        f7632c00 4001a000 00000001 f5519068 f54fc000 f7632c00 f7771e00 f7771e1c
Sep 11 19:47:31 sjoerd kernel: Call Trace: [do_no_page+49/336] [handle_mm_fault+94/240] [do_page_fault+407/1200] [do_page_fault+0/1200] [do_rw_disk+290/832]
Sep 11 19:47:31 sjoerd kernel:    [start_request+312/528] [start_request+405/528] [ide_do_request+711/800] [error_code+56/68] [file_read_actor+96/192] [do_generic_file_read+517/1264]
Sep 11 19:47:31 sjoerd kernel:    [generic_file_read+99/128] [file_read_actor+0/192] [sys_read+150/208] [system_call+51/56]
Sep 11 19:47:31 sjoerd kernel:
Sep 11 19:47:31 sjoerd kernel: Code: 0f 0b 89 f0 2b 05 ec 63 29 c0 69 c0 f1 f0 f0 f0 c1 f8 02 c1
Sep 11 19:47:31 sjoerd kernel:  <6>device eth0 left promiscuous mode
Sep 11 19:47:31 sjoerd kernel: device eth0 entered promiscuous mode
Sep 11 19:47:31 sjoerd kernel: eth0: Enabling Jumbo frame support
Sep 11 19:47:31 sjoerd kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Sep 11 19:47:31 sjoerd kernel: LOOUT REJECT UDP IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=84 TOS=0x02 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=605 DPT=111 LEN=64
Sep 11 19:47:31 sjoerd kernel: LOOUT REJECT UDP IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=84 TOS=0x02 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=607 DPT=111 LEN=64

Sep 11 19:48:43 sjoerd : time=1000230523  unable to read "uptime"   ^I../SRC/dqs_load_avg.c 239 /usr/sbin/dqs_execd sjoerd.sjoerdnet
Sep 11 19:48:43 sjoerd kernel: invalid operand: 0000
Sep 11 19:48:43 sjoerd kernel: CPU:    0
Sep 11 19:48:43 sjoerd kernel: EIP:    0010:[filemap_nopage+284/1168]
Sep 11 19:48:43 sjoerd kernel: EFLAGS: 00010206
Sep 11 19:48:43 sjoerd kernel: eax: c2658524   ebx: 00000001   ecx: c023eee8   edx: c0001ff8
Sep 11 19:48:43 sjoerd kernel: esi: c297d838   edi: 00000015   ebp: c2658524   esp: f4169b3c
Sep 11 19:48:43 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep 11 19:48:43 sjoerd kernel: Process sh (pid: 2708, stackpage=f4169000)
Sep 11 19:48:43 sjoerd kernel: Stack: 40016000 f77153c0 f4372500 f4a67740 00000019 f7b173b8 f7715478 f77153c0
Sep 11 19:48:43 sjoerd kernel:        f4439f40 c0123720 f4372500 40016000 00000001 400162a8 f4a67740 ffffffff
Sep 11 19:48:43 sjoerd kernel:        00000001 c012382e f4a67740 f4372500 400162a8 00000001 f409a058 f4168000
Sep 11 19:48:43 sjoerd kernel: Call Trace: [do_no_page+160/336] [handle_mm_fault+94/240] [do_page_fault+407/1200] [do_page_fault+0/1200] [kunmap_high+86/128]
Sep 11 19:48:43 sjoerd kernel:    [file_read_actor+145/192] [update_atime+68/80] [do_generic_file_read+1246/1264] [do_munmap+86/608] [update_atime+68/80] [error_code+56/68]
Sep 11 19:48:43 sjoerd kernel:    [clear_user+46/64] [padzero+28/32] [load_elf_interp+619/704] [load_elf_binary+1927/2672] [load_elf_binary+0/2672] [kunmap_high+86/128]
Sep 11 19:48:43 sjoerd kernel:    [file_read_actor+145/192] [search_binary_handler+152/496] [do_execve+380/496] [do_execve+403/496] [sys_execve+47/96] [system_call+51/56]
Sep 11 19:48:43 sjoerd kernel:
Sep 11 19:48:43 sjoerd kernel: Code: 0f 0b 89 f0 2b 05 ec 63 29 c0 69 c0 f1 f0 f0 f0 c1 f8 02 c1

Sep 11 19:48:45 sjoerd kernel:  invalid operand: 0000
Sep 11 19:48:45 sjoerd kernel: CPU:    0
Sep 11 19:48:45 sjoerd kernel: EIP:    0010:[do_wp_page+604/912]
Sep 11 19:48:45 sjoerd kernel: EFLAGS: 00010206
Sep 11 19:48:45 sjoerd kernel: eax: c1004520   ebx: c1000010   ecx: c1000010   edx: c0001ff8
Sep 11 19:48:45 sjoerd kernel: esi: c25c0ff8   edi: c25984ec   ebp: 51e5a065   esp: f4001ed4
Sep 11 19:48:45 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep 11 19:48:45 sjoerd kernel: Process oracle (pid: 2718, stackpage=f4001000)
Sep 11 19:48:45 sjoerd kernel: Stack: 093e1a6c f4a678c0 ffffffff 00000001 c0123861 f4a678c0 f4024380 093e1a6c
Sep 11 19:48:45 sjoerd kernel:        f4016f84 51e5a065 f4000000 f4024380 f4a678c0 f4a678dc c0110ca7 f4a678c0
Sep 11 19:48:45 sjoerd kernel:        f4024380 093e1a6c 00000001 f4000000 00000007 c0110b10 bfffca1c f4a678dc
Sep 11 19:48:45 sjoerd kernel: Call Trace: [handle_mm_fault+145/240] [do_page_fault+407/1200] [do_page_fault+0/1200] [dput+25/416] [fput+116/224]
Sep 11 19:48:45 sjoerd kernel:    [filp_close+156/176] [sys_close+93/144] [error_code+56/68]
Sep 11 19:48:45 sjoerd kernel:
Sep 11 19:48:45 sjoerd kernel: Code: 0f 0b 89 f0 29 c8 69 c0 f1 f0 f0 f0 c1 f8 02 c1 e0 0c 0b 05

Sep 11 19:53:04 sjoerd kernel: invalid operand: 0000
Sep 11 19:53:04 sjoerd kernel: CPU:    0
Sep 11 19:53:04 sjoerd kernel: EIP:    0010:[filemap_nopage+284/1168]
Sep 11 19:53:04 sjoerd kernel: EFLAGS: 00010206
Sep 11 19:53:04 sjoerd kernel: eax: c23c8b9c   ebx: 00000001   ecx: c023eee8   edx: c0001ff8
Sep 11 19:53:04 sjoerd kernel: esi: c297d838   edi: 00000015   ebp: c23c8b9c   esp: f34a5b3c
Sep 11 19:53:04 sjoerd kernel: ds: 0018   es: 0018   ss: 0018
Sep 11 19:53:04 sjoerd kernel: Process qmail-remote (pid: 3084, stackpage=f34a5000)
Sep 11 19:53:04 sjoerd kernel: Stack: 40016000 f77153c0 f35da900 f3b70d40 00000019 f7b173b8 f7715478 f77153c0
Sep 11 19:53:04 sjoerd kernel:        f484e240 c0123720 f35da900 40016000 00000001 400162a8 f3b70d40 ffffffff
Sep 11 19:53:04 sjoerd kernel:        00000001 c012382e f3b70d40 f35da900 400162a8 00000001 f34c4058 f34a4000
Sep 11 19:53:04 sjoerd kernel: Call Trace: [do_no_page+160/336] [handle_mm_fault+94/240] [do_page_fault+407/1200] [do_page_fault+0/1200] [kunmap_high+86/128]
Sep 11 19:53:04 sjoerd kernel:    [file_read_actor+145/192] [update_atime+68/80] [do_generic_file_read+1246/1264] [do_munmap+86/608] [update_atime+68/80] [error_code+56/68]
Sep 11 19:53:04 sjoerd kernel:    [clear_user+46/64] [padzero+28/32] [load_elf_interp+619/704] [load_elf_binary+1927/2672] [load_elf_binary+0/2672] [search_binary_handler+152/496]
Sep 11 19:53:04 sjoerd kernel:    [do_execve+380/496] [do_execve+403/496] [sys_execve+47/96] [system_call+51/56]
Sep 11 19:53:04 sjoerd kernel:
Sep 11 19:53:04 sjoerd kernel: Code: 0f 0b 89 f0 2b 05 ec 63 29 c0 69 c0 f1 f0 f0 f0 c1 f8 02 c1


I rebooted the the same patched kernel with the mem=850M option (no
highmem used then), and i'm running it for a few hours but doesn't complain with
any special kernel message. And seems runs _just fine_.



Greetings,


-- 
Arjan Filius
mailto:iafilius@xs4all.nl

Re: Feedback on preemptible kernel patch

[grue]

Hi Rob,

Just finished running some benchmarks on my workstation.
Dual P3-550, 256MB ram, no swap, both kernels with CONFIG_SMP=y, 440BX,
2-20GB 5400rpm drives.

All benchmarks running in an Eterm on E on XF86-4.1.0-DRI with xmms running to
listen for latency probs. Benchmarks run as root, everything else as regular
user.

linux-2.4.10-pre6 with rml-netdev-random patch and rml-preempt patch (pre5 patches
applied to pre6), with CONFIG_PREEMPT=y

dbench 16
Throughput 23.4608 MB/sec (NB=29.326 MB/sec  234.608 MBit/sec)
Throughput 22.6915 MB/sec (NB=28.3644 MB/sec  226.915 MBit/sec) - .5sec hiccup in xmms
Throughput 20.4314 MB/sec (NB=25.5392 MB/sec  204.314 MBit/sec) - .5sec hiccup in xmms
Throughput 27.2849 MB/sec (NB=34.1061 MB/sec  272.849 MBit/sec) - .5sec hiccup in xmms
Throughput 20.5148 MB/sec (NB=25.6435 MB/sec  205.148 MBit/sec) - 2sec and .5sec in xmms
loadavg around 14

Bonnie
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
         1024  9002 98.0 15893 15.1  6519 10.8  6101 78.6 23330 24.4 104.3  2.2

linux-2.4.10-pre6

dbench 16
Throughput 18.1821 MB/sec (NB=22.7276 MB/sec  181.821 MBit/sec)
Throughput 22.4247 MB/sec (NB=28.0309 MB/sec  224.247 MBit/sec) - .5sec hiccup in xmms
Throughput 20.2662 MB/sec (NB=25.3328 MB/sec  202.662 MBit/sec) - 2sec hiccup in xmms
Throughput 28.4072 MB/sec (NB=35.5089 MB/sec  284.072 MBit/sec) - 3sec hiccup in xmms
Throughput 24.0549 MB/sec (NB=30.0686 MB/sec  240.549 MBit/sec)
loadavg around 14

Bonnie
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
         1024  9173 99.4 16104 14.5  6488 10.2  6139 78.9 23260 22.1 105.1  2.3

There's no real difference in speed with either kernel, the hard drives
seem to be the bottleneck on this system.
-(root@g-box)-(~)-
->hdparm /dev/hda

/dev/hda:
 multcount    = 16 (on)
 I/O support  =  1 (32-bit)
 unmaskirq    =  1 (on)
 using_dma    =  1 (on)
 keepsettings =  1 (on)
 nowerr       =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)
 geometry     = 2491/255/63, sectors = 40031712, start = 0

The throughput from bonnie is very close to the max physical throughput
of the drives, so I'm not to concerned that there isn't much of a
difference in speed.

The biggest difference is in the usability of the system under the load.
With 2.4.10-pre6 vanilla, dbench seriously affects the interactivity of
X, as well as causes some long interuptions in xmms. With preempt
enabled, this is much improved. Still some interuptions in xmms, but the
system is still usable, although a little sluggish. On the vanilla
kernel, even bonnie caused a couple of hiccups in xmms, with preempt
enabled, bonnie didn't affect xmms at all. all programs were run without
altering nice values at all.

For a workstation, I like the difference, my system is still usable even
with the load upwards of 14, without preempt, it's like looking at a
couple of screenshots. This is not something that I would recommend for
a server, but for a workstation, it works great.

Out of morbid curiousity, I ran a make -j bzImage to see how well this
would handle being driven into the ground. No oopsen, the OOM killer
worked great, killed everything that wasn't being used. The only prob is
that it hosed my console and killed my ssh daemon. oops ;) Sending a
sysrq-SAK and then a 3-finger-salute rebooted the system perfectly, no
fsck or anything.

I couldn't build that latency test for my system, so no results from it.
I haven't had a chance to look at it's source, so I'm not sure if I can
make it work here or not.

I'll keep up with your patches and keep you posted.

--
Gregory Finch

Re: Feedback on preemptible kernel patch

[Robert Love]

On Sun, 2001-09-09 at 14:57, grue@lakesweb.com wrote:
> Just finished running some benchmarks on my workstation.
> Dual P3-550, 256MB ram, no swap, both kernels with CONFIG_SMP=y, 440BX,
> 2-20GB 5400rpm drives.
> 
> All benchmarks running in an Eterm on E on XF86-4.1.0-DRI with xmms running to
> listen for latency probs. Benchmarks run as root, everything else as regular
> user.

The XMMS bit is a neat idea -- good thinking :)

> linux-2.4.10-pre6 with rml-netdev-random patch and rml-preempt patch (pre5 patches
> applied to pre6), with CONFIG_PREEMPT=y
> 
> dbench 16
> Throughput 23.4608 MB/sec (NB=29.326 MB/sec  234.608 MBit/sec)
> Throughput 22.6915 MB/sec (NB=28.3644 MB/sec  226.915 MBit/sec) - .5sec hiccup in xmms
> Throughput 20.4314 MB/sec (NB=25.5392 MB/sec  204.314 MBit/sec) - .5sec hiccup in xmms
> Throughput 27.2849 MB/sec (NB=34.1061 MB/sec  272.849 MBit/sec) - .5sec hiccup in xmms
> Throughput 20.5148 MB/sec (NB=25.6435 MB/sec  205.148 MBit/sec) - 2sec and .5sec in xmms
> loadavg around 14
> 
> Bonnie
>               -------Sequential Output-------- ---Sequential Input-- --Random--
>               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
> Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
>          1024  9002 98.0 15893 15.1  6519 10.8  6101 78.6 23330 24.4 104.3  2.2
> 
> linux-2.4.10-pre6
> 
> dbench 16
> Throughput 18.1821 MB/sec (NB=22.7276 MB/sec  181.821 MBit/sec)
> Throughput 22.4247 MB/sec (NB=28.0309 MB/sec  224.247 MBit/sec) - .5sec hiccup in xmms
> Throughput 20.2662 MB/sec (NB=25.3328 MB/sec  202.662 MBit/sec) - 2sec hiccup in xmms
> Throughput 28.4072 MB/sec (NB=35.5089 MB/sec  284.072 MBit/sec) - 3sec hiccup in xmms
> Throughput 24.0549 MB/sec (NB=30.0686 MB/sec  240.549 MBit/sec)
> loadavg around 14
> 
> Bonnie
>               -------Sequential Output-------- ---Sequential Input-- --Random--
>               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
> Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
>          1024  9173 99.4 16104 14.5  6488 10.2  6139 78.9 23260 22.1 105.1  2.3
> 

I didn't average it out, but the dbench results seem to be slightly
better with preemption enabled, as they should be.  We have seen results
with significantly better results under preemption than not, but oh
well.  Your system is already SMP, so the benefit would not be as
noticable.

Bonnie should not be much different if it is not threading the I/O
across multiple processes, since there is nothing else to be preempted. 
If anything, the throughput should drop slightly with preemption
enabled.

> The biggest difference is in the usability of the system under the load.
> With 2.4.10-pre6 vanilla, dbench seriously affects the interactivity of
> X, as well as causes some long interuptions in xmms. With preempt
> enabled, this is much improved. Still some interuptions in xmms, but the
> system is still usable, although a little sluggish. On the vanilla
> kernel, even bonnie caused a couple of hiccups in xmms, with preempt
> enabled, bonnie didn't affect xmms at all. all programs were run without
> altering nice values at all.

This is exactly what I want to hear.  I am glad you did the
XMMS/how-it-feels test and that preemption came out ahead.

We can work on cutting the latency even further.  There are still some
long held locks in the kernel, and we can not preempt around them.

> For a workstation, I like the difference, my system is still usable even
> with the load upwards of 14, without preempt, it's like looking at a
> couple of screenshots. This is not something that I would recommend for
> a server, but for a workstation, it works great.

Great.

> Out of morbid curiousity, I ran a make -j bzImage to see how well this
> would handle being driven into the ground. No oopsen, the OOM killer
> worked great, killed everything that wasn't being used. The only prob is
> that it hosed my console and killed my ssh daemon. oops ;) Sending a
> sysrq-SAK and then a 3-finger-salute rebooted the system perfectly, no
> fsck or anything.
> 
> I couldn't build that latency test for my system, so no results from it.
> I haven't had a chance to look at it's source, so I'm not sure if I can
> make it work here or not.

Someone else reported to me privately it did not compile.  I don't think
they looked into it.

I would assume the latency is going to drop in the same manner it does
for a UP system (_very_ much).

> I'll keep up with your patches and keep you posted.

Great, thanks for the feedback.  While it is great to hear "preemption
seems to be an improvement under SMP", I am most exciting it works
without faults.

You can always find the newest diffs at http://tech9.net/rml/linux

Thanks again,

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net