Re: 2.4 mm trouble [possible lru race]

Re: 2.4 mm trouble [possible lru race]

[Richard.Zidlicky]

> 
> The theoretical lru race possibly spotted in the wild...
> 
> >
> > Now I am wondering if that is just coincidence or why m68k hit that 
> > error so reliably.. is it supposed to have any effect at all on
> > UP?
> 
> Are you running UP+preempt?

no preempt or anything fancy, m68k vanila 2.4.19 (well almost).

Richard

Re: 2.4 mm trouble [possible lru race]

[Daniel Phillips]

On Tuesday 01 October 2002 16:20, Richard.Zidlicky@stud.informatik.uni-erlangen.de wrote:
> > 
> > The theoretical lru race possibly spotted in the wild...
> > 
> > >
> > > Now I am wondering if that is just coincidence or why m68k hit that 
> > > error so reliably.. is it supposed to have any effect at all on
> > > UP?
> > 
> > Are you running UP+preempt?
> 
> no preempt or anything fancy, m68k vanila 2.4.19 (well almost).

I'm having real trouble spotting substantive change in the patch that
would affect a UP kernel.  I believe you when you say it fixes your
problem, but we don't know why, and it is worth making some effort to
find out why.

Ah wait, I see one candidate, would you like to try:

                 * the page as well.
                 */
                if (page->buffers) {
                        /* avoid to free a locked page */
-                       get_page(page);
                        spin_unlock(&pagemap_lru_lock);
+                       get_page(page);
 
and see if your bug comes back?  There are a couple of other changes
that could be considered substantive by stretching one's imagination
enough, but this is the leading candidate.

Oh wait, you could also try this, a little further down:

+                                       page_cache_release(page);
                                        spin_lock(&pagemap_lru_lock);
-                                       put_page_nofree(page);

By the way, the original patch you posted was reversed and your editor
apparently took the liberty of cleaning up some whitespace in the file.
Generally, we try do avoid patch chunks that just, e.g., change bogus
spaces to tabs, and save those for official whitespace patches.

-- 
Daniel

Re: 2.4 mm trouble [possible lru race]

[Daniel Phillips]

On Tuesday 01 October 2002 16:20, Richard.Zidlicky@stud.informatik.uni-erlangen.de wrote:
> > 
> > The theoretical lru race possibly spotted in the wild...
> > 
> > >
> > > Now I am wondering if that is just coincidence or why m68k hit that 
> > > error so reliably.. is it supposed to have any effect at all on
> > > UP?
> > 
> > Are you running UP+preempt?
> 
> no preempt or anything fancy, m68k vanila 2.4.19 (well almost).

Vanilla would be CONFIG_SMP=y, is that what you have?  Otherwise please
disregard the post just above (which hasn't appeared on the list yet)
because spin_lock/unlock would be null, and the tests I suggested would
have no effect.

We would then be left with a *very* small number of candidates, which
we will test in accordance with the "what remains must be the truth"
principle.

-- 
Daniel

Re: 2.4 mm trouble [possible lru race]

[Rik van Riel]

On Tue, 1 Oct 2002, Daniel Phillips wrote:
> On Tuesday 01 October 2002 16:20, Richard.Zidlicky@stud.informatik.uni-erlangen.de wrote:

> > no preempt or anything fancy, m68k vanila 2.4.19 (well almost).
>
> Vanilla would be CONFIG_SMP=y, is that what you have?

Somehow I doubt Linux supports m68k SMP machines ;)

Rik
-- 
A: No.
Q: Should I include quotations after my reply?

http://www.surriel.com/		http://distro.conectiva.com/

Re: 2.4 mm trouble [possible lru race]

[Daniel Phillips]

On Tuesday 01 October 2002 18:56, Rik van Riel wrote:
> On Tue, 1 Oct 2002, Daniel Phillips wrote:
> > On Tuesday 01 October 2002 16:20, Richard.Zidlicky@stud.informatik.uni-erlangen.de wrote:
> 
> > > no preempt or anything fancy, m68k vanila 2.4.19 (well almost).
> >
> > Vanilla would be CONFIG_SMP=y, is that what you have?
> 
> Somehow I doubt Linux supports m68k SMP machines ;)

CONFIG_SMP=y works perfectly well on single cpu machines - it forces
the spinlocks to actually exist.  It's not supposed to change any
behaviour, but you never know.  Behaviour is obviously changing here.

-- 
Daniel

Re: 2.4 mm trouble [possible lru race]

[Jens Axboe]

On Tue, Oct 01 2002, Daniel Phillips wrote:
> On Tuesday 01 October 2002 18:56, Rik van Riel wrote:
> > On Tue, 1 Oct 2002, Daniel Phillips wrote:
> > > On Tuesday 01 October 2002 16:20, Richard.Zidlicky@stud.informatik.uni-erlangen.de wrote:
> > 
> > > > no preempt or anything fancy, m68k vanila 2.4.19 (well almost).
> > >
> > > Vanilla would be CONFIG_SMP=y, is that what you have?
> > 
> > Somehow I doubt Linux supports m68k SMP machines ;)
> 
> CONFIG_SMP=y works perfectly well on single cpu machines - it forces
> the spinlocks to actually exist.  It's not supposed to change any
> behaviour, but you never know.  Behaviour is obviously changing here.

Again, m68k was the target.

-- 
Jens Axboe

Re: 2.4 mm trouble [possible lru race]

[Daniel Phillips]

On Tuesday 01 October 2002 19:31, Jens Axboe wrote:
> On Tue, Oct 01 2002, Daniel Phillips wrote:
> > On Tuesday 01 October 2002 18:56, Rik van Riel wrote:
> > > On Tue, 1 Oct 2002, Daniel Phillips wrote:
> > > > On Tuesday 01 October 2002 16:20, Richard.Zidlicky@stud.informatik.uni-erlangen.de wrote:
> > > > > no preempt or anything fancy, m68k vanila 2.4.19 (well almost).
> > > >
> > > > Vanilla would be CONFIG_SMP=y, is that what you have?
> > > 
> > > Somehow I doubt Linux supports m68k SMP machines ;)
> > 
> > CONFIG_SMP=y works perfectly well on single cpu machines - it forces
> > the spinlocks to actually exist.  It's not supposed to change any
> > behaviour, but you never know.  Behaviour is obviously changing here.
> 
> Again, m68k was the target.

Sure fine, no good reason to be cryptic about it though.

   #error "m68k doesn't do SMP yet"

So SMP must be off or the compile would abort.  Well, the only interesting
difference remaining is the extra count for the LRU.  I actually had that
parameterized at one time so you could turn it on/off easily, but akpm
complained about #ifdef's so I took that out ;-)

Richard, before I go making a test patch for you (it's not completely
straightforward) can you confirm that your bug comes back when you back
the lru race patch out?

-- 
Daniel

Re: 2.4 mm trouble [possible lru race]

[Jens Axboe]

On Tue, Oct 01 2002, Daniel Phillips wrote:
> On Tuesday 01 October 2002 19:31, Jens Axboe wrote:
> > On Tue, Oct 01 2002, Daniel Phillips wrote:
> > > On Tuesday 01 October 2002 18:56, Rik van Riel wrote:
> > > > On Tue, 1 Oct 2002, Daniel Phillips wrote:
> > > > > On Tuesday 01 October 2002 16:20, Richard.Zidlicky@stud.informatik.uni-erlangen.de wrote:
> > > > > > no preempt or anything fancy, m68k vanila 2.4.19 (well almost).
> > > > >
> > > > > Vanilla would be CONFIG_SMP=y, is that what you have?
> > > > 
> > > > Somehow I doubt Linux supports m68k SMP machines ;)
> > > 
> > > CONFIG_SMP=y works perfectly well on single cpu machines - it forces
> > > the spinlocks to actually exist.  It's not supposed to change any
> > > behaviour, but you never know.  Behaviour is obviously changing here.
> > 
> > Again, m68k was the target.
> 
> Sure fine, no good reason to be cryptic about it though.
> 
>    #error "m68k doesn't do SMP yet"
> 
> So SMP must be off or the compile would abort.  Well, the only interesting

There's no CONFIG_SMP in the m68k arch config.in. Anyways, enough
beating of dead horse :)

-- 
Jens Axboe

Re: 2.4 mm trouble [possible lru race]

[Daniel Phillips]

On Tuesday 01 October 2002 20:04, Jens Axboe wrote:
> On Tue, Oct 01 2002, Daniel Phillips wrote:
> > On Tuesday 01 October 2002 19:31, Jens Axboe wrote:
> > > Again, m68k was the target.
> > 
> > Sure fine, no good reason to be cryptic about it though.
> > 
> >    #error "m68k doesn't do SMP yet"
> > 
> > So SMP must be off or the compile would abort.  Well, the only interesting
> 
> There's no CONFIG_SMP in the m68k arch config.in. Anyways, enough
> beating of dead horse :)

The horse isn't dead yet, it's still twitching a little.  At this
point we still need to speculate about wny anyone would want an SMP
Dragonball machine ;-)

-- 
Daniel

Re: 2.4 mm trouble [possible lru race]

[Rik van Riel]

On Tue, 1 Oct 2002, Daniel Phillips wrote:

> The horse isn't dead yet, it's still twitching a little.  At this
> point we still need to speculate about wny anyone would want an
> SMP Dragonball machine ;-)

I've seen an SMP 68k box, a DIAB DATA machine. I think
Bull shipped them, too.

What is that coloured spot on the pavement ?
Could it be a horse died there, long ago ?
Now, stop beating the pavement.

cheers,

Rik
-- 
A: No.
Q: Should I include quotations after my reply?

http://www.surriel.com/		http://distro.conectiva.com/

Re: 2.4 mm trouble [possible lru race]

[Geert Uytterhoeven]

On Tue, 1 Oct 2002, Daniel Phillips wrote:
> On Tuesday 01 October 2002 20:04, Jens Axboe wrote:
> > On Tue, Oct 01 2002, Daniel Phillips wrote:
> > > On Tuesday 01 October 2002 19:31, Jens Axboe wrote:
> > > > Again, m68k was the target.
> > > 
> > > Sure fine, no good reason to be cryptic about it though.
> > > 
> > >    #error "m68k doesn't do SMP yet"
> > > 
> > > So SMP must be off or the compile would abort.  Well, the only interesting
> > 
> > There's no CONFIG_SMP in the m68k arch config.in. Anyways, enough
> > beating of dead horse :)
> 
> The horse isn't dead yet, it's still twitching a little.  At this
> point we still need to speculate about wny anyone would want an SMP
> Dragonball machine ;-)

Dragonballs don't have an MMU, so they would run uClinux/m68k, not Linux/m68k.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: 2.4 mm trouble [possible lru race]

[Richard Zidlicky]

On Tue, Oct 01, 2002 at 08:01:10PM +0200, Daniel Phillips wrote:

> Richard, before I go making a test patch for you (it's not completely
> straightforward) can you confirm that your bug comes back when you back
> the lru race patch out?

bad luck, the disappearance of the bug was rather accidental - I have
switched to a different swap partition in the meantime. So backing out
the changes doesn't make the bug reappear, restoring previous IDE
configuration does.

Very likely interrupt related trouble, somewhere a missing spinlock_irqsave 
perhaps. The bug manifests itself so that pages from wrong procesess get 
swapped in for some process, however I have also had the luck to crash the 
kernel (no Oops) so it is not likely to be one of the TLB/cache problems.

What strikes me is that is always related to swap and I 've never got any
strange dmesg so far.

Richard

Re: 2.4 mm trouble [possible lru race]

[Richard Zidlicky]

> 
> On Tuesday 01 October 2002 20:04, Jens Axboe wrote:
> > On Tue, Oct 01 2002, Daniel Phillips wrote:
> > > On Tuesday 01 October 2002 19:31, Jens Axboe wrote:
> > > > Again, m68k was the target.
> > > 
> > > Sure fine, no good reason to be cryptic about it though.
> > > 
> > >    #error "m68k doesn't do SMP yet"
> > > 
> > > So SMP must be off or the compile would abort.  Well, the only interesting
> > 
> > There's no CONFIG_SMP in the m68k arch config.in. Anyways, enough
> > beating of dead horse :)
> 
> The horse isn't dead yet, it's still twitching a little.  At this
> point we still need to speculate about wny anyone would want an SMP
> Dragonball machine ;-)

not on Dragonball but there were many 68040 SMP systems around long 
before Intel had anything SMP capable. In the late 80'ies those were 
considered real number crunchers :)

Richard

Re: 2.4 mm trouble [possible lru race]

[Daniel Phillips]

The theoretical lru race possibly spotted in the wild...

On Tuesday 01 October 2002 11:22, Richard Zidlicky wrote:
> On Sat, Sep 28, 2002 at 04:38:50PM +0200, Roman Zippel wrote:
> > Hi,
> > 
> > On Wed, 25 Sep 2002, Richard Zidlicky wrote:
> > 
> > > First I suspected a stale TLB entry so I've added pretty many
> > > extra flushes througout the code. This does very much reduce the
> > > risk of the problem, but the problem still happens if swapping is
> > > increased so it might very well be something else.
> > >
> > > Any ideas?
> > 
> > It sounds like a cache problem. I had to fix one early 2.4, so maybe there
> > is another one.
> > It would help a lot if you could reproduce it within gdb to get some more
> > information about the context of the crash (e.g. invalid data or code, the
> > type of mapping (from /proc/<pid>/maps)).
> 
> hm, I am now testing the appended patch, which is a backport to 2.4.19
> of this:
> 
<<<<<<<<<<<<
>
> From: Daniel Phillips <phillips@arcor.de>
> To: Marcelo Tosatti <marcelo@conectiva.com.br>,
>         "Christian Ehrhardt" <ehrhardt@mathematik.uni-ulm.de>
> Subject: [CFT] [PATCH] LRU race fix
> Date: 	Tue, 17 Sep 2002 19:03:19 +0200
> X-Mailer: KMail [version 1.3.2]
> Cc: <linux-kernel@vger.kernel.org>
> 
> This patch against 2.4.20-pre7 fixes a theoretical race where a page could
> possibly be freed while still on the lru list.  The details have been
> discussed at length earlier, see "[RFC] [PATCH] Include LRU in page count".
> 
> The race may not even be that theoretical, it's just so rare that when it
> does happen, it might be dismissed as a driver problem or similar...
>
> [...]
>
>>>>>>>>>>>>

> Somehow this does completely fix my problem, I have taken out all
> the tlb related hacks and the testcase that caused the problem 100%
> now runs without any sign of problems :))
>
> Now I am wondering if that is just coincidence or why m68k hit that 
> error so reliably.. is it supposed to have any effect at all on
> UP?

Are you running UP+preempt?

-- 
Daniel