2.6.39-rc5-git4: Reported regressions from 2.6.38

2.6.39-rc5-git4: Reported regressions from 2.6.38

[Rafael J. Wysocki]

[NOTE:
 I have one request for whoever works on fixing the listed regressions.  While
 your work is highly appreciated anyway, if you have a patch fixing a listed
 regression or you know of a commit fixing a listed regression, please drop
 a notice into the corresponding Bugzilla entry.  This will help us a lot.]  

This message contains a list of some regressions from 2.6.38,
for which there are no fixes in the mainline known to the tracking team.
If any of them have been fixed already, please let us know.

If you know of any other unresolved regressions from 2.6.38, please let us
know either and we'll add them to the list.  Also, please let us know
if any of the entries below are invalid.

Each entry from the list will be sent additionally in an automatic reply
to this message with CCs to the people involved in reporting and handling
the issue.


Listed regressions statistics:

  Date          Total  Pending  Unresolved
  ----------------------------------------
  2011-04-30       38       17          16
  2011-04-17       17       11          10


Unresolved regressions
----------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=34012
Subject		: 2.6.39-rc4+: oom-killer busy killing tasks
Submitter	: Christian Kujau <lists@nerdbynature.de>
Date		: 2011-04-22 1:57 (9 days old)
Message-ID	: <alpine.DEB.2.01.1104211841510.18728@trent.utfs.org>
References	: http://marc.info/?l=linux-kernel&m=130343744622331&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=34002
Subject		: [REGRESSION] [2.6.39-rc3] Wrong resolution in framebuffer and X Window
Submitter	: Maciej Rutecki <maciej.rutecki@gmail.com>
Date		: 2011-04-17 16:04 (14 days old)
Message-ID	: <201104171804.04664.maciej.rutecki@gmail.com>
References	: http://marc.info/?l=linux-fbdev&m=130305625114863&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33842
Subject		: NULL pointer dereference in ip_fragment
Submitter	: Tomas Carnecky <tom@dbservice.com>
Date		: 2011-04-23 07:51 (8 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33812
Subject		: radeon with modeset option causes oops as secondary adapter (vgaswitcheroo)
Submitter	: Johannes Engel <jcnengel@googlemail.com>
Date		: 2011-04-21 23:14 (10 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33792
Subject		: lockdep trace when unplugging usb audio (.39rc4)
Submitter	: Dave Jones <davej@redhat.com>
Date		: 2011-04-19 18:07 (12 days old)
Message-ID	: <20110419180745.GA438@redhat.com>
References	: http://marc.info/?l=linux-kernel&m=130323648920431&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33492
Subject		: [bug] 2.6.39-rc3 fails to boot on Alpha
Submitter	: Mikael Pettersson <mikpe@it.uu.se>
Date		: 2011-04-17 15:13 (14 days old)
Message-ID	: <19883.912.266127.538215@pilspetsen.it.uu.se>
References	: http://marc.info/?l=linux-kernel&m=130305321212360&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33482
Subject		: [Regression] Bluetooth pairing does not work anymore in 2.6.39-rc3 (works in 2.6.38.3)
Submitter	: Gottfried Haider <gottfried.haider@gmail.com>
Date		: 2011-04-16 16:38 (15 days old)
Message-ID	: <BANLkTi=LvyZ+7BHfVL849pztfvsYaVM4SQ@mail.gmail.com>
References	: http://marc.info/?l=linux-kernel&m=130297197128328&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33432
Subject		: WARNING at libata-core.c:5015 in 2.6.39-rc3-wl+, then lockup.
Submitter	: Ben Greear <greearb@candelatech.com>
Date		: 2011-04-13 16:29 (18 days old)
Message-ID	: <4DA5CF81.4070908@candelatech.com>
References	: http://marc.info/?l=linux-kernel&m=130271220809443&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33422
Subject		: oops in radeon_ddc_get_modes on first boot of rc3
Submitter	: Steve French <smfrench@gmail.com>
Date		: 2011-04-12 17:54 (19 days old)
Message-ID	: <BANLkTikTVuV9eH4GiLDs0DewGJXo_3daeg@mail.gmail.com>
References	: http://marc.info/?l=linux-kernel&m=130263086322702&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33402
Subject		: [regression] 2.6.39-rc[1-3] fail to boot on G5 PowerMac
Submitter	: Mikael Pettersson <mikpe@it.uu.se>
Date		: 2011-04-12 17:30 (19 days old)
Message-ID	: <19876.35918.322649.256455@pilspetsen.it.uu.se>
References	: http://marc.info/?l=linuxppc-embedded&m=130263128523237&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33302
Subject		: ACPI reboot broken on HP Compaq nx7300.
Submitter	: Alexey Zaytsev <alexey.zaytsev@gmail.com>
Date		: 2011-04-15 14:44 (16 days old)
First-Bad-Commit: http://git.kernel.org/linus/3d35ac346e981162eeba391e496faceed4753e7b


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33272
Subject		: drm related hard-hang
Submitter	: Peter Teoh <htmldeveloper@gmail.com>
Date		: 2011-04-14 01:29 (17 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33242
Subject		: Lockdep splat in autofs with 2.6.39-rc2
Submitter	: Nick Bowler <nbowler@elliptictech.com>
Date		: 2011-04-07 19:44 (24 days old)
Message-ID	: <20110407194403.GA29404@elliptictech.com>
References	: http://marc.info/?l=linux-kernel&m=130220545614682&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33142
Subject		: 2.6.39-rc2 regression: X201s fails to resume b77dcf8460ae57d4eb9fd3633eb4f97b8fb20716
Submitter	: Keith Packard <keithp@keithp.com>
Date		: 2011-04-06 7:44 (25 days old)
Message-ID	: <yun1v1fj024.fsf@aiko.keithp.com>
References	: http://marc.info/?l=linux-kernel&m=130207593728273&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33092
Subject		: [regression] 2.6.39-rc1 - Beagleboard usbnet broken
Submitter	: Mark Jackson <mpfj-list@mimc.co.uk>
Date		: 2011-04-04 9:22 (27 days old)
First-Bad-Commit: http://git.kernel.org/linus/087809fce28f50098d9c3ef1a6865c722f23afd2
Message-ID	: <4D998DC9.3040109@mimc.co.uk>
References	: http://marc.info/?l=linux-kernel&m=130191386508831&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=32982
Subject		: Kernel locks up a few minutes after boot
Submitter	: Bart Van Assche <bart.vanassche@gmail.com>
Date		: 2011-04-10 19:55 (21 days old)


Regressions with patches
------------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33802
Subject		: list_del corruption in sd driver since 2.6.39-rc4
Submitter	: Christian Casteyde <casteyde.christian@free.fr>
Date		: 2011-04-21 21:10 (10 days old)
Handled-By	: James Bottomley <James.Bottomley@suse.de>
Patch		: http://marc.info/?l=linux-kernel&m=130271409412095


For details, please visit the bug entries and follow the links given in
references.

As you can see, there is a Bugzilla entry for each of the listed regressions.
There also is a Bugzilla entry used for tracking the regressions from 2.6.38,
unresolved as well as resolved, at:

http://bugzilla.kernel.org/show_bug.cgi?id=32012

Please let the tracking team know if there are any Bugzilla entries that
should be added to the list in there.

Thanks!

[Bug #32982] Kernel locks up a few minutes after boot

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=32982
Subject		: Kernel locks up a few minutes after boot
Submitter	: Bart Van Assche <bart.vanassche@gmail.com>
Date		: 2011-04-10 19:55 (21 days old)

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Linus Torvalds]

I think we had all assumed that this was the MD problem that should
have been fixed in rc4 (the symptoms matched), but I don't think we
got any confirmation from Bart on that.

Bart? Does the problem still persist in current -git?

                 Linus

On Sat, Apr 30, 2011 at 12:42 PM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> This message has been generated automatically as a part of a summary report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.38.  Please verify if it still should be listed and let the tracking team
> know (either way).
>
>
> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=32982
> Subject         : Kernel locks up a few minutes after boot
> Submitter       : Bart Van Assche <bart.vanassche@gmail.com>
> Date            : 2011-04-10 19:55 (21 days old)
>
>
>

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Linus Torvalds]

On Sun, May 1, 2011 at 2:55 AM, Bart Van Assche <bvanassche@acm.org> wrote:
>
> There is something else and completely unrelated that is puzzling me though:
> on two almost identical systems one always recognizes all internal PCIe
> cards but the other system not. This is something that seldom happened with
> 2.6.34 but happens frequently with 2.6.38 and 2.6.39-rcx. What I see is that
> during boot either both InfiniBand PCIe cards are recognized or that one
> specific card is not recognized and even doesn't show up in the lspci
> output. A BIOS upgrade didn't help. Any idea where I should start looking to
> find the cause of this issue ?

So it has happened sporadically before, but happens much more commonly
now? That very much implies some timing issue in PCI probing.

It could be, for example, that the card has a very slow reset
sequence, and doesn't respond to PCI config cycles until it has
internally booted fully. If so, a faster boot by the kernel might just
cause the Linux PCI enumeration to be done before the card is ready.

(That's a really unlikely scenario - I'm not seriously suggesting that
the card would be quite <i>that</i> stupid and slow. But there might
be similar issues at a much lower level, ie if the Linux pcie port
driver might be resetting the port and then trying to read the card
too quickly afterwards, and you'd want some added delay there).

Have you tried it "pcie_ports=compat" (or "native") makes any difference?

But you should probably contact Jesse Barnes and the linux-pci mailing
list and see if anybody has any smarter ideas.

                       Linus

[Bug #33272] drm related hard-hang

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33272
Subject		: drm related hard-hang
Submitter	: Peter Teoh <htmldeveloper@gmail.com>
Date		: 2011-04-14 01:29 (17 days old)

Re: [Bug #33272] drm related hard-hang

[Rafael J. Wysocki]

On Sunday, May 01, 2011, Peter Teoh wrote:
> sorry, i did a git-update to 2.6.39-rc4 and it failed for me (hard hanged
> too) in other ways:
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=34182
> 
> so i have not been able to test this, but definitely 2.6.39-rc3 is not
> working as far as Intel DRM is concerned.

Thanks for the update.

Rafael

Re: [Bug #33272] drm related hard-hang

[Peter Teoh]

I did an update (2.6.39-rc6+) today and the system seemed very stable,
all previously reported bugs i have has all disappeared.   i tested it
with Chrome browser usage heavily, and kernel tracing with systemtap
and all seemed very stable.   Thanks.


2011/5/2 Rafael J. Wysocki <rjw@sisk.pl>:
> On Sunday, May 01, 2011, Peter Teoh wrote:
>> sorry, i did a git-update to 2.6.39-rc4 and it failed for me (hard hanged
>> too) in other ways:
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=34182
>>
>> so i have not been able to test this, but definitely 2.6.39-rc3 is not
>> working as far as Intel DRM is concerned.
>
> Thanks for the update.
>
> Rafael
>



-- 
Regards,
Peter Teoh

Re: [Bug #33272] drm related hard-hang

[Rafael J. Wysocki]

On Saturday, May 07, 2011, Peter Teoh wrote:
> I did an update (2.6.39-rc6+) today and the system seemed very stable,
> all previously reported bugs i have has all disappeared.   i tested it
> with Chrome browser usage heavily, and kernel tracing with systemtap
> and all seemed very stable.   Thanks.

Good, thanks for the update.

Rafael

[Bug #33142] 2.6.39-rc2 regression: X201s fails to resume b77dcf8460ae57d4eb9fd3633eb4f97b8fb20716

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33142
Subject		: 2.6.39-rc2 regression: X201s fails to resume b77dcf8460ae57d4eb9fd3633eb4f97b8fb20716
Submitter	: Keith Packard <keithp@keithp.com>
Date		: 2011-04-06 7:44 (25 days old)
Message-ID	: <yun1v1fj024.fsf@aiko.keithp.com>
References	: http://marc.info/?l=linux-kernel&m=130207593728273&w=2

[Bug #33302] ACPI reboot broken on HP Compaq nx7300.

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33302
Subject		: ACPI reboot broken on HP Compaq nx7300.
Submitter	: Alexey Zaytsev <alexey.zaytsev@gmail.com>
Date		: 2011-04-15 14:44 (16 days old)
First-Bad-Commit: http://git.kernel.org/linus/3d35ac346e981162eeba391e496faceed4753e7b

[Bug #33242] Lockdep splat in autofs with 2.6.39-rc2

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33242
Subject		: Lockdep splat in autofs with 2.6.39-rc2
Submitter	: Nick Bowler <nbowler@elliptictech.com>
Date		: 2011-04-07 19:44 (24 days old)
Message-ID	: <20110407194403.GA29404@elliptictech.com>
References	: http://marc.info/?l=linux-kernel&m=130220545614682&w=2

[Bug #33092] [regression] 2.6.39-rc1 - Beagleboard usbnet broken

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33092
Subject		: [regression] 2.6.39-rc1 - Beagleboard usbnet broken
Submitter	: Mark Jackson <mpfj-list@mimc.co.uk>
Date		: 2011-04-04 9:22 (27 days old)
First-Bad-Commit: http://git.kernel.org/linus/087809fce28f50098d9c3ef1a6865c722f23afd2
Message-ID	: <4D998DC9.3040109@mimc.co.uk>
References	: http://marc.info/?l=linux-kernel&m=130191386508831&w=2

Re: [Bug #33092] [regression] 2.6.39-rc1 - Beagleboard usbnet broken

[Mark Jackson]

On 30/04/11 20:46, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a summary report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.38.  Please verify if it still should be listed and let the tracking team
> know (either way).

I have just tried the latest linus git and it is still broken.

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33092
> Subject		: [regression] 2.6.39-rc1 - Beagleboard usbnet broken
> Submitter	: Mark Jackson <mpfj-list@mimc.co.uk>
> Date		: 2011-04-04 9:22 (27 days old)
> First-Bad-Commit: http://git.kernel.org/linus/087809fce28f50098d9c3ef1a6865c722f23afd2
> Message-ID	: <4D998DC9.3040109@mimc.co.uk>
> References	: http://marc.info/?l=linux-kernel&m=130191386508831&w=2

Regards
Mark

Re: [Bug #33092] [regression] 2.6.39-rc1 - Beagleboard usbnet broken

[Rafael J. Wysocki]

On Friday, May 06, 2011, Mark Jackson wrote:
> On 30/04/11 20:46, Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a summary report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.38.  Please verify if it still should be listed and let the tracking team
> > know (either way).
> 
> I have just tried the latest linus git and it is still broken.
> 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33092
> > Subject		: [regression] 2.6.39-rc1 - Beagleboard usbnet broken
> > Submitter	: Mark Jackson <mpfj-list@mimc.co.uk>
> > Date		: 2011-04-04 9:22 (27 days old)
> > First-Bad-Commit: http://git.kernel.org/linus/087809fce28f50098d9c3ef1a6865c722f23afd2
> > Message-ID	: <4D998DC9.3040109@mimc.co.uk>
> > References	: http://marc.info/?l=linux-kernel&m=130191386508831&w=2

Thanks for the update.

Rafael

[Bug #33432] WARNING at libata-core.c:5015 in 2.6.39-rc3-wl+, then lockup.

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33432
Subject		: WARNING at libata-core.c:5015 in 2.6.39-rc3-wl+, then lockup.
Submitter	: Ben Greear <greearb@candelatech.com>
Date		: 2011-04-13 16:29 (18 days old)
Message-ID	: <4DA5CF81.4070908@candelatech.com>
References	: http://marc.info/?l=linux-kernel&m=130271220809443&w=2

[Bug #33482] [Regression] Bluetooth pairing does not work anymore in 2.6.39-rc3 (works in 2.6.38.3)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33482
Subject		: [Regression] Bluetooth pairing does not work anymore in 2.6.39-rc3 (works in 2.6.38.3)
Submitter	: Gottfried Haider <gottfried.haider@gmail.com>
Date		: 2011-04-16 16:38 (15 days old)
Message-ID	: <BANLkTi=LvyZ+7BHfVL849pztfvsYaVM4SQ@mail.gmail.com>
References	: http://marc.info/?l=linux-kernel&m=130297197128328&w=2

Re: [Bug #33482] [Regression] Bluetooth pairing does not work anymore in 2.6.39-rc3 (works in 2.6.38.3)

[Gottfried Haider]

Hello Rafael,


> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=33482
> Subject         : [Regression] Bluetooth pairing does not work anymore in 2.6.39-rc3 (works in 2.6.38.3)
> Submitter       : Gottfried Haider <gottfried.haider@gmail.com>
> Date            : 2011-04-16 16:38 (15 days old)
> Message-ID      : <BANLkTi=LvyZ+7BHfVL849pztfvsYaVM4SQ@mail.gmail.com>
> References      : http://marc.info/?l=linux-kernel&m=130297197128328&w=2

This can be removed from the regression list as it turned out to be
only a mis-config-uration (oldconfig listed BT_L2CAP as a new config
option, defaulting to false, when it had been build as a module
before).

regards,

Gottfried

Re: [Bug #33482] [Regression] Bluetooth pairing does not work anymore in 2.6.39-rc3 (works in 2.6.38.3)

[Rafael J. Wysocki]

On Monday, May 02, 2011, Gottfried Haider wrote:
> Hello Rafael,
> 
> 
> > Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=33482
> > Subject         : [Regression] Bluetooth pairing does not work anymore in 2.6.39-rc3 (works in 2.6.38.3)
> > Submitter       : Gottfried Haider <gottfried.haider@gmail.com>
> > Date            : 2011-04-16 16:38 (15 days old)
> > Message-ID      : <BANLkTi=LvyZ+7BHfVL849pztfvsYaVM4SQ@mail.gmail.com>
> > References      : http://marc.info/?l=linux-kernel&m=130297197128328&w=2
> 
> This can be removed from the regression list as it turned out to be
> only a mis-config-uration (oldconfig listed BT_L2CAP as a new config
> option, defaulting to false, when it had been build as a module
> before).

Thanks, closing.

Rafael

[Bug #33402] [regression] 2.6.39-rc[1-3] fail to boot on G5 PowerMac

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33402
Subject		: [regression] 2.6.39-rc[1-3] fail to boot on G5 PowerMac
Submitter	: Mikael Pettersson <mikpe@it.uu.se>
Date		: 2011-04-12 17:30 (19 days old)
Message-ID	: <19876.35918.322649.256455@pilspetsen.it.uu.se>
References	: http://marc.info/?l=linuxppc-embedded&m=130263128523237&w=2

Re: [Bug #33402] [regression] 2.6.39-rc[1-3] fail to boot on G5 PowerMac

[Mikael Pettersson]

On Sat, 30 Apr 2011 21:46:55 +0200 (CEST), "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> The following bug entry is on the current list of known regressions
> from 2.6.38.  Please verify if it still should be listed and let the tracking team
> know (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33402
> Subject		: [regression] 2.6.39-rc[1-3] fail to boot on G5 PowerMac
> Submitter	: Mikael Pettersson <mikpe@it.uu.se>
> Date		: 2011-04-12 17:30 (19 days old)
> Message-ID	: <19876.35918.322649.256455@pilspetsen.it.uu.se>
> References	: http://marc.info/?l=linuxppc-embedded&m=130263128523237&w=2

Fixed in 2.6.39-rc5.

/Mikael

Re: [Bug #33402] [regression] 2.6.39-rc[1-3] fail to boot on G5 PowerMac

[Rafael J. Wysocki]

On Monday, May 02, 2011, Mikael Pettersson wrote:
> On Sat, 30 Apr 2011 21:46:55 +0200 (CEST), "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > The following bug entry is on the current list of known regressions
> > from 2.6.38.  Please verify if it still should be listed and let the tracking team
> > know (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33402
> > Subject		: [regression] 2.6.39-rc[1-3] fail to boot on G5 PowerMac
> > Submitter	: Mikael Pettersson <mikpe@it.uu.se>
> > Date		: 2011-04-12 17:30 (19 days old)
> > Message-ID	: <19876.35918.322649.256455@pilspetsen.it.uu.se>
> > References	: http://marc.info/?l=linuxppc-embedded&m=130263128523237&w=2
> 
> Fixed in 2.6.39-rc5.

Thanks, closing.

Rafael

[Bug #33422] oops in radeon_ddc_get_modes on first boot of rc3

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33422
Subject		: oops in radeon_ddc_get_modes on first boot of rc3
Submitter	: Steve French <smfrench@gmail.com>
Date		: 2011-04-12 17:54 (19 days old)
Message-ID	: <BANLkTikTVuV9eH4GiLDs0DewGJXo_3daeg@mail.gmail.com>
References	: http://marc.info/?l=linux-kernel&m=130263086322702&w=2

[Bug #33802] list_del corruption in sd driver since 2.6.39-rc4

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33802
Subject		: list_del corruption in sd driver since 2.6.39-rc4
Submitter	: Christian Casteyde <casteyde.christian@free.fr>
Date		: 2011-04-21 21:10 (10 days old)
Handled-By	: James Bottomley <James.Bottomley@suse.de>
Patch		: http://marc.info/?l=linux-kernel&m=130271409412095

[Bug #33792] lockdep trace when unplugging usb audio (.39rc4)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33792
Subject		: lockdep trace when unplugging usb audio (.39rc4)
Submitter	: Dave Jones <davej@redhat.com>
Date		: 2011-04-19 18:07 (12 days old)
Message-ID	: <20110419180745.GA438@redhat.com>
References	: http://marc.info/?l=linux-kernel&m=130323648920431&w=2

[Bug #33492] [bug] 2.6.39-rc3 fails to boot on Alpha

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33492
Subject		: [bug] 2.6.39-rc3 fails to boot on Alpha
Submitter	: Mikael Pettersson <mikpe@it.uu.se>
Date		: 2011-04-17 15:13 (14 days old)
Message-ID	: <19883.912.266127.538215@pilspetsen.it.uu.se>
References	: http://marc.info/?l=linux-kernel&m=130305321212360&w=2

Re: [Bug #33492] [bug] 2.6.39-rc3 fails to boot on Alpha

[Mikael Pettersson]

On Sat, 30 Apr 2011 21:46:56 +0200 (CEST), "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> The following bug entry is on the current list of known regressions
> from 2.6.38.  Please verify if it still should be listed and let the tracking team
> know (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33492
> Subject		: [bug] 2.6.39-rc3 fails to boot on Alpha
> Submitter	: Mikael Pettersson <mikpe@it.uu.se>
> Date		: 2011-04-17 15:13 (14 days old)
> Message-ID	: <19883.912.266127.538215@pilspetsen.it.uu.se>
> References	: http://marc.info/?l=linux-kernel&m=130305321212360&w=2

Fixed in 2.6.39-rc5.

/Mikael

Re: [Bug #33492] [bug] 2.6.39-rc3 fails to boot on Alpha

[Rafael J. Wysocki]

On Sunday, May 01, 2011, Mikael Pettersson wrote:
> On Sat, 30 Apr 2011 21:46:56 +0200 (CEST), "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > The following bug entry is on the current list of known regressions
> > from 2.6.38.  Please verify if it still should be listed and let the tracking team
> > know (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33492
> > Subject		: [bug] 2.6.39-rc3 fails to boot on Alpha
> > Submitter	: Mikael Pettersson <mikpe@it.uu.se>
> > Date		: 2011-04-17 15:13 (14 days old)
> > Message-ID	: <19883.912.266127.538215@pilspetsen.it.uu.se>
> > References	: http://marc.info/?l=linux-kernel&m=130305321212360&w=2
> 
> Fixed in 2.6.39-rc5.

Thanks, closing.

Rafael

[Bug #33842] NULL pointer dereference in ip_fragment

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33842
Subject		: NULL pointer dereference in ip_fragment
Submitter	: Tomas Carnecky <tom@dbservice.com>
Date		: 2011-04-23 07:51 (8 days old)

[Bug #34002] [REGRESSION] [2.6.39-rc3] Wrong resolution in framebuffer and X Window

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=34002
Subject		: [REGRESSION] [2.6.39-rc3] Wrong resolution in framebuffer and X Window
Submitter	: Maciej Rutecki <maciej.rutecki@gmail.com>
Date		: 2011-04-17 16:04 (14 days old)
Message-ID	: <201104171804.04664.maciej.rutecki@gmail.com>
References	: http://marc.info/?l=linux-fbdev&m=130305625114863&w=2

[Bug #34012] 2.6.39-rc4+: oom-killer busy killing tasks

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=34012
Subject		: 2.6.39-rc4+: oom-killer busy killing tasks
Submitter	: Christian Kujau <lists@nerdbynature.de>
Date		: 2011-04-22 1:57 (9 days old)
Message-ID	: <alpine.DEB.2.01.1104211841510.18728@trent.utfs.org>
References	: http://marc.info/?l=linux-kernel&m=130343744622331&w=2

[Bug #33812] radeon with modeset option causes oops as secondary adapter (vgaswitcheroo)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33812
Subject		: radeon with modeset option causes oops as secondary adapter (vgaswitcheroo)
Submitter	: Johannes Engel <jcnengel@googlemail.com>
Date		: 2011-04-21 23:14 (10 days old)

Re: 2.6.39-rc5-git4: Reported regressions from 2.6.38

[Linus Torvalds]

On Sat, Apr 30, 2011 at 12:42 PM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
>
> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=34012
> Subject         : 2.6.39-rc4+: oom-killer busy killing tasks
> Submitter       : Christian Kujau <lists@nerdbynature.de>
> Date            : 2011-04-22 1:57 (9 days old)
> Message-ID      : <alpine.DEB.2.01.1104211841510.18728@trent.utfs.org>
> References      : http://marc.info/?l=linux-kernel&m=130343744622331&w=2

Judging by the extended debug info in:

   http://nerdbynature.de/bits/2.6.39-rc4/oom/
   (The -9 files are the current ones)

that thing shows a _lot_ of xfs inodes:

  xfs_inode         479187 479187   1120   14    4 : tunables    0
0    0 : slabdata  34329  34329      0

even though there aren't that many dentries (each inode should have at
least one dentry associated with it under normal circumstances):

  dentry             97896  97900    160   25    1 : tunables    0
0    0 : slabdata   3916   3916      0

and then later when the system gets low on memory, the dentries shrink:

  dentry              1017   3525    160   25    1 : tunables    0
0    0 : slabdata    141    141      0

but the XFS inodes do not:

  xfs_inode         557579 557579   1120   14    4 : tunables    0
0    0 : slabdata  41492  41492      0

so I suspect it's some kind of XFS inode leak.

                      Linus

2.6.39-rc3-git7: Reported regressions from 2.6.38

[Rafael J. Wysocki]

This message contains a list of some regressions from 2.6.38,
for which there are no fixes in the mainline known to the tracking team.
If any of them have been fixed already, please let us know.

If you know of any other unresolved regressions from 2.6.38, please let us
know either and we'll add them to the list.  Also, please let us know
if any of the entries below are invalid.

Each entry from the list will be sent additionally in an automatic reply
to this message with CCs to the people involved in reporting and handling
the issue.


Listed regressions statistics:

  Date          Total  Pending  Unresolved
  ----------------------------------------
  2011-04-17       17       11          10


Unresolved regressions
----------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33342
Subject		: [2.6.39-rc2][bisected] Constant DISK_MEDIA_CHANGE_EVENTS from CDROM drive.
Submitter	: Shaun Ruffell <sruffell@digium.com>
Date		: 2011-04-08 20:15 (10 days old)
Message-ID	: <20110408201513.GA3040@digium.com>
References	: http://marc.info/?l=linux-kernel&m=130229371907209&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33272
Subject		: drm related hard-hang
Submitter	: Peter Teoh <htmldeveloper@gmail.com>
Date		: 2011-04-14 01:29 (4 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33242
Subject		: Lockdep splat in autofs with 2.6.39-rc2
Submitter	: Nick Bowler <nbowler@elliptictech.com>
Date		: 2011-04-07 19:44 (11 days old)
Message-ID	: <20110407194403.GA29404@elliptictech.com>
References	: http://marc.info/?l=linux-kernel&m=130220545614682&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33142
Subject		: 2.6.39-rc2 regression: X201s fails to resume b77dcf8460ae57d4eb9fd3633eb4f97b8fb20716
Submitter	: Keith Packard <keithp@keithp.com>
Date		: 2011-04-06 7:44 (12 days old)
Message-ID	: <yun1v1fj024.fsf@aiko.keithp.com>
References	: http://marc.info/?l=linux-kernel&m=130207593728273&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33102
Subject		: File's copied from client->linux server only copy 1st 64K data;rest is lost
Submitter	: Linda Walsh <lkml@tlinx.org>
Date		: 2011-04-11 22:12 (7 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33092
Subject		: [regression] 2.6.39-rc1 - Beagleboard usbnet broken
Submitter	: Mark Jackson <mpfj-list@mimc.co.uk>
Date		: 2011-04-04 9:22 (14 days old)
First-Bad-Commit: http://git.kernel.org/linus/087809fce28f50098d9c3ef1a6865c722f23afd2
Message-ID	: <4D998DC9.3040109@mimc.co.uk>
References	: http://marc.info/?l=linux-kernel&m=130191386508831&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=32982
Subject		: Kernel locks up a few minutes after boot
Submitter	: Bart Van Assche <bart.vanassche@gmail.com>
Date		: 2011-04-10 19:55 (8 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=32902
Subject		: 2.6.39-rc1 doesn't boot on thinkpad t61p x86_64
Submitter	: Alex Romosan <romosan@sycorax.lbl.gov>
Date		: 2011-04-03 19:41 (15 days old)
Message-ID	: <87k4fbnmw8.fsf@sycorax.lbl.gov>
References	: http://marc.info/?l=linux-kernel&m=130186054431678&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=32892
Subject		: 2.6.39-rc1 data corruption with rtorrent
Submitter	: Jindrich Makovicka <makovick@gmail.com>
Date		: 2011-04-02 20:21 (16 days old)
Message-ID	: <20110402222118.3b5c2fa8@holly>
References	: http://marc.info/?l=linux-kernel&m=130177570309226&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=32262
Subject		: 2.6.38-git15+  IDE hangs boot
Submitter	: Pete Clements <clem@clem.clem-digital.net>
Date		: 2011-03-25 15:38 (24 days old)
Message-ID	: <201103251538.p2PFc11i001674@clem.clem-digital.net>
References	: http://marc.info/?l=linux-kernel&m=130106749313695&w=2


Regressions with patches
------------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=33252
Subject		: [regression 2.6.39-rc2][bisected] "perf, x86: P4 PMU - Read proper MSR register to catch" and NMIs
Submitter	: Shaun Ruffell <sruffell@digium.com>
Date		: 2011-04-06 22:30 (12 days old)
First-Bad-Commit: http://git.kernel.org/linus/242214f9c1eeaae40eca11e3b4d37bfce960a7cd
Message-ID	: <20110406223036.GA15721@digium.com>
References	: http://marc.info/?l=linux-kernel&m=130212907032580&w=2
Handled-By	: Don Zickus <dzickus@redhat.com>
Patch		: http://cache.gmane.org//gmane/linux/kernel/1125621-001.bin


For details, please visit the bug entries and follow the links given in
references.

As you can see, there is a Bugzilla entry for each of the listed regressions.
There also is a Bugzilla entry used for tracking the regressions from 2.6.38,
unresolved as well as resolved, at:

http://bugzilla.kernel.org/show_bug.cgi?id=32012

Please let the tracking team know if there are any Bugzilla entries that
should be added to the list in there.

Thanks!

[Bug #32982] Kernel locks up a few minutes after boot

[Rafael J. Wysocki]

This message has been generated automatically as a part of a summary report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.38.  Please verify if it still should be listed and let the tracking team
know (either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=32982
Subject		: Kernel locks up a few minutes after boot
Submitter	: Bart Van Assche <bart.vanassche@gmail.com>
Date		: 2011-04-10 19:55 (8 days old)

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Linus Torvalds]

Is this machine running a RAID5 setup or something like that?

There is a known interaction with the new block layer plugging code
and MD. The "hung task" report in that bugzilla looks very much like
that issue. And you do have "root=/dev/md0", so clearly there's some
md thing going on.

And bisecting might not work all that well for it, because I suspect
it ends up being very much a matter of IO patterns how it triggers.

Neil supposedly has a patch for it, but I haven't seen it yet. Neil, Jens?

                                   Linus

On Sun, Apr 17, 2011 at 5:57 AM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> This message has been generated automatically as a part of a summary report
> of recent regressions.
>
> The following bug entry is on the current list of known regressions
> from 2.6.38.  Please verify if it still should be listed and let the tracking team
> know (either way).
>
>
> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=32982
> Subject         : Kernel locks up a few minutes after boot
> Submitter       : Bart Van Assche <bart.vanassche@gmail.com>
> Date            : 2011-04-10 19:55 (8 days old)
>
>
>

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Bart Van Assche]

On Sun, Apr 17, 2011 at 7:03 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Sun, Apr 17, 2011 at 5:57 AM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> > This message has been generated automatically as a part of a summary report
> > of recent regressions.
> >
> > The following bug entry is on the current list of known regressions
> > from 2.6.38.  Please verify if it still should be listed and let the tracking team
> > know (either way).
> >
> >
> > Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=32982
> > Subject         : Kernel locks up a few minutes after boot
> > Submitter       : Bart Van Assche <bart.vanassche@gmail.com>
> > Date            : 2011-04-10 19:55 (8 days old)
>
> Is this machine running a RAID5 setup or something like that?
>
> There is a known interaction with the new block layer plugging code
> and MD. The "hung task" report in that bugzilla looks very much like
> that issue. And you do have "root=/dev/md0", so clearly there's some
> md thing going on.
>
> And bisecting might not work all that well for it, because I suspect
> it ends up being very much a matter of IO patterns how it triggers.
>
> Neil supposedly has a patch for it, but I haven't seen it yet. Neil, Jens?

(converted top-posting into bottom-posting)

Hello Linus,

On the system on which bug #32982 has been triggered md0, md1 and md2
have been configured as two-disk RAID1 (mirroring).

I've done my best to trigger enough I/O in order to obtain reliable
bisect results. A difficulty I encountered during bisecting though was
that I encountered unbootable kernels (all skipped revisions).

Bart.

Re: [Bug #32982] Kernel locks up a few minutes after boot

[NeilBrown]

On Sun, 17 Apr 2011 20:37:39 +0200 Bart Van Assche <bvanassche@acm.org> wrote:

> On Sun, Apr 17, 2011 at 7:03 PM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> > On Sun, Apr 17, 2011 at 5:57 AM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> > > This message has been generated automatically as a part of a summary report
> > > of recent regressions.
> > >
> > > The following bug entry is on the current list of known regressions
> > > from 2.6.38.  Please verify if it still should be listed and let the tracking team
> > > know (either way).
> > >
> > >
> > > Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=32982
> > > Subject         : Kernel locks up a few minutes after boot
> > > Submitter       : Bart Van Assche <bart.vanassche@gmail.com>
> > > Date            : 2011-04-10 19:55 (8 days old)
> >
> > Is this machine running a RAID5 setup or something like that?
> >
> > There is a known interaction with the new block layer plugging code
> > and MD. The "hung task" report in that bugzilla looks very much like
> > that issue. And you do have "root=/dev/md0", so clearly there's some
> > md thing going on.
> >
> > And bisecting might not work all that well for it, because I suspect
> > it ends up being very much a matter of IO patterns how it triggers.
> >
> > Neil supposedly has a patch for it, but I haven't seen it yet. Neil, Jens?
> 
> (converted top-posting into bottom-posting)
> 
> Hello Linus,
> 
> On the system on which bug #32982 has been triggered md0, md1 and md2
> have been configured as two-disk RAID1 (mirroring).

If any of those have write-intent bitmaps then I definitely know what the
problem is and I'll be posting patches later today (probably not much later).

If not .. then I'm less sure but it would certainly be worth testing after
applying the promised fixes.

NeilBrown


> 
> I've done my best to trigger enough I/O in order to obtain reliable
> bisect results. A difficulty I encountered during bisecting though was
> that I encountered unbootable kernels (all skipped revisions).
> 
> Bart.

Re: [Bug #32982] Kernel locks up a few minutes after boot

[NeilBrown]

On Mon, 18 Apr 2011 07:07:11 +1000 NeilBrown <neilb@suse.de> wrote:

> On Sun, 17 Apr 2011 20:37:39 +0200 Bart Van Assche <bvanassche@acm.org> wrote:
> 
> > On Sun, Apr 17, 2011 at 7:03 PM, Linus Torvalds
> > <torvalds@linux-foundation.org> wrote:
> > > On Sun, Apr 17, 2011 at 5:57 AM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> > > > This message has been generated automatically as a part of a summary report
> > > > of recent regressions.
> > > >
> > > > The following bug entry is on the current list of known regressions
> > > > from 2.6.38.  Please verify if it still should be listed and let the tracking team
> > > > know (either way).
> > > >
> > > >
> > > > Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=32982
> > > > Subject         : Kernel locks up a few minutes after boot
> > > > Submitter       : Bart Van Assche <bart.vanassche@gmail.com>
> > > > Date            : 2011-04-10 19:55 (8 days old)
> > >
> > > Is this machine running a RAID5 setup or something like that?
> > >
> > > There is a known interaction with the new block layer plugging code
> > > and MD. The "hung task" report in that bugzilla looks very much like
> > > that issue. And you do have "root=/dev/md0", so clearly there's some
> > > md thing going on.
> > >
> > > And bisecting might not work all that well for it, because I suspect
> > > it ends up being very much a matter of IO patterns how it triggers.
> > >
> > > Neil supposedly has a patch for it, but I haven't seen it yet. Neil, Jens?
> > 
> > (converted top-posting into bottom-posting)
> > 
> > Hello Linus,
> > 
> > On the system on which bug #32982 has been triggered md0, md1 and md2
> > have been configured as two-disk RAID1 (mirroring).
> 
> If any of those have write-intent bitmaps then I definitely know what the
> problem is and I'll be posting patches later today (probably not much later).
> 

Actually it won't be today.  The new block device plugging is still unusable
for MD - so I won't be able to fix this until that gets sorted out.

NeilBrown

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Jens Axboe]

On 2011-04-17 20:37, Bart Van Assche wrote:
> On Sun, Apr 17, 2011 at 7:03 PM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>> On Sun, Apr 17, 2011 at 5:57 AM, Rafael J. Wysocki <rjw@sisk.pl> wrote:
>>> This message has been generated automatically as a part of a summary report
>>> of recent regressions.
>>>
>>> The following bug entry is on the current list of known regressions
>>> from 2.6.38.  Please verify if it still should be listed and let the tracking team
>>> know (either way).
>>>
>>>
>>> Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=32982
>>> Subject         : Kernel locks up a few minutes after boot
>>> Submitter       : Bart Van Assche <bart.vanassche@gmail.com>
>>> Date            : 2011-04-10 19:55 (8 days old)
>>
>> Is this machine running a RAID5 setup or something like that?
>>
>> There is a known interaction with the new block layer plugging code
>> and MD. The "hung task" report in that bugzilla looks very much like
>> that issue. And you do have "root=/dev/md0", so clearly there's some
>> md thing going on.
>>
>> And bisecting might not work all that well for it, because I suspect
>> it ends up being very much a matter of IO patterns how it triggers.
>>
>> Neil supposedly has a patch for it, but I haven't seen it yet. Neil, Jens?
> 
> (converted top-posting into bottom-posting)
> 
> Hello Linus,
> 
> On the system on which bug #32982 has been triggered md0, md1 and md2
> have been configured as two-disk RAID1 (mirroring).
> 
> I've done my best to trigger enough I/O in order to obtain reliable
> bisect results. A difficulty I encountered during bisecting though was
> that I encountered unbootable kernels (all skipped revisions).

Bart, can you try and pull:

git://git.kernel.dk/linux-2.6-block.git for-linus

into Linus' tree and see if that works? This has, among other things,
Neils fixes for MD.

-- 
Jens Axboe

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Bart Van Assche]

On Mon, Apr 18, 2011 at 1:44 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
> Bart, can you try and pull:
>
> git://git.kernel.dk/linux-2.6-block.git for-linus
>
> into Linus' tree and see if that works? This has, among other things,
> Neils fixes for MD.

md seems to work stable with the resulting tree, but it looks there is
a performance regression in the block layer not related to the md
issue. If I run a small block IOPS test on a block device created by
ib_srp (NOOP scheduler) I see about 11% less IOPS than with 2.6.38.3
(155.000 IOPS with 2.6.38.3 and 140.000 IOPS with 2.6.39-rc3+).

Bart.

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Jens Axboe]

On 2011-04-18 20:21, Bart Van Assche wrote:
> On Mon, Apr 18, 2011 at 1:44 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
>> Bart, can you try and pull:
>>
>> git://git.kernel.dk/linux-2.6-block.git for-linus
>>
>> into Linus' tree and see if that works? This has, among other things,
>> Neils fixes for MD.
> 
> md seems to work stable with the resulting tree, but it looks there is

OK, that's the most important bit.

> a performance regression in the block layer not related to the md
> issue. If I run a small block IOPS test on a block device created by
> ib_srp (NOOP scheduler) I see about 11% less IOPS than with 2.6.38.3
> (155.000 IOPS with 2.6.38.3 and 140.000 IOPS with 2.6.39-rc3+).

That's not good. What's the test case?

-- 
Jens Axboe

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Bart Van Assche]

On Mon, Apr 18, 2011 at 8:28 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
> On 2011-04-18 20:21, Bart Van Assche wrote:
>> a performance regression in the block layer not related to the md
>> issue. If I run a small block IOPS test on a block device created by
>> ib_srp (NOOP scheduler) I see about 11% less IOPS than with 2.6.38.3
>> (155.000 IOPS with 2.6.38.3 and 140.000 IOPS with 2.6.39-rc3+).
>
> That's not good. What's the test case?

Nothing more than a fio IOPS test:

fio --bs=512 --ioengine=libaio --buffered=0 --rw=read --thread
--iodepth=64 --numjobs=2 --loops=10000 --group_reporting --size=1G
    --gtod_reduce=1 --name=iops-test --filename=/dev/${dev} --invalidate=1

Bart.

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Jens Axboe]

On 2011-04-18 20:32, Bart Van Assche wrote:
> On Mon, Apr 18, 2011 at 8:28 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
>> On 2011-04-18 20:21, Bart Van Assche wrote:
>>> a performance regression in the block layer not related to the md
>>> issue. If I run a small block IOPS test on a block device created by
>>> ib_srp (NOOP scheduler) I see about 11% less IOPS than with 2.6.38.3
>>> (155.000 IOPS with 2.6.38.3 and 140.000 IOPS with 2.6.39-rc3+).
>>
>> That's not good. What's the test case?
> 
> Nothing more than a fio IOPS test:
> 
> fio --bs=512 --ioengine=libaio --buffered=0 --rw=read --thread
> --iodepth=64 --numjobs=2 --loops=10000 --group_reporting --size=1G
>     --gtod_reduce=1 --name=iops-test --filename=/dev/${dev} --invalidate=1

Interesting, I'll have to check if we regressed with all these recent
changes. Comparing your .38 to .39-rc3+, are you using more/less CPU,
more/less sys%, etc?

A quick perf record -fg / perf report -g for both kernels would be nice
to see.

-- 
Jens Axboe

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Jens Axboe]

On 2011-04-18 20:32, Bart Van Assche wrote:
> On Mon, Apr 18, 2011 at 8:28 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
>> On 2011-04-18 20:21, Bart Van Assche wrote:
>>> a performance regression in the block layer not related to the md
>>> issue. If I run a small block IOPS test on a block device created by
>>> ib_srp (NOOP scheduler) I see about 11% less IOPS than with 2.6.38.3
>>> (155.000 IOPS with 2.6.38.3 and 140.000 IOPS with 2.6.39-rc3+).
>>
>> That's not good. What's the test case?
> 
> Nothing more than a fio IOPS test:
> 
> fio --bs=512 --ioengine=libaio --buffered=0 --rw=read --thread
> --iodepth=64 --numjobs=2 --loops=10000 --group_reporting --size=1G
>     --gtod_reduce=1 --name=iops-test --filename=/dev/${dev} --invalidate=1

Bart, can you try the below:

diff --git a/block/blk-core.c b/block/blk-core.c
index 5fa3dd2..9b41da1 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -307,11 +307,7 @@ void __blk_run_queue(struct request_queue *q)
 	 * Only recurse once to avoid overrunning the stack, let the unplug
 	 * handling reinvoke the handler shortly if we already got there.
 	 */
-	if (!queue_flag_test_and_set(QUEUE_FLAG_REENTER, q)) {
-		q->request_fn(q);
-		queue_flag_clear(QUEUE_FLAG_REENTER, q);
-	} else
-		queue_delayed_work(kblockd_workqueue, &q->delay_work, 0);
+	q->request_fn(q);
 }
 EXPORT_SYMBOL(__blk_run_queue);
 

-- 
Jens Axboe

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Jens Axboe]

On 2011-04-19 11:09, Jens Axboe wrote:
> On 2011-04-18 20:32, Bart Van Assche wrote:
>> On Mon, Apr 18, 2011 at 8:28 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
>>> On 2011-04-18 20:21, Bart Van Assche wrote:
>>>> a performance regression in the block layer not related to the md
>>>> issue. If I run a small block IOPS test on a block device created by
>>>> ib_srp (NOOP scheduler) I see about 11% less IOPS than with 2.6.38.3
>>>> (155.000 IOPS with 2.6.38.3 and 140.000 IOPS with 2.6.39-rc3+).
>>>
>>> That's not good. What's the test case?
>>
>> Nothing more than a fio IOPS test:
>>
>> fio --bs=512 --ioengine=libaio --buffered=0 --rw=read --thread
>> --iodepth=64 --numjobs=2 --loops=10000 --group_reporting --size=1G
>>     --gtod_reduce=1 --name=iops-test --filename=/dev/${dev} --invalidate=1
> 
> Bart, can you try the below:

Here's a more complete variant. James, lets get rid of this REENTER
crap. It's completely bogus and triggers falsely for a variety of
reasons. The below will work, but there may be room for improvement on
the SCSI side.

diff --git a/block/blk-core.c b/block/blk-core.c
index 5fa3dd2..4e49665 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -303,15 +303,7 @@ void __blk_run_queue(struct request_queue *q)
 	if (unlikely(blk_queue_stopped(q)))
 		return;
 
-	/*
-	 * Only recurse once to avoid overrunning the stack, let the unplug
-	 * handling reinvoke the handler shortly if we already got there.
-	 */
-	if (!queue_flag_test_and_set(QUEUE_FLAG_REENTER, q)) {
-		q->request_fn(q);
-		queue_flag_clear(QUEUE_FLAG_REENTER, q);
-	} else
-		queue_delayed_work(kblockd_workqueue, &q->delay_work, 0);
+	q->request_fn(q);
 }
 EXPORT_SYMBOL(__blk_run_queue);
 
@@ -328,6 +320,7 @@ void blk_run_queue_async(struct request_queue *q)
 	if (likely(!blk_queue_stopped(q)))
 		queue_delayed_work(kblockd_workqueue, &q->delay_work, 0);
 }
+EXPORT_SYMBOL(blk_run_queue_async);
 
 /**
  * blk_run_queue - run a single device queue
diff --git a/block/blk.h b/block/blk.h
index c9df8fc..6126346 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -22,7 +22,6 @@ void blk_rq_timed_out_timer(unsigned long data);
 void blk_delete_timer(struct request *);
 void blk_add_timer(struct request *);
 void __generic_unplug_device(struct request_queue *);
-void blk_run_queue_async(struct request_queue *q);
 
 /*
  * Internal atomic flags for request handling
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index ab55c2f..e9901b8 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -411,8 +411,6 @@ static void scsi_run_queue(struct request_queue *q)
 	list_splice_init(&shost->starved_list, &starved_list);
 
 	while (!list_empty(&starved_list)) {
-		int flagset;
-
 		/*
 		 * As long as shost is accepting commands and we have
 		 * starved queues, call blk_run_queue. scsi_request_fn
@@ -435,20 +433,7 @@ static void scsi_run_queue(struct request_queue *q)
 			continue;
 		}
 
-		spin_unlock(shost->host_lock);
-
-		spin_lock(sdev->request_queue->queue_lock);
-		flagset = test_bit(QUEUE_FLAG_REENTER, &q->queue_flags) &&
-				!test_bit(QUEUE_FLAG_REENTER,
-					&sdev->request_queue->queue_flags);
-		if (flagset)
-			queue_flag_set(QUEUE_FLAG_REENTER, sdev->request_queue);
-		__blk_run_queue(sdev->request_queue);
-		if (flagset)
-			queue_flag_clear(QUEUE_FLAG_REENTER, sdev->request_queue);
-		spin_unlock(sdev->request_queue->queue_lock);
-
-		spin_lock(shost->host_lock);
+		blk_run_queue_async(sdev->request_queue);
 	}
 	/* put any unprocessed entries back */
 	list_splice(&starved_list, &shost->starved_list);
diff --git a/drivers/scsi/scsi_transport_fc.c b/drivers/scsi/scsi_transport_fc.c
index 28c3350..815069d 100644
--- a/drivers/scsi/scsi_transport_fc.c
+++ b/drivers/scsi/scsi_transport_fc.c
@@ -3816,28 +3816,17 @@ fail_host_msg:
 static void
 fc_bsg_goose_queue(struct fc_rport *rport)
 {
-	int flagset;
-	unsigned long flags;
-
 	if (!rport->rqst_q)
 		return;
 
+	/*
+	 * This get/put dance makes no sense
+	 */
 	get_device(&rport->dev);
-
-	spin_lock_irqsave(rport->rqst_q->queue_lock, flags);
-	flagset = test_bit(QUEUE_FLAG_REENTER, &rport->rqst_q->queue_flags) &&
-		  !test_bit(QUEUE_FLAG_REENTER, &rport->rqst_q->queue_flags);
-	if (flagset)
-		queue_flag_set(QUEUE_FLAG_REENTER, rport->rqst_q);
-	__blk_run_queue(rport->rqst_q);
-	if (flagset)
-		queue_flag_clear(QUEUE_FLAG_REENTER, rport->rqst_q);
-	spin_unlock_irqrestore(rport->rqst_q->queue_lock, flags);
-
+	blk_run_queue_async(rport->rqst_q);
 	put_device(&rport->dev);
 }
 
-
 /**
  * fc_bsg_rport_dispatch - process rport bsg requests and dispatch to LLDD
  * @q:		rport request queue
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index cbbfd98..2ad95fa 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -388,20 +388,19 @@ struct request_queue
 #define	QUEUE_FLAG_SYNCFULL	3	/* read queue has been filled */
 #define QUEUE_FLAG_ASYNCFULL	4	/* write queue has been filled */
 #define QUEUE_FLAG_DEAD		5	/* queue being torn down */
-#define QUEUE_FLAG_REENTER	6	/* Re-entrancy avoidance */
-#define QUEUE_FLAG_ELVSWITCH	7	/* don't use elevator, just do FIFO */
-#define QUEUE_FLAG_BIDI		8	/* queue supports bidi requests */
-#define QUEUE_FLAG_NOMERGES     9	/* disable merge attempts */
-#define QUEUE_FLAG_SAME_COMP   10	/* force complete on same CPU */
-#define QUEUE_FLAG_FAIL_IO     11	/* fake timeout */
-#define QUEUE_FLAG_STACKABLE   12	/* supports request stacking */
-#define QUEUE_FLAG_NONROT      13	/* non-rotational device (SSD) */
+#define QUEUE_FLAG_ELVSWITCH	6	/* don't use elevator, just do FIFO */
+#define QUEUE_FLAG_BIDI		7	/* queue supports bidi requests */
+#define QUEUE_FLAG_NOMERGES     8	/* disable merge attempts */
+#define QUEUE_FLAG_SAME_COMP	9	/* force complete on same CPU */
+#define QUEUE_FLAG_FAIL_IO     10	/* fake timeout */
+#define QUEUE_FLAG_STACKABLE   11	/* supports request stacking */
+#define QUEUE_FLAG_NONROT      12	/* non-rotational device (SSD) */
 #define QUEUE_FLAG_VIRT        QUEUE_FLAG_NONROT /* paravirt device */
-#define QUEUE_FLAG_IO_STAT     15	/* do IO stats */
-#define QUEUE_FLAG_DISCARD     16	/* supports DISCARD */
-#define QUEUE_FLAG_NOXMERGES   17	/* No extended merges */
-#define QUEUE_FLAG_ADD_RANDOM  18	/* Contributes to random pool */
-#define QUEUE_FLAG_SECDISCARD  19	/* supports SECDISCARD */
+#define QUEUE_FLAG_IO_STAT     13	/* do IO stats */
+#define QUEUE_FLAG_DISCARD     14	/* supports DISCARD */
+#define QUEUE_FLAG_NOXMERGES   15	/* No extended merges */
+#define QUEUE_FLAG_ADD_RANDOM  16	/* Contributes to random pool */
+#define QUEUE_FLAG_SECDISCARD  17	/* supports SECDISCARD */
 
 #define QUEUE_FLAG_DEFAULT	((1 << QUEUE_FLAG_IO_STAT) |		\
 				 (1 << QUEUE_FLAG_STACKABLE)	|	\
@@ -699,6 +698,7 @@ extern void blk_sync_queue(struct request_queue *q);
 extern void __blk_stop_queue(struct request_queue *q);
 extern void __blk_run_queue(struct request_queue *q);
 extern void blk_run_queue(struct request_queue *);
+extern void blk_run_queue_async(struct request_queue *q);
 extern int blk_rq_map_user(struct request_queue *, struct request *,
 			   struct rq_map_data *, void __user *, unsigned long,
 			   gfp_t);

-- 
Jens Axboe

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Bart Van Assche]

On Tue, Apr 19, 2011 at 1:16 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
> On 2011-04-19 11:09, Jens Axboe wrote:
> > On 2011-04-18 20:32, Bart Van Assche wrote:
> >> On Mon, Apr 18, 2011 at 8:28 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
> >>> On 2011-04-18 20:21, Bart Van Assche wrote:
> >>>> a performance regression in the block layer not related to the md
> >>>> issue. If I run a small block IOPS test on a block device created by
> >>>> ib_srp (NOOP scheduler) I see about 11% less IOPS than with 2.6.38.3
> >>>> (155.000 IOPS with 2.6.38.3 and 140.000 IOPS with 2.6.39-rc3+).
> >>>
> >>> That's not good. What's the test case?
> >>
> >> Nothing more than a fio IOPS test:
> >>
> >> fio --bs=512 --ioengine=libaio --buffered=0 --rw=read --thread
> >> --iodepth=64 --numjobs=2 --loops=10000 --group_reporting --size=1G
> >>     --gtod_reduce=1 --name=iops-test --filename=/dev/${dev} --invalidate=1
> >
> > Bart, can you try the below:
>
> Here's a more complete variant. James, lets get rid of this REENTER
> crap. It's completely bogus and triggers falsely for a variety of
> reasons. The below will work, but there may be room for improvement on
> the SCSI side.
>
> diff --git a/block/blk-core.c b/block/blk-core.c
> index 5fa3dd2..4e49665 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -303,15 +303,7 @@ void __blk_run_queue(struct request_queue *q)
>        if (unlikely(blk_queue_stopped(q)))
>                return;
>
> -       /*
> -        * Only recurse once to avoid overrunning the stack, let the unplug
> -        * handling reinvoke the handler shortly if we already got there.
> -        */
> -       if (!queue_flag_test_and_set(QUEUE_FLAG_REENTER, q)) {
> -               q->request_fn(q);
> -               queue_flag_clear(QUEUE_FLAG_REENTER, q);
> -       } else
> -               queue_delayed_work(kblockd_workqueue, &q->delay_work, 0);
> +       q->request_fn(q);
>  }
>  EXPORT_SYMBOL(__blk_run_queue);
>
> @@ -328,6 +320,7 @@ void blk_run_queue_async(struct request_queue *q)
>        if (likely(!blk_queue_stopped(q)))
>                queue_delayed_work(kblockd_workqueue, &q->delay_work, 0);
>  }
> +EXPORT_SYMBOL(blk_run_queue_async);
>
>  /**
>  * blk_run_queue - run a single device queue
> diff --git a/block/blk.h b/block/blk.h
> index c9df8fc..6126346 100644
> --- a/block/blk.h
> +++ b/block/blk.h
> @@ -22,7 +22,6 @@ void blk_rq_timed_out_timer(unsigned long data);
>  void blk_delete_timer(struct request *);
>  void blk_add_timer(struct request *);
>  void __generic_unplug_device(struct request_queue *);
> -void blk_run_queue_async(struct request_queue *q);
>
>  /*
>  * Internal atomic flags for request handling
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> index ab55c2f..e9901b8 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -411,8 +411,6 @@ static void scsi_run_queue(struct request_queue *q)
>        list_splice_init(&shost->starved_list, &starved_list);
>
>        while (!list_empty(&starved_list)) {
> -               int flagset;
> -
>                /*
>                 * As long as shost is accepting commands and we have
>                 * starved queues, call blk_run_queue. scsi_request_fn
> @@ -435,20 +433,7 @@ static void scsi_run_queue(struct request_queue *q)
>                        continue;
>                }
>
> -               spin_unlock(shost->host_lock);
> -
> -               spin_lock(sdev->request_queue->queue_lock);
> -               flagset = test_bit(QUEUE_FLAG_REENTER, &q->queue_flags) &&
> -                               !test_bit(QUEUE_FLAG_REENTER,
> -                                       &sdev->request_queue->queue_flags);
> -               if (flagset)
> -                       queue_flag_set(QUEUE_FLAG_REENTER, sdev->request_queue);
> -               __blk_run_queue(sdev->request_queue);
> -               if (flagset)
> -                       queue_flag_clear(QUEUE_FLAG_REENTER, sdev->request_queue);
> -               spin_unlock(sdev->request_queue->queue_lock);
> -
> -               spin_lock(shost->host_lock);
> +               blk_run_queue_async(sdev->request_queue);
>        }
>        /* put any unprocessed entries back */
>        list_splice(&starved_list, &shost->starved_list);
> diff --git a/drivers/scsi/scsi_transport_fc.c b/drivers/scsi/scsi_transport_fc.c
> index 28c3350..815069d 100644
> --- a/drivers/scsi/scsi_transport_fc.c
> +++ b/drivers/scsi/scsi_transport_fc.c
> @@ -3816,28 +3816,17 @@ fail_host_msg:
>  static void
>  fc_bsg_goose_queue(struct fc_rport *rport)
>  {
> -       int flagset;
> -       unsigned long flags;
> -
>        if (!rport->rqst_q)
>                return;
>
> +       /*
> +        * This get/put dance makes no sense
> +        */
>        get_device(&rport->dev);
> -
> -       spin_lock_irqsave(rport->rqst_q->queue_lock, flags);
> -       flagset = test_bit(QUEUE_FLAG_REENTER, &rport->rqst_q->queue_flags) &&
> -                 !test_bit(QUEUE_FLAG_REENTER, &rport->rqst_q->queue_flags);
> -       if (flagset)
> -               queue_flag_set(QUEUE_FLAG_REENTER, rport->rqst_q);
> -       __blk_run_queue(rport->rqst_q);
> -       if (flagset)
> -               queue_flag_clear(QUEUE_FLAG_REENTER, rport->rqst_q);
> -       spin_unlock_irqrestore(rport->rqst_q->queue_lock, flags);
> -
> +       blk_run_queue_async(rport->rqst_q);
>        put_device(&rport->dev);
>  }
>
> -
>  /**
>  * fc_bsg_rport_dispatch - process rport bsg requests and dispatch to LLDD
>  * @q:         rport request queue
> diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
> index cbbfd98..2ad95fa 100644
> --- a/include/linux/blkdev.h
> +++ b/include/linux/blkdev.h
> @@ -388,20 +388,19 @@ struct request_queue
>  #define        QUEUE_FLAG_SYNCFULL     3       /* read queue has been filled */
>  #define QUEUE_FLAG_ASYNCFULL   4       /* write queue has been filled */
>  #define QUEUE_FLAG_DEAD                5       /* queue being torn down */
> -#define QUEUE_FLAG_REENTER     6       /* Re-entrancy avoidance */
> -#define QUEUE_FLAG_ELVSWITCH   7       /* don't use elevator, just do FIFO */
> -#define QUEUE_FLAG_BIDI                8       /* queue supports bidi requests */
> -#define QUEUE_FLAG_NOMERGES     9      /* disable merge attempts */
> -#define QUEUE_FLAG_SAME_COMP   10      /* force complete on same CPU */
> -#define QUEUE_FLAG_FAIL_IO     11      /* fake timeout */
> -#define QUEUE_FLAG_STACKABLE   12      /* supports request stacking */
> -#define QUEUE_FLAG_NONROT      13      /* non-rotational device (SSD) */
> +#define QUEUE_FLAG_ELVSWITCH   6       /* don't use elevator, just do FIFO */
> +#define QUEUE_FLAG_BIDI                7       /* queue supports bidi requests */
> +#define QUEUE_FLAG_NOMERGES     8      /* disable merge attempts */
> +#define QUEUE_FLAG_SAME_COMP   9       /* force complete on same CPU */
> +#define QUEUE_FLAG_FAIL_IO     10      /* fake timeout */
> +#define QUEUE_FLAG_STACKABLE   11      /* supports request stacking */
> +#define QUEUE_FLAG_NONROT      12      /* non-rotational device (SSD) */
>  #define QUEUE_FLAG_VIRT        QUEUE_FLAG_NONROT /* paravirt device */
> -#define QUEUE_FLAG_IO_STAT     15      /* do IO stats */
> -#define QUEUE_FLAG_DISCARD     16      /* supports DISCARD */
> -#define QUEUE_FLAG_NOXMERGES   17      /* No extended merges */
> -#define QUEUE_FLAG_ADD_RANDOM  18      /* Contributes to random pool */
> -#define QUEUE_FLAG_SECDISCARD  19      /* supports SECDISCARD */
> +#define QUEUE_FLAG_IO_STAT     13      /* do IO stats */
> +#define QUEUE_FLAG_DISCARD     14      /* supports DISCARD */
> +#define QUEUE_FLAG_NOXMERGES   15      /* No extended merges */
> +#define QUEUE_FLAG_ADD_RANDOM  16      /* Contributes to random pool */
> +#define QUEUE_FLAG_SECDISCARD  17      /* supports SECDISCARD */
>
>  #define QUEUE_FLAG_DEFAULT     ((1 << QUEUE_FLAG_IO_STAT) |            \
>                                 (1 << QUEUE_FLAG_STACKABLE)    |       \
> @@ -699,6 +698,7 @@ extern void blk_sync_queue(struct request_queue *q);
>  extern void __blk_stop_queue(struct request_queue *q);
>  extern void __blk_run_queue(struct request_queue *q);
>  extern void blk_run_queue(struct request_queue *);
> +extern void blk_run_queue_async(struct request_queue *q);
>  extern int blk_rq_map_user(struct request_queue *, struct request *,
>                           struct rq_map_data *, void __user *, unsigned long,
>                           gfp_t);

Hello Jens,

The same test with an initiator running 2.6.39-rc4 +
git://git.kernel.dk/linux-2.6-block.git for-linus + the above patch
yields about 155.000 IOPS on my test setup, or the same performance as
with 2.6.38.3. I'm running the above patch through an I/O stress test
now.

Bart.

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Linus Torvalds]

On Tue, Apr 19, 2011 at 9:13 AM, Bart Van Assche <bvanassche@acm.org> wrote:
>
> The same test with an initiator running 2.6.39-rc4 +
> git://git.kernel.dk/linux-2.6-block.git for-linus + the above patch
> yields about 155.000 IOPS on my test setup, or the same performance as
> with 2.6.38.3. I'm running the above patch through an I/O stress test
> now.

Goodie. So not only does that patch get back the 11%, it removes the
crazy QUEUE_FLAG_REENTER flag that was broken to begin with. AND it
removes a number of complicated lines.

Halleluja.

                        Linus

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Jens Axboe]

On 2011-04-19 18:32, Linus Torvalds wrote:
> On Tue, Apr 19, 2011 at 9:13 AM, Bart Van Assche <bvanassche@acm.org> wrote:
>>
>> The same test with an initiator running 2.6.39-rc4 +
>> git://git.kernel.dk/linux-2.6-block.git for-linus + the above patch
>> yields about 155.000 IOPS on my test setup, or the same performance as
>> with 2.6.38.3. I'm running the above patch through an I/O stress test
>> now.
> 
> Goodie. So not only does that patch get back the 11%, it removes the
> crazy QUEUE_FLAG_REENTER flag that was broken to begin with. AND it
> removes a number of complicated lines.
> 
> Halleluja.

Indeed, coming your way soonish.

-- 
Jens Axboe

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Jens Axboe]

On 2011-04-19 18:13, Bart Van Assche wrote:
> The same test with an initiator running 2.6.39-rc4 +
> git://git.kernel.dk/linux-2.6-block.git for-linus + the above patch
> yields about 155.000 IOPS on my test setup, or the same performance as
> with 2.6.38.3. I'm running the above patch through an I/O stress test
> now.

OK, so parity, that's good. With the above patch, I can take a single
device from ~400K IOPS on 2.6.38 to ~440K IOPS on 2.6.39-rc4+patches.

-- 
Jens Axboe

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Christoph Hellwig]

> +		blk_run_queue_async(sdev->request_queue);

This doesn't even have to be async except when scsi drivers call
cmd->scsi_done directly.  It seems like if this always went through the
softirq (or kblockd) we could still run it in context for the others.

> +	/*
> +	 * This get/put dance makes no sense
> +	 */
>  	get_device(&rport->dev);
> -
> -	spin_lock_irqsave(rport->rqst_q->queue_lock, flags);
> -	flagset = test_bit(QUEUE_FLAG_REENTER, &rport->rqst_q->queue_flags) &&
> -		  !test_bit(QUEUE_FLAG_REENTER, &rport->rqst_q->queue_flags);
> -	if (flagset)
> -		queue_flag_set(QUEUE_FLAG_REENTER, rport->rqst_q);
> -	__blk_run_queue(rport->rqst_q);
> -	if (flagset)
> -		queue_flag_clear(QUEUE_FLAG_REENTER, rport->rqst_q);
> -	spin_unlock_irqrestore(rport->rqst_q->queue_lock, flags);
> -
> +	blk_run_queue_async(rport->rqst_q);

And the QUEUE_FLAG_REENTER mess here never made sense either as it
tested for a bit beeing set and not set at the same time.  So this one
actually should be able to be replaced by a plain blk_run_queue.

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Jens Axboe]

On 2011-04-19 18:48, Christoph Hellwig wrote:
>> +		blk_run_queue_async(sdev->request_queue);
> 
> This doesn't even have to be async except when scsi drivers call
> cmd->scsi_done directly.  It seems like if this always went through the
> softirq (or kblockd) we could still run it in context for the others.

Exactly. I'll pass an 'optimize' patch past James.

>> +	/*
>> +	 * This get/put dance makes no sense
>> +	 */
>>  	get_device(&rport->dev);
>> -
>> -	spin_lock_irqsave(rport->rqst_q->queue_lock, flags);
>> -	flagset = test_bit(QUEUE_FLAG_REENTER, &rport->rqst_q->queue_flags) &&
>> -		  !test_bit(QUEUE_FLAG_REENTER, &rport->rqst_q->queue_flags);
>> -	if (flagset)
>> -		queue_flag_set(QUEUE_FLAG_REENTER, rport->rqst_q);
>> -	__blk_run_queue(rport->rqst_q);
>> -	if (flagset)
>> -		queue_flag_clear(QUEUE_FLAG_REENTER, rport->rqst_q);
>> -	spin_unlock_irqrestore(rport->rqst_q->queue_lock, flags);
>> -
>> +	blk_run_queue_async(rport->rqst_q);
> 
> And the QUEUE_FLAG_REENTER mess here never made sense either as it
> tested for a bit beeing set and not set at the same time.  So this one
> actually should be able to be replaced by a plain blk_run_queue.

Yep, it's completely broken as-is.

-- 
Jens Axboe

Re: [Bug #32982] Kernel locks up a few minutes after boot

[David Dillow]

On Mon, 2011-04-18 at 20:21 +0200, Bart Van Assche wrote:
> On Mon, Apr 18, 2011 at 1:44 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
> > Bart, can you try and pull:
> >
> > git://git.kernel.dk/linux-2.6-block.git for-linus
> >
> > into Linus' tree and see if that works? This has, among other things,
> > Neils fixes for MD.
> 
> md seems to work stable with the resulting tree, but it looks there is
> a performance regression in the block layer not related to the md
> issue. If I run a small block IOPS test on a block device created by
> ib_srp (NOOP scheduler) I see about 11% less IOPS than with 2.6.38.3
> (155.000 IOPS with 2.6.38.3 and 140.000 IOPS with 2.6.39-rc3+).

The mapping code for ib_srp changed in 2.6.39-rc1, but it showed
improved IOPS for a similar setup in my testing so I'd be surprised if
it is the culprit. Still, it wouldn't hurt to check. Do you have time to
try the new ib_srp code with 2.6.38.3 to eliminate it from the equation?

Thanks,
Dave

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Bart Van Assche]

On Tue, Apr 19, 2011 at 5:32 AM, David Dillow <dave@thedillows.org> wrote:
>
> On Mon, 2011-04-18 at 20:21 +0200, Bart Van Assche wrote:
> > On Mon, Apr 18, 2011 at 1:44 PM, Jens Axboe <jaxboe@fusionio.com> wrote:
> > > Bart, can you try and pull:
> > >
> > > git://git.kernel.dk/linux-2.6-block.git for-linus
> > >
> > > into Linus' tree and see if that works? This has, among other things,
> > > Neils fixes for MD.
> >
> > md seems to work stable with the resulting tree, but it looks there is
> > a performance regression in the block layer not related to the md
> > issue. If I run a small block IOPS test on a block device created by
> > ib_srp (NOOP scheduler) I see about 11% less IOPS than with 2.6.38.3
> > (155.000 IOPS with 2.6.38.3 and 140.000 IOPS with 2.6.39-rc3+).
>
> The mapping code for ib_srp changed in 2.6.39-rc1, but it showed
> improved IOPS for a similar setup in my testing so I'd be surprised if
> it is the culprit. Still, it wouldn't hurt to check. Do you have time to
> try the new ib_srp code with 2.6.38.3 to eliminate it from the equation?

Hello Dave,

I just ran a test with the most important 2.6.39-specific ib_srp
commits reverted but that didn't yield a measurable performance
difference for this specific test:

$ git show --format=format:%s 7f9e5c48c1078507747434d4c182ab10925bf98a
be8b981453a4904399cb090c1660618e250092d8
c07d424d6118d528ef71b22b7424bfc359c307a5
8f26c9ff9cd0317ad867bce972f69e0c6c2cbe3c
961e0be89a5120a1409ebc525cca6f603615a8a8
8c4037b501acd2ec3abc7925e66af8af40a2da9d | grep '^IB'
IB: Increase DMA max_segment_size on Mellanox hardware
IB/srp: try to use larger FMR sizes to cover our mappings
IB/srp: add support for indirect tables that don't fit in SRP_CMD
IB/srp: rework mapping engine to use multiple FMR entries
IB/srp: move IB CM setup completion into its own function
IB/srp: always avoid non-zero offsets into an FMR

Bart.

Re: [Bug #32982] Kernel locks up a few minutes after boot

[Dave Dillow]

On 4/19/2011 12:39 PM, Bart Van Assche wrote:
> On Tue, Apr 19, 2011 at 5:32 AM, David Dillow<dave@thedillows.org>  wrote:
>> The mapping code for ib_srp changed in 2.6.39-rc1, but it showed
>> improved IOPS for a similar setup in my testing so I'd be surprised if
>> it is the culprit. Still, it wouldn't hurt to check. Do you have time to
>> try the new ib_srp code with 2.6.38.3 to eliminate it from the equation?
> Hello Dave,
>
> I just ran a test with the most important 2.6.39-specific ib_srp
> commits reverted but that didn't yield a measurable performance
> difference for this specific test:

Thanks for giving it a whirl,
Dave