linux-next.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Ming Lei <ming.lei@redhat.com>
To: Guenter Roeck <linux@roeck-us.net>
Cc: Ming Lei <tom.leiming@gmail.com>,
	linux-ide@vger.kernel.org, Tejun Heo <tj@kernel.org>,
	James Bottomley <James.Bottomley@hansenpartnership.com>,
	Stephen Rothwell <sfr@canb.auug.org.au>,
	Linux-Next Mailing List <linux-next@vger.kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	linux-scsi <linux-scsi@vger.kernel.org>,
	Christoph Hellwig <hch@lst.de>,
	Josef Bacik <josef@toxicpanda.com>, Jens Axboe <axboe@kernel.dk>
Subject: Re: linux-next: Tree for Aug 1
Date: Fri, 3 Aug 2018 00:27:03 +0800	[thread overview]
Message-ID: <20180802162654.GA8928@ming.t460p> (raw)
In-Reply-To: <a10a509a-7e5c-1706-52ee-79849cad4224@roeck-us.net>

On Thu, Aug 02, 2018 at 06:05:16AM -0700, Guenter Roeck wrote:
> On 08/02/2018 04:35 AM, Ming Lei wrote:
> > On Thu, Aug 2, 2018 at 12:58 PM, Guenter Roeck <linux@roeck-us.net> wrote:
> > > On 08/01/2018 05:03 PM, James Bottomley wrote:
> > > > 
> > > > On Thu, 2018-08-02 at 07:57 +0800, Ming Lei wrote:
> > > > > 
> > > > > On Thu, Aug 2, 2018 at 7:47 AM, Guenter Roeck <linux@roeck-us.net>
> > > > > wrote:
> > > > > > 
> > > > > > On Wed, Aug 01, 2018 at 03:52:45PM -0700, James Bottomley wrote:
> > > > > > > 
> > > > > > > On Wed, 2018-08-01 at 15:48 -0700, Guenter Roeck wrote:
> > > > > > > > 
> > > > > > > > On Wed, Aug 01, 2018 at 05:58:52PM +1000, Stephen Rothwell
> > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > Hi all,
> > > > > > > > > 
> > > > > > > > > Changes since 20180731:
> > > > > > > > > 
> > > > > > > > > The pci tree gained a conflict against the pci-current tree.
> > > > > > > > > 
> > > > > > > > > The net-next tree gained a conflict against the bpf tree.
> > > > > > > > > 
> > > > > > > > > The block tree lost its build failure.
> > > > > > > > > 
> > > > > > > > > The staging tree still had its build failure due to an
> > > > > > > > > interaction
> > > > > > > > > with
> > > > > > > > > the vfs tree for which I disabled CONFIG_EROFS_FS.
> > > > > > > > > 
> > > > > > > > > The kspp tree lost its build failure.
> > > > > > > > > 
> > > > > > > > > Non-merge commits (relative to Linus' tree): 10070
> > > > > > > > >    9137 files changed, 417605 insertions(+), 179996 deletions(-
> > > > > > > > > )
> > > > > > > > > 
> > > > > > > > > -----------------------------------------------------------
> > > > > > > > > ------
> > > > > > > > > -----------
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > The widespread kernel hang issues are still seen. I managed
> > > > > > > > to bisect it after working around the transient build failures.
> > > > > > > > Bisect log is attached below. Unfortunately, it doesn't help
> > > > > > > > much.
> > > > > > > > The culprit is reported as:
> > > > > > > > 
> > > > > > > > 2d542828c5e9 Merge remote-tracking branch 'scsi/for-next'
> > > > > > > > 
> > > > > > > > The preceding merge,
> > > > > > > > 
> > > > > > > > 453f1d821165 Merge remote-tracking branch 'cgroup/for-next'
> > > > > > > > 
> > > > > > > > checks out fine, as does the tip of scsi-next (commit
> > > > > > > > 103c7b7e0184,
> > > > > > > > "Merge branch 'misc' into for-next"). No idea how to proceed.
> > > > > > > 
> > > > > > > 
> > > > > > > This sounds like you may have a problem with this patch:
> > > > > > > 
> > > > > > >       commit d5038a13eca72fb216c07eb717169092e92284f1
> > > > > > >        Author: Johannes Thumshirn <jthumshirn@suse.de>
> > > > > > >        Date:   Wed Jul 4 10:53:56 2018 +0200
> > > > > > > 
> > > > > > >            scsi: core: switch to scsi-mq by default
> > > > > > > 
> > > > > > > To verify, boot with the additional kernel parameter
> > > > > > > 
> > > > > > > scsi_mod.use_blk_mq=0
> > > > > > > 
> > > > > > > Which will reverse the effect of the above patch.
> > > > > > > 
> > > > > > 
> > > > > > Yes, that fixes the problem.
> > > > > 
> > > > > 
> > > > > That may not the root cause, given this issue is only started to
> > > > > see from next-20180731, but d5038a13eca7 (scsi: core: switch to
> > > > > scsi-mq by default)
> > > > > has been in -next for quite a while.
> > > > > 
> > > > > Seems something new causes this issue.
> > > > 
> > > > 
> > > > Read my other email about how to find this.
> > > > 
> > > > https://marc.info/?l=linux-scsi&m=153316446223676
> > > > 
> > > > Now that we've confirmed the issue, Gunter, could you attempt to bisect
> > > > it as that email describes?
> > > > 
> > > 
> > > So, I am more and more baffled.
> > > 
> > > I ran another round of bisect, this time each test executing twice,
> > > once with "scsi_mod.use_blk_mq=1" and once with "scsi_mod.use_blk_mq=0",
> > > requiring both to pass. Bisect still points to the merge as culprit.
> > > 
> > > Ok, one step further: Actually _revert_ commit d5038a13eca72 before running
> > > each test, meaning the default is use_blk_mq=0. Still run both tests.
> > > Bisect _still_ points to the merge of scsi-next as culprit.
> > > 
> > > So, to me it looks like the problem is triggered by _something_ in
> > > scsi-next, combined with _something_ in -next prior to the merge,
> > > not specifically associated with use_blk_mq=[0|1] or d5038a13eca72,
> > > but to a combination of some patch in scsi-next and some other patch.
> > 
> > Today I am a bit busy, and not trace it much.
> > 
> > So far, I found the code hangs in scsi_test_unit_ready()
> > <-get_capabilities()<-sr_probe(), and scsi_queue_rq()/ata_scsi_queuecmd()
> > has queued the command successfully, but never completed.
> > 
> > Also tried to revert commits merged to ata tree on 30th, 31th,
> > but no difference.
> > 
> 
> Looking at my commit logs, the problem started to happen after various DMA
> changes were introduced. The boot tests fail on ppc (few), mips (all 32 bit,
> most 64 bit), i386 (all), x86_64 (most). All other platform pass, even with
> the same type of boot tests. Here is an example from alpha:
> 
> Building alpha:defconfig:initrd ... running .... passed
> Building alpha:defconfig:sata:rootfs ... running ..... passed
> Building alpha:defconfig:usb:rootfs ... running ..... passed
> Building alpha:defconfig:usb-uas:rootfs ... running ...... passed
> Building alpha:defconfig:scsi[AM53C974]:rootfs ... running ....... passed
> Building alpha:defconfig:scsi[DC395]:rootfs ... running ....... passed
> Building alpha:defconfig:scsi[MEGASAS]:rootfs ... running ...... passed
> Building alpha:defconfig:scsi[MEGASAS2]:rootfs ... running ...... passed
> Building alpha:defconfig:scsi[FUSION]:rootfs ... running ...... passed
> Building alpha:defconfig:nvme:rootfs ... running ..... passed
> 
> arm64:
> 
> Building arm64:virt:defconfig:smp:initrd ... running ..... passed
> Building arm64:virt:defconfig:smp:usb:rootfs ... running ..... passed
> Building arm64:virt:defconfig:smp:usb-uas:rootfs ... running ..... passed
> Building arm64:virt:defconfig:smp:virtio:rootfs ... running ..... passed
> Building arm64:virt:defconfig:smp:nvme:rootfs ... running ..... passed
> Building arm64:virt:defconfig:smp:mmc:rootfs ... running ..... passed
> Building arm64:virt:defconfig:smp:scsi[DC395]:rootfs ... running ..... passed
> Building arm64:virt:defconfig:smp:scsi[AM53C974]:rootfs ... running ..... passed
> Building arm64:virt:defconfig:smp:scsi[MEGASAS]:rootfs ... running ..... passed
> Building arm64:virt:defconfig:smp:scsi[MEGASAS2]:rootfs ... running ..... passed
> Building arm64:virt:defconfig:smp:scsi[53C810]:rootfs ... running ...... passed
> Building arm64:virt:defconfig:smp:scsi[53C895A]:rootfs ... running ...... passed
> Building arm64:virt:defconfig:smp:scsi[FUSION]:rootfs ... running ...... passed
> Skipping arm64:xlnx-zcu102:defconfig:smp:initrd:xilinx/zynqmp-ep108 ...
> Skipping arm64:xlnx-zcu102:defconfig:smp:sd:rootfs:xilinx/zynqmp-ep108 ...
> Skipping arm64:xlnx-zcu102:defconfig:smp:sata:rootfs:xilinx/zynqmp-ep108 ...
> Building arm64:xlnx-zcu102:defconfig:smp:initrd:xilinx/zynqmp-zcu102-rev1.0 ... running ....... passed
> Building arm64:xlnx-zcu102:defconfig:smp:sd1:rootfs:xilinx/zynqmp-zcu102-rev1.0 ... running ......... passed
> Building arm64:xlnx-zcu102:defconfig:smp:sata:rootfs:xilinx/zynqmp-zcu102-rev1.0 ... running ...... passed
> Building arm64:raspi3:defconfig:smp:initrd:broadcom/bcm2837-rpi-3-b ... running ..... passed
> Building arm64:raspi3:defconfig:smp:sd:rootfs:broadcom/bcm2837-rpi-3-b ... running ........ passed
> Building arm64:virt:defconfig:nosmp:initrd ... running ..... passed
> Skipping arm64:xlnx-zcu102:defconfig:nosmp:initrd:xilinx/zynqmp-ep108 ...
> Skipping arm64:xlnx-zcu102:defconfig:nosmp:sd:rootfs:xilinx/zynqmp-ep108 ...
> Building arm64:xlnx-zcu102:defconfig:nosmp:initrd:xilinx/zynqmp-zcu102-rev1.0 ... running ......... passed
> Building arm64:xlnx-zcu102:defconfig:nosmp:sd1:rootfs:xilinx/zynqmp-zcu102-rev1.0 ... running ......... passed
> 
> ppc:
> 
> Building powerpc:mac99:qemu_ppc_book3s_defconfig:nosmp:rootfs ... running ....... passed
> Building powerpc:g3beige:qemu_ppc_book3s_defconfig:nosmp:rootfs ... running ...... passed
> Building powerpc:mac99:qemu_ppc_book3s_defconfig:smp:rootfs ... running ....... passed
> Building powerpc:virtex-ml507:44x/virtex5_defconfig:devtmpfs:initrd ... running .... passed
> Building powerpc:mpc8544ds:mpc85xx_defconfig:initrd ... running .... passed
> Building powerpc:mpc8544ds:mpc85xx_defconfig:scsi:rootfs ... running ..... passed
> Building powerpc:mpc8544ds:mpc85xx_defconfig:sata:rootfs ... running .... passed
> Building powerpc:mpc8544ds:mpc85xx_smp_defconfig:initrd ... running .... passed
> Building powerpc:mpc8544ds:mpc85xx_smp_defconfig:scsi:rootfs ... running ..... passed
> Building powerpc:mpc8544ds:mpc85xx_smp_defconfig:sata:rootfs ... running .... passed
> Building powerpc:bamboo:44x/bamboo_defconfig:devtmpfs:initrd ... running .... passed
> Building powerpc:bamboo:44x/bamboo_defconfig:devtmpfs:scsi[AM53C974]:rootfs ... running ..... passed
> Building powerpc:bamboo:44x/bamboo_defconfig:devtmpfs:smp:initrd ... running .... passed
> Building powerpc:bamboo:44x/bamboo_defconfig:devtmpfs:smp:scsi[AM53C974]:rootfs ... running ..... passed
> Building powerpc:sam460ex:44x/canyonlands_defconfig:devtmpfs:initrd ... running ..... passed
> Building powerpc:sam460ex:44x/canyonlands_defconfig:devtmpfs:usbdisk:rootfs ... running ...... passed
> Building powerpc:mac99:pmac32_defconfig:devtmpfs:zilog:initrd ... running .................................. failed (timeout)
> Building powerpc:mac99:pmac32_defconfig:devtmpfs:zilog:rootfs ... running .................................. failed (timeout)
> 
> Maybe that is a coincidence, but it is at least suspicious.

This issue can be fixed by reverting d250bf4e776ff09d5 ("blk-mq: only iterate over
inflight requests in blk_mq_tagset_busy_iter").

This patch looks wrong, because 'blk_mq_rq_state(rq) == MQ_RQ_IN_FLIGHT'
isn't completely same with 'blk_mq_request_started(req)'.

Thanks,
Ming

  reply	other threads:[~2018-08-02 16:27 UTC|newest]

Thread overview: 40+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-08-01  7:58 linux-next: Tree for Aug 1 Stephen Rothwell
2018-08-01 22:48 ` Guenter Roeck
2018-08-01 22:52   ` James Bottomley
2018-08-01 23:00     ` James Bottomley
2018-08-02  0:05       ` Stephen Rothwell
2018-08-02  1:19         ` Guenter Roeck
2018-08-01 23:47     ` Guenter Roeck
2018-08-01 23:57       ` Ming Lei
2018-08-02  0:03         ` James Bottomley
2018-08-02  0:20           ` Guenter Roeck
2018-08-02  4:58           ` Guenter Roeck
2018-08-02  5:04             ` Bart Van Assche
2018-08-02 12:46               ` Guenter Roeck
2018-08-02 12:51                 ` Johannes Thumshirn
2018-08-02 13:00                   ` Guenter Roeck
2018-08-02 13:06                     ` Johannes Thumshirn
2018-08-02 11:35             ` Ming Lei
2018-08-02 13:05               ` Guenter Roeck
2018-08-02 16:27                 ` Ming Lei [this message]
2018-08-02 16:40                   ` Bart Van Assche
2018-08-02 16:50                     ` Ming Lei
2018-08-02 16:57                       ` Bart Van Assche
2018-08-02  0:12         ` Guenter Roeck
  -- strict thread matches above, loose matches on Subject: below --
2023-08-01  5:19 Stephen Rothwell
2019-08-01  6:14 Stephen Rothwell
2017-08-01  7:37 Stephen Rothwell
2017-08-01 12:42 ` Sergey Senozhatsky
2017-08-01 13:20   ` Arnd Bergmann
2017-08-01 13:28     ` Arnd Bergmann
2017-08-01 23:55       ` Sergey Senozhatsky
2017-08-02 11:13         ` Arnd Bergmann
2016-08-01  4:02 Stephen Rothwell
2014-08-01  9:14 Stephen Rothwell
2013-08-01  7:25 Stephen Rothwell
2013-08-01  7:50 ` Sedat Dilek
2011-08-01  4:47 Stephen Rothwell
2011-08-01  7:01 ` Sedat Dilek
2011-08-01 21:54   ` Randy Dunlap
2011-08-02  1:29     ` Stephen Rothwell
2011-08-02 10:38       ` Sedat Dilek

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180802162654.GA8928@ming.t460p \
    --to=ming.lei@redhat.com \
    --cc=James.Bottomley@hansenpartnership.com \
    --cc=axboe@kernel.dk \
    --cc=hch@lst.de \
    --cc=josef@toxicpanda.com \
    --cc=linux-ide@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-next@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=linux@roeck-us.net \
    --cc=sfr@canb.auug.org.au \
    --cc=tj@kernel.org \
    --cc=tom.leiming@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).