Re: [PATCH] blk-mq: fix corruption with direct issue

From: Ming Lei <ming.lei@redhat.com>
To: "Theodore Y. Ts'o" <tytso@mit.edu>
Cc: Jens Axboe <axboe@kernel.dk>,
	"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>
Subject: Re: [PATCH] blk-mq: fix corruption with direct issue
Date: Fri, 7 Dec 2018 17:30:17 +0800	[thread overview]
Message-ID: <20181207093016.GE29027@ming.t460p> (raw)
In-Reply-To: <20181207034437.GB22188@ming.t460p>

On Fri, Dec 07, 2018 at 11:44:39AM +0800, Ming Lei wrote:
> On Thu, Dec 06, 2018 at 09:46:42PM -0500, Theodore Y. Ts'o wrote:
> > On Wed, Dec 05, 2018 at 11:03:01AM +0800, Ming Lei wrote:
> > > 
> > > But at that time, there isn't io scheduler for MQ, so in theory the
> > > issue should be there since v4.11, especially 945ffb60c11d ("mq-deadline:
> > > add blk-mq adaptation of the deadline IO scheduler").
> > 
> > Hi Ming,
> > 
> > How were serious you about this issue being there (theoretically) an
> > issue since 4.11?  Can you talk about how it might get triggered, and
> > how we can test for it?  The reason why I ask is because we're trying
> > to track down a mysterious file system corruption problem on a 4.14.x
> > stable kernel.  The symptoms are *very* eerily similar to kernel
> > bugzilla #201685.
> 
> Hi Theodore,
> 
> It is just a theory analysis.
> 
> blk_mq_try_issue_directly() is called in two branches of blk_mq_make_request(),
> both are on real MQ disks.
> 
> IO merge can be done on none or real io schedulers, so in theory there might
> be the risk from v4.1, but IO merge on sw queue didn't work for a bit long,
> especially it was fixed by ab42f35d9cb5ac49b5a2.
> 
> As Jens mentioned in bugzilla, there are several conditions required
> for triggering the issue:
> 
> - MQ device
> 
> - queue busy can be triggered. It is hard to trigger in NVMe PCI,
>   but may be possible on NVMe FC. However, it can be quite easy to
>   trigger on SCSI devices. We know there are some MQ SCSI HBA,
>   qlogic FC, megaraid_sas.
> 
> - IO merge is enabled. 
> 
> I have setup scsi_debug in the following way:
> 
> modprobe scsi_debug dev_size_mb=4096 clustering=1 \
> 		max_luns=1 submit_queues=2 max_queue=2
> 
> - submit_queues=2 may set this disk as MQ
> - max_queue=4 may trigger the queue busy condition easily
> 
> and run some write IO on ext4 over the disk: fio, kernel building,... for
> some time, but still can't trigger the data corruption once.
> 
> I should have created more LUN, so that queue may be easier to become
> busy, will do that soon.

Actually I should have used SDEBUG_OPT_HOST_BUSY to simulate the queue busy.

Thanks,
Ming