linux-scsi.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Douglas Gilbert <dgilbert@interlog.com>
To: Ming Lei <ming.lei@redhat.com>
Cc: Bart Van Assche <bvanassche@acm.org>,
	John Garry <john.garry@huawei.com>,
	Kashyap Desai <kashyap.desai@broadcom.com>,
	linux-block@vger.kernel.org, linux-scsi@vger.kernel.org,
	"Martin K. Petersen" <martin.petersen@oracle.com>,
	Jens Axboe <axboe@kernel.dk>
Subject: Re: [bug report] shared tags causes IO hang and performance drop
Date: Tue, 20 Apr 2021 16:22:53 -0400	[thread overview]
Message-ID: <acb407c9-c9ab-4783-e526-e5d34876e57b@interlog.com> (raw)
In-Reply-To: <CAFj5m9LPe=TdJgz2iJ7U6UT4=x-5aE=YbRgOQ80RHfpp62GQQQ@mail.gmail.com>

On 2021-04-20 2:52 a.m., Ming Lei wrote:
> On Tue, Apr 20, 2021 at 12:54 PM Douglas Gilbert <dgilbert@interlog.com> wrote:
>>
>> On 2021-04-19 11:22 p.m., Bart Van Assche wrote:
>>> On 4/19/21 8:06 PM, Douglas Gilbert wrote:
>>>> I have always suspected under extreme pressure the block layer (or scsi
>>>> mid-level) does strange things, like an IO hang, attempts to prove that
>>>> usually lead back to my own code :-). But I have one example recently
>>>> where upwards of 10 commands had been submitted (blk_execute_rq_nowait())
>>>> and the following one stalled (all on the same thread). Seconds later
>>>> those 10 commands reported DID_TIME_OUT, the stalled thread awoke, and
>>>> my dd variant went to its conclusion (reporting 10 errors). Following
>>>> copies showed no ill effects.
>>>>
>>>> My weapons of choice are sg_dd, actually sgh_dd and sg_mrq_dd. Those last
>>>> two monitor for stalls during the copy. Each submitted READ and WRITE
>>>> command gets its pack_id from an incrementing atomic and a management
>>>> thread in those copies checks every 300 milliseconds that that atomic
>>>> value is greater than the previous check. If not, dump the state of the
>>>> sg driver. The stalled request was in busy state with a timeout of 1
>>>> nanosecond which indicated that blk_execute_rq_nowait() had not been
>>>> called. So the chief suspect would be blk_get_request() followed by
>>>> the bio setup calls IMO.
>>>>
>>>> So it certainly looked like an IO hang, not a locking, resource nor
>>>> corruption issue IMO. That was with a branch off MKP's
>>>> 5.13/scsi-staging branch taken a few weeks back. So basically
>>>> lk 5.12.0-rc1 .
>>>
>>> Hi Doug,
>>>
>>> If it would be possible to develop a script that reproduces this hang and
>>> if that script can be shared I will help with root-causing and fixing this
>>> hang.
>>
>> Possible, but not very practical:
>>      1) apply supplied 83 patches to sg driver
>>      2) apply pending patch to scsi_debug driver
>>      3) find a stable kernel platform (perhaps not lk 5.12.0-rc1)
>>      4) run supplied scripts for three weeks
>>      5) dig through the output and maybe find one case (there were lots
>>         of EAGAINs from blk_get_request() but they are expected when
>>         thrashing the storage layers)
> 
> Or collecting the debugfs log after IO hang is triggered in your test:
> 
> (cd /sys/kernel/debug/block/$SDEV && find . -type f -exec grep -aH . {} \;)
> 
> $SDEV is the disk on which IO hang is observed.

Thanks. I'll try adding that to my IO hang trigger code.

My patches on the sg driver add debugfs support so these produce
the same output:
     cat /proc/scsi/sg/debug
     cat /sys/kernel/debug/scsi_generic/snapshot

There is also a /sys/kernel/debug/scsi_generic/snapped file whose
contents reflect the driver's state when ioctl(<sg_fd>, SG_DEBUG, &one)
was last called.

When I test, the root file system is usually on a NVMe SSD so the
state of all SCSI disks present should be dumped as they are part
of my test. Also I find the netconsole module extremely useful and
have an old laptop on my network running:
    socat udp-recv:6665 - > socat.txt

picking up the UDP packets from netconsole on port 6665. Not quite as
good as monitoring a hardware serial console, but less fiddly. And
most modern laptops don't have access to a serial console so
netconsole is the only option.

Another observation: upper level issues seem to impact the submission
side of request handling (e.g. the IO hang I was trying to describe)
while error injection I can do (e.g. using the scsi_debug driver)
impacts the completion side (obviously). Are there any tools to inject
errors into the block layer submission code?

Doug Gilbert



  reply	other threads:[~2021-04-20 20:22 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-14  7:50 [bug report] shared tags causes IO hang and performance drop Ming Lei
2021-04-14 10:10 ` John Garry
2021-04-14 10:38   ` Ming Lei
2021-04-14 10:42   ` Kashyap Desai
2021-04-14 11:12     ` Ming Lei
2021-04-14 12:06       ` John Garry
2021-04-15  3:46         ` Ming Lei
2021-04-15 10:41           ` John Garry
2021-04-15 12:18             ` Ming Lei
2021-04-15 15:41               ` John Garry
2021-04-16  0:46                 ` Ming Lei
2021-04-16  8:29                   ` John Garry
2021-04-16  8:39                     ` Ming Lei
2021-04-16 14:59                       ` John Garry
2021-04-20  3:06                         ` Douglas Gilbert
2021-04-20  3:22                           ` Bart Van Assche
2021-04-20  4:54                             ` Douglas Gilbert
2021-04-20  6:52                               ` Ming Lei
2021-04-20 20:22                                 ` Douglas Gilbert [this message]
2021-04-21  1:40                                   ` Ming Lei
2021-04-23  8:43           ` John Garry
2021-04-26 10:53             ` John Garry
2021-04-26 14:48               ` Ming Lei
2021-04-26 15:52                 ` John Garry
2021-04-26 16:03                   ` Ming Lei
2021-04-26 17:02                     ` John Garry
2021-04-26 23:59                       ` Ming Lei
2021-04-27  7:52                         ` John Garry
2021-04-27  9:11                           ` Ming Lei
2021-04-27  9:37                             ` John Garry
2021-04-27  9:52                               ` Ming Lei
2021-04-27 10:15                                 ` John Garry
2021-07-07 17:06                                 ` John Garry
2021-04-14 13:59       ` Kashyap Desai
2021-04-14 17:03         ` Douglas Gilbert
2021-04-14 18:19           ` John Garry
2021-04-14 19:39             ` Douglas Gilbert
2021-04-15  0:58         ` Ming Lei

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=acb407c9-c9ab-4783-e526-e5d34876e57b@interlog.com \
    --to=dgilbert@interlog.com \
    --cc=axboe@kernel.dk \
    --cc=bvanassche@acm.org \
    --cc=john.garry@huawei.com \
    --cc=kashyap.desai@broadcom.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=martin.petersen@oracle.com \
    --cc=ming.lei@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).