From: Song Liu <song@kernel.org>
To: Jens Axboe <axboe@kernel.dk>
Cc: Thomas Deutschmann <whissi@whissi.de>,
Christoph Hellwig <hch@lst.de>,
Vishal Verma <vverma@digitalocean.com>,
Thorsten Leemhuis <regressions@leemhuis.info>,
"stable@vger.kernel.org" <stable@vger.kernel.org>,
"regressions@lists.linux.dev" <regressions@lists.linux.dev>
Subject: Re: [REGRESSION] v5.17-rc1+: FIFREEZE ioctl system call hangs
Date: Thu, 25 Aug 2022 15:24:20 -0700 [thread overview]
Message-ID: <CAPhsuW7AYMXrW4DNRTnGEvySSLHVz_8n0vJDHzZh90o5NR_5_Q@mail.gmail.com> (raw)
In-Reply-To: <3114ed77-3802-8ca4-52e4-0e4b917d88b2@kernel.dk>
On Thu, Aug 25, 2022 at 12:12 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 8/25/22 10:47 AM, Song Liu wrote:
> > On Tue, Aug 23, 2022 at 10:13 AM Song Liu <song@kernel.org> wrote:
> >>
> >> On Mon, Aug 22, 2022 at 8:15 PM Thomas Deutschmann <whissi@whissi.de> wrote:
> >>>
> >>> On 2022-08-23 03:37, Song Liu wrote:
> >>>> Thomas, have you tried to bisect with the fio repro?
> >>>
> >>> Yes, just finished:
> >>>
> >>>> d32d3d0b47f7e34560ae3c55ddfcf68694813501 is the first bad commit
> >>>> commit d32d3d0b47f7e34560ae3c55ddfcf68694813501
> >>>> Author: Christoph Hellwig
> >>>> Date: Mon Jun 14 13:17:34 2021 +0200
> >>>>
> >>>> nvme-multipath: set QUEUE_FLAG_NOWAIT
> >>>>
> >>>> The nvme multipathing code just dispatches bios to one of the blk-mq
> >>>> based paths and never blocks on its own, so set QUEUE_FLAG_NOWAIT
> >>>> to support REQ_NOWAIT bios.
> >>>
> >>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d32d3d0b47f7e34560ae3c55ddfcf68694813501
> >>>
> >>>
> >>> So another NOWAIT issue -- similar to the bad commit which is causing
> >>> the mdraid issue I already found
> >>> (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0f9650bd838efe5c52f7e5f40c3204ad59f1964d).
> >>>
> >>> Reverting the commit, i.e. deleting
> >>>
> >>> blk_queue_flag_set(QUEUE_FLAG_NOWAIT, head->disk->queue);
> >>>
> >>> fixes the problem for me. Well, sort of. Looks like this will disable
> >>> io_uring. fio reproducer fails with
> >>
> >> My system doesn't have multipath enabled. I guess bisect will point to something
> >> else here.
> >>
> >> I am afraid we won't get more information from bisect.
> >
> > OK, I am able to pinpoint the issue, and Jens found the proper fix for
> > it (see below,
> > also available in [1]). It survived 100 runs of the repro fio job.
> >
> > Thomas, please give it a try.
> >
> > Thanks,
> > Song
> >
> > diff --git c/fs/io_uring.c w/fs/io_uring.c
> > index 3f8a79a4affa..72a39f5ec5a5 100644
> > --- c/fs/io_uring.c
> > +++ w/fs/io_uring.c
> > @@ -4551,7 +4551,12 @@ static int io_write(struct io_kiocb *req,
> > unsigned int issue_flags)
> > copy_iov:
> > iov_iter_restore(&s->iter, &s->iter_state);
> > ret = io_setup_async_rw(req, iovec, s, false);
> > - return ret ?: -EAGAIN;
> > + if (!ret) {
> > + if (kiocb->ki_flags & IOCB_WRITE)
> > + kiocb_end_write(req);
> > + return -EAGAIN;
> > + }
> > + return 0;
>
> This should be 'return ret;' for that last line. I had to double check
> the ones I did, but they did get it right. But I did a double take when
> I saw this one :-)
Ah, right... "ret ?: -EAGAIN" is a lot of information..
Song
>
> It'll work fine for testing as we won't hit errors here unless we run
> out of memory, so...
>
> --
> Jens Axboe
next prev parent reply other threads:[~2022-08-25 22:24 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-03 14:35 [REGRESSION] v5.17-rc1+: FIFREEZE ioctl system call hangs Thomas Deutschmann
2022-08-11 12:34 ` Thomas Deutschmann
2022-08-15 10:58 ` Thorsten Leemhuis
2022-08-15 15:46 ` Vishal Verma
2022-08-17 6:19 ` Song Liu
2022-08-17 6:53 ` Thomas Deutschmann
2022-08-17 18:29 ` Thomas Deutschmann
2022-08-19 2:46 ` Thomas Deutschmann
2022-08-20 1:04 ` Song Liu
2022-08-22 15:29 ` Thomas Deutschmann
2022-08-22 16:30 ` Thomas Deutschmann
2022-08-22 21:52 ` Song Liu
2022-08-22 22:44 ` Thomas Deutschmann
2022-08-22 22:59 ` Song Liu
2022-08-23 1:37 ` Song Liu
2022-08-23 3:15 ` Thomas Deutschmann
2022-08-23 17:13 ` Song Liu
2022-08-25 16:47 ` Song Liu
2022-08-25 19:12 ` Jens Axboe
2022-08-25 22:24 ` Song Liu [this message]
2022-08-26 20:10 ` Thomas Deutschmann
2022-09-08 13:25 ` [REGRESSION] v5.17-rc1+: FIFREEZE ioctl system call hangs #forregzbot Thorsten Leemhuis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAPhsuW7AYMXrW4DNRTnGEvySSLHVz_8n0vJDHzZh90o5NR_5_Q@mail.gmail.com \
--to=song@kernel.org \
--cc=axboe@kernel.dk \
--cc=hch@lst.de \
--cc=regressions@leemhuis.info \
--cc=regressions@lists.linux.dev \
--cc=stable@vger.kernel.org \
--cc=vverma@digitalocean.com \
--cc=whissi@whissi.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).