From: Shakeel Butt <shakeelb@google.com>
To: Ming Lei <ming.lei@redhat.com>
Cc: "Theodore Y. Ts'o" <tytso@mit.edu>, Jens Axboe <axboe@kernel.dk>,
linux-ext4@vger.kernel.org,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"open list:BLOCK LAYER" <linux-block@vger.kernel.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Linux MM <linux-mm@kvack.org>, Roman Gushchin <guro@fb.com>,
Andrew Morton <akpm@linux-foundation.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Vlastimil Babka <vbabka@suse.cz>
Subject: Re: REGRESSION: 37f4a24c2469: blk-mq: centralise related handling into blk_mq_get_driver_tag
Date: Fri, 25 Sep 2020 09:47:43 -0700 [thread overview]
Message-ID: <CALvZod4uLs399uGh89RV=QsjOXu0-iFLW8ifgWjC9N6OpRqu-w@mail.gmail.com> (raw)
In-Reply-To: <CALvZod6QUvmD_3xf0FM4qt=9YT=nwuMV3ZqFayND+dUp9oQUsQ@mail.gmail.com>
On Fri, Sep 25, 2020 at 9:32 AM Shakeel Butt <shakeelb@google.com> wrote:
>
> On Fri, Sep 25, 2020 at 9:19 AM Ming Lei <ming.lei@redhat.com> wrote:
> >
> > On Fri, Sep 25, 2020 at 03:31:45PM +0800, Ming Lei wrote:
> > > On Thu, Sep 24, 2020 at 09:13:11PM -0400, Theodore Y. Ts'o wrote:
> > > > On Thu, Sep 24, 2020 at 10:33:45AM -0400, Theodore Y. Ts'o wrote:
> > > > > HOWEVER, thanks to a hint from a colleague at $WORK, and realizing
> > > > > that one of the stack traces had virtio balloon in the trace, I
> > > > > realized that when I switched the GCE VM type from e1-standard-2 to
> > > > > n1-standard-2 (where e1 VM's are cheaper because they use
> > > > > virtio-balloon to better manage host OS memory utilization), problem
> > > > > has become, much, *much* rarer (and possibly has gone away, although
> > > > > I'm going to want to run a lot more tests before I say that
> > > > > conclusively) on my test setup. At the very least, using an n1 VM
> > > > > (which doesn't have virtio-balloon enabled in the hypervisor) is
> > > > > enough to unblock ext4 development.
> > > >
> > > > .... and I spoke too soon. A number of runs using -rc6 are now
> > > > failing even with the n1-standard-2 VM, so virtio-ballon may not be an
> > > > indicator.
> > > >
> > > > This is why debugging this is frustrating; it is very much a heisenbug
> > > > --- although 5.8 seems to work completely reliably, as does commits
> > > > before 37f4a24c2469. Anything after that point will show random
> > > > failures. :-(
> > >
> > > It does not make sense to mention 37f4a24c2469, which is reverted in
> > > 4e2f62e566b5. Later the patch in 37f4a24c2469 is fixed and re-commited
> > > as 568f27006577.
> > >
> > > However, I can _not_ reproduce the issue by running the same test on
> > > kernel built from 568f27006577 directly.
> > >
> > > Also you have confirmed that the issue can't be fixed after reverting
> > > 568f27006577 against v5.9-rc4.
> > >
> > > Looks the real issue(slab list corruption) should be introduced between
> > > 568f27006577 and v5.9-rc4.
> >
> > git bisect shows the first bad commit:
> >
> > [10befea91b61c4e2c2d1df06a2e978d182fcf792] mm: memcg/slab: use a single set of
> > kmem_caches for all allocations
> >
> > And I have double checked that the above commit is really the first bad
> > commit for the list corruption issue of 'list_del corruption, ffffe1c241b00408->next
> > is LIST_POISON1 (dead000000000100)', see the detailed stack trace and
> > kernel oops log in the following link:
> >
> > https://lore.kernel.org/lkml/20200916202026.GC38283@mit.edu/
>
> The failure signature is similar to
> https://lore.kernel.org/lkml/20200901075321.GL4299@shao2-debian/
>
> >
> > And the kernel config is the one(without KASAN) used by Theodore in GCE VM, see
> > the following link:
> >
> > https://lore.kernel.org/lkml/20200917143012.GF38283@mit.edu/
> >
> > The reproducer is xfstests generic/038. In my setting, test device is virtio-scsi, and
> > scratch device is virtio-blk.
Is it possible to check SLUB as well to confirm that the issue is only
happening on SLAB?
next prev parent reply other threads:[~2020-09-25 16:48 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-03 23:02 [PATCH] ext4: flag as supporting buffered async reads Jens Axboe
2020-08-11 14:31 ` Jens Axboe
2020-08-18 18:11 ` Theodore Y. Ts'o
2020-08-18 18:12 ` Jens Axboe
2020-08-21 21:26 ` Jens Axboe
2020-08-22 14:33 ` Theodore Y. Ts'o
2020-08-22 15:48 ` Jens Axboe
2020-08-24 10:56 ` Jens Axboe
2020-08-25 14:18 ` Jens Axboe
2020-08-27 1:54 ` Jens Axboe
2020-09-04 0:10 ` Jens Axboe
2020-09-04 3:55 ` Theodore Y. Ts'o
2020-09-04 14:51 ` Jens Axboe
2020-09-04 15:25 ` Darrick J. Wong
2020-09-15 4:45 ` REGRESSION: 37f4a24c2469: blk-mq: centralise related handling into blk_mq_get_driver_tag Theodore Y. Ts'o
2020-09-15 7:33 ` Ming Lei
2020-09-15 22:45 ` Theodore Y. Ts'o
2020-09-15 23:09 ` Ming Lei
2020-09-16 20:20 ` Theodore Y. Ts'o
2020-09-17 2:20 ` Ming Lei
2020-09-17 14:30 ` Theodore Y. Ts'o
2020-09-17 23:08 ` Ming Lei
2020-09-24 0:59 ` Ming Lei
2020-09-24 14:33 ` Theodore Y. Ts'o
2020-09-25 1:13 ` Theodore Y. Ts'o
2020-09-25 7:31 ` Ming Lei
2020-09-25 16:19 ` Ming Lei
2020-09-25 16:32 ` Shakeel Butt
2020-09-25 16:47 ` Shakeel Butt [this message]
2020-09-25 17:22 ` Roman Gushchin
2020-09-25 17:17 ` Linus Torvalds
2020-09-25 17:22 ` Shakeel Butt
2020-09-25 17:35 ` Shakeel Butt
2020-09-25 17:47 ` Roman Gushchin
2020-09-25 17:58 ` Shakeel Butt
2020-09-25 19:19 ` Shakeel Butt
2020-09-25 20:56 ` Roman Gushchin
2020-09-25 21:18 ` Shakeel Butt
2020-09-27 17:38 ` Theodore Y. Ts'o
2020-09-26 1:43 ` Ming Lei
2020-09-26 6:42 ` Roman Gushchin
2020-09-25 1:14 ` Ming Lei
2020-09-25 2:34 ` Ming Lei
2020-10-02 20:08 ` [PATCH] ext4: flag as supporting buffered async reads Theodore Y. Ts'o
2020-10-02 20:10 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CALvZod4uLs399uGh89RV=QsjOXu0-iFLW8ifgWjC9N6OpRqu-w@mail.gmail.com' \
--to=shakeelb@google.com \
--cc=akpm@linux-foundation.org \
--cc=axboe@kernel.dk \
--cc=guro@fb.com \
--cc=hannes@cmpxchg.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ming.lei@redhat.com \
--cc=torvalds@linux-foundation.org \
--cc=tytso@mit.edu \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).