archive mirror
 help / color / mirror / Atom feed
From: "Theodore Y. Ts'o" <>
To: Ming Lei <>
Cc: Jens Axboe <>,,
	"" <>,,
	Linus Torvalds <>
Subject: Re: REGRESSION: 37f4a24c2469: blk-mq: centralise related handling into blk_mq_get_driver_tag
Date: Tue, 15 Sep 2020 18:45:41 -0400	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <20200915073303.GA754106@T590>

On Tue, Sep 15, 2020 at 03:33:03PM +0800, Ming Lei wrote:
> Hi Theodore,
> On Tue, Sep 15, 2020 at 12:45:19AM -0400, Theodore Y. Ts'o wrote:
> > On Thu, Sep 03, 2020 at 11:55:28PM -0400, Theodore Y. Ts'o wrote:
> > > Worse, right now, -rc1 and -rc2 is causing random crashes in my
> > > gce-xfstests framework.  Sometimes it happens before we've run even a
> > > single xfstests; sometimes it happens after we have successfully
> > > completed all of the tests, and we're doing a shutdown of the VM under
> > > test.  Other times it happens in the middle of a test run.  Given that
> > > I'm seeing this at -rc1, which is before my late ext4 pull request to
> > > Linus, it's probably not an ext4 related bug.  But it also means that
> > > I'm partially blind in terms of my kernel testing at the moment.  So I
> > > can't even tell Linus that I've run lots of tests and I'm 100%
> > > confident your one-line change is 100% safe.
> > 
> > I was finally able to bisect it down to the commit:
> > 
> > 37f4a24c2469: blk-mq: centralise related handling into blk_mq_get_driver_tag
> 37f4a24c2469 has been reverted in:
> 	4e2f62e566b5 Revert "blk-mq: put driver tag when this request is completed"
> And later the patch is committed as the following after being fixed:
> 	568f27006577 blk-mq: centralise related handling into blk_mq_get_driver_tag
> So can you reproduce the issue by running kernel of commit 568f27006577?

Yes.  And things work fine if I try 4e2f62e566b5.

> If yes, can the issue be fixed by reverting 568f27006577?

The problem is it's a bit tricky to revert 568f27006577, since there
is a merge conflict in blk_kick_flush().  I attempted to do the bisect
manually here, but it's clearly not right since the kernel is not
booting after the revert:


Can you send me a patch which correctly reverts 568f27006577?  I can
try it against -rc1 .. -rc4, whichever is most convenient.

> Can you share the exact mount command line for setup the environment?
> and the exact xfstest item?

It's a variety of mount command lines, since I'm using gce-xfstests[1][2]
using a variety of file system scenarios --- but the basic one, which
is ext4 using the default 4k block size is failing (they all are failing).


It's also not one consistent xfstests which is failing, but it does
tend to be tests which are loading up the storage stack with a lot of
small random read/writes, especially involving metadata blocks/writes.
(For example, tests which run fsstress.)

Since this reliably triggers for me, and other people running
kvm-xfstests or are running xfstests on their own test environments
aren't seeing it, I'm assuming it must be some kind of interesting
interaction between virtio-scsi, perhaps with how Google Persistent
Disk is behaving (maybe timing related?  who knows?).  Darrick Wong
did say he saw something like it once using Oracle's Cloud
infrastructure, but as far as I know it hasn't reproduced since.  On
Google Compute Engine VM's, it reproduces *extremely* reliably.

I expect that if you were to set up gce-xfstests, get a free GCE
account with the initial $300 free credits, you could run
"gce-xfstests -c ext4/4k -g auto" and it would reproduce within an
hour or so.  (So under a dollar's worth of VM credits, so long as you
notice that it's hung and shut down the VM after gathering debugging

The instructions are at [2], and the image xfstests-202008311554 in
the xfstests-cloud project is a public copy of the VM test appliance I
was using.

% gcloud compute images describe --project xfstests-cloud xfstests-202008311554
archiveSizeBytes: '1720022528'
creationTimestamp: '2020-09-15T15:09:30.544-07:00'
description: Linux Kernel File System Test Appliance
diskSizeGb: '10'
family: xfstests
id: '1558420969906537845'
kind: compute#image
labelFingerprint: V-2Qgcxt2uw=
  blktests: g8a75bed
  e2fsprogs: v1_45_6
  fio: fio-3_22
  fsverity: v1_2
  ima-evm-utils: v1_3_1
  nvme-cli: v1_12
  quota: g13bb8c2
  util-linux: v2_36
  xfsprogs: v5_8_0-rc1
  xfstests: linux-v3_8-2838-geb439bf2
  xfstests-bld: gb5085ab
- '5543610867827062957'
name: xfstests-202008311554
sourceDiskId: '5824762850044577124'
sourceType: RAW
status: READY
- us


					- Ted

P.S.  As you can see, I can also easily run blktests using this VM
Test Appliance.  If you think it would be helpful for me to try
running one or more of the blktests test groups, I can do that.

I've been focusing on using xfstests mainly because this is my primary
way of running regression tests, and I'm blocked on ext4 testing until
I can figure out this particular problem.  Which is why a proper
revert of 568f27006577 would be extremely helpful, if only so I can
apply that on top of the ext4 git tree temporarily so I can run my
file system regression tests.

  reply	other threads:[~2020-09-15 22:46 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-08-03 23:02 [PATCH] ext4: flag as supporting buffered async reads Jens Axboe
2020-08-11 14:31 ` Jens Axboe
2020-08-18 18:11 ` Theodore Y. Ts'o
2020-08-18 18:12   ` Jens Axboe
2020-08-21 21:26     ` Jens Axboe
2020-08-22 14:33       ` Theodore Y. Ts'o
2020-08-22 15:48         ` Jens Axboe
2020-08-24 10:56           ` Jens Axboe
2020-08-25 14:18             ` Jens Axboe
2020-08-27  1:54               ` Jens Axboe
2020-09-04  0:10                 ` Jens Axboe
2020-09-04  3:55                   ` Theodore Y. Ts'o
2020-09-04 14:51                     ` Jens Axboe
2020-09-04 15:25                       ` Darrick J. Wong
2020-09-15  4:45                     ` REGRESSION: 37f4a24c2469: blk-mq: centralise related handling into blk_mq_get_driver_tag Theodore Y. Ts'o
2020-09-15  7:33                       ` Ming Lei
2020-09-15 22:45                         ` Theodore Y. Ts'o [this message]
2020-09-15 23:09                           ` Ming Lei
2020-09-16 20:20                             ` Theodore Y. Ts'o
2020-09-17  2:20                               ` Ming Lei
2020-09-17 14:30                                 ` Theodore Y. Ts'o
2020-09-17 23:08                                   ` Ming Lei
2020-09-24  0:59                                   ` Ming Lei
2020-09-24 14:33                                     ` Theodore Y. Ts'o
2020-09-25  1:13                                       ` Theodore Y. Ts'o
2020-09-25  7:31                                         ` Ming Lei
2020-09-25 16:19                                           ` Ming Lei
2020-09-25 16:32                                             ` Shakeel Butt
2020-09-25 16:47                                               ` Shakeel Butt
2020-09-25 17:22                                                 ` Roman Gushchin
2020-09-25 17:17                                             ` Linus Torvalds
2020-09-25 17:22                                               ` Shakeel Butt
2020-09-25 17:35                                                 ` Shakeel Butt
2020-09-25 17:47                                                   ` Roman Gushchin
2020-09-25 17:58                                                     ` Shakeel Butt
2020-09-25 19:19                                                       ` Shakeel Butt
2020-09-25 20:56                                                         ` Roman Gushchin
2020-09-25 21:18                                                           ` Shakeel Butt
2020-09-27 17:38                                                             ` Theodore Y. Ts'o
2020-09-26  1:43                                                         ` Ming Lei
2020-09-26  6:42                                                           ` Roman Gushchin
2020-09-25  1:14                                       ` Ming Lei
2020-09-25  2:34                                         ` Ming Lei
2020-10-02 20:08 ` [PATCH] ext4: flag as supporting buffered async reads Theodore Y. Ts'o
2020-10-02 20:10   ` Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).