All of lore.kernel.org
 help / color / mirror / Atom feed
From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "dvyukov@google.com" <dvyukov@google.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
	"jthumshirn@suse.de" <jthumshirn@suse.de>,
	"alan.christopher.jenkins@gmail.com"
	<alan.christopher.jenkins@gmail.com>,
	"syzbot+c4f9cebf9d651f6e54de@syzkaller.appspotmail.com"
	<syzbot+c4f9cebf9d651f6e54de@syzkaller.appspotmail.com>,
	"penguin-kernel@i-love.sakura.ne.jp"
	<penguin-kernel@i-love.sakura.ne.jp>,
	"martin.petersen@oracle.com" <martin.petersen@oracle.com>,
	"axboe@kernel.dk" <axboe@kernel.dk>,
	"dan.j.williams@intel.com" <dan.j.williams@intel.com>,
	"hch@lst.de" <hch@lst.de>,
	"oleksandr@natalenko.name" <oleksandr@natalenko.name>,
	"ming.lei@redhat.com" <ming.lei@redhat.com>,
	"martin@lichtvoll.de" <martin@lichtvoll.de>,
	"hare@suse.com" <hare@suse.com>,
	"syzkaller-bugs@googlegroups.com"
	<syzkaller-bugs@googlegroups.com>,
	"ross.zwisler@linux.intel.com" <ross.zwisler@linux.intel.com>,
	"keith.busch@intel.com" <keith.busch@intel.com>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>
Subject: Re: INFO: task hung in blk_queue_enter
Date: Wed, 16 May 2018 15:37:24 +0000	[thread overview]
Message-ID: <6db16aa3a7c56b6dcca2d10b4e100a780c740081.camel@wdc.com> (raw)
In-Reply-To: <CACT4Y+ZJwvROpBmKWF08fTd2AH=dO-a5+dzCzZua_vVFjiCRBw@mail.gmail.com>

T24gV2VkLCAyMDE4LTA1LTE2IGF0IDE3OjE2ICswMjAwLCBEbWl0cnkgVnl1a292IHdyb3RlOg0K
PiBPbiBXZWQsIE1heSAxNiwgMjAxOCBhdCA0OjU2IFBNLCBCYXJ0IFZhbiBBc3NjaGUgPEJhcnQu
VmFuQXNzY2hlQHdkYy5jb20+IHdyb3RlOg0KPiA+IE9uIFdlZCwgMjAxOC0wNS0xNiBhdCAyMjow
NSArMDkwMCwgVGV0c3VvIEhhbmRhIHdyb3RlOg0KPiA+ID4gZGlmZiAtLWdpdCBhL2Jsb2NrL2Js
ay1jb3JlLmMgYi9ibG9jay9ibGstY29yZS5jDQo+ID4gPiBpbmRleCA4NTkwOWI0Li41OWUyNDk2
IDEwMDY0NA0KPiA+ID4gLS0tIGEvYmxvY2svYmxrLWNvcmUuYw0KPiA+ID4gKysrIGIvYmxvY2sv
YmxrLWNvcmUuYw0KPiA+ID4gQEAgLTk1MSwxMCArOTUxLDEwIEBAIGludCBibGtfcXVldWVfZW50
ZXIoc3RydWN0IHJlcXVlc3RfcXVldWUgKnEsIGJsa19tcV9yZXFfZmxhZ3NfdCBmbGFncykNCj4g
PiA+ICAgICAgICAgICAgICAgc21wX3JtYigpOw0KPiA+ID4gDQo+ID4gPiAgICAgICAgICAgICAg
IHdhaXRfZXZlbnQocS0+bXFfZnJlZXplX3dxLA0KPiA+ID4gLSAgICAgICAgICAgICAgICAgICAg
ICAgIChhdG9taWNfcmVhZCgmcS0+bXFfZnJlZXplX2RlcHRoKSA9PSAwICYmDQo+ID4gPiAtICAg
ICAgICAgICAgICAgICAgICAgICAgIChwcmVlbXB0IHx8ICFibGtfcXVldWVfcHJlZW1wdF9vbmx5
KHEpKSkgfHwNCj4gPiA+ICsgICAgICAgICAgICAgICAgICAgICAgICBhdG9taWNfcmVhZCgmcS0+
bXFfZnJlZXplX2RlcHRoKSB8fA0KPiA+ID4gKyAgICAgICAgICAgICAgICAgICAgICAgIChwcmVl
bXB0IHx8ICFibGtfcXVldWVfcHJlZW1wdF9vbmx5KHEpKSB8fA0KPiA+ID4gICAgICAgICAgICAg
ICAgICAgICAgICAgIGJsa19xdWV1ZV9keWluZyhxKSk7DQo+ID4gPiAtICAgICAgICAgICAgIGlm
IChibGtfcXVldWVfZHlpbmcocSkpDQo+ID4gPiArICAgICAgICAgICAgIGlmIChhdG9taWNfcmVh
ZCgmcS0+bXFfZnJlZXplX2RlcHRoKSB8fCBibGtfcXVldWVfZHlpbmcocSkpDQo+ID4gPiAgICAg
ICAgICAgICAgICAgICAgICAgcmV0dXJuIC1FTk9ERVY7DQo+ID4gPiAgICAgICB9DQo+ID4gPiAg
fQ0KPiA+IA0KPiA+IFRoYXQgY2hhbmdlIGxvb2tzIHdyb25nIHRvIG1lLg0KPiANCj4gSGkgQmFy
dCwNCj4gDQo+IFdoeSBkb2VzIGl0IGxvb2sgd3JvbmcgdG8geW91Pw0KDQpCZWNhdXNlIHRoYXQg
Y2hhbmdlIGNvbmZsaWN0cyB3aXRoIHRoZSBwdXJwb3NlIG9mIHF1ZXVlIGZyZWV6aW5nIGFuZCBh
bHNvIGJlY2F1c2UNCnRoYXQgY2hhbmdlIHdvdWxkIGluamVjdCBJL08gZXJyb3JzIGluIGNvZGUg
cGF0aHMgdGhhdCBzaG91bGRuJ3QgaW5qZWN0IEkvTyBlcnJvcnMuDQpQbGVhc2UgaGF2ZSBhIGxv
b2sgYXQgZS5nLiBnZW5lcmljX21ha2VfcmVxdWVzdCgpLiBGcm9tIHRoZSBzdGFydCBvZiB0aGF0
IGZ1bmN0aW9uOg0KDQoJaWYgKGJsa19xdWV1ZV9lbnRlcihxLCBmbGFncykgPCAwKSB7DQoJCWlm
ICghYmxrX3F1ZXVlX2R5aW5nKHEpICYmIChiaW8tPmJpX29wZiAmIFJFUV9OT1dBSVQpKQ0KCQkJ
YmlvX3dvdWxkYmxvY2tfZXJyb3IoYmlvKTsNCgkJZWxzZQ0KCQkJYmlvX2lvX2Vycm9yKGJpbyk7
DQoJCXJldHVybiByZXQ7DQoJfQ0KDQpUaGUgYWJvdmUgcGF0Y2ggY2hhbmdlcyB0aGUgYmVoYXZp
b3Igb2YgYmxrX3F1ZXVlX2VudGVyKCkgY29kZSBmcm9tIHdhaXRpbmcgd2hpbGUNCnEtPm1xX2Zy
ZWV6ZV9kZXB0aCAhPSAwIGludG8gcmV0dXJuaW5nIC1FTk9ERVYgd2hpbGUgdGhlIHJlcXVlc3Qg
cXVldWUgaXMgZnJvemVuLg0KVGhhdCB3aWxsIGNhdXNlIGdlbmVyaWNfbWFrZV9yZXF1ZXN0KCkg
dG8gY2FsbCBiaW9faW9fZXJyb3IoYmlvKSB3aGlsZSBhIHJlcXVlc3QNCnF1ZXVlIGlzIGZyb3pl
biBpZiBSRVFfTk9XQUlUIGhhcyBub3QgYmVlbiBzZXQsIHdoaWNoIGlzIHRoZSBkZWZhdWx0IGJl
aGF2aW9yLiBTbw0KYW55IG9wZXJhdGlvbiB0aGF0IGZyZWV6ZXMgdGhlIHF1ZXVlIHRlbXBvcmFy
aWx5LCBlLmcuIGNoYW5naW5nIHRoZSBxdWV1ZSBkZXB0aCwNCmNvbmN1cnJlbnRseSB3aXRoIEkv
TyBwcm9jZXNzaW5nIGNhbiBjYXVzZSBJL08gdG8gZmFpbCB3aXRoIC1FTk9ERVYuIEFzIHlvdQ0K
cHJvYmFibHkga25vdyBmYWlsdXJlIG9mIHdyaXRlIHJlcXVlc3RzIGhhcyB2ZXJ5IGFubm95aW5n
IGNvbnNlcXVlbmNlcy4gSXQgZS5nLg0KY2F1c2VzIGZpbGVzeXN0ZW1zIHRvIGdvIGludG8gcmVh
ZC1vbmx5IG1vZGUuIFRoYXQncyB3aHkgSSB0aGluayB0aGF0IHRoZSBhYm92ZQ0KY2hhbmdlIGlz
IGNvbXBsZXRlbHkgd3JvbmcuDQoNCkJhcnQuDQoNCg0K

WARNING: multiple messages have this Message-ID (diff)
From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "dvyukov@google.com" <dvyukov@google.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
	"jthumshirn@suse.de" <jthumshirn@suse.de>,
	"alan.christopher.jenkins@gmail.com"
	<alan.christopher.jenkins@gmail.com>,
	"syzbot+c4f9cebf9d651f6e54de@syzkaller.appspotmail.com"
	<syzbot+c4f9cebf9d651f6e54de@syzkaller.appspotmail.com>,
	"penguin-kernel@i-love.sakura.ne.jp"
	<penguin-kernel@i-love.sakura.ne.jp>,
	"martin.petersen@oracle.com" <martin.petersen@oracle.com>,
	"axboe@kernel.dk" <axboe@kernel.dk>,
	"dan.j.williams@intel.com" <dan.j.williams@intel.com>,
	"hch@lst.de" <hch@lst.de>,
	"oleksandr@natalenko.name" <oleksandr@natalenko.name>,
	"ming.lei@redhat.com" <ming.lei@redhat.com>,
	"martin@lichtvoll.de" <martin@lichtvoll.de>,
	"hare@suse.com" <hare@suse.com>,
	"syzkaller-bugs@googlegroups.com"
	<syzkaller-bugs@googlegroups.com>,
	"ross.zwisler@linux.intel.com" <ross.zwisler@linux.intel.com>,
	"keith.busch@intel.com" <keith.busch@intel.com>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>
Subject: Re: INFO: task hung in blk_queue_enter
Date: Wed, 16 May 2018 15:37:24 +0000	[thread overview]
Message-ID: <6db16aa3a7c56b6dcca2d10b4e100a780c740081.camel@wdc.com> (raw)
In-Reply-To: <CACT4Y+ZJwvROpBmKWF08fTd2AH=dO-a5+dzCzZua_vVFjiCRBw@mail.gmail.com>

On Wed, 2018-05-16 at 17:16 +0200, Dmitry Vyukov wrote:
> On Wed, May 16, 2018 at 4:56 PM, Bart Van Assche <Bart.VanAssche@wdc.com> wrote:
> > On Wed, 2018-05-16 at 22:05 +0900, Tetsuo Handa wrote:
> > > diff --git a/block/blk-core.c b/block/blk-core.c
> > > index 85909b4..59e2496 100644
> > > --- a/block/blk-core.c
> > > +++ b/block/blk-core.c
> > > @@ -951,10 +951,10 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)
> > >               smp_rmb();
> > > 
> > >               wait_event(q->mq_freeze_wq,
> > > -                        (atomic_read(&q->mq_freeze_depth) == 0 &&
> > > -                         (preempt || !blk_queue_preempt_only(q))) ||
> > > +                        atomic_read(&q->mq_freeze_depth) ||
> > > +                        (preempt || !blk_queue_preempt_only(q)) ||
> > >                          blk_queue_dying(q));
> > > -             if (blk_queue_dying(q))
> > > +             if (atomic_read(&q->mq_freeze_depth) || blk_queue_dying(q))
> > >                       return -ENODEV;
> > >       }
> > >  }
> > 
> > That change looks wrong to me.
> 
> Hi Bart,
> 
> Why does it look wrong to you?

Because that change conflicts with the purpose of queue freezing and also because
that change would inject I/O errors in code paths that shouldn't inject I/O errors.
Please have a look at e.g. generic_make_request(). From the start of that function:

	if (blk_queue_enter(q, flags) < 0) {
		if (!blk_queue_dying(q) && (bio->bi_opf & REQ_NOWAIT))
			bio_wouldblock_error(bio);
		else
			bio_io_error(bio);
		return ret;
	}

The above patch changes the behavior of blk_queue_enter() code from waiting while
q->mq_freeze_depth != 0 into returning -ENODEV while the request queue is frozen.
That will cause generic_make_request() to call bio_io_error(bio) while a request
queue is frozen if REQ_NOWAIT has not been set, which is the default behavior. So
any operation that freezes the queue temporarily, e.g. changing the queue depth,
concurrently with I/O processing can cause I/O to fail with -ENODEV. As you
probably know failure of write requests has very annoying consequences. It e.g.
causes filesystems to go into read-only mode. That's why I think that the above
change is completely wrong.

Bart.

WARNING: multiple messages have this Message-ID (diff)
From: Bart Van Assche <Bart.VanAssche@wdc.com>
To: "dvyukov@google.com" <dvyukov@google.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
	"jthumshirn@suse.de" <jthumshirn@suse.de>,
	"alan.christopher.jenkins@gmail.com"
	<alan.christopher.jenkins@gmail.com>,
	"syzbot+c4f9cebf9d651f6e54de@syzkaller.appspotmail.com"
	<syzbot+c4f9cebf9d651f6e54de@syzkaller.appspotmail.com>,
	"penguin-kernel@i-love.sakura.ne.jp"
	<penguin-kernel@i-love.sakura.ne.jp>,
	"martin.petersen@oracle.com" <martin.petersen@oracle.com>,
	"axboe@kernel.dk" <axboe@kernel.dk>,
	"dan.j.williams@intel.com" <dan.j.williams@intel.com>,
	"hch@lst.de" <hch@lst.de>,
	"oleksandr@natalenko.name" <oleksandr@natalenko.name>,
	"ming.lei@redhat.com" <ming.lei@redhat.com>,
	"martin@lichtvoll.de" <martin@lichtvoll.de>,
	"hare@suse.c
Subject: Re: INFO: task hung in blk_queue_enter
Date: Wed, 16 May 2018 15:37:24 +0000	[thread overview]
Message-ID: <6db16aa3a7c56b6dcca2d10b4e100a780c740081.camel@wdc.com> (raw)
In-Reply-To: <CACT4Y+ZJwvROpBmKWF08fTd2AH=dO-a5+dzCzZua_vVFjiCRBw@mail.gmail.com>

On Wed, 2018-05-16 at 17:16 +0200, Dmitry Vyukov wrote:
> On Wed, May 16, 2018 at 4:56 PM, Bart Van Assche <Bart.VanAssche@wdc.com> wrote:
> > On Wed, 2018-05-16 at 22:05 +0900, Tetsuo Handa wrote:
> > > diff --git a/block/blk-core.c b/block/blk-core.c
> > > index 85909b4..59e2496 100644
> > > --- a/block/blk-core.c
> > > +++ b/block/blk-core.c
> > > @@ -951,10 +951,10 @@ int blk_queue_enter(struct request_queue *q, blk_mq_req_flags_t flags)
> > >               smp_rmb();
> > > 
> > >               wait_event(q->mq_freeze_wq,
> > > -                        (atomic_read(&q->mq_freeze_depth) == 0 &&
> > > -                         (preempt || !blk_queue_preempt_only(q))) ||
> > > +                        atomic_read(&q->mq_freeze_depth) ||
> > > +                        (preempt || !blk_queue_preempt_only(q)) ||
> > >                          blk_queue_dying(q));
> > > -             if (blk_queue_dying(q))
> > > +             if (atomic_read(&q->mq_freeze_depth) || blk_queue_dying(q))
> > >                       return -ENODEV;
> > >       }
> > >  }
> > 
> > That change looks wrong to me.
> 
> Hi Bart,
> 
> Why does it look wrong to you?

Because that change conflicts with the purpose of queue freezing and also because
that change would inject I/O errors in code paths that shouldn't inject I/O errors.
Please have a look at e.g. generic_make_request(). From the start of that function:

	if (blk_queue_enter(q, flags) < 0) {
		if (!blk_queue_dying(q) && (bio->bi_opf & REQ_NOWAIT))
			bio_wouldblock_error(bio);
		else
			bio_io_error(bio);
		return ret;
	}

The above patch changes the behavior of blk_queue_enter() code from waiting while
q->mq_freeze_depth != 0 into returning -ENODEV while the request queue is frozen.
That will cause generic_make_request() to call bio_io_error(bio) while a request
queue is frozen if REQ_NOWAIT has not been set, which is the default behavior. So
any operation that freezes the queue temporarily, e.g. changing the queue depth,
concurrently with I/O processing can cause I/O to fail with -ENODEV. As you
probably know failure of write requests has very annoying consequences. It e.g.
causes filesystems to go into read-only mode. That's why I think that the above
change is completely wrong.

Bart.



  reply	other threads:[~2018-05-16 15:37 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-04-28 11:24 INFO: task hung in blk_queue_enter syzbot
2018-05-15 11:45 ` Tetsuo Handa
2018-05-16 13:05   ` Tetsuo Handa
2018-05-16 14:56     ` Bart Van Assche
2018-05-16 14:56       ` Bart Van Assche
2018-05-16 15:16       ` Dmitry Vyukov
2018-05-16 15:16         ` Dmitry Vyukov
2018-05-16 15:37         ` Bart Van Assche [this message]
2018-05-16 15:37           ` Bart Van Assche
2018-05-16 15:37           ` Bart Van Assche
2018-05-21 21:52           ` Tetsuo Handa
2018-05-22 11:20             ` Tetsuo Handa
2018-06-01 10:10               ` Tetsuo Handa
2018-06-01 17:52                 ` Jens Axboe
2018-06-01 23:49                   ` Ming Lei
2018-06-02  0:49                     ` Jens Axboe
2018-06-02  0:56                       ` Jens Axboe
2018-06-02  2:36                       ` Ming Lei
2018-06-02  4:31                         ` Jens Axboe
2018-06-02  4:54                           ` Ming Lei
2018-06-02  8:07                             ` Martin Steigerwald
2018-06-02  8:07                               ` Martin Steigerwald
2018-06-02 13:48                             ` Jens Axboe
2018-06-02 13:48                               ` Jens Axboe
2018-06-05  0:27                   ` Tetsuo Handa
2018-06-05  0:41                     ` Ming Lei
2018-06-07  3:29                       ` Ming Lei
2018-06-07  3:29                         ` Ming Lei
2018-06-07 13:19                         ` Tetsuo Handa
2018-06-04 11:46                 ` Dmitry Vyukov
2018-06-04 13:13                   ` Tetsuo Handa
2018-05-16 17:33     ` Alan Jenkins

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6db16aa3a7c56b6dcca2d10b4e100a780c740081.camel@wdc.com \
    --to=bart.vanassche@wdc.com \
    --cc=alan.christopher.jenkins@gmail.com \
    --cc=axboe@kernel.dk \
    --cc=dan.j.williams@intel.com \
    --cc=dvyukov@google.com \
    --cc=hare@suse.com \
    --cc=hch@lst.de \
    --cc=jthumshirn@suse.de \
    --cc=keith.busch@intel.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-ext4@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=martin.petersen@oracle.com \
    --cc=martin@lichtvoll.de \
    --cc=ming.lei@redhat.com \
    --cc=oleksandr@natalenko.name \
    --cc=penguin-kernel@i-love.sakura.ne.jp \
    --cc=ross.zwisler@linux.intel.com \
    --cc=syzbot+c4f9cebf9d651f6e54de@syzkaller.appspotmail.com \
    --cc=syzkaller-bugs@googlegroups.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.