From: Ming Lei <ming.lei@redhat.com>
To: Bart Van Assche <bvanassche@acm.org>
Cc: linux-nvme@lists.infradead.org, linux-scsi@vger.kernel.org,
Jens Axboe <axboe@kernel.dk>,
linux-block@vger.kernel.org,
"Martin K . Petersen" <martin.petersen@oracle.com>,
Christoph Hellwig <hch@lst.de>,
Khazhy Kumykov <khazhy@google.com>,
Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>,
Hannes Reinecke <hare@suse.de>,
John Garry <john.garry@huawei.com>,
David Jeffery <djeffery@redhat.com>
Subject: Re: [PATCH 7/8] blk-mq: grab rq->refcount before calling ->fn in blk_mq_tagset_busy_iter
Date: Mon, 26 Apr 2021 08:41:52 +0800 [thread overview]
Message-ID: <YIYMUP2ZKLJZ3KoT@T590> (raw)
In-Reply-To: <6c0b0af9-ca71-d143-b1cc-384adfca5438@acm.org>
On Sun, Apr 25, 2021 at 11:55:22AM -0700, Bart Van Assche wrote:
> On 4/25/21 1:57 AM, Ming Lei wrote:
> > However, still one request UAF not covered: refcount_inc_not_zero() may
> > read one freed request, and it will be handled in next patch.
>
> This means that patch "blk-mq: clear stale request in tags->rq[] before
> freeing one request pool" should come before this patch.
It doesn't matter. This patch only can't avoid the UAF too, we need
to grab req->ref to prevent queue from being frozen.
>
> > @@ -276,12 +277,15 @@ static bool bt_tags_iter(struct sbitmap *bitmap, unsigned int bitnr, void *data)
> > rq = tags->static_rqs[bitnr];
> > else
> > rq = tags->rqs[bitnr];
> > - if (!rq)
> > + if (!rq || !refcount_inc_not_zero(&rq->ref))
> > return true;
> > if ((iter_data->flags & BT_TAG_ITER_STARTED) &&
> > !blk_mq_request_started(rq))
> > - return true;
> > - return iter_data->fn(rq, iter_data->data, reserved);
> > + ret = true;
> > + else
> > + ret = iter_data->fn(rq, iter_data->data, reserved);
> > + blk_mq_put_rq_ref(rq);
> > + return ret;
> > }
>
> Even if patches 7/8 and 8/8 would be reordered, the above code
> introduces a new use-after-free, a use-after-free that is much worse
> than the UAF in kernel v5.11. The following sequence can be triggered by
> the above code:
> * bt_tags_iter() reads tags->rqs[bitnr] and stores the request pointer
> in the 'rq' variable.
> * Request 'rq' completes, tags->rqs[bitnr] is cleared and the memory
> that backs that request is freed.
> * The memory that backs 'rq' is used for another purpose and the request
> reference count becomes nonzero.
That means the 'rq' is re-allocated, and it becomes in-flight again.
> * bt_tags_iter() increments the request reference count and thereby
> corrupts memory.
No, When refcount_inc_not_zero() succeeds in bt_tags_iter(), no one can
free the request any more until ->fn() returns, why do you think memory
corrupts? This pattern isn't different with timeout's usage, is it?
If IO activity is allowed during iterating tagset requests, ->fn() and
in-flight IO can always be run concurrently. That is caller's
responsibility to handle the race. That is why you can see lots callers
do quiesce queues before calling blk_mq_tagset_busy_iter(), but
quiesce isn't required if ->fn() just READs request only.
Your patch or current in-tree code has same 'problem' too, if you think
it is a problem. Clearing ->rq[tag] or holding a lock before calling
->fn() can not avoid such thing, can it?
Finally it is a request walking in tagset wide, so it should be safe for
->fn to iterate over request in this way. The thing is just that req->tag may
become not same with 'bitnr' any more. We can handle it simply by checking
if 'req->tag == bitnr' in bt_tags_iter() after the req->ref is grabbed,
still not sure if it is absolutely necessary.
Thanks,
Ming
next prev parent reply other threads:[~2021-04-26 0:42 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-25 8:57 [PATCH 0/8] blk-mq: fix request UAF related with iterating over tagset requests Ming Lei
2021-04-25 8:57 ` [PATCH 1/8] Revert "blk-mq: Fix races between blk_mq_update_nr_hw_queues() and iterating over tags" Ming Lei
2021-04-25 8:57 ` [PATCH 2/8] Revert "blk-mq: Make it safe to use RCU to iterate over blk_mq_tag_set.tag_list" Ming Lei
2021-04-25 8:57 ` [PATCH 3/8] Revert "blk-mq: Fix races between iterating over requests and freeing requests" Ming Lei
2021-04-25 8:57 ` [PATCH 4/8] Revert "blk-mq: Introduce atomic variants of blk_mq_(all_tag|tagset_busy)_iter" Ming Lei
2021-04-25 8:57 ` [PATCH 5/8] blk-mq: blk_mq_complete_request_locally Ming Lei
2021-04-25 8:57 ` [PATCH 6/8] block: drivers: complete request locally from blk_mq_tagset_busy_iter Ming Lei
2021-04-26 3:02 ` Bart Van Assche
2021-04-26 6:24 ` Ming Lei
2021-04-27 8:54 ` Ming Lei
2021-04-25 8:57 ` [PATCH 7/8] blk-mq: grab rq->refcount before calling ->fn in blk_mq_tagset_busy_iter Ming Lei
2021-04-25 18:55 ` Bart Van Assche
2021-04-26 0:41 ` Ming Lei [this message]
2021-04-25 8:57 ` [PATCH 8/8] blk-mq: clear stale request in tags->rq[] before freeing one request pool Ming Lei
2021-04-25 20:42 ` Bart Van Assche
2021-04-26 0:49 ` Ming Lei
2021-04-26 1:50 ` Bart Van Assche
2021-04-26 2:07 ` Ming Lei
2021-04-25 9:27 ` [PATCH 0/8] blk-mq: fix request UAF related with iterating over tagset requests Ming Lei
2021-04-25 20:53 ` Bart Van Assche
2021-04-26 1:19 ` Ming Lei
2021-04-26 1:57 ` Bart Van Assche
2021-04-25 16:17 ` Jens Axboe
2021-04-25 18:39 ` Bart Van Assche
2021-04-25 20:18 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YIYMUP2ZKLJZ3KoT@T590 \
--to=ming.lei@redhat.com \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=djeffery@redhat.com \
--cc=hare@suse.de \
--cc=hch@lst.de \
--cc=john.garry@huawei.com \
--cc=khazhy@google.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-scsi@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=shinichiro.kawasaki@wdc.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).