From: Sasha Levin <sashal@kernel.org>
To: Dexuan Cui <decui@microsoft.com>
Cc: kimbrownkd <kimbrownkd@gmail.com>,
Michael Kelley <mikelley@microsoft.com>,
Long Li <longli@microsoft.com>,
Sasha Levin <Alexander.Levin@microsoft.com>,
Stephen Hemminger <sthemmin@microsoft.com>,
KY Srinivasan <kys@microsoft.com>,
Haiyang Zhang <haiyangz@microsoft.com>,
"devel@linuxdriverproject.org" <devel@linuxdriverproject.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] Drivers: hv: vmbus: Add mutex lock to channel show functions
Date: Thu, 31 Jan 2019 10:19:58 -0500 [thread overview]
Message-ID: <20190131151958.GT3973@sasha-vm> (raw)
In-Reply-To: <PU1P153MB01699B96311157EA9DF515E6BF970@PU1P153MB0169.APCP153.PROD.OUTLOOK.COM>
On Tue, Jan 29, 2019 at 07:20:28PM +0000, Dexuan Cui wrote:
>> From: Kimberly Brown <kimbrownkd@gmail.com>
>> > ...
>> > But as you pointed, at least for sub-channels, channel->ringbuffer_page
>> > can indeed disappear in vmbus_close() -> ... -> vmbus_free_ring(), and
>> > the "attribute->show()" could crash when the race happens.
>> > Adding channel_mutex here seems to be able to fix the race for
>> > sub-channels, as the same channel_mutex is used in
>> vmbus_disconnect_ring().
>> >
>> > For a primary channel, vmbus_close() -> vmbus_free_ring() can still
>> > free channel->ringbuffer_page without notifying the "attribute->show()".
>> > We may also need to acquire/release the channel_mutex in vmbus_close()?
>> >
>> > > Actually, there should be checks that "chan" is not null and that
>> > > "chan->state" is CHANNEL_OPENED_STATE within the locked section. I'll
>> > > need to fix that.
>> > I suppose "chan" can not be NULL here (see the above).
>> >
>> > Checking "chan->state" may not help to completely fix the race, because
>> > there is no locking/synchronization code in
>> > vmbus_close_internal() when we test and change "channel->state".
>> >
>>
>> The calls to vmbus_close_internal() for the subchannels and the primary
>> channel are protected with channel_mutex in vmbus_disconnect_ring().
>> This prevents "channel->state" from changing while "attribute->show()" is
>> running.
>Ah, I think you're right.
>
>> > I guess we may need to check if channel->ringbuffer_page is NULL in
>> > the "attribute->show()".
>> >
>>
>> For the primary channel, vmbus_free_ring() is called after the
>> return from vmbus_disconnect_ring(). Therefore, the primary channel's
>> state is changed before "channel->ringbuffer_page" is set to NULL.
>> Checking the channel state should be sufficient to prevent the ring
>> buffers from being freed while "attribute->show()" is running. The
>> ring buffers can't be freed until the channel's state is changed, and
>> the channel state change is protected by the mutex.
>I think you're right (I noticed in a previous mail you mentioned you would
>improve your patch to check "chan->state" with the mutax held).
>
>> I think checking that "channel->ringbuffer_page" is not NULL would
>> also work, but, as you stated, we would need to aquire/release
>> channel_mutex in vmbus_close().
>Then it looks unnecessary to check "channel->ringbuffer_page".
>
>> > PS, to prove that a race condition does exist and can really cause a panic or
>> > something, I usually add some msleep() delays in different paths so that I
>> > can reproduce the crash every time I take a special action, e.g. when I read
>> > the sysfs files of a NIC, I try to remove hv_netvsc driver. This way, I can prove
>> > a patch can indeed help, at least it can fix the crash which would happen
>> > without the patch. :-)
>> >
>>
>> Thanks! I was able to free the ring buffers while "attribute->show()"
>> was running, which caused a null pointer dereference bug. As expected,
>> the mutex lock fixed it.
>Awesome!
I've queued this one for hyper-fixes, thanks all!
--
Thanks,
Sasha
next prev parent reply other threads:[~2019-01-31 15:20 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-22 2:07 [PATCH] Drivers: hv: vmbus: Add mutex lock to channel show functions Kimberly Brown
2019-01-22 3:46 ` Dexuan Cui
2019-01-22 6:42 ` Kimberly Brown
2019-01-22 18:40 ` Dexuan Cui
2019-01-28 19:58 ` Kimberly Brown
2019-01-29 19:20 ` Dexuan Cui
2019-01-31 15:19 ` Sasha Levin [this message]
2019-01-31 16:45 ` Michael Kelley
2019-01-31 17:47 ` Kimberly Brown
2019-02-01 14:13 ` Sasha Levin
2019-02-01 18:24 ` Dexuan Cui
2019-02-02 20:07 ` Kimberly Brown
2019-02-15 1:54 ` Sasha Levin
2019-02-15 2:27 ` Kimberly Brown
2019-02-22 3:46 ` [PATCH v2 0/2] Fix a race condition vulnerability in "_show" functions Kimberly Brown
2019-02-22 3:47 ` [PATCH v2 1/2] Drivers: hv: vmbus: Refactor chan->state if statement Kimberly Brown
2019-02-24 16:54 ` Michael Kelley
2019-02-22 3:47 ` [PATCH v2 2/2] Drivers: hv: vmbus: Add a channel ring buffer mutex lock Kimberly Brown
2019-02-24 16:53 ` Michael Kelley
2019-02-26 6:24 ` Kimberly Brown
2019-03-14 20:04 ` [PATCH v3 0/3] Drivers: hv: vmbus: Fix a race condition in "_show" functions Kimberly Brown
2019-03-14 20:05 ` [PATCH v3 1/3] Drivers: hv: vmbus: Refactor chan->state if statement Kimberly Brown
2019-03-14 20:05 ` [PATCH v3 2/3] Drivers: hv: vmbus: Set ring_info field to 0 and remove memset Kimberly Brown
2019-03-29 16:01 ` Michael Kelley
2019-03-14 20:05 ` [PATCH v3 3/3] Drivers: hv: vmbus: Fix race condition with new ring_buffer_info mutex Kimberly Brown
2019-03-14 22:45 ` Stephen Hemminger
2019-03-17 1:49 ` Kimberly Brown
2019-03-20 20:06 ` Stephen Hemminger
2019-03-21 3:47 ` Kimberly Brown
2019-03-21 16:04 ` Michael Kelley
2019-03-28 4:30 ` Kimberly Brown
2019-03-28 18:42 ` Stephen Hemminger
2019-03-29 16:04 ` Michael Kelley
2019-04-10 22:59 ` [PATCH v3 0/3] Drivers: hv: vmbus: Fix a race condition in "_show" functions Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190131151958.GT3973@sasha-vm \
--to=sashal@kernel.org \
--cc=Alexander.Levin@microsoft.com \
--cc=decui@microsoft.com \
--cc=devel@linuxdriverproject.org \
--cc=haiyangz@microsoft.com \
--cc=kimbrownkd@gmail.com \
--cc=kys@microsoft.com \
--cc=linux-kernel@vger.kernel.org \
--cc=longli@microsoft.com \
--cc=mikelley@microsoft.com \
--cc=sthemmin@microsoft.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).