From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 996C3C433EF for ; Tue, 8 Mar 2022 08:58:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345262AbiCHI7D (ORCPT ); Tue, 8 Mar 2022 03:59:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345259AbiCHI7A (ORCPT ); Tue, 8 Mar 2022 03:59:00 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B74E3C4AF; Tue, 8 Mar 2022 00:58:03 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1DF69B817D4; Tue, 8 Mar 2022 08:58:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7FA6AC340EB; Tue, 8 Mar 2022 08:58:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1646729880; bh=g8JnrciyA5hVe9iyiMja39wfKFTI4eghy2XVxEq8voQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=t/s29xuMklvlvsfmHRfDg9OiclbQlbVZX8uChLs/iNDYVHYxBhBhmcB/RTVBN8DrM sg4pdWLQEckaaK6banT+xBdd6RHSBthFqKnhND6G/1OFKtgE+z7J909kFBLzLQWXKQ pqo+UZmD8jNT0P7y8UJ53GCHfD2zkk2UVSkJViy8= Date: Tue, 8 Mar 2022 09:57:57 +0100 From: Greg KH To: Lee Jones Cc: mst@redhat.com, jasowang@redhat.com, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, stable@vger.kernel.org, syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com Subject: Re: [PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use Message-ID: References: <20220307191757.3177139-1-lee.jones@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Mar 08, 2022 at 08:10:06AM +0000, Lee Jones wrote: > On Mon, 07 Mar 2022, Greg KH wrote: > > > On Mon, Mar 07, 2022 at 07:17:57PM +0000, Lee Jones wrote: > > > vhost_vsock_handle_tx_kick() already holds the mutex during its call > > > to vhost_get_vq_desc(). All we have to do here is take the same lock > > > during virtqueue clean-up and we mitigate the reported issues. > > > > > > Also WARN() as a precautionary measure. The purpose of this is to > > > capture possible future race conditions which may pop up over time. > > > > > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00 > > > > > > Cc: > > > Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com > > > Signed-off-by: Lee Jones > > > --- > > > drivers/vhost/vhost.c | 10 ++++++++++ > > > 1 file changed, 10 insertions(+) > > > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > > index 59edb5a1ffe28..ef7e371e3e649 100644 > > > --- a/drivers/vhost/vhost.c > > > +++ b/drivers/vhost/vhost.c > > > @@ -693,6 +693,15 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > > int i; > > > > > > for (i = 0; i < dev->nvqs; ++i) { > > > + /* No workers should run here by design. However, races have > > > + * previously occurred where drivers have been unable to flush > > > + * all work properly prior to clean-up. Without a successful > > > + * flush the guest will malfunction, but avoiding host memory > > > + * corruption in those cases does seem preferable. > > > + */ > > > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > > > > So you are trading one syzbot triggered issue for another one in the > > future? :) > > > > If this ever can happen, handle it, but don't log it with a WARN_ON() as > > that will trigger the panic-on-warn boxes, as well as syzbot. Unless > > you want that to happen? > > No, Syzbot doesn't report warnings, only BUGs and memory corruption. Has it changed? Last I looked, it did trigger on WARN_* calls, which has resulted in a huge number of kernel fixes because of that. > > And what happens if the mutex is locked _RIGHT_ after you checked it? > > You still have a race... > > No, we miss a warning that one time. Memory is still protected. Then don't warn on something that doesn't matter. This line can be dropped as there's nothing anyone can do about it, right? thanks, greg k-h From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F2A8C433EF for ; Tue, 8 Mar 2022 08:58:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id DEF6160F2C; Tue, 8 Mar 2022 08:58:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x7H_yoyUDGbp; Tue, 8 Mar 2022 08:58:06 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6BDB360BA3; Tue, 8 Mar 2022 08:58:05 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 47FD5C001D; Tue, 8 Mar 2022 08:58:05 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 92A1CC000B for ; Tue, 8 Mar 2022 08:58:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 8320940193 for ; Tue, 8 Mar 2022 08:58:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=linuxfoundation.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Qyif4LqHYuGE for ; Tue, 8 Mar 2022 08:58:02 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by smtp4.osuosl.org (Postfix) with ESMTPS id A2AE040164 for ; Tue, 8 Mar 2022 08:58:02 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7C63261239; Tue, 8 Mar 2022 08:58:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7FA6AC340EB; Tue, 8 Mar 2022 08:58:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1646729880; bh=g8JnrciyA5hVe9iyiMja39wfKFTI4eghy2XVxEq8voQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=t/s29xuMklvlvsfmHRfDg9OiclbQlbVZX8uChLs/iNDYVHYxBhBhmcB/RTVBN8DrM sg4pdWLQEckaaK6banT+xBdd6RHSBthFqKnhND6G/1OFKtgE+z7J909kFBLzLQWXKQ pqo+UZmD8jNT0P7y8UJ53GCHfD2zkk2UVSkJViy8= Date: Tue, 8 Mar 2022 09:57:57 +0100 From: Greg KH To: Lee Jones Subject: Re: [PATCH 1/1] vhost: Protect the virtqueue from being cleared whilst still in use Message-ID: References: <20220307191757.3177139-1-lee.jones@linaro.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Cc: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com, kvm@vger.kernel.org, mst@redhat.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, virtualization@lists.linux-foundation.org X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" On Tue, Mar 08, 2022 at 08:10:06AM +0000, Lee Jones wrote: > On Mon, 07 Mar 2022, Greg KH wrote: > > > On Mon, Mar 07, 2022 at 07:17:57PM +0000, Lee Jones wrote: > > > vhost_vsock_handle_tx_kick() already holds the mutex during its call > > > to vhost_get_vq_desc(). All we have to do here is take the same lock > > > during virtqueue clean-up and we mitigate the reported issues. > > > > > > Also WARN() as a precautionary measure. The purpose of this is to > > > capture possible future race conditions which may pop up over time. > > > > > > Link: https://syzkaller.appspot.com/bug?extid=279432d30d825e63ba00 > > > > > > Cc: > > > Reported-by: syzbot+adc3cb32385586bec859@syzkaller.appspotmail.com > > > Signed-off-by: Lee Jones > > > --- > > > drivers/vhost/vhost.c | 10 ++++++++++ > > > 1 file changed, 10 insertions(+) > > > > > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c > > > index 59edb5a1ffe28..ef7e371e3e649 100644 > > > --- a/drivers/vhost/vhost.c > > > +++ b/drivers/vhost/vhost.c > > > @@ -693,6 +693,15 @@ void vhost_dev_cleanup(struct vhost_dev *dev) > > > int i; > > > > > > for (i = 0; i < dev->nvqs; ++i) { > > > + /* No workers should run here by design. However, races have > > > + * previously occurred where drivers have been unable to flush > > > + * all work properly prior to clean-up. Without a successful > > > + * flush the guest will malfunction, but avoiding host memory > > > + * corruption in those cases does seem preferable. > > > + */ > > > + WARN_ON(mutex_is_locked(&dev->vqs[i]->mutex)); > > > > So you are trading one syzbot triggered issue for another one in the > > future? :) > > > > If this ever can happen, handle it, but don't log it with a WARN_ON() as > > that will trigger the panic-on-warn boxes, as well as syzbot. Unless > > you want that to happen? > > No, Syzbot doesn't report warnings, only BUGs and memory corruption. Has it changed? Last I looked, it did trigger on WARN_* calls, which has resulted in a huge number of kernel fixes because of that. > > And what happens if the mutex is locked _RIGHT_ after you checked it? > > You still have a race... > > No, we miss a warning that one time. Memory is still protected. Then don't warn on something that doesn't matter. This line can be dropped as there's nothing anyone can do about it, right? thanks, greg k-h _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization