From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19D72C25B08 for ; Mon, 15 Aug 2022 17:46:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230453AbiHORqZ (ORCPT ); Mon, 15 Aug 2022 13:46:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229446AbiHORqX (ORCPT ); Mon, 15 Aug 2022 13:46:23 -0400 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD35627FF1; Mon, 15 Aug 2022 10:46:22 -0700 (PDT) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 91FF25C0138; Mon, 15 Aug 2022 13:46:19 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 15 Aug 2022 13:46:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anarazel.de; h= cc:cc:content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1660585579; x=1660671979; bh=8OMR3TElBe 6FtukCUYXb52i6A/aUd+KCr1sfP2Mcrq4=; b=WWh1N78Lr62Hl69WT1r86CNWh+ roxkNiICEOZ7YJGTtdJZ49uTb0HJG+JMAdmUbCy8eWMufL0xmC7HkDRtwZLXNKPz gzSrEDzp63kP/pleU6F076KJt2zCcamtdnuytaIKqPaTo/6+n0rd6mVBYRagD+vB etZs+OsABrz5kX+2kzllpigxikUAKxEo68L3HqIEw2DR9P8hH6DuBE7JYqltbhXS Lwbvc9EQiPa1jEEUHwStQswg38TH1WaKdEnoO5GGkCPn4bk/NFw95wwCaolItbfs BtyCYp4Rl027xTBsqjUExhbcQRYmYGc1eygzn2x8Q5xWgtOyql9mYbWe2GaQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1660585579; x=1660671979; bh=8OMR3TElBe6FtukCUYXb52i6A/aU d+KCr1sfP2Mcrq4=; b=tEMDvu+R2Aspw36NwuK6kFqZRns/mDPUG4fIwlwUQ/ID wCquAEfl8s9ofbcABUZAS88RihgDaBKF+aIiqXSMkXtEdaDfu42fTV2/97XquZ3b 65Fgk7idAukoCMbQxZkN7SDn+1FdBerVJVOT/SCIgCuENHDpV+aDoiE+d9ex7zAX vfcs79dq2yNOBIkfSksvTzrnTuA138zyh66Rd8HmSL+/YWJSvmPMQPG1PQ2WCo+c DFPUMQQ9alpFTkT1DvLXQBFeRVcIhFNbFnz4MYn95zSrfwpidNRE5UZUsz9I4mN5 rcOxRTb3FCsbdJDgafCSUanZSa+nr3K2vqkSeTu4Kg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdehvddguddujecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpeetnhgu rhgvshcuhfhrvghunhguuceorghnughrvghssegrnhgrrhgriigvlhdruggvqeenucggtf frrghtthgvrhhnpedvffefvefhteevffegieetfefhtddvffejvefhueetgeeludehteev udeitedtudenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhroh hmpegrnhgurhgvshesrghnrghrrgiivghlrdguvg X-ME-Proxy: Feedback-ID: id4a34324:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 15 Aug 2022 13:46:18 -0400 (EDT) Date: Mon, 15 Aug 2022 10:46:17 -0700 From: Andres Freund To: "Michael S. Tsirkin" Cc: Xuan Zhuo , Jason Wang , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, Linus Torvalds , Jens Axboe , James Bottomley , "Martin K. Petersen" , Guenter Roeck , linux-kernel@vger.kernel.org, Greg KH , c@redhat.com Subject: Re: upstream kernel crashes Message-ID: <20220815174617.z4chnftzcbv6frqr@awork3.anarazel.de> References: <20220815013651.mrm7qgklk6sgpkbb@awork3.anarazel.de> <20220815071143.n2t5xsmifnigttq2@awork3.anarazel.de> <20220815034532-mutt-send-email-mst@kernel.org> <20220815081527.soikyi365azh5qpu@awork3.anarazel.de> <20220815042623-mutt-send-email-mst@kernel.org> <20220815113729-mutt-send-email-mst@kernel.org> <20220815164503.jsoezxcm6q4u2b6j@awork3.anarazel.de> <20220815124748-mutt-send-email-mst@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220815124748-mutt-send-email-mst@kernel.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, On 2022-08-15 12:50:52 -0400, Michael S. Tsirkin wrote: > On Mon, Aug 15, 2022 at 09:45:03AM -0700, Andres Freund wrote: > > Hi, > > > > On 2022-08-15 11:40:59 -0400, Michael S. Tsirkin wrote: > > > OK so this gives us a quick revert as a solution for now. > > > Next, I would appreciate it if you just try this simple hack. > > > If it crashes we either have a long standing problem in virtio > > > code or more likely a gcp bug where it can't handle smaller > > > rings than what device requestes. > > > Thanks! > > > > I applied the below and the problem persists. > > > > [...] > > Okay! Just checking - I applied and tested this atop 6.0-rc1, correct? Or did you want me to test it with the 762faee5a267 reverted? I guess what you're trying to test if a smaller queue than what's requested you'd want to do so without the problematic patch applied... > And just to be 100% sure, can you try the following on top of 5.19: > diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c > index 623906b4996c..6f4e54a618bc 100644 > --- a/drivers/virtio/virtio_pci_modern.c > +++ b/drivers/virtio/virtio_pci_modern.c > @@ -208,6 +208,9 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev, > return ERR_PTR(-EINVAL); > } > > + if (num > 1024) > + num = 1024; > + > info->msix_vector = msix_vec; > > /* create the vring */ > > -- Either way, I did this, and there are no issues that I could observe. No oopses, no broken networking. But: To make sure it does something I added a debugging printk - which doesn't show up. I assume this is at a point at least earlyprintk should work (which I see getting enabled via serial)? Greetings, Andres Freund From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 52677C00140 for ; Mon, 15 Aug 2022 17:46:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B5C1760777; Mon, 15 Aug 2022 17:46:24 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org B5C1760777 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key, unprotected) header.d=anarazel.de header.i=@anarazel.de header.a=rsa-sha256 header.s=fm1 header.b=WWh1N78L; dkim=fail reason="signature verification failed" (2048-bit key, unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm1 header.b=tEMDvu+R X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tgmSRimpx3GQ; Mon, 15 Aug 2022 17:46:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 3418C6059C; Mon, 15 Aug 2022 17:46:23 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 3418C6059C Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id D8D82C0033; Mon, 15 Aug 2022 17:46:22 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id BE9B4C002D for ; Mon, 15 Aug 2022 17:46:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 78E2D605B3 for ; Mon, 15 Aug 2022 17:46:21 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 78E2D605B3 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pslObCm3qogy for ; Mon, 15 Aug 2022 17:46:20 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 995616059C Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp3.osuosl.org (Postfix) with ESMTPS id 995616059C for ; Mon, 15 Aug 2022 17:46:20 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 91FF25C0138; Mon, 15 Aug 2022 13:46:19 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 15 Aug 2022 13:46:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anarazel.de; h= cc:cc:content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1660585579; x=1660671979; bh=8OMR3TElBe 6FtukCUYXb52i6A/aUd+KCr1sfP2Mcrq4=; b=WWh1N78Lr62Hl69WT1r86CNWh+ roxkNiICEOZ7YJGTtdJZ49uTb0HJG+JMAdmUbCy8eWMufL0xmC7HkDRtwZLXNKPz gzSrEDzp63kP/pleU6F076KJt2zCcamtdnuytaIKqPaTo/6+n0rd6mVBYRagD+vB etZs+OsABrz5kX+2kzllpigxikUAKxEo68L3HqIEw2DR9P8hH6DuBE7JYqltbhXS Lwbvc9EQiPa1jEEUHwStQswg38TH1WaKdEnoO5GGkCPn4bk/NFw95wwCaolItbfs BtyCYp4Rl027xTBsqjUExhbcQRYmYGc1eygzn2x8Q5xWgtOyql9mYbWe2GaQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1660585579; x=1660671979; bh=8OMR3TElBe6FtukCUYXb52i6A/aU d+KCr1sfP2Mcrq4=; b=tEMDvu+R2Aspw36NwuK6kFqZRns/mDPUG4fIwlwUQ/ID wCquAEfl8s9ofbcABUZAS88RihgDaBKF+aIiqXSMkXtEdaDfu42fTV2/97XquZ3b 65Fgk7idAukoCMbQxZkN7SDn+1FdBerVJVOT/SCIgCuENHDpV+aDoiE+d9ex7zAX vfcs79dq2yNOBIkfSksvTzrnTuA138zyh66Rd8HmSL+/YWJSvmPMQPG1PQ2WCo+c DFPUMQQ9alpFTkT1DvLXQBFeRVcIhFNbFnz4MYn95zSrfwpidNRE5UZUsz9I4mN5 rcOxRTb3FCsbdJDgafCSUanZSa+nr3K2vqkSeTu4Kg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdehvddguddujecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpeetnhgu rhgvshcuhfhrvghunhguuceorghnughrvghssegrnhgrrhgriigvlhdruggvqeenucggtf frrghtthgvrhhnpedvffefvefhteevffegieetfefhtddvffejvefhueetgeeludehteev udeitedtudenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhroh hmpegrnhgurhgvshesrghnrghrrgiivghlrdguvg X-ME-Proxy: Feedback-ID: id4a34324:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 15 Aug 2022 13:46:18 -0400 (EDT) Date: Mon, 15 Aug 2022 10:46:17 -0700 From: Andres Freund To: "Michael S. Tsirkin" Subject: Re: upstream kernel crashes Message-ID: <20220815174617.z4chnftzcbv6frqr@awork3.anarazel.de> References: <20220815013651.mrm7qgklk6sgpkbb@awork3.anarazel.de> <20220815071143.n2t5xsmifnigttq2@awork3.anarazel.de> <20220815034532-mutt-send-email-mst@kernel.org> <20220815081527.soikyi365azh5qpu@awork3.anarazel.de> <20220815042623-mutt-send-email-mst@kernel.org> <20220815113729-mutt-send-email-mst@kernel.org> <20220815164503.jsoezxcm6q4u2b6j@awork3.anarazel.de> <20220815124748-mutt-send-email-mst@kernel.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20220815124748-mutt-send-email-mst@kernel.org> Cc: Jens Axboe , "Martin K. Petersen" , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, James Bottomley , Eric Dumazet , Greg KH , c@redhat.com, Jakub Kicinski , Paolo Abeni , Linus Torvalds , "David S. Miller" , Guenter Roeck X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" Hi, On 2022-08-15 12:50:52 -0400, Michael S. Tsirkin wrote: > On Mon, Aug 15, 2022 at 09:45:03AM -0700, Andres Freund wrote: > > Hi, > > > > On 2022-08-15 11:40:59 -0400, Michael S. Tsirkin wrote: > > > OK so this gives us a quick revert as a solution for now. > > > Next, I would appreciate it if you just try this simple hack. > > > If it crashes we either have a long standing problem in virtio > > > code or more likely a gcp bug where it can't handle smaller > > > rings than what device requestes. > > > Thanks! > > > > I applied the below and the problem persists. > > > > [...] > > Okay! Just checking - I applied and tested this atop 6.0-rc1, correct? Or did you want me to test it with the 762faee5a267 reverted? I guess what you're trying to test if a smaller queue than what's requested you'd want to do so without the problematic patch applied... > And just to be 100% sure, can you try the following on top of 5.19: > diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c > index 623906b4996c..6f4e54a618bc 100644 > --- a/drivers/virtio/virtio_pci_modern.c > +++ b/drivers/virtio/virtio_pci_modern.c > @@ -208,6 +208,9 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev, > return ERR_PTR(-EINVAL); > } > > + if (num > 1024) > + num = 1024; > + > info->msix_vector = msix_vec; > > /* create the vring */ > > -- Either way, I did this, and there are no issues that I could observe. No oopses, no broken networking. But: To make sure it does something I added a debugging printk - which doesn't show up. I assume this is at a point at least earlyprintk should work (which I see getting enabled via serial)? Greetings, Andres Freund _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization