From: "Michael S. Tsirkin" <mst@redhat.com> To: Xuan Zhuo <xuanzhuo@linux.alibaba.com>, Jason Wang <jasowang@redhat.com>, Andres Freund <andres@anarazel.de>, "David S. Miller" <davem@davemloft.net>, Eric Dumazet <edumazet@google.com>, Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, Linus Torvalds <torvalds@linux-foundation.org>, Jens Axboe <axboe@kernel.dk>, James Bottomley <James.Bottomley@hansenpartnership.com>, "Martin K. Petersen" <martin.petersen@oracle.com>, Guenter Roeck <linux@roeck-us.net>, linux-kernel@vger.kernel.org, Greg KH <gregkh@linuxfoundation.org>, c@redhat.com Subject: Re: upstream kernel crashes Date: Mon, 15 Aug 2022 11:40:59 -0400 [thread overview] Message-ID: <20220815113729-mutt-send-email-mst@kernel.org> (raw) In-Reply-To: <FCDC5DDE-3CDD-4B8A-916F-CA7D87B547CE@anarazel.de> On Mon, Aug 15, 2022 at 01:34:41AM -0700, Andres Freund wrote: > Hi, > > On August 15, 2022 1:28:29 AM PDT, "Michael S. Tsirkin" <mst@redhat.com> wrote: > >On Mon, Aug 15, 2022 at 01:15:27AM -0700, Andres Freund wrote: > >> Hi, > >> > >> On 2022-08-15 03:51:34 -0400, Michael S. Tsirkin wrote: > >> > It is possible that GCP gets confused if ring size is smaller than the > >> > device maximum simply because no one did it in the past. > >> > > >> > So I pushed just the revert of 762faee5a267 to the test branch. > >> > Could you give it a spin? > >> > >> Seems to fix the issue, at least to the extent I can determine at 1am... :) > >> > >> Greetings, > >> > >> Andres Freund > > > >So you tested this: > > > >commit 13df5a7eaeb22561d39354b576bc98a7e2c389f9 (HEAD, kernel.org/test) > >Author: Michael S. Tsirkin <mst@redhat.com> > >Date: Mon Aug 15 03:44:38 2022 -0400 > > > > Revert "virtio_net: set the default max ring size by find_vqs()" > > > > This reverts commit 762faee5a2678559d3dc09d95f8f2c54cd0466a7. > > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com> > > > >and it fixes both issues right? No crashes no networking issue? > > Correct. I only did limited testing, but it's survived far longer / more reboots than anything since the commit. > > Andres > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. OK so this gives us a quick revert as a solution for now. Next, I would appreciate it if you just try this simple hack. If it crashes we either have a long standing problem in virtio code or more likely a gcp bug where it can't handle smaller rings than what device requestes. Thanks! diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c index f7965c5dd36b..bdd5f481570b 100644 --- a/drivers/virtio/virtio_pci_modern.c +++ b/drivers/virtio/virtio_pci_modern.c @@ -314,6 +314,9 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev, if (!size || size > num) size = num; + if (size > 1024) + size = 1024; + if (size & (size - 1)) { dev_warn(&vp_dev->pci_dev->dev, "bad queue size %u", size); return ERR_PTR(-EINVAL); -- MST
WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com> To: Xuan Zhuo <xuanzhuo@linux.alibaba.com>, Jason Wang <jasowang@redhat.com>, Andres Freund <andres@anarazel.de>, "David S. Miller" <davem@davemloft.net>, Eric Dumazet <edumazet@google.com>, Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, Linus Torvalds <torvalds@linux-foundation.org>, Jens Axboe <axboe@kernel.dk>, James Bottomley <James.Bottomley@hansenpartnership.com>, "Martin K. Petersen" <martin.petersen@oracle.com>, Guenter Roeck <linux@roeck-us.net>, linux-kernel@vger.kernel.org, Greg KH <gregkh@linuxfoundation.org>, c@redhat.com Subject: Re: upstream kernel crashes Date: Mon, 15 Aug 2022 11:40:59 -0400 [thread overview] Message-ID: <20220815113729-mutt-send-email-mst@kernel.org> (raw) In-Reply-To: <FCDC5DDE-3CDD-4B8A-916F-CA7D87B547CE@anarazel.de> On Mon, Aug 15, 2022 at 01:34:41AM -0700, Andres Freund wrote: > Hi, > > On August 15, 2022 1:28:29 AM PDT, "Michael S. Tsirkin" <mst@redhat.com> wrote: > >On Mon, Aug 15, 2022 at 01:15:27AM -0700, Andres Freund wrote: > >> Hi, > >> > >> On 2022-08-15 03:51:34 -0400, Michael S. Tsirkin wrote: > >> > It is possible that GCP gets confused if ring size is smaller than the > >> > device maximum simply because no one did it in the past. > >> > > >> > So I pushed just the revert of 762faee5a267 to the test branch. > >> > Could you give it a spin? > >> > >> Seems to fix the issue, at least to the extent I can determine at 1am... :) > >> > >> Greetings, > >> > >> Andres Freund > > > >So you tested this: > > > >commit 13df5a7eaeb22561d39354b576bc98a7e2c389f9 (HEAD, kernel.org/test) > >Author: Michael S. Tsirkin <mst@redhat.com> > >Date: Mon Aug 15 03:44:38 2022 -0400 > > > > Revert "virtio_net: set the default max ring size by find_vqs()" > > > > This reverts commit 762faee5a2678559d3dc09d95f8f2c54cd0466a7. > > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com> > > > >and it fixes both issues right? No crashes no networking issue? > > Correct. I only did limited testing, but it's survived far longer / more reboots than anything since the commit. > > Andres > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. OK so this gives us a quick revert as a solution for now. Next, I would appreciate it if you just try this simple hack. If it crashes we either have a long standing problem in virtio code or more likely a gcp bug where it can't handle smaller rings than what device requestes. Thanks! diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c index f7965c5dd36b..bdd5f481570b 100644 --- a/drivers/virtio/virtio_pci_modern.c +++ b/drivers/virtio/virtio_pci_modern.c @@ -314,6 +314,9 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev, if (!size || size > num) size = num; + if (size > 1024) + size = 1024; + if (size & (size - 1)) { dev_warn(&vp_dev->pci_dev->dev, "bad queue size %u", size); return ERR_PTR(-EINVAL); -- MST _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2022-08-15 15:41 UTC|newest] Thread overview: 62+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-08-14 21:26 upstream kernel crashes Guenter Roeck 2022-08-14 21:40 ` Linus Torvalds 2022-08-14 22:37 ` Andres Freund 2022-08-14 22:47 ` Linus Torvalds 2022-08-15 1:04 ` Jens Axboe 2022-08-15 1:36 ` Andres Freund 2022-08-15 3:18 ` Linus Torvalds 2022-08-15 7:11 ` Andres Freund 2022-08-15 7:29 ` Michael S. Tsirkin 2022-08-15 7:46 ` Andres Freund 2022-08-15 7:53 ` Michael S. Tsirkin 2022-08-15 8:02 ` Michael S. Tsirkin 2022-08-15 8:02 ` Michael S. Tsirkin 2022-08-15 7:51 ` Michael S. Tsirkin 2022-08-15 8:15 ` Andres Freund 2022-08-15 8:28 ` Michael S. Tsirkin 2022-08-15 8:34 ` Andres Freund 2022-08-15 15:40 ` Michael S. Tsirkin [this message] 2022-08-15 15:40 ` Michael S. Tsirkin 2022-08-15 16:45 ` Andres Freund 2022-08-15 16:45 ` Andres Freund 2022-08-15 16:50 ` Michael S. Tsirkin 2022-08-15 16:50 ` Michael S. Tsirkin 2022-08-15 17:46 ` Andres Freund 2022-08-15 17:46 ` Andres Freund 2022-08-15 20:21 ` Michael S. Tsirkin 2022-08-15 20:21 ` Michael S. Tsirkin 2022-08-15 20:53 ` Andres Freund 2022-08-15 20:53 ` Andres Freund 2022-08-15 21:04 ` Andres Freund 2022-08-15 21:04 ` Andres Freund 2022-08-15 21:10 ` Andres Freund 2022-08-15 21:10 ` Andres Freund 2022-08-15 21:32 ` Michael S. Tsirkin 2022-08-15 21:32 ` Michael S. Tsirkin 2022-08-16 2:45 ` Xuan Zhuo 2022-08-16 2:45 ` Xuan Zhuo 2022-08-17 6:13 ` Dmitry Vyukov 2022-08-17 6:13 ` Dmitry Vyukov via Virtualization 2022-08-17 6:36 ` Xuan Zhuo 2022-08-17 6:36 ` Xuan Zhuo 2022-08-17 10:53 ` Michael S. Tsirkin 2022-08-17 10:53 ` Michael S. Tsirkin 2022-08-17 15:58 ` Linus Torvalds 2022-08-17 15:58 ` Linus Torvalds 2022-08-18 1:55 ` Xuan Zhuo 2022-08-18 1:55 ` Xuan Zhuo 2022-08-15 20:45 ` Guenter Roeck 2022-08-15 20:45 ` Guenter Roeck 2022-08-15 6:36 ` Michael S. Tsirkin 2022-08-15 7:17 ` Andres Freund 2022-08-15 7:43 ` Michael S. Tsirkin 2022-08-15 1:17 ` Guenter Roeck 2022-08-15 1:29 ` Jens Axboe 2022-08-15 9:43 ` Michael S. Tsirkin 2022-08-15 15:49 ` Guenter Roeck 2022-08-15 16:01 ` Michael S. Tsirkin 2022-08-15 18:22 ` Guenter Roeck 2022-08-15 18:37 ` Linus Torvalds 2022-08-15 20:38 ` Guenter Roeck 2022-08-17 17:12 ` Linus Torvalds 2022-08-18 1:08 ` Andres Freund
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20220815113729-mutt-send-email-mst@kernel.org \ --to=mst@redhat.com \ --cc=James.Bottomley@hansenpartnership.com \ --cc=andres@anarazel.de \ --cc=axboe@kernel.dk \ --cc=c@redhat.com \ --cc=davem@davemloft.net \ --cc=edumazet@google.com \ --cc=gregkh@linuxfoundation.org \ --cc=jasowang@redhat.com \ --cc=kuba@kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux@roeck-us.net \ --cc=martin.petersen@oracle.com \ --cc=netdev@vger.kernel.org \ --cc=pabeni@redhat.com \ --cc=torvalds@linux-foundation.org \ --cc=virtualization@lists.linux-foundation.org \ --cc=xuanzhuo@linux.alibaba.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.