From: Halil Pasic <pasic@linux.ibm.com> To: Jason Wang <jasowang@redhat.com> Cc: "kaplan, david" <david.kaplan@amd.com>, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, "Hetzelt, Felicitas" <f.hetzelt@tu-berlin.de>, linux-kernel <linux-kernel@vger.kernel.org>, virtualization <virtualization@lists.linux-foundation.org>, Halil Pasic <pasic@linux.ibm.com>, mst <mst@redhat.com>, Stefan Hajnoczi <stefanha@redhat.com> Subject: Re: [PATCH V5 1/4] virtio_ring: validate used buffer length Date: Mon, 22 Nov 2021 06:35:18 +0100 [thread overview] Message-ID: <20211122063518.37929c01.pasic@linux.ibm.com> (raw) In-Reply-To: <CACGkMEtja2TPC=ujgMrpaPmdsy+zHowbBTvPj8k7nm_+zB8vig@mail.gmail.com> On Mon, 22 Nov 2021 11:51:09 +0800 Jason Wang <jasowang@redhat.com> wrote: > On Fri, Nov 19, 2021 at 11:10 PM Halil Pasic <pasic@linux.ibm.com> wrote: > > > > On Wed, 27 Oct 2021 10:21:04 +0800 > > Jason Wang <jasowang@redhat.com> wrote: > > > > > This patch validate the used buffer length provided by the device > > > before trying to use it. This is done by record the in buffer length > > > in a new field in desc_state structure during virtqueue_add(), then we > > > can fail the virtqueue_get_buf() when we find the device is trying to > > > give us a used buffer length which is greater than the in buffer > > > length. > > > > > > Since some drivers have already done the validation by themselves, > > > this patch tries to makes the core validation optional. For the driver > > > that doesn't want the validation, it can set the > > > suppress_used_validation to be true (which could be overridden by > > > force_used_validation module parameter). To be more efficient, a > > > dedicate array is used for storing the validate used length, this > > > helps to eliminate the cache stress if validation is done by the > > > driver. > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com> > > > > Hi Jason! > > > > Our CI has detected, that virtio-vsock became unusable with this > > patch on s390x. I didn't test on x86 yet. The guest kernel says > > something like: > > vmw_vsock_virtio_transport virtio1: tx: used len 44 is larger than in buflen 0 > > > > Did you, or anybody else, see something like this on platforms other that > > s390x? > > Adding Stefan and Stefano. > > I think it should be a common issue, looking at > vhost_vsock_handle_tx_kick(), it did: > > len += sizeof(pkt->hdr); > vhost_add_used(vq, head, len); > > which looks like a violation of the spec since it's TX. I'm not sure the lines above look like a violation of the spec. If you examine vhost_vsock_alloc_pkt() I believe that you will agree that: len == pkt->len == pkt->hdr.len which makes sense since according to the spec both tx and rx messages are hdr+payload. And I believe hdr.len is the size of the payload, although that does not seem to be properly documented by the spec. On the other hand tx messages are stated to be device read-only (in the spec) so if the device writes stuff, that is certainly wrong. If that is what happens. Looking at virtqueue_get_buf_ctx_split() I'm not sure that is what happens. My hypothesis is that we just a last descriptor is an 'in' type descriptor (i.e. a device writable one). For tx that assumption would be wrong. I will have another look at this today and send a fix patch if my suspicion is confirmed. > > > > > I had a quick look at this code, and I speculate that it probably > > uncovers a pre-existig bug, rather than introducing a new one. > > I agree. > :) I'm not so sure any more myself. > > > > If somebody is already working on this please reach out to me. > > AFAIK, no. Thanks for the info! Then I will dig a little deeper. I asked in order to avoid doing the debugging and fixing just to see that somebody was faster :D > I think the plan is to fix both the device and drive side > (but I'm not sure we need a new feature for this if we stick to the > validation). > > Thanks > Thank you! Regards, Halil _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
WARNING: multiple messages have this Message-ID (diff)
From: Halil Pasic <pasic@linux.ibm.com> To: Jason Wang <jasowang@redhat.com> Cc: mst <mst@redhat.com>, virtualization <virtualization@lists.linux-foundation.org>, "Hetzelt, Felicitas" <f.hetzelt@tu-berlin.de>, linux-kernel <linux-kernel@vger.kernel.org>, "kaplan, david" <david.kaplan@amd.com>, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, Stefan Hajnoczi <stefanha@redhat.com>, Stefano Garzarella <sgarzare@redhat.com>, Halil Pasic <pasic@linux.ibm.com> Subject: Re: [PATCH V5 1/4] virtio_ring: validate used buffer length Date: Mon, 22 Nov 2021 06:35:18 +0100 [thread overview] Message-ID: <20211122063518.37929c01.pasic@linux.ibm.com> (raw) In-Reply-To: <CACGkMEtja2TPC=ujgMrpaPmdsy+zHowbBTvPj8k7nm_+zB8vig@mail.gmail.com> On Mon, 22 Nov 2021 11:51:09 +0800 Jason Wang <jasowang@redhat.com> wrote: > On Fri, Nov 19, 2021 at 11:10 PM Halil Pasic <pasic@linux.ibm.com> wrote: > > > > On Wed, 27 Oct 2021 10:21:04 +0800 > > Jason Wang <jasowang@redhat.com> wrote: > > > > > This patch validate the used buffer length provided by the device > > > before trying to use it. This is done by record the in buffer length > > > in a new field in desc_state structure during virtqueue_add(), then we > > > can fail the virtqueue_get_buf() when we find the device is trying to > > > give us a used buffer length which is greater than the in buffer > > > length. > > > > > > Since some drivers have already done the validation by themselves, > > > this patch tries to makes the core validation optional. For the driver > > > that doesn't want the validation, it can set the > > > suppress_used_validation to be true (which could be overridden by > > > force_used_validation module parameter). To be more efficient, a > > > dedicate array is used for storing the validate used length, this > > > helps to eliminate the cache stress if validation is done by the > > > driver. > > > > > > Signed-off-by: Jason Wang <jasowang@redhat.com> > > > > Hi Jason! > > > > Our CI has detected, that virtio-vsock became unusable with this > > patch on s390x. I didn't test on x86 yet. The guest kernel says > > something like: > > vmw_vsock_virtio_transport virtio1: tx: used len 44 is larger than in buflen 0 > > > > Did you, or anybody else, see something like this on platforms other that > > s390x? > > Adding Stefan and Stefano. > > I think it should be a common issue, looking at > vhost_vsock_handle_tx_kick(), it did: > > len += sizeof(pkt->hdr); > vhost_add_used(vq, head, len); > > which looks like a violation of the spec since it's TX. I'm not sure the lines above look like a violation of the spec. If you examine vhost_vsock_alloc_pkt() I believe that you will agree that: len == pkt->len == pkt->hdr.len which makes sense since according to the spec both tx and rx messages are hdr+payload. And I believe hdr.len is the size of the payload, although that does not seem to be properly documented by the spec. On the other hand tx messages are stated to be device read-only (in the spec) so if the device writes stuff, that is certainly wrong. If that is what happens. Looking at virtqueue_get_buf_ctx_split() I'm not sure that is what happens. My hypothesis is that we just a last descriptor is an 'in' type descriptor (i.e. a device writable one). For tx that assumption would be wrong. I will have another look at this today and send a fix patch if my suspicion is confirmed. > > > > > I had a quick look at this code, and I speculate that it probably > > uncovers a pre-existig bug, rather than introducing a new one. > > I agree. > :) I'm not so sure any more myself. > > > > If somebody is already working on this please reach out to me. > > AFAIK, no. Thanks for the info! Then I will dig a little deeper. I asked in order to avoid doing the debugging and fixing just to see that somebody was faster :D > I think the plan is to fix both the device and drive side > (but I'm not sure we need a new feature for this if we stick to the > validation). > > Thanks > Thank you! Regards, Halil
next prev parent reply other threads:[~2021-11-22 5:35 UTC|newest] Thread overview: 65+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-10-27 2:21 [PATCH V5 0/4] Validate used buffer length Jason Wang 2021-10-27 2:21 ` Jason Wang 2021-10-27 2:21 ` [PATCH V5 1/4] virtio_ring: validate " Jason Wang 2021-10-27 2:21 ` Jason Wang 2021-11-02 3:18 ` Xuan Zhuo 2021-11-02 3:54 ` Jason Wang 2021-11-02 3:54 ` Jason Wang 2021-11-19 15:09 ` Halil Pasic 2021-11-19 15:09 ` Halil Pasic 2021-11-22 3:51 ` Jason Wang 2021-11-22 3:51 ` Jason Wang 2021-11-22 5:35 ` Halil Pasic [this message] 2021-11-22 5:35 ` Halil Pasic 2021-11-22 5:49 ` Halil Pasic 2021-11-22 5:49 ` Halil Pasic 2021-11-22 6:25 ` Jason Wang 2021-11-22 6:25 ` Jason Wang 2021-11-22 7:55 ` Stefano Garzarella 2021-11-22 7:55 ` Stefano Garzarella 2021-11-22 11:08 ` Stefano Garzarella 2021-11-22 11:08 ` Stefano Garzarella 2021-11-22 14:24 ` Halil Pasic 2021-11-22 14:24 ` Halil Pasic 2021-11-22 16:23 ` Stefano Garzarella 2021-11-22 16:23 ` Stefano Garzarella 2021-11-22 13:50 ` Halil Pasic 2021-11-22 13:50 ` Halil Pasic 2021-11-23 2:30 ` Jason Wang 2021-11-23 2:30 ` Jason Wang 2021-11-23 12:17 ` Michael S. Tsirkin 2021-11-23 12:17 ` Michael S. Tsirkin 2021-11-23 12:43 ` Halil Pasic 2021-11-23 12:43 ` Halil Pasic 2021-11-22 20:23 ` Halil Pasic 2021-11-22 20:23 ` Halil Pasic 2021-11-23 2:25 ` Jason Wang 2021-11-23 2:25 ` Jason Wang 2021-11-23 11:05 ` Michael S. Tsirkin 2021-11-23 11:05 ` Michael S. Tsirkin 2021-11-24 1:30 ` Michael Ellerman 2021-11-24 1:30 ` Michael Ellerman 2021-11-24 2:26 ` Jason Wang 2021-11-24 2:26 ` Jason Wang 2021-11-24 2:33 ` Jason Wang 2021-11-24 2:33 ` Jason Wang 2021-11-24 7:22 ` Michael S. Tsirkin 2021-11-24 7:22 ` Michael S. Tsirkin 2021-11-24 7:59 ` Jason Wang 2021-11-24 7:59 ` Jason Wang 2021-11-24 8:24 ` Michael S. Tsirkin 2021-11-24 8:24 ` Michael S. Tsirkin 2021-11-24 8:28 ` Jason Wang 2021-11-24 8:28 ` Jason Wang 2021-11-24 11:33 ` Halil Pasic 2021-11-24 11:33 ` Halil Pasic 2021-11-25 2:27 ` Jason Wang 2021-11-25 2:27 ` Jason Wang 2021-11-22 7:42 ` Stefano Garzarella 2021-11-22 7:42 ` Stefano Garzarella 2021-10-27 2:21 ` [PATCH V5 2/4] virtio-net: don't let virtio core to validate used length Jason Wang 2021-10-27 2:21 ` Jason Wang 2021-10-27 2:21 ` [PATCH V5 3/4] virtio-blk: " Jason Wang 2021-10-27 2:21 ` Jason Wang 2021-10-27 2:21 ` [PATCH V5 4/4] virtio-scsi: don't let virtio core to validate used buffer length Jason Wang 2021-10-27 2:21 ` Jason Wang
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20211122063518.37929c01.pasic@linux.ibm.com \ --to=pasic@linux.ibm.com \ --cc=david.kaplan@amd.com \ --cc=f.hetzelt@tu-berlin.de \ --cc=jasowang@redhat.com \ --cc=konrad.wilk@oracle.com \ --cc=linux-kernel@vger.kernel.org \ --cc=mst@redhat.com \ --cc=stefanha@redhat.com \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.