[PATCH v2] net/virtio-user: fix not working on 32-bit system

[PATCH v2] net/virtio-user: fix not working on 32-bit system

[Jim Murphy]

Hi,

The fix contained in this patch breaks under the following scenario:

1. A 64 bit host and virtual machine. Therefore all physical addresses are
64 bits.
2. A 32 bit user mode DPDK process running on a 64 bit virtual machine (64
bit kernel).

In this case, the physical address is 64bits but the virtual address of the
user process is 32 bits so uintptr_t is only 32 bits. As a result when:

 (uintptr_t)(mb) + (vq)->offset)

is referenced, only 32 bits are copied into the descriptor but 64 bits are
required because in this scenario that is the size of a physical address.

So it seems like we need a way to determine the size of the physical
address and then VIRTIO_MBUF_ADDR should be written to copy that many bytes
into the uint64_t. Does anyone know how to determine the size of the
physical address?

Thanks,

Jim

Original Post:

virtio-user cannot work on 32-bit system as higher 32-bit of the
addr field (64-bit) in the desc is filled with non-zero value
which should not happen for a 32-bit system.

In case of virtio-user, we use buf_addr of mbuf to fill the
virtqueue desc addr. This is a regression bug. For 32-bit system,
the first 4 bytes of mbuf is buf_addr, with following 8 bytes for
buf_phyaddr. With below wrong definition, both buf_addr and lower
4 bytes buf_phyaddr are obtained to fill the virtqueue desc.
  #define VIRTIO_MBUF_ADDR(mb, vq) \
	(*(uint64_t *)((uintptr_t)(mb) + (vq)->offset))

Fixes: 25f80d108780 ("net/virtio: fix packet corruption")
Cc: stable at dpdk.org <http://dpdk.org/ml/listinfo/dev>

Signed-off-by: Jianfeng Tan <jianfeng.tan at intel.com
<http://dpdk.org/ml/listinfo/dev>>
---
 drivers/net/virtio/virtqueue.h | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
index f9e3736..2e67460 100644
--- a/drivers/net/virtio/virtqueue.h
+++ b/drivers/net/virtio/virtqueue.h
@@ -69,10 +69,16 @@ struct rte_mbuf;

 #ifdef RTE_VIRTIO_USER
 /**
- * Return the physical address (or virtual address in case of
- * virtio-user) of mbuf data buffer.
+ *
+ * Return the physical address of mbuf data buffer for virtio pci:
+ *  on 32-bit system, offset equals 4, return the second four bytes of mbuf;
+ *  on 64-bit system, offset equals 8, return the second eight bytes of mbuf.
+ * Return the virtual address of mbuf data buffer for virtio-user.
+ *  on 32-bit system, offset equals 0, return the first four bytes of mbuf;
+ *  on 64-bit system, offset equals 0, return the first eight bytes of mbuf;
  */
-#define VIRTIO_MBUF_ADDR(mb, vq) (*(uint64_t *)((uintptr_t)(mb) +
(vq)->offset))
+#define VIRTIO_MBUF_ADDR(mb, vq) \
+	((uint64_t)(*(uintptr_t *)((uintptr_t)(mb) + (vq)->offset)))
 #else
 #define VIRTIO_MBUF_ADDR(mb, vq) ((mb)->buf_physaddr)
 #endif
-- 
2.7.4

Re: [PATCH v2] net/virtio-user: fix not working on 32-bit system

[Tan, Jianfeng]

Hi Jim,

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jim Murphy
> Sent: Wednesday, September 20, 2017 6:24 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v2] net/virtio-user: fix not working on 32-bit
> system
> 
> Hi,
> 
> The fix contained in this patch breaks under the following scenario:
> 
> 1. A 64 bit host and virtual machine. Therefore all physical addresses are
> 64 bits.

Host, VM and application are 64bit, then I suppose there's no problem?

> 2. A 32 bit user mode DPDK process running on a 64 bit virtual machine (64
> bit kernel).

Ah, this is a case we fail to cover.

> 
> In this case, the physical address is 64bits but the virtual address of the
> user process is 32 bits so uintptr_t is only 32 bits. As a result when:
> 
>  (uintptr_t)(mb) + (vq)->offset)
> 
> is referenced, only 32 bits are copied into the descriptor but 64 bits are
> required because in this scenario that is the size of a physical address.
> 
> So it seems like we need a way to determine the size of the physical
> address and then VIRTIO_MBUF_ADDR should be written to copy that many
> bytes
> into the uint64_t. Does anyone know how to determine the size of the
> physical address?

It's easy to find a way to decide the length (similar to _offset_), but copying variable length of bytes seems not an efficient way.

Thanks,
Jianfeng

Re: [PATCH v2] net/virtio-user: fix not working on 32-bit system

[Jim Murphy]

On Tue, Sep 19, 2017 at 7:17 PM, Tan, Jianfeng <jianfeng.tan@intel.com>
wrote:

> Hi Jim,
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Jim Murphy
> > Sent: Wednesday, September 20, 2017 6:24 AM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] [PATCH v2] net/virtio-user: fix not working on 32-bit
> > system
> >
> > Hi,
> >
> > The fix contained in this patch breaks under the following scenario:
> >
> > 1. A 64 bit host and virtual machine. Therefore all physical addresses
> are
> > 64 bits.
>
> Host, VM and application are 64bit, then I suppose there's no problem?
>

Right, no problem. I was just describing this as my setup.


>
> > 2. A 32 bit user mode DPDK process running on a 64 bit virtual machine
> (64
> > bit kernel).
>
> Ah, this is a case we fail to cover.
>

Yes.


> >
> > In this case, the physical address is 64bits but the virtual address of
> the
> > user process is 32 bits so uintptr_t is only 32 bits. As a result when:
> >
> >  (uintptr_t)(mb) + (vq)->offset)
> >
> > is referenced, only 32 bits are copied into the descriptor but 64 bits
> are
> > required because in this scenario that is the size of a physical address.
> >
> > So it seems like we need a way to determine the size of the physical
> > address and then VIRTIO_MBUF_ADDR should be written to copy that many
> > bytes
> > into the uint64_t. Does anyone know how to determine the size of the
> > physical address?
>
> It's easy to find a way to decide the length (similar to _offset_), but
> copying variable length of bytes seems not an efficient way.
>
>
I agree regarding the efficiency issue. To get something at compile time
does RTE_MACHINE or something derived from it help us?

Thanks,

Jim

Thanks,
> Jianfeng
>
>

Re: [PATCH v2] net/virtio-user: fix not working on 32-bit system

[Yuanhan Liu]

On Wed, Apr 19, 2017 at 06:21:59AM +0000, Tan, Jianfeng wrote:
> 
> 
> > I've expected it to be plain english, something like following:
> > 
> >     /*
> >      * Return the physical address (or virtual address in case of
> >      * virtio-user) of mbuf data buffer.
> >      *
> >      * The address is firstly casted to the word size (sizeof(uintptr_t))
> >      * before casting it to uint64_t. This is to make it work with different
> >      * combination of word size (64 bit and 32 bit) and virtio device
> >      * (virtio-pci and virtio-user).
> >      */
> > 
> > Okay to you?
> 
> Yep, sounds better. Thanks!

Good. Applied to dpdk-next-virtio, with above change.

Thanks.

	--yliu

Re: [PATCH v2] net/virtio-user: fix not working on 32-bit system

[Tan, Jianfeng]

> -----Original Message-----
> From: Yuanhan Liu [mailto:yuanhan.liu@linux.intel.com]
> Sent: Wednesday, April 19, 2017 1:54 PM
> To: Tan, Jianfeng
> Cc: dev@dpdk.org; olivier.matz@6wind.com; stable@dpdk.org
> Subject: Re: [PATCH v2] net/virtio-user: fix not working on 32-bit system
> 
> On Wed, Apr 19, 2017 at 02:30:33AM +0000, Jianfeng Tan wrote:
> > virtio-user cannot work on 32-bit system as higher 32-bit of the
> > addr field (64-bit) in the desc is filled with non-zero value
> > which should not happen for a 32-bit system.
> >
> > In case of virtio-user, we use buf_addr of mbuf to fill the
> > virtqueue desc addr. This is a regression bug. For 32-bit system,
> > the first 4 bytes of mbuf is buf_addr, with following 8 bytes for
> > buf_phyaddr. With below wrong definition, both buf_addr and lower
> > 4 bytes buf_phyaddr are obtained to fill the virtqueue desc.
> >   #define VIRTIO_MBUF_ADDR(mb, vq) \
> > 	(*(uint64_t *)((uintptr_t)(mb) + (vq)->offset))
> >
> > Fixes: 25f80d108780 ("net/virtio: fix packet corruption")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> > ---
> >  drivers/net/virtio/virtqueue.h | 12 +++++++++---
> >  1 file changed, 9 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
> > index f9e3736..2e67460 100644
> > --- a/drivers/net/virtio/virtqueue.h
> > +++ b/drivers/net/virtio/virtqueue.h
> > @@ -69,10 +69,16 @@ struct rte_mbuf;
> >
> >  #ifdef RTE_VIRTIO_USER
> >  /**
> > - * Return the physical address (or virtual address in case of
> > - * virtio-user) of mbuf data buffer.
> > + *
> > + * Return the physical address of mbuf data buffer for virtio pci:
> > + *  on 32-bit system, offset equals 4, return the second four bytes of mbuf;
> > + *  on 64-bit system, offset equals 8, return the second eight bytes of
> mbuf.
> > + * Return the virtual address of mbuf data buffer for virtio-user.
> > + *  on 32-bit system, offset equals 0, return the first four bytes of mbuf;
> > + *  on 64-bit system, offset equals 0, return the first eight bytes of mbuf;
> 
> 
> I've expected it to be plain english, something like following:
> 
>     /*
>      * Return the physical address (or virtual address in case of
>      * virtio-user) of mbuf data buffer.
>      *
>      * The address is firstly casted to the word size (sizeof(uintptr_t))
>      * before casting it to uint64_t. This is to make it work with different
>      * combination of word size (64 bit and 32 bit) and virtio device
>      * (virtio-pci and virtio-user).
>      */
> 
> Okay to you?

Yep, sounds better. Thanks!

Re: [PATCH v2] net/virtio-user: fix not working on 32-bit system

[Yuanhan Liu]

On Wed, Apr 19, 2017 at 02:30:33AM +0000, Jianfeng Tan wrote:
> virtio-user cannot work on 32-bit system as higher 32-bit of the
> addr field (64-bit) in the desc is filled with non-zero value
> which should not happen for a 32-bit system.
> 
> In case of virtio-user, we use buf_addr of mbuf to fill the
> virtqueue desc addr. This is a regression bug. For 32-bit system,
> the first 4 bytes of mbuf is buf_addr, with following 8 bytes for
> buf_phyaddr. With below wrong definition, both buf_addr and lower
> 4 bytes buf_phyaddr are obtained to fill the virtqueue desc.
>   #define VIRTIO_MBUF_ADDR(mb, vq) \
> 	(*(uint64_t *)((uintptr_t)(mb) + (vq)->offset))
> 
> Fixes: 25f80d108780 ("net/virtio: fix packet corruption")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> ---
>  drivers/net/virtio/virtqueue.h | 12 +++++++++---
>  1 file changed, 9 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
> index f9e3736..2e67460 100644
> --- a/drivers/net/virtio/virtqueue.h
> +++ b/drivers/net/virtio/virtqueue.h
> @@ -69,10 +69,16 @@ struct rte_mbuf;
>  
>  #ifdef RTE_VIRTIO_USER
>  /**
> - * Return the physical address (or virtual address in case of
> - * virtio-user) of mbuf data buffer.
> + *
> + * Return the physical address of mbuf data buffer for virtio pci:
> + *  on 32-bit system, offset equals 4, return the second four bytes of mbuf;
> + *  on 64-bit system, offset equals 8, return the second eight bytes of mbuf.
> + * Return the virtual address of mbuf data buffer for virtio-user.
> + *  on 32-bit system, offset equals 0, return the first four bytes of mbuf;
> + *  on 64-bit system, offset equals 0, return the first eight bytes of mbuf;


I've expected it to be plain english, something like following:

    /*
     * Return the physical address (or virtual address in case of
     * virtio-user) of mbuf data buffer.
     *
     * The address is firstly casted to the word size (sizeof(uintptr_t))
     * before casting it to uint64_t. This is to make it work with different
     * combination of word size (64 bit and 32 bit) and virtio device
     * (virtio-pci and virtio-user).
     */

Okay to you?

	--yliu

[PATCH v2] net/virtio-user: fix not working on 32-bit system

[Jianfeng Tan]

virtio-user cannot work on 32-bit system as higher 32-bit of the
addr field (64-bit) in the desc is filled with non-zero value
which should not happen for a 32-bit system.

In case of virtio-user, we use buf_addr of mbuf to fill the
virtqueue desc addr. This is a regression bug. For 32-bit system,
the first 4 bytes of mbuf is buf_addr, with following 8 bytes for
buf_phyaddr. With below wrong definition, both buf_addr and lower
4 bytes buf_phyaddr are obtained to fill the virtqueue desc.
  #define VIRTIO_MBUF_ADDR(mb, vq) \
	(*(uint64_t *)((uintptr_t)(mb) + (vq)->offset))

Fixes: 25f80d108780 ("net/virtio: fix packet corruption")
Cc: stable@dpdk.org

Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
---
 drivers/net/virtio/virtqueue.h | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
index f9e3736..2e67460 100644
--- a/drivers/net/virtio/virtqueue.h
+++ b/drivers/net/virtio/virtqueue.h
@@ -69,10 +69,16 @@ struct rte_mbuf;
 
 #ifdef RTE_VIRTIO_USER
 /**
- * Return the physical address (or virtual address in case of
- * virtio-user) of mbuf data buffer.
+ *
+ * Return the physical address of mbuf data buffer for virtio pci:
+ *  on 32-bit system, offset equals 4, return the second four bytes of mbuf;
+ *  on 64-bit system, offset equals 8, return the second eight bytes of mbuf.
+ * Return the virtual address of mbuf data buffer for virtio-user.
+ *  on 32-bit system, offset equals 0, return the first four bytes of mbuf;
+ *  on 64-bit system, offset equals 0, return the first eight bytes of mbuf;
  */
-#define VIRTIO_MBUF_ADDR(mb, vq) (*(uint64_t *)((uintptr_t)(mb) + (vq)->offset))
+#define VIRTIO_MBUF_ADDR(mb, vq) \
+	((uint64_t)(*(uintptr_t *)((uintptr_t)(mb) + (vq)->offset)))
 #else
 #define VIRTIO_MBUF_ADDR(mb, vq) ((mb)->buf_physaddr)
 #endif
-- 
2.7.4