[PATCH] vhost: Fix reset_owner message handling not to clear callfd

[PATCH] vhost: Fix reset_owner message handling not to clear callfd

[Tetsuya Mukawa]

The patch fixes reset_owner message handling not to clear callfd,
because callfd will be valid while connection is establihed.

Signed-off-by: Tetsuya Mukawa <mukawa@igel.co.jp>
---
 lib/librte_vhost/virtio-net.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/lib/librte_vhost/virtio-net.c b/lib/librte_vhost/virtio-net.c
index 886c104..ae1e4bd 100644
--- a/lib/librte_vhost/virtio-net.c
+++ b/lib/librte_vhost/virtio-net.c
@@ -322,6 +322,25 @@ init_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
 	init_vring_queue(dev->virtqueue[base_idx + VIRTIO_TXQ], qp_idx);
 }
 
+static void
+reset_vring_queue(struct vhost_virtqueue *vq, int qp_idx)
+{
+	int callfd;
+
+	callfd = vq->callfd;
+	init_vring_queue(vq, qp_idx);
+	vq->callfd = callfd;
+}
+
+static void
+reset_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
+{
+	uint32_t base_idx = qp_idx * VIRTIO_QNUM;
+
+	reset_vring_queue(dev->virtqueue[base_idx + VIRTIO_RXQ], qp_idx);
+	reset_vring_queue(dev->virtqueue[base_idx + VIRTIO_TXQ], qp_idx);
+}
+
 static int
 alloc_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
 {
@@ -362,7 +381,7 @@ reset_device(struct virtio_net *dev)
 	dev->flags = 0;
 
 	for (i = 0; i < dev->virt_qp_nb; i++)
-		init_vring_queue_pair(dev, i);
+		reset_vring_queue_pair(dev, i);
 }
 
 /*
-- 
2.1.4

Re: [PATCH] vhost: Fix reset_owner message handling not to clear callfd

[Yuanhan Liu]

On Thu, Nov 19, 2015 at 03:23:26PM +0900, Tetsuya Mukawa wrote:
> The patch fixes reset_owner message handling not to clear callfd,
> because callfd will be valid while connection is establihed.
> 
> Signed-off-by: Tetsuya Mukawa <mukawa@igel.co.jp>
> ---
>  lib/librte_vhost/virtio-net.c | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_vhost/virtio-net.c b/lib/librte_vhost/virtio-net.c
> index 886c104..ae1e4bd 100644
> --- a/lib/librte_vhost/virtio-net.c
> +++ b/lib/librte_vhost/virtio-net.c
> @@ -322,6 +322,25 @@ init_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
>  	init_vring_queue(dev->virtqueue[base_idx + VIRTIO_TXQ], qp_idx);
>  }
>  
> +static void
> +reset_vring_queue(struct vhost_virtqueue *vq, int qp_idx)
> +{
> +	int callfd;
> +
> +	callfd = vq->callfd;
> +	init_vring_queue(vq, qp_idx);
> +	vq->callfd = callfd;

It may fix your issue, but it's not a proper fix. As we actually closed
callfd at cleanup_vq().

BTW, the name, cleanup_vq/device, is not well taken. For a name like
cleanup, those functions should be invoked at device remove stage (say,
shutting down a VM), but not at reset stage (which is actually treated
as STOP in current QEMU implementation).

I have a plan to correct it, but I guess it's not the right time as
v2.2 is coming soon. However, for a fix, it could be simple: how about
adding another arg to cleanup_vq:

	cleanup_vq(struct vhost_virtqueue *vq, int destory)

And we only close callfd when destroy flag is set. (And of course, this
patch is needed).

What do you think of that?

	--yliu
> +}
> +
> +static void
> +reset_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
> +{
> +	uint32_t base_idx = qp_idx * VIRTIO_QNUM;
> +
> +	reset_vring_queue(dev->virtqueue[base_idx + VIRTIO_RXQ], qp_idx);
> +	reset_vring_queue(dev->virtqueue[base_idx + VIRTIO_TXQ], qp_idx);
> +}
> +
>  static int
>  alloc_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
>  {
> @@ -362,7 +381,7 @@ reset_device(struct virtio_net *dev)
>  	dev->flags = 0;
>  
>  	for (i = 0; i < dev->virt_qp_nb; i++)
> -		init_vring_queue_pair(dev, i);
> +		reset_vring_queue_pair(dev, i);
>  }
>  
>  /*
> -- 
> 2.1.4

Re: [PATCH] vhost: Fix reset_owner message handling not to clear callfd

[Tetsuya Mukawa]

On 2015/11/19 16:03, Yuanhan Liu wrote:
> On Thu, Nov 19, 2015 at 03:23:26PM +0900, Tetsuya Mukawa wrote:
>> The patch fixes reset_owner message handling not to clear callfd,
>> because callfd will be valid while connection is establihed.
>>
>> Signed-off-by: Tetsuya Mukawa <mukawa@igel.co.jp>
>> ---
>>  lib/librte_vhost/virtio-net.c | 21 ++++++++++++++++++++-
>>  1 file changed, 20 insertions(+), 1 deletion(-)
>>
>> diff --git a/lib/librte_vhost/virtio-net.c b/lib/librte_vhost/virtio-net.c
>> index 886c104..ae1e4bd 100644
>> --- a/lib/librte_vhost/virtio-net.c
>> +++ b/lib/librte_vhost/virtio-net.c
>> @@ -322,6 +322,25 @@ init_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
>>  	init_vring_queue(dev->virtqueue[base_idx + VIRTIO_TXQ], qp_idx);
>>  }
>>  
>> +static void
>> +reset_vring_queue(struct vhost_virtqueue *vq, int qp_idx)
>> +{
>> +	int callfd;
>> +
>> +	callfd = vq->callfd;
>> +	init_vring_queue(vq, qp_idx);
>> +	vq->callfd = callfd;
> It may fix your issue, but it's not a proper fix. As we actually closed
> callfd at cleanup_vq().

Yes, you are correct.
We also need to fix above.

>
> BTW, the name, cleanup_vq/device, is not well taken. For a name like
> cleanup, those functions should be invoked at device remove stage (say,
> shutting down a VM), but not at reset stage (which is actually treated
> as STOP in current QEMU implementation).
>
> I have a plan to correct it, but I guess it's not the right time as
> v2.2 is coming soon. However, for a fix, it could be simple: how about
> adding another arg to cleanup_vq:
>
> 	cleanup_vq(struct vhost_virtqueue *vq, int destory)
>
> And we only close callfd when destroy flag is set. (And of course, this
> patch is needed).

Yes I agree with it!
Let me have a few hours, I will send it by today.

Thanks,
Tetsuya

> What do you think of that?
>
> 	--yliu
>> +}
>> +
>> +static void
>> +reset_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
>> +{
>> +	uint32_t base_idx = qp_idx * VIRTIO_QNUM;
>> +
>> +	reset_vring_queue(dev->virtqueue[base_idx + VIRTIO_RXQ], qp_idx);
>> +	reset_vring_queue(dev->virtqueue[base_idx + VIRTIO_TXQ], qp_idx);
>> +}
>> +
>>  static int
>>  alloc_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
>>  {
>> @@ -362,7 +381,7 @@ reset_device(struct virtio_net *dev)
>>  	dev->flags = 0;
>>  
>>  	for (i = 0; i < dev->virt_qp_nb; i++)
>> -		init_vring_queue_pair(dev, i);
>> +		reset_vring_queue_pair(dev, i);
>>  }
>>  
>>  /*
>> -- 
>> 2.1.4

[PATCH v2] vhost: Fix reset_owner message handling not to clear callfd

[Tetsuya Mukawa]

The patch fixes reset_owner message handling not to clear callfd,
because callfd will be valid while connection is establihed.

Signed-off-by: Tetsuya Mukawa <mukawa@igel.co.jp>
---
 lib/librte_vhost/virtio-net.c | 44 +++++++++++++++++++++++++++++++++++++------
 1 file changed, 38 insertions(+), 6 deletions(-)

diff --git a/lib/librte_vhost/virtio-net.c b/lib/librte_vhost/virtio-net.c
index 886c104..3907fb5 100644
--- a/lib/librte_vhost/virtio-net.c
+++ b/lib/librte_vhost/virtio-net.c
@@ -223,9 +223,9 @@ add_config_ll_entry(struct virtio_net_config_ll *new_ll_dev)
 }
 
 static void
-cleanup_vq(struct vhost_virtqueue *vq)
+cleanup_vq(struct vhost_virtqueue *vq, int destroy)
 {
-	if (vq->callfd >= 0)
+	if ((vq->callfd >= 0) && (destroy != 0))
 		close(vq->callfd);
 	if (vq->kickfd >= 0)
 		close(vq->kickfd);
@@ -249,8 +249,8 @@ cleanup_device(struct virtio_net *dev)
 	}
 
 	for (i = 0; i < dev->virt_qp_nb; i++) {
-		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_RXQ]);
-		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_TXQ]);
+		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_RXQ], 1);
+		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_TXQ], 1);
 	}
 }
 
@@ -322,6 +322,25 @@ init_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
 	init_vring_queue(dev->virtqueue[base_idx + VIRTIO_TXQ], qp_idx);
 }
 
+static void
+reset_vring_queue(struct vhost_virtqueue *vq, int qp_idx)
+{
+	int callfd;
+
+	callfd = vq->callfd;
+	init_vring_queue(vq, qp_idx);
+	vq->callfd = callfd;
+}
+
+static void
+reset_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
+{
+	uint32_t base_idx = qp_idx * VIRTIO_QNUM;
+
+	reset_vring_queue(dev->virtqueue[base_idx + VIRTIO_RXQ], qp_idx);
+	reset_vring_queue(dev->virtqueue[base_idx + VIRTIO_TXQ], qp_idx);
+}
+
 static int
 alloc_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
 {
@@ -362,7 +381,7 @@ reset_device(struct virtio_net *dev)
 	dev->flags = 0;
 
 	for (i = 0; i < dev->virt_qp_nb; i++)
-		init_vring_queue_pair(dev, i);
+		reset_vring_queue_pair(dev, i);
 }
 
 /*
@@ -467,6 +486,7 @@ static int
 reset_owner(struct vhost_device_ctx ctx)
 {
 	struct virtio_net *dev;
+	uint32_t i;
 
 	dev = get_device(ctx);
 	if (dev == NULL)
@@ -475,7 +495,19 @@ reset_owner(struct vhost_device_ctx ctx)
 	if (dev->flags & VIRTIO_DEV_RUNNING)
 		notify_destroy_device(dev);
 
-	cleanup_device(dev);
+	/* Unmap QEMU memory file if mapped. */
+	if (dev->mem) {
+		munmap((void *)(uintptr_t)dev->mem->mapped_address,
+			(size_t)dev->mem->mapped_size);
+		free(dev->mem);
+		dev->mem = NULL;
+	}
+
+	for (i = 0; i < dev->virt_qp_nb; i++) {
+		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_RXQ], 0);
+		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_TXQ], 0);
+	}
+
 	reset_device(dev);
 	return 0;
 }
-- 
2.1.4

Re: [PATCH v2] vhost: Fix reset_owner message handling not to clear callfd

[Yuanhan Liu]

On Thu, Nov 19, 2015 at 06:07:00PM +0900, Tetsuya Mukawa wrote:
> The patch fixes reset_owner message handling not to clear callfd,
> because callfd will be valid while connection is establihed.
> 
> Signed-off-by: Tetsuya Mukawa <mukawa@igel.co.jp>
....
>  
>  /*
> @@ -467,6 +486,7 @@ static int
>  reset_owner(struct vhost_device_ctx ctx)
>  {
>  	struct virtio_net *dev;
> +	uint32_t i;
>  
>  	dev = get_device(ctx);
>  	if (dev == NULL)
> @@ -475,7 +495,19 @@ reset_owner(struct vhost_device_ctx ctx)
>  	if (dev->flags & VIRTIO_DEV_RUNNING)
>  		notify_destroy_device(dev);
>  
> -	cleanup_device(dev);

I would suggest to introduce the destory flag to cleanup_device() as
well,  so that we don't have to unfold the cleanup_device code here.

(BTW, I should've reviewed this patch this morning, but I was busy).

	--yliu
> +	/* Unmap QEMU memory file if mapped. */
> +	if (dev->mem) {
> +		munmap((void *)(uintptr_t)dev->mem->mapped_address,
> +			(size_t)dev->mem->mapped_size);
> +		free(dev->mem);
> +		dev->mem = NULL;
> +	}
> +
> +	for (i = 0; i < dev->virt_qp_nb; i++) {
> +		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_RXQ], 0);
> +		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_TXQ], 0);
> +	}
> +
>  	reset_device(dev);
>  	return 0;
>  }
> -- 
> 2.1.4

Re: [PATCH v2] vhost: Fix reset_owner message handling not to clear callfd

[Tetsuya Mukawa]

On 2015/11/20 20:21, Yuanhan Liu wrote:
> On Thu, Nov 19, 2015 at 06:07:00PM +0900, Tetsuya Mukawa wrote:
>> The patch fixes reset_owner message handling not to clear callfd,
>> because callfd will be valid while connection is establihed.
>>
>> Signed-off-by: Tetsuya Mukawa <mukawa@igel.co.jp>
> ....
>>  
>>  /*
>> @@ -467,6 +486,7 @@ static int
>>  reset_owner(struct vhost_device_ctx ctx)
>>  {
>>  	struct virtio_net *dev;
>> +	uint32_t i;
>>  
>>  	dev = get_device(ctx);
>>  	if (dev == NULL)
>> @@ -475,7 +495,19 @@ reset_owner(struct vhost_device_ctx ctx)
>>  	if (dev->flags & VIRTIO_DEV_RUNNING)
>>  		notify_destroy_device(dev);
>>  
>> -	cleanup_device(dev);
> I would suggest to introduce the destory flag to cleanup_device() as
> well,  so that we don't have to unfold the cleanup_device code here.
>
> (BTW, I should've reviewed this patch this morning, but I was busy).

Thanks, I will submit it again.

Tetsuya

[PATCH v3] vhost: Fix reset_owner message handling not to clear callfd

[Tetsuya Mukawa]

The patch fixes reset_owner message handling not to clear callfd,
because callfd will be valid while connection is establihed.

Signed-off-by: Tetsuya Mukawa <mukawa@igel.co.jp>
---
 lib/librte_vhost/virtio-net.c | 39 +++++++++++++++++++++++++++++----------
 1 file changed, 29 insertions(+), 10 deletions(-)

diff --git a/lib/librte_vhost/virtio-net.c b/lib/librte_vhost/virtio-net.c
index 886c104..dc977b7 100644
--- a/lib/librte_vhost/virtio-net.c
+++ b/lib/librte_vhost/virtio-net.c
@@ -223,9 +223,9 @@ add_config_ll_entry(struct virtio_net_config_ll *new_ll_dev)
 }
 
 static void
-cleanup_vq(struct vhost_virtqueue *vq)
+cleanup_vq(struct vhost_virtqueue *vq, int destroy)
 {
-	if (vq->callfd >= 0)
+	if ((vq->callfd >= 0) && (destroy != 0))
 		close(vq->callfd);
 	if (vq->kickfd >= 0)
 		close(vq->kickfd);
@@ -236,7 +236,7 @@ cleanup_vq(struct vhost_virtqueue *vq)
  * free any memory owned by a device.
  */
 static void
-cleanup_device(struct virtio_net *dev)
+cleanup_device(struct virtio_net *dev, int destroy)
 {
 	uint32_t i;
 
@@ -249,8 +249,8 @@ cleanup_device(struct virtio_net *dev)
 	}
 
 	for (i = 0; i < dev->virt_qp_nb; i++) {
-		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_RXQ]);
-		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_TXQ]);
+		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_RXQ], destroy);
+		cleanup_vq(dev->virtqueue[i * VIRTIO_QNUM + VIRTIO_TXQ], destroy);
 	}
 }
 
@@ -278,17 +278,17 @@ rm_config_ll_entry(struct virtio_net_config_ll *ll_dev,
 	/* First remove the device and then clean it up. */
 	if (ll_dev == ll_root) {
 		ll_root = ll_dev->next;
-		cleanup_device(&ll_dev->dev);
+		cleanup_device(&ll_dev->dev, 1);
 		free_device(ll_dev);
 		return ll_root;
 	} else {
 		if (likely(ll_dev_last != NULL)) {
 			ll_dev_last->next = ll_dev->next;
-			cleanup_device(&ll_dev->dev);
+			cleanup_device(&ll_dev->dev, 1);
 			free_device(ll_dev);
 			return ll_dev_last->next;
 		} else {
-			cleanup_device(&ll_dev->dev);
+			cleanup_device(&ll_dev->dev, 1);
 			free_device(ll_dev);
 			RTE_LOG(ERR, VHOST_CONFIG,
 				"Remove entry from config_ll failed\n");
@@ -322,6 +322,25 @@ init_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
 	init_vring_queue(dev->virtqueue[base_idx + VIRTIO_TXQ], qp_idx);
 }
 
+static void
+reset_vring_queue(struct vhost_virtqueue *vq, int qp_idx)
+{
+	int callfd;
+
+	callfd = vq->callfd;
+	init_vring_queue(vq, qp_idx);
+	vq->callfd = callfd;
+}
+
+static void
+reset_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
+{
+	uint32_t base_idx = qp_idx * VIRTIO_QNUM;
+
+	reset_vring_queue(dev->virtqueue[base_idx + VIRTIO_RXQ], qp_idx);
+	reset_vring_queue(dev->virtqueue[base_idx + VIRTIO_TXQ], qp_idx);
+}
+
 static int
 alloc_vring_queue_pair(struct virtio_net *dev, uint32_t qp_idx)
 {
@@ -362,7 +381,7 @@ reset_device(struct virtio_net *dev)
 	dev->flags = 0;
 
 	for (i = 0; i < dev->virt_qp_nb; i++)
-		init_vring_queue_pair(dev, i);
+		reset_vring_queue_pair(dev, i);
 }
 
 /*
@@ -475,7 +494,7 @@ reset_owner(struct vhost_device_ctx ctx)
 	if (dev->flags & VIRTIO_DEV_RUNNING)
 		notify_destroy_device(dev);
 
-	cleanup_device(dev);
+	cleanup_device(dev, 0);
 	reset_device(dev);
 	return 0;
 }
-- 
2.1.4

Re: [PATCH v3] vhost: Fix reset_owner message handling not to clear callfd

[Yuanhan Liu]

On Tue, Nov 24, 2015 at 03:45:35PM +0900, Tetsuya Mukawa wrote:
> The patch fixes reset_owner message handling not to clear callfd,
> because callfd will be valid while connection is establihed.
> 
> Signed-off-by: Tetsuya Mukawa <mukawa@igel.co.jp>

Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

Thanks.

Re: [PATCH v3] vhost: Fix reset_owner message handling not to clear callfd

[Thomas Monjalon]

2015-11-24 15:15, Yuanhan Liu:
> On Tue, Nov 24, 2015 at 03:45:35PM +0900, Tetsuya Mukawa wrote:
> > The patch fixes reset_owner message handling not to clear callfd,
> > because callfd will be valid while connection is establihed.
> > 
> > Signed-off-by: Tetsuya Mukawa <mukawa@igel.co.jp>
> 
> Acked-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>

Applied, thanks