[PATCH] vhost: try to shrink pfdset when fdset_add fails

[PATCH] vhost: try to shrink pfdset when fdset_add fails

[Matthias Gatto]

fdset_add increment pfdset->num, but fdset_del doesn't decrement pfdset->num,
so if we call fdset_add then fdset_del in a loop witout calling
fdset_shrink, we can easily exceed MAX_FDS with only a few number of fds
used.

So my solution is simply to call fdset_shrink in fdset_add
when it exceed MAX_FDS.

Because fdset_shrink and fdset_add locks pfdset->fd_mutex we can't call
fdset_shrink inside fdset_add because that would cause a dead lock,
so this patch split fdset_shrink in two, fdset_shrink amd fdset_shrink_nolock.

Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
---
 lib/librte_vhost/fd_man.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)

diff --git a/lib/librte_vhost/fd_man.c b/lib/librte_vhost/fd_man.c
index 8a075da..c7a4490 100644
--- a/lib/librte_vhost/fd_man.c
+++ b/lib/librte_vhost/fd_man.c
@@ -65,17 +65,12 @@ fdset_move(struct fdset *pfdset, int dst, int src)
 	pfdset->rwfds[dst] = pfdset->rwfds[src];
 }
 
-/*
- * Find deleted fd entries and remove them
- */
 static void
-fdset_shrink(struct fdset *pfdset)
+fdset_shrink_nolock(struct fdset *pfdset)
 {
 	int i;
 	int last_valid_idx = get_last_valid_idx(pfdset, pfdset->num - 1);
 
-	pthread_mutex_lock(&pfdset->fd_mutex);
-
 	for (i = 0; i < last_valid_idx; i++) {
 		if (pfdset->fd[i].fd != -1)
 			continue;
@@ -84,7 +79,16 @@ fdset_shrink(struct fdset *pfdset)
 		last_valid_idx = get_last_valid_idx(pfdset, last_valid_idx - 1);
 	}
 	pfdset->num = last_valid_idx + 1;
+}
 
+/*
+ * Find deleted fd entries and remove them
+ */
+static void
+fdset_shrink(struct fdset *pfdset)
+{
+	pthread_mutex_lock(&pfdset->fd_mutex);
+	fdset_shrink_nolock(pfdset);
 	pthread_mutex_unlock(&pfdset->fd_mutex);
 }
 
@@ -151,8 +155,12 @@ fdset_add(struct fdset *pfdset, int fd, fd_cb rcb, fd_cb wcb, void *dat)
 	pthread_mutex_lock(&pfdset->fd_mutex);
 	i = pfdset->num < MAX_FDS ? pfdset->num++ : -1;
 	if (i == -1) {
-		pthread_mutex_unlock(&pfdset->fd_mutex);
-		return -2;
+		fdset_shrink_nolock(pfdset);
+		i = pfdset->num < MAX_FDS ? pfdset->num++ : -1;
+		if (i == -1) {
+			pthread_mutex_unlock(&pfdset->fd_mutex);
+			return -2;
+		}
 	}
 
 	fdset_add_fd(pfdset, i, fd, rcb, wcb, dat);
-- 
2.10.0

Re: [PATCH] vhost: try to shrink pfdset when fdset_add fails

[Yuanhan Liu]

On Tue, Feb 21, 2017 at 03:25:30PM +0100, Matthias Gatto wrote:
> fdset_add increment pfdset->num, but fdset_del doesn't decrement pfdset->num,
> so if we call fdset_add then fdset_del in a loop witout calling
> fdset_shrink, we can easily exceed MAX_FDS with only a few number of fds
> used.

I'm assuming you were doing some tests like following?

	while true; do
		ovs-vsctl add-port br vhost_n -- ...
		ovs-vsctl del-port vhost_n
	done

> So my solution is simply to call fdset_shrink in fdset_add
> when it exceed MAX_FDS.
> 
> Because fdset_shrink and fdset_add locks pfdset->fd_mutex we can't call
> fdset_shrink inside fdset_add because that would cause a dead lock,
> so this patch split fdset_shrink in two, fdset_shrink amd fdset_shrink_nolock.

Looks good to me.

	--yliu

Re: [PATCH] vhost: try to shrink pfdset when fdset_add fails

[Matthias Gatto]

On Wed, Feb 22, 2017 at 2:59 AM, Yuanhan Liu
<yuanhan.liu@linux.intel.com> wrote:
>
> On Tue, Feb 21, 2017 at 03:25:30PM +0100, Matthias Gatto wrote:
> > fdset_add increment pfdset->num, but fdset_del doesn't decrement pfdset->num,
> > so if we call fdset_add then fdset_del in a loop witout calling
> > fdset_shrink, we can easily exceed MAX_FDS with only a few number of fds
> > used.
>
> I'm assuming you were doing some tests like following?
>
>         while true; do
>                 ovs-vsctl add-port br vhost_n -- ...
>                 ovs-vsctl del-port vhost_n
>         done
>

Something pretty close to this yes.

Re: [PATCH] vhost: try to shrink pfdset when fdset_add fails

[Yuanhan Liu]

On Wed, Feb 22, 2017 at 11:01:43AM +0100, Matthias Gatto wrote:
> On Wed, Feb 22, 2017 at 2:59 AM, Yuanhan Liu
> <yuanhan.liu@linux.intel.com> wrote:
> >
> > On Tue, Feb 21, 2017 at 03:25:30PM +0100, Matthias Gatto wrote:
> > > fdset_add increment pfdset->num, but fdset_del doesn't decrement pfdset->num,
> > > so if we call fdset_add then fdset_del in a loop witout calling
> > > fdset_shrink, we can easily exceed MAX_FDS with only a few number of fds
> > > used.
> >
> > I'm assuming you were doing some tests like following?
> >
> >         while true; do
> >                 ovs-vsctl add-port br vhost_n -- ...
> >                 ovs-vsctl del-port vhost_n
> >         done
> >
> 
> Something pretty close to this yes.

    Fixes: 59317cef249c ("vhost: allow many vhost-user ports")
    
    Cc: stable@dpdk.org

And applied to dpdk-next-virtio, with few typos in commit log fixed.

Thanks!

	--yliu