[RFC 0/5] Clean up error handling in libvhost-user memory mapping

[RFC 0/5] Clean up error handling in libvhost-user memory mapping

[Raphael Norwitz]

Hey Stefan, Marc-Andre, MST, David -

As promised here is a series cleaning up error handling in the
libvhost-user memory mapping path. Most of these cleanups are
straightforward and have been discussed on the mailing list in threads
[1] and [2]. Hopefully there is nothing super controversial in the first
4 patches.

I am concerned about is patch 5 “libvhost-user: handle removal of
identical regions”. From my reading of Stefan's comments in [1], the
proposal seemed to be to remove any duplicate regions. I’d prefer to
prevent duplicate regions from being added in the first place. Thoughts? 

[1] https://lore.kernel.org/qemu-devel/20211018143319.GA11006@raphael-debian-dev/
[2] https://lore.kernel.org/qemu-devel/9391f500-70be-26cf-bcfc-591d3ee84d4e@redhat.com/

Sorry for the delay,
Raphael

David Hildenbrand (1):
  libvhost-user: Simplify VHOST_USER_REM_MEM_REG

Raphael Norwitz (4):
  libvhost-user: Add vu_rem_mem_reg input validation
  libvhost-user: Add vu_add_mem_reg input validation
  libvhost-user: prevent over-running max RAM slots
  libvhost-user: handle removal of identical regions

 subprojects/libvhost-user/libvhost-user.c | 52 +++++++++++++++--------
 1 file changed, 34 insertions(+), 18 deletions(-)

-- 
2.20.1

[RFC 1/5] libvhost-user: Add vu_rem_mem_reg input validation

[Raphael Norwitz]

Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
---
 subprojects/libvhost-user/libvhost-user.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
index 787f4d2d4f..573212a83b 100644
--- a/subprojects/libvhost-user/libvhost-user.c
+++ b/subprojects/libvhost-user/libvhost-user.c
@@ -801,6 +801,12 @@ vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
     VuDevRegion shadow_regions[VHOST_USER_MAX_RAM_SLOTS] = {};
     VhostUserMemoryRegion m = vmsg->payload.memreg.region, *msg_region = &m;
 
+    if (vmsg->fd_num != 1 ||
+        vmsg->size != sizeof(vmsg->payload.memreg)) {
+        vu_panic(dev, "VHOST_USER_REM_MEM_REG received multiple regions");
+        return true;
+    }
+
     DPRINT("Removing region:\n");
     DPRINT("    guest_phys_addr: 0x%016"PRIx64"\n",
            msg_region->guest_phys_addr);
-- 
2.20.1

[RFC 2/5] libvhost-user: Add vu_add_mem_reg input validation

[Raphael Norwitz]

Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
---
 subprojects/libvhost-user/libvhost-user.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
index 573212a83b..80ef335254 100644
--- a/subprojects/libvhost-user/libvhost-user.c
+++ b/subprojects/libvhost-user/libvhost-user.c
@@ -690,6 +690,12 @@ vu_add_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
     VuDevRegion *dev_region = &dev->regions[dev->nregions];
     void *mmap_addr;
 
+    if (vmsg->fd_num != 1 ||
+        vmsg->size != sizeof(vmsg->payload.memreg)) {
+        vu_panic(dev, "VHOST_USER_REM_MEM_REG received multiple regions");
+        return true;
+    }
+
     /*
      * If we are in postcopy mode and we receive a u64 payload with a 0 value
      * we know all the postcopy client bases have been received, and we
-- 
2.20.1

[RFC 3/5] libvhost-user: Simplify VHOST_USER_REM_MEM_REG

[Raphael Norwitz]

From: David Hildenbrand <david@redhat.com>

Let's avoid having to manually copy all elements. Copy only the ones
necessary to close the hole and perform the operation in-place without
a second array.

Signed-off-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
---
 subprojects/libvhost-user/libvhost-user.c | 30 +++++++++++------------
 1 file changed, 14 insertions(+), 16 deletions(-)

diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
index 80ef335254..714cc7e08b 100644
--- a/subprojects/libvhost-user/libvhost-user.c
+++ b/subprojects/libvhost-user/libvhost-user.c
@@ -802,10 +802,8 @@ static inline bool reg_equal(VuDevRegion *vudev_reg,
 
 static bool
 vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
-    int i, j;
-    bool found = false;
-    VuDevRegion shadow_regions[VHOST_USER_MAX_RAM_SLOTS] = {};
     VhostUserMemoryRegion m = vmsg->payload.memreg.region, *msg_region = &m;
+    int i;
 
     if (vmsg->fd_num != 1 ||
         vmsg->size != sizeof(vmsg->payload.memreg)) {
@@ -823,28 +821,28 @@ vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
     DPRINT("    mmap_offset      0x%016"PRIx64"\n",
            msg_region->mmap_offset);
 
-    for (i = 0, j = 0; i < dev->nregions; i++) {
-        if (!reg_equal(&dev->regions[i], msg_region)) {
-            shadow_regions[j].gpa = dev->regions[i].gpa;
-            shadow_regions[j].size = dev->regions[i].size;
-            shadow_regions[j].qva = dev->regions[i].qva;
-            shadow_regions[j].mmap_addr = dev->regions[i].mmap_addr;
-            shadow_regions[j].mmap_offset = dev->regions[i].mmap_offset;
-            j++;
-        } else {
-            found = true;
+    for (i = 0; i < dev->nregions; i++) {
+        if (reg_equal(&dev->regions[i], msg_region)) {
             VuDevRegion *r = &dev->regions[i];
             void *m = (void *) (uintptr_t) r->mmap_addr;
 
             if (m) {
                 munmap(m, r->size + r->mmap_offset);
             }
+
+            break;
         }
     }
 
-    if (found) {
-        memcpy(dev->regions, shadow_regions,
-               sizeof(VuDevRegion) * VHOST_USER_MAX_RAM_SLOTS);
+    if (i < dev->nregions) {
+        /*
+         * Shift all affected entries by 1 to close the hole at index i and
+         * zero out the last entry.
+         */
+        memmove(dev->regions + i, dev->regions + i + 1,
+               sizeof(VuDevRegion) * (dev->nregions - i - 1));
+        memset(dev->regions + dev->nregions - 1, 0,
+               sizeof(VuDevRegion));
         DPRINT("Successfully removed a region\n");
         dev->nregions--;
         vmsg_set_reply_u64(vmsg, 0);
-- 
2.20.1

[RFC 4/5] libvhost-user: prevent over-running max RAM slots

[Raphael Norwitz]

When VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS support was added to
libvhost-user, no guardrails were added to protect against QEMU
attempting to hot-add too many RAM slots to a VM with a libvhost-user
based backed attached.

This change adds the missing error handling by introducing a check on
the number of RAM slots the device has available before proceeding to
process the VHOST_USER_ADD_MEM_REG message.

Suggested-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
---
 subprojects/libvhost-user/libvhost-user.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
index 714cc7e08b..74a9980194 100644
--- a/subprojects/libvhost-user/libvhost-user.c
+++ b/subprojects/libvhost-user/libvhost-user.c
@@ -690,6 +690,11 @@ vu_add_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
     VuDevRegion *dev_region = &dev->regions[dev->nregions];
     void *mmap_addr;
 
+    if (dev->nregions == VHOST_USER_MAX_RAM_SLOTS) {
+        vu_panic(dev, "No free ram slots available");
+        return true;
+    }
+
     if (vmsg->fd_num != 1 ||
         vmsg->size != sizeof(vmsg->payload.memreg)) {
         vu_panic(dev, "VHOST_USER_REM_MEM_REG received multiple regions");
-- 
2.20.1

[RFC 5/5] libvhost-user: handle removal of identical regions

[Raphael Norwitz]

Today if QEMU (or any other VMM) has sent multiple copies of the same
region to a libvhost-user based backend and then attempts to remove the
region, only one instance of the region will be removed, leaving stale
copies of the region in dev->regions[].

This change resolves this by having vu_rem_mem_reg() iterate through all
regions in dev->regions[] and delete all matching regions.

Suggested-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
---
 subprojects/libvhost-user/libvhost-user.c | 27 ++++++++++++-----------
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
index 74a9980194..2f465a4f0e 100644
--- a/subprojects/libvhost-user/libvhost-user.c
+++ b/subprojects/libvhost-user/libvhost-user.c
@@ -809,6 +809,7 @@ static bool
 vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
     VhostUserMemoryRegion m = vmsg->payload.memreg.region, *msg_region = &m;
     int i;
+    bool found = false;
 
     if (vmsg->fd_num != 1 ||
         vmsg->size != sizeof(vmsg->payload.memreg)) {
@@ -831,25 +832,25 @@ vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
             VuDevRegion *r = &dev->regions[i];
             void *m = (void *) (uintptr_t) r->mmap_addr;
 
-            if (m) {
+            if (m && !found) {
                 munmap(m, r->size + r->mmap_offset);
             }
 
-            break;
+            /*
+             * Shift all affected entries by 1 to close the hole at index i and
+             * zero out the last entry.
+             */
+            memmove(dev->regions + i, dev->regions + i + 1,
+                    sizeof(VuDevRegion) * (dev->nregions - i - 1));
+            memset(dev->regions + dev->nregions - 1, 0, sizeof(VuDevRegion));
+            DPRINT("Successfully removed a region\n");
+            dev->nregions--;
+
+            found = true;
         }
     }
 
-    if (i < dev->nregions) {
-        /*
-         * Shift all affected entries by 1 to close the hole at index i and
-         * zero out the last entry.
-         */
-        memmove(dev->regions + i, dev->regions + i + 1,
-               sizeof(VuDevRegion) * (dev->nregions - i - 1));
-        memset(dev->regions + dev->nregions - 1, 0,
-               sizeof(VuDevRegion));
-        DPRINT("Successfully removed a region\n");
-        dev->nregions--;
+    if (found) {
         vmsg_set_reply_u64(vmsg, 0);
     } else {
         vu_panic(dev, "Specified region not found\n");
-- 
2.20.1

Re: [RFC 0/5] Clean up error handling in libvhost-user memory mapping

[Raphael Norwitz]

Ping

On Wed, Dec 15, 2021 at 10:29:46PM +0000, Raphael Norwitz wrote:
> Hey Stefan, Marc-Andre, MST, David -
> 
> As promised here is a series cleaning up error handling in the
> libvhost-user memory mapping path. Most of these cleanups are
> straightforward and have been discussed on the mailing list in threads
> [1] and [2]. Hopefully there is nothing super controversial in the first
> 4 patches.
> 
> I am concerned about patch 5 “libvhost-user: handle removal of
> identical regions”. From my reading of Stefan's comments in [1], the
> proposal seemed to be to remove any duplicate regions. I’d prefer to
> prevent duplicate regions from being added in the first place. Thoughts? 
> 
> [1] https://lore.kernel.org/qemu-devel/20211018143319.GA11006@raphael-debian-dev/
> [2] https://lore.kernel.org/qemu-devel/9391f500-70be-26cf-bcfc-591d3ee84d4e@redhat.com/
> 
> Sorry for the delay,
> Raphael
> 
> David Hildenbrand (1):
>   libvhost-user: Simplify VHOST_USER_REM_MEM_REG
> 
> Raphael Norwitz (4):
>   libvhost-user: Add vu_rem_mem_reg input validation
>   libvhost-user: Add vu_add_mem_reg input validation
>   libvhost-user: prevent over-running max RAM slots
>   libvhost-user: handle removal of identical regions
> 
>  subprojects/libvhost-user/libvhost-user.c | 52 +++++++++++++++--------
>  1 file changed, 34 insertions(+), 18 deletions(-)
> 
> -- 
> 2.20.1

Re: [RFC 1/5] libvhost-user: Add vu_rem_mem_reg input validation

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

On Wed, Dec 15, 2021 at 10:29:48PM +0000, Raphael Norwitz wrote:
> Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
> ---
>  subprojects/libvhost-user/libvhost-user.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
> index 787f4d2d4f..573212a83b 100644
> --- a/subprojects/libvhost-user/libvhost-user.c
> +++ b/subprojects/libvhost-user/libvhost-user.c
> @@ -801,6 +801,12 @@ vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
>      VuDevRegion shadow_regions[VHOST_USER_MAX_RAM_SLOTS] = {};
>      VhostUserMemoryRegion m = vmsg->payload.memreg.region, *msg_region = &m;
>  
> +    if (vmsg->fd_num != 1 ||
> +        vmsg->size != sizeof(vmsg->payload.memreg)) {
> +        vu_panic(dev, "VHOST_USER_REM_MEM_REG received multiple regions");
> +        return true;

Most vu_panic() callers return false to indicate that a reply does not
need to be sent. When the return value is true vu_dispatch() sends a
response, which we don't want.

Note that vu_dispatch() returns true (success) when the message handler
function returns false. The success/failure behavior should probably be
separated from the reply_requested behavior :(.

Anyway, returning false is probably appropriate here.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC 2/5] libvhost-user: Add vu_add_mem_reg input validation

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

On Wed, Dec 15, 2021 at 10:29:51PM +0000, Raphael Norwitz wrote:
> Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
> ---
>  subprojects/libvhost-user/libvhost-user.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
> index 573212a83b..80ef335254 100644
> --- a/subprojects/libvhost-user/libvhost-user.c
> +++ b/subprojects/libvhost-user/libvhost-user.c
> @@ -690,6 +690,12 @@ vu_add_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
>      VuDevRegion *dev_region = &dev->regions[dev->nregions];
>      void *mmap_addr;
>  
> +    if (vmsg->fd_num != 1 ||
> +        vmsg->size != sizeof(vmsg->payload.memreg)) {
> +        vu_panic(dev, "VHOST_USER_REM_MEM_REG received multiple regions");
> +        return true;

return false - we don't want a reply to be sent.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC 3/5] libvhost-user: Simplify VHOST_USER_REM_MEM_REG

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

On Wed, Dec 15, 2021 at 10:29:52PM +0000, Raphael Norwitz wrote:
> From: David Hildenbrand <david@redhat.com>
> 
> Let's avoid having to manually copy all elements. Copy only the ones
> necessary to close the hole and perform the operation in-place without
> a second array.
> 
> Signed-off-by: David Hildenbrand <david@redhat.com>
> Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
> ---
>  subprojects/libvhost-user/libvhost-user.c | 30 +++++++++++------------
>  1 file changed, 14 insertions(+), 16 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC 4/5] libvhost-user: prevent over-running max RAM slots

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

On Wed, Dec 15, 2021 at 10:29:54PM +0000, Raphael Norwitz wrote:
> When VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS support was added to
> libvhost-user, no guardrails were added to protect against QEMU
> attempting to hot-add too many RAM slots to a VM with a libvhost-user
> based backed attached.
> 
> This change adds the missing error handling by introducing a check on
> the number of RAM slots the device has available before proceeding to
> process the VHOST_USER_ADD_MEM_REG message.
> 
> Suggested-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
> ---
>  subprojects/libvhost-user/libvhost-user.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
> index 714cc7e08b..74a9980194 100644
> --- a/subprojects/libvhost-user/libvhost-user.c
> +++ b/subprojects/libvhost-user/libvhost-user.c
> @@ -690,6 +690,11 @@ vu_add_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
>      VuDevRegion *dev_region = &dev->regions[dev->nregions];
>      void *mmap_addr;
>  
> +    if (dev->nregions == VHOST_USER_MAX_RAM_SLOTS) {
> +        vu_panic(dev, "No free ram slots available");
> +        return true;

return false

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC 5/5] libvhost-user: handle removal of identical regions

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1780 bytes --]

On Wed, Dec 15, 2021 at 10:29:55PM +0000, Raphael Norwitz wrote:
> diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
> index 74a9980194..2f465a4f0e 100644
> --- a/subprojects/libvhost-user/libvhost-user.c
> +++ b/subprojects/libvhost-user/libvhost-user.c
> @@ -809,6 +809,7 @@ static bool
>  vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
>      VhostUserMemoryRegion m = vmsg->payload.memreg.region, *msg_region = &m;
>      int i;
> +    bool found = false;
>  
>      if (vmsg->fd_num != 1 ||
>          vmsg->size != sizeof(vmsg->payload.memreg)) {
> @@ -831,25 +832,25 @@ vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
>              VuDevRegion *r = &dev->regions[i];
>              void *m = (void *) (uintptr_t) r->mmap_addr;
>  
> -            if (m) {
> +            if (m && !found) {
>                  munmap(m, r->size + r->mmap_offset);
>              }

Why is only the first region unmapped? My interpretation of
vu_add_mem_reg() is that it mmaps duplicate regions to unique mmap_addr
addresses, so we need to munmap each of them.

>  
> -            break;
> +            /*
> +             * Shift all affected entries by 1 to close the hole at index i and
> +             * zero out the last entry.
> +             */
> +            memmove(dev->regions + i, dev->regions + i + 1,
> +                    sizeof(VuDevRegion) * (dev->nregions - i - 1));
> +            memset(dev->regions + dev->nregions - 1, 0, sizeof(VuDevRegion));
> +            DPRINT("Successfully removed a region\n");
> +            dev->nregions--;
> +
> +            found = true;
>          }

i-- is missing. dev->regions[] has been shortened so we need to check
the same element again.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC 1/5] libvhost-user: Add vu_rem_mem_reg input validation

[Raphael Norwitz]

On Wed, Jan 05, 2022 at 11:00:35AM +0000, Stefan Hajnoczi wrote:
> On Wed, Dec 15, 2021 at 10:29:48PM +0000, Raphael Norwitz wrote:
> > Signed-off-by: Raphael Norwitz <raphael.norwitz@nutanix.com>
> > ---
> >  subprojects/libvhost-user/libvhost-user.c | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
> > index 787f4d2d4f..573212a83b 100644
> > --- a/subprojects/libvhost-user/libvhost-user.c
> > +++ b/subprojects/libvhost-user/libvhost-user.c
> > @@ -801,6 +801,12 @@ vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
> >      VuDevRegion shadow_regions[VHOST_USER_MAX_RAM_SLOTS] = {};
> >      VhostUserMemoryRegion m = vmsg->payload.memreg.region, *msg_region = &m;
> >  
> > +    if (vmsg->fd_num != 1 ||
> > +        vmsg->size != sizeof(vmsg->payload.memreg)) {
> > +        vu_panic(dev, "VHOST_USER_REM_MEM_REG received multiple regions");
> > +        return true;
> 
> Most vu_panic() callers return false to indicate that a reply does not
> need to be sent. When the return value is true vu_dispatch() sends a
> response, which we don't want.
> 
> Note that vu_dispatch() returns true (success) when the message handler
> function returns false. The success/failure behavior should probably be
> separated from the reply_requested behavior :(.
> 
> Anyway, returning false is probably appropriate here.
>

Ack - I'll fix it in all the patches.

> Stefan

Re: [RFC 5/5] libvhost-user: handle removal of identical regions

[Raphael Norwitz]

On Wed, Jan 05, 2022 at 11:18:52AM +0000, Stefan Hajnoczi wrote:
> On Wed, Dec 15, 2021 at 10:29:55PM +0000, Raphael Norwitz wrote:
> > diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
> > index 74a9980194..2f465a4f0e 100644
> > --- a/subprojects/libvhost-user/libvhost-user.c
> > +++ b/subprojects/libvhost-user/libvhost-user.c
> > @@ -809,6 +809,7 @@ static bool
> >  vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
> >      VhostUserMemoryRegion m = vmsg->payload.memreg.region, *msg_region = &m;
> >      int i;
> > +    bool found = false;
> >  
> >      if (vmsg->fd_num != 1 ||
> >          vmsg->size != sizeof(vmsg->payload.memreg)) {
> > @@ -831,25 +832,25 @@ vu_rem_mem_reg(VuDev *dev, VhostUserMsg *vmsg) {
> >              VuDevRegion *r = &dev->regions[i];
> >              void *m = (void *) (uintptr_t) r->mmap_addr;
> >  
> > -            if (m) {
> > +            if (m && !found) {
> >                  munmap(m, r->size + r->mmap_offset);
> >              }
> 
> Why is only the first region unmapped? My interpretation of
> vu_add_mem_reg() is that it mmaps duplicate regions to unique mmap_addr
> addresses, so we need to munmap each of them.

I agree - I will remove the found check here.

>
> >  
> > -            break;
> > +            /*
> > +             * Shift all affected entries by 1 to close the hole at index i and
> > +             * zero out the last entry.
> > +             */
> > +            memmove(dev->regions + i, dev->regions + i + 1,
> > +                    sizeof(VuDevRegion) * (dev->nregions - i - 1));
> > +            memset(dev->regions + dev->nregions - 1, 0, sizeof(VuDevRegion));
> > +            DPRINT("Successfully removed a region\n");
> > +            dev->nregions--;
> > +
> > +            found = true;
> >          }
> 
> i-- is missing. dev->regions[] has been shortened so we need to check
> the same element again.

Ack