All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] migration: fix the memory overwriting risk in add_to_iovec
@ 2021-06-22 11:15 Lin Feng
  2021-06-23  1:48 ` [v2] " Lin Feng
  2021-06-23  1:51 ` [v3] " Lin Feng
  0 siblings, 2 replies; 11+ messages in thread
From: Lin Feng @ 2021-06-22 11:15 UTC (permalink / raw)
  To: qemu-devel; +Cc: wangxinxin.wang, Feng Lin, dgilbert

From: Feng Lin <linfeng23@huawei.com>

When testing migration, a Segmentation fault qemu core is generated.
0  error_free (err=0x1)
1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
2  0x00007f8b8516d59a in migrate_fd_cleanup (s=s@entry=0x55e06c0e1ef0)
3  0x00007f8b8516d66c in migrate_fd_cleanup_bh (opaque=0x55e06c0e1ef0)
4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
7  0x00007f8b866bdba4 in g_main_context_dispatch ()
8  0x00007f8b8626cde9 in glib_pollfds_poll ()
9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized out>)
10 0x00007f8b8626cffd in main_loop_wait (nonblocking=nonblocking@entry=0)
11 0x00007f8b862ef01f in main_loop ()
Using gdb print the struct QEMUFile f = {
  ...,
  iovcnt = 65, last_error = 21984,
  last_error_obj = 0x1, shutdown = true
}
Well iovcnt is overflow, because the max size of MAX_IOV_SIZE is 64.
struct QEMUFile {
    ...;
    struct iovec iov[MAX_IOV_SIZE];
    unsigned int iovcnt;
    int last_error;
    Error *last_error_obj;
    bool shutdown;
};
iovcnt and last_error is overwrited by add_to_iovec().
Right now, add_to_iovec() increase iovcnt before check the limit.
And it seems that add_to_iovec() assumes that iovcnt will set to zero
in qemu_fflush(). But qemu_fflush() will directly return when f->shutdown
is true.

The situation may occur when libvirtd restart during migration, after
f->shutdown is set, before calling qemu_file_set_error() in
qemu_file_shutdown().

So the safiest way is checking the iovcnt before increasing it.

Signed-off-by: Feng Lin <linfeng23@huawei.com>
---
 migration/qemu-file.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index d6e03dbc0e..3dde1a193c 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -419,8 +419,10 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
         if (may_free) {
             set_bit(f->iovcnt, f->may_free);
         }
-        f->iov[f->iovcnt].iov_base = (uint8_t *)buf;
-        f->iov[f->iovcnt++].iov_len = size;
+        if (f->iovcnt < MAX_IOV_SIZE) {
+            f->iov[f->iovcnt].iov_base = (uint8_t *)buf;
+            f->iov[f->iovcnt++].iov_len = size;
+        }
     }
 
     if (f->iovcnt >= MAX_IOV_SIZE) {
-- 
2.23.0



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [v2] migration: fix the memory overwriting risk in add_to_iovec
  2021-06-22 11:15 [PATCH] migration: fix the memory overwriting risk in add_to_iovec Lin Feng
@ 2021-06-23  1:48 ` Lin Feng
  2021-06-23  1:51 ` [v3] " Lin Feng
  1 sibling, 0 replies; 11+ messages in thread
From: Lin Feng @ 2021-06-23  1:48 UTC (permalink / raw)
  To: qemu-devel; +Cc: wangxinxin.wang, Feng Lin, dgilbert

From: Feng Lin <linfeng23@huawei.com>

When testing migration, a Segmentation fault qemu core is generated.
0  error_free (err=0x1)
1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
2  0x00007f8b8516d59a in migrate_fd_cleanup (s=s@entry=0x55e06c0e1ef0)
3  0x00007f8b8516d66c in migrate_fd_cleanup_bh (opaque=0x55e06c0e1ef0)
4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
7  0x00007f8b866bdba4 in g_main_context_dispatch ()
8  0x00007f8b8626cde9 in glib_pollfds_poll ()
9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized out>)
10 0x00007f8b8626cffd in main_loop_wait (nonblocking=nonblocking@entry=0)
11 0x00007f8b862ef01f in main_loop ()
Using gdb print the struct QEMUFile f = {
  ...,
  iovcnt = 65, last_error = 21984,
  last_error_obj = 0x1, shutdown = true
}
Well iovcnt is overflow, because the max size of MAX_IOV_SIZE is 64.
struct QEMUFile {
    ...;
    struct iovec iov[MAX_IOV_SIZE];
    unsigned int iovcnt;
    int last_error;
    Error *last_error_obj;
    bool shutdown;
};
iovcnt and last_error is overwrited by add_to_iovec().
Right now, add_to_iovec() increase iovcnt before check the limit.
And it seems that add_to_iovec() assumes that iovcnt will set to zero
in qemu_fflush(). But qemu_fflush() will directly return when f->shutdown
is true.

The situation may occur when libvirtd restart during migration, after
f->shutdown is set, before calling qemu_file_set_error() in
qemu_file_shutdown().

So the safiest way is checking the iovcnt before increasing it.

Signed-off-by: Feng Lin <linfeng23@huawei.com>
---
 migration/qemu-file.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index d6e03dbc0e..f6486cf7bc 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -416,6 +416,9 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
     {
         f->iov[f->iovcnt - 1].iov_len += size;
     } else {
+        if (f->iovcnt >= MAX_IOV_SIZE) {
+            goto fflush;
+        }
         if (may_free) {
             set_bit(f->iovcnt, f->may_free);
         }
@@ -423,12 +426,12 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
         f->iov[f->iovcnt++].iov_len = size;
     }
 
-    if (f->iovcnt >= MAX_IOV_SIZE) {
-        qemu_fflush(f);
-        return 1;
+    if (f->iovcnt < MAX_IOV_SIZE) {
+        return 0;
     }
-
-    return 0;
+fflush:
+    qemu_fflush();
+    return 1;
 }
 
 static void add_buf_to_iovec(QEMUFile *f, size_t len)
-- 
2.23.0



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [v3] migration: fix the memory overwriting risk in add_to_iovec
  2021-06-22 11:15 [PATCH] migration: fix the memory overwriting risk in add_to_iovec Lin Feng
  2021-06-23  1:48 ` [v2] " Lin Feng
@ 2021-06-23  1:51 ` Lin Feng
  2021-06-24 18:58   ` Dr. David Alan Gilbert
  2021-06-25  6:21   ` [v4] " Lin Feng
  1 sibling, 2 replies; 11+ messages in thread
From: Lin Feng @ 2021-06-23  1:51 UTC (permalink / raw)
  To: qemu-devel; +Cc: wangxinxin.wang, Feng Lin, dgilbert

From: Feng Lin <linfeng23@huawei.com>

When testing migration, a Segmentation fault qemu core is generated.
0  error_free (err=0x1)
1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
2  0x00007f8b8516d59a in migrate_fd_cleanup (s=s@entry=0x55e06c0e1ef0)
3  0x00007f8b8516d66c in migrate_fd_cleanup_bh (opaque=0x55e06c0e1ef0)
4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
7  0x00007f8b866bdba4 in g_main_context_dispatch ()
8  0x00007f8b8626cde9 in glib_pollfds_poll ()
9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized out>)
10 0x00007f8b8626cffd in main_loop_wait (nonblocking=nonblocking@entry=0)
11 0x00007f8b862ef01f in main_loop ()
Using gdb print the struct QEMUFile f = {
  ...,
  iovcnt = 65, last_error = 21984,
  last_error_obj = 0x1, shutdown = true
}
Well iovcnt is overflow, because the max size of MAX_IOV_SIZE is 64.
struct QEMUFile {
    ...;
    struct iovec iov[MAX_IOV_SIZE];
    unsigned int iovcnt;
    int last_error;
    Error *last_error_obj;
    bool shutdown;
};
iovcnt and last_error is overwrited by add_to_iovec().
Right now, add_to_iovec() increase iovcnt before check the limit.
And it seems that add_to_iovec() assumes that iovcnt will set to zero
in qemu_fflush(). But qemu_fflush() will directly return when f->shutdown
is true.

The situation may occur when libvirtd restart during migration, after
f->shutdown is set, before calling qemu_file_set_error() in
qemu_file_shutdown().

So the safiest way is checking the iovcnt before increasing it.

Signed-off-by: Feng Lin <linfeng23@huawei.com>
---
 migration/qemu-file.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index d6e03dbc0e..f6486cf7bc 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -416,6 +416,9 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
     {
         f->iov[f->iovcnt - 1].iov_len += size;
     } else {
+        if (f->iovcnt >= MAX_IOV_SIZE) {
+            goto fflush;
+        }
         if (may_free) {
             set_bit(f->iovcnt, f->may_free);
         }
@@ -423,12 +426,12 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
         f->iov[f->iovcnt++].iov_len = size;
     }
 
-    if (f->iovcnt >= MAX_IOV_SIZE) {
-        qemu_fflush(f);
-        return 1;
+    if (f->iovcnt < MAX_IOV_SIZE) {
+        return 0;
     }
-
-    return 0;
+fflush:
+    qemu_fflush(f);
+    return 1;
 }
 
 static void add_buf_to_iovec(QEMUFile *f, size_t len)
-- 
2.23.0



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* Re: [v3] migration: fix the memory overwriting risk in add_to_iovec
  2021-06-23  1:51 ` [v3] " Lin Feng
@ 2021-06-24 18:58   ` Dr. David Alan Gilbert
  2021-06-25  2:01     ` linfeng (M)
  2021-06-25  6:21   ` [v4] " Lin Feng
  1 sibling, 1 reply; 11+ messages in thread
From: Dr. David Alan Gilbert @ 2021-06-24 18:58 UTC (permalink / raw)
  To: Lin Feng; +Cc: wangxinxin.wang, qemu-devel

* Lin Feng (linfeng23@huawei.com) wrote:
> From: Feng Lin <linfeng23@huawei.com>
> 
> When testing migration, a Segmentation fault qemu core is generated.
> 0  error_free (err=0x1)
> 1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
> 2  0x00007f8b8516d59a in migrate_fd_cleanup (s=s@entry=0x55e06c0e1ef0)
> 3  0x00007f8b8516d66c in migrate_fd_cleanup_bh (opaque=0x55e06c0e1ef0)
> 4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
> 5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
> 6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
> 7  0x00007f8b866bdba4 in g_main_context_dispatch ()
> 8  0x00007f8b8626cde9 in glib_pollfds_poll ()
> 9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized out>)
> 10 0x00007f8b8626cffd in main_loop_wait (nonblocking=nonblocking@entry=0)
> 11 0x00007f8b862ef01f in main_loop ()
> Using gdb print the struct QEMUFile f = {
>   ...,
>   iovcnt = 65, last_error = 21984,
>   last_error_obj = 0x1, shutdown = true
> }
> Well iovcnt is overflow, because the max size of MAX_IOV_SIZE is 64.
> struct QEMUFile {
>     ...;
>     struct iovec iov[MAX_IOV_SIZE];
>     unsigned int iovcnt;
>     int last_error;
>     Error *last_error_obj;
>     bool shutdown;
> };
> iovcnt and last_error is overwrited by add_to_iovec().
> Right now, add_to_iovec() increase iovcnt before check the limit.
> And it seems that add_to_iovec() assumes that iovcnt will set to zero
> in qemu_fflush(). But qemu_fflush() will directly return when f->shutdown
> is true.
> 
> The situation may occur when libvirtd restart during migration, after
> f->shutdown is set, before calling qemu_file_set_error() in
> qemu_file_shutdown().
> 
> So the safiest way is checking the iovcnt before increasing it.
> 
> Signed-off-by: Feng Lin <linfeng23@huawei.com>
> ---
>  migration/qemu-file.c | 13 ++++++++-----
>  1 file changed, 8 insertions(+), 5 deletions(-)
> 
> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> index d6e03dbc0e..f6486cf7bc 100644
> --- a/migration/qemu-file.c
> +++ b/migration/qemu-file.c
> @@ -416,6 +416,9 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>      {
>          f->iov[f->iovcnt - 1].iov_len += size;
>      } else {
> +        if (f->iovcnt >= MAX_IOV_SIZE) {
> +            goto fflush;
> +        }

Why call qemu_fflush in this case?
If I understand what you're saying, then we only get to here if a
previous qemu_fflush has failed, so this should fail as well?

How about, something like:
    if (f->iovcnt >= MAX_IOV_SIZE) {
        /* Should only happen if a previous fflush failed */
        assert(f->shutdown || !qemu_file_is_writeable(f));
        return 1;
    }

?

Dave

>          if (may_free) {
>              set_bit(f->iovcnt, f->may_free);
>          }
> @@ -423,12 +426,12 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>          f->iov[f->iovcnt++].iov_len = size;
>      }
>  
> -    if (f->iovcnt >= MAX_IOV_SIZE) {
> -        qemu_fflush(f);
> -        return 1;
> +    if (f->iovcnt < MAX_IOV_SIZE) {
> +        return 0;
>      }
> -
> -    return 0;
> +fflush:
> +    qemu_fflush(f);
> +    return 1;
>  }
>  
>  static void add_buf_to_iovec(QEMUFile *f, size_t len)
> -- 
> 2.23.0
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

* RE: [v3] migration: fix the memory overwriting risk in add_to_iovec
  2021-06-24 18:58   ` Dr. David Alan Gilbert
@ 2021-06-25  2:01     ` linfeng (M)
  0 siblings, 0 replies; 11+ messages in thread
From: linfeng (M) @ 2021-06-25  2:01 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: Wangxin (Alexander), qemu-devel

* Dr. David Alan Gilbert(mailto:dgilbert@redhat.com) wrote:
> * Lin Feng (linfeng23@huawei.com) wrote:
> > From: Feng Lin <linfeng23@huawei.com>
> >
> > When testing migration, a Segmentation fault qemu core is generated.
> > 0  error_free (err=0x1)
> > 1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
> > 2  0x00007f8b8516d59a in migrate_fd_cleanup (s=s@entry=0x55e06c0e1ef0)
> > 3  0x00007f8b8516d66c in migrate_fd_cleanup_bh (opaque=0x55e06c0e1ef0)
> > 4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
> > 5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
> > 6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>,
> user_data=<optimized out>)
> > 7  0x00007f8b866bdba4 in g_main_context_dispatch ()
> > 8  0x00007f8b8626cde9 in glib_pollfds_poll ()
> > 9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized out>)
> > 10 0x00007f8b8626cffd in main_loop_wait (nonblocking=nonblocking@entry=0)
> > 11 0x00007f8b862ef01f in main_loop ()
> > Using gdb print the struct QEMUFile f = {
> >   ...,
> >   iovcnt = 65, last_error = 21984,
> >   last_error_obj = 0x1, shutdown = true
> > }
> > Well iovcnt is overflow, because the max size of MAX_IOV_SIZE is 64.
> > struct QEMUFile {
> >     ...;
> >     struct iovec iov[MAX_IOV_SIZE];
> >     unsigned int iovcnt;
> >     int last_error;
> >     Error *last_error_obj;
> >     bool shutdown;
> > };
> > iovcnt and last_error is overwrited by add_to_iovec().
> > Right now, add_to_iovec() increase iovcnt before check the limit.
> > And it seems that add_to_iovec() assumes that iovcnt will set to zero
> > in qemu_fflush(). But qemu_fflush() will directly return when f->shutdown
> > is true.
> >
> > The situation may occur when libvirtd restart during migration, after
> > f->shutdown is set, before calling qemu_file_set_error() in
> > qemu_file_shutdown().
> >
> > So the safiest way is checking the iovcnt before increasing it.
> >
> > Signed-off-by: Feng Lin <linfeng23@huawei.com>
> > ---
> >  migration/qemu-file.c | 13 ++++++++-----
> >  1 file changed, 8 insertions(+), 5 deletions(-)
> >
> > diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> > index d6e03dbc0e..f6486cf7bc 100644
> > --- a/migration/qemu-file.c
> > +++ b/migration/qemu-file.c
> > @@ -416,6 +416,9 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> >      {
> >          f->iov[f->iovcnt - 1].iov_len += size;
> >      } else {
> > +        if (f->iovcnt >= MAX_IOV_SIZE) {
> > +            goto fflush;
> > +        }
> 
> Why call qemu_fflush in this case?
> If I understand what you're saying, then we only get to here if a
> previous qemu_fflush has failed, so this should fail as well?
Yes, that's what I mean.

> 
> How about, something like:
>     if (f->iovcnt >= MAX_IOV_SIZE) {
>         /* Should only happen if a previous fflush failed */
>         assert(f->shutdown || !qemu_file_is_writeable(f));
>         return 1;
>     }
> 
> ?
At first, I'm just thinking that overwriting requires qemu_fflush to reset iovcnt and do not consider
the possibility of packet loss caused by other exceptions. It makes more sense to make an assertion
here. Thank you for your suggestions.
> 
> Dave
> 
> >          if (may_free) {
> >              set_bit(f->iovcnt, f->may_free);
> >          }
> > @@ -423,12 +426,12 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> >          f->iov[f->iovcnt++].iov_len = size;
> >      }
> >
> > -    if (f->iovcnt >= MAX_IOV_SIZE) {
> > -        qemu_fflush(f);
> > -        return 1;
> > +    if (f->iovcnt < MAX_IOV_SIZE) {
> > +        return 0;
> >      }
> > -
> > -    return 0;
> > +fflush:
> > +    qemu_fflush(f);
> > +    return 1;
> >  }
> >
> >  static void add_buf_to_iovec(QEMUFile *f, size_t len)
> > --
> > 2.23.0
> >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

* [v4] migration: fix the memory overwriting risk in add_to_iovec
  2021-06-23  1:51 ` [v3] " Lin Feng
  2021-06-24 18:58   ` Dr. David Alan Gilbert
@ 2021-06-25  6:21   ` Lin Feng
  2021-06-30 10:14     ` Dr. David Alan Gilbert
  2021-06-30 17:02     ` Dr. David Alan Gilbert
  1 sibling, 2 replies; 11+ messages in thread
From: Lin Feng @ 2021-06-25  6:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: wangxinxin.wang, Feng Lin, dgilbert

From: Feng Lin <linfeng23@huawei.com>

When testing migration, a Segmentation fault qemu core is generated.
0  error_free (err=0x1)
1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
2  0x00007f8b8516d59a in migrate_fd_cleanup (s=s@entry=0x55e06c0e1ef0)
3  0x00007f8b8516d66c in migrate_fd_cleanup_bh (opaque=0x55e06c0e1ef0)
4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
7  0x00007f8b866bdba4 in g_main_context_dispatch ()
8  0x00007f8b8626cde9 in glib_pollfds_poll ()
9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized out>)
10 0x00007f8b8626cffd in main_loop_wait (nonblocking=nonblocking@entry=0)
11 0x00007f8b862ef01f in main_loop ()
Using gdb print the struct QEMUFile f = {
  ...,
  iovcnt = 65, last_error = 21984,
  last_error_obj = 0x1, shutdown = true
}
Well iovcnt is overflow, because the max size of MAX_IOV_SIZE is 64.
struct QEMUFile {
    ...;
    struct iovec iov[MAX_IOV_SIZE];
    unsigned int iovcnt;
    int last_error;
    Error *last_error_obj;
    bool shutdown;
};
iovcnt and last_error is overwrited by add_to_iovec().
Right now, add_to_iovec() increase iovcnt before check the limit.
And it seems that add_to_iovec() assumes that iovcnt will set to zero
in qemu_fflush(). But qemu_fflush() will directly return when f->shutdown
is true.

The situation may occur when libvirtd restart during migration, after
f->shutdown is set, before calling qemu_file_set_error() in
qemu_file_shutdown().

So the safiest way is checking the iovcnt before increasing it.

Signed-off-by: Feng Lin <linfeng23@huawei.com>
---
 migration/qemu-file.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index d6e03dbc0e..6879615197 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -416,6 +416,11 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
     {
         f->iov[f->iovcnt - 1].iov_len += size;
     } else {
+        if (f->iovcnt >= MAX_IOV_SIZE) {
+            /* Should only happen if a previous fflush failed */
+            assert(f->shutdown || !qemu_file_is_writeable(f));
+            return 1;
+        }
         if (may_free) {
             set_bit(f->iovcnt, f->may_free);
         }
-- 
2.23.0



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* Re: [v4] migration: fix the memory overwriting risk in add_to_iovec
  2021-06-25  6:21   ` [v4] " Lin Feng
@ 2021-06-30 10:14     ` Dr. David Alan Gilbert
  2021-06-30 17:02     ` Dr. David Alan Gilbert
  1 sibling, 0 replies; 11+ messages in thread
From: Dr. David Alan Gilbert @ 2021-06-30 10:14 UTC (permalink / raw)
  To: Lin Feng; +Cc: wangxinxin.wang, qemu-devel

* Lin Feng (linfeng23@huawei.com) wrote:
> From: Feng Lin <linfeng23@huawei.com>
> 
> When testing migration, a Segmentation fault qemu core is generated.
> 0  error_free (err=0x1)
> 1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
> 2  0x00007f8b8516d59a in migrate_fd_cleanup (s=s@entry=0x55e06c0e1ef0)
> 3  0x00007f8b8516d66c in migrate_fd_cleanup_bh (opaque=0x55e06c0e1ef0)
> 4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
> 5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
> 6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
> 7  0x00007f8b866bdba4 in g_main_context_dispatch ()
> 8  0x00007f8b8626cde9 in glib_pollfds_poll ()
> 9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized out>)
> 10 0x00007f8b8626cffd in main_loop_wait (nonblocking=nonblocking@entry=0)
> 11 0x00007f8b862ef01f in main_loop ()
> Using gdb print the struct QEMUFile f = {
>   ...,
>   iovcnt = 65, last_error = 21984,
>   last_error_obj = 0x1, shutdown = true
> }
> Well iovcnt is overflow, because the max size of MAX_IOV_SIZE is 64.
> struct QEMUFile {
>     ...;
>     struct iovec iov[MAX_IOV_SIZE];
>     unsigned int iovcnt;
>     int last_error;
>     Error *last_error_obj;
>     bool shutdown;
> };
> iovcnt and last_error is overwrited by add_to_iovec().
> Right now, add_to_iovec() increase iovcnt before check the limit.
> And it seems that add_to_iovec() assumes that iovcnt will set to zero
> in qemu_fflush(). But qemu_fflush() will directly return when f->shutdown
> is true.
> 
> The situation may occur when libvirtd restart during migration, after
> f->shutdown is set, before calling qemu_file_set_error() in
> qemu_file_shutdown().
> 
> So the safiest way is checking the iovcnt before increasing it.
> 
> Signed-off-by: Feng Lin <linfeng23@huawei.com>

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

> ---
>  migration/qemu-file.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> index d6e03dbc0e..6879615197 100644
> --- a/migration/qemu-file.c
> +++ b/migration/qemu-file.c
> @@ -416,6 +416,11 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>      {
>          f->iov[f->iovcnt - 1].iov_len += size;
>      } else {
> +        if (f->iovcnt >= MAX_IOV_SIZE) {
> +            /* Should only happen if a previous fflush failed */
> +            assert(f->shutdown || !qemu_file_is_writeable(f));
> +            return 1;
> +        }
>          if (may_free) {
>              set_bit(f->iovcnt, f->may_free);
>          }
> -- 
> 2.23.0
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [v4] migration: fix the memory overwriting risk in add_to_iovec
  2021-06-25  6:21   ` [v4] " Lin Feng
  2021-06-30 10:14     ` Dr. David Alan Gilbert
@ 2021-06-30 17:02     ` Dr. David Alan Gilbert
  2021-07-01 11:23       ` Dr. David Alan Gilbert
  1 sibling, 1 reply; 11+ messages in thread
From: Dr. David Alan Gilbert @ 2021-06-30 17:02 UTC (permalink / raw)
  To: Lin Feng; +Cc: wangxinxin.wang, qemu-devel

* Lin Feng (linfeng23@huawei.com) wrote:
> From: Feng Lin <linfeng23@huawei.com>
> 
> When testing migration, a Segmentation fault qemu core is generated.
> 0  error_free (err=0x1)
> 1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
> 2  0x00007f8b8516d59a in migrate_fd_cleanup (s=s@entry=0x55e06c0e1ef0)
> 3  0x00007f8b8516d66c in migrate_fd_cleanup_bh (opaque=0x55e06c0e1ef0)
> 4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
> 5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
> 6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
> 7  0x00007f8b866bdba4 in g_main_context_dispatch ()
> 8  0x00007f8b8626cde9 in glib_pollfds_poll ()
> 9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized out>)
> 10 0x00007f8b8626cffd in main_loop_wait (nonblocking=nonblocking@entry=0)
> 11 0x00007f8b862ef01f in main_loop ()
> Using gdb print the struct QEMUFile f = {
>   ...,
>   iovcnt = 65, last_error = 21984,
>   last_error_obj = 0x1, shutdown = true
> }
> Well iovcnt is overflow, because the max size of MAX_IOV_SIZE is 64.
> struct QEMUFile {
>     ...;
>     struct iovec iov[MAX_IOV_SIZE];
>     unsigned int iovcnt;
>     int last_error;
>     Error *last_error_obj;
>     bool shutdown;
> };
> iovcnt and last_error is overwrited by add_to_iovec().
> Right now, add_to_iovec() increase iovcnt before check the limit.
> And it seems that add_to_iovec() assumes that iovcnt will set to zero
> in qemu_fflush(). But qemu_fflush() will directly return when f->shutdown
> is true.
> 
> The situation may occur when libvirtd restart during migration, after
> f->shutdown is set, before calling qemu_file_set_error() in
> qemu_file_shutdown().
> 
> So the safiest way is checking the iovcnt before increasing it.
> 
> Signed-off-by: Feng Lin <linfeng23@huawei.com>

Queued
> ---
>  migration/qemu-file.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> index d6e03dbc0e..6879615197 100644
> --- a/migration/qemu-file.c
> +++ b/migration/qemu-file.c
> @@ -416,6 +416,11 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
>      {
>          f->iov[f->iovcnt - 1].iov_len += size;
>      } else {
> +        if (f->iovcnt >= MAX_IOV_SIZE) {
> +            /* Should only happen if a previous fflush failed */
> +            assert(f->shutdown || !qemu_file_is_writeable(f));
> +            return 1;
> +        }
>          if (may_free) {
>              set_bit(f->iovcnt, f->may_free);
>          }
> -- 
> 2.23.0
> 
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [v4] migration: fix the memory overwriting risk in add_to_iovec
  2021-06-30 17:02     ` Dr. David Alan Gilbert
@ 2021-07-01 11:23       ` Dr. David Alan Gilbert
  2021-07-02  1:17         ` linfeng (M)
  0 siblings, 1 reply; 11+ messages in thread
From: Dr. David Alan Gilbert @ 2021-07-01 11:23 UTC (permalink / raw)
  To: Lin Feng; +Cc: wangxinxin.wang, qemu-devel

* Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> * Lin Feng (linfeng23@huawei.com) wrote:
> > From: Feng Lin <linfeng23@huawei.com>
> > 
> > When testing migration, a Segmentation fault qemu core is generated.
> > 0  error_free (err=0x1)
> > 1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
> > 2  0x00007f8b8516d59a in migrate_fd_cleanup (s=s@entry=0x55e06c0e1ef0)
> > 3  0x00007f8b8516d66c in migrate_fd_cleanup_bh (opaque=0x55e06c0e1ef0)
> > 4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
> > 5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
> > 6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
> > 7  0x00007f8b866bdba4 in g_main_context_dispatch ()
> > 8  0x00007f8b8626cde9 in glib_pollfds_poll ()
> > 9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized out>)
> > 10 0x00007f8b8626cffd in main_loop_wait (nonblocking=nonblocking@entry=0)
> > 11 0x00007f8b862ef01f in main_loop ()
> > Using gdb print the struct QEMUFile f = {
> >   ...,
> >   iovcnt = 65, last_error = 21984,
> >   last_error_obj = 0x1, shutdown = true
> > }
> > Well iovcnt is overflow, because the max size of MAX_IOV_SIZE is 64.
> > struct QEMUFile {
> >     ...;
> >     struct iovec iov[MAX_IOV_SIZE];
> >     unsigned int iovcnt;
> >     int last_error;
> >     Error *last_error_obj;
> >     bool shutdown;
> > };
> > iovcnt and last_error is overwrited by add_to_iovec().
> > Right now, add_to_iovec() increase iovcnt before check the limit.
> > And it seems that add_to_iovec() assumes that iovcnt will set to zero
> > in qemu_fflush(). But qemu_fflush() will directly return when f->shutdown
> > is true.
> > 
> > The situation may occur when libvirtd restart during migration, after
> > f->shutdown is set, before calling qemu_file_set_error() in
> > qemu_file_shutdown().
> > 
> > So the safiest way is checking the iovcnt before increasing it.
> > 
> > Signed-off-by: Feng Lin <linfeng23@huawei.com>
> 
> Queued

Hmm this didn't actually build because that function is actually
misnamed 'qemu_file_is_writable' (no e!);  I've fixed that, but can you
just reconfirm that you've tested this fixes your original problem?

Dave

> > ---
> >  migration/qemu-file.c | 5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> > index d6e03dbc0e..6879615197 100644
> > --- a/migration/qemu-file.c
> > +++ b/migration/qemu-file.c
> > @@ -416,6 +416,11 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> >      {
> >          f->iov[f->iovcnt - 1].iov_len += size;
> >      } else {
> > +        if (f->iovcnt >= MAX_IOV_SIZE) {
> > +            /* Should only happen if a previous fflush failed */
> > +            assert(f->shutdown || !qemu_file_is_writeable(f));
> > +            return 1;
> > +        }
> >          if (may_free) {
> >              set_bit(f->iovcnt, f->may_free);
> >          }
> > -- 
> > 2.23.0
> > 
> > 
> -- 
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

* RE: [v4] migration: fix the memory overwriting risk in add_to_iovec
  2021-07-01 11:23       ` Dr. David Alan Gilbert
@ 2021-07-02  1:17         ` linfeng (M)
  2021-07-05  8:19           ` Dr. David Alan Gilbert
  0 siblings, 1 reply; 11+ messages in thread
From: linfeng (M) @ 2021-07-02  1:17 UTC (permalink / raw)
  To: Dr. David Alan Gilbert; +Cc: Wangxin (Alexander), qemu-devel


* Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> Subject: Re: [v4] migration: fix the memory overwriting risk in add_to_iovec
> 
> * Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> > * Lin Feng (linfeng23@huawei.com) wrote:
> > > From: Feng Lin <linfeng23@huawei.com>
> > >
> > > When testing migration, a Segmentation fault qemu core is generated.
> > > 0  error_free (err=0x1)
> > > 1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
> > > 2  0x00007f8b8516d59a in migrate_fd_cleanup
> > > (s=s@entry=0x55e06c0e1ef0)
> > > 3  0x00007f8b8516d66c in migrate_fd_cleanup_bh
> > > (opaque=0x55e06c0e1ef0)
> > > 4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
> > > 5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
> > > 6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>,
> > > callback=<optimized out>, user_data=<optimized out>)
> > > 7  0x00007f8b866bdba4 in g_main_context_dispatch ()
> > > 8  0x00007f8b8626cde9 in glib_pollfds_poll ()
> > > 9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized
> > > out>)
> > > 10 0x00007f8b8626cffd in main_loop_wait
> > > (nonblocking=nonblocking@entry=0)
> > > 11 0x00007f8b862ef01f in main_loop () Using gdb print the struct
> > > QEMUFile f = {
> > >   ...,
> > >   iovcnt = 65, last_error = 21984,
> > >   last_error_obj = 0x1, shutdown = true } Well iovcnt is overflow,
> > > because the max size of MAX_IOV_SIZE is 64.
> > > struct QEMUFile {
> > >     ...;
> > >     struct iovec iov[MAX_IOV_SIZE];
> > >     unsigned int iovcnt;
> > >     int last_error;
> > >     Error *last_error_obj;
> > >     bool shutdown;
> > > };
> > > iovcnt and last_error is overwrited by add_to_iovec().
> > > Right now, add_to_iovec() increase iovcnt before check the limit.
> > > And it seems that add_to_iovec() assumes that iovcnt will set to
> > > zero in qemu_fflush(). But qemu_fflush() will directly return when
> > > f->shutdown is true.
> > >
> > > The situation may occur when libvirtd restart during migration,
> > > after
> > > f->shutdown is set, before calling qemu_file_set_error() in
> > > qemu_file_shutdown().
> > >
> > > So the safiest way is checking the iovcnt before increasing it.
> > >
> > > Signed-off-by: Feng Lin <linfeng23@huawei.com>
> >
> > Queued
> 
> Hmm this didn't actually build because that function is actually misnamed 'qemu_file_is_writable' (no e!);
> I've fixed that, but can you just reconfirm that you've tested this fixes your original problem?
Sorry for that rookie mistake. I have tested it again with gdb-fault injection. It can fix my original problem.
Thanks for helping me complete my first qemu patch submission. Really helped a lot.
> 
> Dave
> 
> > > ---
> > >  migration/qemu-file.c | 5 +++++
> > >  1 file changed, 5 insertions(+)
> > >
> > > diff --git a/migration/qemu-file.c b/migration/qemu-file.c index
> > > d6e03dbc0e..6879615197 100644
> > > --- a/migration/qemu-file.c
> > > +++ b/migration/qemu-file.c
> > > @@ -416,6 +416,11 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> > >      {
> > >          f->iov[f->iovcnt - 1].iov_len += size;
> > >      } else {
> > > +        if (f->iovcnt >= MAX_IOV_SIZE) {
> > > +            /* Should only happen if a previous fflush failed */
> > > +            assert(f->shutdown || !qemu_file_is_writeable(f));
> > > +            return 1;
> > > +        }
> > >          if (may_free) {
> > >              set_bit(f->iovcnt, f->may_free);
> > >          }
> > > --
> > > 2.23.0
> > >
> > >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [v4] migration: fix the memory overwriting risk in add_to_iovec
  2021-07-02  1:17         ` linfeng (M)
@ 2021-07-05  8:19           ` Dr. David Alan Gilbert
  0 siblings, 0 replies; 11+ messages in thread
From: Dr. David Alan Gilbert @ 2021-07-05  8:19 UTC (permalink / raw)
  To: linfeng (M); +Cc: Wangxin (Alexander), qemu-devel

* linfeng (M) (linfeng23@huawei.com) wrote:
> 
> * Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> > Subject: Re: [v4] migration: fix the memory overwriting risk in add_to_iovec
> > 
> > * Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> > > * Lin Feng (linfeng23@huawei.com) wrote:
> > > > From: Feng Lin <linfeng23@huawei.com>
> > > >
> > > > When testing migration, a Segmentation fault qemu core is generated.
> > > > 0  error_free (err=0x1)
> > > > 1  0x00007f8b862df647 in qemu_fclose (f=f@entry=0x55e06c247640)
> > > > 2  0x00007f8b8516d59a in migrate_fd_cleanup
> > > > (s=s@entry=0x55e06c0e1ef0)
> > > > 3  0x00007f8b8516d66c in migrate_fd_cleanup_bh
> > > > (opaque=0x55e06c0e1ef0)
> > > > 4  0x00007f8b8626a47f in aio_bh_poll (ctx=ctx@entry=0x55e06b5a16d0)
> > > > 5  0x00007f8b8626e71f in aio_dispatch (ctx=0x55e06b5a16d0)
> > > > 6  0x00007f8b8626a33d in aio_ctx_dispatch (source=<optimized out>,
> > > > callback=<optimized out>, user_data=<optimized out>)
> > > > 7  0x00007f8b866bdba4 in g_main_context_dispatch ()
> > > > 8  0x00007f8b8626cde9 in glib_pollfds_poll ()
> > > > 9  0x00007f8b8626ce62 in os_host_main_loop_wait (timeout=<optimized
> > > > out>)
> > > > 10 0x00007f8b8626cffd in main_loop_wait
> > > > (nonblocking=nonblocking@entry=0)
> > > > 11 0x00007f8b862ef01f in main_loop () Using gdb print the struct
> > > > QEMUFile f = {
> > > >   ...,
> > > >   iovcnt = 65, last_error = 21984,
> > > >   last_error_obj = 0x1, shutdown = true } Well iovcnt is overflow,
> > > > because the max size of MAX_IOV_SIZE is 64.
> > > > struct QEMUFile {
> > > >     ...;
> > > >     struct iovec iov[MAX_IOV_SIZE];
> > > >     unsigned int iovcnt;
> > > >     int last_error;
> > > >     Error *last_error_obj;
> > > >     bool shutdown;
> > > > };
> > > > iovcnt and last_error is overwrited by add_to_iovec().
> > > > Right now, add_to_iovec() increase iovcnt before check the limit.
> > > > And it seems that add_to_iovec() assumes that iovcnt will set to
> > > > zero in qemu_fflush(). But qemu_fflush() will directly return when
> > > > f->shutdown is true.
> > > >
> > > > The situation may occur when libvirtd restart during migration,
> > > > after
> > > > f->shutdown is set, before calling qemu_file_set_error() in
> > > > qemu_file_shutdown().
> > > >
> > > > So the safiest way is checking the iovcnt before increasing it.
> > > >
> > > > Signed-off-by: Feng Lin <linfeng23@huawei.com>
> > >
> > > Queued
> > 
> > Hmm this didn't actually build because that function is actually misnamed 'qemu_file_is_writable' (no e!);
> > I've fixed that, but can you just reconfirm that you've tested this fixes your original problem?
> Sorry for that rookie mistake. I have tested it again with gdb-fault injection. It can fix my original problem.
> Thanks for helping me complete my first qemu patch submission. Really helped a lot.

Thanks for retesting.

Dave

> > 
> > Dave
> > 
> > > > ---
> > > >  migration/qemu-file.c | 5 +++++
> > > >  1 file changed, 5 insertions(+)
> > > >
> > > > diff --git a/migration/qemu-file.c b/migration/qemu-file.c index
> > > > d6e03dbc0e..6879615197 100644
> > > > --- a/migration/qemu-file.c
> > > > +++ b/migration/qemu-file.c
> > > > @@ -416,6 +416,11 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
> > > >      {
> > > >          f->iov[f->iovcnt - 1].iov_len += size;
> > > >      } else {
> > > > +        if (f->iovcnt >= MAX_IOV_SIZE) {
> > > > +            /* Should only happen if a previous fflush failed */
> > > > +            assert(f->shutdown || !qemu_file_is_writeable(f));
> > > > +            return 1;
> > > > +        }
> > > >          if (may_free) {
> > > >              set_bit(f->iovcnt, f->may_free);
> > > >          }
> > > > --
> > > > 2.23.0
> > > >
> > > >
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2021-07-05  8:21 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-22 11:15 [PATCH] migration: fix the memory overwriting risk in add_to_iovec Lin Feng
2021-06-23  1:48 ` [v2] " Lin Feng
2021-06-23  1:51 ` [v3] " Lin Feng
2021-06-24 18:58   ` Dr. David Alan Gilbert
2021-06-25  2:01     ` linfeng (M)
2021-06-25  6:21   ` [v4] " Lin Feng
2021-06-30 10:14     ` Dr. David Alan Gilbert
2021-06-30 17:02     ` Dr. David Alan Gilbert
2021-07-01 11:23       ` Dr. David Alan Gilbert
2021-07-02  1:17         ` linfeng (M)
2021-07-05  8:19           ` Dr. David Alan Gilbert

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.