[Qemu-devel] [PATCH] gdb command: qemu iohandlers

[Qemu-devel] [PATCH] gdb command: qemu iohandlers

[Dr. David Alan Gilbert (git)]

From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>

Add a gdb command to print the current set of IOHandlers and
if one of them is a thread yielding for data print the backtrace.

Useful for debugging why an incoming migration has stalled, e.g.

  ----
  {fd_read = 0x7fd4c8e40d00 <fd_coroutine_enter>, fd_write = 0x0, opaque =
  0x7fd4b8bfeb00, next = {le_next = 0x7fd4cac81b00, le_prev =
  0x7fd4c93d2bd0 <io_handlers>}, fd = 22, pollfds_idx = 0, deleted =
  false}
  #0  qemu_coroutine_switch (from_=from_@entry=0x7fd4cbb33c00,
  to_=to_@entry=0x7fd4c8b13a90,
      action=action@entry=COROUTINE_YIELD) at coroutine-ucontext.c:177
  #1  0x00007fd4c8e40507 in qemu_coroutine_yield () at
  qemu-coroutine.c:145
  #2  0x00007fd4c8e40e75 in yield_until_fd_readable (fd=22) at
  qemu-coroutine-io.c:90
  #3  0x00007fd4c8df347f in qemu_rdma_block_for_wrid
  (rdma=rdma@entry=0x7fd4b8c7e010,
      wrid_requested=wrid_requested@entry=2000,
  byte_len=byte_len@entry=0x0) at migration/rdma.c:1510
  #4  0x00007fd4c8df388f in qemu_rdma_post_send_control
  (rdma=rdma@entry=0x7fd4b8c7e010, buf=buf@entry=0x0,
      head=head@entry=0x7fd4b8bfed00) at migration/rdma.c:1608
  #5  0x00007fd4c8df4b8e in qemu_rdma_exchange_recv (rdma=0x7fd4b8c7e010,
  head=0x7fd4b8bfed50, expecting=3)
      at migration/rdma.c:1814
  #6  0x00007fd4c8df5089 in qemu_rdma_get_buffer (opaque=0x7fd4cba34950,
  buf=0x7fd4cc24fd20 "TR\022",
      pos=<optimized out>, size=32768) at migration/rdma.c:2611
  #7  0x00007fd4c8df000d in qemu_fill_buffer (f=f@entry=0x7fd4cc24fcf0) at
  migration/qemu-file.c:214
  #8  0x00007fd4c8df08d4 in qemu_peek_byte (f=f@entry=0x7fd4cc24fcf0,
  offset=offset@entry=0)
      at migration/qemu-file.c:447
  #9  0x00007fd4c8c2cad1 in qemu_loadvm_state (f=f@entry=0x7fd4cc24fcf0)
  at /root/qemu-world3/migration/savevm.c:1128
  #10 0x00007fd4c8ded895 in process_incoming_migration_co
  (opaque=0x7fd4cc24fcf0) at migration/migration.c:145
  #11 0x00007fd4c8e4112a in coroutine_trampoline (i0=<optimized out>,
  i1=<optimized out>) at coroutine-ucontext.c:80
  #12 0x00007fd4c453e0f0 in ?? () from /lib64/libc.so.6
  #13 0x00007ffee263e870 in ?? ()
  #14 0x0000000000000000 in ?? ()
  ----
  {fd_read = 0x7fd4c8e3ace0 <sigfd_handler>, fd_write = 0x0, opaque = 0x5,
  next = {le_next = 0x0, le_prev = 0x7fd4cb78a3d8}, fd = 5, pollfds_idx =
  1, deleted = false}

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
 scripts/qemu-gdb.py | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/scripts/qemu-gdb.py b/scripts/qemu-gdb.py
index 6c7f4fb..2670625 100644
--- a/scripts/qemu-gdb.py
+++ b/scripts/qemu-gdb.py
@@ -102,6 +102,30 @@ class CoroutineCommand(gdb.Command):
         coroutine_pointer = gdb.parse_and_eval(argv[0]).cast(gdb.lookup_type('CoroutineUContext').pointer())
         bt_jmpbuf(coroutine_pointer['env']['__jmpbuf'])
 
+class IOhandlersCommand(gdb.Command):
+    '''Display a backtrace for io handlers'''
+    def __init__(self):
+        gdb.Command.__init__(self, 'qemu iohandlers', gdb.COMMAND_DATA,
+                             gdb.COMPLETE_NONE)
+
+    def invoke(self, arg, from_tty):
+        cur = gdb.parse_and_eval('io_handlers.lh_first')
+        sym_fd_coroutine_enter = gdb.parse_and_eval('fd_coroutine_enter')
+
+        while not isnull(cur):
+            entry = cur.dereference()
+            gdb.write('----\n%s\n' % entry)
+
+            # For functions we know, extract the coroutine pointer and backtrace
+            if cur['fd_read'] == sym_fd_coroutine_enter:
+                # yield_until_fd_readable
+                coptr = (cur['opaque'].cast(gdb.lookup_type('FDYieldUntilData').pointer()))['co']
+                coptr = coptr.cast(gdb.lookup_type('CoroutineUContext').pointer())
+                bt_jmpbuf(coptr['env']['__jmpbuf'])
+
+            cur = cur['next']['le_next']
+        gdb.write('----\n')
+
 class MtreeCommand(gdb.Command):
     '''Display the memory tree hierarchy'''
     def __init__(self):
@@ -161,4 +185,5 @@ class MtreeCommand(gdb.Command):
 
 QemuCommand()
 CoroutineCommand()
+IOhandlersCommand()
 MtreeCommand()
-- 
2.4.3

Re: [Qemu-devel] [PATCH] gdb command: qemu iohandlers

[Peter Maydell]

On 23 June 2015 at 15:43, Dr. David Alan Gilbert (git)
<dgilbert@redhat.com> wrote:
> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>
> Add a gdb command to print the current set of IOHandlers and
> if one of them is a thread yielding for data print the backtrace.
>
> Useful for debugging why an incoming migration has stalled, e.g.

I'd rather we did the split of qemu-gdb.py into one subfile
per command before adding new ones: see patches 1 and 2 in
this series:
https://lists.gnu.org/archive/html/qemu-devel/2015-05/msg02961.html

thanks
-- PMM

Re: [Qemu-devel] [PATCH] gdb command: qemu iohandlers

[Dr. David Alan Gilbert]

* Peter Maydell (peter.maydell@linaro.org) wrote:
> On 23 June 2015 at 15:43, Dr. David Alan Gilbert (git)
> <dgilbert@redhat.com> wrote:
> > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> >
> > Add a gdb command to print the current set of IOHandlers and
> > if one of them is a thread yielding for data print the backtrace.
> >
> > Useful for debugging why an incoming migration has stalled, e.g.
> 
> I'd rather we did the split of qemu-gdb.py into one subfile
> per command before adding new ones: see patches 1 and 2 in
> this series:
> https://lists.gnu.org/archive/html/qemu-devel/2015-05/msg02961.html

No problem - I'm happy to rework after that split goes in.
Note I'm using the bt_jmpbuf function which that patchset puts in coroutine.py;
maybe there would be a better way to share it.

Dave

> 
> thanks
> -- PMM
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH] gdb command: qemu iohandlers

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3599 bytes --]

On Tue, Jun 23, 2015 at 03:43:53PM +0100, Dr. David Alan Gilbert (git) wrote:
> From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> 
> Add a gdb command to print the current set of IOHandlers and
> if one of them is a thread yielding for data print the backtrace.
> 
> Useful for debugging why an incoming migration has stalled, e.g.
> 
>   ----
>   {fd_read = 0x7fd4c8e40d00 <fd_coroutine_enter>, fd_write = 0x0, opaque =
>   0x7fd4b8bfeb00, next = {le_next = 0x7fd4cac81b00, le_prev =
>   0x7fd4c93d2bd0 <io_handlers>}, fd = 22, pollfds_idx = 0, deleted =
>   false}
>   #0  qemu_coroutine_switch (from_=from_@entry=0x7fd4cbb33c00,
>   to_=to_@entry=0x7fd4c8b13a90,
>       action=action@entry=COROUTINE_YIELD) at coroutine-ucontext.c:177
>   #1  0x00007fd4c8e40507 in qemu_coroutine_yield () at
>   qemu-coroutine.c:145
>   #2  0x00007fd4c8e40e75 in yield_until_fd_readable (fd=22) at
>   qemu-coroutine-io.c:90
>   #3  0x00007fd4c8df347f in qemu_rdma_block_for_wrid
>   (rdma=rdma@entry=0x7fd4b8c7e010,
>       wrid_requested=wrid_requested@entry=2000,
>   byte_len=byte_len@entry=0x0) at migration/rdma.c:1510
>   #4  0x00007fd4c8df388f in qemu_rdma_post_send_control
>   (rdma=rdma@entry=0x7fd4b8c7e010, buf=buf@entry=0x0,
>       head=head@entry=0x7fd4b8bfed00) at migration/rdma.c:1608
>   #5  0x00007fd4c8df4b8e in qemu_rdma_exchange_recv (rdma=0x7fd4b8c7e010,
>   head=0x7fd4b8bfed50, expecting=3)
>       at migration/rdma.c:1814
>   #6  0x00007fd4c8df5089 in qemu_rdma_get_buffer (opaque=0x7fd4cba34950,
>   buf=0x7fd4cc24fd20 "TR\022",
>       pos=<optimized out>, size=32768) at migration/rdma.c:2611
>   #7  0x00007fd4c8df000d in qemu_fill_buffer (f=f@entry=0x7fd4cc24fcf0) at
>   migration/qemu-file.c:214
>   #8  0x00007fd4c8df08d4 in qemu_peek_byte (f=f@entry=0x7fd4cc24fcf0,
>   offset=offset@entry=0)
>       at migration/qemu-file.c:447
>   #9  0x00007fd4c8c2cad1 in qemu_loadvm_state (f=f@entry=0x7fd4cc24fcf0)
>   at /root/qemu-world3/migration/savevm.c:1128
>   #10 0x00007fd4c8ded895 in process_incoming_migration_co
>   (opaque=0x7fd4cc24fcf0) at migration/migration.c:145
>   #11 0x00007fd4c8e4112a in coroutine_trampoline (i0=<optimized out>,
>   i1=<optimized out>) at coroutine-ucontext.c:80
>   #12 0x00007fd4c453e0f0 in ?? () from /lib64/libc.so.6
>   #13 0x00007ffee263e870 in ?? ()
>   #14 0x0000000000000000 in ?? ()
>   ----
>   {fd_read = 0x7fd4c8e3ace0 <sigfd_handler>, fd_write = 0x0, opaque = 0x5,
>   next = {le_next = 0x0, le_prev = 0x7fd4cb78a3d8}, fd = 5, pollfds_idx =
>   1, deleted = false}
> 
> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
>  scripts/qemu-gdb.py | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
> 
> diff --git a/scripts/qemu-gdb.py b/scripts/qemu-gdb.py
> index 6c7f4fb..2670625 100644
> --- a/scripts/qemu-gdb.py
> +++ b/scripts/qemu-gdb.py
> @@ -102,6 +102,30 @@ class CoroutineCommand(gdb.Command):
>          coroutine_pointer = gdb.parse_and_eval(argv[0]).cast(gdb.lookup_type('CoroutineUContext').pointer())
>          bt_jmpbuf(coroutine_pointer['env']['__jmpbuf'])
>  
> +class IOhandlersCommand(gdb.Command):
> +    '''Display a backtrace for io handlers'''
> +    def __init__(self):
> +        gdb.Command.__init__(self, 'qemu iohandlers', gdb.COMMAND_DATA,
> +                             gdb.COMPLETE_NONE)
> +
> +    def invoke(self, arg, from_tty):
> +        cur = gdb.parse_and_eval('io_handlers.lh_first')

Fam is getting rid of io_handlers and replacing it with an AioContext.

I have CCed him.

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH] gdb command: qemu iohandlers

[Fam Zheng]

On Wed, 06/24 16:19, Stefan Hajnoczi wrote:
> On Tue, Jun 23, 2015 at 03:43:53PM +0100, Dr. David Alan Gilbert (git) wrote:
> > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > 
> > Add a gdb command to print the current set of IOHandlers and
> > if one of them is a thread yielding for data print the backtrace.
> > 
> > Useful for debugging why an incoming migration has stalled, e.g.
> > 
> >   ----
> >   {fd_read = 0x7fd4c8e40d00 <fd_coroutine_enter>, fd_write = 0x0, opaque =
> >   0x7fd4b8bfeb00, next = {le_next = 0x7fd4cac81b00, le_prev =
> >   0x7fd4c93d2bd0 <io_handlers>}, fd = 22, pollfds_idx = 0, deleted =
> >   false}
> >   #0  qemu_coroutine_switch (from_=from_@entry=0x7fd4cbb33c00,
> >   to_=to_@entry=0x7fd4c8b13a90,
> >       action=action@entry=COROUTINE_YIELD) at coroutine-ucontext.c:177
> >   #1  0x00007fd4c8e40507 in qemu_coroutine_yield () at
> >   qemu-coroutine.c:145
> >   #2  0x00007fd4c8e40e75 in yield_until_fd_readable (fd=22) at
> >   qemu-coroutine-io.c:90
> >   #3  0x00007fd4c8df347f in qemu_rdma_block_for_wrid
> >   (rdma=rdma@entry=0x7fd4b8c7e010,
> >       wrid_requested=wrid_requested@entry=2000,
> >   byte_len=byte_len@entry=0x0) at migration/rdma.c:1510
> >   #4  0x00007fd4c8df388f in qemu_rdma_post_send_control
> >   (rdma=rdma@entry=0x7fd4b8c7e010, buf=buf@entry=0x0,
> >       head=head@entry=0x7fd4b8bfed00) at migration/rdma.c:1608
> >   #5  0x00007fd4c8df4b8e in qemu_rdma_exchange_recv (rdma=0x7fd4b8c7e010,
> >   head=0x7fd4b8bfed50, expecting=3)
> >       at migration/rdma.c:1814
> >   #6  0x00007fd4c8df5089 in qemu_rdma_get_buffer (opaque=0x7fd4cba34950,
> >   buf=0x7fd4cc24fd20 "TR\022",
> >       pos=<optimized out>, size=32768) at migration/rdma.c:2611
> >   #7  0x00007fd4c8df000d in qemu_fill_buffer (f=f@entry=0x7fd4cc24fcf0) at
> >   migration/qemu-file.c:214
> >   #8  0x00007fd4c8df08d4 in qemu_peek_byte (f=f@entry=0x7fd4cc24fcf0,
> >   offset=offset@entry=0)
> >       at migration/qemu-file.c:447
> >   #9  0x00007fd4c8c2cad1 in qemu_loadvm_state (f=f@entry=0x7fd4cc24fcf0)
> >   at /root/qemu-world3/migration/savevm.c:1128
> >   #10 0x00007fd4c8ded895 in process_incoming_migration_co
> >   (opaque=0x7fd4cc24fcf0) at migration/migration.c:145
> >   #11 0x00007fd4c8e4112a in coroutine_trampoline (i0=<optimized out>,
> >   i1=<optimized out>) at coroutine-ucontext.c:80
> >   #12 0x00007fd4c453e0f0 in ?? () from /lib64/libc.so.6
> >   #13 0x00007ffee263e870 in ?? ()
> >   #14 0x0000000000000000 in ?? ()
> >   ----
> >   {fd_read = 0x7fd4c8e3ace0 <sigfd_handler>, fd_write = 0x0, opaque = 0x5,
> >   next = {le_next = 0x0, le_prev = 0x7fd4cb78a3d8}, fd = 5, pollfds_idx =
> >   1, deleted = false}
> > 
> > Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> > ---
> >  scripts/qemu-gdb.py | 25 +++++++++++++++++++++++++
> >  1 file changed, 25 insertions(+)
> > 
> > diff --git a/scripts/qemu-gdb.py b/scripts/qemu-gdb.py
> > index 6c7f4fb..2670625 100644
> > --- a/scripts/qemu-gdb.py
> > +++ b/scripts/qemu-gdb.py
> > @@ -102,6 +102,30 @@ class CoroutineCommand(gdb.Command):
> >          coroutine_pointer = gdb.parse_and_eval(argv[0]).cast(gdb.lookup_type('CoroutineUContext').pointer())
> >          bt_jmpbuf(coroutine_pointer['env']['__jmpbuf'])
> >  
> > +class IOhandlersCommand(gdb.Command):
> > +    '''Display a backtrace for io handlers'''
> > +    def __init__(self):
> > +        gdb.Command.__init__(self, 'qemu iohandlers', gdb.COMMAND_DATA,
> > +                             gdb.COMPLETE_NONE)
> > +
> > +    def invoke(self, arg, from_tty):
> > +        cur = gdb.parse_and_eval('io_handlers.lh_first')
> 
> Fam is getting rid of io_handlers and replacing it with an AioContext.
> 

Yes, the series is

  [PATCH 0/9] slirp: iohandler: Rebase onto aio

Fam

Re: [Qemu-devel] [PATCH] gdb command: qemu iohandlers

[Dr. David Alan Gilbert]

* Fam Zheng (famz@redhat.com) wrote:
> On Wed, 06/24 16:19, Stefan Hajnoczi wrote:
> > On Tue, Jun 23, 2015 at 03:43:53PM +0100, Dr. David Alan Gilbert (git) wrote:
> > > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > 
> > > Add a gdb command to print the current set of IOHandlers and
> > > if one of them is a thread yielding for data print the backtrace.
> > > 
> > > Useful for debugging why an incoming migration has stalled, e.g.
> > > 
> > >   ----
> > >   {fd_read = 0x7fd4c8e40d00 <fd_coroutine_enter>, fd_write = 0x0, opaque =
> > >   0x7fd4b8bfeb00, next = {le_next = 0x7fd4cac81b00, le_prev =
> > >   0x7fd4c93d2bd0 <io_handlers>}, fd = 22, pollfds_idx = 0, deleted =
> > >   false}
> > >   #0  qemu_coroutine_switch (from_=from_@entry=0x7fd4cbb33c00,
> > >   to_=to_@entry=0x7fd4c8b13a90,
> > >       action=action@entry=COROUTINE_YIELD) at coroutine-ucontext.c:177
> > >   #1  0x00007fd4c8e40507 in qemu_coroutine_yield () at
> > >   qemu-coroutine.c:145
> > >   #2  0x00007fd4c8e40e75 in yield_until_fd_readable (fd=22) at
> > >   qemu-coroutine-io.c:90
> > >   #3  0x00007fd4c8df347f in qemu_rdma_block_for_wrid
> > >   (rdma=rdma@entry=0x7fd4b8c7e010,
> > >       wrid_requested=wrid_requested@entry=2000,
> > >   byte_len=byte_len@entry=0x0) at migration/rdma.c:1510
> > >   #4  0x00007fd4c8df388f in qemu_rdma_post_send_control
> > >   (rdma=rdma@entry=0x7fd4b8c7e010, buf=buf@entry=0x0,
> > >       head=head@entry=0x7fd4b8bfed00) at migration/rdma.c:1608
> > >   #5  0x00007fd4c8df4b8e in qemu_rdma_exchange_recv (rdma=0x7fd4b8c7e010,
> > >   head=0x7fd4b8bfed50, expecting=3)
> > >       at migration/rdma.c:1814
> > >   #6  0x00007fd4c8df5089 in qemu_rdma_get_buffer (opaque=0x7fd4cba34950,
> > >   buf=0x7fd4cc24fd20 "TR\022",
> > >       pos=<optimized out>, size=32768) at migration/rdma.c:2611
> > >   #7  0x00007fd4c8df000d in qemu_fill_buffer (f=f@entry=0x7fd4cc24fcf0) at
> > >   migration/qemu-file.c:214
> > >   #8  0x00007fd4c8df08d4 in qemu_peek_byte (f=f@entry=0x7fd4cc24fcf0,
> > >   offset=offset@entry=0)
> > >       at migration/qemu-file.c:447
> > >   #9  0x00007fd4c8c2cad1 in qemu_loadvm_state (f=f@entry=0x7fd4cc24fcf0)
> > >   at /root/qemu-world3/migration/savevm.c:1128
> > >   #10 0x00007fd4c8ded895 in process_incoming_migration_co
> > >   (opaque=0x7fd4cc24fcf0) at migration/migration.c:145
> > >   #11 0x00007fd4c8e4112a in coroutine_trampoline (i0=<optimized out>,
> > >   i1=<optimized out>) at coroutine-ucontext.c:80
> > >   #12 0x00007fd4c453e0f0 in ?? () from /lib64/libc.so.6
> > >   #13 0x00007ffee263e870 in ?? ()
> > >   #14 0x0000000000000000 in ?? ()
> > >   ----
> > >   {fd_read = 0x7fd4c8e3ace0 <sigfd_handler>, fd_write = 0x0, opaque = 0x5,
> > >   next = {le_next = 0x0, le_prev = 0x7fd4cb78a3d8}, fd = 5, pollfds_idx =
> > >   1, deleted = false}
> > > 
> > > Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> > > ---
> > >  scripts/qemu-gdb.py | 25 +++++++++++++++++++++++++
> > >  1 file changed, 25 insertions(+)
> > > 
> > > diff --git a/scripts/qemu-gdb.py b/scripts/qemu-gdb.py
> > > index 6c7f4fb..2670625 100644
> > > --- a/scripts/qemu-gdb.py
> > > +++ b/scripts/qemu-gdb.py
> > > @@ -102,6 +102,30 @@ class CoroutineCommand(gdb.Command):
> > >          coroutine_pointer = gdb.parse_and_eval(argv[0]).cast(gdb.lookup_type('CoroutineUContext').pointer())
> > >          bt_jmpbuf(coroutine_pointer['env']['__jmpbuf'])
> > >  
> > > +class IOhandlersCommand(gdb.Command):
> > > +    '''Display a backtrace for io handlers'''
> > > +    def __init__(self):
> > > +        gdb.Command.__init__(self, 'qemu iohandlers', gdb.COMMAND_DATA,
> > > +                             gdb.COMPLETE_NONE)
> > > +
> > > +    def invoke(self, arg, from_tty):
> > > +        cur = gdb.parse_and_eval('io_handlers.lh_first')
> > 
> > Fam is getting rid of io_handlers and replacing it with an AioContext.
> > 
> 
> Yes, the series is
> 
>   [PATCH 0/9] slirp: iohandler: Rebase onto aio

OK, I can rework after that goes in.
I guess the equivalent there is to dump the iohandler_ctx->aio_handlers list?

Dave

> 
> Fam
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH] gdb command: qemu iohandlers

[Fam Zheng]

On Thu, 06/25 10:08, Dr. David Alan Gilbert wrote:
> * Fam Zheng (famz@redhat.com) wrote:
> > On Wed, 06/24 16:19, Stefan Hajnoczi wrote:
> > > On Tue, Jun 23, 2015 at 03:43:53PM +0100, Dr. David Alan Gilbert (git) wrote:
> > > > From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> > > > 
> > > > Add a gdb command to print the current set of IOHandlers and
> > > > if one of them is a thread yielding for data print the backtrace.
> > > > 
> > > > Useful for debugging why an incoming migration has stalled, e.g.
> > > > 
> > > >   ----
> > > >   {fd_read = 0x7fd4c8e40d00 <fd_coroutine_enter>, fd_write = 0x0, opaque =
> > > >   0x7fd4b8bfeb00, next = {le_next = 0x7fd4cac81b00, le_prev =
> > > >   0x7fd4c93d2bd0 <io_handlers>}, fd = 22, pollfds_idx = 0, deleted =
> > > >   false}
> > > >   #0  qemu_coroutine_switch (from_=from_@entry=0x7fd4cbb33c00,
> > > >   to_=to_@entry=0x7fd4c8b13a90,
> > > >       action=action@entry=COROUTINE_YIELD) at coroutine-ucontext.c:177
> > > >   #1  0x00007fd4c8e40507 in qemu_coroutine_yield () at
> > > >   qemu-coroutine.c:145
> > > >   #2  0x00007fd4c8e40e75 in yield_until_fd_readable (fd=22) at
> > > >   qemu-coroutine-io.c:90
> > > >   #3  0x00007fd4c8df347f in qemu_rdma_block_for_wrid
> > > >   (rdma=rdma@entry=0x7fd4b8c7e010,
> > > >       wrid_requested=wrid_requested@entry=2000,
> > > >   byte_len=byte_len@entry=0x0) at migration/rdma.c:1510
> > > >   #4  0x00007fd4c8df388f in qemu_rdma_post_send_control
> > > >   (rdma=rdma@entry=0x7fd4b8c7e010, buf=buf@entry=0x0,
> > > >       head=head@entry=0x7fd4b8bfed00) at migration/rdma.c:1608
> > > >   #5  0x00007fd4c8df4b8e in qemu_rdma_exchange_recv (rdma=0x7fd4b8c7e010,
> > > >   head=0x7fd4b8bfed50, expecting=3)
> > > >       at migration/rdma.c:1814
> > > >   #6  0x00007fd4c8df5089 in qemu_rdma_get_buffer (opaque=0x7fd4cba34950,
> > > >   buf=0x7fd4cc24fd20 "TR\022",
> > > >       pos=<optimized out>, size=32768) at migration/rdma.c:2611
> > > >   #7  0x00007fd4c8df000d in qemu_fill_buffer (f=f@entry=0x7fd4cc24fcf0) at
> > > >   migration/qemu-file.c:214
> > > >   #8  0x00007fd4c8df08d4 in qemu_peek_byte (f=f@entry=0x7fd4cc24fcf0,
> > > >   offset=offset@entry=0)
> > > >       at migration/qemu-file.c:447
> > > >   #9  0x00007fd4c8c2cad1 in qemu_loadvm_state (f=f@entry=0x7fd4cc24fcf0)
> > > >   at /root/qemu-world3/migration/savevm.c:1128
> > > >   #10 0x00007fd4c8ded895 in process_incoming_migration_co
> > > >   (opaque=0x7fd4cc24fcf0) at migration/migration.c:145
> > > >   #11 0x00007fd4c8e4112a in coroutine_trampoline (i0=<optimized out>,
> > > >   i1=<optimized out>) at coroutine-ucontext.c:80
> > > >   #12 0x00007fd4c453e0f0 in ?? () from /lib64/libc.so.6
> > > >   #13 0x00007ffee263e870 in ?? ()
> > > >   #14 0x0000000000000000 in ?? ()
> > > >   ----
> > > >   {fd_read = 0x7fd4c8e3ace0 <sigfd_handler>, fd_write = 0x0, opaque = 0x5,
> > > >   next = {le_next = 0x0, le_prev = 0x7fd4cb78a3d8}, fd = 5, pollfds_idx =
> > > >   1, deleted = false}
> > > > 
> > > > Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> > > > ---
> > > >  scripts/qemu-gdb.py | 25 +++++++++++++++++++++++++
> > > >  1 file changed, 25 insertions(+)
> > > > 
> > > > diff --git a/scripts/qemu-gdb.py b/scripts/qemu-gdb.py
> > > > index 6c7f4fb..2670625 100644
> > > > --- a/scripts/qemu-gdb.py
> > > > +++ b/scripts/qemu-gdb.py
> > > > @@ -102,6 +102,30 @@ class CoroutineCommand(gdb.Command):
> > > >          coroutine_pointer = gdb.parse_and_eval(argv[0]).cast(gdb.lookup_type('CoroutineUContext').pointer())
> > > >          bt_jmpbuf(coroutine_pointer['env']['__jmpbuf'])
> > > >  
> > > > +class IOhandlersCommand(gdb.Command):
> > > > +    '''Display a backtrace for io handlers'''
> > > > +    def __init__(self):
> > > > +        gdb.Command.__init__(self, 'qemu iohandlers', gdb.COMMAND_DATA,
> > > > +                             gdb.COMPLETE_NONE)
> > > > +
> > > > +    def invoke(self, arg, from_tty):
> > > > +        cur = gdb.parse_and_eval('io_handlers.lh_first')
> > > 
> > > Fam is getting rid of io_handlers and replacing it with an AioContext.
> > > 
> > 
> > Yes, the series is
> > 
> >   [PATCH 0/9] slirp: iohandler: Rebase onto aio
> 
> OK, I can rework after that goes in.
> I guess the equivalent there is to dump the iohandler_ctx->aio_handlers list?

Yes, I think so.

Fam