Best approach for supporting snapshots for QEMU's gdbstub?

Best approach for supporting snapshots for QEMU's gdbstub?

[Alex Bennée]

Hi,

I've been playing around with QEMU's reverse debugging support which
I have working with Pavel's latest patches for supporting virtio with
record/replay. Once you get the right command line it works well enough
although currently each step backwards requires replaying the entire
execution history until you get to the right point.

QEMU can quite easily snapshot the entire VM state so I was looking to
see what the best way to integrate this would be. As far as I can tell
there are two interfaces gdb supports: bookmarks and checkpoints.

As far as I can tell bookmarks where added as part of GDB's reverse
debugging support but attempting to use them from the gdbstub reports:

  (gdb) bookmark
  You can't do that when your target is `remote'

so I guess that would need an extension to the stub protocol to support?

The other option I found was checkpoints which seem to predate support
for reverse debugging. However:

  (gdb) checkpoint
  checkpoint: can't find fork function in inferior.

I couldn't tell what feature needs to be negotiated but I suspect it's
something like fork-events if the checkpoint mechanism is designed for
user space with a fork/freeze approach.

We could of course just add a custom monitor command like the
qemu.sstep= command which could be used manually. However that would be
a QEMU gdbstub specific approach.

The other thing would be to be more intelligent on QEMU's side and save
snapshots each time we hit an event, for example each time we hit a
given breakpoint. However I do worry that might lead to snapshots
growing quite quickly.

Any thoughts/suggestions?

-- 
Alex Bennée

Re: Best approach for supporting snapshots for QEMU's gdbstub?

[Pavel Dovgalyuk]

On 14.05.2021 19:06, Alex Bennée wrote:
> Hi,
> 
> I've been playing around with QEMU's reverse debugging support which
> I have working with Pavel's latest patches for supporting virtio with
> record/replay. Once you get the right command line it works well enough
> although currently each step backwards requires replaying the entire
> execution history until you get to the right point.
> 
> QEMU can quite easily snapshot the entire VM state so I was looking to
> see what the best way to integrate this would be. As far as I can tell
> there are two interfaces gdb supports: bookmarks and checkpoints.
> 
> As far as I can tell bookmarks where added as part of GDB's reverse
> debugging support but attempting to use them from the gdbstub reports:
> 
>    (gdb) bookmark
>    You can't do that when your target is `remote'
> 
> so I guess that would need an extension to the stub protocol to support?
> 
> The other option I found was checkpoints which seem to predate support
> for reverse debugging. However:
> 
>    (gdb) checkpoint
>    checkpoint: can't find fork function in inferior.
> 
> I couldn't tell what feature needs to be negotiated but I suspect it's
> something like fork-events if the checkpoint mechanism is designed for
> user space with a fork/freeze approach.
> 
> We could of course just add a custom monitor command like the
> qemu.sstep= command which could be used manually. However that would be
> a QEMU gdbstub specific approach.

For now you can just use 'monitor savevm sn1' in gdb.
But something like 'bookmark' seems more convenient.

> The other thing would be to be more intelligent on QEMU's side and save
> snapshots each time we hit an event, for example each time we hit a
> given breakpoint. However I do worry that might lead to snapshots
> growing quite quickly.
> 
> Any thoughts/suggestions?
> 

Re: Best approach for supporting snapshots for QEMU's gdbstub?

[Luis Machado]

Hi,

On 5/14/21 1:06 PM, Alex Bennée wrote:
> Hi,
> 
> I've been playing around with QEMU's reverse debugging support which
> I have working with Pavel's latest patches for supporting virtio with
> record/replay. Once you get the right command line it works well enough
> although currently each step backwards requires replaying the entire
> execution history until you get to the right point.
> 
> QEMU can quite easily snapshot the entire VM state so I was looking to
> see what the best way to integrate this would be. As far as I can tell
> there are two interfaces gdb supports: bookmarks and checkpoints.
> 
> As far as I can tell bookmarks where added as part of GDB's reverse
> debugging support but attempting to use them from the gdbstub reports:
> 
>    (gdb) bookmark
>    You can't do that when your target is `remote'
> 
> so I guess that would need an extension to the stub protocol to support?
> 

Right. We don't support reverse step/next/continue for remote targets. I 
think this would be the most appropriate way to implement this feature 
in GDB. But it is not trivial.

> The other option I found was checkpoints which seem to predate support
> for reverse debugging. However:
> 
>    (gdb) checkpoint
>    checkpoint: can't find fork function in inferior.
> 
> I couldn't tell what feature needs to be negotiated but I suspect it's
> something like fork-events if the checkpoint mechanism is designed for
> user space with a fork/freeze approach.

Checkpoints are an old mechanism for saving a snapshot from a process, 
but they don't support threaded inferiors and they need to be able to 
call fork.

> 
> We could of course just add a custom monitor command like the
> qemu.sstep= command which could be used manually. However that would be
> a QEMU gdbstub specific approach.

That would be an easy and quick way to allow GDB to control things in 
QEMU, but I wouldn't say it is the best. Monitor commands are basically 
a bypass of the RSP where GDB sends/receives commands to/from the remote 
target.

> 
> The other thing would be to be more intelligent on QEMU's side and save
> snapshots each time we hit an event, for example each time we hit a
> given breakpoint. However I do worry that might lead to snapshots
> growing quite quickly.

GDB would need to be aware of such snapshots for them to be useful. 
Otherwise GDB wouldn't be able to use them to restore state.

> 
> Any thoughts/suggestions?
> 

Re: Best approach for supporting snapshots for QEMU's gdbstub?

[Alex Bennée]

Luis Machado <luis.machado@linaro.org> writes:

> Hi,
>
> On 5/14/21 1:06 PM, Alex Bennée wrote:
>> Hi,
>> I've been playing around with QEMU's reverse debugging support which
>> I have working with Pavel's latest patches for supporting virtio with
>> record/replay. Once you get the right command line it works well enough
>> although currently each step backwards requires replaying the entire
>> execution history until you get to the right point.
>> QEMU can quite easily snapshot the entire VM state so I was looking
>> to
>> see what the best way to integrate this would be. As far as I can tell
>> there are two interfaces gdb supports: bookmarks and checkpoints.
>> As far as I can tell bookmarks where added as part of GDB's reverse
>> debugging support but attempting to use them from the gdbstub reports:
>>    (gdb) bookmark
>>    You can't do that when your target is `remote'
>> so I guess that would need an extension to the stub protocol to
>> support?
>> 
>
> Right. We don't support reverse step/next/continue for remote targets.
> I think this would be the most appropriate way to implement this
> feature in GDB. But it is not trivial.

You do because ";ReverseStep+;ReverseContinue+" is part of the gdbstub
negotiation handshake.

Out of interest how is rr implemented? It presents a gdb interface so I
thought it was some implemented using some remote magic.

<snip>

>> We could of course just add a custom monitor command like the
>> qemu.sstep= command which could be used manually. However that would be
>> a QEMU gdbstub specific approach.
>
> That would be an easy and quick way to allow GDB to control things in
> QEMU, but I wouldn't say it is the best. Monitor commands are
> basically a bypass of the RSP where GDB sends/receives commands
> to/from the remote target.

We have some underlying commands we can set via the monitor including:

  monitor info replay
  monitor replay_seek <N>
  monitor replay_break <N>

>
>> The other thing would be to be more intelligent on QEMU's side and
>> save
>> snapshots each time we hit an event, for example each time we hit a
>> given breakpoint. However I do worry that might lead to snapshots
>> growing quite quickly.
>
> GDB would need to be aware of such snapshots for them to be useful.
> Otherwise GDB wouldn't be able to use them to restore state.

What does GDB need to know about them? Does it include something like
the icount at a particular point.

I'm curious at how a break and reverse-continue is meant to work if that
breakpoint is hit multiple times from the start of a run. You need to
know if the last time you hit a particular breakpoint was in fact the
last time before where the user was when they hit reverse-continue.

>
>> Any thoughts/suggestions?
>> 


-- 
Alex Bennée

Re: Best approach for supporting snapshots for QEMU's gdbstub?

[Luis Machado]

On 5/17/21 2:27 PM, Alex Bennée wrote:
> 
> Luis Machado <luis.machado@linaro.org> writes:
> 
>> Hi,
>>
>> On 5/14/21 1:06 PM, Alex Bennée wrote:
>>> Hi,
>>> I've been playing around with QEMU's reverse debugging support which
>>> I have working with Pavel's latest patches for supporting virtio with
>>> record/replay. Once you get the right command line it works well enough
>>> although currently each step backwards requires replaying the entire
>>> execution history until you get to the right point.
>>> QEMU can quite easily snapshot the entire VM state so I was looking
>>> to
>>> see what the best way to integrate this would be. As far as I can tell
>>> there are two interfaces gdb supports: bookmarks and checkpoints.
>>> As far as I can tell bookmarks where added as part of GDB's reverse
>>> debugging support but attempting to use them from the gdbstub reports:
>>>     (gdb) bookmark
>>>     You can't do that when your target is `remote'
>>> so I guess that would need an extension to the stub protocol to
>>> support?
>>>
>>
>> Right. We don't support reverse step/next/continue for remote targets.
>> I think this would be the most appropriate way to implement this
>> feature in GDB. But it is not trivial.
> 
> You do because ";ReverseStep+;ReverseContinue+" is part of the gdbstub
> negotiation handshake.

Interesting... I was looking at the vCont; packets for inferior 
movement. The regular c/C/s/S packet are deprecated and vCont; 
equivalents should be used instead.

It seems the reverse continue (bc) and reverse step (bs) packets can be 
used, but they are not vCont packets.

That's confusing. I suppose nobody took the time to implement bc/bs 
equivalents for vCont.

> 
> Out of interest how is rr implemented? It presents a gdb interface so I
> thought it was some implemented using some remote magic.

I don't know. I have never used rr.

> 
> <snip>
> 
>>> We could of course just add a custom monitor command like the
>>> qemu.sstep= command which could be used manually. However that would be
>>> a QEMU gdbstub specific approach.
>>
>> That would be an easy and quick way to allow GDB to control things in
>> QEMU, but I wouldn't say it is the best. Monitor commands are
>> basically a bypass of the RSP where GDB sends/receives commands
>> to/from the remote target.
> 
> We have some underlying commands we can set via the monitor including:
> 
>    monitor info replay
>    monitor replay_seek <N>
>    monitor replay_break <N>
> 
>>
>>> The other thing would be to be more intelligent on QEMU's side and
>>> save
>>> snapshots each time we hit an event, for example each time we hit a
>>> given breakpoint. However I do worry that might lead to snapshots
>>> growing quite quickly.
>>
>> GDB would need to be aware of such snapshots for them to be useful.
>> Otherwise GDB wouldn't be able to use them to restore state.
> 
> What does GDB need to know about them? Does it include something like
> the icount at a particular point.

GDB needs to know they exist so the user can choose to go back to such 
snapshots. I haven't dealt with remote reverse execution 
implementations, but if this information can be exposed to

> 
> I'm curious at how a break and reverse-continue is meant to work if that
> breakpoint is hit multiple times from the start of a run. You need to
> know if the last time you hit a particular breakpoint was in fact the
> last time before where the user was when they hit reverse-continue.

When you have record/replay on, there is no real "continue". GDB will 
instruction-step everything and will record register values and memory 
changes.

When you reverse instruction-step, GDB will restore the state for the 
previous snapshot. When you reverse continue, GDB will do the same and 
will move the state backwards snapshot by snapshot.

It is not very efficient.

So, in that sense, GDB will hit all of the breakpoints again. It doesn't 
keep track of how many times the breakpoint was hit. It only keeps track 
of how many instructions were recorded and what register/memory changes 
happened.

If you hit an instruction that GDB doesn't know how to calculate 
register/memory changes for, it will stop dead on its tracks. In that 
sense, it is also not very easy to maintain and takes a lot of 
instruction-parsing to work correctly.

No wonder there are more performatic solutions out there. :-)

> 
>>
>>> Any thoughts/suggestions?
>>>
> 
> 

Re: Best approach for supporting snapshots for QEMU's gdbstub?

[Peter Maydell]

On Mon, 17 May 2021 at 18:37, Alex Bennée <alex.bennee@linaro.org> wrote:
> Luis Machado <luis.machado@linaro.org> writes:
> > Right. We don't support reverse step/next/continue for remote targets.
> > I think this would be the most appropriate way to implement this
> > feature in GDB. But it is not trivial.
>
> You do because ";ReverseStep+;ReverseContinue+" is part of the gdbstub
> negotiation handshake.
>
> Out of interest how is rr implemented? It presents a gdb interface so I
> thought it was some implemented using some remote magic.

AIUI rr just implements the reverse-step/reverse-continue parts
of the gdb remote protocol. It makes them fast by internally to
its implementation saying "ah, you wanted to do a reverse-step,
I can do that by starting from the best available checkpoint and
going forwards" and by automatically creating checkpoints at
points that it thinks will be useful. gdb and the remote protocol
know nothing about these checkpoints -- they are purely created and
managed under the hood by rr as an optimisation so that reverse-step
is decently fast. (Given that it's the rr end that knows best about
what checkpoints  are available, how expensive it is to create a
checkpoint, etc, that seems not unreasonable.)

There are also a handful of rr-specific gdb commands kind of
like the QEMU-specific ones, which the user can use to say
things like "go directly to this point in time T" which the
gdb UI doesn't have a concept of. (Also because rr starts the
gdb for you it gets a chance to feed it a few gdb macro
definitions which I think mostly just make the debugging
experience a bit smoother rather than being critical parts
of the gdb-to-stub communication.)

thanks
-- PMM

Re: Best approach for supporting snapshots for QEMU's gdbstub?

[Pavel Dovgalyuk]

On 17.05.2021 20:27, Alex Bennée wrote:
> 
> Luis Machado <luis.machado@linaro.org> writes:
> 
>> Hi,
>>
>> On 5/14/21 1:06 PM, Alex Bennée wrote:
>>> Hi,
>>> I've been playing around with QEMU's reverse debugging support which
>>> I have working with Pavel's latest patches for supporting virtio with
>>> record/replay. Once you get the right command line it works well enough
>>> although currently each step backwards requires replaying the entire
>>> execution history until you get to the right point.
>>> QEMU can quite easily snapshot the entire VM state so I was looking
>>> to
>>> see what the best way to integrate this would be. As far as I can tell
>>> there are two interfaces gdb supports: bookmarks and checkpoints.
>>> As far as I can tell bookmarks where added as part of GDB's reverse
>>> debugging support but attempting to use them from the gdbstub reports:
>>>     (gdb) bookmark
>>>     You can't do that when your target is `remote'
>>> so I guess that would need an extension to the stub protocol to
>>> support?
>>>
>>
>> Right. We don't support reverse step/next/continue for remote targets.
>> I think this would be the most appropriate way to implement this
>> feature in GDB. But it is not trivial.
> 
> You do because ";ReverseStep+;ReverseContinue+" is part of the gdbstub
> negotiation handshake.
> 
> Out of interest how is rr implemented? It presents a gdb interface so I
> thought it was some implemented using some remote magic.
> 
> <snip>
> 
>>> We could of course just add a custom monitor command like the
>>> qemu.sstep= command which could be used manually. However that would be
>>> a QEMU gdbstub specific approach.
>>
>> That would be an easy and quick way to allow GDB to control things in
>> QEMU, but I wouldn't say it is the best. Monitor commands are
>> basically a bypass of the RSP where GDB sends/receives commands
>> to/from the remote target.
> 
> We have some underlying commands we can set via the monitor including:
> 
>    monitor info replay
>    monitor replay_seek <N>
>    monitor replay_break <N>
> 
>>
>>> The other thing would be to be more intelligent on QEMU's side and
>>> save
>>> snapshots each time we hit an event, for example each time we hit a
>>> given breakpoint. However I do worry that might lead to snapshots
>>> growing quite quickly.
>>
>> GDB would need to be aware of such snapshots for them to be useful.
>> Otherwise GDB wouldn't be able to use them to restore state.
> 
> What does GDB need to know about them? Does it include something like
> the icount at a particular point.
> 
> I'm curious at how a break and reverse-continue is meant to work if that
> breakpoint is hit multiple times from the start of a run. You need to
> know if the last time you hit a particular breakpoint was in fact the
> last time before where the user was when they hit reverse-continue.

QEMU does this seamlessly. For reverse continue it replays the 
execution, noticing the last breakpoint that was hit. Then it replays 
again, but to the icount of that hit.


Pavel Dovgalyuk