[Qemu-devel] [Question] dump memory when host pci device is used by guest

[Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

Currently, virsh dump uses monitor command migrate to dump guest's memory
to file, and we can use crash to analyze the file.

Unfortunately, virsh dump can not work if guest uses host pci device. The
reason is that the device's status is also needed to migrate to remote machine,
and the host pci device's status is not stored in qemu. So it is unmigratable.

I think we can  we can add a option to qmp command migrate(eg: skip) to allow
the user to skip the check, and this option should be used only when dumping
the guest's memory.

Thanks
Wen Congyang

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-07 11:46, Wen Congyang wrote:
> Currently, virsh dump uses monitor command migrate to dump guest's memory
> to file, and we can use crash to analyze the file.
> 
> Unfortunately, virsh dump can not work if guest uses host pci device. The
> reason is that the device's status is also needed to migrate to remote machine,
> and the host pci device's status is not stored in qemu. So it is unmigratable.
> 
> I think we can  we can add a option to qmp command migrate(eg: skip) to allow
> the user to skip the check, and this option should be used only when dumping
> the guest's memory.

Why not simply attach gdb? That works independently of migration.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

于 2011/10/7 18:16, Jan Kiszka 写道:
> On 2011-10-07 11:46, Wen Congyang wrote:
>> Currently, virsh dump uses monitor command migrate to dump guest's memory
>> to file, and we can use crash to analyze the file.
>>
>> Unfortunately, virsh dump can not work if guest uses host pci device. The
>> reason is that the device's status is also needed to migrate to remote machine,
>> and the host pci device's status is not stored in qemu. So it is unmigratable.
>>
>> I think we can  we can add a option to qmp command migrate(eg: skip) to allow
>> the user to skip the check, and this option should be used only when dumping
>> the guest's memory.
>
> Why not simply attach gdb? That works independently of migration.

If qemu has some problem, we can use gdb to debug it. But if guest os 
has problem
(eg:kernel panic and kdump does not work), we should dump guest's memory 
and use
crash to analyze.

Thanks
Wen Congyang


>
> Jan
>

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-07 14:25, Wen Congyang wrote:
> 于 2011/10/7 18:16, Jan Kiszka 写道:
>> On 2011-10-07 11:46, Wen Congyang wrote:
>>> Currently, virsh dump uses monitor command migrate to dump guest's memory
>>> to file, and we can use crash to analyze the file.
>>>
>>> Unfortunately, virsh dump can not work if guest uses host pci device. The
>>> reason is that the device's status is also needed to migrate to remote machine,
>>> and the host pci device's status is not stored in qemu. So it is unmigratable.
>>>
>>> I think we can  we can add a option to qmp command migrate(eg: skip) to allow
>>> the user to skip the check, and this option should be used only when dumping
>>> the guest's memory.
>>
>> Why not simply attach gdb? That works independently of migration.
> 
> If qemu has some problem, we can use gdb to debug it. But if guest os 
> has problem
> (eg:kernel panic and kdump does not work), we should dump guest's memory 
> and use
> crash to analyze.

qemu-system-xxx -s (or "gdbserver" via monitor if qemu is already
running), gdb vmlinux, then "target remote :1234".

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

于 2011/10/7 20:56, Jan Kiszka 写道:
> On 2011-10-07 14:25, Wen Congyang wrote:
>> 于 2011/10/7 18:16, Jan Kiszka 写道:
>>> On 2011-10-07 11:46, Wen Congyang wrote:
>>>> Currently, virsh dump uses monitor command migrate to dump guest's memory
>>>> to file, and we can use crash to analyze the file.
>>>>
>>>> Unfortunately, virsh dump can not work if guest uses host pci device. The
>>>> reason is that the device's status is also needed to migrate to remote machine,
>>>> and the host pci device's status is not stored in qemu. So it is unmigratable.
>>>>
>>>> I think we can  we can add a option to qmp command migrate(eg: skip) to allow
>>>> the user to skip the check, and this option should be used only when dumping
>>>> the guest's memory.
>>>
>>> Why not simply attach gdb? That works independently of migration.
>>
>> If qemu has some problem, we can use gdb to debug it. But if guest os
>> has problem
>> (eg:kernel panic and kdump does not work), we should dump guest's memory
>> and use
>> crash to analyze.
>
> qemu-system-xxx -s (or "gdbserver" via monitor if qemu is already
> running), gdb vmlinux, then "target remote :1234".

Hmm, if i use qemu, i can do it as the above. But i can not hope our 
customer
do it because it is difficult for them to debug kernel.
So the customer can use 'virsh dump'(the guest is managed by libvirt) or 
autodump(if
the guest has a watchdog) to dump the memory. The supporter can debug 
kernel in another
machine.

I still think that supporting to dump memory when guest uses host pci 
device is necessay.

Thanks
Wen Congyang
>
> Jan
>

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-07 16:05, Wen Congyang wrote:
> 于 2011/10/7 20:56, Jan Kiszka 写道:
>> On 2011-10-07 14:25, Wen Congyang wrote:
>>> 于 2011/10/7 18:16, Jan Kiszka 写道:
>>>> On 2011-10-07 11:46, Wen Congyang wrote:
>>>>> Currently, virsh dump uses monitor command migrate to dump guest's memory
>>>>> to file, and we can use crash to analyze the file.
>>>>>
>>>>> Unfortunately, virsh dump can not work if guest uses host pci device. The
>>>>> reason is that the device's status is also needed to migrate to remote machine,
>>>>> and the host pci device's status is not stored in qemu. So it is unmigratable.
>>>>>
>>>>> I think we can  we can add a option to qmp command migrate(eg: skip) to allow
>>>>> the user to skip the check, and this option should be used only when dumping
>>>>> the guest's memory.
>>>>
>>>> Why not simply attach gdb? That works independently of migration.
>>>
>>> If qemu has some problem, we can use gdb to debug it. But if guest os
>>> has problem
>>> (eg:kernel panic and kdump does not work), we should dump guest's memory
>>> and use
>>> crash to analyze.
>>
>> qemu-system-xxx -s (or "gdbserver" via monitor if qemu is already
>> running), gdb vmlinux, then "target remote :1234".
> 
> Hmm, if i use qemu, i can do it as the above. But i can not hope our 
> customer
> do it because it is difficult for them to debug kernel.
> So the customer can use 'virsh dump'(the guest is managed by libvirt) or 
> autodump(if
> the guest has a watchdog) to dump the memory. The supporter can debug 
> kernel in another
> machine.

gdb is surely scriptable. It's just a bit more complex than running
"generate-core-file" - gdb does not yet support this for remote
sessions. Ideally, that feature should be added, and you are done,
independent of QEMU migration format X.Y.whatever or limitations due to
unmigratable devices. The current approach is, well, "pragmatic".

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Richard W.M. Jones]

On Fri, Oct 07, 2011 at 12:16:07PM +0200, Jan Kiszka wrote:
> On 2011-10-07 11:46, Wen Congyang wrote:
> > Currently, virsh dump uses monitor command migrate to dump guest's memory
> > to file, and we can use crash to analyze the file.
> > 
> > Unfortunately, virsh dump can not work if guest uses host pci device. The
> > reason is that the device's status is also needed to migrate to remote machine,
> > and the host pci device's status is not stored in qemu. So it is unmigratable.
> > 
> > I think we can  we can add a option to qmp command migrate(eg: skip) to allow
> > the user to skip the check, and this option should be used only when dumping
> > the guest's memory.
> 
> Why not simply attach gdb? That works independently of migration.

It's a lot more hairy and requires dev tools to be installed on the
production host.

The core file is technically simple, has no dependencies, can be moved
to another machine for analysis, and contains all the data (not just a
backtrace).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1369 bytes --]

On 2011-10-08 17:16, Richard W.M. Jones wrote:
> On Fri, Oct 07, 2011 at 12:16:07PM +0200, Jan Kiszka wrote:
>> On 2011-10-07 11:46, Wen Congyang wrote:
>>> Currently, virsh dump uses monitor command migrate to dump guest's memory
>>> to file, and we can use crash to analyze the file.
>>>
>>> Unfortunately, virsh dump can not work if guest uses host pci device. The
>>> reason is that the device's status is also needed to migrate to remote machine,
>>> and the host pci device's status is not stored in qemu. So it is unmigratable.
>>>
>>> I think we can  we can add a option to qmp command migrate(eg: skip) to allow
>>> the user to skip the check, and this option should be used only when dumping
>>> the guest's memory.
>>
>> Why not simply attach gdb? That works independently of migration.
> 
> It's a lot more hairy and requires dev tools to be installed on the
> production host.
> 
> The core file is technically simple, has no dependencies, can be moved
> to another machine for analysis, and contains all the data (not just a
> backtrace).

As explained in the other replies: It is way more future-proof to use an
interface for this which was designed for it (remote gdb) instead of
artificially relaxing reasonable constraints of the migration mechanism
plus having to follow that format with the post-processing tool.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Richard W.M. Jones]

On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
> As explained in the other replies: It is way more future-proof to use an
> interface for this which was designed for it (remote gdb) instead of
> artificially relaxing reasonable constraints of the migration mechanism
> plus having to follow that format with the post-processing tool.

Any interface that isn't "get this information off my production
server *now*" so that I can get the server restarted, and send it to
an expert to analyse -- is a poor interface, whether it was designed
like that or not.  Perhaps we don't have the right interface at all,
but remote gdb is not it.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On 2011-10-09 12:23, Richard W.M. Jones wrote:
> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>> As explained in the other replies: It is way more future-proof to use an
>> interface for this which was designed for it (remote gdb) instead of
>> artificially relaxing reasonable constraints of the migration mechanism
>> plus having to follow that format with the post-processing tool.
> 
> Any interface that isn't "get this information off my production
> server *now*" so that I can get the server restarted, and send it to
> an expert to analyse -- is a poor interface, whether it was designed
> like that or not.  Perhaps we don't have the right interface at all,
> but remote gdb is not it.

Why is it not (for implementing core file write-out - I'm not proposing
to drop that part of libvirt)?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>> As explained in the other replies: It is way more future-proof to use an
>> interface for this which was designed for it (remote gdb) instead of
>> artificially relaxing reasonable constraints of the migration mechanism
>> plus having to follow that format with the post-processing tool.
> 
> Any interface that isn't "get this information off my production
> server *now*" so that I can get the server restarted, and send it to
> an expert to analyse -- is a poor interface, whether it was designed
> like that or not.  Perhaps we don't have the right interface at all,
> but remote gdb is not it.

What about the following idea?

Introduce a new monitor command named dump, and this command accepts a filename.
We can use almost all migration's code. We use this command to dump guest's
memory, so there is no need to check whether the guest has a unmigratable device.

Thanks
Wen Congyang

> 
> Rich.
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1243 bytes --]

On 2011-10-10 04:21, Wen Congyang wrote:
> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>> As explained in the other replies: It is way more future-proof to use an
>>> interface for this which was designed for it (remote gdb) instead of
>>> artificially relaxing reasonable constraints of the migration mechanism
>>> plus having to follow that format with the post-processing tool.
>>
>> Any interface that isn't "get this information off my production
>> server *now*" so that I can get the server restarted, and send it to
>> an expert to analyse -- is a poor interface, whether it was designed
>> like that or not.  Perhaps we don't have the right interface at all,
>> but remote gdb is not it.
> 
> What about the following idea?
> 
> Introduce a new monitor command named dump, and this command accepts a filename.
> We can use almost all migration's code. We use this command to dump guest's
> memory, so there is no need to check whether the guest has a unmigratable device.

I do not want to reject this proposal categorically, but I would like to
see the gdb path fail /wrt essential requirements first. So far I don't
see it would.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/10/2011 02:52 PM, Jan Kiszka Write:
> On 2011-10-10 04:21, Wen Congyang wrote:
>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>> As explained in the other replies: It is way more future-proof to use an
>>>> interface for this which was designed for it (remote gdb) instead of
>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>> plus having to follow that format with the post-processing tool.
>>>
>>> Any interface that isn't "get this information off my production
>>> server *now*" so that I can get the server restarted, and send it to
>>> an expert to analyse -- is a poor interface, whether it was designed
>>> like that or not.  Perhaps we don't have the right interface at all,
>>> but remote gdb is not it.
>>
>> What about the following idea?
>>
>> Introduce a new monitor command named dump, and this command accepts a filename.
>> We can use almost all migration's code. We use this command to dump guest's
>> memory, so there is no need to check whether the guest has a unmigratable device.
> 
> I do not want to reject this proposal categorically, but I would like to
> see the gdb path fail /wrt essential requirements first. So far I don't
> see it would.

‘gdb path fail /wrt essential requirements’

what does it mean?

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

On 2011-10-10 08:59, Wen Congyang wrote:
> At 10/10/2011 02:52 PM, Jan Kiszka Write:
>> On 2011-10-10 04:21, Wen Congyang wrote:
>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>> plus having to follow that format with the post-processing tool.
>>>>
>>>> Any interface that isn't "get this information off my production
>>>> server *now*" so that I can get the server restarted, and send it to
>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>> but remote gdb is not it.
>>>
>>> What about the following idea?
>>>
>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>> We can use almost all migration's code. We use this command to dump guest's
>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>
>> I do not want to reject this proposal categorically, but I would like to
>> see the gdb path fail /wrt essential requirements first. So far I don't
>> see it would.
> 
> ‘gdb path fail /wrt essential requirements’
> 
> what does it mean?

That you explain why reading reading memory and processor states via the
remote gdb interface and dumping it into a proper core file cannot be
made working for you.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Alexander Graf]

Am 10.10.2011 um 08:52 schrieb Jan Kiszka <jan.kiszka@web.de>:

> On 2011-10-10 04:21, Wen Congyang wrote:
>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>> As explained in the other replies: It is way more future-proof to use an
>>>> interface for this which was designed for it (remote gdb) instead of
>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>> plus having to follow that format with the post-processing tool.
>>> 
>>> Any interface that isn't "get this information off my production
>>> server *now*" so that I can get the server restarted, and send it to
>>> an expert to analyse -- is a poor interface, whether it was designed
>>> like that or not.  Perhaps we don't have the right interface at all,
>>> but remote gdb is not it.
>> 
>> What about the following idea?
>> 
>> Introduce a new monitor command named dump, and this command accepts a filename.
>> We can use almost all migration's code. We use this command to dump guest's
>> memory, so there is no need to check whether the guest has a unmigratable device.
> 
> I do not want to reject this proposal categorically, but I would like to
> see the gdb path fail /wrt essential requirements first. So far I don't
> see it would.

Through gdb you access virtual address space, whilein this case you really want physical.

All you need to create a core file is a memory dump and a register dump. For registers, gdb sounds like a good interface. For memory, why not just use pmemsave? Maybe add a monitor/qmp command that tells you all ram regions, so pmemsave can be invoked intelligently.

The alternative to all this is obviously an in-qemu 'core' command that creates core files inside of qemu. That's where I think it really belongs to. Libvirt is a management tool stack, not a "I don't like to work on qemu code so I push code to the layer on top" mesh of things. Or at least that's how it's really supposed to be :).


Alex

> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]

On 2011-10-10 09:08, Alexander Graf wrote:
> 
> Am 10.10.2011 um 08:52 schrieb Jan Kiszka <jan.kiszka@web.de>:
> 
>> On 2011-10-10 04:21, Wen Congyang wrote:
>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>> plus having to follow that format with the post-processing tool.
>>>>
>>>> Any interface that isn't "get this information off my production
>>>> server *now*" so that I can get the server restarted, and send it to
>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>> but remote gdb is not it.
>>>
>>> What about the following idea?
>>>
>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>> We can use almost all migration's code. We use this command to dump guest's
>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>
>> I do not want to reject this proposal categorically, but I would like to
>> see the gdb path fail /wrt essential requirements first. So far I don't
>> see it would.
> 
> Through gdb you access virtual address space, whilein this case you really want physical.

You want to attach a debugger afterwards as well. So this is just about
offlining the gdb session the end user should not have to handle himself.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/10/2011 03:01 PM, Jan Kiszka Write:
> On 2011-10-10 08:59, Wen Congyang wrote:
>> At 10/10/2011 02:52 PM, Jan Kiszka Write:
>>> On 2011-10-10 04:21, Wen Congyang wrote:
>>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>>> plus having to follow that format with the post-processing tool.
>>>>>
>>>>> Any interface that isn't "get this information off my production
>>>>> server *now*" so that I can get the server restarted, and send it to
>>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>>> but remote gdb is not it.
>>>>
>>>> What about the following idea?
>>>>
>>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>>> We can use almost all migration's code. We use this command to dump guest's
>>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>>
>>> I do not want to reject this proposal categorically, but I would like to
>>> see the gdb path fail /wrt essential requirements first. So far I don't
>>> see it would.
>>
>> ‘gdb path fail /wrt essential requirements’
>>
>> what does it mean?
> 
> That you explain why reading reading memory and processor states via the
> remote gdb interface and dumping it into a proper core file cannot be
> made working for you.

First, I think crash can not analyze such core file. But it is not very important.

What is remote gdb interface? Do you mean that: the supporter uses gdb from another machine
to connect to customer's machine and get the data? If so, this way can not be
used when the customer needs to dump the guest's memory automatically when watchdog timeouts.

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2533 bytes --]

On 2011-10-10 09:17, Wen Congyang wrote:
> At 10/10/2011 03:01 PM, Jan Kiszka Write:
>> On 2011-10-10 08:59, Wen Congyang wrote:
>>> At 10/10/2011 02:52 PM, Jan Kiszka Write:
>>>> On 2011-10-10 04:21, Wen Congyang wrote:
>>>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>>>> plus having to follow that format with the post-processing tool.
>>>>>>
>>>>>> Any interface that isn't "get this information off my production
>>>>>> server *now*" so that I can get the server restarted, and send it to
>>>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>>>> but remote gdb is not it.
>>>>>
>>>>> What about the following idea?
>>>>>
>>>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>>>> We can use almost all migration's code. We use this command to dump guest's
>>>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>>>
>>>> I do not want to reject this proposal categorically, but I would like to
>>>> see the gdb path fail /wrt essential requirements first. So far I don't
>>>> see it would.
>>>
>>> ‘gdb path fail /wrt essential requirements’
>>>
>>> what does it mean?
>>
>> That you explain why reading reading memory and processor states via the
>> remote gdb interface and dumping it into a proper core file cannot be
>> made working for you.
> 
> First, I think crash can not analyze such core file. But it is not very important.
> 
> What is remote gdb interface?

man qemu -> gdb.

> Do you mean that: the supporter uses gdb from another machine

Or locally. There are various transports possible.

> to connect to customer's machine and get the data? If so, this way can not be
> used when the customer needs to dump the guest's memory automatically when watchdog timeouts.

It is just another channel that can conceptually be used like the
monitor, by a management app like libvirt, directly or indirectly via a
scripted gdb frontend, or also by a human who wants to save some ongoing
gdb session for later analysis. This dual use make such an approach the
preferred one.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/10/2011 03:22 PM, Jan Kiszka Write:
> On 2011-10-10 09:17, Wen Congyang wrote:
>> At 10/10/2011 03:01 PM, Jan Kiszka Write:
>>> On 2011-10-10 08:59, Wen Congyang wrote:
>>>> At 10/10/2011 02:52 PM, Jan Kiszka Write:
>>>>> On 2011-10-10 04:21, Wen Congyang wrote:
>>>>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>>>>> plus having to follow that format with the post-processing tool.
>>>>>>>
>>>>>>> Any interface that isn't "get this information off my production
>>>>>>> server *now*" so that I can get the server restarted, and send it to
>>>>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>>>>> but remote gdb is not it.
>>>>>>
>>>>>> What about the following idea?
>>>>>>
>>>>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>>>>> We can use almost all migration's code. We use this command to dump guest's
>>>>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>>>>
>>>>> I do not want to reject this proposal categorically, but I would like to
>>>>> see the gdb path fail /wrt essential requirements first. So far I don't
>>>>> see it would.
>>>>
>>>> ‘gdb path fail /wrt essential requirements’
>>>>
>>>> what does it mean?
>>>
>>> That you explain why reading reading memory and processor states via the
>>> remote gdb interface and dumping it into a proper core file cannot be
>>> made working for you.
>>
>> First, I think crash can not analyze such core file. But it is not very important.
>>
>> What is remote gdb interface?
> 
> man qemu -> gdb.
> 
>> Do you mean that: the supporter uses gdb from another machine
> 
> Or locally. There are various transports possible.
> 
>> to connect to customer's machine and get the data? If so, this way can not be
>> used when the customer needs to dump the guest's memory automatically when watchdog timeouts.
> 
> It is just another channel that can conceptually be used like the
> monitor, by a management app like libvirt, directly or indirectly via a
> scripted gdb frontend, or also by a human who wants to save some ongoing
> gdb session for later analysis. This dual use make such an approach the
> preferred one.

Is the following is right?

1. execute the monitor command 'gdbserver'
2. run gdb and then 'target remote :1234'

But, unfortunately, the monitor command gdbserver does not exit when we use json to connect to monitor.

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3073 bytes --]

On 2011-10-10 09:47, Wen Congyang wrote:
> At 10/10/2011 03:22 PM, Jan Kiszka Write:
>> On 2011-10-10 09:17, Wen Congyang wrote:
>>> At 10/10/2011 03:01 PM, Jan Kiszka Write:
>>>> On 2011-10-10 08:59, Wen Congyang wrote:
>>>>> At 10/10/2011 02:52 PM, Jan Kiszka Write:
>>>>>> On 2011-10-10 04:21, Wen Congyang wrote:
>>>>>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>>>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>>>>>> plus having to follow that format with the post-processing tool.
>>>>>>>>
>>>>>>>> Any interface that isn't "get this information off my production
>>>>>>>> server *now*" so that I can get the server restarted, and send it to
>>>>>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>>>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>>>>>> but remote gdb is not it.
>>>>>>>
>>>>>>> What about the following idea?
>>>>>>>
>>>>>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>>>>>> We can use almost all migration's code. We use this command to dump guest's
>>>>>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>>>>>
>>>>>> I do not want to reject this proposal categorically, but I would like to
>>>>>> see the gdb path fail /wrt essential requirements first. So far I don't
>>>>>> see it would.
>>>>>
>>>>> ‘gdb path fail /wrt essential requirements’
>>>>>
>>>>> what does it mean?
>>>>
>>>> That you explain why reading reading memory and processor states via the
>>>> remote gdb interface and dumping it into a proper core file cannot be
>>>> made working for you.
>>>
>>> First, I think crash can not analyze such core file. But it is not very important.
>>>
>>> What is remote gdb interface?
>>
>> man qemu -> gdb.
>>
>>> Do you mean that: the supporter uses gdb from another machine
>>
>> Or locally. There are various transports possible.
>>
>>> to connect to customer's machine and get the data? If so, this way can not be
>>> used when the customer needs to dump the guest's memory automatically when watchdog timeouts.
>>
>> It is just another channel that can conceptually be used like the
>> monitor, by a management app like libvirt, directly or indirectly via a
>> scripted gdb frontend, or also by a human who wants to save some ongoing
>> gdb session for later analysis. This dual use make such an approach the
>> preferred one.
> 
> Is the following is right?
> 
> 1. execute the monitor command 'gdbserver'
> 2. run gdb and then 'target remote :1234'

Yep.

> 
> But, unfortunately, the monitor command gdbserver does not exit when we use json to connect to monitor.

You mean via human-monitor-command (gdbserver is not a qmp command)?
Would be a bug.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/10/2011 03:48 PM, Jan Kiszka Write:
> On 2011-10-10 09:47, Wen Congyang wrote:
>> At 10/10/2011 03:22 PM, Jan Kiszka Write:
>>> On 2011-10-10 09:17, Wen Congyang wrote:
>>>> At 10/10/2011 03:01 PM, Jan Kiszka Write:
>>>>> On 2011-10-10 08:59, Wen Congyang wrote:
>>>>>> At 10/10/2011 02:52 PM, Jan Kiszka Write:
>>>>>>> On 2011-10-10 04:21, Wen Congyang wrote:
>>>>>>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>>>>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>>>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>>>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>>>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>>>>>>> plus having to follow that format with the post-processing tool.
>>>>>>>>>
>>>>>>>>> Any interface that isn't "get this information off my production
>>>>>>>>> server *now*" so that I can get the server restarted, and send it to
>>>>>>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>>>>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>>>>>>> but remote gdb is not it.
>>>>>>>>
>>>>>>>> What about the following idea?
>>>>>>>>
>>>>>>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>>>>>>> We can use almost all migration's code. We use this command to dump guest's
>>>>>>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>>>>>>
>>>>>>> I do not want to reject this proposal categorically, but I would like to
>>>>>>> see the gdb path fail /wrt essential requirements first. So far I don't
>>>>>>> see it would.
>>>>>>
>>>>>> ‘gdb path fail /wrt essential requirements’
>>>>>>
>>>>>> what does it mean?
>>>>>
>>>>> That you explain why reading reading memory and processor states via the
>>>>> remote gdb interface and dumping it into a proper core file cannot be
>>>>> made working for you.
>>>>
>>>> First, I think crash can not analyze such core file. But it is not very important.
>>>>
>>>> What is remote gdb interface?
>>>
>>> man qemu -> gdb.
>>>
>>>> Do you mean that: the supporter uses gdb from another machine
>>>
>>> Or locally. There are various transports possible.
>>>
>>>> to connect to customer's machine and get the data? If so, this way can not be
>>>> used when the customer needs to dump the guest's memory automatically when watchdog timeouts.
>>>
>>> It is just another channel that can conceptually be used like the
>>> monitor, by a management app like libvirt, directly or indirectly via a
>>> scripted gdb frontend, or also by a human who wants to save some ongoing
>>> gdb session for later analysis. This dual use make such an approach the
>>> preferred one.
>>
>> Is the following is right?
>>
>> 1. execute the monitor command 'gdbserver'
>> 2. run gdb and then 'target remote :1234'
> 
> Yep.

Can two guest use such port(1234) at the same time? If not, I think qemu should support to select
a port automatically.

Another problem is that: we should not require dev and debug tools(here is gdb) to be installed on the
production host.

> 
>>
>> But, unfortunately, the monitor command gdbserver does not exit when we use json to connect to monitor.
> 
> You mean via human-monitor-command (gdbserver is not a qmp command)?
Yes.

Thanks
Wen Congyang

> Would be a bug.
> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Daniel P. Berrange]

On Fri, Oct 07, 2011 at 02:56:56PM +0200, Jan Kiszka wrote:
> On 2011-10-07 14:25, Wen Congyang wrote:
> > 于 2011/10/7 18:16, Jan Kiszka 写道:
> >> On 2011-10-07 11:46, Wen Congyang wrote:
> >>> Currently, virsh dump uses monitor command migrate to dump guest's memory
> >>> to file, and we can use crash to analyze the file.
> >>>
> >>> Unfortunately, virsh dump can not work if guest uses host pci device. The
> >>> reason is that the device's status is also needed to migrate to remote machine,
> >>> and the host pci device's status is not stored in qemu. So it is unmigratable.
> >>>
> >>> I think we can  we can add a option to qmp command migrate(eg: skip) to allow
> >>> the user to skip the check, and this option should be used only when dumping
> >>> the guest's memory.
> >>
> >> Why not simply attach gdb? That works independently of migration.
> > 
> > If qemu has some problem, we can use gdb to debug it. But if guest os 
> > has problem
> > (eg:kernel panic and kdump does not work), we should dump guest's memory 
> > and use
> > crash to analyze.
> 
> qemu-system-xxx -s (or "gdbserver" via monitor if qemu is already
> running), gdb vmlinux, then "target remote :1234".

That is already possible, but that is not what we need for 'virsh dump'.

The goal of that API is to provide a coredump of the guest, which can
then be analysed off-node/site. While in theory you could attach GDB
to the QEMU process and use GDB commands to then write out a coredump
this isn't really satisfactory because too many large companies have
security/audit compliance rules which forbid installation of developer
tools (compilers, debuggers, etc) on production servers. So we cannot
assume GDB is available.

We need to be able to create a coredump natively from either QEMU or
libvirt, with minimal of external tools.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Daniel P. Berrange]

On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
> On 2011-10-10 04:21, Wen Congyang wrote:
> > At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
> >> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
> >>> As explained in the other replies: It is way more future-proof to use an
> >>> interface for this which was designed for it (remote gdb) instead of
> >>> artificially relaxing reasonable constraints of the migration mechanism
> >>> plus having to follow that format with the post-processing tool.
> >>
> >> Any interface that isn't "get this information off my production
> >> server *now*" so that I can get the server restarted, and send it to
> >> an expert to analyse -- is a poor interface, whether it was designed
> >> like that or not.  Perhaps we don't have the right interface at all,
> >> but remote gdb is not it.
> > 
> > What about the following idea?
> > 
> > Introduce a new monitor command named dump, and this command accepts a filename.
> > We can use almost all migration's code. We use this command to dump guest's
> > memory, so there is no need to check whether the guest has a unmigratable device.
> 
> I do not want to reject this proposal categorically, but I would like to
> see the gdb path fail /wrt essential requirements first. So far I don't
> see it would.

GDB is often forbidden on production servers, so that path is
clearly not an option, unless we want libvirt to implement the
GDB remote RPC protocol itself which just sounds like a world
of hurt.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Daniel P. Berrange]

On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote:
> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
> > On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
> >> As explained in the other replies: It is way more future-proof to use an
> >> interface for this which was designed for it (remote gdb) instead of
> >> artificially relaxing reasonable constraints of the migration mechanism
> >> plus having to follow that format with the post-processing tool.
> > 
> > Any interface that isn't "get this information off my production
> > server *now*" so that I can get the server restarted, and send it to
> > an expert to analyse -- is a poor interface, whether it was designed
> > like that or not.  Perhaps we don't have the right interface at all,
> > but remote gdb is not it.
> 
> What about the following idea?
> 
> Introduce a new monitor command named dump, and this command accepts a filename.
> We can use almost all migration's code. We use this command to dump guest's
> memory, so there is no need to check whether the guest has a unmigratable device.

I think it would be a good idea of QEMU had a dedicated 'dump' command
for this purpose, even if it was just an alias for 'migrate' initially.
I have never really liked the fact that we abuse the 'migrate' command
to generate a core dump. The resulting data file from this is more
complex than it really needs to be, causing complexity for post-processing
it. The needs of migration, are not entirely aligned with the needs of
core dumping in the long term, so we should allow the possibility of
their impls diverging without impacting apps using them.

So adding a 'dump' command which wrote out data in a format that was
optimized for offline processing by tools like 'crash' (or the windows
equivalent) would be a good improvement, even if it just reuses the
migrate code for now.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Daniel P. Berrange]

On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote:
> On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote:
> > At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
> > > On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
> > >> As explained in the other replies: It is way more future-proof to use an
> > >> interface for this which was designed for it (remote gdb) instead of
> > >> artificially relaxing reasonable constraints of the migration mechanism
> > >> plus having to follow that format with the post-processing tool.
> > > 
> > > Any interface that isn't "get this information off my production
> > > server *now*" so that I can get the server restarted, and send it to
> > > an expert to analyse -- is a poor interface, whether it was designed
> > > like that or not.  Perhaps we don't have the right interface at all,
> > > but remote gdb is not it.
> > 
> > What about the following idea?
> > 
> > Introduce a new monitor command named dump, and this command accepts a filename.
> > We can use almost all migration's code. We use this command to dump guest's
> > memory, so there is no need to check whether the guest has a unmigratable device.
> 
> I think it would be a good idea of QEMU had a dedicated 'dump' command
> for this purpose, even if it was just an alias for 'migrate' initially.
> I have never really liked the fact that we abuse the 'migrate' command
> to generate a core dump. The resulting data file from this is more
> complex than it really needs to be, causing complexity for post-processing
> it. The needs of migration, are not entirely aligned with the needs of
> core dumping in the long term, so we should allow the possibility of
> their impls diverging without impacting apps using them.
> 
> So adding a 'dump' command which wrote out data in a format that was
> optimized for offline processing by tools like 'crash' (or the windows
> equivalent) would be a good improvement, even if it just reuses the
> migrate code for now.

The other reason why it would be good, is that we would then have a clearly
defined standard "QEMU dump format", instead of "libvirt dump format for QEMU"

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-10 11:02, Daniel P. Berrange wrote:
> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>> On 2011-10-10 04:21, Wen Congyang wrote:
>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>> plus having to follow that format with the post-processing tool.
>>>>
>>>> Any interface that isn't "get this information off my production
>>>> server *now*" so that I can get the server restarted, and send it to
>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>> but remote gdb is not it.
>>>
>>> What about the following idea?
>>>
>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>> We can use almost all migration's code. We use this command to dump guest's
>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>
>> I do not want to reject this proposal categorically, but I would like to
>> see the gdb path fail /wrt essential requirements first. So far I don't
>> see it would.
> 
> GDB is often forbidden on production servers, so that path is
> clearly not an option, unless we want libvirt to implement the
> GDB remote RPC protocol itself which just sounds like a world
> of hurt.

Run gdb with "set debug remote 1" and watch the communication, it is not
that complex. But a dump command is probably simpler for those
scenarios, I agree.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Richard W.M. Jones]

On Mon, Oct 10, 2011 at 10:10:21AM +0100, Daniel P. Berrange wrote:
> On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote:
> > On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote:
> > > At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
> > > > On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
> > > >> As explained in the other replies: It is way more future-proof to use an
> > > >> interface for this which was designed for it (remote gdb) instead of
> > > >> artificially relaxing reasonable constraints of the migration mechanism
> > > >> plus having to follow that format with the post-processing tool.
> > > > 
> > > > Any interface that isn't "get this information off my production
> > > > server *now*" so that I can get the server restarted, and send it to
> > > > an expert to analyse -- is a poor interface, whether it was designed
> > > > like that or not.  Perhaps we don't have the right interface at all,
> > > > but remote gdb is not it.
> > > 
> > > What about the following idea?
> > > 
> > > Introduce a new monitor command named dump, and this command accepts a filename.
> > > We can use almost all migration's code. We use this command to dump guest's
> > > memory, so there is no need to check whether the guest has a unmigratable device.
> > 
> > I think it would be a good idea of QEMU had a dedicated 'dump' command
> > for this purpose, even if it was just an alias for 'migrate' initially.
> > I have never really liked the fact that we abuse the 'migrate' command
> > to generate a core dump. The resulting data file from this is more
> > complex than it really needs to be, causing complexity for post-processing
> > it. The needs of migration, are not entirely aligned with the needs of
> > core dumping in the long term, so we should allow the possibility of
> > their impls diverging without impacting apps using them.
> > 
> > So adding a 'dump' command which wrote out data in a format that was
> > optimized for offline processing by tools like 'crash' (or the windows
> > equivalent) would be a good improvement, even if it just reuses the
> > migrate code for now.
> 
> The other reason why it would be good, is that we would then have a clearly
> defined standard "QEMU dump format", instead of "libvirt dump format for QEMU"

Other reasons why this is a good idea:

 - so you don't need 3 processes doing a dance in order to capture a
   core dump

 - so that qemu can initiate a core dump itself (from watchdog events)

 - so that two qemu processes can do a core dump at the same time
   without trying to use conflicting ports

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-10 11:10, Daniel P. Berrange wrote:
> On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote:
>> On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote:
>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>> plus having to follow that format with the post-processing tool.
>>>>
>>>> Any interface that isn't "get this information off my production
>>>> server *now*" so that I can get the server restarted, and send it to
>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>> but remote gdb is not it.
>>>
>>> What about the following idea?
>>>
>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>> We can use almost all migration's code. We use this command to dump guest's
>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>
>> I think it would be a good idea of QEMU had a dedicated 'dump' command
>> for this purpose, even if it was just an alias for 'migrate' initially.
>> I have never really liked the fact that we abuse the 'migrate' command
>> to generate a core dump. The resulting data file from this is more
>> complex than it really needs to be, causing complexity for post-processing
>> it. The needs of migration, are not entirely aligned with the needs of
>> core dumping in the long term, so we should allow the possibility of
>> their impls diverging without impacting apps using them.
>>
>> So adding a 'dump' command which wrote out data in a format that was
>> optimized for offline processing by tools like 'crash' (or the windows
>> equivalent) would be a good improvement, even if it just reuses the
>> migrate code for now.
> 
> The other reason why it would be good, is that we would then have a clearly
> defined standard "QEMU dump format", instead of "libvirt dump format for QEMU"

A core file would be that format - for direct gdb processing. No
proprietary re-inventions please.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-10 11:34, Richard W.M. Jones wrote:
> On Mon, Oct 10, 2011 at 10:10:21AM +0100, Daniel P. Berrange wrote:
>> On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote:
>>> On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote:
>>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>>> plus having to follow that format with the post-processing tool.
>>>>>
>>>>> Any interface that isn't "get this information off my production
>>>>> server *now*" so that I can get the server restarted, and send it to
>>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>>> but remote gdb is not it.
>>>>
>>>> What about the following idea?
>>>>
>>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>>> We can use almost all migration's code. We use this command to dump guest's
>>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>>
>>> I think it would be a good idea of QEMU had a dedicated 'dump' command
>>> for this purpose, even if it was just an alias for 'migrate' initially.
>>> I have never really liked the fact that we abuse the 'migrate' command
>>> to generate a core dump. The resulting data file from this is more
>>> complex than it really needs to be, causing complexity for post-processing
>>> it. The needs of migration, are not entirely aligned with the needs of
>>> core dumping in the long term, so we should allow the possibility of
>>> their impls diverging without impacting apps using them.
>>>
>>> So adding a 'dump' command which wrote out data in a format that was
>>> optimized for offline processing by tools like 'crash' (or the windows
>>> equivalent) would be a good improvement, even if it just reuses the
>>> migrate code for now.
>>
>> The other reason why it would be good, is that we would then have a clearly
>> defined standard "QEMU dump format", instead of "libvirt dump format for QEMU"
> 
> Other reasons why this is a good idea:
> 
>  - so you don't need 3 processes doing a dance in order to capture a
>    core dump

If it made the overall implementation simpler, that would have been no
valid reason.

The key is only that banning of gdb on certain systems (though I wonder
if you cannot configure it to only support remote targets, thus making
it useless for potential attackers).

> 
>  - so that qemu can initiate a core dump itself (from watchdog events)

This is a management tool decision. A dump command would not have any
pre-condition or auto-trigger.

> 
>  - so that two qemu processes can do a core dump at the same time
>    without trying to use conflicting ports

That is a non-issue (just like you can have multiple monitors of
multiple VMs listening on different TCP ports).

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Daniel P. Berrange]

On Mon, Oct 10, 2011 at 11:34:44AM +0200, Jan Kiszka wrote:
> On 2011-10-10 11:10, Daniel P. Berrange wrote:
> > On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote:
> >> On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote:
> >>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
> >>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
> >>>>> As explained in the other replies: It is way more future-proof to use an
> >>>>> interface for this which was designed for it (remote gdb) instead of
> >>>>> artificially relaxing reasonable constraints of the migration mechanism
> >>>>> plus having to follow that format with the post-processing tool.
> >>>>
> >>>> Any interface that isn't "get this information off my production
> >>>> server *now*" so that I can get the server restarted, and send it to
> >>>> an expert to analyse -- is a poor interface, whether it was designed
> >>>> like that or not.  Perhaps we don't have the right interface at all,
> >>>> but remote gdb is not it.
> >>>
> >>> What about the following idea?
> >>>
> >>> Introduce a new monitor command named dump, and this command accepts a filename.
> >>> We can use almost all migration's code. We use this command to dump guest's
> >>> memory, so there is no need to check whether the guest has a unmigratable device.
> >>
> >> I think it would be a good idea of QEMU had a dedicated 'dump' command
> >> for this purpose, even if it was just an alias for 'migrate' initially.
> >> I have never really liked the fact that we abuse the 'migrate' command
> >> to generate a core dump. The resulting data file from this is more
> >> complex than it really needs to be, causing complexity for post-processing
> >> it. The needs of migration, are not entirely aligned with the needs of
> >> core dumping in the long term, so we should allow the possibility of
> >> their impls diverging without impacting apps using them.
> >>
> >> So adding a 'dump' command which wrote out data in a format that was
> >> optimized for offline processing by tools like 'crash' (or the windows
> >> equivalent) would be a good improvement, even if it just reuses the
> >> migrate code for now.
> > 
> > The other reason why it would be good, is that we would then have a clearly
> > defined standard "QEMU dump format", instead of "libvirt dump format for QEMU"
> 
> A core file would be that format - for direct gdb processing. No
> proprietary re-inventions please.

I have no personal attachment to any particular format, so if the standard
core file format is possible, then we should definitely try to use it in
QEMU.


Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Alon Levy]

On Mon, Oct 10, 2011 at 11:34:44AM +0200, Jan Kiszka wrote:
> On 2011-10-10 11:10, Daniel P. Berrange wrote:
> > On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote:
> >> On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote:
> >>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
> >>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
> >>>>> As explained in the other replies: It is way more future-proof to use an
> >>>>> interface for this which was designed for it (remote gdb) instead of
> >>>>> artificially relaxing reasonable constraints of the migration mechanism
> >>>>> plus having to follow that format with the post-processing tool.
> >>>>
> >>>> Any interface that isn't "get this information off my production
> >>>> server *now*" so that I can get the server restarted, and send it to
> >>>> an expert to analyse -- is a poor interface, whether it was designed
> >>>> like that or not.  Perhaps we don't have the right interface at all,
> >>>> but remote gdb is not it.
> >>>
> >>> What about the following idea?
> >>>
> >>> Introduce a new monitor command named dump, and this command accepts a filename.
> >>> We can use almost all migration's code. We use this command to dump guest's
> >>> memory, so there is no need to check whether the guest has a unmigratable device.
> >>
> >> I think it would be a good idea of QEMU had a dedicated 'dump' command
> >> for this purpose, even if it was just an alias for 'migrate' initially.
> >> I have never really liked the fact that we abuse the 'migrate' command
> >> to generate a core dump. The resulting data file from this is more
> >> complex than it really needs to be, causing complexity for post-processing
> >> it. The needs of migration, are not entirely aligned with the needs of
> >> core dumping in the long term, so we should allow the possibility of
> >> their impls diverging without impacting apps using them.
> >>
> >> So adding a 'dump' command which wrote out data in a format that was
> >> optimized for offline processing by tools like 'crash' (or the windows
> >> equivalent) would be a good improvement, even if it just reuses the
> >> migrate code for now.
> > 
> > The other reason why it would be good, is that we would then have a clearly
> > defined standard "QEMU dump format", instead of "libvirt dump format for QEMU"
> 
> A core file would be that format - for direct gdb processing. No
> proprietary re-inventions please.

Just a note: A core file to windows core dump file would be nice for
windows guest crashes.

> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT T DE IT 1
> Corporate Competence Center Embedded Linux
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Paolo Bonzini]

On 10/10/2011 12:21 PM, Alon Levy wrote:
> >  A core file would be that format - for direct gdb processing. No
> >  proprietary re-inventions please.
>
> Just a note: A core file to windows core dump file would be nice for
> windows guest crashes.

That requires cooperation from a kernel driver in the Windows guest. 
The driver must call KeInitializeCrashDumpHeader and write somewhere the 
physical address of the buffer.

Paolo

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Daniel P. Berrange]

On Mon, Oct 10, 2011 at 12:48:52PM +0200, Paolo Bonzini wrote:
> On 10/10/2011 12:21 PM, Alon Levy wrote:
> >>  A core file would be that format - for direct gdb processing. No
> >>  proprietary re-inventions please.
> >
> >Just a note: A core file to windows core dump file would be nice for
> >windows guest crashes.
> 
> That requires cooperation from a kernel driver in the Windows guest.
> The driver must call KeInitializeCrashDumpHeader and write somewhere
> the physical address of the buffer.

That won't be suitable for 'virsh dump' then, because we need this to
work when the guest OS is in a crashed/non-responsive state, and so
we cna't rely on talking to it.


Daniel.
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Paolo Bonzini]

On 10/10/2011 01:04 PM, Daniel P. Berrange wrote:
> On Mon, Oct 10, 2011 at 12:48:52PM +0200, Paolo Bonzini wrote:
>> >  On 10/10/2011 12:21 PM, Alon Levy wrote:
>>>> >  >>    A core file would be that format - for direct gdb processing. No
>>>> >  >>    proprietary re-inventions please.
>>> >  >
>>> >  >Just a note: A core file to windows core dump file would be nice for
>>> >  >windows guest crashes.
>> >
>> >  That requires cooperation from a kernel driver in the Windows guest.
>> >  The driver must call KeInitializeCrashDumpHeader and write somewhere
>> >  the physical address of the buffer.
> That won't be suitable for 'virsh dump' then, because we need this to
> work when the guest OS is in a crashed/non-responsive state, and so
> we cna't rely on talking to it.

Note that the guest can generate the buffer before it crashes.

Paolo

[Qemu-devel] windows crash dump header. was: Re: [Question] dump memory when host pci device is used by guest

[Alon Levy]

On Mon, Oct 10, 2011 at 01:09:17PM +0200, Paolo Bonzini wrote:
> On 10/10/2011 01:04 PM, Daniel P. Berrange wrote:
> >On Mon, Oct 10, 2011 at 12:48:52PM +0200, Paolo Bonzini wrote:
> >>>  On 10/10/2011 12:21 PM, Alon Levy wrote:
> >>>>>  >>    A core file would be that format - for direct gdb processing. No
> >>>>>  >>    proprietary re-inventions please.
> >>>>  >
> >>>>  >Just a note: A core file to windows core dump file would be nice for
> >>>>  >windows guest crashes.
> >>>
> >>>  That requires cooperation from a kernel driver in the Windows guest.
> >>>  The driver must call KeInitializeCrashDumpHeader and write somewhere
> >>>  the physical address of the buffer.
> >That won't be suitable for 'virsh dump' then, because we need this to
> >work when the guest OS is in a crashed/non-responsive state, and so
> >we cna't rely on talking to it.
> 
> Note that the guest can generate the buffer before it crashes.
> 

Thanks. The contents seem to be undocumented (stubbed in reactos).
I did find a link to a tool made for Xen, but it mentions being broken
with Windows 2003 (where KeInitializeCrashDumpHeader was introduced).
 http://www.osronline.com/showThread.CFM?link=203490


> Paolo
> 

Re: [Qemu-devel] windows crash dump header. was: Re: [Question] dump memory when host pci device is used by guest

[Paolo Bonzini]

On 10/10/2011 02:00 PM, Alon Levy wrote:
>> >  Note that the guest can generate the buffer before it crashes.
>> >
> Thanks. The contents seem to be undocumented (stubbed in reactos).

Those lazy reactos authors. :)

32-bit:

http://www.google.com/codesearch#s5CWGGZtI6g/trunk/Volatility/vtypes.py&q=MajorVersion&exact_package=http://volatility.googlecode.com/svn&ct=rc&cd=1

64-bit:

https://singularity.svn.codeplex.com/svn/base/Windows/Inc/Dump.h

Looks like most of the fields can be made up at crash time, with some 
luck the others are not needed for a basic debugging session.

Paolo

Re: [Qemu-devel] windows crash dump header. was: Re: [Question] dump memory when host pci device is used by guest

[Alon Levy]

On Mon, Oct 10, 2011 at 02:38:35PM +0200, Paolo Bonzini wrote:
> On 10/10/2011 02:00 PM, Alon Levy wrote:
> >>>  Note that the guest can generate the buffer before it crashes.
> >>>
> >Thanks. The contents seem to be undocumented (stubbed in reactos).
> 
> Those lazy reactos authors. :)
> 
> 32-bit:
> 
> http://www.google.com/codesearch#s5CWGGZtI6g/trunk/Volatility/vtypes.py&q=MajorVersion&exact_package=http://volatility.googlecode.com/svn&ct=rc&cd=1
> 
> 64-bit:
> 
> https://singularity.svn.codeplex.com/svn/base/Windows/Inc/Dump.h
> 
> Looks like most of the fields can be made up at crash time, with
> some luck the others are not needed for a basic debugging session.
> 

Great. Nice that singularity exposes stuff not in WinDDK.

> Paolo
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/10/2011 06:19 PM, Daniel P. Berrange Write:
> On Mon, Oct 10, 2011 at 11:34:44AM +0200, Jan Kiszka wrote:
>> On 2011-10-10 11:10, Daniel P. Berrange wrote:
>>> On Mon, Oct 10, 2011 at 10:08:26AM +0100, Daniel P. Berrange wrote:
>>>> On Mon, Oct 10, 2011 at 10:21:02AM +0800, Wen Congyang wrote:
>>>>> At 10/09/2011 06:23 PM, Richard W.M. Jones Write:
>>>>>> On Sun, Oct 09, 2011 at 10:49:57AM +0200, Jan Kiszka wrote:
>>>>>>> As explained in the other replies: It is way more future-proof to use an
>>>>>>> interface for this which was designed for it (remote gdb) instead of
>>>>>>> artificially relaxing reasonable constraints of the migration mechanism
>>>>>>> plus having to follow that format with the post-processing tool.
>>>>>>
>>>>>> Any interface that isn't "get this information off my production
>>>>>> server *now*" so that I can get the server restarted, and send it to
>>>>>> an expert to analyse -- is a poor interface, whether it was designed
>>>>>> like that or not.  Perhaps we don't have the right interface at all,
>>>>>> but remote gdb is not it.
>>>>>
>>>>> What about the following idea?
>>>>>
>>>>> Introduce a new monitor command named dump, and this command accepts a filename.
>>>>> We can use almost all migration's code. We use this command to dump guest's
>>>>> memory, so there is no need to check whether the guest has a unmigratable device.
>>>>
>>>> I think it would be a good idea of QEMU had a dedicated 'dump' command
>>>> for this purpose, even if it was just an alias for 'migrate' initially.
>>>> I have never really liked the fact that we abuse the 'migrate' command
>>>> to generate a core dump. The resulting data file from this is more
>>>> complex than it really needs to be, causing complexity for post-processing
>>>> it. The needs of migration, are not entirely aligned with the needs of
>>>> core dumping in the long term, so we should allow the possibility of
>>>> their impls diverging without impacting apps using them.
>>>>
>>>> So adding a 'dump' command which wrote out data in a format that was
>>>> optimized for offline processing by tools like 'crash' (or the windows
>>>> equivalent) would be a good improvement, even if it just reuses the
>>>> migrate code for now.
>>>
>>> The other reason why it would be good, is that we would then have a clearly
>>> defined standard "QEMU dump format", instead of "libvirt dump format for QEMU"
>>
>> A core file would be that format - for direct gdb processing. No
>> proprietary re-inventions please.
> 
> I have no personal attachment to any particular format, so if the standard
> core file format is possible, then we should definitely try to use it in
> QEMU.

I do not know whether there is such standard format. But IIRC, the format of kdump,
netdump, and diskdump is very similar.

If we want to use such format in qemu, I think we can not do live dump in libvirt.

Thanks
Wen Congyang

> 
> 
> Daniel

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/10/2011 05:34 PM, Jan Kiszka Write:
> On 2011-10-10 11:02, Daniel P. Berrange wrote:

> Run gdb with "set debug remote 1" and watch the communication, it is not
> that complex. But a dump command is probably simpler for those
> scenarios, I agree.

We have determined to introduce a new command dump for such scenarios.
Now, let's to discuss the arguments for this command:

Here is the arguments for command migrate:
    {    
        .name       = "migrate",
        .args_type  = "detach:-d,blk:-b,inc:-i,uri:s",
        .params     = "[-d] [-b] [-i] uri",
...
    },   

I think the arguments -b and -i are not required for dump, and -d is useful
for dump.

We can pass the file's path or fd to qemu instead of uri.

So here is my image for command dump(reuse migration's code, not generate a standard
format core):
    {
        .name       = "dump",
        .args_type  = "detach:-d,file:s",
        .params     = "[-d] file",
...
    },

The argument can be 'file:file's path' or 'fd:fd's name'

Any advice is welcome.

Thanks
Wen Congyang
> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

On 2011-10-11 04:20, Wen Congyang wrote:
>>>> The other reason why it would be good, is that we would then have a clearly
>>>> defined standard "QEMU dump format", instead of "libvirt dump format for QEMU"
>>>
>>> A core file would be that format - for direct gdb processing. No
>>> proprietary re-inventions please.
>>
>> I have no personal attachment to any particular format, so if the standard
>> core file format is possible, then we should definitely try to use it in
>> QEMU.
> 
> I do not know whether there is such standard format. But IIRC, the format of kdump,
> netdump, and diskdump is very similar.
> 
> If we want to use such format in qemu, I think we can not do live dump in libvirt.

You said you wanted to perform crash dumps - so there hardly anything
alive at this point.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/11/2011 02:58 PM, Jan Kiszka Write:
> On 2011-10-11 04:20, Wen Congyang wrote:
>>>>> The other reason why it would be good, is that we would then have a clearly
>>>>> defined standard "QEMU dump format", instead of "libvirt dump format for QEMU"
>>>>
>>>> A core file would be that format - for direct gdb processing. No
>>>> proprietary re-inventions please.
>>>
>>> I have no personal attachment to any particular format, so if the standard
>>> core file format is possible, then we should definitely try to use it in
>>> QEMU.
>>
>> I do not know whether there is such standard format. But IIRC, the format of kdump,
>> netdump, and diskdump is very similar.
>>
>> If we want to use such format in qemu, I think we can not do live dump in libvirt.
> 
> You said you wanted to perform crash dumps - so there hardly anything
> alive at this point.

The purpose of 'virsh dump' is for crash dumps, but we can not prevent the user doing
it when the kernel is alive.

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

Hi, Jan Kiszka

At 10/10/2011 05:34 PM, Jan Kiszka Write:
> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:

> 
> Run gdb with "set debug remote 1" and watch the communication, it is not
> that complex. But a dump command is probably simpler for those
> scenarios, I agree.

I have implemented the command dump and reuse migration's code. But I meet a problem
when I test it.

My qemu-kvm's tree is not updated about 2 months ago, because kernel.org is down, and
I forgot to pull from github.

After I pull it from github, I find the following changes:
@@ -1523,9 +1523,7 @@ static void assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
 
 static const VMStateDescription vmstate_assigned_device = {
     .name = "pci-assign",
-    .fields = (VMStateField []) {
-        VMSTATE_END_OF_LIST()
-    }
+    .unmigratable = 1,
 };
 
 static void reset_assigned_device(DeviceState *dev)

Why do you remove fields from vmstate_assigned_device?
It is useful for dump because it does not check unmigratable. If vmstate_assigned_device
does not contain .fields, qemu will crash in vmstate_save_state().

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 09:15, Wen Congyang wrote:
> Hi, Jan Kiszka
> 
> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
> 
>>
>> Run gdb with "set debug remote 1" and watch the communication, it is not
>> that complex. But a dump command is probably simpler for those
>> scenarios, I agree.
> 
> I have implemented the command dump and reuse migration's code. But I meet a problem
> when I test it.

Using migration code for dump is most probably the wrong approach as you
saw through that conflict. All you need are the register states and the
RAM. Reuse gdbstub services for this.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Daniel P. Berrange]

On Tue, Oct 18, 2011 at 03:15:29PM +0800, Wen Congyang wrote:
> Hi, Jan Kiszka
> 
> At 10/10/2011 05:34 PM, Jan Kiszka Write:
> > On 2011-10-10 11:02, Daniel P. Berrange wrote:
> >> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
> 
> > 
> > Run gdb with "set debug remote 1" and watch the communication, it is not
> > that complex. But a dump command is probably simpler for those
> > scenarios, I agree.
> 
> I have implemented the command dump and reuse migration's code. But I meet a problem
> when I test it.
> 
> My qemu-kvm's tree is not updated about 2 months ago, because kernel.org is down, and
> I forgot to pull from github.
> 
> After I pull it from github, I find the following changes:
> @@ -1523,9 +1523,7 @@ static void assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
>  
>  static const VMStateDescription vmstate_assigned_device = {
>      .name = "pci-assign",
> -    .fields = (VMStateField []) {
> -        VMSTATE_END_OF_LIST()
> -    }
> +    .unmigratable = 1,
>  };
>  
>  static void reset_assigned_device(DeviceState *dev)
> 
> Why do you remove fields from vmstate_assigned_device?
> It is useful for dump because it does not check unmigratable. If vmstate_assigned_device
> does not contain .fields, qemu will crash in vmstate_save_state().

Given that '.fields' is allowed to be NULL for some devices, I'd say
even for normal migration, QEMU should be checking for NULL in the
vmstate_save_state() code. This would prevent QEMU crashes in the case
where someone removed the .unmigratable member, but forgot to add back
a .fields member.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/18/2011 03:52 PM, Jan Kiszka Write:
> On 2011-10-18 09:15, Wen Congyang wrote:
>> Hi, Jan Kiszka
>>
>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>
>>>
>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>> that complex. But a dump command is probably simpler for those
>>> scenarios, I agree.
>>
>> I have implemented the command dump and reuse migration's code. But I meet a problem
>> when I test it.
> 
> Using migration code for dump is most probably the wrong approach as you
> saw through that conflict. All you need are the register states and the
> RAM. Reuse gdbstub services for this.

Hmm, if the migration code can not be reused, I think we should define a new
qemu's vmcore format, and add some codes into crash to support such format.

I will read gdbstub services's code now.

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 09:58, Daniel P. Berrange wrote:
> On Tue, Oct 18, 2011 at 03:15:29PM +0800, Wen Congyang wrote:
>> Hi, Jan Kiszka
>>
>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>
>>>
>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>> that complex. But a dump command is probably simpler for those
>>> scenarios, I agree.
>>
>> I have implemented the command dump and reuse migration's code. But I meet a problem
>> when I test it.
>>
>> My qemu-kvm's tree is not updated about 2 months ago, because kernel.org is down, and
>> I forgot to pull from github.
>>
>> After I pull it from github, I find the following changes:
>> @@ -1523,9 +1523,7 @@ static void assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
>>  
>>  static const VMStateDescription vmstate_assigned_device = {
>>      .name = "pci-assign",
>> -    .fields = (VMStateField []) {
>> -        VMSTATE_END_OF_LIST()
>> -    }
>> +    .unmigratable = 1,
>>  };
>>  
>>  static void reset_assigned_device(DeviceState *dev)
>>
>> Why do you remove fields from vmstate_assigned_device?
>> It is useful for dump because it does not check unmigratable. If vmstate_assigned_device
>> does not contain .fields, qemu will crash in vmstate_save_state().
> 
> Given that '.fields' is allowed to be NULL for some devices, I'd say
> even for normal migration, QEMU should be checking for NULL in the
> vmstate_save_state() code. This would prevent QEMU crashes in the case
> where someone removed the .unmigratable member, but forgot to add back
> a .fields member.

That crash wouldn't be bad because removinb unmigratable without adding
proper fields is almost always a bug.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 09:58, Wen Congyang wrote:
> At 10/18/2011 03:52 PM, Jan Kiszka Write:
>> On 2011-10-18 09:15, Wen Congyang wrote:
>>> Hi, Jan Kiszka
>>>
>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>>
>>>>
>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>>> that complex. But a dump command is probably simpler for those
>>>> scenarios, I agree.
>>>
>>> I have implemented the command dump and reuse migration's code. But I meet a problem
>>> when I test it.
>>
>> Using migration code for dump is most probably the wrong approach as you
>> saw through that conflict. All you need are the register states and the
>> RAM. Reuse gdbstub services for this.
> 
> Hmm, if the migration code can not be reused, I think we should define a new
> qemu's vmcore format, and add some codes into crash to support such format.

Please try to avoid defining something new. Unless there is a striking
reason, standard gdb core files should be generated so that you can load
the dump directly into gdb for analysis.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 10:17, Jan Kiszka wrote:
> On 2011-10-18 09:58, Daniel P. Berrange wrote:
>> On Tue, Oct 18, 2011 at 03:15:29PM +0800, Wen Congyang wrote:
>>> Hi, Jan Kiszka
>>>
>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>>
>>>>
>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>>> that complex. But a dump command is probably simpler for those
>>>> scenarios, I agree.
>>>
>>> I have implemented the command dump and reuse migration's code. But I meet a problem
>>> when I test it.
>>>
>>> My qemu-kvm's tree is not updated about 2 months ago, because kernel.org is down, and
>>> I forgot to pull from github.
>>>
>>> After I pull it from github, I find the following changes:
>>> @@ -1523,9 +1523,7 @@ static void assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
>>>  
>>>  static const VMStateDescription vmstate_assigned_device = {
>>>      .name = "pci-assign",
>>> -    .fields = (VMStateField []) {
>>> -        VMSTATE_END_OF_LIST()
>>> -    }
>>> +    .unmigratable = 1,
>>>  };
>>>  
>>>  static void reset_assigned_device(DeviceState *dev)
>>>
>>> Why do you remove fields from vmstate_assigned_device?
>>> It is useful for dump because it does not check unmigratable. If vmstate_assigned_device
>>> does not contain .fields, qemu will crash in vmstate_save_state().
>>
>> Given that '.fields' is allowed to be NULL for some devices, I'd say
>> even for normal migration, QEMU should be checking for NULL in the
>> vmstate_save_state() code. This would prevent QEMU crashes in the case
>> where someone removed the .unmigratable member, but forgot to add back
>> a .fields member.
> 
> That crash wouldn't be bad because removinb unmigratable without adding
> proper fields is almost always a bug.

Err, s/almost//.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Daniel P. Berrange]

On Tue, Oct 18, 2011 at 10:17:27AM +0200, Jan Kiszka wrote:
> On 2011-10-18 09:58, Daniel P. Berrange wrote:
> > On Tue, Oct 18, 2011 at 03:15:29PM +0800, Wen Congyang wrote:
> >> Hi, Jan Kiszka
> >>
> >> At 10/10/2011 05:34 PM, Jan Kiszka Write:
> >>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
> >>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
> >>
> >>>
> >>> Run gdb with "set debug remote 1" and watch the communication, it is not
> >>> that complex. But a dump command is probably simpler for those
> >>> scenarios, I agree.
> >>
> >> I have implemented the command dump and reuse migration's code. But I meet a problem
> >> when I test it.
> >>
> >> My qemu-kvm's tree is not updated about 2 months ago, because kernel.org is down, and
> >> I forgot to pull from github.
> >>
> >> After I pull it from github, I find the following changes:
> >> @@ -1523,9 +1523,7 @@ static void assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
> >>  
> >>  static const VMStateDescription vmstate_assigned_device = {
> >>      .name = "pci-assign",
> >> -    .fields = (VMStateField []) {
> >> -        VMSTATE_END_OF_LIST()
> >> -    }
> >> +    .unmigratable = 1,
> >>  };
> >>  
> >>  static void reset_assigned_device(DeviceState *dev)
> >>
> >> Why do you remove fields from vmstate_assigned_device?
> >> It is useful for dump because it does not check unmigratable. If vmstate_assigned_device
> >> does not contain .fields, qemu will crash in vmstate_save_state().
> > 
> > Given that '.fields' is allowed to be NULL for some devices, I'd say
> > even for normal migration, QEMU should be checking for NULL in the
> > vmstate_save_state() code. This would prevent QEMU crashes in the case
> > where someone removed the .unmigratable member, but forgot to add back
> > a .fields member.
> 
> That crash wouldn't be bad because removinb unmigratable without adding
> proper fields is almost always a bug.

This is the kind of example of why QEMU is too crashy in general. We
should be defensive against such bugs, by checking that preconditions
are valid in the code, even if we don't expect them to happen often.


Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/18/2011 04:19 PM, Jan Kiszka Write:
> On 2011-10-18 09:58, Wen Congyang wrote:
>> At 10/18/2011 03:52 PM, Jan Kiszka Write:
>>> On 2011-10-18 09:15, Wen Congyang wrote:
>>>> Hi, Jan Kiszka
>>>>
>>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>>>
>>>>>
>>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>>>> that complex. But a dump command is probably simpler for those
>>>>> scenarios, I agree.
>>>>
>>>> I have implemented the command dump and reuse migration's code. But I meet a problem
>>>> when I test it.
>>>
>>> Using migration code for dump is most probably the wrong approach as you
>>> saw through that conflict. All you need are the register states and the
>>> RAM. Reuse gdbstub services for this.
>>
>> Hmm, if the migration code can not be reused, I think we should define a new
>> qemu's vmcore format, and add some codes into crash to support such format.
> 
> Please try to avoid defining something new. Unless there is a striking
> reason, standard gdb core files should be generated so that you can load
> the dump directly into gdb for analysis.

I am not sure whehter the standard gdb core files can not be analyzed by crash.
If not, I think we should define something new because it's easier to use
crash than gdb to analyze the core files.

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 10:25, Wen Congyang wrote:
> At 10/18/2011 04:19 PM, Jan Kiszka Write:
>> On 2011-10-18 09:58, Wen Congyang wrote:
>>> At 10/18/2011 03:52 PM, Jan Kiszka Write:
>>>> On 2011-10-18 09:15, Wen Congyang wrote:
>>>>> Hi, Jan Kiszka
>>>>>
>>>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>>>>
>>>>>>
>>>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>>>>> that complex. But a dump command is probably simpler for those
>>>>>> scenarios, I agree.
>>>>>
>>>>> I have implemented the command dump and reuse migration's code. But I meet a problem
>>>>> when I test it.
>>>>
>>>> Using migration code for dump is most probably the wrong approach as you
>>>> saw through that conflict. All you need are the register states and the
>>>> RAM. Reuse gdbstub services for this.
>>>
>>> Hmm, if the migration code can not be reused, I think we should define a new
>>> qemu's vmcore format, and add some codes into crash to support such format.
>>
>> Please try to avoid defining something new. Unless there is a striking
>> reason, standard gdb core files should be generated so that you can load
>> the dump directly into gdb for analysis.
> 
> I am not sure whehter the standard gdb core files can not be analyzed by crash.
> If not, I think we should define something new because it's easier to use
> crash than gdb to analyze the core files.

gdb allows you to walk up the frame and print variables (globals &
local) etc.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 10:25, Daniel P. Berrange wrote:
> On Tue, Oct 18, 2011 at 10:17:27AM +0200, Jan Kiszka wrote:
>> On 2011-10-18 09:58, Daniel P. Berrange wrote:
>>> On Tue, Oct 18, 2011 at 03:15:29PM +0800, Wen Congyang wrote:
>>>> Hi, Jan Kiszka
>>>>
>>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>>>
>>>>>
>>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>>>> that complex. But a dump command is probably simpler for those
>>>>> scenarios, I agree.
>>>>
>>>> I have implemented the command dump and reuse migration's code. But I meet a problem
>>>> when I test it.
>>>>
>>>> My qemu-kvm's tree is not updated about 2 months ago, because kernel.org is down, and
>>>> I forgot to pull from github.
>>>>
>>>> After I pull it from github, I find the following changes:
>>>> @@ -1523,9 +1523,7 @@ static void assigned_dev_unregister_msix_mmio(AssignedDevice *dev)
>>>>  
>>>>  static const VMStateDescription vmstate_assigned_device = {
>>>>      .name = "pci-assign",
>>>> -    .fields = (VMStateField []) {
>>>> -        VMSTATE_END_OF_LIST()
>>>> -    }
>>>> +    .unmigratable = 1,
>>>>  };
>>>>  
>>>>  static void reset_assigned_device(DeviceState *dev)
>>>>
>>>> Why do you remove fields from vmstate_assigned_device?
>>>> It is useful for dump because it does not check unmigratable. If vmstate_assigned_device
>>>> does not contain .fields, qemu will crash in vmstate_save_state().
>>>
>>> Given that '.fields' is allowed to be NULL for some devices, I'd say
>>> even for normal migration, QEMU should be checking for NULL in the
>>> vmstate_save_state() code. This would prevent QEMU crashes in the case
>>> where someone removed the .unmigratable member, but forgot to add back
>>> a .fields member.
>>
>> That crash wouldn't be bad because removinb unmigratable without adding
>> proper fields is almost always a bug.
> 
> This is the kind of example of why QEMU is too crashy in general. We
> should be defensive against such bugs, by checking that preconditions
> are valid in the code, even if we don't expect them to happen often.

A serious bug in a device model is tricky to handle gracefully. Given
that this case is fairly clear (specifically for the code reviewer), I'm
not sure what defensive measures you want to apply and what benefit they
would provide.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 10:31, Wen Congyang wrote:
> At 10/18/2011 04:26 PM, Jan Kiszka Write:
>> On 2011-10-18 10:25, Wen Congyang wrote:
>>> At 10/18/2011 04:19 PM, Jan Kiszka Write:
>>>> On 2011-10-18 09:58, Wen Congyang wrote:
>>>>> At 10/18/2011 03:52 PM, Jan Kiszka Write:
>>>>>> On 2011-10-18 09:15, Wen Congyang wrote:
>>>>>>> Hi, Jan Kiszka
>>>>>>>
>>>>>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>>>>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>>>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>>>>>>> that complex. But a dump command is probably simpler for those
>>>>>>>> scenarios, I agree.
>>>>>>>
>>>>>>> I have implemented the command dump and reuse migration's code. But I meet a problem
>>>>>>> when I test it.
>>>>>>
>>>>>> Using migration code for dump is most probably the wrong approach as you
>>>>>> saw through that conflict. All you need are the register states and the
>>>>>> RAM. Reuse gdbstub services for this.
>>>>>
>>>>> Hmm, if the migration code can not be reused, I think we should define a new
>>>>> qemu's vmcore format, and add some codes into crash to support such format.
>>>>
>>>> Please try to avoid defining something new. Unless there is a striking
>>>> reason, standard gdb core files should be generated so that you can load
>>>> the dump directly into gdb for analysis.
>>>
>>> I am not sure whehter the standard gdb core files can not be analyzed by crash.
>>> If not, I think we should define something new because it's easier to use
>>> crash than gdb to analyze the core files.
>>
>> gdb allows you to walk up the frame and print variables (globals &
>> local) etc.
> 
> Crash uses gdb to provide common function, and you can also use all the gdb commands
> in crash.

That what's the added value here when I can use gdb directly?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/18/2011 04:26 PM, Jan Kiszka Write:
> On 2011-10-18 10:25, Wen Congyang wrote:
>> At 10/18/2011 04:19 PM, Jan Kiszka Write:
>>> On 2011-10-18 09:58, Wen Congyang wrote:
>>>> At 10/18/2011 03:52 PM, Jan Kiszka Write:
>>>>> On 2011-10-18 09:15, Wen Congyang wrote:
>>>>>> Hi, Jan Kiszka
>>>>>>
>>>>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>>>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>>>>>
>>>>>>>
>>>>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>>>>>> that complex. But a dump command is probably simpler for those
>>>>>>> scenarios, I agree.
>>>>>>
>>>>>> I have implemented the command dump and reuse migration's code. But I meet a problem
>>>>>> when I test it.
>>>>>
>>>>> Using migration code for dump is most probably the wrong approach as you
>>>>> saw through that conflict. All you need are the register states and the
>>>>> RAM. Reuse gdbstub services for this.
>>>>
>>>> Hmm, if the migration code can not be reused, I think we should define a new
>>>> qemu's vmcore format, and add some codes into crash to support such format.
>>>
>>> Please try to avoid defining something new. Unless there is a striking
>>> reason, standard gdb core files should be generated so that you can load
>>> the dump directly into gdb for analysis.
>>
>> I am not sure whehter the standard gdb core files can not be analyzed by crash.
>> If not, I think we should define something new because it's easier to use
>> crash than gdb to analyze the core files.
> 
> gdb allows you to walk up the frame and print variables (globals &
> local) etc.

Crash uses gdb to provide common function, and you can also use all the gdb commands
in crash.

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Richard W.M. Jones]

On Tue, Oct 18, 2011 at 10:31:10AM +0200, Jan Kiszka wrote:
> On 2011-10-18 10:31, Wen Congyang wrote:
> > At 10/18/2011 04:26 PM, Jan Kiszka Write:
> >> On 2011-10-18 10:25, Wen Congyang wrote:
> >>> At 10/18/2011 04:19 PM, Jan Kiszka Write:
> >>>> On 2011-10-18 09:58, Wen Congyang wrote:
> >>>>> At 10/18/2011 03:52 PM, Jan Kiszka Write:
> >>>>>> On 2011-10-18 09:15, Wen Congyang wrote:
> >>>>>>> Hi, Jan Kiszka
> >>>>>>>
> >>>>>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
> >>>>>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
> >>>>>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
> >>>>>>>> that complex. But a dump command is probably simpler for those
> >>>>>>>> scenarios, I agree.
> >>>>>>>
> >>>>>>> I have implemented the command dump and reuse migration's code. But I meet a problem
> >>>>>>> when I test it.
> >>>>>>
> >>>>>> Using migration code for dump is most probably the wrong approach as you
> >>>>>> saw through that conflict. All you need are the register states and the
> >>>>>> RAM. Reuse gdbstub services for this.
> >>>>>
> >>>>> Hmm, if the migration code can not be reused, I think we should define a new
> >>>>> qemu's vmcore format, and add some codes into crash to support such format.
> >>>>
> >>>> Please try to avoid defining something new. Unless there is a striking
> >>>> reason, standard gdb core files should be generated so that you can load
> >>>> the dump directly into gdb for analysis.
> >>>
> >>> I am not sure whehter the standard gdb core files can not be analyzed by crash.
> >>> If not, I think we should define something new because it's easier to use
> >>> crash than gdb to analyze the core files.
> >>
> >> gdb allows you to walk up the frame and print variables (globals &
> >> local) etc.
> > 
> > Crash uses gdb to provide common function, and you can also use all the gdb commands
> > in crash.
> 
> That what's the added value here when I can use gdb directly?

Crash can decode kernel structures, providing for example process
listings and debugging into userspace processes.  Crash actually
reuses gdb's code too, so it's kind of a superset.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 10:34, Richard W.M. Jones wrote:
> On Tue, Oct 18, 2011 at 10:31:10AM +0200, Jan Kiszka wrote:
>> On 2011-10-18 10:31, Wen Congyang wrote:
>>> At 10/18/2011 04:26 PM, Jan Kiszka Write:
>>>> On 2011-10-18 10:25, Wen Congyang wrote:
>>>>> At 10/18/2011 04:19 PM, Jan Kiszka Write:
>>>>>> On 2011-10-18 09:58, Wen Congyang wrote:
>>>>>>> At 10/18/2011 03:52 PM, Jan Kiszka Write:
>>>>>>>> On 2011-10-18 09:15, Wen Congyang wrote:
>>>>>>>>> Hi, Jan Kiszka
>>>>>>>>>
>>>>>>>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>>>>>>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>>>>>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>>>>>>>>> that complex. But a dump command is probably simpler for those
>>>>>>>>>> scenarios, I agree.
>>>>>>>>>
>>>>>>>>> I have implemented the command dump and reuse migration's code. But I meet a problem
>>>>>>>>> when I test it.
>>>>>>>>
>>>>>>>> Using migration code for dump is most probably the wrong approach as you
>>>>>>>> saw through that conflict. All you need are the register states and the
>>>>>>>> RAM. Reuse gdbstub services for this.
>>>>>>>
>>>>>>> Hmm, if the migration code can not be reused, I think we should define a new
>>>>>>> qemu's vmcore format, and add some codes into crash to support such format.
>>>>>>
>>>>>> Please try to avoid defining something new. Unless there is a striking
>>>>>> reason, standard gdb core files should be generated so that you can load
>>>>>> the dump directly into gdb for analysis.
>>>>>
>>>>> I am not sure whehter the standard gdb core files can not be analyzed by crash.
>>>>> If not, I think we should define something new because it's easier to use
>>>>> crash than gdb to analyze the core files.
>>>>
>>>> gdb allows you to walk up the frame and print variables (globals &
>>>> local) etc.
>>>
>>> Crash uses gdb to provide common function, and you can also use all the gdb commands
>>> in crash.
>>
>> That what's the added value here when I can use gdb directly?
> 
> Crash can decode kernel structures, providing for example process
> listings and debugging into userspace processes.  Crash actually
> reuses gdb's code too, so it's kind of a superset.

Yeah, I see. Could also be solved via gdb scripts, but crash is already
there.

But let's see if the formats actually differ. In the end, crash is just
parsing the same information that also gdb sees.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 10:36, Jan Kiszka wrote:
>> Crash can decode kernel structures, providing for example process
>> listings and debugging into userspace processes.  Crash actually
>> reuses gdb's code too, so it's kind of a superset.
> 
> Yeah, I see. Could also be solved via gdb scripts, but crash is already
> there.

[ BTW, crash is for the dead. But having those features in form of gdb
scripts would also allow live system analysis. Looks like it's worth
obsoleting crash on the long run. ]

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/18/2011 04:36 PM, Jan Kiszka Write:
> On 2011-10-18 10:34, Richard W.M. Jones wrote:
> 
> Yeah, I see. Could also be solved via gdb scripts, but crash is already
> there.
> 
> But let's see if the formats actually differ. In the end, crash is just
> parsing the same information that also gdb sees.

I think the format can be similar with diskdump/kdump/netdump:

dump_header:     1 block
sub header:      n blocks(n is stored in dump_header)
bitmap:          m blocks(2m is stored in dump_header)
dumpable bitmap: m blocks
memory data(We can know whether a page is stored in the core by bitmap and dumpable bitmap)

The format of dump header(It's like kdump/diskdump):
struct disk_dump_header {
	char			signature[SIG_LEN];	/* = "QEMU    " */
	int			header_version; /* Dump header version */
	struct new_utsname	utsname;	/* copy of system_utsname */
	struct timeval		timestamp;	/* Time stamp */
	unsigned int		status;
	int			block_size;	/* Size of a block in byte */
	int			sub_hdr_size;	/* Size of arch dependent
						   header in blocks */
	unsigned int		bitmap_blocks;	/* Size of Memory bitmap in
						   block */
	unsigned int		max_mapnr;	/* = max_mapnr */
	unsigned int		total_ram_blocks;/* Number of blocks should be
						   written */
	unsigned int		device_blocks;	/* Number of total blocks in
						 * the dump device */
	unsigned int		written_blocks; /* Number of written blocks */
	unsigned int		current_cpu;	/* CPU# which handles dump */
	int			nr_cpus;	/* Number of CPUs */
};

The sub header can contains all registers's value on each vcpu, and other information,
for example:
struct qemu_sub_header {
	unsigned long	start_pfn;
	unsigned long	end_pfn;
	off_t		offset_note;
	unsigned long	size_note;
};

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 11:43, Wen Congyang wrote:
> At 10/18/2011 04:36 PM, Jan Kiszka Write:
>> On 2011-10-18 10:34, Richard W.M. Jones wrote:
>>
>> Yeah, I see. Could also be solved via gdb scripts, but crash is already
>> there.
>>
>> But let's see if the formats actually differ. In the end, crash is just
>> parsing the same information that also gdb sees.
> 
> I think the format can be similar with diskdump/kdump/netdump:
> 
> dump_header:     1 block
> sub header:      n blocks(n is stored in dump_header)
> bitmap:          m blocks(2m is stored in dump_header)
> dumpable bitmap: m blocks
> memory data(We can know whether a page is stored in the core by bitmap and dumpable bitmap)
> 
> The format of dump header(It's like kdump/diskdump):
> struct disk_dump_header {
> 	char			signature[SIG_LEN];	/* = "QEMU    " */
> 	int			header_version; /* Dump header version */
> 	struct new_utsname	utsname;	/* copy of system_utsname */
> 	struct timeval		timestamp;	/* Time stamp */
> 	unsigned int		status;
> 	int			block_size;	/* Size of a block in byte */
> 	int			sub_hdr_size;	/* Size of arch dependent
> 						   header in blocks */
> 	unsigned int		bitmap_blocks;	/* Size of Memory bitmap in
> 						   block */
> 	unsigned int		max_mapnr;	/* = max_mapnr */
> 	unsigned int		total_ram_blocks;/* Number of blocks should be
> 						   written */
> 	unsigned int		device_blocks;	/* Number of total blocks in
> 						 * the dump device */
> 	unsigned int		written_blocks; /* Number of written blocks */
> 	unsigned int		current_cpu;	/* CPU# which handles dump */
> 	int			nr_cpus;	/* Number of CPUs */
> };
> 
> The sub header can contains all registers's value on each vcpu, and other information,
> for example:
> struct qemu_sub_header {
> 	unsigned long	start_pfn;
> 	unsigned long	end_pfn;
> 	off_t		offset_note;
> 	unsigned long	size_note;
> };

So is this a standard format or only "similar" to something? Which tools
will understand it out-of-the-box? If it's not standard, why?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Paolo Bonzini]

On 10/18/2011 10:39 AM, Jan Kiszka wrote:
>> >
>> >  Yeah, I see. Could also be solved via gdb scripts, but crash is already
>> >  there.
> [ BTW, crash is for the dead. But having those features in form of gdb
> scripts would also allow live system analysis. Looks like it's worth
> obsoleting crash on the long run. ]

Crash can also do live system analysis. :)

Paolo

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Christoph Hellwig]

On Tue, Oct 18, 2011 at 12:41:10PM +0200, Paolo Bonzini wrote:
> On 10/18/2011 10:39 AM, Jan Kiszka wrote:
>>> >
>>> >  Yeah, I see. Could also be solved via gdb scripts, but crash is already
>>> >  there.
>> [ BTW, crash is for the dead. But having those features in form of gdb
>> scripts would also allow live system analysis. Looks like it's worth
>> obsoleting crash on the long run. ]
>
> Crash can also do live system analysis. :)

crash in fact is gdb - with a lot of additional commands.

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 12:41, Paolo Bonzini wrote:
> On 10/18/2011 10:39 AM, Jan Kiszka wrote:
>>>>
>>>>  Yeah, I see. Could also be solved via gdb scripts, but crash is already
>>>>  there.
>> [ BTW, crash is for the dead. But having those features in form of gdb
>> scripts would also allow live system analysis. Looks like it's worth
>> obsoleting crash on the long run. ]
> 
> Crash can also do live system analysis. :)

Does it work via remote interface? And it's still the wrong way around:
You have to append "gdb" to the majority of command when doing debugging
instead of issuing additional analysis commands from the gdb shell.
That's understandable given the limited scripting power of old gdbs. But
that's now history thanks to its python interface.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 12:42, Christoph Hellwig wrote:
> On Tue, Oct 18, 2011 at 12:41:10PM +0200, Paolo Bonzini wrote:
>> On 10/18/2011 10:39 AM, Jan Kiszka wrote:
>>>>>
>>>>>  Yeah, I see. Could also be solved via gdb scripts, but crash is already
>>>>>  there.
>>> [ BTW, crash is for the dead. But having those features in form of gdb
>>> scripts would also allow live system analysis. Looks like it's worth
>>> obsoleting crash on the long run. ]
>>
>> Crash can also do live system analysis. :)
> 
> crash in fact is gdb - with a lot of additional commands.

Is there a trick to make it work against a gdbserver?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Christoph Hellwig]

On Tue, Oct 18, 2011 at 02:47:39PM +0200, Jan Kiszka wrote:
> >> Crash can also do live system analysis. :)
> > 
> > crash in fact is gdb - with a lot of additional commands.
> 
> Is there a trick to make it work against a gdbserver?

No idea.  I just noticed that crash was built around a gdb source
tree and additional files when I had to hack it for a new kernel
version a while ago.

The optimum would be if all the crash changes were merged back
into upstream gdb, but I'm not sure the gdb community would like
to see all that very linux kernel specific code (especially given that it's
very version dependent)

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 15:08, Christoph Hellwig wrote:
> On Tue, Oct 18, 2011 at 02:47:39PM +0200, Jan Kiszka wrote:
>>>> Crash can also do live system analysis. :)
>>>
>>> crash in fact is gdb - with a lot of additional commands.
>>
>> Is there a trick to make it work against a gdbserver?
> 
> No idea.  I just noticed that crash was built around a gdb source
> tree and additional files when I had to hack it for a new kernel
> version a while ago.
> 
> The optimum would be if all the crash changes were merged back
> into upstream gdb, but I'm not sure the gdb community would like
> to see all that very linux kernel specific code (especially given that it's
> very version dependent)

That's why I'm advocating gdb scripts, maybe maintained inside the
kernel tree.

But I need to look closer as well to see if vanilla gdb is already able
to provide all registers used by crash so far (surely not all required
for perfect debugging, but that's a long-known issue).

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Richard W.M. Jones]

On Tue, Oct 18, 2011 at 12:47:23PM +0200, Jan Kiszka wrote:
> On 2011-10-18 12:41, Paolo Bonzini wrote:
> > On 10/18/2011 10:39 AM, Jan Kiszka wrote:
> >>>>
> >>>>  Yeah, I see. Could also be solved via gdb scripts, but crash is already
> >>>>  there.
> >> [ BTW, crash is for the dead. But having those features in form of gdb
> >> scripts would also allow live system analysis. Looks like it's worth
> >> obsoleting crash on the long run. ]
> > 
> > Crash can also do live system analysis. :)
> 
> Does it work via remote interface? And it's still the wrong way around:
> You have to append "gdb" to the majority of command when doing debugging
> instead of issuing additional analysis commands from the gdb shell.
> That's understandable given the limited scripting power of old gdbs. But
> that's now history thanks to its python interface.

Before this conversation heads any further into the realm of
absurdity, let's get back to the key requirements for production
systems:

(A) No development tools can be installed.

(B) The core file must be removed from the production system quickly
and analyzed by on another machine (indeed, often by experts from
another company entirely).

Anything that involves live debugging using GDB is simply not
acceptable for production systems, whether you think that's right or
not.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 15:30, Richard W.M. Jones wrote:
> On Tue, Oct 18, 2011 at 12:47:23PM +0200, Jan Kiszka wrote:
>> On 2011-10-18 12:41, Paolo Bonzini wrote:
>>> On 10/18/2011 10:39 AM, Jan Kiszka wrote:
>>>>>>
>>>>>>  Yeah, I see. Could also be solved via gdb scripts, but crash is already
>>>>>>  there.
>>>> [ BTW, crash is for the dead. But having those features in form of gdb
>>>> scripts would also allow live system analysis. Looks like it's worth
>>>> obsoleting crash on the long run. ]
>>>
>>> Crash can also do live system analysis. :)
>>
>> Does it work via remote interface? And it's still the wrong way around:
>> You have to append "gdb" to the majority of command when doing debugging
>> instead of issuing additional analysis commands from the gdb shell.
>> That's understandable given the limited scripting power of old gdbs. But
>> that's now history thanks to its python interface.
> 
> Before this conversation heads any further into the realm of
> absurdity, let's get back to the key requirements for production
> systems:
> 
> (A) No development tools can be installed.
> 
> (B) The core file must be removed from the production system quickly
> and analyzed by on another machine (indeed, often by experts from
> another company entirely).
> 
> Anything that involves live debugging using GDB is simply not
> acceptable for production systems, whether you think that's right or
> not.

My memory is usually that bad, but not in this case. :)

This discussion should also work out a bigger picture than just your
(valid) scenario. We should look at meeting that requirements but also
see what would be a useful foundation for future extensions like having
post-crash analysis and live debugging at a similar feature level with
very similar or (much better) identical tools.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

On 10/18/2011 06:28 PM, Jan Kiszka wrote:
> On 2011-10-18 11:43, Wen Congyang wrote:
>> At 10/18/2011 04:36 PM, Jan Kiszka Write:
>>> On 2011-10-18 10:34, Richard W.M. Jones wrote:
>>>
>>> Yeah, I see. Could also be solved via gdb scripts, but crash is already
>>> there.
>>>
>>> But let's see if the formats actually differ. In the end, crash is just
>>> parsing the same information that also gdb sees.
>>
>> I think the format can be similar with diskdump/kdump/netdump:
>>
>> dump_header:     1 block
>> sub header:      n blocks(n is stored in dump_header)
>> bitmap:          m blocks(2m is stored in dump_header)
>> dumpable bitmap: m blocks
>> memory data(We can know whether a page is stored in the core by bitmap and dumpable bitmap)
>>
>> The format of dump header(It's like kdump/diskdump):
>> struct disk_dump_header {
>> 	char			signature[SIG_LEN];	/* = "QEMU    " */
>> 	int			header_version; /* Dump header version */
>> 	struct new_utsname	utsname;	/* copy of system_utsname */
>> 	struct timeval		timestamp;	/* Time stamp */
>> 	unsigned int		status;
>> 	int			block_size;	/* Size of a block in byte */
>> 	int			sub_hdr_size;	/* Size of arch dependent
>> 						   header in blocks */
>> 	unsigned int		bitmap_blocks;	/* Size of Memory bitmap in
>> 						   block */
>> 	unsigned int		max_mapnr;	/* = max_mapnr */
>> 	unsigned int		total_ram_blocks;/* Number of blocks should be
>> 						   written */
>> 	unsigned int		device_blocks;	/* Number of total blocks in
>> 						 * the dump device */
>> 	unsigned int		written_blocks; /* Number of written blocks */
>> 	unsigned int		current_cpu;	/* CPU# which handles dump */
>> 	int			nr_cpus;	/* Number of CPUs */
>> };
>>
>> The sub header can contains all registers's value on each vcpu, and other information,
>> for example:
>> struct qemu_sub_header {
>> 	unsigned long	start_pfn;
>> 	unsigned long	end_pfn;
>> 	off_t		offset_note;
>> 	unsigned long	size_note;
>> };
>
> So is this a standard format or only "similar" to something? Which tools
> will understand it out-of-the-box? If it's not standard, why?

Only similar to something, and we can add a little codes into crash to 
support such format.

If you have a better format, please tell me.

Thanks
Wen Congyang

>
> Jan
>

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 15:51, Wen Congyang wrote:
> On 10/18/2011 06:28 PM, Jan Kiszka wrote:
>> On 2011-10-18 11:43, Wen Congyang wrote:
>>> At 10/18/2011 04:36 PM, Jan Kiszka Write:
>>>> On 2011-10-18 10:34, Richard W.M. Jones wrote:
>>>>
>>>> Yeah, I see. Could also be solved via gdb scripts, but crash is already
>>>> there.
>>>>
>>>> But let's see if the formats actually differ. In the end, crash is just
>>>> parsing the same information that also gdb sees.
>>>
>>> I think the format can be similar with diskdump/kdump/netdump:
>>>
>>> dump_header:     1 block
>>> sub header:      n blocks(n is stored in dump_header)
>>> bitmap:          m blocks(2m is stored in dump_header)
>>> dumpable bitmap: m blocks
>>> memory data(We can know whether a page is stored in the core by bitmap and dumpable bitmap)
>>>
>>> The format of dump header(It's like kdump/diskdump):
>>> struct disk_dump_header {
>>> 	char			signature[SIG_LEN];	/* = "QEMU    " */
>>> 	int			header_version; /* Dump header version */
>>> 	struct new_utsname	utsname;	/* copy of system_utsname */
>>> 	struct timeval		timestamp;	/* Time stamp */
>>> 	unsigned int		status;
>>> 	int			block_size;	/* Size of a block in byte */
>>> 	int			sub_hdr_size;	/* Size of arch dependent
>>> 						   header in blocks */
>>> 	unsigned int		bitmap_blocks;	/* Size of Memory bitmap in
>>> 						   block */
>>> 	unsigned int		max_mapnr;	/* = max_mapnr */
>>> 	unsigned int		total_ram_blocks;/* Number of blocks should be
>>> 						   written */
>>> 	unsigned int		device_blocks;	/* Number of total blocks in
>>> 						 * the dump device */
>>> 	unsigned int		written_blocks; /* Number of written blocks */
>>> 	unsigned int		current_cpu;	/* CPU# which handles dump */
>>> 	int			nr_cpus;	/* Number of CPUs */
>>> };
>>>
>>> The sub header can contains all registers's value on each vcpu, and other information,
>>> for example:
>>> struct qemu_sub_header {
>>> 	unsigned long	start_pfn;
>>> 	unsigned long	end_pfn;
>>> 	off_t		offset_note;
>>> 	unsigned long	size_note;
>>> };
>>
>> So is this a standard format or only "similar" to something? Which tools
>> will understand it out-of-the-box? If it's not standard, why?
> 
> Only similar to something, and we can add a little codes into crash to 
> support such format.
> 
> If you have a better format, please tell me.

The format crash already processes? What do you need in addition? That
should be discussed first, not the format details.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

On 10/18/2011 09:55 PM, Jan Kiszka wrote:
> On 2011-10-18 15:51, Wen Congyang wrote:
>> On 10/18/2011 06:28 PM, Jan Kiszka wrote:
>>> On 2011-10-18 11:43, Wen Congyang wrote:
>>>> At 10/18/2011 04:36 PM, Jan Kiszka Write:
>>>>> On 2011-10-18 10:34, Richard W.M. Jones wrote:
>>>>>
>>>>> Yeah, I see. Could also be solved via gdb scripts, but crash is already
>>>>> there.
>>>>>
>>>>> But let's see if the formats actually differ. In the end, crash is just
>>>>> parsing the same information that also gdb sees.
>>>>
>>>> I think the format can be similar with diskdump/kdump/netdump:
>>>>
>>>> dump_header:     1 block
>>>> sub header:      n blocks(n is stored in dump_header)
>>>> bitmap:          m blocks(2m is stored in dump_header)
>>>> dumpable bitmap: m blocks
>>>> memory data(We can know whether a page is stored in the core by bitmap and dumpable bitmap)
>>>>
>>>> The format of dump header(It's like kdump/diskdump):
>>>> struct disk_dump_header {
>>>> 	char			signature[SIG_LEN];	/* = "QEMU    " */
>>>> 	int			header_version; /* Dump header version */
>>>> 	struct new_utsname	utsname;	/* copy of system_utsname */
>>>> 	struct timeval		timestamp;	/* Time stamp */
>>>> 	unsigned int		status;
>>>> 	int			block_size;	/* Size of a block in byte */
>>>> 	int			sub_hdr_size;	/* Size of arch dependent
>>>> 						   header in blocks */
>>>> 	unsigned int		bitmap_blocks;	/* Size of Memory bitmap in
>>>> 						   block */
>>>> 	unsigned int		max_mapnr;	/* = max_mapnr */
>>>> 	unsigned int		total_ram_blocks;/* Number of blocks should be
>>>> 						   written */
>>>> 	unsigned int		device_blocks;	/* Number of total blocks in
>>>> 						 * the dump device */
>>>> 	unsigned int		written_blocks; /* Number of written blocks */
>>>> 	unsigned int		current_cpu;	/* CPU# which handles dump */
>>>> 	int			nr_cpus;	/* Number of CPUs */
>>>> };
>>>>
>>>> The sub header can contains all registers's value on each vcpu, and other information,
>>>> for example:
>>>> struct qemu_sub_header {
>>>> 	unsigned long	start_pfn;
>>>> 	unsigned long	end_pfn;
>>>> 	off_t		offset_note;
>>>> 	unsigned long	size_note;
>>>> };
>>>
>>> So is this a standard format or only "similar" to something? Which tools
>>> will understand it out-of-the-box? If it's not standard, why?
>>
>> Only similar to something, and we can add a little codes into crash to
>> support such format.
>>
>> If you have a better format, please tell me.
>
> The format crash already processes? What do you need in addition? That
> should be discussed first, not the format details.

A way to dump the memory to a file, and crash can analyze it. We always 
use crash
to analyze such file because it is easier than gdb to use.

We can use 'virsh dump' to dump the memory to a file, but 'virsh dump' 
can not
work while host pci device is used by guest(The reason is that 'virsh 
dump' uses
monitor command migrate to dump the memory to a file).

gdbserver is not the way need, because the guest should be restarted 
quickly.

I think introducing a new monitor command(named dump) to do such thing, 
and you
have agreed it in another mail.

I first to try to reuse migration's code, but I meet some problem(I have 
pointed it
out in an early mail today).

You advice me to reuse gdbstub's code, but I think crash can not support 
such format.

And I want to implement it without reusing any other codes in qemu(If 
so, we should determine
the core's format).

I am happy if you have some other ideas.

Thanks
Wen Congyang
>
> Jan
>

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 16:17, Wen Congyang wrote:
> On 10/18/2011 09:55 PM, Jan Kiszka wrote:
>> On 2011-10-18 15:51, Wen Congyang wrote:
>>> On 10/18/2011 06:28 PM, Jan Kiszka wrote:
>>>> On 2011-10-18 11:43, Wen Congyang wrote:
>>>>> At 10/18/2011 04:36 PM, Jan Kiszka Write:
>>>>>> On 2011-10-18 10:34, Richard W.M. Jones wrote:
>>>>>>
>>>>>> Yeah, I see. Could also be solved via gdb scripts, but crash is already
>>>>>> there.
>>>>>>
>>>>>> But let's see if the formats actually differ. In the end, crash is just
>>>>>> parsing the same information that also gdb sees.
>>>>>
>>>>> I think the format can be similar with diskdump/kdump/netdump:
>>>>>
>>>>> dump_header:     1 block
>>>>> sub header:      n blocks(n is stored in dump_header)
>>>>> bitmap:          m blocks(2m is stored in dump_header)
>>>>> dumpable bitmap: m blocks
>>>>> memory data(We can know whether a page is stored in the core by bitmap and dumpable bitmap)
>>>>>
>>>>> The format of dump header(It's like kdump/diskdump):
>>>>> struct disk_dump_header {
>>>>> 	char			signature[SIG_LEN];	/* = "QEMU    " */
>>>>> 	int			header_version; /* Dump header version */
>>>>> 	struct new_utsname	utsname;	/* copy of system_utsname */
>>>>> 	struct timeval		timestamp;	/* Time stamp */
>>>>> 	unsigned int		status;
>>>>> 	int			block_size;	/* Size of a block in byte */
>>>>> 	int			sub_hdr_size;	/* Size of arch dependent
>>>>> 						   header in blocks */
>>>>> 	unsigned int		bitmap_blocks;	/* Size of Memory bitmap in
>>>>> 						   block */
>>>>> 	unsigned int		max_mapnr;	/* = max_mapnr */
>>>>> 	unsigned int		total_ram_blocks;/* Number of blocks should be
>>>>> 						   written */
>>>>> 	unsigned int		device_blocks;	/* Number of total blocks in
>>>>> 						 * the dump device */
>>>>> 	unsigned int		written_blocks; /* Number of written blocks */
>>>>> 	unsigned int		current_cpu;	/* CPU# which handles dump */
>>>>> 	int			nr_cpus;	/* Number of CPUs */
>>>>> };
>>>>>
>>>>> The sub header can contains all registers's value on each vcpu, and other information,
>>>>> for example:
>>>>> struct qemu_sub_header {
>>>>> 	unsigned long	start_pfn;
>>>>> 	unsigned long	end_pfn;
>>>>> 	off_t		offset_note;
>>>>> 	unsigned long	size_note;
>>>>> };
>>>>
>>>> So is this a standard format or only "similar" to something? Which tools
>>>> will understand it out-of-the-box? If it's not standard, why?
>>>
>>> Only similar to something, and we can add a little codes into crash to
>>> support such format.
>>>
>>> If you have a better format, please tell me.
>>
>> The format crash already processes? What do you need in addition? That
>> should be discussed first, not the format details.
> 
> A way to dump the memory to a file, and crash can analyze it.

I was talking about information you need in the crash dump file that is
not contained in those files crash processes by default. If there is no
such information, then it looks strange to me defining a new format.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

On 10/18/2011 10:21 PM, Jan Kiszka wrote:
> On 2011-10-18 16:17, Wen Congyang wrote:
>
> I was talking about information you need in the crash dump file that is
> not contained in those files crash processes by default. If there is no
> such information, then it looks strange to me defining a new format.

Sorry for my bad english, I can not understant what information you are
talking about.

Do you mean if the core file does not contains such information and crash
can not process it?

Thanks
Wen Congyang

>
> Jan
>

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-18 17:04, Wen Congyang wrote:
> On 10/18/2011 10:21 PM, Jan Kiszka wrote:
>> On 2011-10-18 16:17, Wen Congyang wrote:
>>
>> I was talking about information you need in the crash dump file that is
>> not contained in those files crash processes by default. If there is no
>> such information, then it looks strange to me defining a new format.
> 
> Sorry for my bad english, I can not understant what information you are
> talking about.
> 
> Do you mean if the core file does not contains such information and crash
> can not process it?

OK, I guess we are just talking past each other:

Are you proposing the kernel dump format (as crash takes for its
"dumpfile" parameter)? Should QEMU write out dumps in that file format?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/18/2011 11:27 PM, Jan Kiszka Write:
> On 2011-10-18 17:04, Wen Congyang wrote:
>> On 10/18/2011 10:21 PM, Jan Kiszka wrote:
>>> On 2011-10-18 16:17, Wen Congyang wrote:
>>>
>>> I was talking about information you need in the crash dump file that is
>>> not contained in those files crash processes by default. If there is no
>>> such information, then it looks strange to me defining a new format.
>>
>> Sorry for my bad english, I can not understant what information you are
>> talking about.
>>
>> Do you mean if the core file does not contains such information and crash
>> can not process it?
> 
> OK, I guess we are just talking past each other:
> 
> Are you proposing the kernel dump format (as crash takes for its
> "dumpfile" parameter)? Should QEMU write out dumps in that file format?

Yes.

We can reuse gdbstub's code, and use gdb scripts to instead of crash. But
I do not like this way, because we should know the guest kernel version
and it's config file. We should do too many things that crash has done
for us.

So I think qemu should write out dumps in that file format.

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[KAMEZAWA Hiroyuki]

On Tue, 18 Oct 2011 10:31:10 +0200
Jan Kiszka <jan.kiszka@siemens.com> wrote:

> On 2011-10-18 10:31, Wen Congyang wrote:
> > At 10/18/2011 04:26 PM, Jan Kiszka Write:
> >> On 2011-10-18 10:25, Wen Congyang wrote:
> >>> At 10/18/2011 04:19 PM, Jan Kiszka Write:
> >>>> On 2011-10-18 09:58, Wen Congyang wrote:
> >>>>> At 10/18/2011 03:52 PM, Jan Kiszka Write:
> >>>>>> On 2011-10-18 09:15, Wen Congyang wrote:
> >>>>>>> Hi, Jan Kiszka
> >>>>>>>
> >>>>>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
> >>>>>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
> >>>>>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
> >>>>>>>> that complex. But a dump command is probably simpler for those
> >>>>>>>> scenarios, I agree.
> >>>>>>>
> >>>>>>> I have implemented the command dump and reuse migration's code. But I meet a problem
> >>>>>>> when I test it.
> >>>>>>
> >>>>>> Using migration code for dump is most probably the wrong approach as you
> >>>>>> saw through that conflict. All you need are the register states and the
> >>>>>> RAM. Reuse gdbstub services for this.
> >>>>>
> >>>>> Hmm, if the migration code can not be reused, I think we should define a new
> >>>>> qemu's vmcore format, and add some codes into crash to support such format.
> >>>>
> >>>> Please try to avoid defining something new. Unless there is a striking
> >>>> reason, standard gdb core files should be generated so that you can load
> >>>> the dump directly into gdb for analysis.
> >>>
> >>> I am not sure whehter the standard gdb core files can not be analyzed by crash.
> >>> If not, I think we should define something new because it's easier to use
> >>> crash than gdb to analyze the core files.
> >>
> >> gdb allows you to walk up the frame and print variables (globals &
> >> local) etc.
> > 
> > Crash uses gdb to provide common function, and you can also use all the gdb commands
> > in crash.
> 
> That what's the added value here when I can use gdb directly?
> 

I didn't read full story but 'crash' is used for investigating kernel core generated
by kdump for several years. Considering support service guys, virsh dump should support
a format for crash because they can't work well at investigating vmcore by gdb.

crash has several functionality useful for them as 'show kerne log', 'focus on a cpu'
'for-each-task', 'for-each-vma', 'extract ftrace log' etc.

Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for
investigating kernel dump, it sounds harmful.

Thanks,
-Kame
 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-19 04:04, KAMEZAWA Hiroyuki wrote:
> On Tue, 18 Oct 2011 10:31:10 +0200
> Jan Kiszka <jan.kiszka@siemens.com> wrote:
> 
>> On 2011-10-18 10:31, Wen Congyang wrote:
>>> At 10/18/2011 04:26 PM, Jan Kiszka Write:
>>>> On 2011-10-18 10:25, Wen Congyang wrote:
>>>>> At 10/18/2011 04:19 PM, Jan Kiszka Write:
>>>>>> On 2011-10-18 09:58, Wen Congyang wrote:
>>>>>>> At 10/18/2011 03:52 PM, Jan Kiszka Write:
>>>>>>>> On 2011-10-18 09:15, Wen Congyang wrote:
>>>>>>>>> Hi, Jan Kiszka
>>>>>>>>>
>>>>>>>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>>>>>>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>>>>>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>>>>>>>>> that complex. But a dump command is probably simpler for those
>>>>>>>>>> scenarios, I agree.
>>>>>>>>>
>>>>>>>>> I have implemented the command dump and reuse migration's code. But I meet a problem
>>>>>>>>> when I test it.
>>>>>>>>
>>>>>>>> Using migration code for dump is most probably the wrong approach as you
>>>>>>>> saw through that conflict. All you need are the register states and the
>>>>>>>> RAM. Reuse gdbstub services for this.
>>>>>>>
>>>>>>> Hmm, if the migration code can not be reused, I think we should define a new
>>>>>>> qemu's vmcore format, and add some codes into crash to support such format.
>>>>>>
>>>>>> Please try to avoid defining something new. Unless there is a striking
>>>>>> reason, standard gdb core files should be generated so that you can load
>>>>>> the dump directly into gdb for analysis.
>>>>>
>>>>> I am not sure whehter the standard gdb core files can not be analyzed by crash.
>>>>> If not, I think we should define something new because it's easier to use
>>>>> crash than gdb to analyze the core files.
>>>>
>>>> gdb allows you to walk up the frame and print variables (globals &
>>>> local) etc.
>>>
>>> Crash uses gdb to provide common function, and you can also use all the gdb commands
>>> in crash.
>>
>> That what's the added value here when I can use gdb directly?
>>
> 
> I didn't read full story but 'crash' is used for investigating kernel core generated
> by kdump for several years. Considering support service guys, virsh dump should support
> a format for crash because they can't work well at investigating vmcore by gdb.
> 
> crash has several functionality useful for them as 'show kerne log', 'focus on a cpu'
> 'for-each-task', 'for-each-vma', 'extract ftrace log' etc.
> 
> Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for
> investigating kernel dump, it sounds harmful.

Right, that's why everything (live debugging & crash analysis) should be
consolidated on the long run over gdb. crash is architecturally obsolete
today - not saying it is useless!

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/19/2011 07:40 PM, Jan Kiszka Write:
> On 2011-10-19 04:04, KAMEZAWA Hiroyuki wrote:
>> On Tue, 18 Oct 2011 10:31:10 +0200
>> Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>
>>> On 2011-10-18 10:31, Wen Congyang wrote:
>>>> At 10/18/2011 04:26 PM, Jan Kiszka Write:
>>>>> On 2011-10-18 10:25, Wen Congyang wrote:
>>>>>> At 10/18/2011 04:19 PM, Jan Kiszka Write:
>>>>>>> On 2011-10-18 09:58, Wen Congyang wrote:
>>>>>>>> At 10/18/2011 03:52 PM, Jan Kiszka Write:
>>>>>>>>> On 2011-10-18 09:15, Wen Congyang wrote:
>>>>>>>>>> Hi, Jan Kiszka
>>>>>>>>>>
>>>>>>>>>> At 10/10/2011 05:34 PM, Jan Kiszka Write:
>>>>>>>>>>> On 2011-10-10 11:02, Daniel P. Berrange wrote:
>>>>>>>>>>>> On Mon, Oct 10, 2011 at 08:52:08AM +0200, Jan Kiszka wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Run gdb with "set debug remote 1" and watch the communication, it is not
>>>>>>>>>>> that complex. But a dump command is probably simpler for those
>>>>>>>>>>> scenarios, I agree.
>>>>>>>>>>
>>>>>>>>>> I have implemented the command dump and reuse migration's code. But I meet a problem
>>>>>>>>>> when I test it.
>>>>>>>>>
>>>>>>>>> Using migration code for dump is most probably the wrong approach as you
>>>>>>>>> saw through that conflict. All you need are the register states and the
>>>>>>>>> RAM. Reuse gdbstub services for this.
>>>>>>>>
>>>>>>>> Hmm, if the migration code can not be reused, I think we should define a new
>>>>>>>> qemu's vmcore format, and add some codes into crash to support such format.
>>>>>>>
>>>>>>> Please try to avoid defining something new. Unless there is a striking
>>>>>>> reason, standard gdb core files should be generated so that you can load
>>>>>>> the dump directly into gdb for analysis.
>>>>>>
>>>>>> I am not sure whehter the standard gdb core files can not be analyzed by crash.
>>>>>> If not, I think we should define something new because it's easier to use
>>>>>> crash than gdb to analyze the core files.
>>>>>
>>>>> gdb allows you to walk up the frame and print variables (globals &
>>>>> local) etc.
>>>>
>>>> Crash uses gdb to provide common function, and you can also use all the gdb commands
>>>> in crash.
>>>
>>> That what's the added value here when I can use gdb directly?
>>>
>>
>> I didn't read full story but 'crash' is used for investigating kernel core generated
>> by kdump for several years. Considering support service guys, virsh dump should support
>> a format for crash because they can't work well at investigating vmcore by gdb.
>>
>> crash has several functionality useful for them as 'show kerne log', 'focus on a cpu'
>> 'for-each-task', 'for-each-vma', 'extract ftrace log' etc.
>>
>> Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for
>> investigating kernel dump, it sounds harmful.
> 
> Right, that's why everything (live debugging & crash analysis) should be
> consolidated on the long run over gdb. crash is architecturally obsolete
> today - not saying it is useless!

I do not know why crash is obsoleted today. Is there a new better tool to instead
crash?

At least, I always use crash to live debugging & crash analysis.

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-20 03:22, Wen Congyang wrote:
>>> I didn't read full story but 'crash' is used for investigating kernel core generated
>>> by kdump for several years. Considering support service guys, virsh dump should support
>>> a format for crash because they can't work well at investigating vmcore by gdb.
>>>
>>> crash has several functionality useful for them as 'show kerne log', 'focus on a cpu'
>>> 'for-each-task', 'for-each-vma', 'extract ftrace log' etc.
>>>
>>> Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for
>>> investigating kernel dump, it sounds harmful.
>>
>> Right, that's why everything (live debugging & crash analysis) should be
>> consolidated on the long run over gdb. crash is architecturally obsolete
>> today - not saying it is useless!
> 
> I do not know why crash is obsoleted today. Is there a new better tool to instead
> crash?

I'm not aware of equally powerful (python) scripts for gdb as
replacement, but I think it's worth starting a porting effort at some point.

> 
> At least, I always use crash to live debugging & crash analysis.

Then you may answer some questions to me:
 - Can you attach to a remote target (kgdb, qemu, etc.) and how?
 - Can you use it with latest gdb versions or is the gdb functionality
   hard-wired due to an embedded gdb core in crash (that's how I
   understood Christoph's reply to this topic)

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/20/2011 05:41 PM, Jan Kiszka Write:
> On 2011-10-20 03:22, Wen Congyang wrote:
>>>> I didn't read full story but 'crash' is used for investigating kernel core generated
>>>> by kdump for several years. Considering support service guys, virsh dump should support
>>>> a format for crash because they can't work well at investigating vmcore by gdb.
>>>>
>>>> crash has several functionality useful for them as 'show kerne log', 'focus on a cpu'
>>>> 'for-each-task', 'for-each-vma', 'extract ftrace log' etc.
>>>>
>>>> Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for
>>>> investigating kernel dump, it sounds harmful.
>>>
>>> Right, that's why everything (live debugging & crash analysis) should be
>>> consolidated on the long run over gdb. crash is architecturally obsolete
>>> today - not saying it is useless!
>>
>> I do not know why crash is obsoleted today. Is there a new better tool to instead
>> crash?
> 
> I'm not aware of equally powerful (python) scripts for gdb as
> replacement, but I think it's worth starting a porting effort at some point.
> 
>>
>> At least, I always use crash to live debugging & crash analysis.
> 
> Then you may answer some questions to me:
>  - Can you attach to a remote target (kgdb, qemu, etc.) and how?

No. crash's live debugging only can work the kernel is live. I can use it get
some var's value, or some other information from kernel. If kernel panics,
we can use gdb to attach to a remote target as you said. But on end user machine,
we can not do it, we should dump the memory into a file and analyze it in another
machine while the end user's guest can be restart.

>  - Can you use it with latest gdb versions or is the gdb functionality
>    hard-wired due to an embedded gdb core in crash (that's how I
>    understood Christoph's reply to this topic)

If I use crash, I can not use latest gdb versions. Do we always need to use
the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it
is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I
think we can update it and rebuild crash.

Thanks
Wen Congyang.

> 
> Thanks,
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-20 12:03, Wen Congyang wrote:
> At 10/20/2011 05:41 PM, Jan Kiszka Write:
>> On 2011-10-20 03:22, Wen Congyang wrote:
>>>>> I didn't read full story but 'crash' is used for investigating kernel core generated
>>>>> by kdump for several years. Considering support service guys, virsh dump should support
>>>>> a format for crash because they can't work well at investigating vmcore by gdb.
>>>>>
>>>>> crash has several functionality useful for them as 'show kerne log', 'focus on a cpu'
>>>>> 'for-each-task', 'for-each-vma', 'extract ftrace log' etc.
>>>>>
>>>>> Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for
>>>>> investigating kernel dump, it sounds harmful.
>>>>
>>>> Right, that's why everything (live debugging & crash analysis) should be
>>>> consolidated on the long run over gdb. crash is architecturally obsolete
>>>> today - not saying it is useless!
>>>
>>> I do not know why crash is obsoleted today. Is there a new better tool to instead
>>> crash?
>>
>> I'm not aware of equally powerful (python) scripts for gdb as
>> replacement, but I think it's worth starting a porting effort at some point.
>>
>>>
>>> At least, I always use crash to live debugging & crash analysis.
>>
>> Then you may answer some questions to me:
>>  - Can you attach to a remote target (kgdb, qemu, etc.) and how?
> 
> No. crash's live debugging only can work the kernel is live. I can use it get
> some var's value, or some other information from kernel. If kernel panics,
> we can use gdb to attach to a remote target as you said. But on end user machine,
> we can not do it, we should dump the memory into a file and analyze it in another
> machine while the end user's guest can be restart.
> 
>>  - Can you use it with latest gdb versions or is the gdb functionality
>>    hard-wired due to an embedded gdb core in crash (that's how I
>>    understood Christoph's reply to this topic)
> 
> If I use crash, I can not use latest gdb versions. Do we always need to use
> the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it
> is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I
> think we can update it and rebuild crash.

crash is simply designed the wrong way around (from today's
perspective): it should augment upstream gdb instead of forking it.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/21/2011 03:11 PM, Jan Kiszka Write:
> On 2011-10-20 12:03, Wen Congyang wrote:
>> At 10/20/2011 05:41 PM, Jan Kiszka Write:
>>> On 2011-10-20 03:22, Wen Congyang wrote:
>>>>>> I didn't read full story but 'crash' is used for investigating kernel core generated
>>>>>> by kdump for several years. Considering support service guys, virsh dump should support
>>>>>> a format for crash because they can't work well at investigating vmcore by gdb.
>>>>>>
>>>>>> crash has several functionality useful for them as 'show kerne log', 'focus on a cpu'
>>>>>> 'for-each-task', 'for-each-vma', 'extract ftrace log' etc.
>>>>>>
>>>>>> Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for
>>>>>> investigating kernel dump, it sounds harmful.
>>>>>
>>>>> Right, that's why everything (live debugging & crash analysis) should be
>>>>> consolidated on the long run over gdb. crash is architecturally obsolete
>>>>> today - not saying it is useless!
>>>>
>>>> I do not know why crash is obsoleted today. Is there a new better tool to instead
>>>> crash?
>>>
>>> I'm not aware of equally powerful (python) scripts for gdb as
>>> replacement, but I think it's worth starting a porting effort at some point.
>>>
>>>>
>>>> At least, I always use crash to live debugging & crash analysis.
>>>
>>> Then you may answer some questions to me:
>>>  - Can you attach to a remote target (kgdb, qemu, etc.) and how?
>>
>> No. crash's live debugging only can work the kernel is live. I can use it get
>> some var's value, or some other information from kernel. If kernel panics,
>> we can use gdb to attach to a remote target as you said. But on end user machine,
>> we can not do it, we should dump the memory into a file and analyze it in another
>> machine while the end user's guest can be restart.
>>
>>>  - Can you use it with latest gdb versions or is the gdb functionality
>>>    hard-wired due to an embedded gdb core in crash (that's how I
>>>    understood Christoph's reply to this topic)
>>
>> If I use crash, I can not use latest gdb versions. Do we always need to use
>> the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it
>> is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I
>> think we can update it and rebuild crash.
> 
> crash is simply designed the wrong way around (from today's
> perspective): it should augment upstream gdb instead of forking it.

Cc Dave Anderson. He knows how crash uses gdb.

I think that crash does not fork a task to execute gdb, and gdb is a part of crash.

Thanks
Wen Congyang

> 
> Jan
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Dave Anderson]

----- Original Message -----
> At 10/21/2011 03:11 PM, Jan Kiszka Write:
> > On 2011-10-20 12:03, Wen Congyang wrote:
> >> At 10/20/2011 05:41 PM, Jan Kiszka Write:
> >>> On 2011-10-20 03:22, Wen Congyang wrote:
> >>>>>> I didn't read full story but 'crash' is used for investigating kernel core generated
> >>>>>> by kdump for several years. Considering support service guys, virsh dump should support
> >>>>>> a format for crash because they can't work well at investigating vmcore by gdb.
> >>>>>>
> >>>>>> crash has several functionality useful for them as 'show kerne log', 'focus on a cpu'
> >>>>>> 'for-each-task', 'for-each-vma', 'extract ftrace log' etc.
> >>>>>>
> >>>>>> Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for
> >>>>>> investigating kernel dump, it sounds harmful.
> >>>>>
> >>>>> Right, that's why everything (live debugging & crash analysis) should be
> >>>>> consolidated on the long run over gdb. crash is architecturally obsolete
> >>>>> today - not saying it is useless!
> >>>>
> >>>> I do not know why crash is obsoleted today. Is there a new better tool to instead
> >>>> crash?
> >>>
> >>> I'm not aware of equally powerful (python) scripts for gdb as
> >>> replacement, but I think it's worth starting a porting effort at
> >>> some point.
> >>>
> >>>>
> >>>> At least, I always use crash to live debugging & crash analysis.
> >>>
> >>> Then you may answer some questions to me:
> >>>  - Can you attach to a remote target (kgdb, qemu, etc.) and how?
> >>
> >> No. crash's live debugging only can work the kernel is live. I can use it get
> >> some var's value, or some other information from kernel. If kernel panics,
> >> we can use gdb to attach to a remote target as you said. But on end user machine,
> >> we can not do it, we should dump the memory into a file and analyze it in another
> >> machine while the end user's guest can be restart.
> >>
> >>>  - Can you use it with latest gdb versions or is the gdb functionality
> >>>    hard-wired due to an embedded gdb core in crash (that's how I
> >>>    understood Christoph's reply to this topic)
> >>
> >> If I use crash, I can not use latest gdb versions. Do we always need to use
> >> the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it
> >> is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I
> >> think we can update it and rebuild crash.
> > 
> > crash is simply designed the wrong way around (from today's
> > perspective): it should augment upstream gdb instead of forking it.
> 
> Cc Dave Anderson. He knows how crash uses gdb.
> 
> I think that crash does not fork a task to execute gdb, and gdb is a
> part of crash.

I'm not sure what the question is, but you can consider crash as a huge
wrapper around its embedded gdb, which it invokes as "gdb vmlinux", and
then takes over the user interface.  It doesn't have a clue as to what
the memory source is, i.e., whether it's one of the almost 20 different
dumpfile formats that it supports (including "virsh dump"), or if it's
running against a live system.  It has its own command set, although
you can enter some gdb commands, write gdb scripts, etc.  But the main
purpose of the embedded gdb is for the crash-level sources to be able
to gather data structure information, disassemble text, add-symbol-file
kernel modules, and so on.  There is no kgdb remote linkage. 

It's currently embedding gdb-7.0, although as we speak I'm updating it
to gdb-7.3.1 because the compiler guys have decided that dwarf4 should be
used by default.

It would be kind of cool if there was a "/dev/mem"-like interface
to a KVM guest's physical memory, so that you could sit on a KVM host
and enter "crash vmlinux-of-guest /dev/mem-of-guest" in order to
run live analysis of a guest.

Anyway, sorry if it doesn't meet your needs...

Dave

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-21 15:02, Dave Anderson wrote:
> 
> 
> ----- Original Message -----
>> At 10/21/2011 03:11 PM, Jan Kiszka Write:
>>> On 2011-10-20 12:03, Wen Congyang wrote:
>>>> At 10/20/2011 05:41 PM, Jan Kiszka Write:
>>>>> On 2011-10-20 03:22, Wen Congyang wrote:
>>>>>>>> I didn't read full story but 'crash' is used for investigating kernel core generated
>>>>>>>> by kdump for several years. Considering support service guys, virsh dump should support
>>>>>>>> a format for crash because they can't work well at investigating vmcore by gdb.
>>>>>>>>
>>>>>>>> crash has several functionality useful for them as 'show kerne log', 'focus on a cpu'
>>>>>>>> 'for-each-task', 'for-each-vma', 'extract ftrace log' etc.
>>>>>>>>
>>>>>>>> Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for
>>>>>>>> investigating kernel dump, it sounds harmful.
>>>>>>>
>>>>>>> Right, that's why everything (live debugging & crash analysis) should be
>>>>>>> consolidated on the long run over gdb. crash is architecturally obsolete
>>>>>>> today - not saying it is useless!
>>>>>>
>>>>>> I do not know why crash is obsoleted today. Is there a new better tool to instead
>>>>>> crash?
>>>>>
>>>>> I'm not aware of equally powerful (python) scripts for gdb as
>>>>> replacement, but I think it's worth starting a porting effort at
>>>>> some point.
>>>>>
>>>>>>
>>>>>> At least, I always use crash to live debugging & crash analysis.
>>>>>
>>>>> Then you may answer some questions to me:
>>>>>  - Can you attach to a remote target (kgdb, qemu, etc.) and how?
>>>>
>>>> No. crash's live debugging only can work the kernel is live. I can use it get
>>>> some var's value, or some other information from kernel. If kernel panics,
>>>> we can use gdb to attach to a remote target as you said. But on end user machine,
>>>> we can not do it, we should dump the memory into a file and analyze it in another
>>>> machine while the end user's guest can be restart.
>>>>
>>>>>  - Can you use it with latest gdb versions or is the gdb functionality
>>>>>    hard-wired due to an embedded gdb core in crash (that's how I
>>>>>    understood Christoph's reply to this topic)
>>>>
>>>> If I use crash, I can not use latest gdb versions. Do we always need to use
>>>> the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it
>>>> is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I
>>>> think we can update it and rebuild crash.
>>>
>>> crash is simply designed the wrong way around (from today's
>>> perspective): it should augment upstream gdb instead of forking it.
>>
>> Cc Dave Anderson. He knows how crash uses gdb.
>>
>> I think that crash does not fork a task to execute gdb, and gdb is a
>> part of crash.
> 
> I'm not sure what the question is, but you can consider crash as a huge
> wrapper around its embedded gdb, which it invokes as "gdb vmlinux", and
> then takes over the user interface.  It doesn't have a clue as to what
> the memory source is, i.e., whether it's one of the almost 20 different
> dumpfile formats that it supports (including "virsh dump"), or if it's
> running against a live system.  It has its own command set, although
> you can enter some gdb commands, write gdb scripts, etc.  But the main
> purpose of the embedded gdb is for the crash-level sources to be able
> to gather data structure information, disassemble text, add-symbol-file
> kernel modules, and so on.  There is no kgdb remote linkage. 
> 
> It's currently embedding gdb-7.0, although as we speak I'm updating it
> to gdb-7.3.1 because the compiler guys have decided that dwarf4 should be
> used by default.
> 
> It would be kind of cool if there was a "/dev/mem"-like interface
> to a KVM guest's physical memory, so that you could sit on a KVM host
> and enter "crash vmlinux-of-guest /dev/mem-of-guest" in order to
> run live analysis of a guest.
> 
> Anyway, sorry if it doesn't meet your needs...

Yes, I'd prefer to have the added value of crash available with standard
gdb, specifically to reuse it for remote target debugging which is
fairly common in embedded scenarios and for guest debugging (via
qemu/kvm). I do not yet see that there is anything preventing this
except that it "needs to be done" - or collected from previous work:
there should be, e.g., dozens of gdb scripts circling around that
automate kernel module symbol loading with gdb.

We do have a proper remote debugging interface already, no need to
invent a new one for crash-on-qemu. We do lack complete x86 system-level
debugging support, but that's an absolutely generic gdb problem,
independent of Linux as a debugging target. And, AFAIK, we do not have
such issues with common non-x86 targets.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Richard W.M. Jones]

On Fri, Oct 21, 2011 at 09:02:37AM -0400, Dave Anderson wrote:
> It would be kind of cool if there was a "/dev/mem"-like interface
> to a KVM guest's physical memory, so that you could sit on a KVM host
> and enter "crash vmlinux-of-guest /dev/mem-of-guest" in order to
> run live analysis of a guest.

OT for this thread, but this sort of thing does exist, kind of.

You can send monitor commands to qemu to read physical and virtual
memory ("pmemsave" and "memsave" respectively).  At least one, and
possibly now both of these are bound through libvirt APIs:

http://libvirt.org/html/libvirt-libvirt.html#virDomainMemoryPeek

Here was your previous response about 4 years ago:

https://www.redhat.com/archives/crash-utility/2008-August/msg00032.html

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/21/2011 09:02 PM, Dave Anderson Write:
> 
> 
> ----- Original Message -----
>> At 10/21/2011 03:11 PM, Jan Kiszka Write:
>>> On 2011-10-20 12:03, Wen Congyang wrote:
>>>> At 10/20/2011 05:41 PM, Jan Kiszka Write:
>>>>> On 2011-10-20 03:22, Wen Congyang wrote:
>>>>>>>> I didn't read full story but 'crash' is used for investigating kernel core generated
>>>>>>>> by kdump for several years. Considering support service guys, virsh dump should support
>>>>>>>> a format for crash because they can't work well at investigating vmcore by gdb.
>>>>>>>>
>>>>>>>> crash has several functionality useful for them as 'show kerne log', 'focus on a cpu'
>>>>>>>> 'for-each-task', 'for-each-vma', 'extract ftrace log' etc.
>>>>>>>>
>>>>>>>> Anyway, if a man, who is not developper of qemu/kvm, should learn 2 tools for
>>>>>>>> investigating kernel dump, it sounds harmful.
>>>>>>>
>>>>>>> Right, that's why everything (live debugging & crash analysis) should be
>>>>>>> consolidated on the long run over gdb. crash is architecturally obsolete
>>>>>>> today - not saying it is useless!
>>>>>>
>>>>>> I do not know why crash is obsoleted today. Is there a new better tool to instead
>>>>>> crash?
>>>>>
>>>>> I'm not aware of equally powerful (python) scripts for gdb as
>>>>> replacement, but I think it's worth starting a porting effort at
>>>>> some point.
>>>>>
>>>>>>
>>>>>> At least, I always use crash to live debugging & crash analysis.
>>>>>
>>>>> Then you may answer some questions to me:
>>>>>  - Can you attach to a remote target (kgdb, qemu, etc.) and how?
>>>>
>>>> No. crash's live debugging only can work the kernel is live. I can use it get
>>>> some var's value, or some other information from kernel. If kernel panics,
>>>> we can use gdb to attach to a remote target as you said. But on end user machine,
>>>> we can not do it, we should dump the memory into a file and analyze it in another
>>>> machine while the end user's guest can be restart.
>>>>
>>>>>  - Can you use it with latest gdb versions or is the gdb functionality
>>>>>    hard-wired due to an embedded gdb core in crash (that's how I
>>>>>    understood Christoph's reply to this topic)
>>>>
>>>> If I use crash, I can not use latest gdb versions. Do we always need to use
>>>> the latest gdb versions? Currently, gdb-7.0 is embedded into crash, and it
>>>> is enough to me. If the gdb embedded into crash cannot anaylze the vmcore, I
>>>> think we can update it and rebuild crash.
>>>
>>> crash is simply designed the wrong way around (from today's
>>> perspective): it should augment upstream gdb instead of forking it.
>>
>> Cc Dave Anderson. He knows how crash uses gdb.
>>
>> I think that crash does not fork a task to execute gdb, and gdb is a
>> part of crash.
> 
> I'm not sure what the question is, but you can consider crash as a huge

The question is that: 'virsh dump' can not be used when host pci device
is used by guest. We are discussing how to fix the problem. We have determined
that introduce a new monitor command dump. Jan suggested that the core file's
format is gdb standard core format. Does crash support such format? If no,
is it possible to support such format?

Thanks
Wen Congyang

> wrapper around its embedded gdb, which it invokes as "gdb vmlinux", and
> then takes over the user interface.  It doesn't have a clue as to what
> the memory source is, i.e., whether it's one of the almost 20 different
> dumpfile formats that it supports (including "virsh dump"), or if it's
> running against a live system.  It has its own command set, although
> you can enter some gdb commands, write gdb scripts, etc.  But the main
> purpose of the embedded gdb is for the crash-level sources to be able
> to gather data structure information, disassemble text, add-symbol-file
> kernel modules, and so on.  There is no kgdb remote linkage. 
> 
> It's currently embedding gdb-7.0, although as we speak I'm updating it
> to gdb-7.3.1 because the compiler guys have decided that dwarf4 should be
> used by default.
> 
> It would be kind of cool if there was a "/dev/mem"-like interface
> to a KVM guest's physical memory, so that you could sit on a KVM host
> and enter "crash vmlinux-of-guest /dev/mem-of-guest" in order to
> run live analysis of a guest.
> 
> Anyway, sorry if it doesn't meet your needs...
> 
> Dave
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Dave Anderson]

----- Original Message -----

> The question is that: 'virsh dump' can not be used when host pci device
> is used by guest. We are discussing how to fix the problem. We have determined
> that introduce a new monitor command dump. Jan suggested that the core file's
> format is gdb standard core format. Does crash support such format?  If no,
> is it possible to support such format?

If you are talking about an ELF core dump of the user-space qemu-kvm process
running on the host, then it's certainly not supported.  

As to whether it's possible, I suppose it could be done if a phyical-memory-read
function could be created for it, similar to what I asked about for live analysis
of a guest kernel run on/from the KVM host.  

Dave
 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Avi Kivity]

On 10/24/2011 04:25 PM, Dave Anderson wrote:
> > The question is that: 'virsh dump' can not be used when host pci device
> > is used by guest. We are discussing how to fix the problem. We have determined
> > that introduce a new monitor command dump. Jan suggested that the core file's
> > format is gdb standard core format. Does crash support such format?  If no,
> > is it possible to support such format?
>
> If you are talking about an ELF core dump of the user-space qemu-kvm process
> running on the host, then it's certainly not supported.  

No, an ELF image of the guest's physical memory.

> As to whether it's possible, I suppose it could be done if a phyical-memory-read
> function could be created for it, similar to what I asked about for live analysis
> of a guest kernel run on/from the KVM host.  

We could use the gdb protocol for this.  Currently it reads/writes
virtual memory, but we could extend the gdbserver command to make these
commands apply to physical memory.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Dave Anderson]

----- Original Message -----
> On 10/24/2011 04:25 PM, Dave Anderson wrote:
> > > The question is that: 'virsh dump' can not be used when host pci device
> > > is used by guest. We are discussing how to fix the problem. We have determined
> > > that introduce a new monitor command dump. Jan suggested that the core file's
> > > format is gdb standard core format. Does crash support such format?  If no,
> > > is it possible to support such format?
> >
> > If you are talking about an ELF core dump of the user-space qemu-kvm process
> > running on the host, then it's certainly not supported.
> 
> No, an ELF image of the guest's physical memory.

Well then that should be pretty straight forward to support.  Depending upon
how similar it would be to the "standard" kdump ELF format, the only other
issue is how to determine the physical base address at which the kernel is
loaded, in order to be able to translate the mapped kernel-text/static-data
virtual region of the x86_64 arch (the __START_KERNEL_map region).
 
> > As to whether it's possible, I suppose it could be done if a phyical-memory-read
> > function could be created for it, similar to what I asked about for live analysis
> > of a guest kernel run on/from the KVM host.
> 
> We could use the gdb protocol for this.  Currently it reads/writes
> virtual memory, but we could extend the gdbserver command to make these
> commands apply to physical memory.

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Avi Kivity]

On 10/24/2011 05:25 PM, Dave Anderson wrote:
>
> ----- Original Message -----
> > On 10/24/2011 04:25 PM, Dave Anderson wrote:
> > > > The question is that: 'virsh dump' can not be used when host pci device
> > > > is used by guest. We are discussing how to fix the problem. We have determined
> > > > that introduce a new monitor command dump. Jan suggested that the core file's
> > > > format is gdb standard core format. Does crash support such format?  If no,
> > > > is it possible to support such format?
> > >
> > > If you are talking about an ELF core dump of the user-space qemu-kvm process
> > > running on the host, then it's certainly not supported.
> > 
> > No, an ELF image of the guest's physical memory.
>
> Well then that should be pretty straight forward to support.  Depending upon
> how similar it would be to the "standard" kdump ELF format, the only other
> issue is how to determine the physical base address at which the kernel is
> loaded, in order to be able to translate the mapped kernel-text/static-data
> virtual region of the x86_64 arch (the __START_KERNEL_map region).
>

I guess an elf note would work for that?

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Dave Anderson]

----- Original Message -----

> > > No, an ELF image of the guest's physical memory.
> >
> > Well then that should be pretty straight forward to support.  Depending upon
> > how similar it would be to the "standard" kdump ELF format, the only other
> > issue is how to determine the physical base address at which the kernel is
> > loaded, in order to be able to translate the mapped kernel-text/static-data
> > virtual region of the x86_64 arch (the __START_KERNEL_map region).
> >
> 
> I guess an elf note would work for that?

Right -- here is an example of a RHEL6 ELF kdump header:

$ readelf -a vmcore
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              CORE (Core file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          0 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         6
  Size of section headers:           0 (bytes)
  Number of section headers:         0
  Section header string table index: 0

There are no sections in this file.

There are no sections in this file.

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  NOTE           0x0000000000000190 0x0000000000000000 0x0000000000000000
                 0x000000000000083c 0x000000000000083c         0
  LOAD           0x00000000000009cc 0xffffffff81000000 0x0000000001000000
                 0x0000000000ba3000 0x0000000000ba3000  RWE    0
  LOAD           0x0000000000ba39cc 0xffff810000000000 0x0000000000000000
                 0x00000000000a0000 0x00000000000a0000  RWE    0
  LOAD           0x0000000000c439cc 0xffff810000100000 0x0000000000100000
                 0x0000000001f00000 0x0000000001f00000  RWE    0
  LOAD           0x0000000002b439cc 0xffff81000a000000 0x000000000a000000
                 0x00000000c5fc2840 0x00000000c5fc2840  RWE    0
  LOAD           0x00000000c8b0620c 0xffff810100000000 0x0000000100000000
                 0x0000000030000000 0x0000000030000000  RWE    0

There is no dynamic section in this file.

There are no relocations in this file.

There are no unwind sections in this file.

No version information found in this file.

Notes at offset 0x00000190 with length 0x0000083c:
  Owner         Data size       Description
  CORE          0x00000150      NT_PRSTATUS (prstatus structure)
  CORE          0x00000150      NT_PRSTATUS (prstatus structure)
  VMCOREINFO            0x0000055b      Unknown note type: (0x00000000)
$

In this example, the phys_base (of zero) can be determined by looking 
at the first PT_LOAD segment, and comparing the PhysAddr and the VirtAddr
values -- given that __START_KERNEL_map region is based at ffffffff800000000.
The remaining physical memory chunks are described by the subsequent 
unity-mapped segments.

The NT_PRSTATUS notes are register dumps of each cpu, where this vmcore
was from a 2-cpu system.  But the crash utility is capable of surviving
without them.  It can also get by without the VMCOREINFO note, which is
primarily there for use by the "makedumpfile" utility, which is used to
compress ELF kdumps and filter out unwanted pages, and then make a different 
dumpfile format entirely.

This may be another stupid question -- but does the guest failure mode
render it incapable of using kdump?

Dave

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/24/2011 11:58 PM, Dave Anderson Write:
> 
> 
> ----- Original Message -----
> 
>>>> No, an ELF image of the guest's physical memory.
>>>
>>> Well then that should be pretty straight forward to support.  Depending upon
>>> how similar it would be to the "standard" kdump ELF format, the only other
>>> issue is how to determine the physical base address at which the kernel is
>>> loaded, in order to be able to translate the mapped kernel-text/static-data
>>> virtual region of the x86_64 arch (the __START_KERNEL_map region).
>>>
>>
>> I guess an elf note would work for that?
> 
> Right -- here is an example of a RHEL6 ELF kdump header:

Hi, Jan
Is gdb standard core file like the following format? Does gdb support this
format?

If yes, I think the dump command can output the guest physical memory in
the following format, and we can use both gdb and crash to analyze it.

> 
> $ readelf -a vmcore
> ELF Header:
>   Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
>   Class:                             ELF64
>   Data:                              2's complement, little endian
>   Version:                           1 (current)
>   OS/ABI:                            UNIX - System V
>   ABI Version:                       0
>   Type:                              CORE (Core file)
>   Machine:                           Advanced Micro Devices X86-64
>   Version:                           0x1
>   Entry point address:               0x0
>   Start of program headers:          64 (bytes into file)
>   Start of section headers:          0 (bytes into file)
>   Flags:                             0x0
>   Size of this header:               64 (bytes)
>   Size of program headers:           56 (bytes)
>   Number of program headers:         6
>   Size of section headers:           0 (bytes)
>   Number of section headers:         0
>   Section header string table index: 0
> 
> There are no sections in this file.
> 
> There are no sections in this file.
> 
> Program Headers:
>   Type           Offset             VirtAddr           PhysAddr
>                  FileSiz            MemSiz              Flags  Align
>   NOTE           0x0000000000000190 0x0000000000000000 0x0000000000000000
>                  0x000000000000083c 0x000000000000083c         0
>   LOAD           0x00000000000009cc 0xffffffff81000000 0x0000000001000000
>                  0x0000000000ba3000 0x0000000000ba3000  RWE    0
>   LOAD           0x0000000000ba39cc 0xffff810000000000 0x0000000000000000
>                  0x00000000000a0000 0x00000000000a0000  RWE    0
>   LOAD           0x0000000000c439cc 0xffff810000100000 0x0000000000100000
>                  0x0000000001f00000 0x0000000001f00000  RWE    0
>   LOAD           0x0000000002b439cc 0xffff81000a000000 0x000000000a000000
>                  0x00000000c5fc2840 0x00000000c5fc2840  RWE    0
>   LOAD           0x00000000c8b0620c 0xffff810100000000 0x0000000100000000
>                  0x0000000030000000 0x0000000030000000  RWE    0
> 
> There is no dynamic section in this file.
> 
> There are no relocations in this file.
> 
> There are no unwind sections in this file.
> 
> No version information found in this file.
> 
> Notes at offset 0x00000190 with length 0x0000083c:
>   Owner         Data size       Description
>   CORE          0x00000150      NT_PRSTATUS (prstatus structure)
>   CORE          0x00000150      NT_PRSTATUS (prstatus structure)
>   VMCOREINFO            0x0000055b      Unknown note type: (0x00000000)
> $
> 
> In this example, the phys_base (of zero) can be determined by looking 
> at the first PT_LOAD segment, and comparing the PhysAddr and the VirtAddr
> values -- given that __START_KERNEL_map region is based at ffffffff800000000.
> The remaining physical memory chunks are described by the subsequent 
> unity-mapped segments.
> 
> The NT_PRSTATUS notes are register dumps of each cpu, where this vmcore
> was from a 2-cpu system.  But the crash utility is capable of surviving
> without them.  It can also get by without the VMCOREINFO note, which is
> primarily there for use by the "makedumpfile" utility, which is used to
> compress ELF kdumps and filter out unwanted pages, and then make a different 
> dumpfile format entirely.
> 
> This may be another stupid question -- but does the guest failure mode
> render it incapable of using kdump?

Guest failure mode? I do not what does it mean. But 'virsh dump' can be used
when kdump fails or kdump service is not started.

Thanks
Wen Congyang

> 
> Dave
> 
> 

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-25 03:37, Wen Congyang wrote:
> At 10/24/2011 11:58 PM, Dave Anderson Write:
>>
>>
>> ----- Original Message -----
>>
>>>>> No, an ELF image of the guest's physical memory.
>>>>
>>>> Well then that should be pretty straight forward to support.  Depending upon
>>>> how similar it would be to the "standard" kdump ELF format, the only other
>>>> issue is how to determine the physical base address at which the kernel is
>>>> loaded, in order to be able to translate the mapped kernel-text/static-data
>>>> virtual region of the x86_64 arch (the __START_KERNEL_map region).
>>>>
>>>
>>> I guess an elf note would work for that?
>>
>> Right -- here is an example of a RHEL6 ELF kdump header:
> 
> Hi, Jan
> Is gdb standard core file like the following format? Does gdb support this
> format?
> 
> If yes, I think the dump command can output the guest physical memory in
> the following format, and we can use both gdb and crash to analyze it.

That would be ideal. But given lacking crash-like add-ons for vanilla
gdb, it makes sense to focus on crash compatibility for now. I suspect
we will need some changes to gdb anyway to make use of all information
the kdump file contains.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Wen Congyang]

At 10/24/2011 11:30 PM, Avi Kivity Write:
> On 10/24/2011 05:25 PM, Dave Anderson wrote:
>>
>> ----- Original Message -----
>>> On 10/24/2011 04:25 PM, Dave Anderson wrote:
>>>>> The question is that: 'virsh dump' can not be used when host pci device
>>>>> is used by guest. We are discussing how to fix the problem. We have determined
>>>>> that introduce a new monitor command dump. Jan suggested that the core file's
>>>>> format is gdb standard core format. Does crash support such format?  If no,
>>>>> is it possible to support such format?
>>>>
>>>> If you are talking about an ELF core dump of the user-space qemu-kvm process
>>>> running on the host, then it's certainly not supported.
>>>
>>> No, an ELF image of the guest's physical memory.
>>
>> Well then that should be pretty straight forward to support.  Depending upon
>> how similar it would be to the "standard" kdump ELF format, the only other
>> issue is how to determine the physical base address at which the kernel is
>> loaded, in order to be able to translate the mapped kernel-text/static-data
>> virtual region of the x86_64 arch (the __START_KERNEL_map region).
>>
> 
> I guess an elf note would work for that?

Hi, Avi Kivity, Dave Anderson

I have two questions about it:

1. How to know the guest's physical base address in qemu?

2. Does kdump format(not ELF) include such information in its head?

Thanks
Wen Congyang

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Avi Kivity]

On 10/25/2011 10:06 AM, Wen Congyang wrote:
> Hi, Avi Kivity, Dave Anderson
>
> I have two questions about it:
>
> 1. How to know the guest's physical base address in qemu?

In fact, it's impossible.

Perhaps crash can scan through the core looking for a signature.  Need
to be careful since multiple signatures may be detected (a kdump kernel,
a kernel image in pagecache, or a nested guest).

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Paolo Bonzini]

On 10/25/2011 10:23 AM, Avi Kivity wrote:
> On 10/25/2011 10:06 AM, Wen Congyang wrote:
>> Hi, Avi Kivity, Dave Anderson
>>
>> I have two questions about it:
>>
>> 1. How to know the guest's physical base address in qemu?
>
> In fact, it's impossible.
>
> Perhaps crash can scan through the core looking for a signature.  Need
> to be careful since multiple signatures may be detected (a kdump kernel,
> a kernel image in pagecache, or a nested guest).

Crash uses a heuristic based on the IDT base to identify the base from 
migrate-to-file output.  It also needs CR3 in order to read from virtual 
address.

Paolo

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Avi Kivity]

On 10/25/2011 10:35 AM, Paolo Bonzini wrote:
> On 10/25/2011 10:23 AM, Avi Kivity wrote:
>> On 10/25/2011 10:06 AM, Wen Congyang wrote:
>>> Hi, Avi Kivity, Dave Anderson
>>>
>>> I have two questions about it:
>>>
>>> 1. How to know the guest's physical base address in qemu?
>>
>> In fact, it's impossible.
>>
>> Perhaps crash can scan through the core looking for a signature.  Need
>> to be careful since multiple signatures may be detected (a kdump kernel,
>> a kernel image in pagecache, or a nested guest).
>
> Crash uses a heuristic based on the IDT base to identify the base from
> migrate-to-file output.  It also needs CR3 in order to read from
> virtual address.

Does elf not store cr3?  Perhaps a note with the contents of the
privileged cpu registers is needed.

How will gdb cope with module sections?  They are virtually contiguous
but physically scattered.  AFAICT gdb doesn't do translations itself. 
The kernel proper can be handled with a simple offset, but modules cannot.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Paolo Bonzini]

On 10/25/2011 10:52 AM, Avi Kivity wrote:
>> >  Crash uses a heuristic based on the IDT base to identify the base from
>> >  migrate-to-file output.  It also needs CR3 in order to read from
>> >  virtual address.
> Does elf not store cr3?  Perhaps a note with the contents of the
> privileged cpu registers is needed.

Yes.  It would be nice if this note could be somewhat standardized.

Jan, is there a code sample somewhere for the x86-gdb summer of code 
project?

Paolo

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Jan Kiszka]

On 2011-10-25 10:56, Paolo Bonzini wrote:
> On 10/25/2011 10:52 AM, Avi Kivity wrote:
>>>>  Crash uses a heuristic based on the IDT base to identify the base from
>>>>  migrate-to-file output.  It also needs CR3 in order to read from
>>>>  virtual address.
>> Does elf not store cr3?  Perhaps a note with the contents of the
>> privileged cpu registers is needed.
> 
> Yes.  It would be nice if this note could be somewhat standardized.
> 
> Jan, is there a code sample somewhere for the x86-gdb summer of code 
> project?

I'm not aware of a public repos, CC'ing Anthony and Tom.

We definitely need all x86 system registers handled by gdb long-term,
already for live debugging and for segmented guests / proper real mode
support. But if crash depends on those values already today, we should
head for its current format in a first step, then enhance gdb, and
finally check how to integrate that information best with vanilla gdb.

For me it is important that we agree on consolidating over (enhanced)
gdb on the long run, both /wrt features and supported core formats.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Dave Anderson]

----- Original Message -----
> On 10/25/2011 10:23 AM, Avi Kivity wrote:
> > On 10/25/2011 10:06 AM, Wen Congyang wrote:
> >> Hi, Avi Kivity, Dave Anderson
> >>
> >> I have two questions about it:
> >>
> >> 1. How to know the guest's physical base address in qemu?
> >
> > In fact, it's impossible.
> >
> > Perhaps crash can scan through the core looking for a signature.  Need
> > to be careful since multiple signatures may be detected (a kdump kernel,
> > a kernel image in pagecache, or a nested guest).
> 
> Crash uses a heuristic based on the IDT base to identify the base from
> migrate-to-file output.  It also needs CR3 in order to read from virtual
> address.

Just to clarify -- the need for the IDT base and cr3 is *only* true for 
KVM dumps using the "savevm" format.  

With ELF-format kdumps, the phys_base is simply determined from the PT_LOAD
segment that describes the kernel's __START_KERNEL_map region.  I don't know
why that couldn't be done in this proposed case.  
 
Dave

Re: [Qemu-devel] [Question] dump memory when host pci device is used by guest

[Avi Kivity]

On 10/25/2011 03:18 PM, Dave Anderson wrote:
> > 
> > Crash uses a heuristic based on the IDT base to identify the base from
> > migrate-to-file output.  It also needs CR3 in order to read from virtual
> > address.
>
> Just to clarify -- the need for the IDT base and cr3 is *only* true for 
> KVM dumps using the "savevm" format.  
>
> With ELF-format kdumps, the phys_base is simply determined from the PT_LOAD
> segment that describes the kernel's __START_KERNEL_map region.  I don't know
> why that couldn't be done in this proposed case.  
>  
>

kvm has no idea where the kernel (if any) is running.  The dump is taken
without any guest cooperation.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.