[Qemu-devel] dump-guest-memory enhancement.

[Qemu-devel] dump-guest-memory enhancement.

[Phi Debian]

Hi All,

I implemented guest-dump-memory for arm32, and bumped in something
strange, the PT_LOADs generated from dump.c are not target page
aligned. There are some advantage to get PT_LOAD aligned.

This mail is to ask advise about patch I could submit later if wanted.

Would it be desirable to get the PT_LOAD aligned on target page size
always, for all arch ?
If so I could fix dump.c, but may be people using dump-guest-memory on
existing current implementation may object, dunno if some of there
tools (debuggers) are dependent of the non aligned current setup (gdb
would nt complain)

If not wanted for existing arch (s390, ppc, i386), may be I could do
it for arm only.

Or ultimatly may be this is not wanted at all, in this case I keep my
dumper for myself :)

Cheers,
Phi

Re: [Qemu-devel] dump-guest-memory enhancement.

[Laszlo Ersek]

On 11/10/13 10:10, Phi Debian wrote:
> Hi All,
> 
> I implemented guest-dump-memory for arm32, and bumped in something
> strange, the PT_LOADs generated from dump.c are not target page
> aligned. There are some advantage to get PT_LOAD aligned.
> 
> This mail is to ask advise about patch I could submit later if wanted.
> 
> Would it be desirable to get the PT_LOAD aligned on target page size
> always, for all arch ?

Can you please explain the problem in more detail? Preferably
illustrated by "readelf" output.

Is it about the placement of the PT_LOAD entries in the core file (eg.
their distance of the beginning of the file, or some such), or about the
contents of the PT_LOAD entries? (Ie. the target-phys ranges they
describe and where those are located in the core file.)

Thanks
Laszlo

> If so I could fix dump.c, but may be people using dump-guest-memory on
> existing current implementation may object, dunno if some of there
> tools (debuggers) are dependent of the non aligned current setup (gdb
> would nt complain)
> 
> If not wanted for existing arch (s390, ppc, i386), may be I could do
> it for arm only.
> 
> Or ultimatly may be this is not wanted at all, in this case I keep my
> dumper for myself :)
> 
> Cheers,
> Phi
> 

Re: [Qemu-devel] dump-guest-memory enhancement.

[Phi Debian]

Hi All,

Sorry Laszlo for flooding your mailbox, I missed the 'reply to all' so
I redo the post here.

And I added some more comment at the end to answer your questions.
Phi

==========================================================
CU82$ /usr/bin/readelf -a vmcore
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              CORE (Core file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          52 (bytes into file)
  Start of section headers:          0 (bytes into file)
  Flags:                             0x0
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         2
  Size of section headers:           0 (bytes)
  Number of section headers:         0
  Section header string table index: 0

There are no sections in this file.

There are no sections to group in this file.

Program Headers:
  Type     Offset      VirtAddr        PhysAddr   FileSiz    MemSiz  Flg Align
  NOTE   0x000074 0x00000000 0x00000000 0x000a0 0x000a0     0
  LOAD   0x000114 0x60000000 0x60000000 0x40000000 0x40000000     0

There is no dynamic section in this file.

There are no relocations in this file.

No version information found in this file.

Notes at offset 0x00000074 with length 0x000000a0:
  Owner                 Data size       Description
  CORE                 0x0000008c       NT_PRSTATUS (prstatus structure)


========================================================

The Align fot the PT_LOAD is ZERO, then the offset is 0x114, having an
Align set to TARGET_PAGE_BITS, (or at least 4Kb) would provide a
chance for any debugger to do page align copy (either lseek/read, or
mmap) as they trip on the core, marginal detail, but may help.

As an example, a userland main(){abort();} kind of prog would produce
a core file like this.

CM01$ readelf -a core.2000
...
    LOAD off    0x0000000000001000 vaddr 0x0000000000400000 paddr
0x0000000000000000 align 2**12
Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  NOTE           0x0001d4 0x00000000 0x00000000 0x001d8 0x00000     0
  LOAD           0x001000 0x00a42000 0x00000000 0x00000 0x1b000 R E 0x1000
  LOAD           0x002000 0x00a5e000 0x00000000 0x01000 0x01000 RW  0x1000

The align for LOAD's is 0x1000 thus the file offset is 0x01000, 0x2000 etc...

==========================================================

I guess dump-guest-memory is of a marginal use, yet it can be usefull
when kexec/kdump is broken or non existant on some new architecture
(os/arch bring up).

So to answer your question, the content of the PT_LOAD is ok, only its
offset is non aligned.

I got to precise I obtained this vmcore by implementing the arc_arm
part of the qemu dump-guest-memory, and planing to do the same for
arm64, I may have mis-used the QEMU API's, but for what I can read,
the align member is left non initialised after a memset(zero) of the
phdr/shdr i.e it is left at zero, and I got the impression that the
wayt the elf is produced, section/progs alignment was not in mind. So
I guess other arch are not aligned either, I did not test that.

Cheers,
Phi

Re: [Qemu-devel] dump-guest-memory enhancement.

[Laszlo Ersek]

On 11/11/13 04:28, Phi Debian wrote:

> CU82$ /usr/bin/readelf -a vmcore

> Program Headers:
>   Type     Offset      VirtAddr        PhysAddr   FileSiz    MemSiz  Flg Align
>   NOTE   0x000074 0x00000000 0x00000000 0x000a0 0x000a0     0
>   LOAD   0x000114 0x60000000 0x60000000 0x40000000 0x40000000     0

> The Align fot the PT_LOAD is ZERO, then the offset is 0x114, having an
> Align set to TARGET_PAGE_BITS, (or at least 4Kb) would provide a
> chance for any debugger to do page align copy (either lseek/read, or
> mmap) as they trip on the core, marginal detail, but may help.

I checked <http://refspecs.linuxbase.org/elf/gabi4+/ch5.pheader.html>,
and now it starts to make sense to me.

But first, I think you meant TARGET_PAGE_SIZE, not TARGET_PAGE_BITS, for
the p_align field.

So, the specs say for p_align:

> As ``Program Loading'' describes in this chapter of the processor
> supplement, loadable process segments must have congruent values for
> p_vaddr and p_offset, modulo the page size. This member gives the
> value to which the segments are aligned in memory and in the file.
> Values 0 and 1 mean no alignment is required. Otherwise, p_align
> should be a positive, integral power of 2, and p_vaddr should equal
> p_offset, modulo p_align.

In one sentence, p_vaddr and p_offset should produce the same remainder
when divided by p_align. I guess you could implement this by adding some
buffer space between the headers and the "meat" of the file.

IMHO this requirement isn't necessarily a hard one for vmcores, as we're
not trying to map-and-execute these segments.

> As an example, a userland main(){abort();} kind of prog would produce
> a core file like this.
> 
> CM01$ readelf -a core.2000
> ...
>     LOAD off    0x0000000000001000 vaddr 0x0000000000400000 paddr
> 0x0000000000000000 align 2**12
> Program Headers:
>   Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
>   NOTE           0x0001d4 0x00000000 0x00000000 0x001d8 0x00000     0
>   LOAD           0x001000 0x00a42000 0x00000000 0x00000 0x1b000 R E 0x1000
>   LOAD           0x002000 0x00a5e000 0x00000000 0x01000 0x01000 RW  0x1000
> 
> The align for LOAD's is 0x1000 thus the file offset is 0x01000, 0x2000 etc...

Understood. Seems to be consistent with the above.

> I guess dump-guest-memory is of a marginal use,

Strongly disagree :)

> yet it can be usefull
> when kexec/kdump is broken or non existant on some new architecture
> (os/arch bring up).

It's also important when you want to control the dumping process from
the host side (including the location of the vmcore). You might want to
grab a dump without a guest kernel panic too.

> So to answer your question, the content of the PT_LOAD is ok, only its
> offset is non aligned.

I actually meant the question the other way around. Apparently, the
offset of the PT_LOAD entry itself (ie. where it is in the file) is OK;
its contents could be improved perhaps (the p_offset and p_align fields).

> I got to precise I obtained this vmcore by implementing the arc_arm
> part of the qemu dump-guest-memory, and planing to do the same for
> arm64, I may have mis-used the QEMU API's, but for what I can read,
> the align member is left non initialised after a memset(zero) of the
> phdr/shdr i.e it is left at zero, and I got the impression that the
> wayt the elf is produced, section/progs alignment was not in mind.

I haven't been there at the original creation of this functionality, but
I tend to agree with you. For analyzing the vmcore with gdb or crash,
the alignment doesn't seem to be important, so it was probably ignored.

> So
> I guess other arch are not aligned either, I did not test that.

It seems to be so, yes.

I think that implementing the arch-specific bits (currently: stubs) for
a new architectire, for dump-guest-memory, would be very useful.

OTOH reworking the generic dump code for the alignment (non-)issue seems
to be more risk than worth to me, unless the produced vmcore is actually
unusable on a specific platform. I'm not opposing or trying to block it
of course, it's just plain ol' "don't fix it if it ain't broken" and
"keep the complexity down" for me.

If I understand correctly, the benefit of getting the alignment
pedantically right would be "a chance for any debugger to do page align
copy (either lseek/read, or mmap)". Is that right?

I can't imagine it would be noticeable in performance on x86_64 with
crash or gdb. Is the performance significantly different on ARM?

Thanks
Laszlo

Re: [Qemu-devel] dump-guest-memory enhancement.

[Phi Debian]

Hi Laszlo,


On Mon, Nov 11, 2013 at 8:38 PM, Laszlo Ersek <lersek@redhat.com> wrote:

>
> But first, I think you meant TARGET_PAGE_SIZE, not TARGET_PAGE_BITS, for
> the p_align field.
>
> So, the specs say for p_align:
>
>> As ``Program Loading'' describes in this chapter of the processor
>> supplement, loadable process segments must have congruent values for
>> p_vaddr and p_offset, modulo the page size. This member gives the
>> value to which the segments are aligned in memory and in the file.
>> Values 0 and 1 mean no alignment is required. Otherwise, p_align
>> should be a positive, integral power of 2, and p_vaddr should equal
>> p_offset, modulo p_align.
>
> In one sentence, p_vaddr and p_offset should produce the same remainder
> when divided by p_align. I guess you could implement this by adding some
> buffer space between the headers and the "meat" of the file.

I said TARGET_PAGE_BITS, because as you noticed in your spec text,
p_align is a power of 2, if you want a 4096 byte aligned section (or
prog header) you would stick 12 in there, i.e the number of bits.

>
> IMHO this requirement isn't necessarily a hard one for vmcores, as we're
> not trying to map-and-execute these segments.

Not map-and-execute, but map-and-read sure enough :)



>
> I think that implementing the arch-specific bits (currently: stubs) for
> a new architectire, for dump-guest-memory, would be very useful.
>
> OTOH reworking the generic dump code for the alignment (non-)issue seems
> to be more risk than worth to me, unless the produced vmcore is actually
> unusable on a specific platform. I'm not opposing or trying to block it
> of course, it's just plain ol' "don't fix it if it ain't broken" and
> "keep the complexity down" for me.

Yes, when fixing code you may simply break it :)

>
> If I understand correctly, the benefit of getting the alignment
> pedantically right would be "a chance for any debugger to do page align
> copy (either lseek/read, or mmap)". Is that right?

yes.

>
> I can't imagine it would be noticeable in performance on x86_64 with
> crash or gdb. Is the performance significantly different on ARM?
>

When number of crashdump pages to bring in start to be 'big'  having
it non aligned double the number of page faults, if the panic dump
path would generate non aligned gigantic dump I would voice a bit more
:)

For qemu, we can safely ignore all this, as cross-arch qemu is mainly
for bring up, and then I guess it will never run gigantic models, so
never dump gigantic dump.

You right, qemu dump-guest-memory is much more than a 'dump on panic'
is more like get a 'live-dump' you can snapshot at different point of
time in your OS load.

So in conclusion we stay the way it is, the ELF is semantically
correct and debuggers can deal with it.

Thanx for taking time to look at it.

Cheers
Phi

Re: [Qemu-devel] dump-guest-memory enhancement.

[Peter Maydell]

On 11 November 2013 03:28, Phi Debian <phi.debian@gmail.com> wrote:
> The Align fot the PT_LOAD is ZERO, then the offset is 0x114, having an
> Align set to TARGET_PAGE_BITS, (or at least 4Kb) would provide a
> chance for any debugger to do page align copy (either lseek/read, or
> mmap) as they trip on the core, marginal detail, but may help.

You almost certainly don't want TARGET_PAGE_BITS or
TARGET_PAGE_SIZE. These refer to the *minimum* page size
for the target CPU architecture, which may well be smaller than
the actual page size being used by either the host or the target OS.
In particular, for ARM TARGET_PAGE_SIZE is 1K, but both host
and guest are probably using 4K pages.

thanks
-- PMM

Re: [Qemu-devel] dump-guest-memory enhancement.

[Dave Anderson]

----- Original Message -----

> I haven't been there at the original creation of this functionality, but
> I tend to agree with you. For analyzing the vmcore with gdb or crash,
> the alignment doesn't seem to be important, so it was probably ignored.

With respect to the crash utility, the p_align field is actually used
to differentiate kdump ELF vmcores from the old netdump ELF vmcores.
There are a couple subtle differences between the old netdump ELF vmcores
and kdump ELF vmcores.  Netdump ELF vmcores set the the p_align field to
the architecture page size, whereas kdump ELF vmcores have always had a p_align
field of zero.

Ideally these virsh-dump vmcores would continue to be essentially kdump
clones.  But if it's changed, then there will need to be an update to the
crash utility to work around it.

Dave