[Qemu-devel] qemu in full emulation on win32

[Qemu-devel] qemu in full emulation on win32

[Alexey Kardashevskiy]

Hi!

Found 2 problems while I was debugging qemu/ppc64-softmmu/qemu-system-ppc64.exe
WindowsXP SP3 Pro, 32bit, i686-pc-mingw32-gcc (GCC) 4.5.2.


1. The size of the following is 7 bytes on linux and 8 bytes on Windows:
struct {
    uint32_t hi;
    uint64_t child;
    uint64_t parent;
    uint64_t size;
} __attribute__((packed)) ranges[];

The structure is used between QEMU and Open Firmware (powerpc bios) so it is important.

The Feature is described here:
http://stackoverflow.com/questions/7789668/why-would-the-size-of-a-packed-structure-be-different-on-linux-and-windows-when
Shortly there is packing and ms-packing and they are different :)

The solutions are:
1. Add MS-specific #pragma pack(push,1) and #pragma pack(pop).
2. Add -mno-ms-bitfields (gcc >= 4.7.0)
3. Change the structure above to use only uint32_t.

What is the common way of solving such problems in QEMU?


2. QEMU cannot allocate 1024MB for the guest RAM. Literally, VirtualAlloc() fails on 1024MB BUT it does not if I allocate 1023MB and 64MB by 2 subsequent calls. We allocate RAM via memory_region_init_ram(). I am pretty sure this is not happening on 64bit Windows and I suspect that it is happening with qemu-system-x86.exe, is not it?

Do we care that there is actually enough RAM and we could allocate it in several chunks?



-- 
Alexey

Re: [Qemu-devel] qemu in full emulation on win32

[Peter Maydell]

On 18 July 2012 07:30, Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
> 1. The size of the following is 7 bytes on linux and 8 bytes on Windows:
> struct {
>     uint32_t hi;
>     uint64_t child;
>     uint64_t parent;
>     uint64_t size;
> } __attribute__((packed)) ranges[];
>
> The structure is used between QEMU and Open Firmware (powerpc bios) so it is important.

I think this struct should use QEMU_PACKED, which will
ensure that it is packed to GCC rules rather than MS
rules.

We also seem to have let a pile of new uses of attribute((packed))
slip in in hw/mfi.h. Those are probably bugs too.

-- PMM

Re: [Qemu-devel] qemu in full emulation on win32

[Stefan Weil]

Am 18.07.2012 10:01, schrieb Peter Maydell:
> On 18 July 2012 07:30, Alexey Kardashevskiy <aik@ozlabs.ru> wrote:
>> 1. The size of the following is 7 bytes on linux and 8 bytes on Windows:
>> struct {
>>      uint32_t hi;
>>      uint64_t child;
>>      uint64_t parent;
>>      uint64_t size;
>> } __attribute__((packed)) ranges[];
>>
>> The structure is used between QEMU and Open Firmware (powerpc bios) so it is important.
>
> I think this struct should use QEMU_PACKED, which will
> ensure that it is packed to GCC rules rather than MS
> rules.
>
> We also seem to have let a pile of new uses of attribute((packed))
> slip in in hw/mfi.h. Those are probably bugs too.
>
> -- PMM

They are bugs (for w32 / w64 hosts). I just sent a patch to fix them.

Some more which I did not fix are in the TCG debugger interface.
Maybe those also need to be fixed for w32 / w64, but that needs
more tests and reading of the debugger interface documentation.
Maybe Richard Henderson knows whether they should use QEMU_PACKED,
too.

Regards,

Stefan W.

Re: [Qemu-devel] qemu in full emulation on win32

[Stefan Weil]

Am 18.07.2012 08:30, schrieb Alexey Kardashevskiy:
> Hi!
>
> Found 2 problems while I was debugging qemu/ppc64-softmmu/qemu-system-ppc64.exe
> WindowsXP SP3 Pro, 32bit, i686-pc-mingw32-gcc (GCC) 4.5.2.
>
>
> 1. The size of the following is 7 bytes on linux and 8 bytes on Windows:
> struct {
>      uint32_t hi;
>      uint64_t child;
>      uint64_t parent;
>      uint64_t size;
> } __attribute__((packed)) ranges[];
>
> The structure is used between QEMU and Open Firmware (powerpc bios) so it is important.
>
> The Feature is described here:
> http://stackoverflow.com/questions/7789668/why-would-the-size-of-a-packed-structure-be-different-on-linux-and-windows-when
> Shortly there is packing and ms-packing and they are different :)
>
> The solutions are:
> 1. Add MS-specific #pragma pack(push,1) and #pragma pack(pop).
> 2. Add -mno-ms-bitfields (gcc >= 4.7.0)
> 3. Change the structure above to use only uint32_t.
>
> What is the common way of solving such problems in QEMU?

Problem 1 is solved with solution 4 (your own patch) although
that patch does not change the structure size to 7 bytes :-)


>
>
>
> 2. QEMU cannot allocate 1024MB for the guest RAM. Literally, VirtualAlloc() fails on 1024MB BUT it does not if I allocate 1023MB and 64MB by 2 subsequent calls. We allocate RAM via memory_region_init_ram(). I am pretty sure this is not happening on 64bit Windows and I suspect that it is happening with qemu-system-x86.exe, is not it?
>
> Do we care that there is actually enough RAM and we could allocate it in several chunks?


Please try the patch which I'm going to send.

On w64, VirtualAlloc() _can_ allocate large quantities of contiguous 
virtual memory.

On w32, it is normally restricted to the lower 2 GiB which are already 
fragmented
by the code (executable, shared libraries) and data. Larger quantities 
are available
when the executable is allowed to use the upper 2 GiB, too. That's what 
my patch does.

Regards,

Stefan W.

Re: [Qemu-devel] qemu in full emulation on win32

[Peter Maydell]

On 18 July 2012 17:29, Stefan Weil <sw@weilnetz.de> wrote:
> Am 18.07.2012 10:01, schrieb Peter Maydell:
>> I think this struct should use QEMU_PACKED, which will
>> ensure that it is packed to GCC rules rather than MS
>> rules.
>>
>> We also seem to have let a pile of new uses of attribute((packed))
>> slip in in hw/mfi.h. Those are probably bugs too.

> They are bugs (for w32 / w64 hosts). I just sent a patch to fix them.
>
> Some more which I did not fix are in the TCG debugger interface.
> Maybe those also need to be fixed for w32 / w64, but that needs
> more tests and reading of the debugger interface documentation.
> Maybe Richard Henderson knows whether they should use QEMU_PACKED,
> too.

So, I think none of these structs will get actually used on
Windows, because they're ELF structure layouts which only
get used on hosts with ELF support. However I think it would
be nice to use QEMU_PACKED for consistency.

We can clearly straightforwardly switch the DebugInfo struct
in tcg/tcg.c to use QEMU_PACKED.

The remaining cases are all the same, in tcg/*/tcg-target.c:

typedef struct {
    uint32_t len __attribute__((aligned((sizeof(void *)))));
    uint32_t cie_offset;
    tcg_target_long func_start __attribute__((packed));
    tcg_target_long func_len __attribute__((packed));
    uint8_t def_cfa[4];
    uint8_t reg_ofs[14];
} DebugFrameFDE;

I think we can just remove the packed attributes from the
struct member fields and apply QEMU_PACKED to the whole struct:
I don't think this will change any of the alignment or packing
in the not-windows case.

(It's not clear to me why the alignment attribute is applied
to the len field rather than to the whole struct, but we
don't need to change that I guess.)

-- PMM

Re: [Qemu-devel] qemu in full emulation on win32

[Alexey Kardashevskiy]

On 19/07/12 02:35, Stefan Weil wrote:
> Am 18.07.2012 08:30, schrieb Alexey Kardashevskiy:
>> Hi!
>>
>> Found 2 problems while I was debugging qemu/ppc64-softmmu/qemu-system-ppc64.exe
>> WindowsXP SP3 Pro, 32bit, i686-pc-mingw32-gcc (GCC) 4.5.2.
>>
>>
>> 1. The size of the following is 7 bytes on linux and 8 bytes on Windows:
>> struct {
>>      uint32_t hi;
>>      uint64_t child;
>>      uint64_t parent;
>>      uint64_t size;
>> } __attribute__((packed)) ranges[];
>>
>> The structure is used between QEMU and Open Firmware (powerpc bios) so it is important.
>>
>> The Feature is described here:
>> http://stackoverflow.com/questions/7789668/why-would-the-size-of-a-packed-structure-be-different-on-linux-and-windows-when
>> Shortly there is packing and ms-packing and they are different :)
>>
>> The solutions are:
>> 1. Add MS-specific #pragma pack(push,1) and #pragma pack(pop).
>> 2. Add -mno-ms-bitfields (gcc >= 4.7.0)
>> 3. Change the structure above to use only uint32_t.
>>
>> What is the common way of solving such problems in QEMU?
> 
> Problem 1 is solved with solution 4 (your own patch) although
> that patch does not change the structure size to 7 bytes :-)


The weblink here is just for explanation :)
My struct is 7 32bit values but on Windows it was 8 32bit values, 32->28 bytes.



>> 2. QEMU cannot allocate 1024MB for the guest RAM. Literally, VirtualAlloc() fails on 1024MB BUT it does not if I allocate 1023MB and 64MB by 2 subsequent calls. We allocate RAM via memory_region_init_ram(). I am pretty sure this is not happening on 64bit Windows and I suspect that it is happening with qemu-system-x86.exe, is not it?
>>
>> Do we care that there is actually enough RAM and we could allocate it in several chunks?
> 
> 
> Please try the patch which I'm going to send.
> 
> On w64, VirtualAlloc() _can_ allocate large quantities of contiguous 
> virtual memory.
> 
> On w32, it is normally restricted to the lower 2 GiB which are already 
> fragmented
> by the code (executable, shared libraries) and data. Larger quantities 
> are available
> when the executable is allowed to use the upper 2 GiB, too. That's what 
> my patch does.


Looking forward, thanks. I am surprised nobody hit it before.



-- 
Alexey