[Qemu-devel] Debugging a 64-bit kernel in qemu

[Qemu-devel] Debugging a 64-bit kernel in qemu

[Markus Duft]

Hi!

I have been playing a little with this: I'm writing a kernel for both x86 and x86-64. While doing so, i'd like to debug the kernel using qemu (and it's gdb stub) and gdb. This worked very well until qemu-0.11.1 (gdb version does not seem to play any role...). From there on, debugging the 64 bit version no longer works. My sessions look like this with qemu-0.13.50 (and any version above 0.11.1, actually...):

mduft@s01en22 /big/Privat/osdev/tachyon2 $ x86_64-pc-linux-gnu-gdb .build/x86_64/x86_64-tachyon
GNU gdb (GDB) 7.2.50.20110103-cvs
[snip]
Reading symbols from /big/Privat/osdev/tachyon2/.build/x86_64/x86_64-tachyon...done.
(gdb) target remote:1234
Remote debugging using :1234
0x0000000000000000 in ?? ()
(gdb) b boot
Breakpoint 1 at 0xffffffff80119000: file /big/Privat/osdev/tachyon2/core/tachyon.boot/Entry.cc, line 25.
(gdb) c
Continuing.
Remote 'g' packet reply is too long: 09ea1180ffffffff00950000000000000000000000000000000010000000000002b0ad2b009500000095000000000000d6101180ffffffffce101180ffffffff0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000901180ffffffff4600000018000000100000001000000010000000100000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000
(gdb) quit

actuallly, i _can_ debug the kernel, nut only until the long mode switch occurs. after this point: no luck... :[

now for the questions:

1) is this a problem with qemu or was qemu "fixed" and gdb has a problem?
   (that's why i CCd the gdb list ;)).
2) is there any plan to fix this issue?
3) is there some kind of workaround i can use (i'd be happy with an ugly/unsupported patch too, since i build all my stuff myself ;)).

Thanks in advance!
Regards, Markus

[Qemu-devel] Re: Debugging a 64-bit kernel in qemu

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2742 bytes --]

Am 03.01.2011 11:27, Markus Duft wrote:
> Hi!
> 
> I have been playing a little with this: I'm writing a kernel for both x86 and x86-64. While doing so, i'd like to debug the kernel using qemu (and it's gdb stub) and gdb. This worked very well until qemu-0.11.1 (gdb version does not seem to play any role...). From there on, debugging the 64 bit version no longer works. My sessions look like this with qemu-0.13.50 (and any version above 0.11.1, actually...):
> 
> mduft@s01en22 /big/Privat/osdev/tachyon2 $ x86_64-pc-linux-gnu-gdb .build/x86_64/x86_64-tachyon
> GNU gdb (GDB) 7.2.50.20110103-cvs
> [snip]
> Reading symbols from /big/Privat/osdev/tachyon2/.build/x86_64/x86_64-tachyon...done.
> (gdb) target remote:1234
> Remote debugging using :1234
> 0x0000000000000000 in ?? ()
> (gdb) b boot
> Breakpoint 1 at 0xffffffff80119000: file /big/Privat/osdev/tachyon2/core/tachyon.boot/Entry.cc, line 25.
> (gdb) c
> Continuing.
> Remote 'g' packet reply is too long: 09ea1180ffffffff00950000000000000000000000000000000010000000000002b0ad2b009500000095000000000000d6101180ffffffffce101180ffffffff0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000901180ffffffff4600000018000000100000001000000010000000100000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f030000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
>  0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
> 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000
> (gdb) quit
> 
> actuallly, i _can_ debug the kernel, nut only until the long mode switch occurs. after this point: no luck... :[
> 
> now for the questions:
> 
> 1) is this a problem with qemu or was qemu "fixed" and gdb has a problem?
>    (that's why i CCd the gdb list ;)).
> 2) is there any plan to fix this issue?
> 3) is there some kind of workaround i can use (i'd be happy with an ugly/unsupported patch too, since i build all my stuff myself ;)).

You see the effects of a workaround in qemu for gdb shortcomings on x86.
See e.g. http://thread.gmane.org/gmane.comp.emulators.qemu/80327, there
should be some threads on the gdb list as well.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: Debugging a 64-bit kernel in qemu

[Markus Duft]

On 01/03/2011 12:15 PM, Jan Kiszka wrote:
[snip]
>>
>> 1) is this a problem with qemu or was qemu "fixed" and gdb has a problem?
>>    (that's why i CCd the gdb list ;)).
>> 2) is there any plan to fix this issue?
>> 3) is there some kind of workaround i can use (i'd be happy with an ugly/unsupported patch too, since i build all my stuff myself ;)).
> 
> You see the effects of a workaround in qemu for gdb shortcomings on x86.
> See e.g. http://thread.gmane.org/gmane.comp.emulators.qemu/80327, there
> should be some threads on the gdb list as well.

Hmm, from the thread i read, that it should be possible to interrupt the code from gdb (by pressing CTRL-C), and set a 64 bit breakpoint then? In case this should work: it doesn't. I'm getting the very same packet too long error, as soon as i try to do _anything_ with the debugger. so, actually, the state this is in is completely unusable to debug 64 bit code. i cannot find any way to do it. 

actually, i find that Ted Harkington was right: in 0.11.1 i can debug 32 bit code with qemu-system-x86_64 well enough (which means i debugged all the 32 bit part of my kernel without ever seen _any_ problem/non-working feature/whatever). wouldn't it be better to have 64 bit debugging working in the 64 bit version, with 32 bit mode working mostly (with whatever small issues), rather than just completely dooming 64 bit debugging...?

any further ideas?

BTW, i did look around a little for this problem before (although i wasn't aware of the thread you pointed me to). All those threads don't propose any solution other than "fix gdb", which may or may not take ages.

Regards, Markus

> 
> Jan
> 

Re: [Qemu-devel] Re: Debugging a 64-bit kernel in qemu

[Markus Duft]

On 01/03/2011 01:15 PM, Markus Duft wrote:
> On 01/03/2011 12:15 PM, Jan Kiszka wrote:
> [snip]
[snip]
> actually, i find that Ted Harkington was right: in 0.11.1 i can debug 32 bit code with qemu-system-x86_64 well enough (which means i debugged all the 32 bit part of my kernel without ever seen _any_ problem/non-working feature/whatever). wouldn't it be better to have 64 bit debugging working in the 64 bit version, with 32 bit mode working mostly (with whatever small issues), rather than just completely dooming 64 bit debugging...?
> 

owh - spoke too soon. there must be more to it: i tried reverting 5f30fa18ad043a841fe9f0c3917ac60f2519ebd1, which restores ability to debug my 64 bit kernel just fine, but now i get the packet too long when trying to debug 32 bit code....

wouldn't it be possible to implement some kind of explicit switch with qemu in the meantime, so i can choose what bitness i want to debug? I know, it's a problem with gdb under the hood, but still - it's really uncool debugging doesn't work in either of the two cases.

Regards, Markus

> any further ideas?
> 
> BTW, i did look around a little for this problem before (although i wasn't aware of the thread you pointed me to). All those threads don't propose any solution other than "fix gdb", which may or may not take ages.
> 
> Regards, Markus
> 
>>
>> Jan
>>
> 
> 

[Qemu-devel] Re: Debugging a 64-bit kernel in qemu

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1720 bytes --]

[ please keep CCs ]

Am 03.01.2011 13:27, Markus Duft wrote:
> On 01/03/2011 01:15 PM, Markus Duft wrote:
>> On 01/03/2011 12:15 PM, Jan Kiszka wrote:
>> [snip]
> [snip]
>> actually, i find that Ted Harkington was right: in 0.11.1 i can debug 32 bit code with qemu-system-x86_64 well enough (which means i debugged all the 32 bit part of my kernel without ever seen _any_ problem/non-working feature/whatever). wouldn't it be better to have 64 bit debugging working in the 64 bit version, with 32 bit mode working mostly (with whatever small issues), rather than just completely dooming 64 bit debugging...?
>>
> 
> owh - spoke too soon. there must be more to it: i tried reverting 5f30fa18ad043a841fe9f0c3917ac60f2519ebd1, which restores ability to debug my 64 bit kernel just fine, but now i get the packet too long when trying to debug 32 bit code....

Hmm, that's new. You definitely loose stack unwinding when using the
wrong mode, thus source-level debugging.

I thought that thread suggested to set the arch explicitly, maybe I
misremembered that:

set arch i386:x86_64
tar rem :1234

If that is required, you probably load a 32-bit binary into gdb that
also contains 64-bit code in some section. I guess this is even more
confusing for gdb.

> 
> wouldn't it be possible to implement some kind of explicit switch with qemu in the meantime, so i can choose what bitness i want to debug? I know, it's a problem with gdb under the hood, but still - it's really uncool debugging doesn't work in either of the two cases.

Wasn't required so far. If you debug in either mode, "set arch" should
do the job. If you have to debug across mode switches, that knob won't
help anyway.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: Debugging a 64-bit kernel in qemu

[Markus Duft]

On 01/03/2011 02:00 PM, Jan Kiszka wrote:
> [ please keep CCs ]
> 
> Am 03.01.2011 13:27, Markus Duft wrote:
>> On 01/03/2011 01:15 PM, Markus Duft wrote:
>>> On 01/03/2011 12:15 PM, Jan Kiszka wrote:
>>> [snip]
>> [snip]
>>> actually, i find that Ted Harkington was right: in 0.11.1 i can debug 32 bit code with qemu-system-x86_64 well enough (which means i debugged all the 32 bit part of my kernel without ever seen _any_ problem/non-working feature/whatever). wouldn't it be better to have 64 bit debugging working in the 64 bit version, with 32 bit mode working mostly (with whatever small issues), rather than just completely dooming 64 bit debugging...?
>>>
>>
>> owh - spoke too soon. there must be more to it: i tried reverting 5f30fa18ad043a841fe9f0c3917ac60f2519ebd1, which restores ability to debug my 64 bit kernel just fine, but now i get the packet too long when trying to debug 32 bit code....
> 
> Hmm, that's new. You definitely loose stack unwinding when using the
> wrong mode, thus source-level debugging.

hmmm... ok - that could be. my "source" in that case is all assembly for the 32 bit part ;) i didn't have such a close look at stack unwinding, as i'm all in one single 32 bit procedure. the next call is already a far call to 64 bit mode, which re-sets the stack anyway.

> 
> I thought that thread suggested to set the arch explicitly, maybe I
> misremembered that:
> 
> set arch i386:x86_64
> tar rem :1234

arch is automatically at x86_64, as i start gdb giving it my elf64 kernel to load (which switches gdb to x86_64). however the first few instructions are 32 bit, switching to long mode then.

> 
> If that is required, you probably load a 32-bit binary into gdb that
> also contains 64-bit code in some section. I guess this is even more
> confusing for gdb.

the other way round: i have a elf64 binary, containing all 64 bit code, but with exactly _one_ section containing 32 bit bootstrap code, which switches to long mode.

> 
>>
>> wouldn't it be possible to implement some kind of explicit switch with qemu in the meantime, so i can choose what bitness i want to debug? I know, it's a problem with gdb under the hood, but still - it's really uncool debugging doesn't work in either of the two cases.
> 
> Wasn't required so far. If you debug in either mode, "set arch" should
> do the job. If you have to debug across mode switches, that knob won't
> help anyway.

that definitely doesn't help in either of my cases... :( behaviour stays the same, no matter if i'm currently breaking in 32 bit or 64 bit code, and setting either architecture in any of the situations.

Regards, Markus

> 
> Jan
>