* [Qemu-devel] Booting kernels with PVHVM documentation?
@ 2019-03-06 17:51 Alex Bennée
2019-03-08 9:57 ` Liam Merwick
2019-03-11 9:29 ` Stefano Garzarella
0 siblings, 2 replies; 6+ messages in thread
From: Alex Bennée @ 2019-03-06 17:51 UTC (permalink / raw)
To: Stefano Garzarella, Liam Merwick, Paolo Bonzini; +Cc: qemu-devel
Hi,
I've been looking at using PVH as an alternative to a long bios boot
sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
hard to piece together all the bits but I naively thought it would just
be a case of adding a few ELF NOTES to my boot.S with something like:
ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE, _ASM_PTR 0x100000)
ELFNOTE(Xen, XEN_ELFNOTE_ENTRY, _ASM_PTR _start)
ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY, _ASM_PTR 0) /* entry == virtbase */
ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET, _ASM_PTR 0)
.code64
.section .text
/* Kernel Entry Point */
.global _start
_start:
// Setup stack ASAP
movq $stack_end,%rsp
However I'm running into lots of head scratching as the get_elf_note
code seems to skip over the notes before failing:
./qemu-system-x86_64 -monitor none -display none \
-chardev stdio,id=out -device isa-debugcon,chardev=out \
-device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello
load_elf64: processing hdr:0 of type 1
load_elf64: processing hdr:1 of type 4
get_elf_note_type64: looking for type 18, first is 3
get_elf_note_type64: 4/20
get_elf_note_type64: offset is 36
get_elf_note_type64: note is 0
get_elf_note_type64: 0/123713
get_elf_note_type64: offset is 123728
load_elf64: processing hdr:2 of type 1685382481
qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note
So I thought I'd go back to the Linux kernel and see if I could get it
to boot up. So I built an x86_64 kernel with:
CONFIG_XEN_PVHVM=y
CONFIG_XEN_PVHVM_SMP=y
CONFIG_XEN_PVH=y
CONFIG_PVH=y
And tried to boot that, it certainly gets a lot further but in detecting
the note 18 it's looking for but then doesn't provide any output. So I
started digging around the patches and saw talk of a PVH option ROM
which does all the x86 mode escalation before booting the kernel.
However I was unable to find any documentation about if I should be
adding this manually to my command line or if it is auto-magiced into
place. So I have a number of questions:
* what's the canonical command line for booting a Linux PVHVM kernel?
* should this work in TCG as well?
* are they any special linker rules required for the Xen.notes?
And finally:
* is this idea of mine a weird abuse of the PVHVM boot protocol or
does it make sense?
Thanks in advance for any elucidation ;-)
--
Alex Bennée
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] Booting kernels with PVHVM documentation?
2019-03-06 17:51 [Qemu-devel] Booting kernels with PVHVM documentation? Alex Bennée
@ 2019-03-08 9:57 ` Liam Merwick
2019-03-08 10:43 ` Alex Bennée
2019-03-11 9:29 ` Stefano Garzarella
1 sibling, 1 reply; 6+ messages in thread
From: Liam Merwick @ 2019-03-08 9:57 UTC (permalink / raw)
To: Alex Bennée, Stefano Garzarella, Paolo Bonzini
Cc: qemu-devel, liam.merwick
On 06/03/2019 17:51, Alex Bennée wrote:
>
> Hi,
>
> I've been looking at using PVH as an alternative to a long bios boot
> sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
> hard to piece together all the bits but I naively thought it would just
> be a case of adding a few ELF NOTES to my boot.S with something like:
>
> ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE, _ASM_PTR 0x100000)
> ELFNOTE(Xen, XEN_ELFNOTE_ENTRY, _ASM_PTR _start)
> ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY, _ASM_PTR 0) /* entry == virtbase */
> ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET, _ASM_PTR 0)
>
> .code64
> .section .text
> /* Kernel Entry Point */
> .global _start
> _start:
> // Setup stack ASAP
> movq $stack_end,%rsp
>
> However I'm running into lots of head scratching as the get_elf_note
> code seems to skip over the notes before failing:
>
> ./qemu-system-x86_64 -monitor none -display none \
> -chardev stdio,id=out -device isa-debugcon,chardev=out \
> -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello
> load_elf64: processing hdr:0 of type 1
> load_elf64: processing hdr:1 of type 4
> get_elf_note_type64: looking for type 18, first is 3
> get_elf_note_type64: 4/20
> get_elf_note_type64: offset is 36
> get_elf_note_type64: note is 0
> get_elf_note_type64: 0/123713
> get_elf_note_type64: offset is 123728
> load_elf64: processing hdr:2 of type 1685382481
> qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note
What does 'readelf -Wn ./tests/hello' or 'readelf -p .notes
./tests/hello' show?
>
> So I thought I'd go back to the Linux kernel and see if I could get it
> to boot up. So I built an x86_64 kernel with:
>
> CONFIG_XEN_PVHVM=y
> CONFIG_XEN_PVHVM_SMP=y
> CONFIG_XEN_PVH=y
> CONFIG_PVH=y
>
> And tried to boot that, it certainly gets a lot further but in detecting
> the note 18 it's looking for but then doesn't provide any output. So I
> started digging around the patches and saw talk of a PVH option ROM
> which does all the x86 mode escalation before booting the kernel.
> However I was unable to find any documentation about if I should be
> adding this manually to my command line or if it is auto-magiced into
> place. So I have a number of questions:
>
> * what's the canonical command line for booting a Linux PVHVM kernel?
I had been using '-bios ./qboot.bin' (using
https://github.com/bonzini/qboot/pull/17/files) but Stefano's changes
mean that isn't necessary. Here's a command line I used
sudo $QEMU \
-name testvm01 \
-machine
q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit \
-cpu host \
-m 1024,maxmem=20G,slots=2 \
-smp 1 \
-no-user-config \
-nodefaults \
-object
memory-backend-file,id=mem0,share,mem-path=$IMAGE,size=235929600 \
-device nvdimm,memdev=mem0,id=nv0 \
-append 'console=ttyS0,115200,8n1 root=/dev/pmem0p1 panic=1 rw
tsc=reliable no_timer_check noreplace-smp init=/usr/lib/systemd/systemd
rootfstype=ext4 rcupdate.rcu_expedited=1 reboot=f' \
-no-reboot \
-serial mon:stdio \
-kernel $KERNEL
> * should this work in TCG as well?
> * are they any special linker rules required for the Xen.notes?
>
> And finally:
>
> * is this idea of mine a weird abuse of the PVHVM boot protocol or
> does it make sense?
>
Regards,
Liam
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] Booting kernels with PVHVM documentation?
2019-03-08 9:57 ` Liam Merwick
@ 2019-03-08 10:43 ` Alex Bennée
2019-03-08 11:48 ` Paolo Bonzini
0 siblings, 1 reply; 6+ messages in thread
From: Alex Bennée @ 2019-03-08 10:43 UTC (permalink / raw)
To: Liam Merwick; +Cc: Stefano Garzarella, Paolo Bonzini, qemu-devel
Liam Merwick <liam.merwick@oracle.com> writes:
> On 06/03/2019 17:51, Alex Bennée wrote:
>>
>> Hi,
>>
>> I've been looking at using PVH as an alternative to a long bios boot
>> sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
>> hard to piece together all the bits but I naively thought it would just
>> be a case of adding a few ELF NOTES to my boot.S with something like:
>>
>> ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE, _ASM_PTR 0x100000)
>> ELFNOTE(Xen, XEN_ELFNOTE_ENTRY, _ASM_PTR _start)
>> ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY, _ASM_PTR 0) /* entry == virtbase */
>> ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET, _ASM_PTR 0)
>>
>> .code64
>> .section .text
>> /* Kernel Entry Point */
>> .global _start
>> _start:
>> // Setup stack ASAP
>> movq $stack_end,%rsp
>>
>> However I'm running into lots of head scratching as the get_elf_note
>> code seems to skip over the notes before failing:
>>
>> ./qemu-system-x86_64 -monitor none -display none \
>> -chardev stdio,id=out -device isa-debugcon,chardev=out \
>> -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello
>> load_elf64: processing hdr:0 of type 1
>> load_elf64: processing hdr:1 of type 4
>> get_elf_note_type64: looking for type 18, first is 3
>> get_elf_note_type64: 4/20
>> get_elf_note_type64: offset is 36
>> get_elf_note_type64: note is 0
>> get_elf_note_type64: 0/123713
>> get_elf_note_type64: offset is 123728
>> load_elf64: processing hdr:2 of type 1685382481
>> qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note
>
>
>
> What does 'readelf -Wn ./tests/hello' or 'readelf -p .notes
> ./tests/hello' show?
$ readelf -Wn tests/hello
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: 919289306d162384e3c8a38441088cf6c511f623
Displaying notes found in: .note.Xen
Owner Data size Description
Xen 0x00000008 Unknown note type: (0x00000003) description data: 00 00 10 00 00 00 00 00
Xen 0x00000008 NT_VERSION (version) description data: 40 00 10 00 00 00 00 00
Xen 0x00000008 Unknown note type: (0x00000012) description data: 00 00 00 00 00 00 00 00
Xen 0x00000008 Unknown note type: (0x00000004) description data: 00 00 00 00 00 00 00 00
$ readelf -p .notes tests/hello
readelf: Warning: Section '.notes' was not dumped because it does not exist!
>>
>> So I thought I'd go back to the Linux kernel and see if I could get it
>> to boot up. So I built an x86_64 kernel with:
>>
>> CONFIG_XEN_PVHVM=y
>> CONFIG_XEN_PVHVM_SMP=y
>> CONFIG_XEN_PVH=y
>> CONFIG_PVH=y
>>
>> And tried to boot that, it certainly gets a lot further but in detecting
>> the note 18 it's looking for but then doesn't provide any output. So I
>> started digging around the patches and saw talk of a PVH option ROM
>> which does all the x86 mode escalation before booting the kernel.
>> However I was unable to find any documentation about if I should be
>> adding this manually to my command line or if it is auto-magiced into
>> place. So I have a number of questions:
>>
>> * what's the canonical command line for booting a Linux PVHVM kernel?
>
> I had been using '-bios ./qboot.bin' (using
> https://github.com/bonzini/qboot/pull/17/files) but Stefano's changes
> mean that isn't necessary. Here's a command line I used
I think they are all in.
>
> sudo $QEMU \
> -name testvm01 \
> -machine
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit \
Hmm if I use the q35 machine type I might have to rethink the approach
for getting output and results as I'm currently using the isa debugcon
and debug exit devices.
I wonder how much framework I would need for a minimal virtio or PV
serial implementation? Currently the ISA stuff is dumb as bricks and
simple:
/* Output a single character to serial port */
.global __sys_outc
__sys_outc:
pushl %ebp
movl %esp, %ebp
out %al,$0xE9
movl %ebp, %esp
popl %ebp
ret
and:
/* output any non-zero result in eax to isa-debug-exit device */
test %al, %al
jz 1f
out %ax, $0xf4
1: /* QEMU ACPI poweroff */
mov $0x604,%edx
mov $0x2000,%eax
out %ax,%dx
hlt
jmp 1b
> -cpu host \
> -m 1024,maxmem=20G,slots=2 \
> -smp 1 \
> -no-user-config \
> -nodefaults \
> -object
> memory-backend-file,id=mem0,share,mem-path=$IMAGE,size=235929600 \
> -device nvdimm,memdev=mem0,id=nv0 \
> -append 'console=ttyS0,115200,8n1 root=/dev/pmem0p1 panic=1 rw
> tsc=reliable no_timer_check noreplace-smp
> init=/usr/lib/systemd/systemd rootfstype=ext4 rcupdate.rcu_expedited=1
> reboot=f' \
> -no-reboot \
> -serial mon:stdio \
> -kernel $KERNEL
>
>
>> * should this work in TCG as well?
>> * are they any special linker rules required for the Xen.notes?
>>
>> And finally:
>>
>> * is this idea of mine a weird abuse of the PVHVM boot protocol or
>> does it make sense?
>>
--
Alex Bennée
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] Booting kernels with PVHVM documentation?
2019-03-08 10:43 ` Alex Bennée
@ 2019-03-08 11:48 ` Paolo Bonzini
0 siblings, 0 replies; 6+ messages in thread
From: Paolo Bonzini @ 2019-03-08 11:48 UTC (permalink / raw)
To: Alex Bennée, Liam Merwick; +Cc: Stefano Garzarella, qemu-devel
On 08/03/19 11:43, Alex Bennée wrote:
>>
>> sudo $QEMU \
>> -name testvm01 \
>> -machine
>> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit \
>
> Hmm if I use the q35 machine type I might have to rethink the approach
> for getting output and results as I'm currently using the isa debugcon
> and debug exit devices.
q35 supports ISA, but it shouldn't be needed. pc should work.
Paolo
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] Booting kernels with PVHVM documentation?
2019-03-06 17:51 [Qemu-devel] Booting kernels with PVHVM documentation? Alex Bennée
2019-03-08 9:57 ` Liam Merwick
@ 2019-03-11 9:29 ` Stefano Garzarella
2019-06-04 18:34 ` Alex Bennée
1 sibling, 1 reply; 6+ messages in thread
From: Stefano Garzarella @ 2019-03-11 9:29 UTC (permalink / raw)
To: Alex Bennée; +Cc: Liam Merwick, Paolo Bonzini, qemu-devel
Hi Alex,
sorry for the big delay, but I was traveling without my PC.
On Wed, Mar 06, 2019 at 05:51:05PM +0000, Alex Bennée wrote:
>
> Hi,
>
> I've been looking at using PVH as an alternative to a long bios boot
> sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
> hard to piece together all the bits but I naively thought it would just
> be a case of adding a few ELF NOTES to my boot.S with something like:
>
> ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE, _ASM_PTR 0x100000)
> ELFNOTE(Xen, XEN_ELFNOTE_ENTRY, _ASM_PTR _start)
> ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY, _ASM_PTR 0) /* entry == virtbase */
> ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET, _ASM_PTR 0)
>
> .code64
Can you try to use .code32?
The pvh.bin optionrom will jump to the image in 32-bit mode.
I don't have a lot of experience, but looking at arch/x86/platform/pvh/head.S
(Linux), I saw that entry point is under .code32, than it will switch to
64-bit mode.
> .section .text
> /* Kernel Entry Point */
> .global _start
> _start:
> // Setup stack ASAP
> movq $stack_end,%rsp
>
> However I'm running into lots of head scratching as the get_elf_note
> code seems to skip over the notes before failing:
>
> ./qemu-system-x86_64 -monitor none -display none \
> -chardev stdio,id=out -device isa-debugcon,chardev=out \
> -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello
> load_elf64: processing hdr:0 of type 1
> load_elf64: processing hdr:1 of type 4
> get_elf_note_type64: looking for type 18, first is 3
> get_elf_note_type64: 4/20
> get_elf_note_type64: offset is 36
> get_elf_note_type64: note is 0
> get_elf_note_type64: 0/123713
> get_elf_note_type64: offset is 123728
> load_elf64: processing hdr:2 of type 1685382481
> qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note
>
> So I thought I'd go back to the Linux kernel and see if I could get it
> to boot up. So I built an x86_64 kernel with:
>
> CONFIG_XEN_PVHVM=y
> CONFIG_XEN_PVHVM_SMP=y
> CONFIG_XEN_PVH=y
> CONFIG_PVH=y
I enabled only CONFIG_PVH to boot a vmlinux image with PVH support.
>
> And tried to boot that, it certainly gets a lot further but in detecting
> the note 18 it's looking for but then doesn't provide any output. So I
> started digging around the patches and saw talk of a PVH option ROM
> which does all the x86 mode escalation before booting the kernel.
> However I was unable to find any documentation about if I should be
> adding this manually to my command line or if it is auto-magiced into
> place. So I have a number of questions:
Sorry for that, I'll wrote some docs to cover this feature.
>
> * what's the canonical command line for booting a Linux PVHVM kernel?
You can use the standard -kernel parameter specifying the path to the
vmlinux image compiled with CONFIG_PVH=y. For example I'm using this
command:
./x86_64-softmmu/qemu-system-x86_64 -machine pc,accel=kvm \
-kernel /path/to/vmlinux \
-drive file=/path/to/rootfs.ext2,if=virtio,format=raw \
-append 'root=/dev/vda console=ttyS0' -vga none -display none \
-serial mon:stdio
QEMU will detect the PVH image and it will use SeaBIOS with the new pvh.bin
optionrom to boot the image.
> * should this work in TCG as well?
Yes, I tried the following command and it works:
./x86_64-softmmu/qemu-system-x86_64 -machine pc,accel=tcg \
-kernel /path/to/vmlinux \
-drive file=/path/to/rootfs.ext2,if=virtio,format=raw \
-append 'root=/dev/vda ro console=ttyS0' -vga none -display none \
-serial mon:stdio
> * are they any special linker rules required for the Xen.notes?
I don't know, but I'll investigate on it.
>
> And finally:
>
> * is this idea of mine a weird abuse of the PVHVM boot protocol or
> does it make sense?
IMHO it isn't an abuse and make sense to boot a bare-metal application
directly in 32-bit mode using the PVH loader.
If you want to share with me a part of your code, I'll try to play with
it.
Cheers,
Stefano
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Qemu-devel] Booting kernels with PVHVM documentation?
2019-03-11 9:29 ` Stefano Garzarella
@ 2019-06-04 18:34 ` Alex Bennée
0 siblings, 0 replies; 6+ messages in thread
From: Alex Bennée @ 2019-06-04 18:34 UTC (permalink / raw)
To: Stefano Garzarella; +Cc: Paolo Bonzini, qemu-devel
Stefano Garzarella <sgarzare@redhat.com> writes:
> Hi Alex,
> sorry for the big delay, but I was traveling without my PC.
>
> On Wed, Mar 06, 2019 at 05:51:05PM +0000, Alex Bennée wrote:
>>
>> Hi,
>>
>> I've been looking at using PVH as an alternative to a long bios boot
>> sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
>> hard to piece together all the bits but I naively thought it would just
>> be a case of adding a few ELF NOTES to my boot.S with something like:
>>
>> ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE, _ASM_PTR 0x100000)
>> ELFNOTE(Xen, XEN_ELFNOTE_ENTRY, _ASM_PTR _start)
>> ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY, _ASM_PTR 0) /* entry == virtbase */
>> ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET, _ASM_PTR 0)
>>
>> .code64
>
> Can you try to use .code32?
> The pvh.bin optionrom will jump to the image in 32-bit mode.
> I don't have a lot of experience, but looking at arch/x86/platform/pvh/head.S
> (Linux), I saw that entry point is under .code32, than it will switch to
> 64-bit mode.
That doesn't seem to make any difference:
./qemu-system-x86_64 -monitor none -display none -chardev stdio,id=out -device isa-debugcon,chardev=out -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello -vga none
qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note
<snip>
>
>> * are they any special linker rules required for the Xen.notes?
>
> I don't know, but I'll investigate on it.
>
>>
>> And finally:
>>
>> * is this idea of mine a weird abuse of the PVHVM boot protocol or
>> does it make sense?
>
> IMHO it isn't an abuse and make sense to boot a bare-metal application
> directly in 32-bit mode using the PVH loader.
>
> If you want to share with me a part of your code, I'll try to play with
> it.
My current hacking branch is:
https://github.com/stsquad/qemu/tree/testing/next-with-x86-64-tests
--
Alex Bennée
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-06-04 18:35 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-06 17:51 [Qemu-devel] Booting kernels with PVHVM documentation? Alex Bennée
2019-03-08 9:57 ` Liam Merwick
2019-03-08 10:43 ` Alex Bennée
2019-03-08 11:48 ` Paolo Bonzini
2019-03-11 9:29 ` Stefano Garzarella
2019-06-04 18:34 ` Alex Bennée
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.