Xen PVH domU start-of-day VCPU state

Xen PVH domU start-of-day VCPU state

[Martin Lucina]

Hi,

I'm trying to bootstrap a new PVH-only Xen domU OS "from scratch", to
replace our existing use of Mini-OS for the early boot/low-level support
layer in MirageOS. I've done this by creating new Xen bindings for Solo5
[1], basing them on our existing virtio code [2].

Unfortunately, I can't seem to get past the first few instructions on VCPU
boot. Here's what I have at the moment (abridged):

    .section .note.solo5.xen

            .align  4
            .long   4
            .long   4
            .long   XEN_ELFNOTE_PHYS32_ENTRY
            .ascii "Xen\0"
            .long   _start32

    /* ... */

    .code32

    ENTRY(_start32)
            cld

            lgdt (gdt64_ptr)
            ljmp $0x10, $1f

    1:      movl $0x18, %eax
            movl %eax, %ds
            movl %eax, %es
            movl %eax, %ss

            xorl %eax, %eax
            movl %eax, %fs
            movl %eax, %gs

I have verified, via xl -v create -c ..., that the domain builder appears
to be doing the right thing, and is interpreting the ELF NOTE correctly.
However, for some reason I cannot fathom, I get a triple fault on the ljmp
following the lgdt instruction above:

    (XEN) d31v0 Triple fault - invoking HVM shutdown action 1
    (XEN) *** Dumping Dom31 vcpu#0 state: ***
    (XEN) ----[ Xen-4.11.4-pre  x86_64  debug=n   Not tainted ]----
    (XEN) CPU:    0
    (XEN) RIP:    0000:[<0000000000100028>]
    (XEN) RFLAGS: 0000000000010002   CONTEXT: hvm guest (d31v0)
    (XEN) rax: 0000000000000000   rbx: 0000000000116000   rcx: 0000000000000000
    (XEN) rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
    (XEN) rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
    (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
    (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
    (XEN) r15: 0000000000000000   cr0: 0000000000000011   cr4: 0000000000000000
    (XEN) cr3: 0000000000000000   cr2: 0000000000000000
    (XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
    (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000

Cross-checking 0x100028 via gdb:

    Dump of assembler code for function _start32:
       0x00100020 <+0>:	cld
       0x00100021 <+1>:	lgdtl  0x108040
       0x00100028 <+8>:	ljmp   $0x10,$0x10002f
       0x0010002f <+15>:	mov    $0x18,%eax

I've spent a couple of days trying various things and cross-checking both
with the Mini-OS PVH/HVM startup [3] and the Intel SDM, but no joy. I've
also re-checked the GDT selector values used by the original virtio code
which this is based on, and they appear to be fine.

This is not helped by the fact that the Xen domU PVH start-of-day VCPU
state does not seem to be documented anywhere, with the exception of
"struct hvm_start_info is passed in %ebx" as stated in
arch-x86/hvm/start_info.h.

In case it's relevant, I'm testing with Xen 4.11.4 as shipped with Debian
10, on an Intel Broadwell CPU.

Any ideas? Any help much appreciated.

Thanks,

-mato

[1] https://github.com/mato/solo5/tree/xen/bindings/xen / https://github.com/mato/solo5/commit/f2539d588883a2e8854998c75bdea9b10f113ed6
[2] https://github.com/mato/solo5/tree/xen/bindings/virtio
[3] https://xenbits.xen.org/gitweb/?p=mini-os.git;a=blob;f=arch/x86/x86_hvm.S;h=6e8ad983a16adbe97b343f7dbc17e281ee0c389f;hb=HEAD

Re: Xen PVH domU start-of-day VCPU state

[Jürgen Groß]

On 25.05.20 18:04, Martin Lucina wrote:
> Hi,
> 
> I'm trying to bootstrap a new PVH-only Xen domU OS "from scratch", to
> replace our existing use of Mini-OS for the early boot/low-level support
> layer in MirageOS. I've done this by creating new Xen bindings for Solo5
> [1], basing them on our existing virtio code [2].
> 
> Unfortunately, I can't seem to get past the first few instructions on VCPU
> boot. Here's what I have at the moment (abridged):
> 
>      .section .note.solo5.xen
> 
>              .align  4
>              .long   4
>              .long   4
>              .long   XEN_ELFNOTE_PHYS32_ENTRY
>              .ascii "Xen\0"
>              .long   _start32
> 
>      /* ... */
> 
>      .code32
> 
>      ENTRY(_start32)
>              cld
> 
>              lgdt (gdt64_ptr)
>              ljmp $0x10, $1f

You need to setup virtual addressing and enable 64 bit mode before using
64-bit GDT.

See Mini-OS source arch/x86/x86_hvm.S


Juergen

Re: Xen PVH domU start-of-day VCPU state

[Andrew Cooper]

On 25/05/2020 17:42, Jürgen Groß wrote:
> On 25.05.20 18:04, Martin Lucina wrote:
>> Hi,
>>
>> I'm trying to bootstrap a new PVH-only Xen domU OS "from scratch", to
>> replace our existing use of Mini-OS for the early boot/low-level support
>> layer in MirageOS. I've done this by creating new Xen bindings for Solo5
>> [1], basing them on our existing virtio code [2].
>>
>> Unfortunately, I can't seem to get past the first few instructions on
>> VCPU
>> boot. Here's what I have at the moment (abridged):
>>
>>      .section .note.solo5.xen
>>
>>              .align  4
>>              .long   4
>>              .long   4
>>              .long   XEN_ELFNOTE_PHYS32_ENTRY
>>              .ascii "Xen\0"
>>              .long   _start32
>>
>>      /* ... */
>>
>>      .code32
>>
>>      ENTRY(_start32)
>>              cld
>>
>>              lgdt (gdt64_ptr)
>>              ljmp $0x10, $1f
>
> You need to setup virtual addressing and enable 64 bit mode before using
> 64-bit GDT.
>
> See Mini-OS source arch/x86/x86_hvm.S

Or
https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD

But yes - Juergen is correct.  Until you have enabled long mode, lgdt
will only load the bottom 32 bits of GDTR.base.

Is there a less abridged version to look at?

~Andrew

Re: Xen PVH domU start-of-day VCPU state

[Andrew Cooper]

On 25/05/2020 17:04, Martin Lucina wrote:
> Hi,
>
> I'm trying to bootstrap a new PVH-only Xen domU OS "from scratch", to
> replace our existing use of Mini-OS for the early boot/low-level support
> layer in MirageOS. I've done this by creating new Xen bindings for Solo5
> [1], basing them on our existing virtio code [2].
>
> Unfortunately, I can't seem to get past the first few instructions on VCPU
> boot. Here's what I have at the moment (abridged):
>
>     .section .note.solo5.xen
>
>             .align  4
>             .long   4
>             .long   4
>             .long   XEN_ELFNOTE_PHYS32_ENTRY
>             .ascii "Xen\0"
>             .long   _start32
>
>     /* ... */
>
>     .code32
>
>     ENTRY(_start32)
>             cld
>
>             lgdt (gdt64_ptr)
>             ljmp $0x10, $1f
>
>     1:      movl $0x18, %eax
>             movl %eax, %ds
>             movl %eax, %es
>             movl %eax, %ss
>
>             xorl %eax, %eax
>             movl %eax, %fs
>             movl %eax, %gs
>
> I have verified, via xl -v create -c ..., that the domain builder appears
> to be doing the right thing, and is interpreting the ELF NOTE correctly.
> However, for some reason I cannot fathom, I get a triple fault on the ljmp
> following the lgdt instruction above:
>
>     (XEN) d31v0 Triple fault - invoking HVM shutdown action 1
>     (XEN) *** Dumping Dom31 vcpu#0 state: ***
>     (XEN) ----[ Xen-4.11.4-pre  x86_64  debug=n   Not tainted ]----
>     (XEN) CPU:    0
>     (XEN) RIP:    0000:[<0000000000100028>]
>     (XEN) RFLAGS: 0000000000010002   CONTEXT: hvm guest (d31v0)
>     (XEN) rax: 0000000000000000   rbx: 0000000000116000   rcx: 0000000000000000
>     (XEN) rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
>     (XEN) rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
>     (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
>     (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
>     (XEN) r15: 0000000000000000   cr0: 0000000000000011   cr4: 0000000000000000
>     (XEN) cr3: 0000000000000000   cr2: 0000000000000000
>     (XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
>     (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000

For extra help debugging this, you can dump the vmcs here:

andrewcoop@andrewcoop:/local/xen.git/xen$ git d
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 74c9f84462..8ae23545ae 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -1687,6 +1687,7 @@ void hvm_triple_fault(void)
             "Triple fault - invoking HVM shutdown action %d\n",
             reason);
     vcpu_show_execution_state(v);
+    vmcs_dump_vcpu(v);
     domain_shutdown(d, reason);
 }
 

which will include the segment cache, including the just loaded GDT details.

> Cross-checking 0x100028 via gdb:
>
>     Dump of assembler code for function _start32:
>        0x00100020 <+0>:	cld
>        0x00100021 <+1>:	lgdtl  0x108040
>        0x00100028 <+8>:	ljmp   $0x10,$0x10002f
>        0x0010002f <+15>:	mov    $0x18,%eax
>
> I've spent a couple of days trying various things and cross-checking both
> with the Mini-OS PVH/HVM startup [3] and the Intel SDM, but no joy. I've
> also re-checked the GDT selector values used by the original virtio code
> which this is based on, and they appear to be fine.
>
> This is not helped by the fact that the Xen domU PVH start-of-day VCPU
> state does not seem to be documented anywhere, with the exception of
> "struct hvm_start_info is passed in %ebx" as stated in
> arch-x86/hvm/start_info.h.

https://xenbits.xen.org/docs/unstable/misc/pvh.html

The starting state is described there.  It is 32bit flat mode, very
similar to multiboot's entry.

> In case it's relevant, I'm testing with Xen 4.11.4 as shipped with Debian
> 10, on an Intel Broadwell CPU.
>
> Any ideas?

Sadly no.

From
https://github.com/mato/solo5/commit/f2539d588883a2e8854998c75bdea9b10f113ed6

all data looks to be linked below the 4G boundary, so the 32/64bitness
of lgdt shouldn't matter in this case.

Reordering the logic as per MiniOS/XTF will avoid the need for a 32bit
CS selector - it is safe to run on the ABI-provided %cs until you switch
into 64bit mode.

It might also be interesting to see exactly what value is in gdt64_ptr,
just to check that the base an limit are set sensibly.

~Andrew

Re: Xen PVH domU start-of-day VCPU state

[Martin Lucina]

On Monday, 25.05.2020 at 18:41, Andrew Cooper wrote:
> On 25/05/2020 17:04, Martin Lucina wrote:
> > Hi,
> >
> > I'm trying to bootstrap a new PVH-only Xen domU OS "from scratch", to
> > replace our existing use of Mini-OS for the early boot/low-level support
> > layer in MirageOS. I've done this by creating new Xen bindings for Solo5
> > [1], basing them on our existing virtio code [2].
> >
> > Unfortunately, I can't seem to get past the first few instructions on VCPU
> > boot. Here's what I have at the moment (abridged):
> >
> >     .section .note.solo5.xen
> >
> >             .align  4
> >             .long   4
> >             .long   4
> >             .long   XEN_ELFNOTE_PHYS32_ENTRY
> >             .ascii "Xen\0"
> >             .long   _start32
> >
> >     /* ... */
> >
> >     .code32
> >
> >     ENTRY(_start32)
> >             cld
> >
> >             lgdt (gdt64_ptr)
> >             ljmp $0x10, $1f
> >
> >     1:      movl $0x18, %eax
> >             movl %eax, %ds
> >             movl %eax, %es
> >             movl %eax, %ss
> >
> >             xorl %eax, %eax
> >             movl %eax, %fs
> >             movl %eax, %gs
> >
> > I have verified, via xl -v create -c ..., that the domain builder appears
> > to be doing the right thing, and is interpreting the ELF NOTE correctly.
> > However, for some reason I cannot fathom, I get a triple fault on the ljmp
> > following the lgdt instruction above:
> >
> >     (XEN) d31v0 Triple fault - invoking HVM shutdown action 1
> >     (XEN) *** Dumping Dom31 vcpu#0 state: ***
> >     (XEN) ----[ Xen-4.11.4-pre  x86_64  debug=n   Not tainted ]----
> >     (XEN) CPU:    0
> >     (XEN) RIP:    0000:[<0000000000100028>]
> >     (XEN) RFLAGS: 0000000000010002   CONTEXT: hvm guest (d31v0)
> >     (XEN) rax: 0000000000000000   rbx: 0000000000116000   rcx: 0000000000000000
> >     (XEN) rdx: 0000000000000000   rsi: 0000000000000000   rdi: 0000000000000000
> >     (XEN) rbp: 0000000000000000   rsp: 0000000000000000   r8:  0000000000000000
> >     (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
> >     (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
> >     (XEN) r15: 0000000000000000   cr0: 0000000000000011   cr4: 0000000000000000
> >     (XEN) cr3: 0000000000000000   cr2: 0000000000000000
> >     (XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
> >     (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
> 
> For extra help debugging this, you can dump the vmcs here:
> 
> andrewcoop@andrewcoop:/local/xen.git/xen$ git d
> diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
> index 74c9f84462..8ae23545ae 100644
> --- a/xen/arch/x86/hvm/hvm.c
> +++ b/xen/arch/x86/hvm/hvm.c
> @@ -1687,6 +1687,7 @@ void hvm_triple_fault(void)
>              "Triple fault - invoking HVM shutdown action %d\n",
>              reason);
>      vcpu_show_execution_state(v);
> +    vmcs_dump_vcpu(v);
>      domain_shutdown(d, reason);
>  }
>  
> 
> which will include the segment cache, including the just loaded GDT details.

Thanks, I'll try that and report back.

> 
> > Cross-checking 0x100028 via gdb:
> >
> >     Dump of assembler code for function _start32:
> >        0x00100020 <+0>:	cld
> >        0x00100021 <+1>:	lgdtl  0x108040
> >        0x00100028 <+8>:	ljmp   $0x10,$0x10002f
> >        0x0010002f <+15>:	mov    $0x18,%eax
> >
> > I've spent a couple of days trying various things and cross-checking both
> > with the Mini-OS PVH/HVM startup [3] and the Intel SDM, but no joy. I've
> > also re-checked the GDT selector values used by the original virtio code
> > which this is based on, and they appear to be fine.
> >
> > This is not helped by the fact that the Xen domU PVH start-of-day VCPU
> > state does not seem to be documented anywhere, with the exception of
> > "struct hvm_start_info is passed in %ebx" as stated in
> > arch-x86/hvm/start_info.h.
> 
> https://xenbits.xen.org/docs/unstable/misc/pvh.html
> 
> The starting state is described there.  It is 32bit flat mode, very
> similar to multiboot's entry.
> 
> > In case it's relevant, I'm testing with Xen 4.11.4 as shipped with Debian
> > 10, on an Intel Broadwell CPU.
> >
> > Any ideas?
> 
> Sadly no.
> 
> From
> https://github.com/mato/solo5/commit/f2539d588883a2e8854998c75bdea9b10f113ed6
> 
> all data looks to be linked below the 4G boundary, so the 32/64bitness
> of lgdt shouldn't matter in this case.

That's correct, the virtio code this is based on doesn't use anything above
1GB.

> Reordering the logic as per MiniOS/XTF will avoid the need for a 32bit
> CS selector - it is safe to run on the ABI-provided %cs until you switch
> into 64bit mode.

I can try poking at the order some more, but was aiming for a minimal diff
against virtio to start with.

> It might also be interesting to see exactly what value is in gdt64_ptr,
> just to check that the base an limit are set sensibly.

Seems fine:

    (gdb) info address gdt64_ptr
    Symbol "gdt64_ptr" is at 0x108040 in a file compiled without debugging.
    (gdb) x /1xg 0x108040
    0x108040:	0x000000108000002f
    (gdb) p/x (struct gdtptr)gdt64_ptr
    $3 = {limit = 0x2f, base = 0x108000}

-mato

Re: Xen PVH domU start-of-day VCPU state

[Martin Lucina]

On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
> On 25/05/2020 17:42, Jürgen Groß wrote:
> > You need to setup virtual addressing and enable 64 bit mode before using
> > 64-bit GDT.
> >
> > See Mini-OS source arch/x86/x86_hvm.S
> 
> Or
> https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
> 
> But yes - Juergen is correct.  Until you have enabled long mode, lgdt
> will only load the bottom 32 bits of GDTR.base.

Ah, I missed Jurgen's and your reply here.

LGDT loading only the bottom 32 bits of GDTR.base shouldn't matter.
Examining gdt_ptr some more:

    (gdb) set architecture i386
    The target architecture is assumed to be i386
    (gdb) x /xh 0x108040
    0x108040:	0x002f
    (gdb) x /xw 0x108042
    0x108042:	0x00108000
    (gdb) x /6xb 0x108040
    0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00
    (gdb) x /8xb 0x108040
    0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00	0x00	0x00

> Is there a less abridged version to look at?

https://github.com/mato/solo5/blob/xen/bindings/xen/boot.S

As I wrote in another reply, this boot.S is based off the virtio
(multiboot) boot.S, which has worked fine for years and should have the
same environment (32-bit flat protected mode).

-mato

Re: Xen PVH domU start-of-day VCPU state

[Roger Pau Monné]

On Tue, May 26, 2020 at 10:52:21AM +0200, Martin Lucina wrote:
> On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
> > On 25/05/2020 17:42, Jürgen Groß wrote:
> > > You need to setup virtual addressing and enable 64 bit mode before using
> > > 64-bit GDT.
> > >
> > > See Mini-OS source arch/x86/x86_hvm.S
> > 
> > Or
> > https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
> > 
> > But yes - Juergen is correct.  Until you have enabled long mode, lgdt
> > will only load the bottom 32 bits of GDTR.base.
> 
> Ah, I missed Jurgen's and your reply here.
> 
> LGDT loading only the bottom 32 bits of GDTR.base shouldn't matter.
> Examining gdt_ptr some more:
> 
>     (gdb) set architecture i386
>     The target architecture is assumed to be i386
>     (gdb) x /xh 0x108040
>     0x108040:	0x002f
>     (gdb) x /xw 0x108042
>     0x108042:	0x00108000
>     (gdb) x /6xb 0x108040
>     0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00
>     (gdb) x /8xb 0x108040
>     0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00	0x00	0x00

Could you also print the GDT entry at 0x10 (ie: 0x108000 + 0x10), just
to make sure it contains the right descriptor?

Roger.

Re: Xen PVH domU start-of-day VCPU state

[Roger Pau Monné]

On Tue, May 26, 2020 at 11:34:21AM +0200, Roger Pau Monné wrote:
> On Tue, May 26, 2020 at 10:52:21AM +0200, Martin Lucina wrote:
> > On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
> > > On 25/05/2020 17:42, Jürgen Groß wrote:
> > > > You need to setup virtual addressing and enable 64 bit mode before using
> > > > 64-bit GDT.
> > > >
> > > > See Mini-OS source arch/x86/x86_hvm.S
> > > 
> > > Or
> > > https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
> > > 
> > > But yes - Juergen is correct.  Until you have enabled long mode, lgdt
> > > will only load the bottom 32 bits of GDTR.base.
> > 
> > Ah, I missed Jurgen's and your reply here.
> > 
> > LGDT loading only the bottom 32 bits of GDTR.base shouldn't matter.
> > Examining gdt_ptr some more:
> > 
> >     (gdb) set architecture i386
> >     The target architecture is assumed to be i386
> >     (gdb) x /xh 0x108040
> >     0x108040:	0x002f
> >     (gdb) x /xw 0x108042
> >     0x108042:	0x00108000
> >     (gdb) x /6xb 0x108040
> >     0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00
> >     (gdb) x /8xb 0x108040
> >     0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00	0x00	0x00
> 
> Could you also print the GDT entry at 0x10 (ie: 0x108000 + 0x10), just
> to make sure it contains the right descriptor?

Forgot to ask, but can you also add the output of readelf -lW
<kernel>?

Roger.

Re: Xen PVH domU start-of-day VCPU state

[Martin Lucina]

On Tuesday, 26.05.2020 at 11:34, Roger Pau Monné wrote:
> On Tue, May 26, 2020 at 10:52:21AM +0200, Martin Lucina wrote:
> > On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
> > > On 25/05/2020 17:42, Jürgen Groß wrote:
> > > > You need to setup virtual addressing and enable 64 bit mode before using
> > > > 64-bit GDT.
> > > >
> > > > See Mini-OS source arch/x86/x86_hvm.S
> > > 
> > > Or
> > > https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
> > > 
> > > But yes - Juergen is correct.  Until you have enabled long mode, lgdt
> > > will only load the bottom 32 bits of GDTR.base.
> > 
> > Ah, I missed Jurgen's and your reply here.
> > 
> > LGDT loading only the bottom 32 bits of GDTR.base shouldn't matter.
> > Examining gdt_ptr some more:
> > 
> >     (gdb) set architecture i386
> >     The target architecture is assumed to be i386
> >     (gdb) x /xh 0x108040
> >     0x108040:	0x002f
> >     (gdb) x /xw 0x108042
> >     0x108042:	0x00108000
> >     (gdb) x /6xb 0x108040
> >     0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00
> >     (gdb) x /8xb 0x108040
> >     0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00	0x00	0x00
> 
> Could you also print the GDT entry at 0x10 (ie: 0x108000 + 0x10), just
> to make sure it contains the right descriptor?

I triple-checked that on Friday, but here you go:

    (gdb) x /xg 0x108010
    0x108010:	0x00cf9b000000ffff
    (gdb) x /tg 0x108010
    0x108010:	0000000011001111100110110000000000000000000000001111111111111111

Translates to:

base_31_24 = { 0 }
g = 1 (4 kB)
b = 1 (32-bit)
l = 0 (32-bit)
avl = 0
limit_19_16 = { 1 }
p = 1
dpl = 0
s = 1
type = 1011 (code, exec/read, accessed)
base23_16 = { 0 }
base15_0 = { 0 }
limit_15_0 = { 1 }

type should technically not include accessed, but that shouldn't matter.
In any case, changing it to 1010 does not help.

Looks like I'll have to build a patched Xen as per Andrew's suggestion, and
dump the VMCS on the triple fault.

-mato

Re: Xen PVH domU start-of-day VCPU state

[Martin Lucina]

On Tuesday, 26.05.2020 at 12:03, Roger Pau Monné wrote:
> On Tue, May 26, 2020 at 11:34:21AM +0200, Roger Pau Monné wrote:
> > On Tue, May 26, 2020 at 10:52:21AM +0200, Martin Lucina wrote:
> > > On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
> > > > On 25/05/2020 17:42, Jürgen Groß wrote:
> > > > > You need to setup virtual addressing and enable 64 bit mode before using
> > > > > 64-bit GDT.
> > > > >
> > > > > See Mini-OS source arch/x86/x86_hvm.S
> > > > 
> > > > Or
> > > > https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
> > > > 
> > > > But yes - Juergen is correct.  Until you have enabled long mode, lgdt
> > > > will only load the bottom 32 bits of GDTR.base.
> > > 
> > > Ah, I missed Jurgen's and your reply here.
> > > 
> > > LGDT loading only the bottom 32 bits of GDTR.base shouldn't matter.
> > > Examining gdt_ptr some more:
> > > 
> > >     (gdb) set architecture i386
> > >     The target architecture is assumed to be i386
> > >     (gdb) x /xh 0x108040
> > >     0x108040:	0x002f
> > >     (gdb) x /xw 0x108042
> > >     0x108042:	0x00108000
> > >     (gdb) x /6xb 0x108040
> > >     0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00
> > >     (gdb) x /8xb 0x108040
> > >     0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00	0x00	0x00
> > 
> > Could you also print the GDT entry at 0x10 (ie: 0x108000 + 0x10), just
> > to make sure it contains the right descriptor?
> 
> Forgot to ask, but can you also add the output of readelf -lW
> <kernel>?

    Elf file type is EXEC (Executable file)
    Entry point 0x1001e0
    There are 7 program headers, starting at offset 64

    Program Headers:
      Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
      INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 0x000018 R   0x8
          [Requesting program interpreter: /nonexistent/solo5/]
      LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00626c 0x00626c R E 0x1000
      LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 0x00ed48 RW  0x1000
      NOTE           0x0080ac 0x00000000001070ac 0x00000000001070ac 0x000018 0x000018 R   0x4
      NOTE           0x00f120 0x00000000001070c4 0x00000000001070c4 0x000014 0x000000 R   0x4
      NOTE           0x008088 0x0000000000107088 0x0000000000107088 0x000024 0x000024 R   0x4
      NOTE           0x008000 0x0000000000107000 0x0000000000107000 0x000088 0x000088 R   0x4

     Section to Segment mapping:
      Segment Sections...
       00     .interp
       01     .interp .text .rodata .eh_frame
       02     .note.solo5.manifest .note.solo5.abi .note.solo5.not-openbsd .data .bss
       03     .note.solo5.not-openbsd
       04     .note.solo5.xen
       05     .note.solo5.abi
       06     .note.solo5.manifest

The PT_INTERP and multiple PT_NOTE headers are that way due to specifics of
how Solo5 ABIs work, but I've verified that the domain builder is
interpreting XEN_ELFNOTE_PHYS32_ENTRY correctly.

-mato

Re: Xen PVH domU start-of-day VCPU state

[Roger Pau Monné]

BTW, not sure why but my MUA (Mutt 11.0.3) seems to add everyone on Cc
to the To: field on reply, and drops your email address from the list.
I don't see a 'Reply-to:' on the headers, so I'm not sure why it does
that, but you might want to check your config.

I have to manually fix the headers to properly set the To: field to
your address and the Cc to everyone else.

On Tue, May 26, 2020 at 12:12:03PM +0200, Martin Lucina wrote:
> On Tuesday, 26.05.2020 at 12:03, Roger Pau Monné wrote:
> > On Tue, May 26, 2020 at 11:34:21AM +0200, Roger Pau Monné wrote:
> > > On Tue, May 26, 2020 at 10:52:21AM +0200, Martin Lucina wrote:
> > > > On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
> > > > > On 25/05/2020 17:42, Jürgen Groß wrote:
> > > > > > You need to setup virtual addressing and enable 64 bit mode before using
> > > > > > 64-bit GDT.
> > > > > >
> > > > > > See Mini-OS source arch/x86/x86_hvm.S
> > > > > 
> > > > > Or
> > > > > https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
> > > > > 
> > > > > But yes - Juergen is correct.  Until you have enabled long mode, lgdt
> > > > > will only load the bottom 32 bits of GDTR.base.
> > > > 
> > > > Ah, I missed Jurgen's and your reply here.
> > > > 
> > > > LGDT loading only the bottom 32 bits of GDTR.base shouldn't matter.
> > > > Examining gdt_ptr some more:
> > > > 
> > > >     (gdb) set architecture i386
> > > >     The target architecture is assumed to be i386
> > > >     (gdb) x /xh 0x108040
> > > >     0x108040:	0x002f
> > > >     (gdb) x /xw 0x108042
> > > >     0x108042:	0x00108000
> > > >     (gdb) x /6xb 0x108040
> > > >     0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00
> > > >     (gdb) x /8xb 0x108040
> > > >     0x108040:	0x2f	0x00	0x00	0x80	0x10	0x00	0x00	0x00
> > > 
> > > Could you also print the GDT entry at 0x10 (ie: 0x108000 + 0x10), just
> > > to make sure it contains the right descriptor?
> > 
> > Forgot to ask, but can you also add the output of readelf -lW
> > <kernel>?
> 
>     Elf file type is EXEC (Executable file)
>     Entry point 0x1001e0
>     There are 7 program headers, starting at offset 64
> 
>     Program Headers:
>       Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
>       INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 0x000018 R   0x8
>           [Requesting program interpreter: /nonexistent/solo5/]
>       LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00626c 0x00626c R E 0x1000
>       LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 0x00ed48 RW  0x1000
>       NOTE           0x0080ac 0x00000000001070ac 0x00000000001070ac 0x000018 0x000018 R   0x4
>       NOTE           0x00f120 0x00000000001070c4 0x00000000001070c4 0x000014 0x000000 R   0x4
>       NOTE           0x008088 0x0000000000107088 0x0000000000107088 0x000024 0x000024 R   0x4
>       NOTE           0x008000 0x0000000000107000 0x0000000000107000 0x000088 0x000088 R   0x4
> 
>      Section to Segment mapping:
>       Segment Sections...
>        00     .interp
>        01     .interp .text .rodata .eh_frame
>        02     .note.solo5.manifest .note.solo5.abi .note.solo5.not-openbsd .data .bss
>        03     .note.solo5.not-openbsd
>        04     .note.solo5.xen
>        05     .note.solo5.abi
>        06     .note.solo5.manifest
> 
> The PT_INTERP and multiple PT_NOTE headers are that way due to specifics of
> how Solo5 ABIs work, but I've verified that the domain builder is
> interpreting XEN_ELFNOTE_PHYS32_ENTRY correctly.

Right, just wanted to double check that virtaddr == physaddr since you
didn't use any offset to get the physical address of symbols, but I
guess that if this wasn't correct you won't be even able to execute
the first instruction anyway.

Thanks, Roger.

Re: Xen PVH domU start-of-day VCPU state

[Andrew Cooper]

On 26/05/2020 09:52, Martin Lucina wrote:
> On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
>> On 25/05/2020 17:42, Jürgen Groß wrote:
>>> You need to setup virtual addressing and enable 64 bit mode before using
>>> 64-bit GDT.
>>>
>>> See Mini-OS source arch/x86/x86_hvm.S
>> Or
>> https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
>>
>> But yes - Juergen is correct.  Until you have enabled long mode, lgdt
>> will only load the bottom 32 bits of GDTR.base.
> Ah, I missed Jurgen's and your reply here.

So the mailing list is doing something evil and setting:

Mail-Followup-To: Andrew Cooper <andrew.cooper3@citrix.com>,
    =?iso-8859-1?Q?J=FCrgen_Gro=DF?= <jgross@suse.com>,
    xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org,
    anil@recoil.org, dave@recoil.org

which causes normal replies to cut you out.

~Andrew

Mail-Followup-To (was Re: Xen PVH domU start-of-day VCPU state)

[Martin Lucina]

On Tuesday, 26.05.2020 at 11:58, Andrew Cooper wrote:
> On 26/05/2020 09:52, Martin Lucina wrote:
> > On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
> >> On 25/05/2020 17:42, Jürgen Groß wrote:
> >>> You need to setup virtual addressing and enable 64 bit mode before using
> >>> 64-bit GDT.
> >>>
> >>> See Mini-OS source arch/x86/x86_hvm.S
> >> Or
> >> https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
> >>
> >> But yes - Juergen is correct.  Until you have enabled long mode, lgdt
> >> will only load the bottom 32 bits of GDTR.base.
> > Ah, I missed Jurgen's and your reply here.
> 
> So the mailing list is doing something evil and setting:
> 
> Mail-Followup-To: Andrew Cooper <andrew.cooper3@citrix.com>,
>     =?iso-8859-1?Q?J=FCrgen_Gro=DF?= <jgross@suse.com>,
>     xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org,
>     anil@recoil.org, dave@recoil.org
> 
> which causes normal replies to cut you out.

I _think_ I've fixed this, it was due to ancient Mutt configuration (using
xensource.com / xen.org !?) for xen-devel at my end.

Dropped the other direct Ccs to lessen the noise, but I have no real way of
testing without replying, so here goes.

-mato

Re: Mail-Followup-To (was Re: Xen PVH domU start-of-day VCPU state)

[Andrew Cooper]

On 26/05/2020 12:54, Martin Lucina wrote:
> On Tuesday, 26.05.2020 at 11:58, Andrew Cooper wrote:
>> On 26/05/2020 09:52, Martin Lucina wrote:
>>> On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
>>>> On 25/05/2020 17:42, Jürgen Groß wrote:
>>>>> You need to setup virtual addressing and enable 64 bit mode before using
>>>>> 64-bit GDT.
>>>>>
>>>>> See Mini-OS source arch/x86/x86_hvm.S
>>>> Or
>>>> https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
>>>>
>>>> But yes - Juergen is correct.  Until you have enabled long mode, lgdt
>>>> will only load the bottom 32 bits of GDTR.base.
>>> Ah, I missed Jurgen's and your reply here.
>> So the mailing list is doing something evil and setting:
>>
>> Mail-Followup-To: Andrew Cooper <andrew.cooper3@citrix.com>,
>>     =?iso-8859-1?Q?J=FCrgen_Gro=DF?= <jgross@suse.com>,
>>     xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org,
>>     anil@recoil.org, dave@recoil.org
>>
>> which causes normal replies to cut you out.
> I _think_ I've fixed this, it was due to ancient Mutt configuration (using
> xensource.com / xen.org !?) for xen-devel at my end.
>
> Dropped the other direct Ccs to lessen the noise, but I have no real way of
> testing without replying, so here goes.

Sorry - still no luck.  Had to add you back in manually.

~Andrew

Re: Mail-Followup-To (was Re: Xen PVH domU start-of-day VCPU state)

[Martin Lucina]

On Tuesday, 26.05.2020 at 12:58, Andrew Cooper wrote:
> On 26/05/2020 12:54, Martin Lucina wrote:
> > On Tuesday, 26.05.2020 at 11:58, Andrew Cooper wrote:
> >> On 26/05/2020 09:52, Martin Lucina wrote:
> >>> On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
> >>>> On 25/05/2020 17:42, Jürgen Groß wrote:
> >>>>> You need to setup virtual addressing and enable 64 bit mode before using
> >>>>> 64-bit GDT.
> >>>>>
> >>>>> See Mini-OS source arch/x86/x86_hvm.S
> >>>> Or
> >>>> https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
> >>>>
> >>>> But yes - Juergen is correct.  Until you have enabled long mode, lgdt
> >>>> will only load the bottom 32 bits of GDTR.base.
> >>> Ah, I missed Jurgen's and your reply here.
> >> So the mailing list is doing something evil and setting:
> >>
> >> Mail-Followup-To: Andrew Cooper <andrew.cooper3@citrix.com>,
> >>     =?iso-8859-1?Q?J=FCrgen_Gro=DF?= <jgross@suse.com>,
> >>     xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org,
> >>     anil@recoil.org, dave@recoil.org
> >>
> >> which causes normal replies to cut you out.
> > I _think_ I've fixed this, it was due to ancient Mutt configuration (using
> > xensource.com / xen.org !?) for xen-devel at my end.
> >
> > Dropped the other direct Ccs to lessen the noise, but I have no real way of
> > testing without replying, so here goes.
> 
> Sorry - still no luck.  Had to add you back in manually.

How about now?

-mato

Re: Mail-Followup-To (was Re: Xen PVH domU start-of-day VCPU state)

[Andrew Cooper]

On 26/05/2020 13:41, Martin Lucina wrote:
> On Tuesday, 26.05.2020 at 12:58, Andrew Cooper wrote:
>> On 26/05/2020 12:54, Martin Lucina wrote:
>>> On Tuesday, 26.05.2020 at 11:58, Andrew Cooper wrote:
>>>> On 26/05/2020 09:52, Martin Lucina wrote:
>>>>> On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
>>>>>> On 25/05/2020 17:42, Jürgen Groß wrote:
>>>>>>> You need to setup virtual addressing and enable 64 bit mode before using
>>>>>>> 64-bit GDT.
>>>>>>>
>>>>>>> See Mini-OS source arch/x86/x86_hvm.S
>>>>>> Or
>>>>>> https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
>>>>>>
>>>>>> But yes - Juergen is correct.  Until you have enabled long mode, lgdt
>>>>>> will only load the bottom 32 bits of GDTR.base.
>>>>> Ah, I missed Jurgen's and your reply here.
>>>> So the mailing list is doing something evil and setting:
>>>>
>>>> Mail-Followup-To: Andrew Cooper <andrew.cooper3@citrix.com>,
>>>>     =?iso-8859-1?Q?J=FCrgen_Gro=DF?= <jgross@suse.com>,
>>>>     xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org,
>>>>     anil@recoil.org, dave@recoil.org
>>>>
>>>> which causes normal replies to cut you out.
>>> I _think_ I've fixed this, it was due to ancient Mutt configuration (using
>>> xensource.com / xen.org !?) for xen-devel at my end.
>>>
>>> Dropped the other direct Ccs to lessen the noise, but I have no real way of
>>> testing without replying, so here goes.
>> Sorry - still no luck.  Had to add you back in manually.
> How about now?

That works.

~Andrew

Re: Mail-Followup-To (was Re: Xen PVH domU start-of-day VCPU state)

[Martin Lucina]

On Tuesday, 26.05.2020 at 13:42, Andrew Cooper wrote:
> On 26/05/2020 13:41, Martin Lucina wrote:
> > On Tuesday, 26.05.2020 at 12:58, Andrew Cooper wrote:
> >> On 26/05/2020 12:54, Martin Lucina wrote:
> >>> On Tuesday, 26.05.2020 at 11:58, Andrew Cooper wrote:
> >>>> On 26/05/2020 09:52, Martin Lucina wrote:
> >>>>> On Monday, 25.05.2020 at 17:59, Andrew Cooper wrote:
> >>>>>> On 25/05/2020 17:42, Jürgen Groß wrote:
> >>>>>>> You need to setup virtual addressing and enable 64 bit mode before using
> >>>>>>> 64-bit GDT.
> >>>>>>>
> >>>>>>> See Mini-OS source arch/x86/x86_hvm.S
> >>>>>> Or
> >>>>>> https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen-test-framework.git;a=blob;f=arch/x86/hvm/head.S;h=f7dc72b58ab9ec68538f0087969ab6f72d181d80;hb=HEAD
> >>>>>>
> >>>>>> But yes - Juergen is correct.  Until you have enabled long mode, lgdt
> >>>>>> will only load the bottom 32 bits of GDTR.base.
> >>>>> Ah, I missed Jurgen's and your reply here.
> >>>> So the mailing list is doing something evil and setting:
> >>>>
> >>>> Mail-Followup-To: Andrew Cooper <andrew.cooper3@citrix.com>,
> >>>>     =?iso-8859-1?Q?J=FCrgen_Gro=DF?= <jgross@suse.com>,
> >>>>     xen-devel@lists.xenproject.org, mirageos-devel@lists.xenproject.org,
> >>>>     anil@recoil.org, dave@recoil.org
> >>>>
> >>>> which causes normal replies to cut you out.
> >>> I _think_ I've fixed this, it was due to ancient Mutt configuration (using
> >>> xensource.com / xen.org !?) for xen-devel at my end.
> >>>
> >>> Dropped the other direct Ccs to lessen the noise, but I have no real way of
> >>> testing without replying, so here goes.
> >> Sorry - still no luck.  Had to add you back in manually.
> > How about now?
> 
> That works.

Ok, TIL that I've been doing "subscribe" vs. "lists" in Mutt wrong. Here's
a good explanation of how it's intended to work:

https://lists.debian.org/debian-user/2003/05/msg05016.html

TL;DR if you want a Mail-Followup-To, use "lists", otherwise use
"subscribe".

-mato

Re: Xen PVH domU start-of-day VCPU state

[Martin Lucina]

Oh! I think I've found a solution, even though I don't entirely understand the
problem/root cause:

On Tuesday, 26.05.2020 at 12:12, Martin Lucina wrote:
> > On Tue, May 26, 2020 at 11:34:21AM +0200, Roger Pau Monné wrote:
> > Forgot to ask, but can you also add the output of readelf -lW
> > <kernel>?
> 
>     Elf file type is EXEC (Executable file)
>     Entry point 0x1001e0
>     There are 7 program headers, starting at offset 64
> 
>     Program Headers:
>       Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
>       INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 0x000018 R   0x8
>           [Requesting program interpreter: /nonexistent/solo5/]
>       LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00626c 0x00626c R E 0x1000
>       LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 0x00ed48 RW  0x1000
>       NOTE           0x0080ac 0x00000000001070ac 0x00000000001070ac 0x000018 0x000018 R   0x4
>       NOTE           0x00f120 0x00000000001070c4 0x00000000001070c4 0x000014 0x000000 R   0x4
                                                                               ^^^^^^^^

I should have picked up on the above, but thought it wasn't relevant.

>       NOTE           0x008088 0x0000000000107088 0x0000000000107088 0x000024 0x000024 R   0x4
>       NOTE           0x008000 0x0000000000107000 0x0000000000107000 0x000088 0x000088 R   0x4
> 
>      Section to Segment mapping:
>       Segment Sections...
>        00     .interp
>        01     .interp .text .rodata .eh_frame
>        02     .note.solo5.manifest .note.solo5.abi .note.solo5.not-openbsd .data .bss

And also the missing .note.solo5.xen above.

>        03     .note.solo5.not-openbsd
>        04     .note.solo5.xen
>        05     .note.solo5.abi
>        06     .note.solo5.manifest

Turns out that the .note.solo5.xen section as defined in boot.S was not
marked allocatable, and that was doing <something> that was confusing our
linker script[1] (?).

If I make this simple change:

--- a/bindings/xen/boot.S
+++ b/bindings/xen/boot.S
@@ -32,7 +32,7 @@
 #define ENTRY(x) .text; .globl x; .type x,%function; x:
 #define END(x)   .size x, . - x

-.section .note.solo5.xen
+.section .note.solo5.xen, "a", @note

        .align  4
        .long   4

then I get the expected output from readelf -lW, and I can get as far as
the C _start() with no issues!

FWIW, here's the diff of readelf -lW before/after:

--- before	2020-05-26 17:36:46.117885855 +0200
+++ after	2020-05-26 17:38:07.090508322 +0200
@@ -8,9 +8,9 @@
   INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 0x000018 R   0x8
       [Requesting program interpreter: /nonexistent/solo5/]
   LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00615c 0x00615c R E 0x1000
-  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 0x00ed28 RW  0x1000
+  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x006120 0x00dd28 RW  0x1000
   NOTE           0x0080ac 0x00000000001070ac 0x00000000001070ac 0x000018 0x000018 R   0x4
-  NOTE           0x00f120 0x00000000001070c4 0x00000000001070c4 0x000014 0x000000 R   0x4
+  NOTE           0x0080c4 0x00000000001070c4 0x00000000001070c4 0x000014 0x000014 R   0x4
   NOTE           0x008088 0x0000000000107088 0x0000000000107088 0x000024 0x000024 R   0x4
   NOTE           0x008000 0x0000000000107000 0x0000000000107000 0x000088 0x000088 R   0x4

@@ -18,7 +18,7 @@
   Segment Sections...
    00     .interp
    01     .interp .text .rodata .eh_frame
-   02     .note.solo5.manifest .note.solo5.abi .note.solo5.not-openbsd .data .bss
+   02     .note.solo5.manifest .note.solo5.abi .note.solo5.not-openbsd .note.solo5.xen .data .bss
    03     .note.solo5.not-openbsd
    04     .note.solo5.xen
    05     .note.solo5.abi

-mato

[1] https://github.com/mato/solo5/blob/xen/bindings/xen/solo5_xen.lds

Re: Xen PVH domU start-of-day VCPU state

[Roger Pau Monné]

On Tue, May 26, 2020 at 05:42:24PM +0200, Martin Lucina wrote:
> Oh! I think I've found a solution, even though I don't entirely understand the
> problem/root cause:
> 
> On Tuesday, 26.05.2020 at 12:12, Martin Lucina wrote:
> > > On Tue, May 26, 2020 at 11:34:21AM +0200, Roger Pau Monné wrote:
> > > Forgot to ask, but can you also add the output of readelf -lW
> > > <kernel>?
> > 
> >     Elf file type is EXEC (Executable file)
> >     Entry point 0x1001e0
> >     There are 7 program headers, starting at offset 64
> > 
> >     Program Headers:
> >       Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
> >       INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 0x000018 R   0x8
> >           [Requesting program interpreter: /nonexistent/solo5/]
> >       LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00626c 0x00626c R E 0x1000
> >       LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 0x00ed48 RW  0x1000
> >       NOTE           0x0080ac 0x00000000001070ac 0x00000000001070ac 0x000018 0x000018 R   0x4
> >       NOTE           0x00f120 0x00000000001070c4 0x00000000001070c4 0x000014 0x000000 R   0x4
>                                                                                ^^^^^^^^
> 
> I should have picked up on the above, but thought it wasn't relevant.
> 
> >       NOTE           0x008088 0x0000000000107088 0x0000000000107088 0x000024 0x000024 R   0x4
> >       NOTE           0x008000 0x0000000000107000 0x0000000000107000 0x000088 0x000088 R   0x4
> > 
> >      Section to Segment mapping:
> >       Segment Sections...
> >        00     .interp
> >        01     .interp .text .rodata .eh_frame
> >        02     .note.solo5.manifest .note.solo5.abi .note.solo5.not-openbsd .data .bss
> 
> And also the missing .note.solo5.xen above.
> 
> >        03     .note.solo5.not-openbsd
> >        04     .note.solo5.xen
> >        05     .note.solo5.abi
> >        06     .note.solo5.manifest
> 
> Turns out that the .note.solo5.xen section as defined in boot.S was not
> marked allocatable, and that was doing <something> that was confusing our
> linker script[1] (?).

Hm, I would have said there was no need to load notes into memory, and
hence using a MemSize of 0 would be fine.

Maybe libelf loader was somehow getting confused and not loading the
image properly?

Can you paste the output of `xl -vvv create ...` when using the broken
image?

> 
> If I make this simple change:
> 
> --- a/bindings/xen/boot.S
> +++ b/bindings/xen/boot.S
> @@ -32,7 +32,7 @@
>  #define ENTRY(x) .text; .globl x; .type x,%function; x:
>  #define END(x)   .size x, . - x
> 
> -.section .note.solo5.xen
> +.section .note.solo5.xen, "a", @note
> 
>         .align  4
>         .long   4
> 
> then I get the expected output from readelf -lW, and I can get as far as
> the C _start() with no issues!
> 
> FWIW, here's the diff of readelf -lW before/after:
> 
> --- before	2020-05-26 17:36:46.117885855 +0200
> +++ after	2020-05-26 17:38:07.090508322 +0200
> @@ -8,9 +8,9 @@
>    INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 0x000018 R   0x8
>        [Requesting program interpreter: /nonexistent/solo5/]
>    LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00615c 0x00615c R E 0x1000
> -  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 0x00ed28 RW  0x1000
> +  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x006120 0x00dd28 RW  0x1000

This seems suspicious, there's a change of the size of the LOAD
section, but your change to the note type should not affect the LOAD
section?

Hm, maybe it does because the .note.solo5.xen was considered writable
by default?

Roger.

Re: Xen PVH domU start-of-day VCPU state

[Martin Lucina]

On Tuesday, 26.05.2020 at 18:30, Roger Pau Monné wrote:
> > Turns out that the .note.solo5.xen section as defined in boot.S was not
> > marked allocatable, and that was doing <something> that was confusing our
> > linker script[1] (?).
> 
> Hm, I would have said there was no need to load notes into memory, and
> hence using a MemSize of 0 would be fine.
> 
> Maybe libelf loader was somehow getting confused and not loading the
> image properly?
> 
> Can you paste the output of `xl -vvv create ...` when using the broken
> image?

Here you go:

Parsing config from ./test_hello.xl
libxl: debug: libxl_create.c:1671:do_domain_create: Domain 0:ao 0x5593c42e7e30: create: how=(nil) callback=(nil) poller=0x5593c42e7670
libxl: debug: libxl_create.c:1007:initiate_domain_create: Domain 2:running bootloader
libxl: debug: libxl_bootloader.c:335:libxl__bootloader_run: Domain 2:no bootloader configured, using user supplied kernel
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x5593c42e9590: deregister unregistered
libxl: debug: libxl_sched.c:82:libxl__set_vcpuaffinity: Domain 2:New soft affinity for vcpu 0 has unreachable cpus
domainbuilder: detail: xc_dom_allocate: cmdline="", features=""
domainbuilder: detail: xc_dom_kernel_file: filename="test_hello.xen"
domainbuilder: detail: xc_dom_malloc_filemap    : 191 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.11, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ...
domainbuilder: detail: loader probe OK
xc: detail: ELF: phdr: paddr=0x100000 memsz=0x6264
xc: detail: ELF: phdr: paddr=0x107000 memsz=0xed48
xc: detail: ELF: memory: 0x100000 -> 0x115d48
xc: detail: ELF: note: PHYS32_ENTRY = 0x100020
xc: detail: ELF: Found PVH image
xc: detail: ELF: VIRT_BASE unset, using 0
xc: detail: ELF_PADDR_OFFSET unset, using 0
xc: detail: ELF: addresses:
xc: detail:     virt_base        = 0x0
xc: detail:     elf_paddr_offset = 0x0
xc: detail:     virt_offset      = 0x0
xc: detail:     virt_kstart      = 0x100000
xc: detail:     virt_kend        = 0x115d48
xc: detail:     virt_entry       = 0x1001e0
xc: detail:     p2m_base         = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: hvm-3.0-x86_32: 0x100000 -> 0x115d48
domainbuilder: detail: xc_dom_mem_init: mem 256 MB, pages 0x10000 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x10000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: range: start=0x0 end=0x10000400
domainbuilder: detail: xc_dom_malloc            : 512 kB
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000c00
xc: detail:   2MB PAGES: 0x000000000000007a
xc: detail:   1GB PAGES: 0x0000000000000000
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x100+0x16 at 0x7f5609445000
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x100000 -> 0x116000  (pfn 0x100 + 0x16 pages)
xc: detail: ELF: phdr 1 at 0x7f5609445000 -> 0x7f560944b264
xc: detail: ELF: phdr 2 at 0x7f560944c000 -> 0x7f5609453120
domainbuilder: detail: xc_dom_load_acpi: 64 bytes at address fc008000
domainbuilder: detail: xc_dom_load_acpi: 4096 bytes at address fc000000
domainbuilder: detail: xc_dom_load_acpi: 28672 bytes at address fc001000
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x116+0x1 at 0x7f5609ace000
domainbuilder: detail: xc_dom_alloc_segment:   HVM start info : 0x116000 -> 0x117000  (pfn 0x116 + 0x1 pages)
domainbuilder: detail: alloc_pgtables_hvm: doing nothing
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x117000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 515 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 191 kB
domainbuilder: detail:       domU mmap          : 92 kB
domainbuilder: detail: vcpu_hvm: called
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff001
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_event.c:2194:libxl__ao_progress_report: ao 0x5593c42e7e30: progress report: callback queued aop=0x5593c42fea10
libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x5593c42e7e30: complete, rc=0
libxl: debug: libxl_event.c:1404:egc_run_callbacks: ao 0x5593c42e7e30: progress report: callback aop=0x5593c42fea10
libxl: debug: libxl_create.c:1708:do_domain_create: Domain 0:ao 0x5593c42e7e30: inprogress: poller=0x5593c42e7670, flags=ic
libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x5593c42e7e30: destroy
xencall:buffer: debug: total allocations:233 total releases:233
xencall:buffer: debug: current allocations:0 maximum allocations:3
xencall:buffer: debug: cache current size:3
xencall:buffer: debug: cache hits:215 misses:3 toobig:15
xencall:buffer: debug: total allocations:0 total releases:0
xencall:buffer: debug: current allocations:0 maximum allocations:0
xencall:buffer: debug: cache current size:0
xencall:buffer: debug: cache hits:0 misses:0 toobig:0

> 
> > 
> > If I make this simple change:
> > 
> > --- a/bindings/xen/boot.S
> > +++ b/bindings/xen/boot.S
> > @@ -32,7 +32,7 @@
> >  #define ENTRY(x) .text; .globl x; .type x,%function; x:
> >  #define END(x)   .size x, . - x
> > 
> > -.section .note.solo5.xen
> > +.section .note.solo5.xen, "a", @note
> > 
> >         .align  4
> >         .long   4
> > 
> > then I get the expected output from readelf -lW, and I can get as far as
> > the C _start() with no issues!
> > 
> > FWIW, here's the diff of readelf -lW before/after:
> > 
> > --- before	2020-05-26 17:36:46.117885855 +0200
> > +++ after	2020-05-26 17:38:07.090508322 +0200
> > @@ -8,9 +8,9 @@
> >    INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 0x000018 R   0x8
> >        [Requesting program interpreter: /nonexistent/solo5/]
> >    LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00615c 0x00615c R E 0x1000
> > -  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 0x00ed28 RW  0x1000
> > +  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x006120 0x00dd28 RW  0x1000
> 
> This seems suspicious, there's a change of the size of the LOAD
> section, but your change to the note type should not affect the LOAD
> section?

Indeed.

> 
> Hm, maybe it does because the .note.solo5.xen was considered writable
> by default?

I don't think so. From the broken image:

  [ 8] .note.solo5.xen   NOTE             00000000001070c4  0000f120
       0000000000000014  0000000000000000           0     0     4

From the good image:

  [ 8] .note.solo5.xen   NOTE             00000000001070c4  000080c4
       0000000000000014  0000000000000000   A       0     0     4

-mato

Re: Xen PVH domU start-of-day VCPU state

[Roger Pau Monné]

On Wed, May 27, 2020 at 10:00:08AM +0200, Martin Lucina wrote:
> On Tuesday, 26.05.2020 at 18:30, Roger Pau Monné wrote:
> > > Turns out that the .note.solo5.xen section as defined in boot.S was not
> > > marked allocatable, and that was doing <something> that was confusing our
> > > linker script[1] (?).
> > 
> > Hm, I would have said there was no need to load notes into memory, and
> > hence using a MemSize of 0 would be fine.
> > 
> > Maybe libelf loader was somehow getting confused and not loading the
> > image properly?
> > 
> > Can you paste the output of `xl -vvv create ...` when using the broken
> > image?
> 
> Here you go:
> 
> Parsing config from ./test_hello.xl
> libxl: debug: libxl_create.c:1671:do_domain_create: Domain 0:ao 0x5593c42e7e30: create: how=(nil) callback=(nil) poller=0x5593c42e7670
> libxl: debug: libxl_create.c:1007:initiate_domain_create: Domain 2:running bootloader
> libxl: debug: libxl_bootloader.c:335:libxl__bootloader_run: Domain 2:no bootloader configured, using user supplied kernel
> libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x5593c42e9590: deregister unregistered
> libxl: debug: libxl_sched.c:82:libxl__set_vcpuaffinity: Domain 2:New soft affinity for vcpu 0 has unreachable cpus
> domainbuilder: detail: xc_dom_allocate: cmdline="", features=""
> domainbuilder: detail: xc_dom_kernel_file: filename="test_hello.xen"
> domainbuilder: detail: xc_dom_malloc_filemap    : 191 kB
> domainbuilder: detail: xc_dom_boot_xen_init: ver 4.11, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
> domainbuilder: detail: xc_dom_parse_image: called
> domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
> domainbuilder: detail: loader probe failed
> domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ...
> domainbuilder: detail: loader probe failed
> domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
> domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage
> domainbuilder: detail: loader probe failed
> domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ...
> domainbuilder: detail: loader probe OK
> xc: detail: ELF: phdr: paddr=0x100000 memsz=0x6264
> xc: detail: ELF: phdr: paddr=0x107000 memsz=0xed48
> xc: detail: ELF: memory: 0x100000 -> 0x115d48
> xc: detail: ELF: note: PHYS32_ENTRY = 0x100020
> xc: detail: ELF: Found PVH image
> xc: detail: ELF: VIRT_BASE unset, using 0
> xc: detail: ELF_PADDR_OFFSET unset, using 0
> xc: detail: ELF: addresses:
> xc: detail:     virt_base        = 0x0
> xc: detail:     elf_paddr_offset = 0x0
> xc: detail:     virt_offset      = 0x0
> xc: detail:     virt_kstart      = 0x100000
> xc: detail:     virt_kend        = 0x115d48
> xc: detail:     virt_entry       = 0x1001e0
> xc: detail:     p2m_base         = 0xffffffffffffffff
> domainbuilder: detail: xc_dom_parse_elf_kernel: hvm-3.0-x86_32: 0x100000 -> 0x115d48
> domainbuilder: detail: xc_dom_mem_init: mem 256 MB, pages 0x10000 pages, 4k each
> domainbuilder: detail: xc_dom_mem_init: 0x10000 pages
> domainbuilder: detail: xc_dom_boot_mem_init: called
> domainbuilder: detail: range: start=0x0 end=0x10000400
> domainbuilder: detail: xc_dom_malloc            : 512 kB
> xc: detail: PHYSICAL MEMORY ALLOCATION:
> xc: detail:   4KB PAGES: 0x0000000000000c00
> xc: detail:   2MB PAGES: 0x000000000000007a
> xc: detail:   1GB PAGES: 0x0000000000000000
> domainbuilder: detail: xc_dom_build_image: called
> domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x100+0x16 at 0x7f5609445000
> domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x100000 -> 0x116000  (pfn 0x100 + 0x16 pages)
> xc: detail: ELF: phdr 1 at 0x7f5609445000 -> 0x7f560944b264
> xc: detail: ELF: phdr 2 at 0x7f560944c000 -> 0x7f5609453120
> domainbuilder: detail: xc_dom_load_acpi: 64 bytes at address fc008000
> domainbuilder: detail: xc_dom_load_acpi: 4096 bytes at address fc000000
> domainbuilder: detail: xc_dom_load_acpi: 28672 bytes at address fc001000
> domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x116+0x1 at 0x7f5609ace000
> domainbuilder: detail: xc_dom_alloc_segment:   HVM start info : 0x116000 -> 0x117000  (pfn 0x116 + 0x1 pages)
> domainbuilder: detail: alloc_pgtables_hvm: doing nothing
> domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x117000
> domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
> domainbuilder: detail: xc_dom_boot_image: called
> domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64
> domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
> domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 <= matches
> domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
> domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
> domainbuilder: detail: domain builder memory footprint
> domainbuilder: detail:    allocated
> domainbuilder: detail:       malloc             : 515 kB
> domainbuilder: detail:       anon mmap          : 0 bytes
> domainbuilder: detail:    mapped
> domainbuilder: detail:       file mmap          : 191 kB
> domainbuilder: detail:       domU mmap          : 92 kB
> domainbuilder: detail: vcpu_hvm: called
> domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
> domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff001
> domainbuilder: detail: xc_dom_release: called
> libxl: debug: libxl_event.c:2194:libxl__ao_progress_report: ao 0x5593c42e7e30: progress report: callback queued aop=0x5593c42fea10
> libxl: debug: libxl_event.c:1869:libxl__ao_complete: ao 0x5593c42e7e30: complete, rc=0
> libxl: debug: libxl_event.c:1404:egc_run_callbacks: ao 0x5593c42e7e30: progress report: callback aop=0x5593c42fea10
> libxl: debug: libxl_create.c:1708:do_domain_create: Domain 0:ao 0x5593c42e7e30: inprogress: poller=0x5593c42e7670, flags=ic
> libxl: debug: libxl_event.c:1838:libxl__ao__destroy: ao 0x5593c42e7e30: destroy
> xencall:buffer: debug: total allocations:233 total releases:233
> xencall:buffer: debug: current allocations:0 maximum allocations:3
> xencall:buffer: debug: cache current size:3
> xencall:buffer: debug: cache hits:215 misses:3 toobig:15
> xencall:buffer: debug: total allocations:0 total releases:0
> xencall:buffer: debug: current allocations:0 maximum allocations:0
> xencall:buffer: debug: cache current size:0
> xencall:buffer: debug: cache hits:0 misses:0 toobig:0

That looks fine AFAICT. The program headers seems to be correctly
identified and sized.

> > 
> > > 
> > > If I make this simple change:
> > > 
> > > --- a/bindings/xen/boot.S
> > > +++ b/bindings/xen/boot.S
> > > @@ -32,7 +32,7 @@
> > >  #define ENTRY(x) .text; .globl x; .type x,%function; x:
> > >  #define END(x)   .size x, . - x
> > > 
> > > -.section .note.solo5.xen
> > > +.section .note.solo5.xen, "a", @note
> > > 
> > >         .align  4
> > >         .long   4
> > > 
> > > then I get the expected output from readelf -lW, and I can get as far as
> > > the C _start() with no issues!
> > > 
> > > FWIW, here's the diff of readelf -lW before/after:
> > > 
> > > --- before	2020-05-26 17:36:46.117885855 +0200
> > > +++ after	2020-05-26 17:38:07.090508322 +0200
> > > @@ -8,9 +8,9 @@
> > >    INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 0x000018 R   0x8
> > >        [Requesting program interpreter: /nonexistent/solo5/]
> > >    LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00615c 0x00615c R E 0x1000
> > > -  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 0x00ed28 RW  0x1000
> > > +  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x006120 0x00dd28 RW  0x1000
> > 
> > This seems suspicious, there's a change of the size of the LOAD
> > section, but your change to the note type should not affect the LOAD
> > section?
> 
> Indeed.

You could try to disassemble the text sections with objdump -d (or -D
for all sections) and see if there's a difference between both
versions, but having solved the issue maybe you just want to move
on.

Roger.

Re: Xen PVH domU start-of-day VCPU state

[Martin Lucina]

On Wednesday, 27.05.2020 at 16:41, Roger Pau Monné wrote:
> > > > If I make this simple change:
> > > > 
> > > > --- a/bindings/xen/boot.S
> > > > +++ b/bindings/xen/boot.S
> > > > @@ -32,7 +32,7 @@
> > > >  #define ENTRY(x) .text; .globl x; .type x,%function; x:
> > > >  #define END(x)   .size x, . - x
> > > > 
> > > > -.section .note.solo5.xen
> > > > +.section .note.solo5.xen, "a", @note
> > > > 
> > > >         .align  4
> > > >         .long   4
> > > > 
> > > > then I get the expected output from readelf -lW, and I can get as far as
> > > > the C _start() with no issues!
> > > > 
> > > > FWIW, here's the diff of readelf -lW before/after:
> > > > 
> > > > --- before	2020-05-26 17:36:46.117885855 +0200
> > > > +++ after	2020-05-26 17:38:07.090508322 +0200
> > > > @@ -8,9 +8,9 @@
> > > >    INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 0x000018 R   0x8
> > > >        [Requesting program interpreter: /nonexistent/solo5/]
> > > >    LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00615c 0x00615c R E 0x1000
> > > > -  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 0x00ed28 RW  0x1000
> > > > +  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x006120 0x00dd28 RW  0x1000
> > > 
> > > This seems suspicious, there's a change of the size of the LOAD
> > > section, but your change to the note type should not affect the LOAD
> > > section?
> > 
> > Indeed.
> 
> You could try to disassemble the text sections with objdump -d (or -D
> for all sections) and see if there's a difference between both
> versions, but having solved the issue maybe you just want to move
> on.

I have moved on, making good progress:

    domainbuilder: detail: xc_dom_release: called
    Hello, world!
    Solo5: Xen hvm_start_info @0x0000000000119000
    Solo5: magic=0x336ec578 version=1
    Solo5: cmdline_paddr=0x0
    Solo5: memmap_paddr=0x119878 entries=5
    Solo5: memmap[0] = { 0x0, 0x10000400, 1 }
    Solo5: mem_size=0x10000000
                |      ___|
      __|  _ \  |  _ \ __ \
    \__ \ (   | | (   |  ) |
    ____/\___/ _|\___/____/
    Solo5: Bindings version v0.6.5-4-g57724f8-dirty
    Solo5: Memory map: 256 MB addressable:
    Solo5:   reserved @ (0x0 - 0xfffff)
    Solo5:       text @ (0x100000 - 0x104fff)
    Solo5:     rodata @ (0x105000 - 0x106fff)
    Solo5:       data @ (0x107000 - 0x118fff)
    Solo5:       heap >= 0x119000 < stack < 0x10000000
    Solo5: Clock source: KVM paravirtualized clock
    Solo5: trap: type=#GP ec=0x0 rip=0x103a96 rsp=0xfffff90 rflags=0x2
    Solo5: ABORT: cpu_x86_64.c:181: Fatal trap
    Solo5: Halted

(The #GP is due to the timekeeping code not yet ported to Xen).

Random question: With memory="256" in the xl config, why is the size of the
first XEN_HVM_MEMMAP_TYPE_RAM memmap entry not a multiple of PAGE_SIZE? I
had to align it down, since we put the stack at the top of RAM. 0x10000400
seems... odd.

Thanks all for your help so far, I'm sure I'll run into some more details
that will need clarifying. Enough for today, now going for a walk in the
woods :-)

-mato

Re: Xen PVH domU start-of-day VCPU state

[Roger Pau Monné]

On Wed, May 27, 2020 at 04:57:02PM +0200, Martin Lucina wrote:
> On Wednesday, 27.05.2020 at 16:41, Roger Pau Monné wrote:
> > > > > If I make this simple change:
> > > > > 
> > > > > --- a/bindings/xen/boot.S
> > > > > +++ b/bindings/xen/boot.S
> > > > > @@ -32,7 +32,7 @@
> > > > >  #define ENTRY(x) .text; .globl x; .type x,%function; x:
> > > > >  #define END(x)   .size x, . - x
> > > > > 
> > > > > -.section .note.solo5.xen
> > > > > +.section .note.solo5.xen, "a", @note
> > > > > 
> > > > >         .align  4
> > > > >         .long   4
> > > > > 
> > > > > then I get the expected output from readelf -lW, and I can get as far as
> > > > > the C _start() with no issues!
> > > > > 
> > > > > FWIW, here's the diff of readelf -lW before/after:
> > > > > 
> > > > > --- before	2020-05-26 17:36:46.117885855 +0200
> > > > > +++ after	2020-05-26 17:38:07.090508322 +0200
> > > > > @@ -8,9 +8,9 @@
> > > > >    INTERP         0x001000 0x0000000000100000 0x0000000000100000 0x000018 0x000018 R   0x8
> > > > >        [Requesting program interpreter: /nonexistent/solo5/]
> > > > >    LOAD           0x001000 0x0000000000100000 0x0000000000100000 0x00615c 0x00615c R E 0x1000
> > > > > -  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x007120 0x00ed28 RW  0x1000
> > > > > +  LOAD           0x008000 0x0000000000107000 0x0000000000107000 0x006120 0x00dd28 RW  0x1000
> > > > 
> > > > This seems suspicious, there's a change of the size of the LOAD
> > > > section, but your change to the note type should not affect the LOAD
> > > > section?
> > > 
> > > Indeed.
> > 
> > You could try to disassemble the text sections with objdump -d (or -D
> > for all sections) and see if there's a difference between both
> > versions, but having solved the issue maybe you just want to move
> > on.
> 
> I have moved on, making good progress:
> 
>     domainbuilder: detail: xc_dom_release: called
>     Hello, world!
>     Solo5: Xen hvm_start_info @0x0000000000119000
>     Solo5: magic=0x336ec578 version=1
>     Solo5: cmdline_paddr=0x0
>     Solo5: memmap_paddr=0x119878 entries=5
>     Solo5: memmap[0] = { 0x0, 0x10000400, 1 }
>     Solo5: mem_size=0x10000000
>                 |      ___|
>       __|  _ \  |  _ \ __ \
>     \__ \ (   | | (   |  ) |
>     ____/\___/ _|\___/____/
>     Solo5: Bindings version v0.6.5-4-g57724f8-dirty
>     Solo5: Memory map: 256 MB addressable:
>     Solo5:   reserved @ (0x0 - 0xfffff)
>     Solo5:       text @ (0x100000 - 0x104fff)
>     Solo5:     rodata @ (0x105000 - 0x106fff)
>     Solo5:       data @ (0x107000 - 0x118fff)
>     Solo5:       heap >= 0x119000 < stack < 0x10000000
>     Solo5: Clock source: KVM paravirtualized clock
>     Solo5: trap: type=#GP ec=0x0 rip=0x103a96 rsp=0xfffff90 rflags=0x2
>     Solo5: ABORT: cpu_x86_64.c:181: Fatal trap
>     Solo5: Halted
> 
> (The #GP is due to the timekeeping code not yet ported to Xen).
> 
> Random question: With memory="256" in the xl config, why is the size of the
> first XEN_HVM_MEMMAP_TYPE_RAM memmap entry not a multiple of PAGE_SIZE? I
> had to align it down, since we put the stack at the top of RAM. 0x10000400
> seems... odd.

IIRC some memory is stolen by the domain builder to place stuff (like
the ACPI tables), and the end is not aligned to a page size boundary
because it's not mandatory.

Roger.