hvm crash on hypercall event channel

hvm crash on hypercall event channel

[Daniel Castro]

Hello All,

I am writing the PV-Drivers for Seabios.

When I put a request on the front ring and issue the hypercall to
notify, the hvm guest crashes.

Here is the dmesg output:

(XEN) realmode.c:116:d10 Failed to emulate insn.
(XEN) realmode.c:166:d10 Real-mode emulation failed @ f000:00001c4b:
0f aa ba b2 00 ec
(XEN) domain_crash called from realmode.c:167
(XEN) Domain 10 (vcpu#0) crashed on cpu#1:
(XEN) ----[ Xen-4.2-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    f000:[<0000000000001c4b>]
(XEN) RFLAGS: 0000000000000097   CONTEXT: hvm guest
(XEN) rax: 00000000000a0000   rbx: 000000000003fef8   rcx: 0000000000000320
(XEN) rdx: 00000000000000b3   rsi: 00000000000fd600   rdi: 0000000000000340
(XEN) rbp: 000000000009a040   rsp: 0000000000000308   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000000000010   cr4: 0000000000000000
(XEN) cr3: 0000000000800000   cr2: 0000000000000000
(XEN) ds: 9940   es: 9940   fs: 0000   gs: 0000   ss: 9940   cs: f000

Here is the code for issue the hypercall:
        	dprintf(1,"Start notify procedure\n");
        	evtchn_send_t send;
        	send.port = GET_GLOBALFLAT(bi->port);
        	dprintf(1,"In notify before hypercall port is %d = %d",send.port);
        	//hypercall_event_channel_op(EVTCHNOP_send, &send);
        	dprintf(1,"read operation notify res %d\n",
hypercall_event_channel_op(EVTCHNOP_send, &send));
Nothing out of the ordinary. Except that the hypercall is issued under
16bit, It works under 32bit.

Any ideas what could be wrong?

-- 
+-=====---------------------------+
| +---------------------------------+ | This space intentionally blank
for notetaking.
| |   | Daniel Castro,                |
| |   | Consultant/Programmer.|
| |   | U Andes                         |
+-------------------------------------+

Re: hvm crash on hypercall event channel

[Keir Fraser]

On 10/04/2012 12:30, "Daniel Castro" <evil.dani@gmail.com> wrote:

> Hello All,
> 
> I am writing the PV-Drivers for Seabios.
> 
> When I put a request on the front ring and issue the hypercall to
> notify, the hvm guest crashes.
> 
> Here is the dmesg output:
> 
> (XEN) realmode.c:116:d10 Failed to emulate insn.
> (XEN) realmode.c:166:d10 Real-mode emulation failed @ f000:00001c4b:
> 0f aa ba b2 00 ec

Looks like instruction RSM (return from SMM mode). Seems unlikely!

However, even if you are trying to run VMCALL (opcode 0F 01 C1) from
realmode it may not work as we emulate real mode for older Intel CPUs, and
our emulator does not include the vmcall instruction. Also the hypercall
stub code we provide to guests is only correct for 32-bit and 64-bit modes.
You can't legitimately use the hypercall stubs from real mode, vm86 mode, or
16-bit protected mode.

Could you just do the hypercalls from 32-bit mode? Our old rombios had a
32-bit code area for stuff like this, quite probably seabios has similar. Or
perhaps if not it could gain this functionality. Hypercalls from 16-bit mode
are not something we care to add support for, I think.

 -- Keir

> (XEN) domain_crash called from realmode.c:167
> (XEN) Domain 10 (vcpu#0) crashed on cpu#1:
> (XEN) ----[ Xen-4.2-unstable  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    1
> (XEN) RIP:    f000:[<0000000000001c4b>]
> (XEN) RFLAGS: 0000000000000097   CONTEXT: hvm guest
> (XEN) rax: 00000000000a0000   rbx: 000000000003fef8   rcx: 0000000000000320
> (XEN) rdx: 00000000000000b3   rsi: 00000000000fd600   rdi: 0000000000000340
> (XEN) rbp: 000000000009a040   rsp: 0000000000000308   r8:  0000000000000000
> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
> (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 0000000000000010   cr4: 0000000000000000
> (XEN) cr3: 0000000000800000   cr2: 0000000000000000
> (XEN) ds: 9940   es: 9940   fs: 0000   gs: 0000   ss: 9940   cs: f000
> 
> Here is the code for issue the hypercall:
> dprintf(1,"Start notify procedure\n");
> evtchn_send_t send;
> send.port = GET_GLOBALFLAT(bi->port);
> dprintf(1,"In notify before hypercall port is %d = %d",send.port);
> //hypercall_event_channel_op(EVTCHNOP_send, &send);
> dprintf(1,"read operation notify res %d\n",
> hypercall_event_channel_op(EVTCHNOP_send, &send));
> Nothing out of the ordinary. Except that the hypercall is issued under
> 16bit, It works under 32bit.
> 
> Any ideas what could be wrong?

Re: hvm crash on hypercall event channel

[Tim Deegan]

At 20:30 +0900 on 10 Apr (1334089815), Daniel Castro wrote:
> Hello All,
> 
> I am writing the PV-Drivers for Seabios.
> 
> When I put a request on the front ring and issue the hypercall to
> notify, the hvm guest crashes.
> 
> Here is the dmesg output:
> 
> (XEN) realmode.c:116:d10 Failed to emulate insn.
> (XEN) realmode.c:166:d10 Real-mode emulation failed @ f000:00001c4b:
> 0f aa ba b2 00 ec

0F AA is RSM, which is a pretty surprising instruction to find in a
hypercall invocation -- or indeed anywhere outside machine-specific SMM
code.  Is there SMM code in SeaBIOS?  It may be that you've ended up
jumping to a misaligned instruction boundary.

> Nothing out of the ordinary. Except that the hypercall is issued under
> 16bit, It works under 32bit.

Are you using the hypercall page to make your hypercall?  Its contents
don't make sense in 16-bit mode, only in 32-bit and 64-bit.  Since the
register arguments are 32-bit anyway you might want to make all your
hypercalls from 32-bit code anyway; otherwise you'll need to make your
own 16-bit stubs, with the right prefixes for the MOV imm32.

Cheers,

Tim.

Re: hvm crash on hypercall event channel

[Daniel Castro]

On Tue, Apr 10, 2012 at 9:13 PM, Tim Deegan <tim@xen.org> wrote:
> At 20:30 +0900 on 10 Apr (1334089815), Daniel Castro wrote:
>> Hello All,
>>
>> I am writing the PV-Drivers for Seabios.
>>
>> When I put a request on the front ring and issue the hypercall to
>> notify, the hvm guest crashes.
>>
>> Here is the dmesg output:
>>
>> (XEN) realmode.c:116:d10 Failed to emulate insn.
>> (XEN) realmode.c:166:d10 Real-mode emulation failed @ f000:00001c4b:
>> 0f aa ba b2 00 ec
>
> 0F AA is RSM, which is a pretty surprising instruction to find in a
> hypercall invocation -- or indeed anywhere outside machine-specific SMM
> code.  Is there SMM code in SeaBIOS?  It may be that you've ended up
> jumping to a misaligned instruction boundary.
>
>> Nothing out of the ordinary. Except that the hypercall is issued under
>> 16bit, It works under 32bit.
>
> Are you using the hypercall page to make your hypercall?  Its contents
> don't make sense in 16-bit mode, only in 32-bit and 64-bit.  Since the
> register arguments are 32-bit anyway you might want to make all your
> hypercalls from 32-bit code anyway; otherwise you'll need to make your
> own 16-bit stubs, with the right prefixes for the MOV imm32.

I have no idea how to fix this :(
>
> Cheers,
>
> Tim.



-- 
+-=====---------------------------+
| +---------------------------------+ | This space intentionally blank
for notetaking.
| |   | Daniel Castro,                |
| |   | Consultant/Programmer.|
| |   | U Andes                         |
+-------------------------------------+

Re: hvm crash on hypercall event channel

[Ian Campbell]

On Wed, 2012-04-11 at 13:02 +0100, Daniel Castro wrote:
> On Tue, Apr 10, 2012 at 9:13 PM, Tim Deegan <tim@xen.org> wrote:
> > At 20:30 +0900 on 10 Apr (1334089815), Daniel Castro wrote:
> >> Hello All,
> >>
> >> I am writing the PV-Drivers for Seabios.
> >>
> >> When I put a request on the front ring and issue the hypercall to
> >> notify, the hvm guest crashes.
> >>
> >> Here is the dmesg output:
> >>
> >> (XEN) realmode.c:116:d10 Failed to emulate insn.
> >> (XEN) realmode.c:166:d10 Real-mode emulation failed @ f000:00001c4b:
> >> 0f aa ba b2 00 ec
> >
> > 0F AA is RSM, which is a pretty surprising instruction to find in a
> > hypercall invocation -- or indeed anywhere outside machine-specific SMM
> > code.  Is there SMM code in SeaBIOS?  It may be that you've ended up
> > jumping to a misaligned instruction boundary.
> >
> >> Nothing out of the ordinary. Except that the hypercall is issued under
> >> 16bit, It works under 32bit.
> >
> > Are you using the hypercall page to make your hypercall?  Its contents
> > don't make sense in 16-bit mode, only in 32-bit and 64-bit.  Since the
> > register arguments are 32-bit anyway you might want to make all your
> > hypercalls from 32-bit code anyway; otherwise you'll need to make your
> > own 16-bit stubs, with the right prefixes for the MOV imm32.
> 
> I have no idea how to fix this :(

It's looking likely that you'll have to go to 32 bit mode to do all of
the actual I/O associated with the PV devices. That ought to simplify a
bunch of stuff WRT handling the rings etc too.

Ian.

Re: hvm crash on hypercall event channel

[Daniel Castro]

On Wed, Apr 11, 2012 at 9:06 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Wed, 2012-04-11 at 13:02 +0100, Daniel Castro wrote:
>> On Tue, Apr 10, 2012 at 9:13 PM, Tim Deegan <tim@xen.org> wrote:
>> > At 20:30 +0900 on 10 Apr (1334089815), Daniel Castro wrote:
>> >> Hello All,
>> >>
>> >> I am writing the PV-Drivers for Seabios.
>> >>
>> >> When I put a request on the front ring and issue the hypercall to
>> >> notify, the hvm guest crashes.
>> >>
>> >> Here is the dmesg output:
>> >>
>> >> (XEN) realmode.c:116:d10 Failed to emulate insn.
>> >> (XEN) realmode.c:166:d10 Real-mode emulation failed @ f000:00001c4b:
>> >> 0f aa ba b2 00 ec
>> >
>> > 0F AA is RSM, which is a pretty surprising instruction to find in a
>> > hypercall invocation -- or indeed anywhere outside machine-specific SMM
>> > code.  Is there SMM code in SeaBIOS?  It may be that you've ended up
>> > jumping to a misaligned instruction boundary.
>> >
>> >> Nothing out of the ordinary. Except that the hypercall is issued under
>> >> 16bit, It works under 32bit.
>> >
>> > Are you using the hypercall page to make your hypercall?  Its contents
>> > don't make sense in 16-bit mode, only in 32-bit and 64-bit.  Since the
>> > register arguments are 32-bit anyway you might want to make all your
>> > hypercalls from 32-bit code anyway; otherwise you'll need to make your
>> > own 16-bit stubs, with the right prefixes for the MOV imm32.
>>
>> I have no idea how to fix this :(
>
> It's looking likely that you'll have to go to 32 bit mode to do all of
> the actual I/O associated with the PV devices. That ought to simplify a
> bunch of stuff WRT handling the rings etc too.

What are my choices?
1. Add support for 16bit hypercalls on Xen.
2. Jump to 32 bit
3. make a 16bit "special" hypercall on SeaBIOS
4. Any other choice?

Thanks to all the comments.

>
> Ian.
>
>
>



-- 
+-=====---------------------------+
| +---------------------------------+ | This space intentionally blank
for notetaking.
| |   | Daniel Castro,                |
| |   | Consultant/Programmer.|
| |   | U Andes                         |
+-------------------------------------+

Re: hvm crash on hypercall event channel

[Keir Fraser]

On 11/04/2012 13:24, "Daniel Castro" <evil.dani@gmail.com> wrote:

>> It's looking likely that you'll have to go to 32 bit mode to do all of
>> the actual I/O associated with the PV devices. That ought to simplify a
>> bunch of stuff WRT handling the rings etc too.
> 
> What are my choices?
> 1. Add support for 16bit hypercalls on Xen.
> 2. Jump to 32 bit

I think this is best. Compile the 32-bit drivers as 32-bit code, then use
stubs to transfer between 16-bit and 32-bit execution modes at run time. Our
old rombios has code you could borrow for this, if seabios doesn't already
have such functionality.

 -- Keir

> 3. make a 16bit "special" hypercall on SeaBIOS
> 4. Any other choice?
> 
> Thanks to all the comments.

Re: hvm crash on hypercall event channel

[Daniel Castro]

On Wed, Apr 11, 2012 at 9:44 PM, Keir Fraser <keir@xen.org> wrote:
> On 11/04/2012 13:24, "Daniel Castro" <evil.dani@gmail.com> wrote:
>
>>> It's looking likely that you'll have to go to 32 bit mode to do all of
>>> the actual I/O associated with the PV devices. That ought to simplify a
>>> bunch of stuff WRT handling the rings etc too.
>>
>> What are my choices?
>> 1. Add support for 16bit hypercalls on Xen.
>> 2. Jump to 32 bit
>
> I think this is best. Compile the 32-bit drivers as 32-bit code, then use
> stubs to transfer between 16-bit and 32-bit execution modes at run time. Our
> old rombios has code you could borrow for this, if seabios doesn't already
> have such functionality.

SeaBIOS allows me to call a function wrapped in 32 bit.
Also I found that I can not write the ring indexes in 16BIT mode
because the memory is marked read-only. So jumping to 32 bit is my
only option.

>
>  -- Keir
>
>> 3. make a 16bit "special" hypercall on SeaBIOS
>> 4. Any other choice?
>>
>> Thanks to all the comments.
>
>



-- 
+-=====---------------------------+
| +---------------------------------+ | This space intentionally blank
for notetaking.
| |   | Daniel Castro,                |
| |   | Consultant/Programmer.|
| |   | U Andes                         |
+-------------------------------------+