Re: [ARM] gvirt_to_maddr fails when DomU is created

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Julien Grall]

Hi Juergen,

I have noticed on the last few threads that your e-mails don't get threaded
correctly. Looking at the source, I can't find the In-Reply-To tag. Do you have 
any issue with your e-mail?

On 29/11/2018 09:51, Juergen Gross wrote:
> On 29/11/2018 10:39, Jan Beulich wrote:
>>>>> On 28.11.18 at 01:05, <andrew.cooper3@citrix.com> wrote:
>>> update_runstate_area() using a virtual address is a complete misfeature,
>>> and the sooner we can replace it, the better.  It's history is with x86
>>> PV guests, where the early ABIs were designed in terms of Linux's
>>> copy_{to,from}_user().
>>>
>>> It is similarly broken in x86 with meltdown mitigations, as well as SMAP
>>> considerations (PAN in ARM, iirc).
>>>
>>> We've got two options.  Invent a new API which takes a gfn/gaddr, or
>>> retrofit the API to be "you pass a virtual address, we translate to
>>> gfn/gaddr, then update that".  Perhaps both.
>>>
>>> When this was last discussed, I think the "onetime translate to
>>> gfn/gaddr" was a good enough compatibility to cope with existing guests,
>>> but that we should have a more clean way for modern guests.
>>
>> I don't think a one-time translate can be a replacement without
>> the guest giving its consent, at which point the guest could as
>> well be switched to whatever the replacement interface is going
>> to be. Aiui (the introduction of the functionality predating my
>> involvement with Xen) the original idea was that guests would
>> specifically be allowed to context switch the mapping of the
>> involved linear address.
>>
>> Furthermore for x86-64 guests we already have logic to deal
>> with the case where there is no present mapping at the time
>> the write is to occur, as that's a common situation with x86-64
>> guest user mode running with kernel page tables removed.
>> For x86-32 guests with Meltdown mitigation in place something
>> similar might indeed be needed. Whether something like this is
>> doable on ARM depends on whether Xen has a way to know
>> at which point missing mappings re-appear.
> 
> In any case we want some interface using gfn/gaddr for performance
> reasons: Always having to do a vaddr->gaddr translation is expensive
> (especially for HVM/PVH and probably on ARM, too), so we should try to
> avoid that.

On Arm, performance is one of the reason, but not the main one. Using Virtual 
Address is by default unreliable as you have no way to prevent the guest to play 
with its page-tables. For instance, Arm requires to go through an invalid state 
when updating the page-tables entry under certain condition (e.g superpage <-> 
small mapping).

The long-term goal is to use guest physical address for all hypercalls rather 
than guest virtual address.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Jan Beulich]

>>> On 29.11.18 at 11:03, <andrew.cooper3@citrix.com> wrote:
> Furthermore, looking at the content held in the runstate area, what do
> guests actually use the information for?  It looks like it is of
> marginal use for diagnostics within the guest.

Correct steal time accounting, at the very least, depends on this
iirc. Steal time accounting being wrong has lead to multiple support
calls here in the past, so I don't think I'd subscribe to the "marginal"
part.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Andrew Cooper]

On 29/11/2018 09:39, Jan Beulich wrote:
>>>> On 28.11.18 at 01:05, <andrew.cooper3@citrix.com> wrote:
>> update_runstate_area() using a virtual address is a complete misfeature,
>> and the sooner we can replace it, the better.  It's history is with x86
>> PV guests, where the early ABIs were designed in terms of Linux's
>> copy_{to,from}_user().
>>
>> It is similarly broken in x86 with meltdown mitigations, as well as SMAP
>> considerations (PAN in ARM, iirc).
>>
>> We've got two options.  Invent a new API which takes a gfn/gaddr, or
>> retrofit the API to be "you pass a virtual address, we translate to
>> gfn/gaddr, then update that".  Perhaps both.
>>
>> When this was last discussed, I think the "onetime translate to
>> gfn/gaddr" was a good enough compatibility to cope with existing guests,
>> but that we should have a more clean way for modern guests.
> I don't think a one-time translate can be a replacement without
> the guest giving its consent, at which point the guest could as
> well be switched to whatever the replacement interface is going
> to be. Aiui (the introduction of the functionality predating my
> involvement with Xen) the original idea was that guests would
> specifically be allowed to context switch the mapping of the
> involved linear address.

That entirely depends on how guests currently use the address they set
up.  The purpose of investigating an option like this is specifically to
avoid needing to alter the guest...

>
> Furthermore for x86-64 guests we already have logic to deal
> with the case where there is no present mapping at the time
> the write is to occur, as that's a common situation with x86-64
> guest user mode running with kernel page tables removed.
> For x86-32 guests with Meltdown mitigation in place something
> similar might indeed be needed. Whether something like this is
> doable on ARM depends on whether Xen has a way to know
> at which point missing mappings re-appear.

... so we can get something which works without having to do a full
pagewalk (and nested vmexit!) on every context switch and each time we
toggle the guest pagetables.

Furthermore, looking at the content held in the runstate area, what do
guests actually use the information for?  It looks like it is of
marginal use for diagnostics within the guest.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Juergen Gross]

On 29/11/2018 10:39, Jan Beulich wrote:
>>>> On 28.11.18 at 01:05, <andrew.cooper3@citrix.com> wrote:
>> update_runstate_area() using a virtual address is a complete misfeature,
>> and the sooner we can replace it, the better.  It's history is with x86
>> PV guests, where the early ABIs were designed in terms of Linux's
>> copy_{to,from}_user().
>>
>> It is similarly broken in x86 with meltdown mitigations, as well as SMAP
>> considerations (PAN in ARM, iirc).
>>
>> We've got two options.  Invent a new API which takes a gfn/gaddr, or
>> retrofit the API to be "you pass a virtual address, we translate to
>> gfn/gaddr, then update that".  Perhaps both.
>>
>> When this was last discussed, I think the "onetime translate to
>> gfn/gaddr" was a good enough compatibility to cope with existing guests,
>> but that we should have a more clean way for modern guests.
> 
> I don't think a one-time translate can be a replacement without
> the guest giving its consent, at which point the guest could as
> well be switched to whatever the replacement interface is going
> to be. Aiui (the introduction of the functionality predating my
> involvement with Xen) the original idea was that guests would
> specifically be allowed to context switch the mapping of the
> involved linear address.
> 
> Furthermore for x86-64 guests we already have logic to deal
> with the case where there is no present mapping at the time
> the write is to occur, as that's a common situation with x86-64
> guest user mode running with kernel page tables removed.
> For x86-32 guests with Meltdown mitigation in place something
> similar might indeed be needed. Whether something like this is
> doable on ARM depends on whether Xen has a way to know
> at which point missing mappings re-appear.

In any case we want some interface using gfn/gaddr for performance
reasons: Always having to do a vaddr->gaddr translation is expensive
(especially for HVM/PVH and probably on ARM, too), so we should try to
avoid that.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Jan Beulich]

>>> On 28.11.18 at 01:05, <andrew.cooper3@citrix.com> wrote:
> update_runstate_area() using a virtual address is a complete misfeature,
> and the sooner we can replace it, the better.  It's history is with x86
> PV guests, where the early ABIs were designed in terms of Linux's
> copy_{to,from}_user().
> 
> It is similarly broken in x86 with meltdown mitigations, as well as SMAP
> considerations (PAN in ARM, iirc).
> 
> We've got two options.  Invent a new API which takes a gfn/gaddr, or
> retrofit the API to be "you pass a virtual address, we translate to
> gfn/gaddr, then update that".  Perhaps both.
> 
> When this was last discussed, I think the "onetime translate to
> gfn/gaddr" was a good enough compatibility to cope with existing guests,
> but that we should have a more clean way for modern guests.

I don't think a one-time translate can be a replacement without
the guest giving its consent, at which point the guest could as
well be switched to whatever the replacement interface is going
to be. Aiui (the introduction of the functionality predating my
involvement with Xen) the original idea was that guests would
specifically be allowed to context switch the mapping of the
involved linear address.

Furthermore for x86-64 guests we already have logic to deal
with the case where there is no present mapping at the time
the write is to occur, as that's a common situation with x86-64
guest user mode running with kernel page tables removed.
For x86-32 guests with Meltdown mitigation in place something
similar might indeed be needed. Whether something like this is
doable on ARM depends on whether Xen has a way to know
at which point missing mappings re-appear.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Julien Grall]

On 11/28/18 6:10 PM, Volodymyr Babchuk wrote:
> Hi Julien,

Hi Volodymyr,

> On Tue, 27 Nov 2018 at 21:40, Julien Grall <julien.grall@arm.com> wrote:
>>> After creating domU, I'm seeing lots of this messages from hypervisor:
>>>
>>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
>>> flags=0x1 par=0x809
>>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f00
>>> flags=0x1 par=0x809
>>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
>>> flags=0x1 par=0x809
>>>
>>> Interestingly, I'm getting them from both Dom0 and DomU:
>>>
>>> (XEN) p2m.c:1442: d0v0: gvirt_to_maddr failed va=0xffff80003efd7f0f
>>> flags=0x1 par=0x809
>>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
>>> flags=0x1 par=0x809
>>>
>>> But only after DomU is created.
>>>
>>> I attached GDB and found that this is caused by update_runstate_area:
>>>
>>> (gdb) bt
>>> #0  get_page_from_gva (v=0x80005dbe2000, v@entry=0x22f2c8 <schedule+1236>,
>>>       va=va@entry=18446603337277996815, flags=flags@entry=1) at p2m.c:1440
>>> #1  0x000000000024e320 in translate_get_page (write=true, linear=true,
>>> addr=18446603337277996815,
>>>       info=...) at guestcopy.c:37
>>> #2  copy_guest (buf=buf@entry=0x80005dbe20d7,
>>> addr=addr@entry=18446603337277996815, len=len@entry=1,
>>>       info=..., flags=flags@entry=6) at guestcopy.c:69
>>> #3  0x000000000024e45c in raw_copy_to_guest (to=to@entry=0xffff80003efd7f0f,
>>>       from=from@entry=0x80005dbe20d7, len=len@entry=1) at guestcopy.c:110
>>> #4  0x00000000002497b4 in update_runstate_area
>>> (v=v@entry=0x80005dbe2000) at domain.c:287
>>> #5  0x0000000000249eb8 in context_switch (prev=prev@entry=0x80005dbe2000,
>>>       next=next@entry=0x80005bf3c000) at domain.c:344
>>> #6  0x000000000022f2c8 in schedule () at schedule.c:1583
>>> #7  0x0000000000232c10 in __do_softirq
>>> (ignore_mask=ignore_mask@entry=0) at softirq.c:50
>>> #8  0x0000000000232ca4 in do_softirq () at softirq.c:64
>>> #9  0x0000000000258254 in leave_hypervisor_tail () at traps.c:2302
>>>
>>> This issue is encountered on QEMU-ARMv8. Dom0 kernel is Linux 4.19.0
>>> My XEN master is at d8ffac1f7 "xen/arm: gic: Remove duplicated comment
>>> in do_sgi"
>>>
>>> The same setup worked perfectly with Xen 4.10.2
>>
>> The message is only printed in debug build. Do you have CONFIG_DEBUG
>> enabled?
>>
> Yes, I do.
> 
>> update_runstate_area is using a guest virtual address to update the vCPU
>> runstate. It blindly assumes the vCPU runstate will always be mapped in
>> stage-1 page-tables. However, if KPTI (Kernel Page Table Isolation) is
>> enabled the kernel address space (and therefore the vCPU runstate) will
>> not be mapped when running at EL0.
> I tried to disable KPTI for both Dom0 and DomU kernels (with nopti
> option) and this didn't helped at all.

nopti is x86 specific. So did you mean kpti=no?

> I can verify, that kernel does not print "CPU features: detected:
> Kernel page table isolation (KPTI)", but that's all.

So you should see something similar to:

CPU features: kernel page table isolation forced OFF by command line option

Correct?

> 
> Strangely, I'm starting to see this messages only after I create DomU.
> If this really would be triggered
> by KPTI, then I should see those errors right from the boot, right?

Not necessarily, you need to have a context switch happening while you 
are at EL0 to trigger the issue. That's unlikely going to happen if you 
have less vCPUs running than available pCPUs. There are more chance to 
happen when starting you DomU.

Anyway, it is quite interesting because I also managed to reproduce it 
with KPTI turned off (i.e kpti=no).

The PAR_EL1 contains 0x809 which tells us this is a level 0 translation 
fault when walking stage-1. So the virtual address is definitely not 
mapped. I added some code to dump the guest vCPU registers on the fault. 
All the fault happen at EL0 so somehow the address is getting unmapped 
when running at EL0.

I have the feeling that kpti=no does not fully disable the feature. I 
will have the chat tomorrow with my team to see how the option should be 
behave.

In any case, passing a virtual address is just the wrong things to do as 
the guest is free to do whatever it wants in term of page-tables. The 
discussion in this thread is an example of what could go wrong :).

So we still want to fix the hypercall no matter the outcome of the 
discussion regarding kpti=no.

Finally, for the sake of clarification turning off kpti=no is not 
recommended unless you really trust your userspace applications. I was 
interested to know whether the problem was related to the feature or 
something different :).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Volodymyr Babchuk]

Hi Julien,

On Tue, 27 Nov 2018 at 21:40, Julien Grall <julien.grall@arm.com> wrote:
> > After creating domU, I'm seeing lots of this messages from hypervisor:
> >
> > (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
> > flags=0x1 par=0x809
> > (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f00
> > flags=0x1 par=0x809
> > (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
> > flags=0x1 par=0x809
> >
> > Interestingly, I'm getting them from both Dom0 and DomU:
> >
> > (XEN) p2m.c:1442: d0v0: gvirt_to_maddr failed va=0xffff80003efd7f0f
> > flags=0x1 par=0x809
> > (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
> > flags=0x1 par=0x809
> >
> > But only after DomU is created.
> >
> > I attached GDB and found that this is caused by update_runstate_area:
> >
> > (gdb) bt
> > #0  get_page_from_gva (v=0x80005dbe2000, v@entry=0x22f2c8 <schedule+1236>,
> >      va=va@entry=18446603337277996815, flags=flags@entry=1) at p2m.c:1440
> > #1  0x000000000024e320 in translate_get_page (write=true, linear=true,
> > addr=18446603337277996815,
> >      info=...) at guestcopy.c:37
> > #2  copy_guest (buf=buf@entry=0x80005dbe20d7,
> > addr=addr@entry=18446603337277996815, len=len@entry=1,
> >      info=..., flags=flags@entry=6) at guestcopy.c:69
> > #3  0x000000000024e45c in raw_copy_to_guest (to=to@entry=0xffff80003efd7f0f,
> >      from=from@entry=0x80005dbe20d7, len=len@entry=1) at guestcopy.c:110
> > #4  0x00000000002497b4 in update_runstate_area
> > (v=v@entry=0x80005dbe2000) at domain.c:287
> > #5  0x0000000000249eb8 in context_switch (prev=prev@entry=0x80005dbe2000,
> >      next=next@entry=0x80005bf3c000) at domain.c:344
> > #6  0x000000000022f2c8 in schedule () at schedule.c:1583
> > #7  0x0000000000232c10 in __do_softirq
> > (ignore_mask=ignore_mask@entry=0) at softirq.c:50
> > #8  0x0000000000232ca4 in do_softirq () at softirq.c:64
> > #9  0x0000000000258254 in leave_hypervisor_tail () at traps.c:2302
> >
> > This issue is encountered on QEMU-ARMv8. Dom0 kernel is Linux 4.19.0
> > My XEN master is at d8ffac1f7 "xen/arm: gic: Remove duplicated comment
> > in do_sgi"
> >
> > The same setup worked perfectly with Xen 4.10.2
>
> The message is only printed in debug build. Do you have CONFIG_DEBUG
> enabled?
>
Yes, I do.

> update_runstate_area is using a guest virtual address to update the vCPU
> runstate. It blindly assumes the vCPU runstate will always be mapped in
> stage-1 page-tables. However, if KPTI (Kernel Page Table Isolation) is
> enabled the kernel address space (and therefore the vCPU runstate) will
> not be mapped when running at EL0.
I tried to disable KPTI for both Dom0 and DomU kernels (with nopti
option) and this didn't helped at all.
I can verify, that kernel does not print "CPU features: detected:
Kernel page table isolation (KPTI)", but that's all.

Strangely, I'm starting to see this messages only after I create DomU.
If this really would be triggered
by KPTI, then I should see those errors right from the boot, right?

--
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@gmail.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Andrew Cooper]

On 28/11/2018 00:05, Andrew Cooper wrote:
> On 27/11/2018 19:40, Julien Grall wrote:
>> (+ Stefano)
>>
>> On 11/27/18 5:12 PM, Volodymyr Babchuk wrote:
>>> Hello community,
>> Hi Volodymyr,
>>
>>> After creating domU, I'm seeing lots of this messages from hypervisor:
>>>
>>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
>>> flags=0x1 par=0x809
>>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f00
>>> flags=0x1 par=0x809
>>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
>>> flags=0x1 par=0x809
>>>
>>> Interestingly, I'm getting them from both Dom0 and DomU:
>>>
>>> (XEN) p2m.c:1442: d0v0: gvirt_to_maddr failed va=0xffff80003efd7f0f
>>> flags=0x1 par=0x809
>>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
>>> flags=0x1 par=0x809
>>>
>>> But only after DomU is created.
>>>
>>> I attached GDB and found that this is caused by update_runstate_area:
>>>
>>> (gdb) bt
>>> #0  get_page_from_gva (v=0x80005dbe2000, v@entry=0x22f2c8
>>> <schedule+1236>,
>>>      va=va@entry=18446603337277996815, flags=flags@entry=1) at
>>> p2m.c:1440
>>> #1  0x000000000024e320 in translate_get_page (write=true, linear=true,
>>> addr=18446603337277996815,
>>>      info=...) at guestcopy.c:37
>>> #2  copy_guest (buf=buf@entry=0x80005dbe20d7,
>>> addr=addr@entry=18446603337277996815, len=len@entry=1,
>>>      info=..., flags=flags@entry=6) at guestcopy.c:69
>>> #3  0x000000000024e45c in raw_copy_to_guest
>>> (to=to@entry=0xffff80003efd7f0f,
>>>      from=from@entry=0x80005dbe20d7, len=len@entry=1) at guestcopy.c:110
>>> #4  0x00000000002497b4 in update_runstate_area
>>> (v=v@entry=0x80005dbe2000) at domain.c:287
>>> #5  0x0000000000249eb8 in context_switch
>>> (prev=prev@entry=0x80005dbe2000,
>>>      next=next@entry=0x80005bf3c000) at domain.c:344
>>> #6  0x000000000022f2c8 in schedule () at schedule.c:1583
>>> #7  0x0000000000232c10 in __do_softirq
>>> (ignore_mask=ignore_mask@entry=0) at softirq.c:50
>>> #8  0x0000000000232ca4 in do_softirq () at softirq.c:64
>>> #9  0x0000000000258254 in leave_hypervisor_tail () at traps.c:2302
>>>
>>> This issue is encountered on QEMU-ARMv8. Dom0 kernel is Linux 4.19.0
>>> My XEN master is at d8ffac1f7 "xen/arm: gic: Remove duplicated comment
>>> in do_sgi"
>>>
>>> The same setup worked perfectly with Xen 4.10.2
>> The message is only printed in debug build. Do you have CONFIG_DEBUG
>> enabled?
>>
>>> Julien, I saw on mailing list, that you paid attention to issues with
>>> gvirt_to_maddr,
>>> so maybe you can be interested in this.
>> Which thread are you speaking about? The problem is not because of
>> gvirt_to_maddr but of how update_runstate_area is working at the moment.
>>
>> update_runstate_area is using a guest virtual address to update the
>> vCPU runstate. It blindly assumes the vCPU runstate will always be
>> mapped in stage-1 page-tables. However, if KPTI (Kernel Page Table
>> Isolation) is enabled the kernel address space (and therefore the vCPU
>> runstate) will not be mapped when running at EL0.
>>
>> So if you are restoring a vCPU that was executing code at EL0 then
>> update_runstate_area will fail as the address is not mapped. There are
>> a few solution suggested on the ML (see [1]). However I haven't had
>> time to look at properly how to implement them.
>>
>> KPTI is getting used more widely (e.g meltdown and KASLR). So it would
>> be good if we try to solve this problem sooner. I would be happy to
>> review patches and/or provide advice if you want to tackle the problem.
>>
>> Cheers,
>>
>> [1] https://lists.xen.org/archives/html/xen-devel/2018-03/msg00223.html
>>
> update_runstate_area() using a virtual address is a complete misfeature,
> and the sooner we can replace it, the better.  It's history is with x86
> PV guests, where the early ABIs were designed in terms of Linux's
> copy_{to,from}_user().
>
> It is similarly broken in x86 with meltdown mitigations, as well as SMAP
> considerations (PAN in ARM, iirc).
>
> We've got two options.  Invent a new API which takes a gfn/gaddr, or
> retrofit the API to be "you pass a virtual address, we translate to
> gfn/gaddr, then update that".  Perhaps both.
>
> When this was last discussed, I think the "onetime translate to
> gfn/gaddr" was a good enough compatibility to cope with existing guests,
> but that we should have a more clean way for modern guests.

Or alternatively, see if we can actually get away without it.  A lot of
the early Xen paravirtual functionality can probably be done without, or
designed in a better way entirely.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Andrew Cooper]

On 27/11/2018 19:40, Julien Grall wrote:
> (+ Stefano)
>
> On 11/27/18 5:12 PM, Volodymyr Babchuk wrote:
>> Hello community,
>
> Hi Volodymyr,
>
>>
>> After creating domU, I'm seeing lots of this messages from hypervisor:
>>
>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
>> flags=0x1 par=0x809
>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f00
>> flags=0x1 par=0x809
>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
>> flags=0x1 par=0x809
>>
>> Interestingly, I'm getting them from both Dom0 and DomU:
>>
>> (XEN) p2m.c:1442: d0v0: gvirt_to_maddr failed va=0xffff80003efd7f0f
>> flags=0x1 par=0x809
>> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
>> flags=0x1 par=0x809
>>
>> But only after DomU is created.
>>
>> I attached GDB and found that this is caused by update_runstate_area:
>>
>> (gdb) bt
>> #0  get_page_from_gva (v=0x80005dbe2000, v@entry=0x22f2c8
>> <schedule+1236>,
>>      va=va@entry=18446603337277996815, flags=flags@entry=1) at
>> p2m.c:1440
>> #1  0x000000000024e320 in translate_get_page (write=true, linear=true,
>> addr=18446603337277996815,
>>      info=...) at guestcopy.c:37
>> #2  copy_guest (buf=buf@entry=0x80005dbe20d7,
>> addr=addr@entry=18446603337277996815, len=len@entry=1,
>>      info=..., flags=flags@entry=6) at guestcopy.c:69
>> #3  0x000000000024e45c in raw_copy_to_guest
>> (to=to@entry=0xffff80003efd7f0f,
>>      from=from@entry=0x80005dbe20d7, len=len@entry=1) at guestcopy.c:110
>> #4  0x00000000002497b4 in update_runstate_area
>> (v=v@entry=0x80005dbe2000) at domain.c:287
>> #5  0x0000000000249eb8 in context_switch
>> (prev=prev@entry=0x80005dbe2000,
>>      next=next@entry=0x80005bf3c000) at domain.c:344
>> #6  0x000000000022f2c8 in schedule () at schedule.c:1583
>> #7  0x0000000000232c10 in __do_softirq
>> (ignore_mask=ignore_mask@entry=0) at softirq.c:50
>> #8  0x0000000000232ca4 in do_softirq () at softirq.c:64
>> #9  0x0000000000258254 in leave_hypervisor_tail () at traps.c:2302
>>
>> This issue is encountered on QEMU-ARMv8. Dom0 kernel is Linux 4.19.0
>> My XEN master is at d8ffac1f7 "xen/arm: gic: Remove duplicated comment
>> in do_sgi"
>>
>> The same setup worked perfectly with Xen 4.10.2
>
> The message is only printed in debug build. Do you have CONFIG_DEBUG
> enabled?
>
>>
>> Julien, I saw on mailing list, that you paid attention to issues with
>> gvirt_to_maddr,
>> so maybe you can be interested in this.
>
> Which thread are you speaking about? The problem is not because of
> gvirt_to_maddr but of how update_runstate_area is working at the moment.
>
> update_runstate_area is using a guest virtual address to update the
> vCPU runstate. It blindly assumes the vCPU runstate will always be
> mapped in stage-1 page-tables. However, if KPTI (Kernel Page Table
> Isolation) is enabled the kernel address space (and therefore the vCPU
> runstate) will not be mapped when running at EL0.
>
> So if you are restoring a vCPU that was executing code at EL0 then
> update_runstate_area will fail as the address is not mapped. There are
> a few solution suggested on the ML (see [1]). However I haven't had
> time to look at properly how to implement them.
>
> KPTI is getting used more widely (e.g meltdown and KASLR). So it would
> be good if we try to solve this problem sooner. I would be happy to
> review patches and/or provide advice if you want to tackle the problem.
>
> Cheers,
>
> [1] https://lists.xen.org/archives/html/xen-devel/2018-03/msg00223.html
>

update_runstate_area() using a virtual address is a complete misfeature,
and the sooner we can replace it, the better.  It's history is with x86
PV guests, where the early ABIs were designed in terms of Linux's
copy_{to,from}_user().

It is similarly broken in x86 with meltdown mitigations, as well as SMAP
considerations (PAN in ARM, iirc).

We've got two options.  Invent a new API which takes a gfn/gaddr, or
retrofit the API to be "you pass a virtual address, we translate to
gfn/gaddr, then update that".  Perhaps both.

When this was last discussed, I think the "onetime translate to
gfn/gaddr" was a good enough compatibility to cope with existing guests,
but that we should have a more clean way for modern guests.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Stefano Stabellini]

On Tue, 27 Nov 2018, Julien Grall wrote:
> (+ Stefano)
> 
> On 11/27/18 5:12 PM, Volodymyr Babchuk wrote:
> > Hello community,
> 
> Hi Volodymyr,
> 
> > 
> > After creating domU, I'm seeing lots of this messages from hypervisor:
> > 
> > (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
> > flags=0x1 par=0x809
> > (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f00
> > flags=0x1 par=0x809
> > (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
> > flags=0x1 par=0x809
> > 
> > Interestingly, I'm getting them from both Dom0 and DomU:
> > 
> > (XEN) p2m.c:1442: d0v0: gvirt_to_maddr failed va=0xffff80003efd7f0f
> > flags=0x1 par=0x809
> > (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
> > flags=0x1 par=0x809
> > 
> > But only after DomU is created.
> > 
> > I attached GDB and found that this is caused by update_runstate_area:
> > 
> > (gdb) bt
> > #0  get_page_from_gva (v=0x80005dbe2000, v@entry=0x22f2c8 <schedule+1236>,
> >      va=va@entry=18446603337277996815, flags=flags@entry=1) at p2m.c:1440
> > #1  0x000000000024e320 in translate_get_page (write=true, linear=true,
> > addr=18446603337277996815,
> >      info=...) at guestcopy.c:37
> > #2  copy_guest (buf=buf@entry=0x80005dbe20d7,
> > addr=addr@entry=18446603337277996815, len=len@entry=1,
> >      info=..., flags=flags@entry=6) at guestcopy.c:69
> > #3  0x000000000024e45c in raw_copy_to_guest (to=to@entry=0xffff80003efd7f0f,
> >      from=from@entry=0x80005dbe20d7, len=len@entry=1) at guestcopy.c:110
> > #4  0x00000000002497b4 in update_runstate_area
> > (v=v@entry=0x80005dbe2000) at domain.c:287
> > #5  0x0000000000249eb8 in context_switch (prev=prev@entry=0x80005dbe2000,
> >      next=next@entry=0x80005bf3c000) at domain.c:344
> > #6  0x000000000022f2c8 in schedule () at schedule.c:1583
> > #7  0x0000000000232c10 in __do_softirq
> > (ignore_mask=ignore_mask@entry=0) at softirq.c:50
> > #8  0x0000000000232ca4 in do_softirq () at softirq.c:64
> > #9  0x0000000000258254 in leave_hypervisor_tail () at traps.c:2302
> > 
> > This issue is encountered on QEMU-ARMv8. Dom0 kernel is Linux 4.19.0
> > My XEN master is at d8ffac1f7 "xen/arm: gic: Remove duplicated comment
> > in do_sgi"
> > 
> > The same setup worked perfectly with Xen 4.10.2
> 
> The message is only printed in debug build. Do you have CONFIG_DEBUG enabled?
> 
> > 
> > Julien, I saw on mailing list, that you paid attention to issues with
> > gvirt_to_maddr,
> > so maybe you can be interested in this.
> 
> Which thread are you speaking about? The problem is not because of
> gvirt_to_maddr but of how update_runstate_area is working at the moment.
> 
> update_runstate_area is using a guest virtual address to update the vCPU
> runstate. It blindly assumes the vCPU runstate will always be mapped in
> stage-1 page-tables. However, if KPTI (Kernel Page Table Isolation) is enabled
> the kernel address space (and therefore the vCPU runstate) will not be mapped
> when running at EL0.
> 
> So if you are restoring a vCPU that was executing code at EL0 then
> update_runstate_area will fail as the address is not mapped. There are a few
> solution suggested on the ML (see [1]). However I haven't had time to look at
> properly how to implement them.
> 
> KPTI is getting used more widely (e.g meltdown and KASLR). So it would be good
> if we try to solve this problem sooner. I would be happy to review patches
> and/or provide advice if you want to tackle the problem.
> 
> Cheers,
> 
> [1] https://lists.xen.org/archives/html/xen-devel/2018-03/msg00223.html

I have seen this issue too, it is quite painful. I haven't had the time
to fix it, but like Julien, I would be happy to help somebody else fix
it.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [ARM] gvirt_to_maddr fails when DomU is created

[Julien Grall]

(+ Stefano)

On 11/27/18 5:12 PM, Volodymyr Babchuk wrote:
> Hello community,

Hi Volodymyr,

> 
> After creating domU, I'm seeing lots of this messages from hypervisor:
> 
> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
> flags=0x1 par=0x809
> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f00
> flags=0x1 par=0x809
> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
> flags=0x1 par=0x809
> 
> Interestingly, I'm getting them from both Dom0 and DomU:
> 
> (XEN) p2m.c:1442: d0v0: gvirt_to_maddr failed va=0xffff80003efd7f0f
> flags=0x1 par=0x809
> (XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
> flags=0x1 par=0x809
> 
> But only after DomU is created.
> 
> I attached GDB and found that this is caused by update_runstate_area:
> 
> (gdb) bt
> #0  get_page_from_gva (v=0x80005dbe2000, v@entry=0x22f2c8 <schedule+1236>,
>      va=va@entry=18446603337277996815, flags=flags@entry=1) at p2m.c:1440
> #1  0x000000000024e320 in translate_get_page (write=true, linear=true,
> addr=18446603337277996815,
>      info=...) at guestcopy.c:37
> #2  copy_guest (buf=buf@entry=0x80005dbe20d7,
> addr=addr@entry=18446603337277996815, len=len@entry=1,
>      info=..., flags=flags@entry=6) at guestcopy.c:69
> #3  0x000000000024e45c in raw_copy_to_guest (to=to@entry=0xffff80003efd7f0f,
>      from=from@entry=0x80005dbe20d7, len=len@entry=1) at guestcopy.c:110
> #4  0x00000000002497b4 in update_runstate_area
> (v=v@entry=0x80005dbe2000) at domain.c:287
> #5  0x0000000000249eb8 in context_switch (prev=prev@entry=0x80005dbe2000,
>      next=next@entry=0x80005bf3c000) at domain.c:344
> #6  0x000000000022f2c8 in schedule () at schedule.c:1583
> #7  0x0000000000232c10 in __do_softirq
> (ignore_mask=ignore_mask@entry=0) at softirq.c:50
> #8  0x0000000000232ca4 in do_softirq () at softirq.c:64
> #9  0x0000000000258254 in leave_hypervisor_tail () at traps.c:2302
> 
> This issue is encountered on QEMU-ARMv8. Dom0 kernel is Linux 4.19.0
> My XEN master is at d8ffac1f7 "xen/arm: gic: Remove duplicated comment
> in do_sgi"
> 
> The same setup worked perfectly with Xen 4.10.2

The message is only printed in debug build. Do you have CONFIG_DEBUG 
enabled?

> 
> Julien, I saw on mailing list, that you paid attention to issues with
> gvirt_to_maddr,
> so maybe you can be interested in this.

Which thread are you speaking about? The problem is not because of 
gvirt_to_maddr but of how update_runstate_area is working at the moment.

update_runstate_area is using a guest virtual address to update the vCPU 
runstate. It blindly assumes the vCPU runstate will always be mapped in 
stage-1 page-tables. However, if KPTI (Kernel Page Table Isolation) is 
enabled the kernel address space (and therefore the vCPU runstate) will 
not be mapped when running at EL0.

So if you are restoring a vCPU that was executing code at EL0 then 
update_runstate_area will fail as the address is not mapped. There are a 
few solution suggested on the ML (see [1]). However I haven't had time 
to look at properly how to implement them.

KPTI is getting used more widely (e.g meltdown and KASLR). So it would 
be good if we try to solve this problem sooner. I would be happy to 
review patches and/or provide advice if you want to tackle the problem.

Cheers,

[1] https://lists.xen.org/archives/html/xen-devel/2018-03/msg00223.html

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[ARM] gvirt_to_maddr fails when DomU is created

[Volodymyr Babchuk]

Hello community,

After creating domU, I'm seeing lots of this messages from hypervisor:

(XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
flags=0x1 par=0x809
(XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f00
flags=0x1 par=0x809
(XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
flags=0x1 par=0x809

Interestingly, I'm getting them from both Dom0 and DomU:

(XEN) p2m.c:1442: d0v0: gvirt_to_maddr failed va=0xffff80003efd7f0f
flags=0x1 par=0x809
(XEN) p2m.c:1442: d1v0: gvirt_to_maddr failed va=0xffff80000efc7f0f
flags=0x1 par=0x809

But only after DomU is created.

I attached GDB and found that this is caused by update_runstate_area:

(gdb) bt
#0  get_page_from_gva (v=0x80005dbe2000, v@entry=0x22f2c8 <schedule+1236>,
    va=va@entry=18446603337277996815, flags=flags@entry=1) at p2m.c:1440
#1  0x000000000024e320 in translate_get_page (write=true, linear=true,
addr=18446603337277996815,
    info=...) at guestcopy.c:37
#2  copy_guest (buf=buf@entry=0x80005dbe20d7,
addr=addr@entry=18446603337277996815, len=len@entry=1,
    info=..., flags=flags@entry=6) at guestcopy.c:69
#3  0x000000000024e45c in raw_copy_to_guest (to=to@entry=0xffff80003efd7f0f,
    from=from@entry=0x80005dbe20d7, len=len@entry=1) at guestcopy.c:110
#4  0x00000000002497b4 in update_runstate_area
(v=v@entry=0x80005dbe2000) at domain.c:287
#5  0x0000000000249eb8 in context_switch (prev=prev@entry=0x80005dbe2000,
    next=next@entry=0x80005bf3c000) at domain.c:344
#6  0x000000000022f2c8 in schedule () at schedule.c:1583
#7  0x0000000000232c10 in __do_softirq
(ignore_mask=ignore_mask@entry=0) at softirq.c:50
#8  0x0000000000232ca4 in do_softirq () at softirq.c:64
#9  0x0000000000258254 in leave_hypervisor_tail () at traps.c:2302

This issue is encountered on QEMU-ARMv8. Dom0 kernel is Linux 4.19.0
My XEN master is at d8ffac1f7 "xen/arm: gic: Remove duplicated comment
in do_sgi"

The same setup worked perfectly with Xen 4.10.2

Julien, I saw on mailing list, that you paid attention to issues with
gvirt_to_maddr,
so maybe you can be interested in this.

-- 
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@gmail.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel