Hand over of the Xen shared info page

Hand over of the Xen shared info page

[Anastasiia Lukianenko]

Hi all,

The problem described below concerns cases when a shared info page
needs to be handed over from one entity in the system to another, for
example, there is a bootloader or any other code that may run before
the guest OS' kernel.
Normally, to map the shared info page guests allocate a memory page
from their RAM and map the shared info on top of it. Specifically we
use XENMAPSPACE_shared_info memory space in  XENMEM_add_to_physmap
hypercall.  As the info page exists throughout the guest existence this
doesn't hurt the guest, but when the page gets out of accounting, e.g.
after bootloader jumps to Linux and the page is not handed over to it,
the mapped page becomes a problem.
Consider the case with U-boot bootloader which already has Xen support.
The U-boot’s Xen guest implementation allocates a shared info page
between Xen and the guest domain and U-boot uses domain's RAM address
space to create and map the shared info page by using
XENMEM_add_to_physmap hypercall [1].

After U-boot transfers control to the operating system (Linux, Android,
etc), the shared info page is still mapped in domain’s address space,
e.g. its RAM. So, after we leave U-boot, this page becomes just an
ordinary memory page from Linux POV while it is still a shared info
page from Xen POV. This can lead to undefined behavior, errors etc as
Xen can write something to the shared info page and when Linux tries to
use it - data corruption may happen.
This happens because there is no unmap function in Xen API to remove an
existing shared info page mapping. We could use only hypercall
XENMEM_remove_from_physmap which eventually will create a hole in the
domain's RAM address space which may also lead to guest’s crash while
accessing that memory.

We noticed this problem and the workaround was implemented using the
special GUEST_MAGIC memory region [2].

Now we want to make a proper solution based on GUEST_MAGIC_BASE, which
does not belong to the guest’s RAM address space [3]. Using the example
of how offsets for the console and xenstore are implemented, we can add
a new shared_info offset and increase the number of magic pages [4] and
implement related functionality, so there is a similar API to query
that magic page location as it is done for console PFN and others.
This approach would allow the use of the XENMEM_remove_from_physmap
hypercall without creating gaps in the RAM address space for Xen guest
OS [5].

[1] - 
https://github.com/u-boot/u-boot/blob/master/drivers/xen/hypervisor.c#L141
[2] - 
https://github.com/xen-troops/u-boot/commit/f759b151116af204a5ab02ace82c09300cd6233a
[3] - 
https://github.com/xen-project/xen/blob/master/xen/include/public/arch-arm.h#L432
[4] - 
https://github.com/xen-project/xen/blob/25849c8b16f2a5b7fcd0a823e80a5f1b590291f9/tools/libs/guest/xg_dom_arm.c#L29
[5] - 
https://github.com/xen-troops/u-boot/blob/android-master/drivers/xen/hypervisor.c#L162

Regards,
Anastasiia Lukianenko

Re: Hand over of the Xen shared info page

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

Am Thu, 13 May 2021 08:03:48 +0000
schrieb Anastasiia Lukianenko <anastasiia_lukianenko@epam.com>:

> shared info page needs to be handed over from one entity in the system to another

The same issue exists with kexec. Not sure why it was not addressed as you proposed, "soft_reset" was implemented instead.

Good luck.


Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Hand over of the Xen shared info page

[Julien Grall]

On 13/05/2021 09:03, Anastasiia Lukianenko wrote:
> Hi all,

Hi,

> The problem described below concerns cases when a shared info page
> needs to be handed over from one entity in the system to another, for
> example, there is a bootloader or any other code that may run before
> the guest OS' kernel.
> Normally, to map the shared info page guests allocate a memory page
> from their RAM and map the shared info on top of it. Specifically we
> use XENMAPSPACE_shared_info memory space in  XENMEM_add_to_physmap
> hypercall.  As the info page exists throughout the guest existence this
> doesn't hurt the guest, but when the page gets out of accounting, e.g.
> after bootloader jumps to Linux and the page is not handed over to it,
> the mapped page becomes a problem.
> Consider the case with U-boot bootloader which already has Xen support.
> The U-boot’s Xen guest implementation allocates a shared info page
> between Xen and the guest domain and U-boot uses domain's RAM address
> space to create and map the shared info page by using
> XENMEM_add_to_physmap hypercall [1].
> 
> After U-boot transfers control to the operating system (Linux, Android,
> etc), the shared info page is still mapped in domain’s address space,
> e.g. its RAM. So, after we leave U-boot, this page becomes just an
> ordinary memory page from Linux POV while it is still a shared info
> page from Xen POV. This can lead to undefined behavior, errors etc as
> Xen can write something to the shared info page and when Linux tries to
> use it - data corruption may happen.
> This happens because there is no unmap function in Xen API to remove an
> existing shared info page mapping. We could use only hypercall
> XENMEM_remove_from_physmap which eventually will create a hole in the
> domain's RAM address space which may also lead to guest’s crash while
> accessing that memory.

The hypercall XENMEM_remove_from_physmap is the correct hypercall here 
and work as intented. It is not Xen business to keep track what was the 
original page (it may have been RAM, device...).

The problem here is the hypercall XENMEM_add_to_physmap is misused in 
U-boot. When you give an address for the mapping you are telling Xen 
"Here a free region to map the share paged". IOW, Xen will throw away 
whatever was before because that was you asked.

If you want to map in place of the RAM page, then the correct approach 
is to:
   1) Request Xen to remove the RAM page from the P2M
   2) Map the shared page
   /* Use it */
   3) Unmap the shared page
   4) Allocate the memory

You can avoid 1) and 4) by finding free region in the address space.

> 
> We noticed this problem and the workaround was implemented using the
> special GUEST_MAGIC memory region [2].
> 
> Now we want to make a proper solution based on GUEST_MAGIC_BASE, which
> does not belong to the guest’s RAM address space [3]. Using the example
> of how offsets for the console and xenstore are implemented, we can add
> a new shared_info offset and increase the number of magic pages [4] and
> implement related functionality, so there is a similar API to query
> that magic page location as it is done for console PFN and others.

They are not the same type. The console PFN points memory already 
populated in the guest address space.

For the domain shared page, this is memory belonging to Xen that you 
will map in your address space. A domain can map it anywhere it wants.

> This approach would allow the use of the XENMEM_remove_from_physmap
> hypercall without creating gaps in the RAM address space for Xen guest
> OS [5].

See above to prevent the gap. I appreciate this means a superpage may 
get shattered.

The alternative is for U-boot to go through the DT and infer which 
regions are free (IOW any region not described).

Cheers,

-- 
Julien Grall

Re: Hand over of the Xen shared info page

[Anastasiia Lukianenko]

Hi Julien!

On Thu, 2021-05-13 at 09:37 +0100, Julien Grall wrote:
> 
> On 13/05/2021 09:03, Anastasiia Lukianenko wrote:
> > Hi all,
> 
> Hi,
> 
> > The problem described below concerns cases when a shared info page
> > needs to be handed over from one entity in the system to another,
> > for
> > example, there is a bootloader or any other code that may run
> > before
> > the guest OS' kernel.
> > Normally, to map the shared info page guests allocate a memory page
> > from their RAM and map the shared info on top of it. Specifically
> > we
> > use XENMAPSPACE_shared_info memory space in  XENMEM_add_to_physmap
> > hypercall.  As the info page exists throughout the guest existence
> > this
> > doesn't hurt the guest, but when the page gets out of accounting,
> > e.g.
> > after bootloader jumps to Linux and the page is not handed over to
> > it,
> > the mapped page becomes a problem.
> > Consider the case with U-boot bootloader which already has Xen
> > support.
> > The U-boot’s Xen guest implementation allocates a shared info page
> > between Xen and the guest domain and U-boot uses domain's RAM
> > address
> > space to create and map the shared info page by using
> > XENMEM_add_to_physmap hypercall [1].
> > 
> > After U-boot transfers control to the operating system (Linux,
> > Android,
> > etc), the shared info page is still mapped in domain’s address
> > space,
> > e.g. its RAM. So, after we leave U-boot, this page becomes just an
> > ordinary memory page from Linux POV while it is still a shared info
> > page from Xen POV. This can lead to undefined behavior, errors etc
> > as
> > Xen can write something to the shared info page and when Linux
> > tries to
> > use it - data corruption may happen.
> > This happens because there is no unmap function in Xen API to
> > remove an
> > existing shared info page mapping. We could use only hypercall
> > XENMEM_remove_from_physmap which eventually will create a hole in
> > the
> > domain's RAM address space which may also lead to guest’s crash
> > while
> > accessing that memory.
> 
> The hypercall XENMEM_remove_from_physmap is the correct hypercall
> here 
> and work as intented. It is not Xen business to keep track what was
> the 
> original page (it may have been RAM, device...).
> 
> The problem here is the hypercall XENMEM_add_to_physmap is misused
> in 
> U-boot. When you give an address for the mapping you are telling Xen 
> "Here a free region to map the share paged". IOW, Xen will throw
> away 
> whatever was before because that was you asked.
> 
> If you want to map in place of the RAM page, then the correct
> approach 
> is to:
>    1) Request Xen to remove the RAM page from the P2M
>    2) Map the shared page
>    /* Use it */
>    3) Unmap the shared page
>    4) Allocate the memory
> 
> You can avoid 1) and 4) by finding free region in the address space.
> 
> > 
> > We noticed this problem and the workaround was implemented using
> > the
> > special GUEST_MAGIC memory region [2].
> > 
> > Now we want to make a proper solution based on GUEST_MAGIC_BASE,
> > which
> > does not belong to the guest’s RAM address space [3]. Using the
> > example
> > of how offsets for the console and xenstore are implemented, we can
> > add
> > a new shared_info offset and increase the number of magic pages [4]
> > and
> > implement related functionality, so there is a similar API to query
> > that magic page location as it is done for console PFN and others.
> 
> They are not the same type. The console PFN points memory already 
> populated in the guest address space.
> 
> For the domain shared page, this is memory belonging to Xen that you 
> will map in your address space. A domain can map it anywhere it
> wants.
> 
> > This approach would allow the use of the XENMEM_remove_from_physmap
> > hypercall without creating gaps in the RAM address space for Xen
> > guest
> > OS [5].
> 
> See above to prevent the gap. I appreciate this means a superpage
> may 
> get shattered.
> 
> The alternative is for U-boot to go through the DT and infer which 
> regions are free (IOW any region not described).

Thank you for interest in the problem and advice on how to solve it.
Could you please clarify how we could find free regions using DT in U-
boot?

Regards,
Anastasiia
> 
> Cheers,
> 

Re: Hand over of the Xen shared info page

[Julien Grall]

On 14/05/2021 10:50, Anastasiia Lukianenko wrote:
> Hi Julien!

Hello,

> On Thu, 2021-05-13 at 09:37 +0100, Julien Grall wrote:
>>
>> On 13/05/2021 09:03, Anastasiia Lukianenko wrote:
>> The alternative is for U-boot to go through the DT and infer which
>> regions are free (IOW any region not described).
> 
> Thank you for interest in the problem and advice on how to solve it.
> Could you please clarify how we could find free regions using DT in U-
> boot?

I don't know U-boot code, so I can't tell whether what I suggest would work.

In theory, the device-tree should described every region allocated in 
address space. So if you parse the device-tree and create a list (or any 
datastructure) with the regions, then any range not present in the list 
would be free region you could use.

However, I realized a few days ago that the magic pages are not 
described in the DT. We probably want to fix it by marking the page as 
"reserved" or create a specific bindings.

So you will need a specific quirk for them.

I have posted some more idea a separate thread [1] related to FreeBSD 
support for Arm.

Cheers,

[1] 
https://lore.kernel.org/xen-devel/f7360dac-5d83-733b-7ec5-c73d4dc0350d@xen.org/

-- 
Julien Grall

Re: Hand over of the Xen shared info page

[Stefano Stabellini]

On Wed, 19 May 2021, Julien Grall wrote:
> On 14/05/2021 10:50, Anastasiia Lukianenko wrote:
> > Hi Julien!
> 
> Hello,
> 
> > On Thu, 2021-05-13 at 09:37 +0100, Julien Grall wrote:
> > > 
> > > On 13/05/2021 09:03, Anastasiia Lukianenko wrote:
> > > The alternative is for U-boot to go through the DT and infer which
> > > regions are free (IOW any region not described).
> > 
> > Thank you for interest in the problem and advice on how to solve it.
> > Could you please clarify how we could find free regions using DT in U-
> > boot?
> 
> I don't know U-boot code, so I can't tell whether what I suggest would work.
> 
> In theory, the device-tree should described every region allocated in address
> space. So if you parse the device-tree and create a list (or any
> datastructure) with the regions, then any range not present in the list would
> be free region you could use.

Yes, any "empty" memory region which is neither memory nor MMIO should
work.


> However, I realized a few days ago that the magic pages are not described in
> the DT. We probably want to fix it by marking the page as "reserved" or create
> a specific bindings.
> 
> So you will need a specific quirk for them.

It should also be possible to keep the shared info page allocated and
simply pass the address to the kernel by adding the right node to device
tree. To do that, we would have to add a description of the magic pages
to device tree, which I think would be good to have in any case. In that
case it would be best to add a proper binding for it under the "xen,xen"
node.

Re: Hand over of the Xen shared info page

[Oleksandr Andrushchenko]

Hi, all!

On 5/20/21 2:11 AM, Stefano Stabellini wrote:
> On Wed, 19 May 2021, Julien Grall wrote:
>> On 14/05/2021 10:50, Anastasiia Lukianenko wrote:
>>> Hi Julien!
>> Hello,
>>
>>> On Thu, 2021-05-13 at 09:37 +0100, Julien Grall wrote:
>>>> On 13/05/2021 09:03, Anastasiia Lukianenko wrote:
>>>> The alternative is for U-boot to go through the DT and infer which
>>>> regions are free (IOW any region not described).
>>> Thank you for interest in the problem and advice on how to solve it.
>>> Could you please clarify how we could find free regions using DT in U-
>>> boot?
>> I don't know U-boot code, so I can't tell whether what I suggest would work.
>>
>> In theory, the device-tree should described every region allocated in address
>> space. So if you parse the device-tree and create a list (or any
>> datastructure) with the regions, then any range not present in the list would
>> be free region you could use.
> Yes, any "empty" memory region which is neither memory nor MMIO should
> work.

You need to account on many things while creating the list of regions:

device register mappings, reserved nodes, memory nodes, device tree

overlays parsing etc. It all seem to be a not-that-trivial task and after

all if implemented it only lives in U-boot and you have to maintain that

code. So, if some other entity needs the same you need to implement

that as well. And also you can imagine a system without device tree at all...

So, I am not saying it is not possible to implement, but I just question if

such an implementation is really a good way forward.

>
>
>> However, I realized a few days ago that the magic pages are not described in
>> the DT. We probably want to fix it by marking the page as "reserved" or create
>> a specific bindings.
>>
>> So you will need a specific quirk for them.
> It should also be possible to keep the shared info page allocated and
> simply pass the address to the kernel by adding the right node to device
> tree.
And then you need to modify your OS and this is not only Linux...
> To do that, we would have to add a description of the magic pages
> to device tree, which I think would be good to have in any case. In that
> case it would be best to add a proper binding for it under the "xen,xen"
> node.

I would say that querying Xen for such a memory page seems much

more cleaner and allows the guest OS either to continue using the existing

method with memory allocation or using the page provided by Xen.

Re: Hand over of the Xen shared info page

[Julien Grall]

On 20/05/2021 06:21, Oleksandr Andrushchenko wrote:
> Hi, all!

Hi,


> On 5/20/21 2:11 AM, Stefano Stabellini wrote:
>> On Wed, 19 May 2021, Julien Grall wrote:
>>> On 14/05/2021 10:50, Anastasiia Lukianenko wrote:
>>>> Hi Julien!
>>> Hello,
>>>
>>>> On Thu, 2021-05-13 at 09:37 +0100, Julien Grall wrote:
>>>>> On 13/05/2021 09:03, Anastasiia Lukianenko wrote:
>>>>> The alternative is for U-boot to go through the DT and infer which
>>>>> regions are free (IOW any region not described).
>>>> Thank you for interest in the problem and advice on how to solve it.
>>>> Could you please clarify how we could find free regions using DT in U-
>>>> boot?
>>> I don't know U-boot code, so I can't tell whether what I suggest would work.
>>>
>>> In theory, the device-tree should described every region allocated in address
>>> space. So if you parse the device-tree and create a list (or any
>>> datastructure) with the regions, then any range not present in the list would
>>> be free region you could use.
>> Yes, any "empty" memory region which is neither memory nor MMIO should
>> work.
> 
> You need to account on many things while creating the list of regions:
> 
> device register mappings, reserved nodes, memory nodes, device tree
> 
> overlays parsing etc. It all seem to be a not-that-trivial task and after
> 
> all if implemented it only lives in U-boot and you have to maintain that
> 
> code. So, if some other entity needs the same you need to implement
> 
> that as well.

Yes, there are some complexity. I have suggested other approach in a 
separate thread. Did you look at them?

> And also you can imagine a system without device tree at all...
Xen doesn't provide a stable guest layout. Such system would have to be 
tailored to a given guest configuration, Xen version (can be custom)...

Therefore, hardcoding the value in the system (not in Xen headers!) is 
not going to make it much worse.

> So, I am not saying it is not possible to implement, but I just question if
> 
> such an implementation is really a good way forward.
> 
>>
>>
>>> However, I realized a few days ago that the magic pages are not described in
>>> the DT. We probably want to fix it by marking the page as "reserved" or create
>>> a specific bindings.
>>>
>>> So you will need a specific quirk for them.
>> It should also be possible to keep the shared info page allocated and
>> simply pass the address to the kernel by adding the right node to device
>> tree.
> And then you need to modify your OS and this is not only Linux...
>> To do that, we would have to add a description of the magic pages
>> to device tree, which I think would be good to have in any case. In that
>> case it would be best to add a proper binding for it under the "xen,xen"
>> node.
> 
> I would say that querying Xen for such a memory page seems much
> 
> more cleaner and allows the guest OS either to continue using the existing
> 
> method with memory allocation or using the page provided by Xen.

IIUC your proposal, you are asking to add an hypercall to query which 
guest physical region can be used for the shared info page.

This may solve the problem you have at hand, but this is not scalable. 
There are a few other regions which regions unallocated memory (e.g. 
grant table, grant mapping, foreign memory map,....).

So if we were going to involve Xen, then I think it is better to have a 
generic way to ask Xen for unallocated space.

Cheers,

-- 
Julien Grall

Re: Hand over of the Xen shared info page

[Oleksandr Andrushchenko]

Hi,

On 5/20/21 12:46 PM, Julien Grall wrote:
>
>
> On 20/05/2021 06:21, Oleksandr Andrushchenko wrote:
>> Hi, all!
>
> Hi,
>
>
>> On 5/20/21 2:11 AM, Stefano Stabellini wrote:
>>> On Wed, 19 May 2021, Julien Grall wrote:
>>>> On 14/05/2021 10:50, Anastasiia Lukianenko wrote:
>>>>> Hi Julien!
>>>> Hello,
>>>>
>>>>> On Thu, 2021-05-13 at 09:37 +0100, Julien Grall wrote:
>>>>>> On 13/05/2021 09:03, Anastasiia Lukianenko wrote:
>>>>>> The alternative is for U-boot to go through the DT and infer which
>>>>>> regions are free (IOW any region not described).
>>>>> Thank you for interest in the problem and advice on how to solve it.
>>>>> Could you please clarify how we could find free regions using DT in U-
>>>>> boot?
>>>> I don't know U-boot code, so I can't tell whether what I suggest would work.
>>>>
>>>> In theory, the device-tree should described every region allocated in address
>>>> space. So if you parse the device-tree and create a list (or any
>>>> datastructure) with the regions, then any range not present in the list would
>>>> be free region you could use.
>>> Yes, any "empty" memory region which is neither memory nor MMIO should
>>> work.
>>
>> You need to account on many things while creating the list of regions:
>>
>> device register mappings, reserved nodes, memory nodes, device tree
>>
>> overlays parsing etc. It all seem to be a not-that-trivial task and after
>>
>> all if implemented it only lives in U-boot and you have to maintain that
>>
>> code. So, if some other entity needs the same you need to implement
>>
>> that as well.
>
> Yes, there are some complexity. I have suggested other approach in a separate thread. Did you look at them?

Yes, so probably we can re-use the solution from that thread when it comes to some conclusion

and implementation.

>
>> And also you can imagine a system without device tree at all...
> Xen doesn't provide a stable guest layout. Such system would have to be tailored to a given guest configuration, Xen version (can be custom)...
>
> Therefore, hardcoding the value in the system (not in Xen headers!) is not going to make it much worse.
Agree to some extent
>
>> So, I am not saying it is not possible to implement, but I just question if
>>
>> such an implementation is really a good way forward.
>>
>>>
>>>
>>>> However, I realized a few days ago that the magic pages are not described in
>>>> the DT. We probably want to fix it by marking the page as "reserved" or create
>>>> a specific bindings.
>>>>
>>>> So you will need a specific quirk for them.
>>> It should also be possible to keep the shared info page allocated and
>>> simply pass the address to the kernel by adding the right node to device
>>> tree.
>> And then you need to modify your OS and this is not only Linux...
>>> To do that, we would have to add a description of the magic pages
>>> to device tree, which I think would be good to have in any case. In that
>>> case it would be best to add a proper binding for it under the "xen,xen"
>>> node.
>>
>> I would say that querying Xen for such a memory page seems much
>>
>> more cleaner and allows the guest OS either to continue using the existing
>>
>> method with memory allocation or using the page provided by Xen.
>
> IIUC your proposal, you are asking to add an hypercall to query which guest physical region can be used for the shared info page.
>
> This may solve the problem you have at hand, but this is not scalable. There are a few other regions which regions unallocated memory (e.g. grant table, grant mapping, foreign memory map,....).
>
> So if we were going to involve Xen, then I think it is better to have a generic way to ask Xen for unallocated space.
Agree
>
> Cheers,
>
Thank you,

Oleksandr