Xen on Arm guest memory mapping

Xen on Arm guest memory mapping

[Sander Bogaert]

Hi,

I went over xen/arch/arm/domain_build.c (especially set_memory_reg(..)
) trying to figure out how Xen maps memory to guest and assigns the
virtual addresses. I couldn't find documentation on this, might have
been looking in the wrong place?

I understand the x86 start_page isn't used and everything is passed
using a device tree. This tree is in the linux kernel repo and
attached to the zImage. Xen reads the dtb and adapts it. The guest
then parses the dtb to get info about the assigned memory.

* What if there is no dtb attached? My guest ( ELF, not zImage ) seems
to boot anyway but the domain_build code seems to always use a
devicetree. where does it get one?

* From what I understand of the set_memory_reg() function mentioned
before it does the following ( as long as there is unassigned memory &
banks left ):
    1. read the value from the dt
    2. if there is more room then unassigned memory only use the
amount unassigned
    3. write out to the dt again
    4. actually allocate it using p2m_populate_ram

 => This seems to indicate the xen doesn't adapt the dt if it fits and
just allocates those addresses. This doesn't seem to be true since
they are set at 0x8000000 (xenvm-4.2.dts) and that's a machine
address. The guest needs to receive/work with virtual ones does it
not? Is the dt adapted before this call, I couldn't find where?

* How does Xen pick the virtual addresses?

Thanks,
Sander

Re: Xen on Arm guest memory mapping

[Ian Campbell]

On Tue, 2013-05-07 at 15:25 +0100, Sander Bogaert wrote:
> Hi,
> 
> I went over xen/arch/arm/domain_build.c (especially set_memory_reg(..)
> ) trying to figure out how Xen maps memory to guest and assigns the
> virtual addresses. I couldn't find documentation on this, might have
> been looking in the wrong place?
> 
> I understand the x86 start_page isn't used and everything is passed
> using a device tree. This tree is in the linux kernel repo and
> attached to the zImage. Xen reads the dtb and adapts it. The guest
> then parses the dtb to get info about the assigned memory.

Xen currently only does this read+adapt for dom0.

For domU the appended dtb is currently just passed on. Eventually the
Xen tools will generate the dtb themselves from first principals to
reflect the configuration of the guest. This is (very briefly) mentioned
at
http://wiki.xen.org/wiki/Xen_ARM_TODO#Autogenerate_DomU.27s_device_trees

> * What if there is no dtb attached? My guest ( ELF, not zImage ) seems
> to boot anyway but the domain_build code seems to always use a
> devicetree. where does it get one?

Device Tree is just a way to describe a hardware platform to a kernel.
You could equally well (FSV of well) hardcode all the relevant details
into the kernel, since your ELF guest likely doesn't know anything about
DT you must be (effectively) hardcoding things.

If you are hardcoding instead of parsing DT then your guest cfg file and
your guest kernel need to agree some of the dynamic stuff, such as the
amount of RAM the guest is going to have and its location in the
physical address map.

In the absence of the dtb generation code mentioned above the Xen guest
domain builder places the RAM at 0x80000000 up to whatever size you give
in your guest configuration.

> * From what I understand of the set_memory_reg() function mentioned

Note that this function is only called when building domain 0.
xen/arch/arm/domain_build.c is the initial domain builder only,
subsequent guests are built by the userspace stuff in tools/libxc.

> before it does the following ( as long as there is unassigned memory &
> banks left ):
>     1. read the value from the dt
>     2. if there is more room then unassigned memory only use the
> amount unassigned
>     3. write out to the dt again
>     4. actually allocate it using p2m_populate_ram
> 
>  => This seems to indicate the xen doesn't adapt the dt if it fits and
> just allocates those addresses.

For dom0 Xen effectively trucates the underlying platform's memory to
only be as large as the specified dom0_mem.

>  This doesn't seem to be true since
> they are set at 0x8000000 (xenvm-4.2.dts) and that's a machine

NB xenvm-4.2.dts is a domU dts not a dom0 one. For dom0 you would use
the DTS for your actual platform.

> address. The guest needs to receive/work with virtual ones does it
> not?

No, the guest gets physical addresses here.

>  Is the dt adapted before this call, I couldn't find where?
> 
> * How does Xen pick the virtual addresses?

It doesn't. On ARM the guest is in complete control of its own virtual
address space. Guests are started with paging disabled. For the most
part Xen doesn't care about the guest's virtual address mappings.

Ian.

Re: Xen on Arm guest memory mapping

[Sander Bogaert]

On 7 May 2013 17:39, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Tue, 2013-05-07 at 15:25 +0100, Sander Bogaert wrote:
>> Hi,
>>
>> I went over xen/arch/arm/domain_build.c (especially set_memory_reg(..)
>> ) trying to figure out how Xen maps memory to guest and assigns the
>> virtual addresses. I couldn't find documentation on this, might have
>> been looking in the wrong place?
>>
>> I understand the x86 start_page isn't used and everything is passed
>> using a device tree. This tree is in the linux kernel repo and
>> attached to the zImage. Xen reads the dtb and adapts it. The guest
>> then parses the dtb to get info about the assigned memory.
>
> Xen currently only does this read+adapt for dom0.
>
> For domU the appended dtb is currently just passed on. Eventually the
> Xen tools will generate the dtb themselves from first principals to
> reflect the configuration of the guest. This is (very briefly) mentioned
> at
> http://wiki.xen.org/wiki/Xen_ARM_TODO#Autogenerate_DomU.27s_device_trees
>
>> * What if there is no dtb attached? My guest ( ELF, not zImage ) seems
>> to boot anyway but the domain_build code seems to always use a
>> devicetree. where does it get one?
>
> Device Tree is just a way to describe a hardware platform to a kernel.
> You could equally well (FSV of well) hardcode all the relevant details
> into the kernel, since your ELF guest likely doesn't know anything about
> DT you must be (effectively) hardcoding things.
>
> If you are hardcoding instead of parsing DT then your guest cfg file and
> your guest kernel need to agree some of the dynamic stuff, such as the
> amount of RAM the guest is going to have and its location in the
> physical address map.
>
> In the absence of the dtb generation code mentioned above the Xen guest
> domain builder places the RAM at 0x80000000 up to whatever size you give
> in your guest configuration.
>
>> * From what I understand of the set_memory_reg() function mentioned
>
> Note that this function is only called when building domain 0.
> xen/arch/arm/domain_build.c is the initial domain builder only,
> subsequent guests are built by the userspace stuff in tools/libxc.
>
>> before it does the following ( as long as there is unassigned memory &
>> banks left ):
>>     1. read the value from the dt
>>     2. if there is more room then unassigned memory only use the
>> amount unassigned
>>     3. write out to the dt again
>>     4. actually allocate it using p2m_populate_ram
>>
>>  => This seems to indicate the xen doesn't adapt the dt if it fits and
>> just allocates those addresses.
>
> For dom0 Xen effectively trucates the underlying platform's memory to
> only be as large as the specified dom0_mem.
>
>>  This doesn't seem to be true since
>> they are set at 0x8000000 (xenvm-4.2.dts) and that's a machine
>
> NB xenvm-4.2.dts is a domU dts not a dom0 one. For dom0 you would use
> the DTS for your actual platform.
>
>> address. The guest needs to receive/work with virtual ones does it
>> not?
>
> No, the guest gets physical addresses here.
>
>>  Is the dt adapted before this call, I couldn't find where?
>>
>> * How does Xen pick the virtual addresses?
>
> It doesn't. On ARM the guest is in complete control of its own virtual
> address space. Guests are started with paging disabled. For the most
> part Xen doesn't care about the guest's virtual address mappings.
>
> Ian.
>
>

Thanks a lot for these answers, they are very useful! So if I
understand this correctly in order to have multiple guests in Xen on
Arm one has to adapt the dts files so they all have a different part
of the memory.

Sander

Re: Xen on Arm guest memory mapping

[Ian Campbell]

On Wed, 2013-05-08 at 11:18 +0100, Sander Bogaert wrote:
> On 7 May 2013 17:39, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Tue, 2013-05-07 at 15:25 +0100, Sander Bogaert wrote:
> >> Hi,
> >>
> >> I went over xen/arch/arm/domain_build.c (especially set_memory_reg(..)
> >> ) trying to figure out how Xen maps memory to guest and assigns the
> >> virtual addresses. I couldn't find documentation on this, might have
> >> been looking in the wrong place?
> >>
> >> I understand the x86 start_page isn't used and everything is passed
> >> using a device tree. This tree is in the linux kernel repo and
> >> attached to the zImage. Xen reads the dtb and adapts it. The guest
> >> then parses the dtb to get info about the assigned memory.
> >
> > Xen currently only does this read+adapt for dom0.
> >
> > For domU the appended dtb is currently just passed on. Eventually the
> > Xen tools will generate the dtb themselves from first principals to
> > reflect the configuration of the guest. This is (very briefly) mentioned
> > at
> > http://wiki.xen.org/wiki/Xen_ARM_TODO#Autogenerate_DomU.27s_device_trees
> >
> >> * What if there is no dtb attached? My guest ( ELF, not zImage ) seems
> >> to boot anyway but the domain_build code seems to always use a
> >> devicetree. where does it get one?
> >
> > Device Tree is just a way to describe a hardware platform to a kernel.
> > You could equally well (FSV of well) hardcode all the relevant details
> > into the kernel, since your ELF guest likely doesn't know anything about
> > DT you must be (effectively) hardcoding things.
> >
> > If you are hardcoding instead of parsing DT then your guest cfg file and
> > your guest kernel need to agree some of the dynamic stuff, such as the
> > amount of RAM the guest is going to have and its location in the
> > physical address map.
> >
> > In the absence of the dtb generation code mentioned above the Xen guest
> > domain builder places the RAM at 0x80000000 up to whatever size you give
> > in your guest configuration.
> >
> >> * From what I understand of the set_memory_reg() function mentioned
> >
> > Note that this function is only called when building domain 0.
> > xen/arch/arm/domain_build.c is the initial domain builder only,
> > subsequent guests are built by the userspace stuff in tools/libxc.
> >
> >> before it does the following ( as long as there is unassigned memory &
> >> banks left ):
> >>     1. read the value from the dt
> >>     2. if there is more room then unassigned memory only use the
> >> amount unassigned
> >>     3. write out to the dt again
> >>     4. actually allocate it using p2m_populate_ram
> >>
> >>  => This seems to indicate the xen doesn't adapt the dt if it fits and
> >> just allocates those addresses.
> >
> > For dom0 Xen effectively trucates the underlying platform's memory to
> > only be as large as the specified dom0_mem.
> >
> >>  This doesn't seem to be true since
> >> they are set at 0x8000000 (xenvm-4.2.dts) and that's a machine
> >
> > NB xenvm-4.2.dts is a domU dts not a dom0 one. For dom0 you would use
> > the DTS for your actual platform.
> >
> >> address. The guest needs to receive/work with virtual ones does it
> >> not?
> >
> > No, the guest gets physical addresses here.
> >
> >>  Is the dt adapted before this call, I couldn't find where?
> >>
> >> * How does Xen pick the virtual addresses?
> >
> > It doesn't. On ARM the guest is in complete control of its own virtual
> > address space. Guests are started with paging disabled. For the most
> > part Xen doesn't care about the guest's virtual address mappings.
> >
> > Ian.
> >
> >
> 
> Thanks a lot for these answers, they are very useful! So if I
> understand this correctly in order to have multiple guests in Xen on
> Arm one has to adapt the dts files so they all have a different part
> of the memory.

No, absolutely not.

The physical addresses in the DTS (and elsewhere) are virtualised via
the hardware second stage paging.

We tend to refer to the actual underlying host RAM as "machine memory"
to distinguish this from the physical memory abstraction as seen by
guest domains. Sorry if this made my original reply confusing!

Ian.