Re: [RFC] Add support for Xen ARM guest on FreeBSD

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Warner Losh]

On Jan 14, 2014, at 2:01 PM, Julien Grall wrote:
> This new support brings 2 open questions (for both Xen and FreeBSD community).
> When a new guest is created, the toolstack will generated a device tree which
> will contains:
> 	- The amount of memory
> 	- The description of the platform (gic, timer, hypervisor node)
> 	- PSCI node for SMP bringup
> 
> Until now, Xen on ARM supported only Linux-based OS. When the support of
> device tree was added in Xen for guest, we chose to use Linux device tree
> bindings (gic, timer,...). It seems that FreeBSD choose a different way to
> implement device tree:
> 	- strictly respect ePAR (for interrupt-parent property)
> 	- gic bindings different (only 1 interrupt cell)
> 
> I would like to come with a common device tree specification (bindings, ...)
> across every operating system. I know the Linux community is working on having
> a device tree bindings out of the kernel tree. Does FreeBSD community plan to
> work with Linux community for this purpose?

We generally try to follow the common definitions for the FDT stuff. There are a few cases where we either lack the feature set of Linux, or where the Linux folks are moving quickly and changing the underlying definitions where we wait for the standards to mature before we implement. In some cases, where maturity hasn't happened, or where the bindings are overly Linux centric (which in theory they aren't supposed to be, but sometimes wind up that way). where we've not implemented things.

> As specified early, the device tree can vary accross Xen version and user input
> (eg the memory). I have noticed few places (mainly the timer) where the IRQs
> number are harcoded.

These cases should be viewed as bugs in FreeBSD, I believe. One of the goals that has wide support, at leas tin theory, is that we can boot with an unaltered Linux FDT. This goal is some time in the future.

> In the long-term, it would be nice to see FreeBSD booting out-of-box (eg the
> device tree is directly provided by the board firmware). I plan to add support
> for Device Tree loading via Linux Boot ABI, it the way that Xen use to boot a
> new guest.

That would be most welcome.

> The second question is related to memory attribute for page table. The early
> page table setup by FreeBSD are using Write-Through memory attribute which
> result to a likely random (not every time the same place) crash before the
> real page table are initialized.
> Replacing Write-Through by Write-Back made FreeBSD boot correctly. Even today,
> I have no idea if it's a requirement from Xen or a bug (either in Xen or
> FreeBSD).

There were some problems with pages being setup improperly for the mutexes to operate properly. Have you confirmed this on a recent version of FreeBSD?

> The code is taking its first faltering steps, therefore the TODO is quite big:
> 	- Try the code on x86 architecture. I did lots of rework for the event
> 	channels code and didn't even try to compile
> 	- Clean up event channel code
> 	- Clean up xen control code
> 	- Add support for Device Tree loading via Linux Boot ABI
> 	- Fixing crashing on userspace. Could be related to the patch
> 	series "arm SMP on Cortex-A15"
> 	- Add guest SMP support
> 	- DOM0 support?
> 
> Any help, comments, questions are welcomed.

I think this is great! We'd love to work with you to make this happen.

Warner

> Sincerely yours,
> 
> ============= Instruction to test FreeBSD on Xen on ARM ===========
[trimmed]

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Julien Grall]

On 01/15/2014 01:26 AM, Warner Losh wrote:
> 
> On Jan 14, 2014, at 2:01 PM, Julien Grall wrote:
>> This new support brings 2 open questions (for both Xen and FreeBSD community).
>> When a new guest is created, the toolstack will generated a device tree which
>> will contains:
>> 	- The amount of memory
>> 	- The description of the platform (gic, timer, hypervisor node)
>> 	- PSCI node for SMP bringup
>>
>> Until now, Xen on ARM supported only Linux-based OS. When the support of
>> device tree was added in Xen for guest, we chose to use Linux device tree
>> bindings (gic, timer,...). It seems that FreeBSD choose a different way to
>> implement device tree:
>> 	- strictly respect ePAR (for interrupt-parent property)
>> 	- gic bindings different (only 1 interrupt cell)
>>
>> I would like to come with a common device tree specification (bindings, ...)
>> across every operating system. I know the Linux community is working on having
>> a device tree bindings out of the kernel tree. Does FreeBSD community plan to
>> work with Linux community for this purpose?
> 
> We generally try to follow the common definitions for the FDT stuff.
> There are a few cases where we either lack the feature set of Linux, or
> where the Linux folks are moving quickly and changing the underlying
definitions
> where we wait for the standards to mature before we implement. In some
cases, where
> maturity hasn't happened, or where the bindings are overly Linux
centric (which in
> theory they aren't supposed to be, but sometimes wind up that way).
where we've not
> implemented things.

As I understand main bindings (gic, timer) are set in stone and won't
change. Ian Campbell has a repository with all the ARM bindings here:
http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;
a=tree;f=Bindings/arm

To compare the difference between the DT provided by Xen, and the one
correctly parsed by FreeBSD
	- Xen: http://xenbits.xen.org/people/julieng/xenvm-4.2.dts
        - FreeBSD: http://xenbits.xen.org/people/julieng/xenvm-bsd.dts

>From Xen side:
	- Every device should move under a simple-bus. I think it's harmless
for Linux side.
	- How about hypervisor node? IHMO this node should also live under the
simple-bus.

>From FreeBSD side:
	- GIC and Timer bindings needs to be handled correctly (see below the
problem for the GIC)
	- Less stricter about interrupt-parent property. Eg, look at the
grant-parent if there is no property interrupt-parent
	- If the hypervisor doesn't live under simple-bus, the interrupt/memory
retrieving should be moved from simple-bus to nexus?

Before the revision r260282 (I have added Nathan on cc), I was able to
use the Linux GIC bindings (see
http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;a=blob;f=Bindings/arm/gic.txt)
without any issue.

It seems the fdt bus, now consider the number of interrupt cells is
always 1.

>> As specified early, the device tree can vary accross Xen version and user input
>> (eg the memory). I have noticed few places (mainly the timer) where the IRQs
>> number are harcoded.
> 
> These cases should be viewed as bugs in FreeBSD, I believe. One of the goals
> that has wide support, at leas tin theory, is that we can boot with an
unaltered
> Linux FDT. This goal is some time in the future.

The timer interrupt used by FreeBSD doesn't match the one used by Xen
(secure vs non-secure interrupt). This should be easily resolved by
retrieving the interrupt from the device tree.

Is there a particular reason to use the secure interrupt for the timer?

> 
>> The second question is related to memory attribute for page table. The early
>> page table setup by FreeBSD are using Write-Through memory attribute which
>> result to a likely random (not every time the same place) crash before the
>> real page table are initialized.
>> Replacing Write-Through by Write-Back made FreeBSD boot correctly. Even today,
>> I have no idea if it's a requirement from Xen or a bug (either in Xen or
>> FreeBSD).
> 
> There were some problems with pages being setup improperly for the mutexes
> to operate properly. Have you confirmed this on a recent version of
FreeBSD?

I have checkout the freeBSD branch last week.

I'm not sure to understand the relation with mutexes. The symptoms I
have is mostly data/prefetch abort receive to FreeBSD. Could it be related?

>> The code is taking its first faltering steps, therefore the TODO is quite big:
>> 	- Try the code on x86 architecture. I did lots of rework for the event
>> 	channels code and didn't even try to compile
>> 	- Clean up event channel code
>> 	- Clean up xen control code
>> 	- Add support for Device Tree loading via Linux Boot ABI
>> 	- Fixing crashing on userspace. Could be related to the patch
>> 	series "arm SMP on Cortex-A15"
>> 	- Add guest SMP support
>> 	- DOM0 support?
>>
>> Any help, comments, questions are welcomed.
> 
> I think this is great! We'd love to work with you to make this happen.
> 
> Warner
> 
>> Sincerely yours,
>>
>> ============= Instruction to test FreeBSD on Xen on ARM ===========
> [trimmed]
> 


-- 
Julien Grall

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Nathan Whitehorn]

On 01/15/14 10:24, Julien Grall wrote:
> On 01/15/2014 01:26 AM, Warner Losh wrote:
>> On Jan 14, 2014, at 2:01 PM, Julien Grall wrote:
>>> This new support brings 2 open questions (for both Xen and FreeBSD community).
>>> When a new guest is created, the toolstack will generated a device tree which
>>> will contains:
>>> 	- The amount of memory
>>> 	- The description of the platform (gic, timer, hypervisor node)
>>> 	- PSCI node for SMP bringup
>>>
>>> Until now, Xen on ARM supported only Linux-based OS. When the support of
>>> device tree was added in Xen for guest, we chose to use Linux device tree
>>> bindings (gic, timer,...). It seems that FreeBSD choose a different way to
>>> implement device tree:
>>> 	- strictly respect ePAR (for interrupt-parent property)
>>> 	- gic bindings different (only 1 interrupt cell)
>>>
>>> I would like to come with a common device tree specification (bindings, ...)
>>> across every operating system. I know the Linux community is working on having
>>> a device tree bindings out of the kernel tree. Does FreeBSD community plan to
>>> work with Linux community for this purpose?
>> We generally try to follow the common definitions for the FDT stuff.
>> There are a few cases where we either lack the feature set of Linux, or
>> where the Linux folks are moving quickly and changing the underlying
> definitions
>> where we wait for the standards to mature before we implement. In some
> cases, where
>> maturity hasn't happened, or where the bindings are overly Linux
> centric (which in
>> theory they aren't supposed to be, but sometimes wind up that way).
> where we've not
>> implemented things.
> As I understand main bindings (gic, timer) are set in stone and won't
> change. Ian Campbell has a repository with all the ARM bindings here:
> http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;
> a=tree;f=Bindings/arm
>
> To compare the difference between the DT provided by Xen, and the one
> correctly parsed by FreeBSD
> 	- Xen: http://xenbits.xen.org/people/julieng/xenvm-4.2.dts
>          - FreeBSD: http://xenbits.xen.org/people/julieng/xenvm-bsd.dts
>
> >From Xen side:
> 	- Every device should move under a simple-bus. I think it's harmless
> for Linux side.
> 	- How about hypervisor node? IHMO this node should also live under the
> simple-bus.
>
> >From FreeBSD side:
> 	- GIC and Timer bindings needs to be handled correctly (see below the
> problem for the GIC)
> 	- Less stricter about interrupt-parent property. Eg, look at the
> grant-parent if there is no property interrupt-parent
> 	- If the hypervisor doesn't live under simple-bus, the interrupt/memory
> retrieving should be moved from simple-bus to nexus?
>
> Before the revision r260282 (I have added Nathan on cc), I was able to
> use the Linux GIC bindings (see
> http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;a=blob;f=Bindings/arm/gic.txt)
> without any issue.
>
> It seems the fdt bus, now consider the number of interrupt cells is
> always 1.
>
>

Thanks for the CC. Could you explain what you mean by "grant-parent" 
etc? "interrupt-parent" is a fundamental part of the way PAPR (and 
ePAPR) work, so I'm very very hesitant to start guessing. I think things 
have broken for you because some (a lot, actually) of OF code does not 
expect #interrupt-cells to be more than 2. Some APIs might need 
updating, which I'll try to take care of. Could you tell me what the 
difference between SPI and PPI is, by the way?

On the subject of simple-bus, they usually aren't necessary. For 
example, all hypervisor devices on IBM hardware live under /vdevice, 
which is attached to the device tree root. They don't use MMIO, so 
simple-bus doesn't really make sense. How does Xen communicate with the 
OS in these devices?
-Nathan

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Stefano Stabellini]

On Wed, 15 Jan 2014, Nathan Whitehorn wrote:
> On 01/15/14 10:24, Julien Grall wrote:
> > On 01/15/2014 01:26 AM, Warner Losh wrote:
> > > On Jan 14, 2014, at 2:01 PM, Julien Grall wrote:
> > > > This new support brings 2 open questions (for both Xen and FreeBSD
> > > > community).
> > > > When a new guest is created, the toolstack will generated a device tree
> > > > which
> > > > will contains:
> > > > 	- The amount of memory
> > > > 	- The description of the platform (gic, timer, hypervisor node)
> > > > 	- PSCI node for SMP bringup
> > > > 
> > > > Until now, Xen on ARM supported only Linux-based OS. When the support of
> > > > device tree was added in Xen for guest, we chose to use Linux device
> > > > tree
> > > > bindings (gic, timer,...). It seems that FreeBSD choose a different way
> > > > to
> > > > implement device tree:
> > > > 	- strictly respect ePAR (for interrupt-parent property)
> > > > 	- gic bindings different (only 1 interrupt cell)
> > > > 
> > > > I would like to come with a common device tree specification (bindings,
> > > > ...)
> > > > across every operating system. I know the Linux community is working on
> > > > having
> > > > a device tree bindings out of the kernel tree. Does FreeBSD community
> > > > plan to
> > > > work with Linux community for this purpose?
> > > We generally try to follow the common definitions for the FDT stuff.
> > > There are a few cases where we either lack the feature set of Linux, or
> > > where the Linux folks are moving quickly and changing the underlying
> > definitions
> > > where we wait for the standards to mature before we implement. In some
> > cases, where
> > > maturity hasn't happened, or where the bindings are overly Linux
> > centric (which in
> > > theory they aren't supposed to be, but sometimes wind up that way).
> > where we've not
> > > implemented things.
> > As I understand main bindings (gic, timer) are set in stone and won't
> > change. Ian Campbell has a repository with all the ARM bindings here:
> > http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;
> > a=tree;f=Bindings/arm
> > 
> > To compare the difference between the DT provided by Xen, and the one
> > correctly parsed by FreeBSD
> > 	- Xen: http://xenbits.xen.org/people/julieng/xenvm-4.2.dts
> >          - FreeBSD: http://xenbits.xen.org/people/julieng/xenvm-bsd.dts
> > 
> > >From Xen side:
> > 	- Every device should move under a simple-bus. I think it's harmless
> > for Linux side.
> > 	- How about hypervisor node? IHMO this node should also live under the
> > simple-bus.
> > 
> > >From FreeBSD side:
> > 	- GIC and Timer bindings needs to be handled correctly (see below the
> > problem for the GIC)
> > 	- Less stricter about interrupt-parent property. Eg, look at the
> > grant-parent if there is no property interrupt-parent
> > 	- If the hypervisor doesn't live under simple-bus, the
> > interrupt/memory
> > retrieving should be moved from simple-bus to nexus?
> > 
> > Before the revision r260282 (I have added Nathan on cc), I was able to
> > use the Linux GIC bindings (see
> > http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;a=blob;f=Bindings/arm/gic.txt)
> > without any issue.
> > 
> > It seems the fdt bus, now consider the number of interrupt cells is
> > always 1.
> > 
> > 
> 
> Thanks for the CC. Could you explain what you mean by "grant-parent" etc?
> "interrupt-parent" is a fundamental part of the way PAPR (and ePAPR) work, so
> I'm very very hesitant to start guessing. I think things have broken for you
> because some (a lot, actually) of OF code does not expect #interrupt-cells to
> be more than 2. Some APIs might need updating, which I'll try to take care of.
> Could you tell me what the difference between SPI and PPI is, by the way?

SPI: Shared Peripheral Interrupt
PPI: Private Peripheral Interrupt

PPIs are per processor interrupts.


> On the subject of simple-bus, they usually aren't necessary. For example, all
> hypervisor devices on IBM hardware live under /vdevice, which is attached to
> the device tree root. They don't use MMIO, so simple-bus doesn't really make
> sense. How does Xen communicate with the OS in these devices?

The PPI and the memory region advertized under the Xen hypervisor node
on device tree are used to setup the basic infrastructure (grant table,
event channels) needed by Xen paravirtualized devices.

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Ian Campbell]

On Wed, 2014-01-15 at 16:24 +0000, Julien Grall wrote:
> As I understand main bindings (gic, timer) are set in stone and won't
> change. Ian Campbell has a repository with all the ARM bindings here:
> http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;a=tree;f=Bindings/arm 

FYI this tree is automatically extracted from the bindings in the Linux
source, because long term we would like to split them out from Linux and
try and standardize some (if not all) of them. This is exactly because
these bindings should not be OS specific...

Ian.

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Julien Grall]

On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
>
> Thanks for the CC. Could you explain what you mean by "grant-parent"
> etc? "interrupt-parent" is a fundamental part of the way PAPR (and
> ePAPR) work, so I'm very very hesitant to start guessing. I think things
> have broken for you because some (a lot, actually) of OF code does not
> expect #interrupt-cells to be more than 2. Some APIs might need
> updating, which I'll try to take care of. Could you tell me what the
> difference between SPI and PPI is, by the way?

Sorry, I also made some typoes in my explanation so it was not clear.

interrupt-parent is a property in a device node which links this node to 
an interrupt controller (in our case the GIC controller).

The way to handle it on Linux and the ePAR is different:
    - ePAR (chapter 2.4) says:
The physical wiring of an interrupt source to an interrupt controller is 
represented in the device tree with the interrupt-parent property. Nodes 
that represent interrupt-generating devices contain an
interrupt-parent property which has a phandle value that points to the 
device to which the device's interrupts are routed, typically an 
interrupt controller. If an interrupt-generating device does not have
an interrupt-parent property, its interrupt parent is assumed to be its 
device tree parent.
 From my understanding, it's mandatory to have an interrupt-parent 
property on each device node which is using interrupts. If it doesn't 
exist it will assume that the parent is interrupt controller.
If I'm mistaken, at least FreeBSD handle the interrupt-parent property 
in this way.
    - Linux implementation will look at to the node, if the property 
doesn't exists, it will check if the ancestor has this property ...

So the device tree below is valid on Linux, but not on FreeBSD:

/ {
   interrupt-parent = &gic

   gic: gic@10
   {
     ...
   }

   timer@1
   {
     interrupts = <...>
   }
}

Most of shipped device tree use this trick.

IanC: I was reading the linux binding documentation 
(devicetree/booting-without-of.txt VII.2) and it seems that the 
explanation differs from the implementation, right?

For the #interrupt-cells property, the problem starts in fdt_intr_to_rl 
(sys/dev/fdt/fdt_common.c:476). ofw_bus_map_intr is called always with 
the first cells of the interrupt no matter the number of cells specified 
by #interrupt-cells.

> On the subject of simple-bus, they usually aren't necessary. For
> example, all hypervisor devices on IBM hardware live under /vdevice,
> which is attached to the device tree root. They don't use MMIO, so
> simple-bus doesn't really make sense. How does Xen communicate with the
> OS in these devices?
> -Nathan

As I understand, only the simple bus code (see simplebus_attach) is 
translating the interrupts in the device on a resource.
So if you have a node directly attached to the root node with interrupts 
and MMIO, the driver won't be able to retrieve and translate the 
interrupts via bus_alloc_resources.

In the Xen device tree, we have an hypervisor node directly attach to 
the root which contains both MMIO and interrupt used by Xen to 
communicate with the guest.

-- 
Julien Grall

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Nathan Whitehorn]

On 01/16/14 18:36, Julien Grall wrote:
>
>
> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
>>
>> Thanks for the CC. Could you explain what you mean by "grant-parent"
>> etc? "interrupt-parent" is a fundamental part of the way PAPR (and
>> ePAPR) work, so I'm very very hesitant to start guessing. I think things
>> have broken for you because some (a lot, actually) of OF code does not
>> expect #interrupt-cells to be more than 2. Some APIs might need
>> updating, which I'll try to take care of. Could you tell me what the
>> difference between SPI and PPI is, by the way?
>
> Sorry, I also made some typoes in my explanation so it was not clear.
>
> interrupt-parent is a property in a device node which links this node 
> to an interrupt controller (in our case the GIC controller).
>
> The way to handle it on Linux and the ePAR is different:
>    - ePAR (chapter 2.4) says:
> The physical wiring of an interrupt source to an interrupt controller 
> is represented in the device tree with the interrupt-parent property. 
> Nodes that represent interrupt-generating devices contain an
> interrupt-parent property which has a phandle value that points to the 
> device to which the device's interrupts are routed, typically an 
> interrupt controller. If an interrupt-generating device does not have
> an interrupt-parent property, its interrupt parent is assumed to be 
> its device tree parent.
> From my understanding, it's mandatory to have an interrupt-parent 
> property on each device node which is using interrupts. If it doesn't 
> exist it will assume that the parent is interrupt controller.
> If I'm mistaken, at least FreeBSD handle the interrupt-parent property 
> in this way.
>    - Linux implementation will look at to the node, if the property 
> doesn't exists, it will check if the ancestor has this property ...



> So the device tree below is valid on Linux, but not on FreeBSD:
>
> / {
>   interrupt-parent = &gic
>
>   gic: gic@10
>   {
>     ...
>   }
>
>   timer@1
>   {
>     interrupts = <...>
>   }
> }
>
> Most of shipped device tree use this trick.
>
> IanC: I was reading the linux binding documentation 
> (devicetree/booting-without-of.txt VII.2) and it seems that the 
> explanation differs from the implementation, right?
>
> For the #interrupt-cells property, the problem starts in 
> fdt_intr_to_rl (sys/dev/fdt/fdt_common.c:476). ofw_bus_map_intr is 
> called always with the first cells of the interrupt no matter the 
> number of cells specified by #interrupt-cells.

The specification is actually a little unclear on this point, but 
FreeBSD follows the same rules as Linux in any case. Most, if not all, 
FreeBSD code should check any ancestor at this point as well. In 
particular fdt_intr_to_rl does this. What it *doesn't* do is allow 
#interrupt-cells to be larger than 2. I'll fix this this weekend.

>> On the subject of simple-bus, they usually aren't necessary. For
>> example, all hypervisor devices on IBM hardware live under /vdevice,
>> which is attached to the device tree root. They don't use MMIO, so
>> simple-bus doesn't really make sense. How does Xen communicate with the
>> OS in these devices?
>> -Nathan
>
> As I understand, only the simple bus code (see simplebus_attach) is 
> translating the interrupts in the device on a resource.
> So if you have a node directly attached to the root node with 
> interrupts and MMIO, the driver won't be able to retrieve and 
> translate the interrupts via bus_alloc_resources.

Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this.

> In the Xen device tree, we have an hypervisor node directly attach to 
> the root which contains both MMIO and interrupt used by Xen to 
> communicate with the guest.
>

OK. This should be fine, though simplebus would also work if you use MMIO.
-Nathan

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Ian Campbell]

On Fri, 2014-01-17 at 00:36 +0000, Julien Grall wrote:
> 
> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
> >
> > Thanks for the CC. Could you explain what you mean by "grant-parent"
> > etc? "interrupt-parent" is a fundamental part of the way PAPR (and
> > ePAPR) work, so I'm very very hesitant to start guessing. I think things
> > have broken for you because some (a lot, actually) of OF code does not
> > expect #interrupt-cells to be more than 2. Some APIs might need
> > updating, which I'll try to take care of. Could you tell me what the
> > difference between SPI and PPI is, by the way?
> 
> Sorry, I also made some typoes in my explanation so it was not clear.
> 
> interrupt-parent is a property in a device node which links this node to 
> an interrupt controller (in our case the GIC controller).
> 
> The way to handle it on Linux and the ePAR is different:
>     - ePAR (chapter 2.4) says:
> The physical wiring of an interrupt source to an interrupt controller is 
> represented in the device tree with the interrupt-parent property. Nodes 
> that represent interrupt-generating devices contain an
> interrupt-parent property which has a phandle value that points to the 
> device to which the device's interrupts are routed, typically an 
> interrupt controller. If an interrupt-generating device does not have
> an interrupt-parent property, its interrupt parent is assumed to be its 
> device tree parent.
>  From my understanding, it's mandatory to have an interrupt-parent 
> property on each device node which is using interrupts. If it doesn't 
> exist it will assume that the parent is interrupt controller.
> If I'm mistaken, at least FreeBSD handle the interrupt-parent property 
> in this way.
>     - Linux implementation will look at to the node, if the property 
> doesn't exists, it will check if the ancestor has this property ...
> 
> So the device tree below is valid on Linux, but not on FreeBSD:
> 
> / {
>    interrupt-parent = &gic
> 
>    gic: gic@10
>    {
>      ...
>    }
> 
>    timer@1
>    {
>      interrupts = <...>
>    }
> }
> 
> Most of shipped device tree use this trick.
> 
> IanC: I was reading the linux binding documentation 
> (devicetree/booting-without-of.txt VII.2) and it seems that the 
> explanation differs from the implementation, right?

I vaguely recall someone saying that the Linux behaviour was a quirk of
some real PPC system which supplied a DTB which required this behaviour
which has leaked into the other platforms. It does also sound like a
useful extension to the spec which makes the dtb easier to write.

The right place to discuss this would probably be either #devicetree on
freenode or devicetree@vger.kernel.org.

> > On the subject of simple-bus, they usually aren't necessary. For
> > example, all hypervisor devices on IBM hardware live under /vdevice,
> > which is attached to the device tree root. They don't use MMIO, so
> > simple-bus doesn't really make sense. How does Xen communicate with the
> > OS in these devices?
> > -Nathan
> 
> As I understand, only the simple bus code (see simplebus_attach) is 
> translating the interrupts in the device on a resource.
> So if you have a node directly attached to the root node with interrupts 
> and MMIO, the driver won't be able to retrieve and translate the 
> interrupts via bus_alloc_resources.

Is the root node not considered to be a "top-level simple-bus" with a
1:1 mapping of MMIO and interrupts? (Linux seems to treat it this way,
but I haven't trawled the docs for a spec reference to back that
behaviour up). I take it BSD doesn't do this?

Ian.

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Julien Grall]

On 01/17/2014 03:04 AM, Nathan Whitehorn wrote:
> On 01/16/14 18:36, Julien Grall wrote:
>
> The specification is actually a little unclear on this point, but
> FreeBSD follows the same rules as Linux in any case. Most, if not all,
> FreeBSD code should check any ancestor at this point as well. In
> particular fdt_intr_to_rl does this. What it *doesn't* do is allow
> #interrupt-cells to be larger than 2. I'll fix this this weekend.

Thanks, for working on this part.

Another things to take into account: the first cell doesn't always 
contain the interrupt.
With the Linux binding (#interrupt-cells == 3)
   - cell 1: 1 or 0 (PPI vs SPI)
   - cell 2: relative IRQ number to the start of PPI/SPI
   - cell 3: cpu mask + interrupt flags (edge/level...)

>>> On the subject of simple-bus, they usually aren't necessary. For
>>> example, all hypervisor devices on IBM hardware live under /vdevice,
>>> which is attached to the device tree root. They don't use MMIO, so
>>> simple-bus doesn't really make sense. How does Xen communicate with the
>>> OS in these devices?
>>> -Nathan
>>
>> As I understand, only the simple bus code (see simplebus_attach) is
>> translating the interrupts in the device on a resource.
>> So if you have a node directly attached to the root node with
>> interrupts and MMIO, the driver won't be able to retrieve and
>> translate the interrupts via bus_alloc_resources.
>
> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this.

I have noticed at least one issue (which is not related to my problem):
   - When the OFW nexus translate IRQ (with #interrupt-cells > 1), the 
rid will be equal to 0, 0 + #interrupt-cells, ... So the number will be 
discontinued. Rather than on simple-bus for the same device, the rid 
will be 0, 1, 2...

For my issue, I will look at it again this week-end.

BTW when I look to the FDT (sys/dev/fdt_common.c) and the ofw 
(sys/dev/ofw_nexus.c) code, I have notice that lots of code are duplicated.

It would be nice to have common helper to avoid duplicate code and issue 
for the future :).

-- 
Julien Grall

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Julien Grall]

On 01/17/2014 09:29 AM, Ian Campbell wrote:
> On Fri, 2014-01-17 at 00:36 +0000, Julien Grall wrote:
>>
>> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
>>> On the subject of simple-bus, they usually aren't necessary. For
>>> example, all hypervisor devices on IBM hardware live under /vdevice,
>>> which is attached to the device tree root. They don't use MMIO, so
>>> simple-bus doesn't really make sense. How does Xen communicate with the
>>> OS in these devices?
>>> -Nathan
>>
>> As I understand, only the simple bus code (see simplebus_attach) is
>> translating the interrupts in the device on a resource.
>> So if you have a node directly attached to the root node with interrupts
>> and MMIO, the driver won't be able to retrieve and translate the
>> interrupts via bus_alloc_resources.
>
> Is the root node not considered to be a "top-level simple-bus" with a
> 1:1 mapping of MMIO and interrupts? (Linux seems to treat it this way,
> but I haven't trawled the docs for a spec reference to back that
> behaviour up). I take it BSD doesn't do this?

There is 2 different paths on FreeBSD to decode interrupt/MMIO 
(depending if you are under the root node or a simple-bus node). Most of 
the code is duplicated but there are some parts which differs (for 
instance interrupt decoding, see my answer to Nathan).

I will look at closer to the code this week-end and see if I can fix it.

-- 
Julien Grall

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Nathan Whitehorn]

On 01/17/14 07:45, Julien Grall wrote:
>
>
> On 01/17/2014 03:04 AM, Nathan Whitehorn wrote:
>> On 01/16/14 18:36, Julien Grall wrote:
>>
>> The specification is actually a little unclear on this point, but
>> FreeBSD follows the same rules as Linux in any case. Most, if not all,
>> FreeBSD code should check any ancestor at this point as well. In
>> particular fdt_intr_to_rl does this. What it *doesn't* do is allow
>> #interrupt-cells to be larger than 2. I'll fix this this weekend.
>
> Thanks, for working on this part.
>
> Another things to take into account: the first cell doesn't always 
> contain the interrupt.
> With the Linux binding (#interrupt-cells == 3)
>   - cell 1: 1 or 0 (PPI vs SPI)
>   - cell 2: relative IRQ number to the start of PPI/SPI
>   - cell 3: cpu mask + interrupt flags (edge/level...)
> `

Yep. This will require a little API redesign, but shouldn't be that bad.

>>>> On the subject of simple-bus, they usually aren't necessary. For
>>>> example, all hypervisor devices on IBM hardware live under /vdevice,
>>>> which is attached to the device tree root. They don't use MMIO, so
>>>> simple-bus doesn't really make sense. How does Xen communicate with 
>>>> the
>>>> OS in these devices?
>>>> -Nathan
>>>
>>> As I understand, only the simple bus code (see simplebus_attach) is
>>> translating the interrupts in the device on a resource.
>>> So if you have a node directly attached to the root node with
>>> interrupts and MMIO, the driver won't be able to retrieve and
>>> translate the interrupts via bus_alloc_resources.
>>
>> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this.
>
> I have noticed at least one issue (which is not related to my problem):
>   - When the OFW nexus translate IRQ (with #interrupt-cells > 1), the 
> rid will be equal to 0, 0 + #interrupt-cells, ... So the number will 
> be discontinued. Rather than on simple-bus for the same device, the 
> rid will be 0, 1, 2...

Interesting. I'll investigate.

> For my issue, I will look at it again this week-end.
>
> BTW when I look to the FDT (sys/dev/fdt_common.c) and the ofw 
> (sys/dev/ofw_nexus.c) code, I have notice that lots of code are 
> duplicated.
>
> It would be nice to have common helper to avoid duplicate code and 
> issue for the future :).
>

I'm in the middle of cleaning all this up (which is how I'm on the hook 
for breaking your case!). Most of fdt_common.c is not long for this world.
-Nathan

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Julien Grall]

Hello Nathan,

On 01/17/2014 03:04 AM, Nathan Whitehorn wrote:
> On 01/16/14 18:36, Julien Grall wrote:
>>
>>
>> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
>> As I understand, only the simple bus code (see simplebus_attach) is
>> translating the interrupts in the device on a resource.
>> So if you have a node directly attached to the root node with
>> interrupts and MMIO, the driver won't be able to retrieve and
>> translate the interrupts via bus_alloc_resources.

> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this.

I have digged into the code to find the reason of my issue. FreeBSD is 
receiving a VM fault when the driver (xen-dt) is trying to setup the IRQ.

This is because the GIC is not yet initialized but FreeBSD asks to 
unmask the IRQ (sys/arm/arm/gic.c:306).

With this problem, all device nodes that are before the GIC in the 
device tree can't have interrupts. For instance this simple device will 
segfault on FreeBSD:

/ {

   mybus {
      compatible = "simple-bus";

      mynode {
         interrupt-parent = &gic;
         interrupts = <...>;
      };

      gic: gic@xxxx {
         interrupt-controller;
      }
   };
};

The node "mynode" will have to move after the GIC to be able to work 
correctly.

-- 
Julien Grall

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Julien Grall]

On 01/17/2014 09:29 AM, Ian Campbell wrote:
> On Fri, 2014-01-17 at 00:36 +0000, Julien Grall wrote:
>> IanC: I was reading the linux binding documentation
>> (devicetree/booting-without-of.txt VII.2) and it seems that the
>> explanation differs from the implementation, right?
>
> I vaguely recall someone saying that the Linux behaviour was a quirk of
> some real PPC system which supplied a DTB which required this behaviour
> which has leaked into the other platforms. It does also sound like a
> useful extension to the spec which makes the dtb easier to write.

I've read twice the ePAR and did some test. I was completely wrong, this 
file is valid with the ePAR (there is even an example like that at the 
end at the specification). And FreeBSD doesn't complain about this since 
Nathan's ofw/fdt rework.

Sorry for the waste time.

-- 
Julien Grall

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Nathan Whitehorn]

On 01/18/14 17:41, Julien Grall wrote:
>
> Hello Nathan,
>
> On 01/17/2014 03:04 AM, Nathan Whitehorn wrote:
>> On 01/16/14 18:36, Julien Grall wrote:
>>>
>>>
>>> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
>>> As I understand, only the simple bus code (see simplebus_attach) is
>>> translating the interrupts in the device on a resource.
>>> So if you have a node directly attached to the root node with
>>> interrupts and MMIO, the driver won't be able to retrieve and
>>> translate the interrupts via bus_alloc_resources.
>
>> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this.
>
> I have digged into the code to find the reason of my issue. FreeBSD is
> receiving a VM fault when the driver (xen-dt) is trying to setup the IRQ.
>
> This is because the GIC is not yet initialized but FreeBSD asks to
> unmask the IRQ (sys/arm/arm/gic.c:306).
>
> With this problem, all device nodes that are before the GIC in the
> device tree can't have interrupts. For instance this simple device
> will segfault on FreeBSD:
>
> / {
>
>   mybus {
>      compatible = "simple-bus";
>
>      mynode {
>         interrupt-parent = &gic;
>         interrupts = <...>;
>      };
>
>      gic: gic@xxxx {
>         interrupt-controller;
>      }
>   };
> };
>
> The node "mynode" will have to move after the GIC to be able to work
> correctly.
>

Ah, that sounds like a bug in the interrupt handling code. The PPC code
is designed to handle this problem by deferring interrupt setup, as well
as a number of other latent issues, and I think would make a good match
for ARM as well. I made an experimental branch to port it to MIPS (the
code is almost entirely machine-independent) but am waiting for testing.
A general solution to this problem has to involve deferred setup.
-Nathan

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Warner Losh]

On Jan 18, 2014, at 4:41 PM, Julien Grall wrote:

> 
> Hello Nathan,
> 
> On 01/17/2014 03:04 AM, Nathan Whitehorn wrote:
>> On 01/16/14 18:36, Julien Grall wrote:
>>> 
>>> 
>>> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
>>> As I understand, only the simple bus code (see simplebus_attach) is
>>> translating the interrupts in the device on a resource.
>>> So if you have a node directly attached to the root node with
>>> interrupts and MMIO, the driver won't be able to retrieve and
>>> translate the interrupts via bus_alloc_resources.
> 
>> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this.
> 
> I have digged into the code to find the reason of my issue. FreeBSD is receiving a VM fault when the driver (xen-dt) is trying to setup the IRQ.
> 
> This is because the GIC is not yet initialized but FreeBSD asks to unmask the IRQ (sys/arm/arm/gic.c:306).
> 
> With this problem, all device nodes that are before the GIC in the device tree can't have interrupts. For instance this simple device will segfault on FreeBSD:
> 
> / {
> 
>  mybus {
>     compatible = "simple-bus";
> 
>     mynode {
>        interrupt-parent = &gic;
>        interrupts = <...>;
>     };
> 
>     gic: gic@xxxx {
>        interrupt-controller;
>     }
>  };
> };
> 
> The node "mynode" will have to move after the GIC to be able to work correctly.

This stems from a difference in enumeration between FreeBSD and Linux. FreeBSD enumerates the devices in DTB order, while Linux does a partial ordering based on dependencies.

Warner

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Nathan Whitehorn]

On 01/18/14 20:44, Warner Losh wrote:
> On Jan 18, 2014, at 4:41 PM, Julien Grall wrote:
>
>> Hello Nathan,
>>
>> On 01/17/2014 03:04 AM, Nathan Whitehorn wrote:
>>> On 01/16/14 18:36, Julien Grall wrote:
>>>>
>>>> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
>>>> As I understand, only the simple bus code (see simplebus_attach) is
>>>> translating the interrupts in the device on a resource.
>>>> So if you have a node directly attached to the root node with
>>>> interrupts and MMIO, the driver won't be able to retrieve and
>>>> translate the interrupts via bus_alloc_resources.
>>> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this.
>> I have digged into the code to find the reason of my issue. FreeBSD is receiving a VM fault when the driver (xen-dt) is trying to setup the IRQ.
>>
>> This is because the GIC is not yet initialized but FreeBSD asks to unmask the IRQ (sys/arm/arm/gic.c:306).
>>
>> With this problem, all device nodes that are before the GIC in the device tree can't have interrupts. For instance this simple device will segfault on FreeBSD:
>>
>> / {
>>
>>  mybus {
>>     compatible = "simple-bus";
>>
>>     mynode {
>>        interrupt-parent = &gic;
>>        interrupts = <...>;
>>     };
>>
>>     gic: gic@xxxx {
>>        interrupt-controller;
>>     }
>>  };
>> };
>>
>> The node "mynode" will have to move after the GIC to be able to work correctly.
> This stems from a difference in enumeration between FreeBSD and Linux. FreeBSD enumerates the devices in DTB order, while Linux does a partial ordering based on dependencies.
>
> Warner

Enumerating in some other order doesn't necessarily help: since the
interrupt and bus trees are independent, circular dependencies can
happen. This is not a hypothetical: on most powermacs, the main
interrupt controller is a functional unit on a PCI device -- a PCI
device whose other units have interrupt lines that eventually connect
back to itself. There is no way to fix that with ordering. So I think we
still need to defer interrupt setup. It's not that bad -- PPC already
does this to handle the powermac case.
-Nathan

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Warner Losh]

On Jan 18, 2014, at 8:01 PM, Nathan Whitehorn wrote:

> On 01/18/14 20:44, Warner Losh wrote:
>> On Jan 18, 2014, at 4:41 PM, Julien Grall wrote:
>> 
>>> Hello Nathan,
>>> 
>>> On 01/17/2014 03:04 AM, Nathan Whitehorn wrote:
>>>> On 01/16/14 18:36, Julien Grall wrote:
>>>>> 
>>>>> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
>>>>> As I understand, only the simple bus code (see simplebus_attach) is
>>>>> translating the interrupts in the device on a resource.
>>>>> So if you have a node directly attached to the root node with
>>>>> interrupts and MMIO, the driver won't be able to retrieve and
>>>>> translate the interrupts via bus_alloc_resources.
>>>> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this.
>>> I have digged into the code to find the reason of my issue. FreeBSD is receiving a VM fault when the driver (xen-dt) is trying to setup the IRQ.
>>> 
>>> This is because the GIC is not yet initialized but FreeBSD asks to unmask the IRQ (sys/arm/arm/gic.c:306).
>>> 
>>> With this problem, all device nodes that are before the GIC in the device tree can't have interrupts. For instance this simple device will segfault on FreeBSD:
>>> 
>>> / {
>>> 
>>> mybus {
>>>    compatible = "simple-bus";
>>> 
>>>    mynode {
>>>       interrupt-parent = &gic;
>>>       interrupts = <...>;
>>>    };
>>> 
>>>    gic: gic@xxxx {
>>>       interrupt-controller;
>>>    }
>>> };
>>> };
>>> 
>>> The node "mynode" will have to move after the GIC to be able to work correctly.
>> This stems from a difference in enumeration between FreeBSD and Linux. FreeBSD enumerates the devices in DTB order, while Linux does a partial ordering based on dependencies.
>> 
>> Warner
> 
> Enumerating in some other order doesn't necessarily help: since the
> interrupt and bus trees are independent, circular dependencies can
> happen. This is not a hypothetical: on most powermacs, the main
> interrupt controller is a functional unit on a PCI device -- a PCI
> device whose other units have interrupt lines that eventually connect
> back to itself. There is no way to fix that with ordering. So I think we
> still need to defer interrupt setup. It's not that bad -- PPC already
> does this to handle the powermac case.

I guess I've looked at simpler cases where interrupts and GPIO pins need to be up before anything can work on Atmel...  We kinda fake it now, but there's some ordering issues that are solved in this way. But I've not finished the work on bringing Atmel into the FDT world yet. Deferred setup isn't always an option, but I'll keep that in mind in case I hit that case...

The other way to cope is to use the multi-pass enumeration stuff that John Baldwin put into the tree a while ago. In that case, you could enumerate bridges, interrupt controllers, gpio providers, etc first, and then do  a second pass that catches the rest of the devices and the interrupt processing for the first pass devices. This is a variation on the deferred enumeration stuff you are talking about, so that might be a better, more general solution to these sorts of problems.

Warner

Re: [RFC] Add support for Xen ARM guest on FreeBSD

[Nathan Whitehorn]

On 01/18/14 21:08, Warner Losh wrote:
> On Jan 18, 2014, at 8:01 PM, Nathan Whitehorn wrote:
>
>> On 01/18/14 20:44, Warner Losh wrote:
>>> On Jan 18, 2014, at 4:41 PM, Julien Grall wrote:
>>>
>>>> Hello Nathan,
>>>>
>>>> On 01/17/2014 03:04 AM, Nathan Whitehorn wrote:
>>>>> On 01/16/14 18:36, Julien Grall wrote:
>>>>>> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote:
>>>>>> As I understand, only the simple bus code (see simplebus_attach) is
>>>>>> translating the interrupts in the device on a resource.
>>>>>> So if you have a node directly attached to the root node with
>>>>>> interrupts and MMIO, the driver won't be able to retrieve and
>>>>>> translate the interrupts via bus_alloc_resources.
>>>>> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this.
>>>> I have digged into the code to find the reason of my issue. FreeBSD is receiving a VM fault when the driver (xen-dt) is trying to setup the IRQ.
>>>>
>>>> This is because the GIC is not yet initialized but FreeBSD asks to unmask the IRQ (sys/arm/arm/gic.c:306).
>>>>
>>>> With this problem, all device nodes that are before the GIC in the device tree can't have interrupts. For instance this simple device will segfault on FreeBSD:
>>>>
>>>> / {
>>>>
>>>> mybus {
>>>>    compatible = "simple-bus";
>>>>
>>>>    mynode {
>>>>       interrupt-parent = &gic;
>>>>       interrupts = <...>;
>>>>    };
>>>>
>>>>    gic: gic@xxxx {
>>>>       interrupt-controller;
>>>>    }
>>>> };
>>>> };
>>>>
>>>> The node "mynode" will have to move after the GIC to be able to work correctly.
>>> This stems from a difference in enumeration between FreeBSD and Linux. FreeBSD enumerates the devices in DTB order, while Linux does a partial ordering based on dependencies.
>>>
>>> Warner
>> Enumerating in some other order doesn't necessarily help: since the
>> interrupt and bus trees are independent, circular dependencies can
>> happen. This is not a hypothetical: on most powermacs, the main
>> interrupt controller is a functional unit on a PCI device -- a PCI
>> device whose other units have interrupt lines that eventually connect
>> back to itself. There is no way to fix that with ordering. So I think we
>> still need to defer interrupt setup. It's not that bad -- PPC already
>> does this to handle the powermac case.
> I guess I've looked at simpler cases where interrupts and GPIO pins need to be up before anything can work on Atmel...  We kinda fake it now, but there's some ordering issues that are solved in this way. But I've not finished the work on bringing Atmel into the FDT world yet. Deferred setup isn't always an option, but I'll keep that in mind in case I hit that case...
>
> The other way to cope is to use the multi-pass enumeration stuff that John Baldwin put into the tree a while ago. In that case, you could enumerate bridges, interrupt controllers, gpio providers, etc first, and then do  a second pass that catches the rest of the devices and the interrupt processing for the first pass devices. This is a variation on the deferred enumeration stuff you are talking about, so that might be a better, more general solution to these sorts of problems.
>
> Warner
>
>

I'm still not sure that helps. Take the powermac case. The PCI parent
bridge assigns and routes interrupts as part of its probe stage. One
device (called mac-io) is the one that actually has the interrupt
controller, along with other things like ATA and sound. It has several
interrupts to support e.g. ATA. How can the PCI bus attachment sensibly
attach the macio device? If it delays attaching it until the interrupt
controller registers, the bus probing will deadlock, since the interrupt
controller is itself a child of macio! If it attaches it immediately, it
will not be able to route the interrupts of the macio device. It's a
catch-22.

The only solution we were able to come up with when this situation arose
was to treat the bus and interrupt trees as entirely separate things,
built independently, which allows the kernel to break the loop. GPIOs
can have similar problems. The basic issue is that newbus ultimately
assumes that the system topology can be described by a single tree.
Interconnections between the branches -- especially if it isn't even a
DAG -- fundamentally break the model and multipass doesn't necessarily help.
-Nathan

[RFC] Add support for Xen ARM guest on FreeBSD

[Julien Grall]

Hello,

During the last couple of months I have been working on porting FreeBSD on
Xen on ARM.

It's my first big project on FreeBSD and I'm not sure if I have cc the right
MLs and people. If not, could somebody point me to the right email address?

With this patch series I'm able to boot FreeBSD with a single VCPU on every
32 bits platform supported by Xen.

The patch series is divided in 5 parts:
	* #1-#12: Clean up and bug fix with some options
	* #13-#17: Update the Xen interface to Xen 4.4 and fix compilation
	* #18-#32: Update Xen code to support ARM guest
	* #33-#38: Update ARM code to support Xen guest
	* #39-#40: Add platform code to support ARM guest

I have pushed a branch on my git based on Royger's pvh work version 10:
	git clone git://xenbits.xen.org/people/julieng/freebsd.git -b xen-arm
	http://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm

If needed, I can send the patch one by one the different mailing list.

This new support brings 2 open questions (for both Xen and FreeBSD community).
When a new guest is created, the toolstack will generated a device tree which
will contains:
	- The amount of memory
	- The description of the platform (gic, timer, hypervisor node)
	- PSCI node for SMP bringup

Until now, Xen on ARM supported only Linux-based OS. When the support of
device tree was added in Xen for guest, we chose to use Linux device tree
bindings (gic, timer,...). It seems that FreeBSD choose a different way to
implement device tree:
	- strictly respect ePAR (for interrupt-parent property)
	- gic bindings different (only 1 interrupt cell)

I would like to come with a common device tree specification (bindings, ...)
across every operating system. I know the Linux community is working on having
a device tree bindings out of the kernel tree. Does FreeBSD community plan to
work with Linux community for this purpose?

As specified early, the device tree can vary accross Xen version and user input
(eg the memory). I have noticed few places (mainly the timer) where the IRQs
number are harcoded.

In the long-term, it would be nice to see FreeBSD booting out-of-box (eg the
device tree is directly provided by the board firmware). I plan to add support
for Device Tree loading via Linux Boot ABI, it the way that Xen use to boot a
new guest.

The second question is related to memory attribute for page table. The early
page table setup by FreeBSD are using Write-Through memory attribute which
result to a likely random (not every time the same place) crash before the
real page table are initialized.
Replacing Write-Through by Write-Back made FreeBSD boot correctly. Even today,
I have no idea if it's a requirement from Xen or a bug (either in Xen or
FreeBSD).

The code is taking its first faltering steps, therefore the TODO is quite big:
	- Try the code on x86 architecture. I did lots of rework for the event
	channels code and didn't even try to compile
	- Clean up event channel code
	- Clean up xen control code
	- Add support for Device Tree loading via Linux Boot ABI
	- Fixing crashing on userspace. Could be related to the patch
	series "arm SMP on Cortex-A15"
	- Add guest SMP support
	- DOM0 support?

Any help, comments, questions are welcomed.

Sincerely yours,

============= Instruction to test FreeBSD on Xen on ARM ===========

Xen upstream tree doesn't fully support ELF loading for ARM. You will need to
recompile the tools with the following 2 patches:
	- https://patches.linaro.org/22228/
	- https://patches.linaro.org/22227/

To compile and boot Xen on your board, you can refer to the wiki page:
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions

The instruction to compile FreeBSD for Xen on ARM:
$ truncate -s 512 xenvm.img
$ sudo mdconfig -f xenvm.img -u0
$ sudo newfs /dev/md0
$ sudo mount /dev/md0 /mnt

$ sudo make TARGET_ARCH=armv6 kernel-toolchain
$ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
$ sudo make TARGET_ARCH=armv6 buildworld
$ sudo make TARGET_ARCH=armv6 DESTDIR=/mnt installworld distribution

$ echo "/dev/xbd0	/	ufs	rw	1	1" > /mnt/etc/fstab
$ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")

$ sudo umount /mnt
$ sudo mdconfig -d u0

Then you can copy the rootfs and the kernel to DOM 0 on your board.

To boot the a FreeBSD your will required the following configuration file
$ cat freebsd.xl
kernel="kernel"
memory=64
name="freebsd"
vcpus=1
autoballon="off"
disk=[ 'phy:/dev/loop0,xvda,w' ]
$ losetup /dev/loop0 xenvm.img
$ xl create freebsd.xl
$ xl console freebsd

Julien Grall (40):
  xen/netfront: Don't need to include machine/intr_machdep.h
  xen/blkfront: Don't need to include machine/intr_machdep.h
  xen/control: Remove include machine/intr_machdep.h
  xen: Define xen_intr_handle_upcall in common headers
  xen: Remove duplicate features.h header in i386 arch
  xen/hypercall: Allow HYPERVISOR_console_write to take a const string
  xen/timer: Make xen timer optional
  xen/console: Fix build when DDB is enabled
  xen/console: clean up identify callback
  xen/xenstore: xs_probe should return BUS_PROBE_NOWILDCARD
  xen/control: xctlr_probe shoud return BUS_PROBE_NOWILDCARD
  xen/hypervisor: Be sure to set __XEN_INTERFACE_VERSION__
  xen/interface: Update interface to Xen 4.4 headers
  xen: Bump __XEN_INTERFACE_VERSION__ to 4.3
  xen/baloon: Fix compilation with Xen 4.4 headers
  xen/netfront: Fix compilation with Xen 4.4 headers
  xen/gnttab: Fix compilation with Xen 4.4 headers
  xen/ballon: Use correct type for frame list
  xen/netback: Fix printf format for xen_pfn_t
  xen/netfront: Use the correct type for rx_pfn_array
  xen/gnttab: Use the right type for the frames
  xen/gnttab: Export resume_frames
  xen/gnttab: Add a guard for xenpci functions
  xen/netfront: Define PTE flags per architecture
  xen/control: Implement suspend has panic for ARM
  xen: Introduce xen_pmap
  xen: move x86/xen/xenpv.c in dev/xen/xenpv.c
  xen/xenpv: Only add isa for x86 architecture
  xen/xenpv: Protect xenpci code by DEV_XENPCI
  xen: move xen_intr.c to common code
  xen/evtchn: Rework the event channels handler to make it generic
  xen/console: handle console for HVM
  arm/timer: Handle timer with no clock-frequency property
  arm/timer: WORKAROUND The timer should support DT...
  arm: Detect new revision for cortex A15
  arm: add ELFNOTE macro
  arm: Implement disable_intr
  arm: Use Write-Back as memory attribute for early page table
  dts: Add xenvm.dts
  arm: Add xenhvm platform

 sys/amd64/conf/GENERIC                  |    4 +-
 sys/amd64/include/apicvar.h             |    1 -
 sys/amd64/include/xen/hypercall.h       |    2 +-
 sys/amd64/include/xen/xen-os.h          |    2 +
 sys/amd64/include/xen/xenpmap.h         |    2 +-
 sys/arm/arm/cpufunc.c                   |    2 +
 sys/arm/arm/generic_timer.c             |   12 +-
 sys/arm/arm/locore.S                    |    4 +-
 sys/arm/conf/XENHVM                     |   97 +++
 sys/arm/include/armreg.h                |    2 +
 sys/arm/include/asmacros.h              |   26 +
 sys/arm/include/cpufunc.h               |    6 +
 sys/arm/include/xen/hypercall.h         |   50 ++
 sys/arm/include/xen/synch_bitops.h      |   52 ++
 sys/arm/include/xen/xen-os.h            |   29 +
 sys/arm/include/xen/xenfunc.h           |    4 +
 sys/arm/include/xen/xenvar.h            |   14 +
 sys/arm/xenhvm/elf.S                    |   12 +
 sys/arm/xenhvm/files.xenhvm             |   18 +
 sys/arm/xenhvm/hypercall.S              |  107 +++
 sys/arm/xenhvm/xen-dt.c                 |  148 ++++
 sys/arm/xenhvm/xenhvm_machdep.c         |  132 ++++
 sys/boot/fdt/dts/xenvm.dts              |   77 ++
 sys/conf/files                          |    4 +-
 sys/conf/options                        |    3 +
 sys/conf/options.arm                    |    1 +
 sys/dev/xen/balloon/balloon.c           |    6 +-
 sys/dev/xen/blkfront/blkfront.c         |    1 -
 sys/dev/xen/console/console.c           |   28 +-
 sys/dev/xen/console/xencons_ring.c      |   41 +-
 sys/dev/xen/control/control.c           |   19 +-
 sys/dev/xen/netback/netback.c           |    6 +-
 sys/dev/xen/netfront/netfront.c         |   15 +-
 sys/dev/xen/xenpci/xenpci.c             |    2 +-
 sys/dev/xen/xenpv.c                     |  133 ++++
 sys/i386/conf/GENERIC                   |    4 +-
 sys/i386/conf/XEN                       |    1 +
 sys/i386/include/apicvar.h              |    1 -
 sys/i386/include/xen/features.h         |   22 -
 sys/i386/include/xen/hypercall.h        |    2 +-
 sys/i386/include/xen/xen-os.h           |    2 +
 sys/i386/include/xen/xenvar.h           |    2 +-
 sys/i386/xen/xen_machdep.c              |    2 +-
 sys/x86/xen/xen_intr.c                  | 1241 -------------------------------
 sys/x86/xen/xenpv.c                     |  128 ----
 sys/xen/gnttab.c                        |   23 +-
 sys/xen/hypervisor.h                    |    3 +-
 sys/xen/interface/arch-arm.h            |  259 +++++--
 sys/xen/interface/arch-arm/hvm/save.h   |    2 +-
 sys/xen/interface/arch-x86/hvm/save.h   |   25 +-
 sys/xen/interface/arch-x86/xen-mca.h    |    2 +-
 sys/xen/interface/arch-x86/xen-x86_32.h |    2 +-
 sys/xen/interface/arch-x86/xen-x86_64.h |    2 +-
 sys/xen/interface/arch-x86/xen.h        |   31 +-
 sys/xen/interface/callback.h            |    4 +-
 sys/xen/interface/dom0_ops.h            |    2 +-
 sys/xen/interface/domctl.h              |   65 +-
 sys/xen/interface/elfnote.h             |   10 +-
 sys/xen/interface/event_channel.h       |    5 +-
 sys/xen/interface/features.h            |   16 +-
 sys/xen/interface/gcov.h                |  115 +++
 sys/xen/interface/grant_table.h         |    8 +-
 sys/xen/interface/hvm/hvm_xs_strings.h  |   80 ++
 sys/xen/interface/hvm/ioreq.h           |   20 +-
 sys/xen/interface/hvm/params.h          |   14 +-
 sys/xen/interface/hvm/pvdrivers.h       |   47 ++
 sys/xen/interface/hvm/save.h            |    4 +-
 sys/xen/interface/io/blkif.h            |  104 ++-
 sys/xen/interface/io/console.h          |    2 +-
 sys/xen/interface/io/fbif.h             |    2 +-
 sys/xen/interface/io/kbdif.h            |    2 +-
 sys/xen/interface/io/netif.h            |   47 +-
 sys/xen/interface/io/pciif.h            |    3 +-
 sys/xen/interface/io/protocols.h        |    5 +-
 sys/xen/interface/io/tpmif.h            |   68 +-
 sys/xen/interface/io/usbif.h            |    1 -
 sys/xen/interface/io/vscsiif.h          |   42 +-
 sys/xen/interface/io/xenbus.h           |   20 +-
 sys/xen/interface/io/xs_wire.h          |    7 +-
 sys/xen/interface/kexec.h               |    7 +-
 sys/xen/interface/mem_event.h           |    4 +-
 sys/xen/interface/memory.h              |   80 +-
 sys/xen/interface/nmi.h                 |    7 +-
 sys/xen/interface/physdev.h             |   33 +-
 sys/xen/interface/platform.h            |   27 +-
 sys/xen/interface/sched.h               |   87 ++-
 sys/xen/interface/sysctl.h              |   47 +-
 sys/xen/interface/tmem.h                |    2 +-
 sys/xen/interface/trace.h               |   93 ++-
 sys/xen/interface/vcpu.h                |    2 +-
 sys/xen/interface/version.h             |    6 +-
 sys/xen/interface/xen-compat.h          |    2 +-
 sys/xen/interface/xen.h                 |  128 +++-
 sys/xen/interface/xenoprof.h            |    2 +-
 sys/xen/interface/xsm/flask_op.h        |    8 +
 sys/xen/xen-os.h                        |    2 +-
 sys/xen/xen_intr.c                      | 1072 ++++++++++++++++++++++++++
 sys/xen/xen_intr.h                      |    8 +-
 sys/xen/xenstore/xenstore.c             |    4 +-
 99 files changed, 3339 insertions(+), 1791 deletions(-)
 create mode 100644 sys/arm/conf/XENHVM
 create mode 100644 sys/arm/include/xen/hypercall.h
 create mode 100644 sys/arm/include/xen/synch_bitops.h
 create mode 100644 sys/arm/include/xen/xen-os.h
 create mode 100644 sys/arm/include/xen/xenfunc.h
 create mode 100644 sys/arm/include/xen/xenvar.h
 create mode 100644 sys/arm/xenhvm/elf.S
 create mode 100644 sys/arm/xenhvm/files.xenhvm
 create mode 100644 sys/arm/xenhvm/hypercall.S
 create mode 100644 sys/arm/xenhvm/xen-dt.c
 create mode 100644 sys/arm/xenhvm/xenhvm_machdep.c
 create mode 100644 sys/boot/fdt/dts/xenvm.dts
 create mode 100644 sys/dev/xen/xenpv.c
 delete mode 100644 sys/i386/include/xen/features.h
 delete mode 100644 sys/x86/xen/xen_intr.c
 delete mode 100644 sys/x86/xen/xenpv.c
 create mode 100644 sys/xen/interface/gcov.h
 create mode 100644 sys/xen/interface/hvm/hvm_xs_strings.h
 create mode 100644 sys/xen/interface/hvm/pvdrivers.h
 create mode 100644 sys/xen/xen_intr.c

-- 
1.8.0