Using DT overlays for adding virtual hardware

Using DT overlays for adding virtual hardware

[Jan Kiszka]

Hi all,

already started the discussion off-list with Pantelis, but it's better
done in public:

I'm currently exploring ways to make Linux recognize dynamically added
virtual hardware when running under the Jailhouse hypervisor [1]. We
need to load drivers for inter-partition communication devices that only
appear after Jailhouse started (which is done from within Linux, i.e.
long after boot) or when a partition was added later on. Probably, we
will simply add a virtual PCI host bridge on systems without physical
PCI and let the IPC device be explored that way (already works on x86).
Still, that leaves us with hotplug and unplug on hypervisor activation
and deactivation.

Inspired by how FPGA bitstream loaders and the caps/hats for BeagleBones
and RasPis work, I thought of using device tree overlays for this.
However, it seems there are still some gaps in upstream Linux /wrt
overlays. So I would like to find out what already works and what may
need further patches.

The plan (open for suggestions) is as follows:

- have an overlay template for a virtual platform device
- compile it into the Jailhouse loader kernel module
- adjust key parameters (e.g. register base addresses) after
  instantiating a device (of_update_property?)
- push modified dt into of_overlay_create

To my understanding, the last step will trigger driver probing which
will cause the device being activated (provided the driver is available,
of course).

I played with the fragment from
Documentation/devicetree/overlay-notes.txt, but already that failed
because the in-kernel dtc does not support /plugin/ yet, right? (Why
not, btw?) What else do I need to patch/update to make the above work?
Or are the simpler ways to achieve what we need?

Thanks,
Jan

[1] https://github.com/siemens/jailhouse

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Mark Rutland]

On Wed, Jun 08, 2016 at 04:16:32PM +0200, Jan Kiszka wrote:
> Hi all,
> 
> already started the discussion off-list with Pantelis, but it's better
> done in public:
> 
> I'm currently exploring ways to make Linux recognize dynamically added
> virtual hardware when running under the Jailhouse hypervisor [1]. We
> need to load drivers for inter-partition communication devices that only
> appear after Jailhouse started (which is done from within Linux, i.e.
> long after boot) or when a partition was added later on. Probably, we
> will simply add a virtual PCI host bridge on systems without physical
> PCI and let the IPC device be explored that way (already works on x86).
> Still, that leaves us with hotplug and unplug on hypervisor activation
> and deactivation.

If I've understood correctly you want to use overlays to inject the
virtual PCI host bridge?

Given that you know precisely what you want to inject, I'm not sure I
see the value of using an overlay. 

Is there some reason you can't just create a device without having to go
via an intermediate step? As I understand it, Xen does that for (some)
virtual devices provided to Dom0 and DomU.

Thanks,
Mark.

Re: Using DT overlays for adding virtual hardware

[Jan Kiszka]

On 2016-06-08 17:17, Mark Rutland wrote:
> On Wed, Jun 08, 2016 at 04:16:32PM +0200, Jan Kiszka wrote:
>> Hi all,
>>
>> already started the discussion off-list with Pantelis, but it's better
>> done in public:
>>
>> I'm currently exploring ways to make Linux recognize dynamically added
>> virtual hardware when running under the Jailhouse hypervisor [1]. We
>> need to load drivers for inter-partition communication devices that only
>> appear after Jailhouse started (which is done from within Linux, i.e.
>> long after boot) or when a partition was added later on. Probably, we
>> will simply add a virtual PCI host bridge on systems without physical
>> PCI and let the IPC device be explored that way (already works on x86).
>> Still, that leaves us with hotplug and unplug on hypervisor activation
>> and deactivation.
> 
> If I've understood correctly you want to use overlays to inject the
> virtual PCI host bridge?
> 
> Given that you know precisely what you want to inject, I'm not sure I
> see the value of using an overlay. 
> 
> Is there some reason you can't just create a device without having to go
> via an intermediate step? As I understand it, Xen does that for (some)
> virtual devices provided to Dom0 and DomU.

I don't want to write yet another virtual driver but ideally reuse
pci-host-generic. Possibly a dumb question, but bow can I make Linux
instantiate a device node compatible with that driver and with the
desired resources? That's what made me think of overlays.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Pantelis Antoniou]

Hi Mark,

> On Jun 8, 2016, at 18:17 , Mark Rutland <mark.rutland@arm.com> wrote:
> 
> On Wed, Jun 08, 2016 at 04:16:32PM +0200, Jan Kiszka wrote:
>> Hi all,
>> 
>> already started the discussion off-list with Pantelis, but it's better
>> done in public:
>> 
>> I'm currently exploring ways to make Linux recognize dynamically added
>> virtual hardware when running under the Jailhouse hypervisor [1]. We
>> need to load drivers for inter-partition communication devices that only
>> appear after Jailhouse started (which is done from within Linux, i.e.
>> long after boot) or when a partition was added later on. Probably, we
>> will simply add a virtual PCI host bridge on systems without physical
>> PCI and let the IPC device be explored that way (already works on x86).
>> Still, that leaves us with hotplug and unplug on hypervisor activation
>> and deactivation.
> 
> If I've understood correctly you want to use overlays to inject the
> virtual PCI host bridge?
> 
> Given that you know precisely what you want to inject, I'm not sure I
> see the value of using an overlay. 
> 
> Is there some reason you can't just create a device without having to go
> via an intermediate step? As I understand it, Xen does that for (some)
> virtual devices provided to Dom0 and DomU.
> 

As far as I understand it PCI is just one of the cases. You could conceivably
inject any kind of virtio device like serial/storage networking etc.

The question is since overlays exist and do work, why should he do anything else
besides using them? I doubt he’s interested in writing yet another driver for
something that already exists and works.

> Thanks,
> Mark.

Regards

— Pantelis

Re: Using DT overlays for adding virtual hardware

[Mark Rutland]

On Wed, Jun 08, 2016 at 06:57:37PM +0300, Pantelis Antoniou wrote:
> Hi Mark,
> 
> > On Jun 8, 2016, at 18:17 , Mark Rutland <mark.rutland@arm.com> wrote:
> > 
> > On Wed, Jun 08, 2016 at 04:16:32PM +0200, Jan Kiszka wrote:
> >> Hi all,
> >> 
> >> already started the discussion off-list with Pantelis, but it's better
> >> done in public:
> >> 
> >> I'm currently exploring ways to make Linux recognize dynamically added
> >> virtual hardware when running under the Jailhouse hypervisor [1]. We
> >> need to load drivers for inter-partition communication devices that only
> >> appear after Jailhouse started (which is done from within Linux, i.e.
> >> long after boot) or when a partition was added later on. Probably, we
> >> will simply add a virtual PCI host bridge on systems without physical
> >> PCI and let the IPC device be explored that way (already works on x86).
> >> Still, that leaves us with hotplug and unplug on hypervisor activation
> >> and deactivation.
> > 
> > If I've understood correctly you want to use overlays to inject the
> > virtual PCI host bridge?
> > 
> > Given that you know precisely what you want to inject, I'm not sure I
> > see the value of using an overlay. 
> > 
> > Is there some reason you can't just create a device without having to go
> > via an intermediate step? As I understand it, Xen does that for (some)
> > virtual devices provided to Dom0 and DomU.
> 
> As far as I understand it PCI is just one of the cases. You could conceivably
> inject any kind of virtio device like serial/storage networking etc.

Sure, but we already have PCI transport for virtio devices, and per the
above PCI is the transport used on x86, so I assume that the devices we
really care about are going to be PCI anyhow.

> The question is since overlays exist and do work, why should he do anything else
> besides using them?

For one thing, they only work with DT, and there are ACPI ARM server
platforms out there, for which people may wish to use jailhouse. Tying
this to DT is not necessarily the best idea.

To be clear, I'm not arguing *against* overlays as such, just making
sure that we're not prematurely choosing a solution just becasue it's
the one we're aware of.

Thanks,
Mark.

Re: Using DT overlays for adding virtual hardware

[Pantelis Antoniou]

Hi Mark,

> On Jun 8, 2016, at 19:23 , Mark Rutland <mark.rutland@arm.com> wrote:
> 
> On Wed, Jun 08, 2016 at 06:57:37PM +0300, Pantelis Antoniou wrote:
>> Hi Mark,
>> 
>>> On Jun 8, 2016, at 18:17 , Mark Rutland <mark.rutland@arm.com> wrote:
>>> 
>>> On Wed, Jun 08, 2016 at 04:16:32PM +0200, Jan Kiszka wrote:
>>>> Hi all,
>>>> 
>>>> already started the discussion off-list with Pantelis, but it's better
>>>> done in public:
>>>> 
>>>> I'm currently exploring ways to make Linux recognize dynamically added
>>>> virtual hardware when running under the Jailhouse hypervisor [1]. We
>>>> need to load drivers for inter-partition communication devices that only
>>>> appear after Jailhouse started (which is done from within Linux, i.e.
>>>> long after boot) or when a partition was added later on. Probably, we
>>>> will simply add a virtual PCI host bridge on systems without physical
>>>> PCI and let the IPC device be explored that way (already works on x86).
>>>> Still, that leaves us with hotplug and unplug on hypervisor activation
>>>> and deactivation.
>>> 
>>> If I've understood correctly you want to use overlays to inject the
>>> virtual PCI host bridge?
>>> 
>>> Given that you know precisely what you want to inject, I'm not sure I
>>> see the value of using an overlay. 
>>> 
>>> Is there some reason you can't just create a device without having to go
>>> via an intermediate step? As I understand it, Xen does that for (some)
>>> virtual devices provided to Dom0 and DomU.
>> 
>> As far as I understand it PCI is just one of the cases. You could conceivably
>> inject any kind of virtio device like serial/storage networking etc.
> 
> Sure, but we already have PCI transport for virtio devices, and per the
> above PCI is the transport used on x86, so I assume that the devices we
> really care about are going to be PCI anyhow.
> 

PCI on VMs is a hack, it’s all emulated.

We’re using it as crutch because it’s ubiquitous and is capable
of probing, but it comes with a considerable amount of baggage.
Jailhouse is a particular kind of a hypervisor where it is intended for
safety critical applications and designed to be certified as such.
The less amount of code it contains the better, and much easier to certify. 

>> The question is since overlays exist and do work, why should he do anything else
>> besides using them?
> 
> For one thing, they only work with DT, and there are ACPI ARM server
> platforms out there, for which people may wish to use jailhouse. Tying
> this to DT is not necessarily the best idea.
> 

I just don’t see how an ACPI based hypervisor can ever be certified for
safety critical applications. It might be possible but it should be
an enormous undertaking; perhaps a subset without AML, but then again
can you even boot an ACPI box without it?

DT is safer since it contains state only.

> To be clear, I'm not arguing *against* overlays as such, just making
> sure that we're not prematurely choosing a solution just becasue it's
> the one we're aware of.
> 
> Thanks,
> Mark.

Regards

— Pantelis

Re: Using DT overlays for adding virtual hardware

[Jan Kiszka]

On 2016-06-08 18:31, Pantelis Antoniou wrote:
> Hi Mark,
> 
>> On Jun 8, 2016, at 19:23 , Mark Rutland <mark.rutland@arm.com> wrote:
>>
>> On Wed, Jun 08, 2016 at 06:57:37PM +0300, Pantelis Antoniou wrote:
>>> Hi Mark,
>>>
>>>> On Jun 8, 2016, at 18:17 , Mark Rutland <mark.rutland@arm.com> wrote:
>>>>
>>>> On Wed, Jun 08, 2016 at 04:16:32PM +0200, Jan Kiszka wrote:
>>>>> Hi all,
>>>>>
>>>>> already started the discussion off-list with Pantelis, but it's better
>>>>> done in public:
>>>>>
>>>>> I'm currently exploring ways to make Linux recognize dynamically added
>>>>> virtual hardware when running under the Jailhouse hypervisor [1]. We
>>>>> need to load drivers for inter-partition communication devices that only
>>>>> appear after Jailhouse started (which is done from within Linux, i.e.
>>>>> long after boot) or when a partition was added later on. Probably, we
>>>>> will simply add a virtual PCI host bridge on systems without physical
>>>>> PCI and let the IPC device be explored that way (already works on x86).
>>>>> Still, that leaves us with hotplug and unplug on hypervisor activation
>>>>> and deactivation.
>>>>
>>>> If I've understood correctly you want to use overlays to inject the
>>>> virtual PCI host bridge?
>>>>
>>>> Given that you know precisely what you want to inject, I'm not sure I
>>>> see the value of using an overlay. 
>>>>
>>>> Is there some reason you can't just create a device without having to go
>>>> via an intermediate step? As I understand it, Xen does that for (some)
>>>> virtual devices provided to Dom0 and DomU.
>>>
>>> As far as I understand it PCI is just one of the cases. You could conceivably
>>> inject any kind of virtio device like serial/storage networking etc.
>>
>> Sure, but we already have PCI transport for virtio devices, and per the
>> above PCI is the transport used on x86, so I assume that the devices we
>> really care about are going to be PCI anyhow.
>>
> 
> PCI on VMs is a hack, it’s all emulated.
> 
> We’re using it as crutch because it’s ubiquitous and is capable
> of probing, but it comes with a considerable amount of baggage.
> Jailhouse is a particular kind of a hypervisor where it is intended for
> safety critical applications and designed to be certified as such.
> The less amount of code it contains the better, and much easier to certify. 

That's true, but we already have to live with PCI on x86, thus the code
is there, and it's more and more present on ARM[64] as well.

We are trying hard to make it simple, primarily for the hypervisor, and
if it is simpler to plug a trivial virtual PCI bridge than to emulate
individual platform IPC devices - also fine.

> 
>>> The question is since overlays exist and do work, why should he do anything else
>>> besides using them?
>>
>> For one thing, they only work with DT, and there are ACPI ARM server
>> platforms out there, for which people may wish to use jailhouse. Tying
>> this to DT is not necessarily the best idea.
>>
> 
> I just don’t see how an ACPI based hypervisor can ever be certified for
> safety critical applications. It might be possible but it should be
> an enormous undertaking; perhaps a subset without AML, but then again
> can you even boot an ACPI box without it?

ACPI is out of scope for us. We will probably continue to feed the
hypervisor with static platform information, generated in advance and
validated. Can be DT-based one day, but even that is more complex to
parse than our current structures.

But does ACPI usually mean that the kernel no longer has DT support and
would not be able to handle any overlay? That could be a killer.

> 
> DT is safer since it contains state only.
> 
>> To be clear, I'm not arguing *against* overlays as such, just making
>> sure that we're not prematurely choosing a solution just becasue it's
>> the one we're aware of.

I'm open for any suggestion that is simple. Maybe we can extend a
trivial existing pci host driver (like pci-host-generic) to work also
without DT overlays - also fine, at least from Jailhose POV. However,
any unneeded kernel patch is even better.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Jan Kiszka]

On 2016-06-08 18:39, Jan Kiszka wrote:
>>>> The question is since overlays exist and do work, why should he do anything else
>>>> besides using them?
>>>
>>> For one thing, they only work with DT, and there are ACPI ARM server
>>> platforms out there, for which people may wish to use jailhouse. Tying
>>> this to DT is not necessarily the best idea.
>>>
>>
>> I just don’t see how an ACPI based hypervisor can ever be certified for
>> safety critical applications. It might be possible but it should be
>> an enormous undertaking; perhaps a subset without AML, but then again
>> can you even boot an ACPI box without it?
> 
> ACPI is out of scope for us. We will probably continue to feed the
> hypervisor with static platform information, generated in advance and
> validated. Can be DT-based one day, but even that is more complex to
> parse than our current structures.
> 
> But does ACPI usually mean that the kernel no longer has DT support and
> would not be able to handle any overlay? That could be a killer.

However, I suspect that those machines with ACPI will also come with
PCI, in which case we do not need the virtual host bridge anyway.

> 
>>
>> DT is safer since it contains state only.
>>
>>> To be clear, I'm not arguing *against* overlays as such, just making
>>> sure that we're not prematurely choosing a solution just becasue it's
>>> the one we're aware of.
> 
> I'm open for any suggestion that is simple. Maybe we can extend a
> trivial existing pci host driver (like pci-host-generic) to work also
> without DT overlays - also fine, at least from Jailhose POV. However,
> any unneeded kernel patch is even better.

OK, trial and error, and some interesting insights: I've played with DT
fragments and the overlay configfs patch of Pantelis [1] to have a
convenient start. Interestingly, I wasn't able to load a fragment that
followed the format specification for overlays ("Failed to resolve
tree"). By chance, I got this one working:

/dts-v1/;
/ {
	fragment {
		target-path = "/soc@01c00000";
		__overlay__ {
			#address-cells = <2>;
			#size-cells = <2>;

			vpci@0x2000000 {
				compatible = "pci-host-cam-generic";
				device_type = "pci";
				#address-cells = <3>;
				#size-cells = <2>;
				reg = <0 0x2000000 0 0x1000000>;
				ranges =
					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
			};
		};
	};
};

It successfully makes a BananaPi kernel add a pci host with the
specified config space and MMIO window.

[   81.619583] PCI host bridge /soc@01c00000/vpci@0x2000000 ranges:
[   81.619610]   No bus range found for /soc@01c00000/vpci@0x2000000, using [bus 00-ff]
[   81.619634]   MEM 0x10000000..0x3fffffff -> 0x10000000
[   81.620482] pci-host-generic 2000000.vpci: ECAM at [mem 0x02000000-0x02ffffff] for [bus 00-ff]
[   81.620779] pci-host-generic 2000000.vpci: PCI host bridge to bus 0000:00
[   81.620801] pci_bus 0000:00: root bus resource [bus 00-ff]
[   81.620814] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3fffffff]
[   81.620851] PCI: bus0: Fast back to back transfers enabled

So, no /plugin/ statement, no phandles resolution. This format even
builds with the in-kernel dtc. Any explanations? Does the code make
sense (at least it builds without warnings)?

Now I need to back this with some code in Jailhouse.

Jan

[1] https://github.com/pantoniou/linux-beagle-track-mainline/commit/160e68ec89eca33e8ed0abb13d52c07c54d7fc10

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Jan Kiszka]

Hi Pantelis,

coming back to this topic:

On 2016-06-09 08:03, Jan Kiszka wrote:
> OK, trial and error, and some interesting insights: I've played with DT
> fragments and the overlay configfs patch of Pantelis [1] to have a
> convenient start. Interestingly, I wasn't able to load a fragment that
> followed the format specification for overlays ("Failed to resolve
> tree"). By chance, I got this one working:
> 
> /dts-v1/;
> / {
> 	fragment {
> 		target-path = "/soc@01c00000";
> 		__overlay__ {
> 			#address-cells = <2>;
> 			#size-cells = <2>;
> 
> 			vpci@0x2000000 {
> 				compatible = "pci-host-cam-generic";
> 				device_type = "pci";
> 				#address-cells = <3>;
> 				#size-cells = <2>;
> 				reg = <0 0x2000000 0 0x1000000>;
> 				ranges =
> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
> 			};
> 		};
> 	};
> };
> 
> It successfully makes a BananaPi kernel add a pci host with the
> specified config space and MMIO window.
> 
> [   81.619583] PCI host bridge /soc@01c00000/vpci@0x2000000 ranges:
> [   81.619610]   No bus range found for /soc@01c00000/vpci@0x2000000, using [bus 00-ff]
> [   81.619634]   MEM 0x10000000..0x3fffffff -> 0x10000000
> [   81.620482] pci-host-generic 2000000.vpci: ECAM at [mem 0x02000000-0x02ffffff] for [bus 00-ff]
> [   81.620779] pci-host-generic 2000000.vpci: PCI host bridge to bus 0000:00
> [   81.620801] pci_bus 0000:00: root bus resource [bus 00-ff]
> [   81.620814] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3fffffff]
> [   81.620851] PCI: bus0: Fast back to back transfers enabled
> 
> So, no /plugin/ statement, no phandles resolution. This format even
> builds with the in-kernel dtc. Any explanations? Does the code make
> sense (at least it builds without warnings)?
> 
> Now I need to back this with some code in Jailhouse.

Meanwhile I got a virtual PCI device recognized by Linux when running
over Jailhouse. However, my hack above doesn't get me to proper
interrupt mapping yet. This is what I was trying with upstream dtc:

/dts-v1/;
/ {
	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";

	fragment@0 {
		target-path = "/soc@01c00000";
		__overlay__ {
			#address-cells = <2>;
			#size-cells = <2>;

			vpci@2000000 {
				compatible = "pci-host-ecam-generic";
				device_type = "pci";
				bus-range = <0 0>;
				#address-cells = <3>;
				#size-cells = <2>;
				#interrupt-cells = <1>;
				interrupt-map-mask = <0 0 0 7>;
				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
						<0 0 0 2 &gic 0 0 0 124 4>,
						<0 0 0 3 &gic 0 0 0 125 4>,
						<0 0 0 4 &gic 0 0 0 126 4>;
				reg = <0 0x2000000 0 0x100000>;
				ranges =
					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
			};
		};
	};

	gic: fragment@1 {
		target-path = "/soc@01c00000/interrupt-controller@01c81000";
		__overlay__ {
		};
	};
};

And this is what Linux detects on that PCI bus:

00:0f.0 RAM memory: Red Hat, Inc Inter-VM shared memory
        Subsystem: Red Hat, Inc Inter-VM shared memory
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 17
        Region 0: Memory at 10000000 (64-bit, non-prefetchable) [size=256]
        Region 4: Memory at 10000100 (64-bit, non-prefetchable) [size=32]
        Kernel modules: ivshmem_net

However, registering a handler on IRQ 17 fails (-EINVAL).


With your dtc [2] (dtc -@ -I dts -O dtb -o vpci-bpi.dtb vpci-bpi.dts)
and this fragment, I'm still not able to load the blob:

/dts-v1/;
/plugin/;
/ {
	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";

	fragment@0 {
		target-path = "/soc@01c00000";
		__overlay__ {
			#address-cells = <2>;
			#size-cells = <2>;

			vpci@2000000 {
				compatible = "pci-host-ecam-generic";
				device_type = "pci";
				bus-range = <0 0>;
				#address-cells = <3>;
				#size-cells = <2>;
				#interrupt-cells = <1>;
				interrupt-map-mask = <0 0 0 7>;
				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
						<0 0 0 2 &gic 0 0 0 124 4>,
						<0 0 0 3 &gic 0 0 0 125 4>,
						<0 0 0 4 &gic 0 0 0 126 4>;
				reg = <0 0x2000000 0 0x100000>;
				ranges =
					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
			};
		};
	};
};

Any suggestions? Any patches I'm missing in upstream to make that work?

Jan

[1] https://github.com/pantoniou/linux-beagle-track-mainline/commit/160e68ec89eca33e8ed0abb13d52c07c54d7fc10
[2] https://github.com/pantoniou/dtc/tree/dgibson-overlay-panto

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Pantelis Antoniou]

Hi Jan,

> On Jun 21, 2016, at 13:13 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
> 
> Hi Pantelis,
> 
> coming back to this topic:
> 
> On 2016-06-09 08:03, Jan Kiszka wrote:
>> OK, trial and error, and some interesting insights: I've played with DT
>> fragments and the overlay configfs patch of Pantelis [1] to have a
>> convenient start. Interestingly, I wasn't able to load a fragment that
>> followed the format specification for overlays ("Failed to resolve
>> tree"). By chance, I got this one working:
>> 
>> /dts-v1/;
>> / {
>> 	fragment {
>> 		target-path = "/soc@01c00000";
>> 		__overlay__ {
>> 			#address-cells = <2>;
>> 			#size-cells = <2>;
>> 
>> 			vpci@0x2000000 {
>> 				compatible = "pci-host-cam-generic";
>> 				device_type = "pci";
>> 				#address-cells = <3>;
>> 				#size-cells = <2>;
>> 				reg = <0 0x2000000 0 0x1000000>;
>> 				ranges =
>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>> 			};
>> 		};
>> 	};
>> };
>> 
>> It successfully makes a BananaPi kernel add a pci host with the
>> specified config space and MMIO window.
>> 
>> [   81.619583] PCI host bridge /soc@01c00000/vpci@0x2000000 ranges:
>> [   81.619610]   No bus range found for /soc@01c00000/vpci@0x2000000, using [bus 00-ff]
>> [   81.619634]   MEM 0x10000000..0x3fffffff -> 0x10000000
>> [   81.620482] pci-host-generic 2000000.vpci: ECAM at [mem 0x02000000-0x02ffffff] for [bus 00-ff]
>> [   81.620779] pci-host-generic 2000000.vpci: PCI host bridge to bus 0000:00
>> [   81.620801] pci_bus 0000:00: root bus resource [bus 00-ff]
>> [   81.620814] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3fffffff]
>> [   81.620851] PCI: bus0: Fast back to back transfers enabled
>> 
>> So, no /plugin/ statement, no phandles resolution. This format even
>> builds with the in-kernel dtc. Any explanations? Does the code make
>> sense (at least it builds without warnings)?
>> 
>> Now I need to back this with some code in Jailhouse.
> 
> Meanwhile I got a virtual PCI device recognized by Linux when running
> over Jailhouse. However, my hack above doesn't get me to proper
> interrupt mapping yet. This is what I was trying with upstream dtc:
> 
> /dts-v1/;
> / {
> 	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
> 
> 	fragment@0 {
> 		target-path = "/soc@01c00000";
> 		__overlay__ {
> 			#address-cells = <2>;
> 			#size-cells = <2>;
> 
> 			vpci@2000000 {
> 				compatible = "pci-host-ecam-generic";
> 				device_type = "pci";
> 				bus-range = <0 0>;
> 				#address-cells = <3>;
> 				#size-cells = <2>;
> 				#interrupt-cells = <1>;
> 				interrupt-map-mask = <0 0 0 7>;
> 				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
> 						<0 0 0 2 &gic 0 0 0 124 4>,
> 						<0 0 0 3 &gic 0 0 0 125 4>,
> 						<0 0 0 4 &gic 0 0 0 126 4>;
> 				reg = <0 0x2000000 0 0x100000>;
> 				ranges =
> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
> 			};
> 		};
> 	};
> 
> 	gic: fragment@1 {
> 		target-path = "/soc@01c00000/interrupt-controller@01c81000";
> 		__overlay__ {
> 		};
> 	};
> };
> 

^ This is not going to work: You need the reference to the real gic not the empty fragment
here that has a target there.

You need to compile with the correct dtc, and you also need to compile the base dts
with dtc too, using the -@ flag. You can hack around it by adding something like

__symbols__ {
	gic = "/soc@01c00000/interrupt-controller@01c81000”;
};

But you really need the __symbols__ node of the base dts generated by the dtc proper cause
the above is a dirty hack.


> And this is what Linux detects on that PCI bus:
> 
> 00:0f.0 RAM memory: Red Hat, Inc Inter-VM shared memory
>        Subsystem: Red Hat, Inc Inter-VM shared memory
>        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>        Interrupt: pin A routed to IRQ 17
>        Region 0: Memory at 10000000 (64-bit, non-prefetchable) [size=256]
>        Region 4: Memory at 10000100 (64-bit, non-prefetchable) [size=32]
>        Kernel modules: ivshmem_net
> 
> However, registering a handler on IRQ 17 fails (-EINVAL).
> 
> 
> With your dtc [2] (dtc -@ -I dts -O dtb -o vpci-bpi.dtb vpci-bpi.dts)
> and this fragment, I'm still not able to load the blob:
> 
> /dts-v1/;
> /plugin/;
> / {
> 	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
> 
> 	fragment@0 {
> 		target-path = "/soc@01c00000";
> 		__overlay__ {
> 			#address-cells = <2>;
> 			#size-cells = <2>;
> 
> 			vpci@2000000 {
> 				compatible = "pci-host-ecam-generic";
> 				device_type = "pci";
> 				bus-range = <0 0>;
> 				#address-cells = <3>;
> 				#size-cells = <2>;
> 				#interrupt-cells = <1>;
> 				interrupt-map-mask = <0 0 0 7>;
> 				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
> 						<0 0 0 2 &gic 0 0 0 124 4>,
> 						<0 0 0 3 &gic 0 0 0 125 4>,
> 						<0 0 0 4 &gic 0 0 0 126 4>;
> 				reg = <0 0x2000000 0 0x100000>;
> 				ranges =
> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
> 			};
> 		};
> 	};
> };
> 

Please try this branch of dtc; it’s a wip of what I’m trying to push to upstream.

https://github.com/pantoniou/dtc/tree/dgibson-overlay-panto

> Any suggestions? Any patches I'm missing in upstream to make that work?
> 
> Jan
> 

Regards

— Pantelis

> [1] https://github.com/pantoniou/linux-beagle-track-mainline/commit/160e68ec89eca33e8ed0abb13d52c07c54d7fc10
> [2] https://github.com/pantoniou/dtc/tree/dgibson-overlay-panto
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA ITP SES-DE
> Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Jan Kiszka]

On 2016-06-21 12:24, Pantelis Antoniou wrote:
> Hi Jan,
> 
>> On Jun 21, 2016, at 13:13 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>
>> Hi Pantelis,
>>
>> coming back to this topic:
>>
>> On 2016-06-09 08:03, Jan Kiszka wrote:
>>> OK, trial and error, and some interesting insights: I've played with DT
>>> fragments and the overlay configfs patch of Pantelis [1] to have a
>>> convenient start. Interestingly, I wasn't able to load a fragment that
>>> followed the format specification for overlays ("Failed to resolve
>>> tree"). By chance, I got this one working:
>>>
>>> /dts-v1/;
>>> / {
>>> 	fragment {
>>> 		target-path = "/soc@01c00000";
>>> 		__overlay__ {
>>> 			#address-cells = <2>;
>>> 			#size-cells = <2>;
>>>
>>> 			vpci@0x2000000 {
>>> 				compatible = "pci-host-cam-generic";
>>> 				device_type = "pci";
>>> 				#address-cells = <3>;
>>> 				#size-cells = <2>;
>>> 				reg = <0 0x2000000 0 0x1000000>;
>>> 				ranges =
>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>> 			};
>>> 		};
>>> 	};
>>> };
>>>
>>> It successfully makes a BananaPi kernel add a pci host with the
>>> specified config space and MMIO window.
>>>
>>> [   81.619583] PCI host bridge /soc@01c00000/vpci@0x2000000 ranges:
>>> [   81.619610]   No bus range found for /soc@01c00000/vpci@0x2000000, using [bus 00-ff]
>>> [   81.619634]   MEM 0x10000000..0x3fffffff -> 0x10000000
>>> [   81.620482] pci-host-generic 2000000.vpci: ECAM at [mem 0x02000000-0x02ffffff] for [bus 00-ff]
>>> [   81.620779] pci-host-generic 2000000.vpci: PCI host bridge to bus 0000:00
>>> [   81.620801] pci_bus 0000:00: root bus resource [bus 00-ff]
>>> [   81.620814] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3fffffff]
>>> [   81.620851] PCI: bus0: Fast back to back transfers enabled
>>>
>>> So, no /plugin/ statement, no phandles resolution. This format even
>>> builds with the in-kernel dtc. Any explanations? Does the code make
>>> sense (at least it builds without warnings)?
>>>
>>> Now I need to back this with some code in Jailhouse.
>>
>> Meanwhile I got a virtual PCI device recognized by Linux when running
>> over Jailhouse. However, my hack above doesn't get me to proper
>> interrupt mapping yet. This is what I was trying with upstream dtc:
>>
>> /dts-v1/;
>> / {
>> 	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>>
>> 	fragment@0 {
>> 		target-path = "/soc@01c00000";
>> 		__overlay__ {
>> 			#address-cells = <2>;
>> 			#size-cells = <2>;
>>
>> 			vpci@2000000 {
>> 				compatible = "pci-host-ecam-generic";
>> 				device_type = "pci";
>> 				bus-range = <0 0>;
>> 				#address-cells = <3>;
>> 				#size-cells = <2>;
>> 				#interrupt-cells = <1>;
>> 				interrupt-map-mask = <0 0 0 7>;
>> 				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
>> 						<0 0 0 2 &gic 0 0 0 124 4>,
>> 						<0 0 0 3 &gic 0 0 0 125 4>,
>> 						<0 0 0 4 &gic 0 0 0 126 4>;
>> 				reg = <0 0x2000000 0 0x100000>;
>> 				ranges =
>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>> 			};
>> 		};
>> 	};
>>
>> 	gic: fragment@1 {
>> 		target-path = "/soc@01c00000/interrupt-controller@01c81000";
>> 		__overlay__ {
>> 		};
>> 	};
>> };
>>
> 
> ^ This is not going to work: You need the reference to the real gic not the empty fragment
> here that has a target there.
> 
> You need to compile with the correct dtc, and you also need to compile the base dts
> with dtc too, using the -@ flag. You can hack around it by adding something like
> 
> __symbols__ {
> 	gic = "/soc@01c00000/interrupt-controller@01c81000”;
> };
> 
> But you really need the __symbols__ node of the base dts generated by the dtc proper cause
> the above is a dirty hack.
> 

OK, re-building the kernel with DTC="/your/dtc -@", thus building the
base dtb with symbols, fixes proper overlay format loading.

However, no luck yet with the interrupt topic - maybe a different issue.
Digging deeper...

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Pantelis Antoniou]

Hi Jan,

> On Jun 21, 2016, at 14:22 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
> 
> On 2016-06-21 12:24, Pantelis Antoniou wrote:
>> Hi Jan,
>> 
>>> On Jun 21, 2016, at 13:13 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>> 
>>> Hi Pantelis,
>>> 
>>> coming back to this topic:
>>> 
>>> On 2016-06-09 08:03, Jan Kiszka wrote:
>>>> OK, trial and error, and some interesting insights: I've played with DT
>>>> fragments and the overlay configfs patch of Pantelis [1] to have a
>>>> convenient start. Interestingly, I wasn't able to load a fragment that
>>>> followed the format specification for overlays ("Failed to resolve
>>>> tree"). By chance, I got this one working:
>>>> 
>>>> /dts-v1/;
>>>> / {
>>>> 	fragment {
>>>> 		target-path = "/soc@01c00000";
>>>> 		__overlay__ {
>>>> 			#address-cells = <2>;
>>>> 			#size-cells = <2>;
>>>> 
>>>> 			vpci@0x2000000 {
>>>> 				compatible = "pci-host-cam-generic";
>>>> 				device_type = "pci";
>>>> 				#address-cells = <3>;
>>>> 				#size-cells = <2>;
>>>> 				reg = <0 0x2000000 0 0x1000000>;
>>>> 				ranges =
>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>> 			};
>>>> 		};
>>>> 	};
>>>> };
>>>> 
>>>> It successfully makes a BananaPi kernel add a pci host with the
>>>> specified config space and MMIO window.
>>>> 
>>>> [   81.619583] PCI host bridge /soc@01c00000/vpci@0x2000000 ranges:
>>>> [   81.619610]   No bus range found for /soc@01c00000/vpci@0x2000000, using [bus 00-ff]
>>>> [   81.619634]   MEM 0x10000000..0x3fffffff -> 0x10000000
>>>> [   81.620482] pci-host-generic 2000000.vpci: ECAM at [mem 0x02000000-0x02ffffff] for [bus 00-ff]
>>>> [   81.620779] pci-host-generic 2000000.vpci: PCI host bridge to bus 0000:00
>>>> [   81.620801] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>> [   81.620814] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3fffffff]
>>>> [   81.620851] PCI: bus0: Fast back to back transfers enabled
>>>> 
>>>> So, no /plugin/ statement, no phandles resolution. This format even
>>>> builds with the in-kernel dtc. Any explanations? Does the code make
>>>> sense (at least it builds without warnings)?
>>>> 
>>>> Now I need to back this with some code in Jailhouse.
>>> 
>>> Meanwhile I got a virtual PCI device recognized by Linux when running
>>> over Jailhouse. However, my hack above doesn't get me to proper
>>> interrupt mapping yet. This is what I was trying with upstream dtc:
>>> 
>>> /dts-v1/;
>>> / {
>>> 	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>>> 
>>> 	fragment@0 {
>>> 		target-path = "/soc@01c00000";
>>> 		__overlay__ {
>>> 			#address-cells = <2>;
>>> 			#size-cells = <2>;
>>> 
>>> 			vpci@2000000 {
>>> 				compatible = "pci-host-ecam-generic";
>>> 				device_type = "pci";
>>> 				bus-range = <0 0>;
>>> 				#address-cells = <3>;
>>> 				#size-cells = <2>;
>>> 				#interrupt-cells = <1>;
>>> 				interrupt-map-mask = <0 0 0 7>;
>>> 				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
>>> 						<0 0 0 2 &gic 0 0 0 124 4>,
>>> 						<0 0 0 3 &gic 0 0 0 125 4>,
>>> 						<0 0 0 4 &gic 0 0 0 126 4>;
>>> 				reg = <0 0x2000000 0 0x100000>;
>>> 				ranges =
>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>> 			};
>>> 		};
>>> 	};
>>> 
>>> 	gic: fragment@1 {
>>> 		target-path = "/soc@01c00000/interrupt-controller@01c81000";
>>> 		__overlay__ {
>>> 		};
>>> 	};
>>> };
>>> 
>> 
>> ^ This is not going to work: You need the reference to the real gic not the empty fragment
>> here that has a target there.
>> 
>> You need to compile with the correct dtc, and you also need to compile the base dts
>> with dtc too, using the -@ flag. You can hack around it by adding something like
>> 
>> __symbols__ {
>> 	gic = "/soc@01c00000/interrupt-controller@01c81000”;
>> };
>> 
>> But you really need the __symbols__ node of the base dts generated by the dtc proper cause
>> the above is a dirty hack.
>> 
> 
> OK, re-building the kernel with DTC="/your/dtc -@", thus building the
> base dtb with symbols, fixes proper overlay format loading.
> 
> However, no luck yet with the interrupt topic - maybe a different issue.
> Digging deeper…
> 
Remove the gic: fragment and build both the kernel and the overlay with the -@ option.
That’s what makes it not to work.


> Thanks,
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA ITP SES-DE
> Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Jan Kiszka]

On 2016-06-21 13:35, Pantelis Antoniou wrote:
> Hi Jan,
> 
>> On Jun 21, 2016, at 14:22 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>
>> On 2016-06-21 12:24, Pantelis Antoniou wrote:
>>> Hi Jan,
>>>
>>>> On Jun 21, 2016, at 13:13 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>>
>>>> Hi Pantelis,
>>>>
>>>> coming back to this topic:
>>>>
>>>> On 2016-06-09 08:03, Jan Kiszka wrote:
>>>>> OK, trial and error, and some interesting insights: I've played with DT
>>>>> fragments and the overlay configfs patch of Pantelis [1] to have a
>>>>> convenient start. Interestingly, I wasn't able to load a fragment that
>>>>> followed the format specification for overlays ("Failed to resolve
>>>>> tree"). By chance, I got this one working:
>>>>>
>>>>> /dts-v1/;
>>>>> / {
>>>>> 	fragment {
>>>>> 		target-path = "/soc@01c00000";
>>>>> 		__overlay__ {
>>>>> 			#address-cells = <2>;
>>>>> 			#size-cells = <2>;
>>>>>
>>>>> 			vpci@0x2000000 {
>>>>> 				compatible = "pci-host-cam-generic";
>>>>> 				device_type = "pci";
>>>>> 				#address-cells = <3>;
>>>>> 				#size-cells = <2>;
>>>>> 				reg = <0 0x2000000 0 0x1000000>;
>>>>> 				ranges =
>>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>>> 			};
>>>>> 		};
>>>>> 	};
>>>>> };
>>>>>
>>>>> It successfully makes a BananaPi kernel add a pci host with the
>>>>> specified config space and MMIO window.
>>>>>
>>>>> [   81.619583] PCI host bridge /soc@01c00000/vpci@0x2000000 ranges:
>>>>> [   81.619610]   No bus range found for /soc@01c00000/vpci@0x2000000, using [bus 00-ff]
>>>>> [   81.619634]   MEM 0x10000000..0x3fffffff -> 0x10000000
>>>>> [   81.620482] pci-host-generic 2000000.vpci: ECAM at [mem 0x02000000-0x02ffffff] for [bus 00-ff]
>>>>> [   81.620779] pci-host-generic 2000000.vpci: PCI host bridge to bus 0000:00
>>>>> [   81.620801] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>>> [   81.620814] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3fffffff]
>>>>> [   81.620851] PCI: bus0: Fast back to back transfers enabled
>>>>>
>>>>> So, no /plugin/ statement, no phandles resolution. This format even
>>>>> builds with the in-kernel dtc. Any explanations? Does the code make
>>>>> sense (at least it builds without warnings)?
>>>>>
>>>>> Now I need to back this with some code in Jailhouse.
>>>>
>>>> Meanwhile I got a virtual PCI device recognized by Linux when running
>>>> over Jailhouse. However, my hack above doesn't get me to proper
>>>> interrupt mapping yet. This is what I was trying with upstream dtc:
>>>>
>>>> /dts-v1/;
>>>> / {
>>>> 	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>>>>
>>>> 	fragment@0 {
>>>> 		target-path = "/soc@01c00000";
>>>> 		__overlay__ {
>>>> 			#address-cells = <2>;
>>>> 			#size-cells = <2>;
>>>>
>>>> 			vpci@2000000 {
>>>> 				compatible = "pci-host-ecam-generic";
>>>> 				device_type = "pci";
>>>> 				bus-range = <0 0>;
>>>> 				#address-cells = <3>;
>>>> 				#size-cells = <2>;
>>>> 				#interrupt-cells = <1>;
>>>> 				interrupt-map-mask = <0 0 0 7>;
>>>> 				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
>>>> 						<0 0 0 2 &gic 0 0 0 124 4>,
>>>> 						<0 0 0 3 &gic 0 0 0 125 4>,
>>>> 						<0 0 0 4 &gic 0 0 0 126 4>;
>>>> 				reg = <0 0x2000000 0 0x100000>;
>>>> 				ranges =
>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>> 			};
>>>> 		};
>>>> 	};
>>>>
>>>> 	gic: fragment@1 {
>>>> 		target-path = "/soc@01c00000/interrupt-controller@01c81000";
>>>> 		__overlay__ {
>>>> 		};
>>>> 	};
>>>> };
>>>>
>>>
>>> ^ This is not going to work: You need the reference to the real gic not the empty fragment
>>> here that has a target there.
>>>
>>> You need to compile with the correct dtc, and you also need to compile the base dts
>>> with dtc too, using the -@ flag. You can hack around it by adding something like
>>>
>>> __symbols__ {
>>> 	gic = "/soc@01c00000/interrupt-controller@01c81000”;
>>> };
>>>
>>> But you really need the __symbols__ node of the base dts generated by the dtc proper cause
>>> the above is a dirty hack.
>>>
>>
>> OK, re-building the kernel with DTC="/your/dtc -@", thus building the
>> base dtb with symbols, fixes proper overlay format loading.
>>
>> However, no luck yet with the interrupt topic - maybe a different issue.
>> Digging deeper…
>>
> Remove the gic: fragment and build both the kernel and the overlay with the -@ option.
> That’s what makes it not to work.

That's what I did, of course.

I've now

/dts-v1/;
// magic:		0xd00dfeed
// totalsize:		0x448 (1096)
// off_dt_struct:	0x38
// off_dt_strings:	0x3c0
// off_mem_rsvmap:	0x28
// version:		17
// last_comp_version:	16
// boot_cpuid_phys:	0x0
// size_dt_strings:	0x88
// size_dt_struct:	0x388

/ {
    compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
    fragment@0 {
        target = <0xffffffff>;
        __overlay__ {
            #address-cells = <0x00000002>;
            #size-cells = <0x00000002>;
            vpci@2000000 {
                compatible = "pci-host-ecam-generic";
                device_type = "pci";
                bus-range = <0x00000000 0x00000000>;
                #address-cells = <0x00000003>;
                #size-cells = <0x00000002>;
                #interrupt-cells = <0x00000001>;
                interrupt-map-mask = <0x00000000 0x00000000 0x00000000 0x00000007>;
                interrupt-map = <0x00000000 0x00000000 0x00000000 0x00000001 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007b 0x00000004 0x00000000 0x00000000 0x00000000 0x00000002 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007c 0x00000004 0x00000000 0x00000000 0x00000000 0x00000003 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007d 0x00000004 0x00000000 0x00000000 0x00000000 0x00000004 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007e 0x00000004>;
                reg = <0x00000000 0x02000000 0x00000000 0x00100000>;
                ranges = <0x02000000 0x00000000 0x10000000 0x00000000 0x10000000 0x00000000 0x30000000>;
            };
        };
    };
    __symbols__ {
    };
    __fixups__ {
        soc = "/fragment@0:target:0";
        gic = "/fragment@0/__overlay__/vpci@2000000:interrupt-map:16", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:56", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:96", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:136";
    };
    __local_fixups__ {
    };
};

and a base dtb with a symbols section and all the required ones
included.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Pantelis Antoniou]

Hi Jan,

> On Jun 21, 2016, at 14:43 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
> 
> On 2016-06-21 13:35, Pantelis Antoniou wrote:
>> Hi Jan,
>> 
>>> On Jun 21, 2016, at 14:22 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>> 
>>> On 2016-06-21 12:24, Pantelis Antoniou wrote:
>>>> Hi Jan,
>>>> 
>>>>> On Jun 21, 2016, at 13:13 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>>> 
>>>>> Hi Pantelis,
>>>>> 
>>>>> coming back to this topic:
>>>>> 
>>>>> On 2016-06-09 08:03, Jan Kiszka wrote:
>>>>>> OK, trial and error, and some interesting insights: I've played with DT
>>>>>> fragments and the overlay configfs patch of Pantelis [1] to have a
>>>>>> convenient start. Interestingly, I wasn't able to load a fragment that
>>>>>> followed the format specification for overlays ("Failed to resolve
>>>>>> tree"). By chance, I got this one working:
>>>>>> 
>>>>>> /dts-v1/;
>>>>>> / {
>>>>>> 	fragment {
>>>>>> 		target-path = "/soc@01c00000";
>>>>>> 		__overlay__ {
>>>>>> 			#address-cells = <2>;
>>>>>> 			#size-cells = <2>;
>>>>>> 
>>>>>> 			vpci@0x2000000 {
>>>>>> 				compatible = "pci-host-cam-generic";
>>>>>> 				device_type = "pci";
>>>>>> 				#address-cells = <3>;
>>>>>> 				#size-cells = <2>;
>>>>>> 				reg = <0 0x2000000 0 0x1000000>;
>>>>>> 				ranges =
>>>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>>>> 			};
>>>>>> 		};
>>>>>> 	};
>>>>>> };
>>>>>> 
>>>>>> It successfully makes a BananaPi kernel add a pci host with the
>>>>>> specified config space and MMIO window.
>>>>>> 
>>>>>> [   81.619583] PCI host bridge /soc@01c00000/vpci@0x2000000 ranges:
>>>>>> [   81.619610]   No bus range found for /soc@01c00000/vpci@0x2000000, using [bus 00-ff]
>>>>>> [   81.619634]   MEM 0x10000000..0x3fffffff -> 0x10000000
>>>>>> [   81.620482] pci-host-generic 2000000.vpci: ECAM at [mem 0x02000000-0x02ffffff] for [bus 00-ff]
>>>>>> [   81.620779] pci-host-generic 2000000.vpci: PCI host bridge to bus 0000:00
>>>>>> [   81.620801] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>>>> [   81.620814] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3fffffff]
>>>>>> [   81.620851] PCI: bus0: Fast back to back transfers enabled
>>>>>> 
>>>>>> So, no /plugin/ statement, no phandles resolution. This format even
>>>>>> builds with the in-kernel dtc. Any explanations? Does the code make
>>>>>> sense (at least it builds without warnings)?
>>>>>> 
>>>>>> Now I need to back this with some code in Jailhouse.
>>>>> 
>>>>> Meanwhile I got a virtual PCI device recognized by Linux when running
>>>>> over Jailhouse. However, my hack above doesn't get me to proper
>>>>> interrupt mapping yet. This is what I was trying with upstream dtc:
>>>>> 
>>>>> /dts-v1/;
>>>>> / {
>>>>> 	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>>>>> 
>>>>> 	fragment@0 {
>>>>> 		target-path = "/soc@01c00000";
>>>>> 		__overlay__ {
>>>>> 			#address-cells = <2>;
>>>>> 			#size-cells = <2>;
>>>>> 
>>>>> 			vpci@2000000 {
>>>>> 				compatible = "pci-host-ecam-generic";
>>>>> 				device_type = "pci";
>>>>> 				bus-range = <0 0>;
>>>>> 				#address-cells = <3>;
>>>>> 				#size-cells = <2>;
>>>>> 				#interrupt-cells = <1>;
>>>>> 				interrupt-map-mask = <0 0 0 7>;
>>>>> 				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
>>>>> 						<0 0 0 2 &gic 0 0 0 124 4>,
>>>>> 						<0 0 0 3 &gic 0 0 0 125 4>,
>>>>> 						<0 0 0 4 &gic 0 0 0 126 4>;
>>>>> 				reg = <0 0x2000000 0 0x100000>;
>>>>> 				ranges =
>>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>>> 			};
>>>>> 		};
>>>>> 	};
>>>>> 
>>>>> 	gic: fragment@1 {
>>>>> 		target-path = "/soc@01c00000/interrupt-controller@01c81000";
>>>>> 		__overlay__ {
>>>>> 		};
>>>>> 	};
>>>>> };
>>>>> 
>>>> 
>>>> ^ This is not going to work: You need the reference to the real gic not the empty fragment
>>>> here that has a target there.
>>>> 
>>>> You need to compile with the correct dtc, and you also need to compile the base dts
>>>> with dtc too, using the -@ flag. You can hack around it by adding something like
>>>> 
>>>> __symbols__ {
>>>> 	gic = "/soc@01c00000/interrupt-controller@01c81000”;
>>>> };
>>>> 
>>>> But you really need the __symbols__ node of the base dts generated by the dtc proper cause
>>>> the above is a dirty hack.
>>>> 
>>> 
>>> OK, re-building the kernel with DTC="/your/dtc -@", thus building the
>>> base dtb with symbols, fixes proper overlay format loading.
>>> 
>>> However, no luck yet with the interrupt topic - maybe a different issue.
>>> Digging deeper…
>>> 
>> Remove the gic: fragment and build both the kernel and the overlay with the -@ option.
>> That’s what makes it not to work.
> 
> That's what I did, of course.
> 
> I've now
> 
> /dts-v1/;
> // magic:		0xd00dfeed
> // totalsize:		0x448 (1096)
> // off_dt_struct:	0x38
> // off_dt_strings:	0x3c0
> // off_mem_rsvmap:	0x28
> // version:		17
> // last_comp_version:	16
> // boot_cpuid_phys:	0x0
> // size_dt_strings:	0x88
> // size_dt_struct:	0x388
> 
> / {
>    compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>    fragment@0 {
>        target = <0xffffffff>;
>        __overlay__ {
>            #address-cells = <0x00000002>;
>            #size-cells = <0x00000002>;
>            vpci@2000000 {
>                compatible = "pci-host-ecam-generic";
>                device_type = "pci";
>                bus-range = <0x00000000 0x00000000>;
>                #address-cells = <0x00000003>;
>                #size-cells = <0x00000002>;
>                #interrupt-cells = <0x00000001>;
>                interrupt-map-mask = <0x00000000 0x00000000 0x00000000 0x00000007>;
>                interrupt-map = <0x00000000 0x00000000 0x00000000 0x00000001 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007b 0x00000004 0x00000000 0x00000000 0x00000000 0x00000002 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007c 0x00000004 0x00000000 0x00000000 0x00000000 0x00000003 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007d 0x00000004 0x00000000 0x00000000 0x00000000 0x00000004 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007e 0x00000004>;
>                reg = <0x00000000 0x02000000 0x00000000 0x00100000>;
>                ranges = <0x02000000 0x00000000 0x10000000 0x00000000 0x10000000 0x00000000 0x30000000>;
>            };
>        };
>    };
>    __symbols__ {
>    };
>    __fixups__ {
>        soc = "/fragment@0:target:0";
>        gic = "/fragment@0/__overlay__/vpci@2000000:interrupt-map:16", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:56", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:96", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:136";
>    };
>    __local_fixups__ {
>    };
> };
> 
> and a base dtb with a symbols section and all the required ones
> included.
> 

OK, this should make things work. Did you try sticking this in the base dts and see if it works there?


> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA ITP SES-DE
> Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Jan Kiszka]

On 2016-06-21 13:45, Pantelis Antoniou wrote:
> Hi Jan,
> 
>> On Jun 21, 2016, at 14:43 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>
>> On 2016-06-21 13:35, Pantelis Antoniou wrote:
>>> Hi Jan,
>>>
>>>> On Jun 21, 2016, at 14:22 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>>
>>>> On 2016-06-21 12:24, Pantelis Antoniou wrote:
>>>>> Hi Jan,
>>>>>
>>>>>> On Jun 21, 2016, at 13:13 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>>>>
>>>>>> Hi Pantelis,
>>>>>>
>>>>>> coming back to this topic:
>>>>>>
>>>>>> On 2016-06-09 08:03, Jan Kiszka wrote:
>>>>>>> OK, trial and error, and some interesting insights: I've played with DT
>>>>>>> fragments and the overlay configfs patch of Pantelis [1] to have a
>>>>>>> convenient start. Interestingly, I wasn't able to load a fragment that
>>>>>>> followed the format specification for overlays ("Failed to resolve
>>>>>>> tree"). By chance, I got this one working:
>>>>>>>
>>>>>>> /dts-v1/;
>>>>>>> / {
>>>>>>> 	fragment {
>>>>>>> 		target-path = "/soc@01c00000";
>>>>>>> 		__overlay__ {
>>>>>>> 			#address-cells = <2>;
>>>>>>> 			#size-cells = <2>;
>>>>>>>
>>>>>>> 			vpci@0x2000000 {
>>>>>>> 				compatible = "pci-host-cam-generic";
>>>>>>> 				device_type = "pci";
>>>>>>> 				#address-cells = <3>;
>>>>>>> 				#size-cells = <2>;
>>>>>>> 				reg = <0 0x2000000 0 0x1000000>;
>>>>>>> 				ranges =
>>>>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>>>>> 			};
>>>>>>> 		};
>>>>>>> 	};
>>>>>>> };
>>>>>>>
>>>>>>> It successfully makes a BananaPi kernel add a pci host with the
>>>>>>> specified config space and MMIO window.
>>>>>>>
>>>>>>> [   81.619583] PCI host bridge /soc@01c00000/vpci@0x2000000 ranges:
>>>>>>> [   81.619610]   No bus range found for /soc@01c00000/vpci@0x2000000, using [bus 00-ff]
>>>>>>> [   81.619634]   MEM 0x10000000..0x3fffffff -> 0x10000000
>>>>>>> [   81.620482] pci-host-generic 2000000.vpci: ECAM at [mem 0x02000000-0x02ffffff] for [bus 00-ff]
>>>>>>> [   81.620779] pci-host-generic 2000000.vpci: PCI host bridge to bus 0000:00
>>>>>>> [   81.620801] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>>>>> [   81.620814] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3fffffff]
>>>>>>> [   81.620851] PCI: bus0: Fast back to back transfers enabled
>>>>>>>
>>>>>>> So, no /plugin/ statement, no phandles resolution. This format even
>>>>>>> builds with the in-kernel dtc. Any explanations? Does the code make
>>>>>>> sense (at least it builds without warnings)?
>>>>>>>
>>>>>>> Now I need to back this with some code in Jailhouse.
>>>>>>
>>>>>> Meanwhile I got a virtual PCI device recognized by Linux when running
>>>>>> over Jailhouse. However, my hack above doesn't get me to proper
>>>>>> interrupt mapping yet. This is what I was trying with upstream dtc:
>>>>>>
>>>>>> /dts-v1/;
>>>>>> / {
>>>>>> 	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>>>>>>
>>>>>> 	fragment@0 {
>>>>>> 		target-path = "/soc@01c00000";
>>>>>> 		__overlay__ {
>>>>>> 			#address-cells = <2>;
>>>>>> 			#size-cells = <2>;
>>>>>>
>>>>>> 			vpci@2000000 {
>>>>>> 				compatible = "pci-host-ecam-generic";
>>>>>> 				device_type = "pci";
>>>>>> 				bus-range = <0 0>;
>>>>>> 				#address-cells = <3>;
>>>>>> 				#size-cells = <2>;
>>>>>> 				#interrupt-cells = <1>;
>>>>>> 				interrupt-map-mask = <0 0 0 7>;
>>>>>> 				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
>>>>>> 						<0 0 0 2 &gic 0 0 0 124 4>,
>>>>>> 						<0 0 0 3 &gic 0 0 0 125 4>,
>>>>>> 						<0 0 0 4 &gic 0 0 0 126 4>;
>>>>>> 				reg = <0 0x2000000 0 0x100000>;
>>>>>> 				ranges =
>>>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>>>> 			};
>>>>>> 		};
>>>>>> 	};
>>>>>>
>>>>>> 	gic: fragment@1 {
>>>>>> 		target-path = "/soc@01c00000/interrupt-controller@01c81000";
>>>>>> 		__overlay__ {
>>>>>> 		};
>>>>>> 	};
>>>>>> };
>>>>>>
>>>>>
>>>>> ^ This is not going to work: You need the reference to the real gic not the empty fragment
>>>>> here that has a target there.
>>>>>
>>>>> You need to compile with the correct dtc, and you also need to compile the base dts
>>>>> with dtc too, using the -@ flag. You can hack around it by adding something like
>>>>>
>>>>> __symbols__ {
>>>>> 	gic = "/soc@01c00000/interrupt-controller@01c81000”;
>>>>> };
>>>>>
>>>>> But you really need the __symbols__ node of the base dts generated by the dtc proper cause
>>>>> the above is a dirty hack.
>>>>>
>>>>
>>>> OK, re-building the kernel with DTC="/your/dtc -@", thus building the
>>>> base dtb with symbols, fixes proper overlay format loading.
>>>>
>>>> However, no luck yet with the interrupt topic - maybe a different issue.
>>>> Digging deeper…
>>>>
>>> Remove the gic: fragment and build both the kernel and the overlay with the -@ option.
>>> That’s what makes it not to work.
>>
>> That's what I did, of course.
>>
>> I've now
>>
>> /dts-v1/;
>> // magic:		0xd00dfeed
>> // totalsize:		0x448 (1096)
>> // off_dt_struct:	0x38
>> // off_dt_strings:	0x3c0
>> // off_mem_rsvmap:	0x28
>> // version:		17
>> // last_comp_version:	16
>> // boot_cpuid_phys:	0x0
>> // size_dt_strings:	0x88
>> // size_dt_struct:	0x388
>>
>> / {
>>    compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>>    fragment@0 {
>>        target = <0xffffffff>;
>>        __overlay__ {
>>            #address-cells = <0x00000002>;
>>            #size-cells = <0x00000002>;
>>            vpci@2000000 {
>>                compatible = "pci-host-ecam-generic";
>>                device_type = "pci";
>>                bus-range = <0x00000000 0x00000000>;
>>                #address-cells = <0x00000003>;
>>                #size-cells = <0x00000002>;
>>                #interrupt-cells = <0x00000001>;
>>                interrupt-map-mask = <0x00000000 0x00000000 0x00000000 0x00000007>;
>>                interrupt-map = <0x00000000 0x00000000 0x00000000 0x00000001 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007b 0x00000004 0x00000000 0x00000000 0x00000000 0x00000002 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007c 0x00000004 0x00000000 0x00000000 0x00000000 0x00000003 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007d 0x00000004 0x00000000 0x00000000 0x00000000 0x00000004 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007e 0x00000004>;
>>                reg = <0x00000000 0x02000000 0x00000000 0x00100000>;
>>                ranges = <0x02000000 0x00000000 0x10000000 0x00000000 0x10000000 0x00000000 0x30000000>;
>>            };
>>        };
>>    };
>>    __symbols__ {
>>    };
>>    __fixups__ {
>>        soc = "/fragment@0:target:0";
>>        gic = "/fragment@0/__overlay__/vpci@2000000:interrupt-map:16", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:56", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:96", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:136";
>>    };
>>    __local_fixups__ {
>>    };
>> };
>>
>> and a base dtb with a symbols section and all the required ones
>> included.
>>
> 
> OK, this should make things work. Did you try sticking this in the base dts and see if it works there?

Can't do this because Linux needs that tree to boot and Jailhouse with
its virtual PCI controller comes after that only.

I'll debug what's happing behind the curtains regarding PCI interrupt
mapping, maybe that will give a clue.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Jan Kiszka]

On 2016-06-21 13:59, Jan Kiszka wrote:
> On 2016-06-21 13:45, Pantelis Antoniou wrote:
>> Hi Jan,
>>
>>> On Jun 21, 2016, at 14:43 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>
>>> On 2016-06-21 13:35, Pantelis Antoniou wrote:
>>>> Hi Jan,
>>>>
>>>>> On Jun 21, 2016, at 14:22 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>>>
>>>>> On 2016-06-21 12:24, Pantelis Antoniou wrote:
>>>>>> Hi Jan,
>>>>>>
>>>>>>> On Jun 21, 2016, at 13:13 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>>>>>
>>>>>>> Hi Pantelis,
>>>>>>>
>>>>>>> coming back to this topic:
>>>>>>>
>>>>>>> On 2016-06-09 08:03, Jan Kiszka wrote:
>>>>>>>> OK, trial and error, and some interesting insights: I've played with DT
>>>>>>>> fragments and the overlay configfs patch of Pantelis [1] to have a
>>>>>>>> convenient start. Interestingly, I wasn't able to load a fragment that
>>>>>>>> followed the format specification for overlays ("Failed to resolve
>>>>>>>> tree"). By chance, I got this one working:
>>>>>>>>
>>>>>>>> /dts-v1/;
>>>>>>>> / {
>>>>>>>> 	fragment {
>>>>>>>> 		target-path = "/soc@01c00000";
>>>>>>>> 		__overlay__ {
>>>>>>>> 			#address-cells = <2>;
>>>>>>>> 			#size-cells = <2>;
>>>>>>>>
>>>>>>>> 			vpci@0x2000000 {
>>>>>>>> 				compatible = "pci-host-cam-generic";
>>>>>>>> 				device_type = "pci";
>>>>>>>> 				#address-cells = <3>;
>>>>>>>> 				#size-cells = <2>;
>>>>>>>> 				reg = <0 0x2000000 0 0x1000000>;
>>>>>>>> 				ranges =
>>>>>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>>>>>> 			};
>>>>>>>> 		};
>>>>>>>> 	};
>>>>>>>> };
>>>>>>>>
>>>>>>>> It successfully makes a BananaPi kernel add a pci host with the
>>>>>>>> specified config space and MMIO window.
>>>>>>>>
>>>>>>>> [   81.619583] PCI host bridge /soc@01c00000/vpci@0x2000000 ranges:
>>>>>>>> [   81.619610]   No bus range found for /soc@01c00000/vpci@0x2000000, using [bus 00-ff]
>>>>>>>> [   81.619634]   MEM 0x10000000..0x3fffffff -> 0x10000000
>>>>>>>> [   81.620482] pci-host-generic 2000000.vpci: ECAM at [mem 0x02000000-0x02ffffff] for [bus 00-ff]
>>>>>>>> [   81.620779] pci-host-generic 2000000.vpci: PCI host bridge to bus 0000:00
>>>>>>>> [   81.620801] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>>>>>> [   81.620814] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3fffffff]
>>>>>>>> [   81.620851] PCI: bus0: Fast back to back transfers enabled
>>>>>>>>
>>>>>>>> So, no /plugin/ statement, no phandles resolution. This format even
>>>>>>>> builds with the in-kernel dtc. Any explanations? Does the code make
>>>>>>>> sense (at least it builds without warnings)?
>>>>>>>>
>>>>>>>> Now I need to back this with some code in Jailhouse.
>>>>>>>
>>>>>>> Meanwhile I got a virtual PCI device recognized by Linux when running
>>>>>>> over Jailhouse. However, my hack above doesn't get me to proper
>>>>>>> interrupt mapping yet. This is what I was trying with upstream dtc:
>>>>>>>
>>>>>>> /dts-v1/;
>>>>>>> / {
>>>>>>> 	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>>>>>>>
>>>>>>> 	fragment@0 {
>>>>>>> 		target-path = "/soc@01c00000";
>>>>>>> 		__overlay__ {
>>>>>>> 			#address-cells = <2>;
>>>>>>> 			#size-cells = <2>;
>>>>>>>
>>>>>>> 			vpci@2000000 {
>>>>>>> 				compatible = "pci-host-ecam-generic";
>>>>>>> 				device_type = "pci";
>>>>>>> 				bus-range = <0 0>;
>>>>>>> 				#address-cells = <3>;
>>>>>>> 				#size-cells = <2>;
>>>>>>> 				#interrupt-cells = <1>;
>>>>>>> 				interrupt-map-mask = <0 0 0 7>;
>>>>>>> 				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
>>>>>>> 						<0 0 0 2 &gic 0 0 0 124 4>,
>>>>>>> 						<0 0 0 3 &gic 0 0 0 125 4>,
>>>>>>> 						<0 0 0 4 &gic 0 0 0 126 4>;
>>>>>>> 				reg = <0 0x2000000 0 0x100000>;
>>>>>>> 				ranges =
>>>>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>>>>> 			};
>>>>>>> 		};
>>>>>>> 	};
>>>>>>>
>>>>>>> 	gic: fragment@1 {
>>>>>>> 		target-path = "/soc@01c00000/interrupt-controller@01c81000";
>>>>>>> 		__overlay__ {
>>>>>>> 		};
>>>>>>> 	};
>>>>>>> };
>>>>>>>
>>>>>>
>>>>>> ^ This is not going to work: You need the reference to the real gic not the empty fragment
>>>>>> here that has a target there.
>>>>>>
>>>>>> You need to compile with the correct dtc, and you also need to compile the base dts
>>>>>> with dtc too, using the -@ flag. You can hack around it by adding something like
>>>>>>
>>>>>> __symbols__ {
>>>>>> 	gic = "/soc@01c00000/interrupt-controller@01c81000”;
>>>>>> };
>>>>>>
>>>>>> But you really need the __symbols__ node of the base dts generated by the dtc proper cause
>>>>>> the above is a dirty hack.
>>>>>>
>>>>>
>>>>> OK, re-building the kernel with DTC="/your/dtc -@", thus building the
>>>>> base dtb with symbols, fixes proper overlay format loading.
>>>>>
>>>>> However, no luck yet with the interrupt topic - maybe a different issue.
>>>>> Digging deeper…
>>>>>
>>>> Remove the gic: fragment and build both the kernel and the overlay with the -@ option.
>>>> That’s what makes it not to work.
>>>
>>> That's what I did, of course.
>>>
>>> I've now
>>>
>>> /dts-v1/;
>>> // magic:		0xd00dfeed
>>> // totalsize:		0x448 (1096)
>>> // off_dt_struct:	0x38
>>> // off_dt_strings:	0x3c0
>>> // off_mem_rsvmap:	0x28
>>> // version:		17
>>> // last_comp_version:	16
>>> // boot_cpuid_phys:	0x0
>>> // size_dt_strings:	0x88
>>> // size_dt_struct:	0x388
>>>
>>> / {
>>>    compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>>>    fragment@0 {
>>>        target = <0xffffffff>;
>>>        __overlay__ {
>>>            #address-cells = <0x00000002>;
>>>            #size-cells = <0x00000002>;
>>>            vpci@2000000 {
>>>                compatible = "pci-host-ecam-generic";
>>>                device_type = "pci";
>>>                bus-range = <0x00000000 0x00000000>;
>>>                #address-cells = <0x00000003>;
>>>                #size-cells = <0x00000002>;
>>>                #interrupt-cells = <0x00000001>;
>>>                interrupt-map-mask = <0x00000000 0x00000000 0x00000000 0x00000007>;
>>>                interrupt-map = <0x00000000 0x00000000 0x00000000 0x00000001 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007b 0x00000004 0x00000000 0x00000000 0x00000000 0x00000002 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007c 0x00000004 0x00000000 0x00000000 0x00000000 0x00000003 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007d 0x00000004 0x00000000 0x00000000 0x00000000 0x00000004 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007e 0x00000004>;
>>>                reg = <0x00000000 0x02000000 0x00000000 0x00100000>;
>>>                ranges = <0x02000000 0x00000000 0x10000000 0x00000000 0x10000000 0x00000000 0x30000000>;
>>>            };
>>>        };
>>>    };
>>>    __symbols__ {
>>>    };
>>>    __fixups__ {
>>>        soc = "/fragment@0:target:0";
>>>        gic = "/fragment@0/__overlay__/vpci@2000000:interrupt-map:16", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:56", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:96", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:136";
>>>    };
>>>    __local_fixups__ {
>>>    };
>>> };
>>>
>>> and a base dtb with a symbols section and all the required ones
>>> included.
>>>
>>
>> OK, this should make things work. Did you try sticking this in the base dts and see if it works there?
> 
> Can't do this because Linux needs that tree to boot and Jailhouse with
> its virtual PCI controller comes after that only.
> 
> I'll debug what's happing behind the curtains regarding PCI interrupt
> mapping, maybe that will give a clue.

That happens when someone without much clue about this device tree
thingy hacks fragments together from all over the place (wouldn't have
happened if the related specs were more readable). The problem was with
the width of the interrupt-map parent part. This now works:

/dts-v1/;
/plugin/;
/ {
	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";

	fragment {
		target-path = "/soc@01c00000";
		__overlay__ {
			#address-cells = <1>;
			#size-cells = <1>;

			vpci@2000000 {
				compatible = "pci-host-ecam-generic";
				device_type = "pci";
				bus-range = <0 0>;
				#address-cells = <3>;
				#size-cells = <2>;
				#interrupt-cells = <1>;
				interrupt-map-mask = <0 0 0 7>;
				interrupt-map = <0 0 0 1 &gic 0 123 4>,
						<0 0 0 2 &gic 0 124 4>,
						<0 0 0 3 &gic 0 125 4>,
						<0 0 0 4 &gic 0 126 4>;
				reg = <0x2000000 0x100000>;
				ranges =
					<0x02000000 0x00 0x10000000 0x10000000 0x00 0x30000000>;
			};
		};
	};
};

I'm sure there are more hidden bugs in this fragment, anyway:

linux:~ # cat /proc/interrupts 
           CPU0       CPU1       
...
126:          0          0     GIC-0 155 Level     ivshm-net

Yeah... Time to wire things up in the hypervisor.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Pantelis Antoniou]

Hi Jan,

> On Jun 21, 2016, at 16:12 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
> 
> On 2016-06-21 13:59, Jan Kiszka wrote:
>> On 2016-06-21 13:45, Pantelis Antoniou wrote:
>>> Hi Jan,
>>> 
>>>> On Jun 21, 2016, at 14:43 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>> 
>>>> On 2016-06-21 13:35, Pantelis Antoniou wrote:
>>>>> Hi Jan,
>>>>> 
>>>>>> On Jun 21, 2016, at 14:22 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>>>> 
>>>>>> On 2016-06-21 12:24, Pantelis Antoniou wrote:
>>>>>>> Hi Jan,
>>>>>>> 
>>>>>>>> On Jun 21, 2016, at 13:13 , Jan Kiszka <jan.kiszka@siemens.com> wrote:
>>>>>>>> 
>>>>>>>> Hi Pantelis,
>>>>>>>> 
>>>>>>>> coming back to this topic:
>>>>>>>> 
>>>>>>>> On 2016-06-09 08:03, Jan Kiszka wrote:
>>>>>>>>> OK, trial and error, and some interesting insights: I've played with DT
>>>>>>>>> fragments and the overlay configfs patch of Pantelis [1] to have a
>>>>>>>>> convenient start. Interestingly, I wasn't able to load a fragment that
>>>>>>>>> followed the format specification for overlays ("Failed to resolve
>>>>>>>>> tree"). By chance, I got this one working:
>>>>>>>>> 
>>>>>>>>> /dts-v1/;
>>>>>>>>> / {
>>>>>>>>> 	fragment {
>>>>>>>>> 		target-path = "/soc@01c00000";
>>>>>>>>> 		__overlay__ {
>>>>>>>>> 			#address-cells = <2>;
>>>>>>>>> 			#size-cells = <2>;
>>>>>>>>> 
>>>>>>>>> 			vpci@0x2000000 {
>>>>>>>>> 				compatible = "pci-host-cam-generic";
>>>>>>>>> 				device_type = "pci";
>>>>>>>>> 				#address-cells = <3>;
>>>>>>>>> 				#size-cells = <2>;
>>>>>>>>> 				reg = <0 0x2000000 0 0x1000000>;
>>>>>>>>> 				ranges =
>>>>>>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>>>>>>> 			};
>>>>>>>>> 		};
>>>>>>>>> 	};
>>>>>>>>> };
>>>>>>>>> 
>>>>>>>>> It successfully makes a BananaPi kernel add a pci host with the
>>>>>>>>> specified config space and MMIO window.
>>>>>>>>> 
>>>>>>>>> [   81.619583] PCI host bridge /soc@01c00000/vpci@0x2000000 ranges:
>>>>>>>>> [   81.619610]   No bus range found for /soc@01c00000/vpci@0x2000000, using [bus 00-ff]
>>>>>>>>> [   81.619634]   MEM 0x10000000..0x3fffffff -> 0x10000000
>>>>>>>>> [   81.620482] pci-host-generic 2000000.vpci: ECAM at [mem 0x02000000-0x02ffffff] for [bus 00-ff]
>>>>>>>>> [   81.620779] pci-host-generic 2000000.vpci: PCI host bridge to bus 0000:00
>>>>>>>>> [   81.620801] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>>>>>>> [   81.620814] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3fffffff]
>>>>>>>>> [   81.620851] PCI: bus0: Fast back to back transfers enabled
>>>>>>>>> 
>>>>>>>>> So, no /plugin/ statement, no phandles resolution. This format even
>>>>>>>>> builds with the in-kernel dtc. Any explanations? Does the code make
>>>>>>>>> sense (at least it builds without warnings)?
>>>>>>>>> 
>>>>>>>>> Now I need to back this with some code in Jailhouse.
>>>>>>>> 
>>>>>>>> Meanwhile I got a virtual PCI device recognized by Linux when running
>>>>>>>> over Jailhouse. However, my hack above doesn't get me to proper
>>>>>>>> interrupt mapping yet. This is what I was trying with upstream dtc:
>>>>>>>> 
>>>>>>>> /dts-v1/;
>>>>>>>> / {
>>>>>>>> 	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>>>>>>>> 
>>>>>>>> 	fragment@0 {
>>>>>>>> 		target-path = "/soc@01c00000";
>>>>>>>> 		__overlay__ {
>>>>>>>> 			#address-cells = <2>;
>>>>>>>> 			#size-cells = <2>;
>>>>>>>> 
>>>>>>>> 			vpci@2000000 {
>>>>>>>> 				compatible = "pci-host-ecam-generic";
>>>>>>>> 				device_type = "pci";
>>>>>>>> 				bus-range = <0 0>;
>>>>>>>> 				#address-cells = <3>;
>>>>>>>> 				#size-cells = <2>;
>>>>>>>> 				#interrupt-cells = <1>;
>>>>>>>> 				interrupt-map-mask = <0 0 0 7>;
>>>>>>>> 				interrupt-map = <0 0 0 1 &gic 0 0 0 123 4>,
>>>>>>>> 						<0 0 0 2 &gic 0 0 0 124 4>,
>>>>>>>> 						<0 0 0 3 &gic 0 0 0 125 4>,
>>>>>>>> 						<0 0 0 4 &gic 0 0 0 126 4>;
>>>>>>>> 				reg = <0 0x2000000 0 0x100000>;
>>>>>>>> 				ranges =
>>>>>>>> 					<0x02000000 0x00 0x10000000 0x00 0x10000000 0x00 0x30000000>;
>>>>>>>> 			};
>>>>>>>> 		};
>>>>>>>> 	};
>>>>>>>> 
>>>>>>>> 	gic: fragment@1 {
>>>>>>>> 		target-path = "/soc@01c00000/interrupt-controller@01c81000";
>>>>>>>> 		__overlay__ {
>>>>>>>> 		};
>>>>>>>> 	};
>>>>>>>> };
>>>>>>>> 
>>>>>>> 
>>>>>>> ^ This is not going to work: You need the reference to the real gic not the empty fragment
>>>>>>> here that has a target there.
>>>>>>> 
>>>>>>> You need to compile with the correct dtc, and you also need to compile the base dts
>>>>>>> with dtc too, using the -@ flag. You can hack around it by adding something like
>>>>>>> 
>>>>>>> __symbols__ {
>>>>>>> 	gic = "/soc@01c00000/interrupt-controller@01c81000”;
>>>>>>> };
>>>>>>> 
>>>>>>> But you really need the __symbols__ node of the base dts generated by the dtc proper cause
>>>>>>> the above is a dirty hack.
>>>>>>> 
>>>>>> 
>>>>>> OK, re-building the kernel with DTC="/your/dtc -@", thus building the
>>>>>> base dtb with symbols, fixes proper overlay format loading.
>>>>>> 
>>>>>> However, no luck yet with the interrupt topic - maybe a different issue.
>>>>>> Digging deeper…
>>>>>> 
>>>>> Remove the gic: fragment and build both the kernel and the overlay with the -@ option.
>>>>> That’s what makes it not to work.
>>>> 
>>>> That's what I did, of course.
>>>> 
>>>> I've now
>>>> 
>>>> /dts-v1/;
>>>> // magic:		0xd00dfeed
>>>> // totalsize:		0x448 (1096)
>>>> // off_dt_struct:	0x38
>>>> // off_dt_strings:	0x3c0
>>>> // off_mem_rsvmap:	0x28
>>>> // version:		17
>>>> // last_comp_version:	16
>>>> // boot_cpuid_phys:	0x0
>>>> // size_dt_strings:	0x88
>>>> // size_dt_struct:	0x388
>>>> 
>>>> / {
>>>>   compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
>>>>   fragment@0 {
>>>>       target = <0xffffffff>;
>>>>       __overlay__ {
>>>>           #address-cells = <0x00000002>;
>>>>           #size-cells = <0x00000002>;
>>>>           vpci@2000000 {
>>>>               compatible = "pci-host-ecam-generic";
>>>>               device_type = "pci";
>>>>               bus-range = <0x00000000 0x00000000>;
>>>>               #address-cells = <0x00000003>;
>>>>               #size-cells = <0x00000002>;
>>>>               #interrupt-cells = <0x00000001>;
>>>>               interrupt-map-mask = <0x00000000 0x00000000 0x00000000 0x00000007>;
>>>>               interrupt-map = <0x00000000 0x00000000 0x00000000 0x00000001 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007b 0x00000004 0x00000000 0x00000000 0x00000000 0x00000002 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007c 0x00000004 0x00000000 0x00000000 0x00000000 0x00000003 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007d 0x00000004 0x00000000 0x00000000 0x00000000 0x00000004 0xffffffff 0x00000000 0x00000000 0x00000000 0x0000007e 0x00000004>;
>>>>               reg = <0x00000000 0x02000000 0x00000000 0x00100000>;
>>>>               ranges = <0x02000000 0x00000000 0x10000000 0x00000000 0x10000000 0x00000000 0x30000000>;
>>>>           };
>>>>       };
>>>>   };
>>>>   __symbols__ {
>>>>   };
>>>>   __fixups__ {
>>>>       soc = "/fragment@0:target:0";
>>>>       gic = "/fragment@0/__overlay__/vpci@2000000:interrupt-map:16", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:56", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:96", "/fragment@0/__overlay__/vpci@2000000:interrupt-map:136";
>>>>   };
>>>>   __local_fixups__ {
>>>>   };
>>>> };
>>>> 
>>>> and a base dtb with a symbols section and all the required ones
>>>> included.
>>>> 
>>> 
>>> OK, this should make things work. Did you try sticking this in the base dts and see if it works there?
>> 
>> Can't do this because Linux needs that tree to boot and Jailhouse with
>> its virtual PCI controller comes after that only.
>> 
>> I'll debug what's happing behind the curtains regarding PCI interrupt
>> mapping, maybe that will give a clue.
> 
> That happens when someone without much clue about this device tree
> thingy hacks fragments together from all over the place (wouldn't have
> happened if the related specs were more readable). The problem was with
> the width of the interrupt-map parent part. This now works:
> 

+1 on the unreadable specs. I would vote for a more verbose definition if that
would make things more readable.

> /dts-v1/;
> /plugin/;
> / {
> 	compatible = "lemaker,bananapi", "allwinner,sun7i-a20";
> 
> 	fragment {
> 		target-path = "/soc@01c00000";
> 		__overlay__ {
> 			#address-cells = <1>;
> 			#size-cells = <1>;
> 
> 			vpci@2000000 {
> 				compatible = "pci-host-ecam-generic";
> 				device_type = "pci";
> 				bus-range = <0 0>;
> 				#address-cells = <3>;
> 				#size-cells = <2>;
> 				#interrupt-cells = <1>;
> 				interrupt-map-mask = <0 0 0 7>;
> 				interrupt-map = <0 0 0 1 &gic 0 123 4>,
> 						<0 0 0 2 &gic 0 124 4>,
> 						<0 0 0 3 &gic 0 125 4>,
> 						<0 0 0 4 &gic 0 126 4>;
> 				reg = <0x2000000 0x100000>;
> 				ranges =
> 					<0x02000000 0x00 0x10000000 0x10000000 0x00 0x30000000>;
> 			};
> 		};
> 	};
> };
> 
> I'm sure there are more hidden bugs in this fragment, anyway:
> 
> linux:~ # cat /proc/interrupts 
>           CPU0       CPU1       
> ...
> 126:          0          0     GIC-0 155 Level     ivshm-net
> 
> Yeah... Time to wire things up in the hypervisor.
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA ITP SES-DE
> Corporate Competence Center Embedded Linux

Re: Using DT overlays for adding virtual hardware

[Arnd Bergmann]

On Wednesday, June 8, 2016 6:39:08 PM CEST Jan Kiszka wrote:
> >>
> > 
> > I just don’t see how an ACPI based hypervisor can ever be certified for
> > safety critical applications. It might be possible but it should be
> > an enormous undertaking; perhaps a subset without AML, but then again
> > can you even boot an ACPI box without it?
> 
> ACPI is out of scope for us. We will probably continue to feed the
> hypervisor with static platform information, generated in advance and
> validated. Can be DT-based one day, but even that is more complex to
> parse than our current structures.
> 
> But does ACPI usually mean that the kernel no longer has DT support and
> would not be able to handle any overlay? That could be a killer.

The kernel always has DT support built-in, but there may be some code
paths that do not look at DT properties when it was booted from ACPI.

In particular, communicating things like interrupt mappings may be
hard, as they are represented very differently on ACPI, so you no
longer have an 'interrupt-parent' node to point to from your overlay.

It's hard to say how things would work out when trying to load DT
overlays in this configuration. My guess is that it's actually
easier to do on x86 (which doesn't normally rely on ACPI for
describing the core system) than on arm64.

> > DT is safer since it contains state only.
> > 
> >> To be clear, I'm not arguing *against* overlays as such, just making
> >> sure that we're not prematurely choosing a solution just becasue it's
> >> the one we're aware of.
> 
> I'm open for any suggestion that is simple. Maybe we can extend a
> trivial existing pci host driver (like pci-host-generic) to work also
> without DT overlays - also fine, at least from Jailhose POV. However,
> any unneeded kernel patch is even better.

A few more observations:

- you can easily have an arbitrary number of PCI host bridges, so you
  can always add another PCI bridge just for the virtual devices even
  on systems that have access to physical PCI devices in passthrough.

- PCIe hotplugging seems well-defined enough to just make that work,
  without needing DT overlays.

- The really tricky question is what to do about passthrough of
  host devices that are not PCI. The current generation of server
  class arm64 machines tend to have a bunch of those, and the
  expectation seems to be that hardware passthrough is the only
  way to get decent I/O performance to make up for the relatively
  slow CPU cores. If you are only concerned about emulated devices,
  that won't be a problem though.

	Arnd

Re: Using DT overlays for adding virtual hardware

[Jan Kiszka]

On 2016-06-09 09:22, Arnd Bergmann wrote:
> On Wednesday, June 8, 2016 6:39:08 PM CEST Jan Kiszka wrote:
>>>>
>>>
>>> I just don’t see how an ACPI based hypervisor can ever be certified for
>>> safety critical applications. It might be possible but it should be
>>> an enormous undertaking; perhaps a subset without AML, but then again
>>> can you even boot an ACPI box without it?
>>
>> ACPI is out of scope for us. We will probably continue to feed the
>> hypervisor with static platform information, generated in advance and
>> validated. Can be DT-based one day, but even that is more complex to
>> parse than our current structures.
>>
>> But does ACPI usually mean that the kernel no longer has DT support and
>> would not be able to handle any overlay? That could be a killer.
> 
> The kernel always has DT support built-in, but there may be some code
> paths that do not look at DT properties when it was booted from ACPI.
> 
> In particular, communicating things like interrupt mappings may be
> hard, as they are represented very differently on ACPI, so you no
> longer have an 'interrupt-parent' node to point to from your overlay.
> 
> It's hard to say how things would work out when trying to load DT
> overlays in this configuration. My guess is that it's actually
> easier to do on x86 (which doesn't normally rely on ACPI for
> describing the core system) than on arm64.

OK. But let's see if there are really systems with ACPI and without
pre-existing PCI. Currently, I would say the probability is low, because
ACPI means server, and servers love PCI...

> 
>>> DT is safer since it contains state only.
>>>
>>>> To be clear, I'm not arguing *against* overlays as such, just making
>>>> sure that we're not prematurely choosing a solution just becasue it's
>>>> the one we're aware of.
>>
>> I'm open for any suggestion that is simple. Maybe we can extend a
>> trivial existing pci host driver (like pci-host-generic) to work also
>> without DT overlays - also fine, at least from Jailhose POV. However,
>> any unneeded kernel patch is even better.
> 
> A few more observations:
> 
> - you can easily have an arbitrary number of PCI host bridges, so you
>   can always add another PCI bridge just for the virtual devices even
>   on systems that have access to physical PCI devices in passthrough.
> 
> - PCIe hotplugging seems well-defined enough to just make that work,
>   without needing DT overlays.

The point is about adding virtual devices when there is no physical PCI
- when there is, we can already sneak them in between physical ones.

Granted, when we run out of free slots, there is a need to do more,
either via virtual bridges (but hypervisor is the last place we'd like
to touch), by enforcing Linux to scan on slots outside of the physical
topology or by making it create bridge stubs for virtual devices that
are not assigned to a physical bus. But that's all PCI topics, not
directly related to the original point of adding the host bridge.

> 
> - The really tricky question is what to do about passthrough of
>   host devices that are not PCI. The current generation of server
>   class arm64 machines tend to have a bunch of those, and the
>   expectation seems to be that hardware passthrough is the only
>   way to get decent I/O performance to make up for the relatively
>   slow CPU cores. If you are only concerned about emulated devices,
>   that won't be a problem though.

Yes, that is tricky, but more from the analytical POV: which devices or
which parts of devices can we hand out to guests without jeopardizing
the system integrity? No generic answers here, for sure.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux