Re: Xen data from meta-virtualization layer

From: Stefano Stabellini <stefano.stabellini@xilinx.com>
To: Michael Walle <michael@walle.cc>
Cc: <stefano.stabellini@xilinx.com>, <Bertrand.Marquis@arm.com>,
	<Zhiqiang.Hou@nxp.com>, <brucea@xilinx.com>,
	<cornelia.bruelhart@zal.aero>, <julien@xen.org>,
	<leo.krueger@zal.aero>, <oleksandr_andrushchenko@epam.com>,
	<peng.fan@nxp.com>, <xen-devel@lists.xenproject.org>
Subject: Re: Xen data from meta-virtualization layer
Date: Fri, 4 Feb 2022 13:11:37 -0800	[thread overview]
Message-ID: <alpine.DEB.2.22.394.2202041256040.4074808@ubuntu-linux-20-04-desktop> (raw)
In-Reply-To: <20220204135814.1033356-1-michael@walle.cc>

On Fri, 4 Feb 2022, Michael Walle wrote:
> > In regards to the reserved-memory regions, maybe we are not seeing them
> > because Leo posted the host device tree, not the one passed at runtime
> > from u-boot to Linux?
> > 
> > If so, Leo, could you please boot Linux on native (no Xen) and get the
> > device tree from there at runtime using dtc -I fs -O dts
> > /proc/device-tree ?
> > 
> > 
> > However, the name of the reserved-memory region created by u-boot seems
> > to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
> > Linux kernel tree either.
> > 
> > Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
> > throws errors in regards to GIC/ITS initialization. On other hardware
> > Xen can use and virtualize GICv3 and ITS just fine. Could you please
> > explain what is different about sAL28 and how Xen/Linux is expected to
> > use the lpi_rd_table reserved-memory region?
> 
> I actually stumbled across this thread after trying out Xen myself. I'm
> using lastest vanilla u-boot (with pending PSCI patches), vanilla kernel
> and vanilla Xen.
> 
> So far I've discovered, that xen complains that it cannot route IRQ64 to
> dom0. That is because on the LS1028A there is a dual UART (two 16550
> with one shared interrupt) and xen takes the first UART and then tries
> to map the interrupt of the second UART to linux. For now, I don't know
> how this is solved correctly. As a quick hack, I removed the second uart
> node from the device tree.

This is an interesting problem. Removing the second UART is a good
workaround for now as there is no obvious solution I think.

> But what is more severe is that the iommu isn't set up correctly. I'm
> getting the following faults:
> 
> (XEN) smmu: /soc/iommu@5000000: Unexpected global fault, this could be serious
> (XEN) smmu: /soc/iommu@5000000: 	GFSR 0x80000002, GFSYNR0 0x00000000, GFSYNR1 0x0000042a, GFSYNR2 0x00000000
> 
> If I decode it correctly, the streamid should be 0x2a which would be one
> of the PCI devices on the internal root complex. Probably the network
> card.

Yes there is DMA transaction with an "unknown" StreamID. I think the
StreamID is 0x42a. It means that there is a DMA master on the board with
StreamID 0x42a that is either:

- not described in device tree
- described in device tree with a different StreamID
- the right StreamID is described device tree, but it is not picked up
  by Xen

> This is the first developer experience with Xen, so please bear with me
> :) It seems that Xen doesn't add the master to the IOMMU. To me it seems
> that only devices with a 'iommus' dt property are added. But in case of
> PCI devices the parent only has a iommu-map property.
> 
> And it makes me wonder why Leo has an almost working setup. Maybe I'm
> missing some patches though.

Xen 4.16 is able to parse StreamID in the "iommus" property and also
"mmu-masters" property. But It is not able to parse the "iommu-map"
property yet. So if 0x42a is described in device tree using "iommu-map"
then the error makes sense.

A simple solution is to replace iommu-map with iommus in device tree.
It is possible that someone in CC to this email might already have a
patch to introduce parsing of iommu-map in Xen.