Re: [RFC PATCH v2 00/26] arm64: Dom0 ITS emulation

[Julien Grall <julien.grall@arm.com>]

Hello Vijay,

On 18/01/2017 08:13, Vijay Kilari wrote:
> On Thu, Dec 22, 2016 at 11:54 PM, Andre Przywara <andre.przywara@arm.com> wrote:
>> Hi,
>>
>> this is a reworked version of the Dom0 GICv3-ITS emulation series.
>> This is still not fully where I want it and has some loose bits and
>> pieces still, but since there are significant changes in the architecture
>> I wanted to have an opinion before going ahead and replacing every single
>> number with a named constant ;-) If that smells like a "send out before
>> the end of the year", you are spot on.
>>
>> This series introduces ARM GICv3 ITS emulation, for now restricted to
>> Dom0 only. The ITS is an interrupt controller widget providing a
>> sophisticated way to deal with MSIs in a scalable manner.
>> For hardware which relies on the ITS to provide interrupts for its
>> peripherals this code is needed to get a machine booted into Dom0 at all.
>> ITS emulation for DomUs is only really useful with PCI passthrough,
>> which is not yet available for ARM. It is expected that this feature
>> will be co-developed with the ITS DomU code. However this code drop here
>> considered DomU emulation already, to keep later architectural changes
>> to a minimum.
>>
>> Some generic design principles:
>>
>> * The current GIC code statically allocates structures for each supported
>> IRQ (both for the host and the guest), which due to the potentially
>> millions of LPI interrupts is not feasible to copy for the ITS.
>> So we refrain from introducing the ITS as a first class Xen interrupt
>> controller, also we don't hold struct irq_desc's or struct pending_irq's
>> for each possible LPI.
>> Fortunately LPIs are only interesting to guests, so we get away with
>> storing only the virtual IRQ number and the guest VCPU for each allocated
>> host LPI, which can be stashed into one uint64_t. This data is stored in
>> a two-level table, which is both memory efficient and quick to access.
>> We hook into the existing IRQ handling and VGIC code to avoid accessing
>> the normal structures, providing alternative methods for getting the
>> needed information (priority, is enabled?) for LPIs.
>> For interrupts which are queued to or are actually in a guest we
>> allocate struct pending_irq's on demand. As it is expected that only a
>> very small number of interrupts is ever on a VCPU at the same time, this
>> seems like the best approach. For now allocated structs are re-used and
>> held in a linked list.
>>
>> * On the guest side we (later will) have to deal with malicious guests
>> trying to hog Xen with mapping requests for a lot of LPIs, for instance.
>> As the ITS actually uses system memory for storing status information,
>> we use this memory (which the guest has to provide) to naturally limit
>> a guest. For those tables which are page sized (devices, collections (CPUs),
>> LPI properties) we map those pages into Xen, so we can easily access
>> them from the virtual GIC code.
>> Unfortunately the actual interrupt mapping tables are not necessarily
>> page aligned, also can be much smaller than a page, so mapping all of
>> them permanently is fiddly. As ITS commands in need to iterate those
>> tables are pretty rare after all, we for now map them on demand upon
>> emulating a virtual ITS command.
>>
>> * An obvious approach to handling some guest ITS commands would be to
>> propagate them to the host, for instance to map devices and LPIs and
>> to enable or disable LPIs.
>> However this (later with DomU support) will create an attack vector, as
>> a malicious guest could try to fill the host command queue with
>> propagated commands.
>> So in contrast to the previous RFC post this version now completely avoids
>> this situation. For mapping devices and LPIs we rely on this being done
>> via a hypercall prior to the actual guest run. For enabling and disabling
>> LPIs we keep this bit on the virtual side and let LPIs always be enabled
>> on the host side, dealing with the consequences this approach creates.
>>
>> This series is still a draft, with some known and many unknown issues.
>> I made ITS support a Kconfig option, also it is only supported on arm64.
>> This leads to some hideous constructs like an #ifdef'ed header file with
>> empty function stubs, but I guess we can clean this up later in the
>> upstreaming process.
>>
>> There are numerous changes compared to the last post, mainly affecting
>> the now missing ITS command progagation. I also added locking to the
>> "usual suspects" data structures.
>> I picked some low hanging fruits from the review comments.
>> Things I haven't addresses well is the whole memory management, in terms
>> of marking pages r/o for a guest or allocating Xen memory from the proper
>> bucket. This will be addresses with the next post.
>>
>> For now this code happens to boot Dom0 on an ARM fast model with ITS
>> support. I still haven't had the chance to get hold of a Xen supported
>> hardware platform with an ITS yet, so running on real hardware is a bit
>> terra incognita.
>>
>> The code can also be found on the its/rfc-v2 branch here:
>> git://linux-arm.org/xen-ap.git
>> http://www.linux-arm.org/git?p=xen-ap.git;a=shortlog;h=refs/heads/its/rfc-v2
>
> What is the kernel version that you tried to boot dom0?.

I haven't tried the ITS series, but any kernel version which boot 
baremetal on your platform should boot on Xen. If not, you need to 
figure out why.

Regarding the kernel config, it will depend on your platform. In general 
your platform options + classic xen options should work.

> Have you tried with smmu and pci devices?.  Please share your kernel config.

The SMMU driver in Xen does not yet support PCI devices.

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel