Re: Discussion about virtual iommu support for Xen guest

["Lan, Tianyu" <tianyu.lan@intel.com>]

On 6/3/2016 7:17 PM, Tian, Kevin wrote:
>> From: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
>> Sent: Friday, June 03, 2016 2:59 AM
>>
>> On 02/06/16 16:03, Lan, Tianyu wrote:
>>> On 5/27/2016 4:19 PM, Lan Tianyu wrote:
>>>> On 2016年05月26日 19:35, Andrew Cooper wrote:
>>>>> On 26/05/16 09:29, Lan Tianyu wrote:
>>>>>
>>>>> To be viable going forwards, any solution must work with PVH/HVMLite as
>>>>> much as HVM.  This alone negates qemu as a viable option.
>>>>>
>>>>> From a design point of view, having Xen needing to delegate to qemu to
>>>>> inject an interrupt into a guest seems backwards.
>>>>>
>>>>
>>>> Sorry, I am not familiar with HVMlite. HVMlite doesn't use Qemu and
>>>> the qemu virtual iommu can't work for it. We have to rewrite virtual
>>>> iommu in the Xen, right?
>>>>
>>>>>
>>>>> A whole lot of this would be easier to reason about if/when we get a
>>>>> basic root port implementation in Xen, which is necessary for HVMLite,
>>>>> and which will make the interaction with qemu rather more clean.  It is
>>>>> probably worth coordinating work in this area.
>>>>
>>>> The virtual iommu also should be under basic root port in Xen, right?
>>>>
>>>>>
>>>>> As for the individual issue of 288vcpu support, there are already
>>>>> issues
>>>>> with 64vcpu guests at the moment. While it is certainly fine to remove
>>>>> the hard limit at 255 vcpus, there is a lot of other work required to
>>>>> even get 128vcpu guests stable.
>>>>
>>>>
>>>> Could you give some points to these issues? We are enabling more vcpus
>>>> support and it can boot up 255 vcpus without IR support basically. It's
>>>> very helpful to learn about known issues.
>>>>
>>>> We will also add more tests for 128 vcpus into our regular test to find
>>>> related bugs. Increasing max vcpu to 255 should be a good start.
>>>
>>> Hi Andrew:
>>> Could you give more inputs about issues with 64 vcpus and what needs to
>>> be done to make 128vcpu guest stable? We hope to do somethings to
>>> improve them.
>>>
>>> What's progress of PCI host bridge in Xen? From your opinion, we should
>>> do that first, right? Thanks.
>>
>> Very sorry for the delay.
>>
>> There are multiple interacting issues here.  On the one side, it would
>> be useful if we could have a central point of coordination on
>> PVH/HVMLite work.  Roger - as the person who last did HVMLite work,
>> would you mind organising that?
>>
>> For the qemu/xen interaction, the current state is woeful and a tangled
>> mess.  I wish to ensure that we don't make any development decisions
>> which makes the situation worse.
>>
>> In your case, the two motivations are quite different I would recommend
>> dealing with them independently.
>>
>> IIRC, the issue with more than 255 cpus and interrupt remapping is that
>> you can only use x2apic mode with more than 255 cpus, and IOAPIC RTEs
>> can't be programmed to generate x2apic interrupts?  In principle, if you
>> don't have an IOAPIC, are there any other issues to be considered?  What
>> happens if you configure the LAPICs in x2apic mode, but have the IOAPIC
>> deliver xapic interrupts?
>
> The key is the APIC ID. There is no modification to existing PCI MSI and
> IOAPIC with the introduction of x2apic. PCI MSI/IOAPIC can only send
> interrupt message containing 8bit APIC ID, which cannot address >255
> cpus. Interrupt remapping supports 32bit APIC ID so it's necessary to
> enable >255 cpus with x2apic mode.
>
> If LAPIC is in x2apic while interrupt remapping is disabled, IOAPIC cannot
> deliver interrupts to all cpus in the system if #cpu > 255.

Another key factor, Linux kernel disables x2apic mode when MAX APIC id
is > 255 if no interrupt remapping function. The reason for this is what
Kevin said. So booting up >255 cpus relies on the interrupt remapping.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel