From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756013Ab3EVPPB (ORCPT ); Wed, 22 May 2013 11:15:01 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:48265 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753226Ab3EVPPA (ORCPT ); Wed, 22 May 2013 11:15:00 -0400 Date: Wed, 22 May 2013 11:14:38 -0400 From: Konrad Rzeszutek Wilk To: Jan Beulich Cc: Stefano Stabellini , David Vrabel , Thomas Gleixner , "xen-devel@lists.xensource.com" , Chien Yen , Feng Jin , Yuval Shaia , Zhenzhong Duan , Ingo Molnar , "linux-kernel@vger.kernel.org" , "H. Peter Anvin" Subject: Re: [Xen-devel] [PATCH] xen: reuse the same pirq allocated when driver load first time Message-ID: <20130522151437.GC8162@phenom.dumpdata.com> References: <20130520175706.GA27973@phenom.dumpdata.com> <20130520203855.GA30616@phenom.dumpdata.com> <519B474E.4000202@citrix.com> <20130521134059.GE492@phenom.dumpdata.com> <20130521204245.GA7073@phenom.dumpdata.com> <20130521224125.GA3483@phenom.dumpdata.com> <519CAE0302000078000D807A@nat28.tlf.novell.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <519CAE0302000078000D807A@nat28.tlf.novell.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Source-IP: acsinet22.oracle.com [141.146.126.238] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, May 22, 2013 at 10:37:39AM +0100, Jan Beulich wrote: > >>> On 22.05.13 at 00:41, Konrad Rzeszutek Wilk wrote: > > On Tue, May 21, 2013 at 10:50:09PM +0100, Stefano Stabellini wrote: > >> We have to be careful about this: the point of PHYSDEVOP_get_free_pirq is > >> that Linux can know for sure the pirq that is going to be used to map the > >> MSI by QEMU. If you modify is_free_pirq that way, suddenly the pirq > >> could be allocated for something else after Linux called > >> PHYSDEVOP_get_free_pirq and before QEMU called xc_physdev_map_pirq_msi. > > > > Yes. And I think the 'is_free_pirq' modification above is incorrect. > > > > I think the fix should be in the unmap_pirq code (hypervisor) to check > > if the arch.irq is either zero or PIRQ_ALLOCATED. Right now it only > > checks for zero. > > Which check are you talking about? Looking at physdev_unmap_pirq() Sorry about being so haphazard here. I am still digging in the code and trying to get a sense of how QEMU and hypervisor are suppose to dance together. The check was on the PHYSDEVOP_get_free_pirq, which calls get_free_pirq and uses the is_free_pirq check. After the get_free_pirq call, the logic in PHYSDEVOP_get_free_pirq sets info->arch.pirq = PIRQ_ALLOCATED to protect itself from giving the same PIRQ twice. The physdev_unmap_pirq (from PHYSDEVOP_unmap_pirq), only has this check: if (domain_pirq_to_emuirq(d, pirq) != IRQ_UNBOUND) and since the arch.hvm.emuirq is IRQ_UNBOUND (-1), it does not call unmap_domain_pirq_emuirq. It probably shouldn't, but it should at least remove the info->arch.pirq = PIRQ_ALLOCATED as we are telling the hypervisor: "hey, I am done with this, return to the pool." But since that is not cleared, the PHYSDEVOP_get_free_pirq will skip this pirq as arch.pirq is still set to PIRQ_ALLOCATED. > I see none at all, unmap_domain_pirq() has a <= 0 check, and > unmap_domain_pirq_emuirq() again doesn't appear to have any. The 'unmap_domain_pirq' path would be if dom0 (so QEMU) did the unmap for the guest. That is via the PHYSDEVOP_unmap_pirq. And I think if that path was taken (as Stefano suggests QEMU should do when a MSI or MSI-X driver is unloaded and zero is writen as an PIRQ), we would end up calling clear_domain_irq_pirq, which would set arch.pirq = 0. Or to a negative value as you pointed out later. Which then means we won't be ever able to re-use the PIRQ (as PHYSDEVOP_get_free_pirq or rather get_free_pirq would skip over it as arch.pirq != 0). > > If you're talking about unmap_domain_pirq(), then you'll need to > be careful: A negative value here doesn't necessarily mean > PIRQ_ALLOCATED, but could also come from another run that > found it necessary to force the unbind. Hence the definition of > PIRQ_ALLOCATED would then collide with the (unlikely?) case of > IRQ1 having got assigned to a guest. To be on the safe side, we > should therefore redefine PIRQ_ALLOCATED to say INT_MIN. You are right about being cautious - this is a bit of complex code interaction between Xen, QEMU, and Linux kernel.