From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E194FC4361B for ; Fri, 11 Dec 2020 23:39:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ADC4923359 for ; Fri, 11 Dec 2020 23:39:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2436976AbgLKW57 (ORCPT ); Fri, 11 Dec 2020 17:57:59 -0500 Received: from Galois.linutronix.de ([193.142.43.55]:38364 "EHLO galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2407062AbgLKW5d (ORCPT ); Fri, 11 Dec 2020 17:57:33 -0500 From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1607727400; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RMdJ5cNpvG94MqmN/2d6QVEWUP/YrLyBgV8MUkzsNlA=; b=0WcDMUhLUE5DmQJjljMHI2zWwird6I5Vzppg/zn6AZ6e1OM3STU+JeGE05b5oqLFOvFfmh CDXWZX+LKXvz41MQk+jw0mabsf/xLxFNSlsc7D+IqZgppXn+tyA6NewoUrmvCeXtmt2AST EViRmHdbhCgpLeje9DpFYTQlqeoEtc+eq14ZwN+Y4G4vhAzpgACbffqabmLR4OWIgQ5TkH 0732UhUwEAjs3Gs9iSFScNlfSc9uVrpQKD7eElRAgICAQIyVgSIk6IrtHhKDjUooShlbLw 6lnH9jzEFDJiTz3F4zTjotppmf5h8vHSXnbS356QRRnsZMAn1kegUSilRX0RVQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1607727400; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RMdJ5cNpvG94MqmN/2d6QVEWUP/YrLyBgV8MUkzsNlA=; b=ljcajSXupjVbHPbFKhUP57PPTY2aSW+D5nnua5xYh57kSUlRXkx3yzKeC3jH07q5vSS64l BGsVaGm3g05BhRAQ== To: Andrew Cooper , boris.ostrovsky@oracle.com, =?utf-8?B?SsO8cmdlbiBHcm/Dnw==?= , LKML Cc: Peter Zijlstra , Marc Zyngier , Stefano Stabellini , xen-devel@lists.xenproject.org, "James E.J. Bottomley" , Helge Deller , afzal mohammed , linux-parisc@vger.kernel.org, Russell King , linux-arm-kernel@lists.infradead.org, Mark Rutland , Catalin Marinas , Will Deacon , Christian Borntraeger , Heiko Carstens , linux-s390@vger.kernel.org, Jani Nikula , Joonas Lahtinen , Rodrigo Vivi , David Airlie , Daniel Vetter , Pankaj Bharadiya , Chris Wilson , Wambui Karuga , intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, Tvrtko Ursulin , Linus Walleij , linux-gpio@vger.kernel.org, Lee Jones , Jon Mason , Dave Jiang , Allen Hubbe , linux-ntb@googlegroups.com, Lorenzo Pieralisi , Rob Herring , Bjorn Helgaas , Michal Simek , linux-pci@vger.kernel.org, Karthikeyan Mitran , Hou Zhiqiang , Tariq Toukan , "David S. Miller" , Jakub Kicinski , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, Saeed Mahameed , Leon Romanovsky Subject: Re: [patch 27/30] xen/events: Only force affinity mask for percpu interrupts In-Reply-To: References: <20201210192536.118432146@linutronix.de> <20201210194045.250321315@linutronix.de> <7f7af60f-567f-cdef-f8db-8062a44758ce@oracle.com> <2164a0ce-0e0d-c7dc-ac97-87c8f384ad82@suse.com> <871rfwiknd.fsf@nanos.tec.linutronix.de> <9806692f-24a3-4b6f-ae55-86bd66481271@oracle.com> <877dpoghio.fsf@nanos.tec.linutronix.de> Date: Fri, 11 Dec 2020 23:56:40 +0100 Message-ID: <87y2i4eytz.fsf@nanos.tec.linutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Andrew, On Fri, Dec 11 2020 at 22:21, Andrew Cooper wrote: > On 11/12/2020 21:27, Thomas Gleixner wrote: >> It's not any different from the hardware example at least not as far as >> I understood the code. > > Xen's event channels do have a couple of quirks. Why am I not surprised? > Binding an event channel always results in one spurious event being > delivered.=C2=A0 This is to cover notifications which can get lost during= the > bidirectional setup, or re-setups in certain configurations. > > Binding an interdomain or pirq event channel always defaults to vCPU0.=C2= =A0 > There is no way to atomically set the affinity while binding.=C2=A0 I bel= ieve > the API predates SMP guest support in Xen, and noone has fixed it up > since. That's fine. I'm not changing that. What I'm changing is the unwanted and unnecessary overwriting of the actual affinity mask. We have a similar issue on real hardware where we can only target _one_ CPU and not all CPUs in the affinity mask. So we still can preserve the (user) requested mask and just affine it to one CPU which is reflected in the effective affinity mask. This the right thing to do for two reasons: 1) It allows proper interrupt distribution 2) It does not break (user) requested affinity when the effective target CPU goes offline and the affinity mask still contains online CPUs. If you overwrite it you lost track of the requested broader mask. > As a consequence, the guest will observe the event raised on vCPU0 as > part of setting up the event, even if it attempts to set a different > affinity immediately afterwards.=C2=A0 A little bit of care needs to be t= aken > when binding an event channel on vCPUs other than 0, to ensure that the > callback is safe with respect to any remaining state needing > initialisation. That's preserved for all non percpu interrupts. The percpu variant of VIRQ and IPIs did binding to vCPU !=3D 0 already before this change. > Beyond this, there is nothing magic I'm aware of. > > We have seen soft lockups before in certain scenarios, simply due to the > quantity of events hitting vCPU0 before irqbalance gets around to > spreading the load.=C2=A0 This is why there is an attempt to round-robin = the > userspace event channel affinities by default, but I still don't see why > this would need custom affinity logic itself. Just the previous attempt makes no sense for the reasons I outlined in the changelog. So now with this new spreading mechanics you get the distribution for all cases: 1) Post setup using and respecting the default affinity mask which can be set as a kernel commandline parameter. 2) Runtime (user) requested affinity change with a mask which contains more than one vCPU. The previous logic always chose the first one in the mask. So assume userspace affines 4 irqs to a CPU 0-3 and 4 irqs to CPU 4-7 then 4 irqs end up on CPU0 and 4 on CPU4 The new algorithm which is similar to what we have on x86 (minus the vector space limitation) picks the CPU which has the least number of channels affine to it at that moment. If e.g. all 8 CPUs have the same number of vectors before that change then in the example above the first 4 are spread to CPU0-3 and the second 4 to CPU4-7 Thanks, tglx =20=20=20