From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from Galois.linutronix.de ([146.0.238.70]:54001 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726014AbeIBQSK (ORCPT ); Sun, 2 Sep 2018 12:18:10 -0400 Date: Sun, 2 Sep 2018 14:02:30 +0200 (CEST) From: Thomas Gleixner To: Kashyap Desai cc: Ming Lei , Sumit Saxena , Ming Lei , Christoph Hellwig , Linux Kernel Mailing List , Shivasharan Srikanteshwara , linux-block Subject: RE: Affinity managed interrupts vs non-managed interrupts In-Reply-To: <602cee6381b9f435a938bbaf852d07f9@mail.gmail.com> Message-ID: References: <20180829084618.GA24765@ming.t460p> <300d6fef733ca76ced581f8c6304bac6@mail.gmail.com> <615d78004495aebc53807156d04d988c@mail.gmail.com> <486f94a563d63c4779498fe8829a546c@mail.gmail.com> <602cee6381b9f435a938bbaf852d07f9@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Sender: linux-block-owner@vger.kernel.org List-Id: linux-block@vger.kernel.org On Fri, 31 Aug 2018, Kashyap Desai wrote: > > Ok. I misunderstood the whole thing a bit. So your real issue is that you > > want to have reply queues which are instantaneous, the per cpu ones, and > > then the extra 16 which do batching and are shared over a set of CPUs, > > right? > > Yes that is correct. Extra 16 or whatever should be shared over set of > CPUs of *local* numa node of the PCI device. Why restricting it to the local NUMA node of the device? That doesn't really make sense if you queue lots of requests from CPUs on a different node. Why don't you spread these extra interrupts accross all nodes and keep the locality for the request/reply? That also would allow to make them properly managed interrupts as you could shutdown the per node batching interrupts when all CPUs of that node are offlined and you'd avoid the whole affinity hint irq balancer hackery. Thanks, tglx From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E063C43334 for ; Sun, 2 Sep 2018 12:02:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3ACB920658 for ; Sun, 2 Sep 2018 12:02:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3ACB920658 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727186AbeIBQSK (ORCPT ); Sun, 2 Sep 2018 12:18:10 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:54001 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726014AbeIBQSK (ORCPT ); Sun, 2 Sep 2018 12:18:10 -0400 Received: from p4fea45ac.dip0.t-ipconnect.de ([79.234.69.172] helo=[192.168.0.145]) by Galois.linutronix.de with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1fwR5L-000109-QU; Sun, 02 Sep 2018 14:02:31 +0200 Date: Sun, 2 Sep 2018 14:02:30 +0200 (CEST) From: Thomas Gleixner To: Kashyap Desai cc: Ming Lei , Sumit Saxena , Ming Lei , Christoph Hellwig , Linux Kernel Mailing List , Shivasharan Srikanteshwara , linux-block Subject: RE: Affinity managed interrupts vs non-managed interrupts In-Reply-To: <602cee6381b9f435a938bbaf852d07f9@mail.gmail.com> Message-ID: References: <20180829084618.GA24765@ming.t460p> <300d6fef733ca76ced581f8c6304bac6@mail.gmail.com> <615d78004495aebc53807156d04d988c@mail.gmail.com> <486f94a563d63c4779498fe8829a546c@mail.gmail.com> <602cee6381b9f435a938bbaf852d07f9@mail.gmail.com> User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Linutronix-Spam-Score: -1.0 X-Linutronix-Spam-Level: - X-Linutronix-Spam-Status: No , -1.0 points, 5.0 required, ALL_TRUSTED=-1,SHORTCIRCUIT=-0.0001 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 31 Aug 2018, Kashyap Desai wrote: > > Ok. I misunderstood the whole thing a bit. So your real issue is that you > > want to have reply queues which are instantaneous, the per cpu ones, and > > then the extra 16 which do batching and are shared over a set of CPUs, > > right? > > Yes that is correct. Extra 16 or whatever should be shared over set of > CPUs of *local* numa node of the PCI device. Why restricting it to the local NUMA node of the device? That doesn't really make sense if you queue lots of requests from CPUs on a different node. Why don't you spread these extra interrupts accross all nodes and keep the locality for the request/reply? That also would allow to make them properly managed interrupts as you could shutdown the per node batching interrupts when all CPUs of that node are offlined and you'd avoid the whole affinity hint irq balancer hackery. Thanks, tglx