From: Peter Zijlstra <peterz@infradead.org> To: Nitesh Narayan Lal <nitesh@redhat.com> Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-pci@vger.kernel.org, intel-wired-lan@lists.osuosl.org, frederic@kernel.org, mtosatti@redhat.com, sassmann@redhat.com, jesse.brandeburg@intel.com, lihong.yang@intel.com, helgaas@kernel.org, jeffrey.t.kirsher@intel.com, jacob.e.keller@intel.com, jlelli@redhat.com, hch@infradead.org, bhelgaas@google.com, mike.marciniszyn@intel.com, dennis.dalessandro@intel.com, thomas.lendacky@amd.com, jiri@nvidia.com, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, lgoncalv@redhat.com Subject: Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs Date: Mon, 19 Oct 2020 13:11:37 +0200 Message-ID: <20201019111137.GL2628@hirez.programming.kicks-ass.net> (raw) In-Reply-To: <79f382a7-883d-ff42-394d-ec4ce81fed6a@redhat.com> On Sun, Oct 18, 2020 at 02:14:46PM -0400, Nitesh Narayan Lal wrote: > >> + hk_cpus = housekeeping_num_online_cpus(HK_FLAG_MANAGED_IRQ); > >> + > >> + /* > >> + * If we have isolated CPUs for use by real-time tasks, to keep the > >> + * latency overhead to a minimum, device-specific IRQ vectors are moved > >> + * to the housekeeping CPUs from the userspace by changing their > >> + * affinity mask. Limit the vector usage to keep housekeeping CPUs from > >> + * running out of IRQ vectors. > >> + */ > >> + if (hk_cpus < num_online_cpus()) { > >> + if (hk_cpus < min_vecs) > >> + max_vecs = min_vecs; > >> + else if (hk_cpus < max_vecs) > >> + max_vecs = hk_cpus; > > is that: > > > > max_vecs = clamp(hk_cpus, min_vecs, max_vecs); > > Yes, I think this will do. > > > > > Also, do we really need to have that conditional on hk_cpus < > > num_online_cpus()? That is, why can't we do this unconditionally? > > FWIU most of the drivers using this API already restricts the number of > vectors based on the num_online_cpus, if we do it unconditionally we can > unnecessary duplicate the restriction for cases where we don't have any > isolated CPUs. unnecessary isn't really a concern here, this is a slow path. What's important is code clarity. > Also, different driver seems to take different factors into consideration > along with num_online_cpus while finding the max_vecs to request, for > example in the case of mlx5: > MLX5_CAP_GEN(dev, num_ports) * num_online_cpus() + > MLX5_EQ_VEC_COMP_BASE > > Having hk_cpus < num_online_cpus() helps us ensure that we are only > changing the behavior when we have isolated CPUs. > > Does that make sense? That seems to want to allocate N interrupts per cpu (plus some random static amount, which seems weird, but whatever). This patch breaks that. So I think it is important to figure out what that driver really wants in the nohz_full case. If it wants to retain N interrupts per CPU, and only reduce the number of CPUs, the proposed interface is wrong. > > And what are the (desired) semantics vs hotplug? Using a cpumask without > > excluding hotplug is racy. > > The housekeeping_mask should still remain constant, isn't? > In any case, I can double check this. The goal is very much to have that dynamically configurable.
next prev parent reply index Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-09-28 18:35 [PATCH v4 0/4] isolation: limit msix vectors " Nitesh Narayan Lal 2020-09-28 18:35 ` [PATCH v4 1/4] sched/isolation: API to get number of " Nitesh Narayan Lal 2020-09-28 18:35 ` [PATCH v4 2/4] sched/isolation: Extend nohz_full to isolate managed IRQs Nitesh Narayan Lal 2020-10-23 13:25 ` Peter Zijlstra 2020-10-23 13:29 ` Frederic Weisbecker 2020-10-23 13:57 ` Nitesh Narayan Lal 2020-10-23 13:45 ` Nitesh Narayan Lal 2020-09-28 18:35 ` [PATCH v4 3/4] i40e: Limit msix vectors to housekeeping CPUs Nitesh Narayan Lal 2020-09-28 18:35 ` [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() " Nitesh Narayan Lal 2020-09-28 21:59 ` Bjorn Helgaas 2020-09-29 17:46 ` Christoph Hellwig 2020-10-16 12:20 ` Peter Zijlstra 2020-10-18 18:14 ` Nitesh Narayan Lal 2020-10-19 11:11 ` Peter Zijlstra [this message] 2020-10-19 14:00 ` Marcelo Tosatti 2020-10-19 14:25 ` Nitesh Narayan Lal 2020-10-20 7:30 ` Peter Zijlstra 2020-10-20 13:00 ` Nitesh Narayan Lal 2020-10-20 13:41 ` Peter Zijlstra 2020-10-20 14:39 ` Nitesh Narayan Lal 2020-10-22 17:47 ` Nitesh Narayan Lal 2020-10-23 8:58 ` Peter Zijlstra 2020-10-23 13:10 ` Nitesh Narayan Lal 2020-10-23 21:00 ` Thomas Gleixner 2020-10-26 13:35 ` Nitesh Narayan Lal 2020-10-26 13:57 ` Thomas Gleixner 2020-10-26 17:30 ` Marcelo Tosatti 2020-10-26 19:00 ` Thomas Gleixner 2020-10-26 19:11 ` Marcelo Tosatti 2020-10-26 19:21 ` Jacob Keller 2020-10-26 20:11 ` Thomas Gleixner 2020-10-26 21:11 ` Jacob Keller 2020-10-26 21:50 ` Thomas Gleixner 2020-10-26 22:13 ` Jakub Kicinski 2020-10-26 22:46 ` Thomas Gleixner 2020-10-26 22:52 ` Jacob Keller 2020-10-26 22:22 ` Nitesh Narayan Lal 2020-10-26 22:49 ` Thomas Gleixner 2020-10-26 23:08 ` Jacob Keller 2020-10-27 14:28 ` Thomas Gleixner 2020-10-27 11:47 ` Marcelo Tosatti 2020-10-27 14:43 ` Thomas Gleixner 2020-10-19 14:21 ` Frederic Weisbecker 2020-10-20 14:16 ` Thomas Gleixner 2020-10-20 16:18 ` Nitesh Narayan Lal 2020-10-20 18:07 ` Thomas Gleixner 2020-10-21 20:25 ` Thomas Gleixner 2020-10-21 21:04 ` Nitesh Narayan Lal 2020-10-22 0:02 ` Jakub Kicinski 2020-10-22 0:27 ` Jacob Keller 2020-10-22 8:28 ` Thomas Gleixner 2020-10-22 12:28 ` Marcelo Tosatti 2020-10-22 22:39 ` Thomas Gleixner 2020-10-01 15:49 ` [PATCH v4 0/4] isolation: limit msix vectors " Frederic Weisbecker 2020-10-08 21:40 ` Nitesh Narayan Lal
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20201019111137.GL2628@hirez.programming.kicks-ass.net \ --to=peterz@infradead.org \ --cc=bhelgaas@google.com \ --cc=dennis.dalessandro@intel.com \ --cc=frederic@kernel.org \ --cc=hch@infradead.org \ --cc=helgaas@kernel.org \ --cc=intel-wired-lan@lists.osuosl.org \ --cc=jacob.e.keller@intel.com \ --cc=jeffrey.t.kirsher@intel.com \ --cc=jesse.brandeburg@intel.com \ --cc=jiri@nvidia.com \ --cc=jlelli@redhat.com \ --cc=juri.lelli@redhat.com \ --cc=lgoncalv@redhat.com \ --cc=lihong.yang@intel.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-pci@vger.kernel.org \ --cc=mike.marciniszyn@intel.com \ --cc=mingo@redhat.com \ --cc=mtosatti@redhat.com \ --cc=netdev@vger.kernel.org \ --cc=nitesh@redhat.com \ --cc=sassmann@redhat.com \ --cc=thomas.lendacky@amd.com \ --cc=vincent.guittot@linaro.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Linux-PCI Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/linux-pci/0 linux-pci/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 linux-pci linux-pci/ https://lore.kernel.org/linux-pci \ linux-pci@vger.kernel.org public-inbox-index linux-pci Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-pci AGPL code for this site: git clone https://public-inbox.org/public-inbox.git