linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Nitesh Lal <nilal@redhat.com>
To: Nitesh Narayan Lal <nitesh@redhat.com>,
	linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
	netdev@vger.kernel.org, linux-api@vger.kernel.org,
	linux-pci@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>,
	Jesse Brandeburg <jesse.brandeburg@intel.com>,
	Robin Murphy <robin.murphy@arm.com>,
	Marcelo Tosatti <mtosatti@redhat.com>,
	Ingo Molnar <mingo@kernel.org>,
	jbrandeb@kernel.org, frederic@kernel.org,
	Juri Lelli <juri.lelli@redhat.com>,
	Alex Belits <abelits@marvell.com>,
	Bjorn Helgaas <bhelgaas@google.com>,
	rostedt@goodmis.org, peterz@infradead.org, davem@davemloft.net,
	akpm@linux-foundation.org, sfr@canb.auug.org.au,
	stephen@networkplumber.org, rppt@linux.vnet.ibm.com,
	chris.friesen@windriver.com, Marc Zyngier <maz@kernel.org>,
	Neil Horman <nhorman@tuxdriver.com>,
	pjwaskiewicz@gmail.com, Stefan Assmann <sassmann@redhat.com>,
	Tomas Henzl <thenzl@redhat.com>,
	kashyap.desai@broadcom.com,
	Sumit Saxena <sumit.saxena@broadcom.com>,
	shivasharan.srikanteshwara@broadcom.com,
	sathya.prakash@broadcom.com,
	Sreekanth Reddy <sreekanth.reddy@broadcom.com>,
	suganath-prabu.subramani@broadcom.com, james.smart@broadcom.com,
	dick.kennedy@broadcom.com, Ken Cox <jkc@redhat.com>,
	faisal.latif@intel.com, shiraz.saleem@intel.com,
	tariqt@nvidia.com, Alaa Hleihel <ahleihel@redhat.com>,
	Kamal Heib <kheib@redhat.com>,
	borisp@nvidia.com, saeedm@nvidia.com, benve@cisco.com,
	govind@gmx.com, jassisinghbrar@gmail.com,
	ajit.khaparde@broadcom.com, sriharsha.basavapatna@broadcom.com,
	somnath.kotur@broadcom.com, "Nikolova,
	Tatyana E" <tatyana.e.nikolova@intel.com>,
	"Ismail, Mustafa" <mustafa.ismail@intel.com>,
	Al Stone <ahs3@redhat.com>,
	leonro@nvidia.com
Subject: Re: [PATCH v2 00/14] genirq: Cleanup the usage of irq_set_affinity_hint
Date: Thu, 8 Jul 2021 15:24:20 -0400	[thread overview]
Message-ID: <CAFki+LnUGiEE-7Uf-x8-TQZYZ+3Migrr=81gGLYszxaK-6A9WQ@mail.gmail.com> (raw)
In-Reply-To: <20210629152746.2953364-1-nitesh@redhat.com>

On Tue, Jun 29, 2021 at 11:28 AM Nitesh Narayan Lal <nitesh@redhat.com> wrote:
>
> The drivers currently rely on irq_set_affinity_hint() to either set the
> affinity_hint that is consumed by the userspace and/or to enforce a custom
> affinity.
>
> irq_set_affinity_hint() as the name suggests is originally introduced to
> only set the affinity_hint to help the userspace in guiding the interrupts
> and not the affinity itself. However, since the commit
>
>         e2e64a932556 "genirq: Set initial affinity in irq_set_affinity_hint()"
>
> irq_set_affinity_hint() also started applying the provided cpumask (if not
> NULL) as the affinity for the interrupts. The issue that this commit was
> trying to solve is to allow the drivers to enforce their affinity mask to
> distribute the interrupts across the CPUs such that they don't always end
> up on CPU0. This issue has been resolved within the irq subsystem since the
> commit
>
>         a0c9259dc4e1 "irq/matrix: Spread interrupts on allocation"
>
> Hence, there is no need for the drivers to overwrite the affinity to spread
> as it is dynamically performed at the time of allocation.
>
> Also, irq_set_affinity_hint() setting affinity unconditionally introduces
> issues for the drivers that only want to set their affinity_hint and not the
> affinity itself as for these driver interrupts the default_smp_affinity_mask
> is completely ignored (for detailed investigation please refer to [1]).
>
> Unfortunately reverting the commit e2e64a932556 is not an option at this
> point for two reasons [2]:
>
> - Several drivers for a valid reason (performance) rely on this API to
>   enforce their affinity mask
>
> - Until very recently this was the only exported interface that was
>   available
>
> To clear this out Thomas has come up with the following interfaces:
>
> - irq_set_affinity(): only sets affinity of an IRQ [3]
> - irq_update_affinity_hint(): Only sets the hint [4]
> - irq_set_affinity_and_hint(): Sets both affinity and the hint mask [4]
>
> The first API is already merged in the linux-next tree and the patch
> that introduces the other two interfaces are included with this patch-set.
>
> To move to the stage where we can safely get rid of the
> irq_set_affinity_hint(), which has been marked deprecated, we have to
> move all its consumers to these new interfaces. In this patch-set, I have
> done that for a few drivers and will hopefully try to move the remaining of
> them in the coming days.
>
> Testing
> -------
> In terms of testing, I have performed some basic testing on x86 to verify
> things such as the interrupts are evenly spread on all CPUs, hint mask is
> correctly set etc. for the drivers - i40e, iavf, mlx5, mlx4, ixgbe, i40iw
> and enic on top of:
>
>         git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
>
> So more testing is probably required for these and the drivers that I didn't
> test and any help will be much appreciated.
>
>
> Notes
> -----
> - I was told that i40iw driver is going to be replaced by irdma, however,
>   the new driver didn't land in Linus's tree yet. Once it does I will send
>   a follow up patch for that as well.
>
> - For the mpt3sas driver I decided to go with the usage of
>   irq_set_affinity_and_hint over irq_set_affinity based on my little
>   analysis of it and the megaraid driver. However, if we are sure that it
>   is not required then I can replace it with just irq_set_affinity as one
>   of its comment suggests.
>
>
> Change from v1 [5]
> ------------------
> - Fixed compilation error by adding the new interface definitions for cases
>   where CONFIG_SMP is not defined
>
> - Fixed function usage in megaraid_sas and removed unnecessary variable
>   (Robin Murphy)
>
> - Removed unwanted #if/endif from mlx4 (Leon Romanovsky)
>
> - Other indentation related fixes
>
>
> [1] https://lore.kernel.org/lkml/1a044a14-0884-eedb-5d30-28b4bec24b23@redhat.com/
> [2] https://lore.kernel.org/linux-pci/d1d5e797-49ee-4968-88c6-c07119343492@arm.com/
> [3] https://lore.kernel.org/linux-arm-kernel/20210518091725.046774792@linutronix.de/
> [4] https://lore.kernel.org/patchwork/patch/1434326/
> [5] https://lore.kernel.org/linux-scsi/20210617182242.8637-1-nitesh@redhat.com/
>
>
> Nitesh Narayan Lal (13):
>   iavf: Use irq_update_affinity_hint
>   i40e: Use irq_update_affinity_hint
>   scsi: megaraid_sas: Use irq_set_affinity_and_hint
>   scsi: mpt3sas: Use irq_set_affinity_and_hint
>   RDMA/i40iw: Use irq_update_affinity_hint
>   enic: Use irq_update_affinity_hint
>   be2net: Use irq_update_affinity_hint
>   ixgbe: Use irq_update_affinity_hint
>   mailbox: Use irq_update_affinity_hint
>   scsi: lpfc: Use irq_set_affinity
>   hinic: Use irq_set_affinity_and_hint
>   net/mlx5: Use irq_update_affinity_hint
>   net/mlx4: Use irq_update_affinity_hint
>
> Thomas Gleixner (1):
>   genirq: Provide new interfaces for affinity hints
>
>  drivers/infiniband/hw/i40iw/i40iw_main.c      |  4 +-
>  drivers/mailbox/bcm-flexrm-mailbox.c          |  4 +-
>  drivers/net/ethernet/cisco/enic/enic_main.c   |  8 +--
>  drivers/net/ethernet/emulex/benet/be_main.c   |  4 +-
>  drivers/net/ethernet/huawei/hinic/hinic_rx.c  |  4 +-
>  drivers/net/ethernet/intel/i40e/i40e_main.c   |  8 +--
>  drivers/net/ethernet/intel/iavf/iavf_main.c   |  8 +--
>  drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 10 ++--
>  drivers/net/ethernet/mellanox/mlx4/eq.c       |  8 ++-
>  .../net/ethernet/mellanox/mlx5/core/pci_irq.c |  6 +--
>  drivers/scsi/lpfc/lpfc_init.c                 |  4 +-
>  drivers/scsi/megaraid/megaraid_sas_base.c     | 27 +++++-----
>  drivers/scsi/mpt3sas/mpt3sas_base.c           | 21 ++++----
>  include/linux/interrupt.h                     | 53 ++++++++++++++++++-
>  kernel/irq/manage.c                           |  8 +--
>  15 files changed, 113 insertions(+), 64 deletions(-)
>
> --
>
>

Gentle ping.
Any comments or suggestions on any of the patches included in this series?

-- 
Thanks
Nitesh


  parent reply	other threads:[~2021-07-08 19:24 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-29 15:27 [PATCH v2 00/14] genirq: Cleanup the usage of irq_set_affinity_hint Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 01/14] genirq: Provide new interfaces for affinity hints Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 02/14] iavf: Use irq_update_affinity_hint Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 03/14] i40e: " Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 04/14] scsi: megaraid_sas: Use irq_set_affinity_and_hint Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 05/14] scsi: mpt3sas: " Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 06/14] RDMA/i40iw: Use irq_update_affinity_hint Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 07/14] enic: " Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 08/14] be2net: " Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 09/14] ixgbe: " Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 10/14] mailbox: " Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 11/14] scsi: lpfc: Use irq_set_affinity Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 12/14] hinic: Use irq_set_affinity_and_hint Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 13/14] net/mlx5: Use irq_update_affinity_hint Nitesh Narayan Lal
2021-06-29 15:27 ` [PATCH v2 14/14] net/mlx4: " Nitesh Narayan Lal
2021-07-08 19:24 ` Nitesh Lal [this message]
2021-07-11 11:31   ` [PATCH v2 00/14] genirq: Cleanup the usage of irq_set_affinity_hint Leon Romanovsky
2021-07-12 14:13     ` Nitesh Lal
2021-07-12 21:27     ` Nitesh Lal
2021-07-13  5:01       ` Leon Romanovsky
2021-07-13 13:29         ` Nitesh Lal

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAFki+LnUGiEE-7Uf-x8-TQZYZ+3Migrr=81gGLYszxaK-6A9WQ@mail.gmail.com' \
    --to=nilal@redhat.com \
    --cc=abelits@marvell.com \
    --cc=ahleihel@redhat.com \
    --cc=ahs3@redhat.com \
    --cc=ajit.khaparde@broadcom.com \
    --cc=akpm@linux-foundation.org \
    --cc=benve@cisco.com \
    --cc=bhelgaas@google.com \
    --cc=borisp@nvidia.com \
    --cc=chris.friesen@windriver.com \
    --cc=davem@davemloft.net \
    --cc=dick.kennedy@broadcom.com \
    --cc=faisal.latif@intel.com \
    --cc=frederic@kernel.org \
    --cc=govind@gmx.com \
    --cc=james.smart@broadcom.com \
    --cc=jassisinghbrar@gmail.com \
    --cc=jbrandeb@kernel.org \
    --cc=jesse.brandeburg@intel.com \
    --cc=jkc@redhat.com \
    --cc=juri.lelli@redhat.com \
    --cc=kashyap.desai@broadcom.com \
    --cc=kheib@redhat.com \
    --cc=leonro@nvidia.com \
    --cc=linux-api@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=maz@kernel.org \
    --cc=mingo@kernel.org \
    --cc=mtosatti@redhat.com \
    --cc=mustafa.ismail@intel.com \
    --cc=netdev@vger.kernel.org \
    --cc=nhorman@tuxdriver.com \
    --cc=nitesh@redhat.com \
    --cc=peterz@infradead.org \
    --cc=pjwaskiewicz@gmail.com \
    --cc=robin.murphy@arm.com \
    --cc=rostedt@goodmis.org \
    --cc=rppt@linux.vnet.ibm.com \
    --cc=saeedm@nvidia.com \
    --cc=sassmann@redhat.com \
    --cc=sathya.prakash@broadcom.com \
    --cc=sfr@canb.auug.org.au \
    --cc=shiraz.saleem@intel.com \
    --cc=shivasharan.srikanteshwara@broadcom.com \
    --cc=somnath.kotur@broadcom.com \
    --cc=sreekanth.reddy@broadcom.com \
    --cc=sriharsha.basavapatna@broadcom.com \
    --cc=stephen@networkplumber.org \
    --cc=suganath-prabu.subramani@broadcom.com \
    --cc=sumit.saxena@broadcom.com \
    --cc=tariqt@nvidia.com \
    --cc=tatyana.e.nikolova@intel.com \
    --cc=tglx@linutronix.de \
    --cc=thenzl@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).