linux-scsi.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Nitesh Narayan Lal <nitesh@redhat.com>
To: linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
	intel-wired-lan@lists.osuosl.org, netdev@vger.kernel.org,
	linux-api@vger.kernel.org, linux-pci@vger.kernel.org,
	tglx@linutronix.de, jesse.brandeburg@intel.com,
	robin.murphy@arm.com, mtosatti@redhat.com, mingo@kernel.org,
	jbrandeb@kernel.org, frederic@kernel.org, juri.lelli@redhat.com,
	abelits@marvell.com, bhelgaas@google.com, rostedt@goodmis.org,
	peterz@infradead.org, davem@davemloft.net,
	akpm@linux-foundation.org, sfr@canb.auug.org.au,
	stephen@networkplumber.org, rppt@linux.vnet.ibm.com,
	chris.friesen@windriver.com, maz@kernel.org,
	nhorman@tuxdriver.com, pjwaskiewicz@gmail.com,
	sassmann@redhat.com, thenzl@redhat.com,
	kashyap.desai@broadcom.com, sumit.saxena@broadcom.com,
	shivasharan.srikanteshwara@broadcom.com,
	sathya.prakash@broadcom.com, sreekanth.reddy@broadcom.com,
	suganath-prabu.subramani@broadcom.com, james.smart@broadcom.com,
	dick.kennedy@broadcom.com, jkc@redhat.com,
	faisal.latif@intel.com, shiraz.saleem@intel.com,
	tariqt@nvidia.com, ahleihel@redhat.com, kheib@redhat.com,
	borisp@nvidia.com, saeedm@nvidia.com, benve@cisco.com,
	govind@gmx.com, jassisinghbrar@gmail.com,
	ajit.khaparde@broadcom.com, sriharsha.basavapatna@broadcom.com,
	somnath.kotur@broadcom.com, nilal@redhat.com,
	tatyana.e.nikolova@intel.com, mustafa.ismail@intel.com,
	ahs3@redhat.com, leonro@nvidia.com,
	chandrakanth.patil@broadcom.com
Subject: [PATCH v3 00/14] genirq: Cleanup the usage of irq_set_affinity_hint
Date: Tue, 13 Jul 2021 17:14:48 -0400	[thread overview]
Message-ID: <20210713211502.464259-1-nitesh@redhat.com> (raw)

The drivers currently rely on irq_set_affinity_hint() to either set the
affinity_hint that is consumed by the userspace and/or to enforce a custom
affinity.

irq_set_affinity_hint() as the name suggests is originally introduced to
only set the affinity_hint to help the userspace in guiding the interrupts
and not the affinity itself. However, since the commit

        e2e64a932556 "genirq: Set initial affinity in irq_set_affinity_hint()"

irq_set_affinity_hint() also started applying the provided cpumask (if not
NULL) as the affinity for the interrupts. The issue that this commit was
trying to solve is to allow the drivers to enforce their affinity mask to
distribute the interrupts across the CPUs such that they don't always end
up on CPU0. This issue has been resolved within the irq subsystem since the
commit

        a0c9259dc4e1 "irq/matrix: Spread interrupts on allocation"

Hence, there is no need for the drivers to overwrite the affinity to spread
as it is dynamically performed at the time of allocation.

Also, irq_set_affinity_hint() setting affinity unconditionally introduces
issues for the drivers that only want to set their affinity_hint and not the
affinity itself as for these driver interrupts the default_smp_affinity_mask
is completely ignored (for detailed investigation please refer to [1]).

Unfortunately reverting the commit e2e64a932556 is not an option at this
point for two reasons [2]:

- Several drivers for a valid reason (performance) rely on this API to
  enforce their affinity mask

- Until very recently this was the only exported interface that was
  available

To clear this out Thomas has come up with the following interfaces:

- irq_set_affinity(): only sets affinity of an IRQ [3]
- irq_update_affinity_hint(): Only sets the hint [4]
- irq_set_affinity_and_hint(): Sets both affinity and the hint mask [4]

The first API is already merged in the linus's tree and the patch
that introduces the other two interfaces is included with this patch-set.

To move to the stage where we can safely get rid of the
irq_set_affinity_hint(), which has been marked deprecated, we have to
move all its consumers to these new interfaces. In this patch-set, I have
done that for a few drivers and will hopefully try to move the remaining of
them in the coming days.

Testing
-------
In terms of testing, I have performed some basic testing on x86 to verify
things such as the interrupts are evenly spread on all CPUs, hint mask is
correctly set etc. for the drivers - i40e, iavf, mlx5, mlx4, ixgbe, and
enic on top of:

        git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git

So more testing is probably required for these and the drivers that I didn't
test and any help will be much appreciated.


Notes
-----
- For the mpt3sas driver I decided to go with the usage of
  irq_set_affinity_and_hint over irq_set_affinity based on my analysis of
  it and the megaraid driver. However, if we are sure that it is not
  required then I can replace it with just irq_set_affinity as one of its
  comment suggests.

Change from v2 [5]
------------------

- Rebased on top of 5.14-rc1 (Leon Romanovsky)
  + After discussion with Leon [6], made changes in the mlx5 patch to use
    irq_set_affinity_and_hint over irq_update_affinity_hint
  + i40iw is replaced with irdma driver, hence made the respective changes
    in irdma (also replcaed irq_update_affinity_hint with
    irq_set_affinity_and_hint).

Change from v1 [7]
------------------
- Fixed compilation error by adding the new interface definitions for cases
  where CONFIG_SMP is not defined

- Fixed function usage in megaraid_sas and removed unnecessary variable
  (Robin Murphy)

- Removed unwanted #if/endif from mlx4 (Leon Romanovsky)

- Other indentation related fixes

 
[1] https://lore.kernel.org/lkml/1a044a14-0884-eedb-5d30-28b4bec24b23@redhat.com/
[2] https://lore.kernel.org/linux-pci/d1d5e797-49ee-4968-88c6-c07119343492@arm.com/
[3] https://lore.kernel.org/linux-arm-kernel/20210518091725.046774792@linutronix.de/
[4] https://lore.kernel.org/patchwork/patch/1434326/
[5] https://lore.kernel.org/lkml/20210629152746.2953364-1-nitesh@redhat.com/
[6] https://lore.kernel.org/lkml/YO0eKv2GJcADQTHH@unreal/
[7] https://lore.kernel.org/linux-scsi/20210617182242.8637-1-nitesh@redhat.com/


Nitesh Narayan Lal (13):
  iavf: Use irq_update_affinity_hint
  i40e: Use irq_update_affinity_hint
  scsi: megaraid_sas: Use irq_set_affinity_and_hint
  scsi: mpt3sas: Use irq_set_affinity_and_hint
  RDMA/irdma: Use irq_set_affinity_and_hint
  enic: Use irq_update_affinity_hint
  be2net: Use irq_update_affinity_hint
  ixgbe: Use irq_update_affinity_hint
  mailbox: Use irq_update_affinity_hint
  scsi: lpfc: Use irq_set_affinity
  hinic: Use irq_set_affinity_and_hint
  net/mlx5: Use irq_set_affinity_and_hint
  net/mlx4: Use irq_update_affinity_hint

Thomas Gleixner (1):
  genirq: Provide new interfaces for affinity hints

 drivers/infiniband/hw/irdma/hw.c              |  4 +-
 drivers/mailbox/bcm-flexrm-mailbox.c          |  4 +-
 drivers/net/ethernet/cisco/enic/enic_main.c   |  8 +--
 drivers/net/ethernet/emulex/benet/be_main.c   |  4 +-
 drivers/net/ethernet/huawei/hinic/hinic_rx.c  |  4 +-
 drivers/net/ethernet/intel/i40e/i40e_main.c   |  8 +--
 drivers/net/ethernet/intel/iavf/iavf_main.c   |  8 +--
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 10 ++--
 drivers/net/ethernet/mellanox/mlx4/eq.c       |  8 ++-
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c |  8 +--
 drivers/scsi/lpfc/lpfc_init.c                 |  4 +-
 drivers/scsi/megaraid/megaraid_sas_base.c     | 27 +++++-----
 drivers/scsi/mpt3sas/mpt3sas_base.c           | 21 ++++----
 include/linux/interrupt.h                     | 53 ++++++++++++++++++-
 kernel/irq/manage.c                           |  8 +--
 15 files changed, 114 insertions(+), 65 deletions(-)

-- 



             reply	other threads:[~2021-07-13 21:15 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-13 21:14 Nitesh Narayan Lal [this message]
2021-07-13 21:14 ` [PATCH v3 01/14] genirq: Provide new interfaces for affinity hints Nitesh Narayan Lal
2021-07-13 21:14 ` [PATCH v3 02/14] iavf: Use irq_update_affinity_hint Nitesh Narayan Lal
2021-07-13 21:51   ` Jesse Brandeburg
2021-07-13 21:14 ` [PATCH v3 03/14] i40e: " Nitesh Narayan Lal
2021-07-13 21:52   ` Jesse Brandeburg
2021-07-13 21:14 ` [PATCH v3 04/14] scsi: megaraid_sas: Use irq_set_affinity_and_hint Nitesh Narayan Lal
2021-07-13 21:14 ` [PATCH v3 05/14] scsi: mpt3sas: " Nitesh Narayan Lal
2021-07-13 21:14 ` [PATCH v3 06/14] RDMA/irdma: " Nitesh Narayan Lal
2021-07-14 12:03   ` Leon Romanovsky
2021-07-14 12:56     ` Nitesh Lal
2021-07-14 13:02       ` Leon Romanovsky
2021-07-15 13:25         ` Nitesh Lal
2021-07-13 21:14 ` [PATCH v3 07/14] enic: Use irq_update_affinity_hint Nitesh Narayan Lal
2021-07-13 21:14 ` [PATCH v3 08/14] be2net: " Nitesh Narayan Lal
2021-07-13 21:14 ` [PATCH v3 09/14] ixgbe: " Nitesh Narayan Lal
2021-07-13 22:01   ` Jesse Brandeburg
2021-07-13 21:14 ` [PATCH v3 10/14] mailbox: " Nitesh Narayan Lal
2021-07-13 21:14 ` [PATCH v3 11/14] scsi: lpfc: Use irq_set_affinity Nitesh Narayan Lal
2021-07-13 21:15 ` [PATCH v3 12/14] hinic: Use irq_set_affinity_and_hint Nitesh Narayan Lal
2021-07-13 21:15 ` [PATCH v3 13/14] net/mlx5: " Nitesh Narayan Lal
2021-07-14 12:01   ` Leon Romanovsky
2021-07-13 21:15 ` [PATCH v3 14/14] net/mlx4: Use irq_update_affinity_hint Nitesh Narayan Lal
2021-07-14 11:19   ` Tariq Toukan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210713211502.464259-1-nitesh@redhat.com \
    --to=nitesh@redhat.com \
    --cc=abelits@marvell.com \
    --cc=ahleihel@redhat.com \
    --cc=ahs3@redhat.com \
    --cc=ajit.khaparde@broadcom.com \
    --cc=akpm@linux-foundation.org \
    --cc=benve@cisco.com \
    --cc=bhelgaas@google.com \
    --cc=borisp@nvidia.com \
    --cc=chandrakanth.patil@broadcom.com \
    --cc=chris.friesen@windriver.com \
    --cc=davem@davemloft.net \
    --cc=dick.kennedy@broadcom.com \
    --cc=faisal.latif@intel.com \
    --cc=frederic@kernel.org \
    --cc=govind@gmx.com \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=james.smart@broadcom.com \
    --cc=jassisinghbrar@gmail.com \
    --cc=jbrandeb@kernel.org \
    --cc=jesse.brandeburg@intel.com \
    --cc=jkc@redhat.com \
    --cc=juri.lelli@redhat.com \
    --cc=kashyap.desai@broadcom.com \
    --cc=kheib@redhat.com \
    --cc=leonro@nvidia.com \
    --cc=linux-api@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=maz@kernel.org \
    --cc=mingo@kernel.org \
    --cc=mtosatti@redhat.com \
    --cc=mustafa.ismail@intel.com \
    --cc=netdev@vger.kernel.org \
    --cc=nhorman@tuxdriver.com \
    --cc=nilal@redhat.com \
    --cc=peterz@infradead.org \
    --cc=pjwaskiewicz@gmail.com \
    --cc=robin.murphy@arm.com \
    --cc=rostedt@goodmis.org \
    --cc=rppt@linux.vnet.ibm.com \
    --cc=saeedm@nvidia.com \
    --cc=sassmann@redhat.com \
    --cc=sathya.prakash@broadcom.com \
    --cc=sfr@canb.auug.org.au \
    --cc=shiraz.saleem@intel.com \
    --cc=shivasharan.srikanteshwara@broadcom.com \
    --cc=somnath.kotur@broadcom.com \
    --cc=sreekanth.reddy@broadcom.com \
    --cc=sriharsha.basavapatna@broadcom.com \
    --cc=stephen@networkplumber.org \
    --cc=suganath-prabu.subramani@broadcom.com \
    --cc=sumit.saxena@broadcom.com \
    --cc=tariqt@nvidia.com \
    --cc=tatyana.e.nikolova@intel.com \
    --cc=tglx@linutronix.de \
    --cc=thenzl@redhat.com \
    --subject='Re: [PATCH v3 00/14] genirq: Cleanup the usage of irq_set_affinity_hint' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).