Re: [PATCH v2 00/14] genirq: Cleanup the usage of irq_set_affinity_hint

From: Leon Romanovsky <leon@kernel.org>
To: Nitesh Lal <nilal@redhat.com>
Cc: linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
	netdev@vger.kernel.org, linux-api@vger.kernel.org,
	linux-pci@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>,
	Jesse Brandeburg <jesse.brandeburg@intel.com>,
	Robin Murphy <robin.murphy@arm.com>,
	Marcelo Tosatti <mtosatti@redhat.com>,
	Ingo Molnar <mingo@kernel.org>,
	jbrandeb@kernel.org, frederic@kernel.org,
	Juri Lelli <juri.lelli@redhat.com>,
	Alex Belits <abelits@marvell.com>,
	Bjorn Helgaas <bhelgaas@google.com>,
	rostedt@goodmis.org, peterz@infradead.org, davem@davemloft.net,
	akpm@linux-foundation.org, sfr@canb.auug.org.au,
	stephen@networkplumber.org, rppt@linux.vnet.ibm.com,
	chris.friesen@windriver.com, Marc Zyngier <maz@kernel.org>,
	Neil Horman <nhorman@tuxdriver.com>,
	pjwaskiewicz@gmail.com, Stefan Assmann <sassmann@redhat.com>,
	Tomas Henzl <thenzl@redhat.com>,
	kashyap.desai@broadcom.com,
	Sumit Saxena <sumit.saxena@broadcom.com>,
	shivasharan.srikanteshwara@broadcom.com,
	sathya.prakash@broadcom.com,
	Sreekanth Reddy <sreekanth.reddy@broadcom.com>,
	suganath-prabu.subramani@broadcom.com, james.smart@broadcom.com,
	dick.kennedy@broadcom.com, Ken Cox <jkc@redhat.com>,
	faisal.latif@intel.com, shiraz.saleem@intel.com,
	tariqt@nvidia.com, Alaa Hleihel <ahleihel@redhat.com>,
	Kamal Heib <kheib@redhat.com>,
	borisp@nvidia.com, saeedm@nvidia.com, benve@cisco.com,
	govind@gmx.com, jassisinghbrar@gmail.com,
	ajit.khaparde@broadcom.com, sriharsha.basavapatna@broadcom.com,
	somnath.kotur@broadcom.com, "Nikolova,
	Tatyana E" <tatyana.e.nikolova@intel.com>,
	"Ismail, Mustafa" <mustafa.ismail@intel.com>,
	Al Stone <ahs3@redhat.com>
Subject: Re: [PATCH v2 00/14] genirq: Cleanup the usage of irq_set_affinity_hint
Date: Tue, 13 Jul 2021 08:01:30 +0300	[thread overview]
Message-ID: <YO0eKv2GJcADQTHH@unreal> (raw)
In-Reply-To: <CAFki+L=FYOTQ1+-MHWmTuA6ZxTUcZA9t41HRL2URYgv03oFbDg@mail.gmail.com>

On Mon, Jul 12, 2021 at 05:27:05PM -0400, Nitesh Lal wrote:
> Hi Leon,
> 
> On Sun, Jul 11, 2021 at 7:32 AM Leon Romanovsky <leonro@nvidia.com> wrote:
> >
> > On Thu, Jul 08, 2021 at 03:24:20PM -0400, Nitesh Lal wrote:
> > > On Tue, Jun 29, 2021 at 11:28 AM Nitesh Narayan Lal <nitesh@redhat.com> wrote:
> >
> > <...>
> >
> > > >
> > > >  drivers/infiniband/hw/i40iw/i40iw_main.c      |  4 +-
> > > >  drivers/mailbox/bcm-flexrm-mailbox.c          |  4 +-
> > > >  drivers/net/ethernet/cisco/enic/enic_main.c   |  8 +--
> > > >  drivers/net/ethernet/emulex/benet/be_main.c   |  4 +-
> > > >  drivers/net/ethernet/huawei/hinic/hinic_rx.c  |  4 +-
> > > >  drivers/net/ethernet/intel/i40e/i40e_main.c   |  8 +--
> > > >  drivers/net/ethernet/intel/iavf/iavf_main.c   |  8 +--
> > > >  drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 10 ++--
> > > >  drivers/net/ethernet/mellanox/mlx4/eq.c       |  8 ++-
> > > >  .../net/ethernet/mellanox/mlx5/core/pci_irq.c |  6 +--
> > > >  drivers/scsi/lpfc/lpfc_init.c                 |  4 +-
> > > >  drivers/scsi/megaraid/megaraid_sas_base.c     | 27 +++++-----
> > > >  drivers/scsi/mpt3sas/mpt3sas_base.c           | 21 ++++----
> > > >  include/linux/interrupt.h                     | 53 ++++++++++++++++++-
> > > >  kernel/irq/manage.c                           |  8 +--
> > > >  15 files changed, 113 insertions(+), 64 deletions(-)
> > > >
> > > > --
> > > >
> > > >
> > >
> > > Gentle ping.
> > > Any comments or suggestions on any of the patches included in this series?
> >
> > Please wait for -rc1, rebase and resend.
> > At least i40iw was deleted during merge window.
> >
> 
> In -rc1 some non-trivial mlx5 changes also went in.  I was going through
> these changes and it seems after your patch
> 
> e4e3f24b822f: ("net/mlx5: Provide cpumask at EQ creation phase")
> 
> we do want to control the affinity for the mlx5 interrupts from the driver.
> Is that correct? 

We would like to create devices with correct affinity from the
beginning. For this, we will introduce extension to devlink to control
affinity that will be used prior initialization sequence.

Currently, netdev users who don't want irqbalance are digging into
their procfs, reconfigure affinity on already existing devices and
hope for the best. 

This is even more cumbersome for the SIOV use case, where every physical
NIC PCI device will/can create thousands of lightweights netdevs that will
be forwarded to the containers later. These containers are limited to known
CPU cores, so no reason do not limit netdev device too.

The same goes for other sub-functions of that PCI device, like RDMA,
vdpa e.t.c.

> This would mean that we should use irq_set_affinity_and_hint() instead
> of irq_update_affinity_hint().

I think so.

Thanks

> 
> -- 
> Thanks
> Nitesh
> 