linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Nitesh Narayan Lal <nitesh@redhat.com>
To: linux-kernel@vger.kernel.org, frederic@kernel.org,
	mtosatti@redhat.com, sassmann@redhat.com,
	jeffrey.t.kirsher@intel.com, jacob.e.keller@intel.com,
	jlelli@redhat.com
Subject: [PATCH v1 0/1] limit the i40e msix vectors based on housekeeping CPUs
Date: Mon, 15 Jun 2020 16:21:24 -0400	[thread overview]
Message-ID: <20200615202125.27831-1-nitesh@redhat.com> (raw)

Issue
=====
With the current implementation at the time of i40e_init_msix(), i40e
creates vectors only based on the number of online CPUs. This would
be problematic for RT setup that includes a large number of isolated
but very few housekeeping CPUs. This is because in those setups
an attempt to move all IRQs from isolated to housekeeping CPUs may
easily fail due to per CPU vector limit.

Setup For The Issue
===================
I have triggered this issue on a setup that had a total of 72
cores among which 68 were isolated and only 4 were left for
housekeeping tasks. I was using tuned's realtime-virtual-host profile
to configure the system. However, Tuned reported the error message
'Failed to set SMP affinity of IRQ xxx to '00000040,00000010,00000005':
[Errno 28] No space left on the device' for several IRQs in tuned.log.
Note: There were other IRQs as well pinned to the housekeeping CPUs that
      were generated by other drivers.

Fix
===
- In this proposed fix I have replaced num_online_cpus in i40e_init_msix()
  with the number of housekeeping CPUs.
- The reason why I chose to include both HK_FLAG_DOMAIN & HK_FLAG_WQ is
  because we would also need IRQ isolation with something like systemd's
  CPU affinity.


Testing
=======
To test this change I had added a tracepoint in i40e_init_msix() to
find the number of CPUs derived for vector creation with and without
tuned's realtime-virtual-host profile. As per expectation with the profile
applied I was only getting the number of housekeeping CPUs and all
available CPUs without it.


Nitesh Narayan Lal (1):
  i40e: limit the msix vectors based on housekeeping CPUs

 drivers/net/ethernet/intel/i40e/i40e_main.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

-- 


             reply	other threads:[~2020-06-15 20:21 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-15 20:21 Nitesh Narayan Lal [this message]
2020-06-15 20:21 ` [Patch v1] i40e: limit the " Nitesh Narayan Lal
2020-06-15 20:48   ` Keller, Jacob E
2020-06-15 20:55     ` Nitesh Narayan Lal
2020-06-16  8:03   ` Christoph Hellwig
2020-06-16 17:29     ` Nitesh Narayan Lal
2020-06-26 20:11     ` Nitesh Narayan Lal

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200615202125.27831-1-nitesh@redhat.com \
    --to=nitesh@redhat.com \
    --cc=frederic@kernel.org \
    --cc=jacob.e.keller@intel.com \
    --cc=jeffrey.t.kirsher@intel.com \
    --cc=jlelli@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mtosatti@redhat.com \
    --cc=sassmann@redhat.com \
    --subject='Re: [PATCH v1 0/1] limit the i40e msix vectors based on housekeeping CPUs' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).