linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Nitesh Narayan Lal <nitesh@redhat.com>
To: Thomas Gleixner <tglx@linutronix.de>,
	Marcelo Tosatti <mtosatti@redhat.com>,
	Jesse Brandeburg <jesse.brandeburg@intel.com>
Cc: Robin Murphy <robin.murphy@arm.com>,
	linux-kernel@vger.kernel.org, linux-api@vger.kernel.org,
	frederic@kernel.org, juri.lelli@redhat.com, abelits@marvell.com,
	bhelgaas@google.com, linux-pci@vger.kernel.org,
	rostedt@goodmis.org, mingo@kernel.org, peterz@infradead.org,
	davem@davemloft.net, akpm@linux-foundation.org,
	sfr@canb.auug.org.au, stephen@networkplumber.org,
	rppt@linux.vnet.ibm.com, jinyuqi@huawei.com,
	zhangshaokun@hisilicon.com
Subject: Re: [Patch v4 1/3] lib: Restrict cpumask_local_spread to houskeeping CPUs
Date: Thu, 4 Mar 2021 13:15:31 -0500	[thread overview]
Message-ID: <07c04bc7-27f0-9c07-9f9e-2d1a450714ef@redhat.com> (raw)
In-Reply-To: <7780ae60-efbd-2902-caaa-0249a1f277d9@redhat.com>


On 2/11/21 10:55 AM, Nitesh Narayan Lal wrote:
> On 2/6/21 7:43 PM, Nitesh Narayan Lal wrote:
>> On 2/5/21 5:23 PM, Thomas Gleixner wrote:
>>> On Thu, Feb 04 2021 at 14:17, Nitesh Narayan Lal wrote:
>>>> On 2/4/21 2:06 PM, Marcelo Tosatti wrote:
>>>>>>> How about adding a new flag for isolcpus instead?
>>>>>>>
>>>>>> Do you mean a flag based on which we can switch the affinity mask to
>>>>>> housekeeping for all the devices at the time of IRQ distribution?
>>>>> Yes a new flag for isolcpus. HK_FLAG_IRQ_SPREAD or some better name.
>>>> Does sounds like a nice idea to explore, lets see what Thomas thinks about it.
> <snip>
>
>>>> When the affinity mask of the interrupt at the time when it is actually
>>>> requested contains an isolated CPU then nothing prevents the kernel from
>>>> steering it at an isolated CPU. But that has absolutely nothing to do
>>>> with that spreading thingy.
>>>>
>>>> The only difference which this change makes is the fact that the
>>>> affinity hint changes. Nothing else.
>>>>
>> Thanks for the detailed explanation.
>>
>> Before I posted this patch, I was doing some debugging on a setup where I
>> was observing some latency issues due to the iavf IRQs that were pinned on
>> the isolated CPUs.
>>
>> Based on some initial traces I had this impression that the affinity hint
>> or cpumask_local_spread was somehow playing a role in deciding the affinity
>> mask of these IRQs. Although, that does look incorrect after going through
>> your explanation.
>> For some reason, with a kernel that had this patch when I tried creating
>> VFs iavf IRQs always ended up on the HK CPUs.
>>
>> The reasoning for the above is still not very clear to me. I will investigate
>> this further to properly understand this behavior.
>>
>>
> After a little more digging, I found out why cpumask_local_spread change
> affects the general/initial smp_affinity for certain device IRQs.
>
> After the introduction of the commit:
>
>     e2e64a932 genirq: Set initial affinity in irq_set_affinity_hint()
>

Continuing the conversation about the above commit and adding Jesse.
I was trying to understand the problem that the commit message explains
"The default behavior of the kernel is somewhat undesirable as all
requested interrupts end up on CPU0 after registration.", I have also been
trying to reproduce this behavior without the patch but I failed in doing
so, maybe because I am missing something here.

@Jesse Can you please explain? FWIU IRQ affinity should be decided based on
the default affinity mask.

The problem with the commit is that when we overwrite the affinity mask
based on the hinting mask we completely ignore the default SMP affinity
mask. If we do want to overwrite the affinity based on the hint mask we
should atleast consider the default SMP affinity.

-- 
Thanks
Nitesh


  reply	other threads:[~2021-03-04 18:18 UTC|newest]

Thread overview: 48+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-25 22:34 [PATCH v4 0/3] Preventing job distribution to isolated CPUs Nitesh Narayan Lal
2020-06-25 22:34 ` [Patch v4 1/3] lib: Restrict cpumask_local_spread to houskeeping CPUs Nitesh Narayan Lal
2020-06-29 16:11   ` Nitesh Narayan Lal
2020-07-01  0:32     ` Andrew Morton
2020-07-01  0:47       ` Nitesh Narayan Lal
2021-01-27 11:57   ` Robin Murphy
2021-01-27 12:19     ` Marcelo Tosatti
2021-01-27 12:36       ` Robin Murphy
2021-01-27 13:09         ` Marcelo Tosatti
2021-01-27 13:49           ` Robin Murphy
2021-01-27 14:16           ` Nitesh Narayan Lal
2021-01-28 15:56           ` Thomas Gleixner
2021-01-28 16:33             ` Marcelo Tosatti
     [not found]             ` <02ac9d85-7ddd-96da-1252-4663feea7c9f@marvell.com>
2021-02-01 17:50               ` [EXT] " Marcelo Tosatti
2021-01-28 16:02       ` Thomas Gleixner
2021-01-28 16:59         ` Marcelo Tosatti
2021-01-28 17:35           ` Nitesh Narayan Lal
2021-01-28 20:01           ` Thomas Gleixner
     [not found]             ` <d2a4dc97-a9ed-e0e7-3b9c-c56ae46f6608@redhat.com>
     [not found]               ` <20210129142356.GB40876@fuller.cnet>
2021-01-29 17:34                 ` [EXT] " Alex Belits
     [not found]                 ` <18584612-868c-0f88-5de2-dc93c8638816@redhat.com>
2021-02-05 19:56                   ` Thomas Gleixner
2021-02-04 18:15             ` Marcelo Tosatti
2021-02-04 18:47               ` Nitesh Narayan Lal
2021-02-04 19:06                 ` Marcelo Tosatti
2021-02-04 19:17                   ` Nitesh Narayan Lal
2021-02-05 22:23                     ` Thomas Gleixner
2021-02-05 22:26                       ` Thomas Gleixner
2021-02-07  0:43                       ` Nitesh Narayan Lal
2021-02-11 15:55                         ` Nitesh Narayan Lal
2021-03-04 18:15                           ` Nitesh Narayan Lal [this message]
     [not found]                             ` <faa8d84e-db67-7fbe-891e-f4987f106b20@marvell.com>
2021-03-04 23:23                               ` [EXT] " Nitesh Narayan Lal
2021-04-06 17:22                             ` Jesse Brandeburg
2021-04-07 15:18                               ` Nitesh Narayan Lal
2021-04-08 18:49                                 ` Nitesh Narayan Lal
2021-04-14 16:11                                 ` Jesse Brandeburg
2021-04-15 22:11                                   ` Nitesh Narayan Lal
2021-04-29 21:44                                     ` Nitesh Lal
2021-04-30  1:48                                       ` Jesse Brandeburg
2021-04-30 13:10                                         ` Nitesh Lal
2021-04-30  7:10                                       ` Thomas Gleixner
2021-04-30 16:14                                         ` Nitesh Lal
2021-04-30 18:21                                           ` Thomas Gleixner
2021-04-30 21:07                                             ` Nitesh Lal
2021-05-01  2:21                                               ` Jesse Brandeburg
2021-05-03 13:15                                                 ` Nitesh Lal
2020-06-25 22:34 ` [Patch v4 2/3] PCI: Restrict probe functions to housekeeping CPUs Nitesh Narayan Lal
2020-06-25 22:34 ` [Patch v4 3/3] net: Restrict receive packets queuing " Nitesh Narayan Lal
2020-06-26 11:14   ` Peter Zijlstra
2020-06-26 17:20     ` David Miller

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=07c04bc7-27f0-9c07-9f9e-2d1a450714ef@redhat.com \
    --to=nitesh@redhat.com \
    --cc=abelits@marvell.com \
    --cc=akpm@linux-foundation.org \
    --cc=bhelgaas@google.com \
    --cc=davem@davemloft.net \
    --cc=frederic@kernel.org \
    --cc=jesse.brandeburg@intel.com \
    --cc=jinyuqi@huawei.com \
    --cc=juri.lelli@redhat.com \
    --cc=linux-api@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=mtosatti@redhat.com \
    --cc=peterz@infradead.org \
    --cc=robin.murphy@arm.com \
    --cc=rostedt@goodmis.org \
    --cc=rppt@linux.vnet.ibm.com \
    --cc=sfr@canb.auug.org.au \
    --cc=stephen@networkplumber.org \
    --cc=tglx@linutronix.de \
    --cc=zhangshaokun@hisilicon.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).