Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt

[John Garry <john.garry@huawei.com>]

Hi Marc,

>> JFYI, we're still testing this and the patch itself seems to work as
>> intended.
>>
>> Here's the kernel log if you just want to see how the interrupts are
>> getting assigned:
>> https://pastebin.com/hh3r810g
> 
> It is a bit hard to make sense of this dump, specially on such a wide
> machine (I want one!) 

So do I :) That's the newer "D06CS" board.

without really knowing the topology of the system.

So it's 2x socket, each socket has 2x CPU dies, and each die has 6 
clusters of 4 CPUs, which gives 96 in total.

> 
>> For me, I did get a performance boost for NVMe testing, but my
>> colleague Xiang Chen saw a drop for our storage test of interest  -
>> that's the HiSi SAS controller. We're trying to make sense of it now.
> 
> One of the difference is that with this patch, the initial affinity
> is picked inside the NUMA node that matches the ITS. 

Is that even for managed interrupts? We're testing the storage 
controller which uses managed interrupts. I should have made that clearer.

In your case,
> that's either node 0 or 2. But it is unclear whether which CPUs these
> map to.
> 
> Given that I see interrupts mapped to CPUs 0-23 on one side, and 48-71
> on the other, it looks like half of your machine gets starved, 

Seems that way.

So this is a mystery to me:

[   23.584192] picked CPU62 IRQ147

147:          0          0          0          0          0          0 
        0          0          0          0          0          0 
  0          0          0          0          0          0          0 
       0          0          0          0          0          0 
0          0          0          0          0          0          0 
     0          0          0          0          0          0          0 
          0          0          0          0          0          0 
    0          0          0          0          0          0          0 
         0          0          0          0          0          0 
   0          0          0          0          0          0          0 
        0          0          0          0          0          0 
  0          0          0          0          0          0          0 
       0          0          0          0          0          0 
0          0          0          0          0          0          0 
     0          0          0          0          0   ITS-MSI 94404626 
Edge      hisi_sas_v3_hw cq


and

[   25.896728] picked CPU62 IRQ183

183:          0          0          0          0          0          0 
        0          0          0          0          0          0 
  0          0          0          0          0          0          0 
       0          0          0          0          0          0 
0          0          0          0          0          0          0 
     0          0          0          0          0          0          0 
          0          0          0          0          0          0 
    0          0          0          0          0          0          0 
         0          0          0          0          0          0 
   0          0          0          0          0          0          0 
        0          0          0          0          0          0 
  0          0          0          0          0          0          0 
       0          0          0          0          0          0 
0          0          0          0          0          0          0 
     0          0          0          0          0   ITS-MSI 94437398 
Edge      hisi_sas_v3_hw cq


But mpstat reports for CPU62:

12:44:58 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal 
  %guest  %gnice   %idle
12:45:00 AM   62    6.54    0.00   42.99    0.00    6.54   12.15    0.00 
    0.00    6.54   25.23

I don't know what interrupts they are...

It's the "hisi_sas_v3_hw cq" interrupts which we're interested in.

and that
> may be because no ITS targets the NUMA nodes they are part of.

So both storage controllers (which we're interested in for this test) 
are on socket #0, node #0.

  It would
> be interesting to see what happens if you manually set the affinity
> of the interrupts outside of the NUMA node.
> 

Again, managed, so I don't think it's possible.

Thanks,
John