aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 3971 bytes --]

Dear Greg,


Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
for Linux 4.14.56 causes the aacraid module to not detect the attached devices
anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.

```
$ dmesg | grep raid
[    0.269768] raid6: sse2x1   gen()  7179 MB/s
[    0.290069] raid6: sse2x1   xor()  5636 MB/s
[    0.311068] raid6: sse2x2   gen()  9160 MB/s
[    0.332076] raid6: sse2x2   xor()  6375 MB/s
[    0.353075] raid6: sse2x4   gen() 11164 MB/s
[    0.374064] raid6: sse2x4   xor()  7429 MB/s
[    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
[    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
[    0.391008] raid6: using ssse3x2 recovery algorithm
[    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
[    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
[   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
[   10.743295] aacraid: Comm Interface type3 enabled
$ lspci -nn | grep Adaptec
04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
```

But, it still works with a Dell PowerEdge R715 with two eight core AMD
Opteron 6136, the card below.

```
$ lspci -nn | grep Adaptec
22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
```

Reverting the commit fixes the issue.

commit ef86f3a72adb8a7931f67335560740a7ad696d1d
Author: Christoph Hellwig <hch@lst.de>
Date:   Fri Jan 12 10:53:05 2018 +0800

    genirq/affinity: assign vectors to all possible CPUs
    
    commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
    
    Currently we assign managed interrupt vectors to all present CPUs.  This
    works fine for systems were we only online/offline CPUs.  But in case of
    systems that support physical CPU hotplug (or the virtualized version of
    it) this means the additional CPUs covered for in the ACPI tables or on
    the command line are not catered for.  To fix this we'd either need to
    introduce new hotplug CPU states just for this case, or we can start
    assining vectors to possible but not present CPUs.
    
    Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
    Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
    Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
    Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
    Cc: linux-kernel@vger.kernel.org
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Signed-off-by: Christoph Hellwig <hch@lst.de>
    Signed-off-by: Jens Axboe <axboe@kernel.dk>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

The problem doesn’t happen with Linux 4.17.11, so there are commits in
Linux master fixing this. Unfortunately, my attempts to find out failed.

I was able to cherry-pick the three commits below on top of 4.14.62,
but the problem persists.

6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
355d7ecdea35 scsi: hpsa: fix selection of reply queue
e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity

Trying to cherry-pick the commits below, referencing the commit
in question, gave conflicts.

1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible

To avoid further trial and error with the server with a slow firmware,
do you know what commits should fix the issue?


Kind regards,

Paul


PS: I couldn’t find, who suggested this for stable, that means how
it was picked to be added to stable. Is there an easy way to find
that out?


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Greg Kroah-Hartman]

On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote:
> Dear Greg,
> 
> 
> Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
> for Linux 4.14.56 causes the aacraid module to not detect the attached devices
> anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.
> 
> ```
> $ dmesg | grep raid
> [    0.269768] raid6: sse2x1   gen()  7179 MB/s
> [    0.290069] raid6: sse2x1   xor()  5636 MB/s
> [    0.311068] raid6: sse2x2   gen()  9160 MB/s
> [    0.332076] raid6: sse2x2   xor()  6375 MB/s
> [    0.353075] raid6: sse2x4   gen() 11164 MB/s
> [    0.374064] raid6: sse2x4   xor()  7429 MB/s
> [    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
> [    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
> [    0.391008] raid6: using ssse3x2 recovery algorithm
> [    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
> [    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> [   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
> [   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
> [   10.743295] aacraid: Comm Interface type3 enabled
> $ lspci -nn | grep Adaptec
> 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
> 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
> ```
> 
> But, it still works with a Dell PowerEdge R715 with two eight core AMD
> Opteron 6136, the card below.
> 
> ```
> $ lspci -nn | grep Adaptec
> 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
> ```
> 
> Reverting the commit fixes the issue.
> 
> commit ef86f3a72adb8a7931f67335560740a7ad696d1d
> Author: Christoph Hellwig <hch@lst.de>
> Date:   Fri Jan 12 10:53:05 2018 +0800
> 
>     genirq/affinity: assign vectors to all possible CPUs
>     
>     commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
>     
>     Currently we assign managed interrupt vectors to all present CPUs.  This
>     works fine for systems were we only online/offline CPUs.  But in case of
>     systems that support physical CPU hotplug (or the virtualized version of
>     it) this means the additional CPUs covered for in the ACPI tables or on
>     the command line are not catered for.  To fix this we'd either need to
>     introduce new hotplug CPU states just for this case, or we can start
>     assining vectors to possible but not present CPUs.
>     
>     Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
>     Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
>     Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
>     Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
>     Cc: linux-kernel@vger.kernel.org
>     Cc: Thomas Gleixner <tglx@linutronix.de>
>     Signed-off-by: Christoph Hellwig <hch@lst.de>
>     Signed-off-by: Jens Axboe <axboe@kernel.dk>
>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> 
> The problem doesn’t happen with Linux 4.17.11, so there are commits in
> Linux master fixing this. Unfortunately, my attempts to find out failed.
> 
> I was able to cherry-pick the three commits below on top of 4.14.62,
> but the problem persists.
> 
> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
> 355d7ecdea35 scsi: hpsa: fix selection of reply queue
> e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> 
> Trying to cherry-pick the commits below, referencing the commit
> in question, gave conflicts.
> 
> 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
> 2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
> 
> To avoid further trial and error with the server with a slow firmware,
> do you know what commits should fix the issue?

Look at the email on the stable mailing list:
	Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y
it should help you out here.  Can you try the patches listed there?

> PS: I couldn’t find, who suggested this for stable, that means how
> it was picked to be added to stable. Is there an easy way to find
> that out?

Dig through the archives is usually the best way.  Looks like it came in
through a suggestion from Sasha.

thanks,

greg k-h

Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 5374 bytes --]

Dear Greg,


On 08/10/18 15:36, Greg Kroah-Hartman wrote:
> On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote:
>> Dear Greg,
>>
>>
>> Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
>> for Linux 4.14.56 causes the aacraid module to not detect the attached devices
>> anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.
>>
>> ```
>> $ dmesg | grep raid
>> [    0.269768] raid6: sse2x1   gen()  7179 MB/s
>> [    0.290069] raid6: sse2x1   xor()  5636 MB/s
>> [    0.311068] raid6: sse2x2   gen()  9160 MB/s
>> [    0.332076] raid6: sse2x2   xor()  6375 MB/s
>> [    0.353075] raid6: sse2x4   gen() 11164 MB/s
>> [    0.374064] raid6: sse2x4   xor()  7429 MB/s
>> [    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
>> [    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
>> [    0.391008] raid6: using ssse3x2 recovery algorithm
>> [    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
>> [    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
>> [   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
>> [   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
>> [   10.743295] aacraid: Comm Interface type3 enabled
>> $ lspci -nn | grep Adaptec
>> 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
>> 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
>> ```
>>
>> But, it still works with a Dell PowerEdge R715 with two eight core AMD
>> Opteron 6136, the card below.
>>
>> ```
>> $ lspci -nn | grep Adaptec
>> 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
>> ```
>>
>> Reverting the commit fixes the issue.
>>
>> commit ef86f3a72adb8a7931f67335560740a7ad696d1d
>> Author: Christoph Hellwig <hch@lst.de>
>> Date:   Fri Jan 12 10:53:05 2018 +0800
>>
>>     genirq/affinity: assign vectors to all possible CPUs
>>     
>>     commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
>>     
>>     Currently we assign managed interrupt vectors to all present CPUs.  This
>>     works fine for systems were we only online/offline CPUs.  But in case of
>>     systems that support physical CPU hotplug (or the virtualized version of
>>     it) this means the additional CPUs covered for in the ACPI tables or on
>>     the command line are not catered for.  To fix this we'd either need to
>>     introduce new hotplug CPU states just for this case, or we can start
>>     assining vectors to possible but not present CPUs.
>>     
>>     Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>     Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>     Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
>>     Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
>>     Cc: linux-kernel@vger.kernel.org
>>     Cc: Thomas Gleixner <tglx@linutronix.de>
>>     Signed-off-by: Christoph Hellwig <hch@lst.de>
>>     Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>>
>> The problem doesn’t happen with Linux 4.17.11, so there are commits in
>> Linux master fixing this. Unfortunately, my attempts to find out failed.
>>
>> I was able to cherry-pick the three commits below on top of 4.14.62,
>> but the problem persists.
>>
>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
>> 355d7ecdea35 scsi: hpsa: fix selection of reply queue
>> e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>
>> Trying to cherry-pick the commits below, referencing the commit
>> in question, gave conflicts.
>>
>> 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
>> 2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
>>
>> To avoid further trial and error with the server with a slow firmware,
>> do you know what commits should fix the issue?
> 
> Look at the email on the stable mailing list:
> 	Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y
> it should help you out here.

Ah, I didn’t see that [1] yet. Also I can’t find the original message, and a
way to reply to that thread. Therefore, here is my reply.

> Can you try the patches listed there?

I tried some of these already without success.

b5b6e8c8d3b4 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
2f31115e940c scsi: core: introduce force_blk_mq
adbe552349f2 scsi: megaraid_sas: fix selection of reply queue

The commit above is already in v4.14.56.

8b834bff1b73 scsi: hpsa: fix selection of reply queue

The problem persists.

The problem also persists with the state below.

3528f73a4e5d scsi: core: introduce force_blk_mq
16dc4d8215f3 scsi: hpsa: fix selection of reply queue
f0a7ab12232d scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
1aa1166eface (tag: v4.14.62, stable/linux-4.14.y) Linux 4.14.62

So, some more commits are necessary.


Kind regards,

Paul


[1]: https://www.spinics.net/lists/stable/msg251103.html


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Greg Kroah-Hartman]

On Fri, Aug 10, 2018 at 04:11:23PM +0200, Paul Menzel wrote:
> Dear Greg,
> 
> 
> On 08/10/18 15:36, Greg Kroah-Hartman wrote:
> > On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote:
> >> Dear Greg,
> >>
> >>
> >> Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
> >> for Linux 4.14.56 causes the aacraid module to not detect the attached devices
> >> anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.
> >>
> >> ```
> >> $ dmesg | grep raid
> >> [    0.269768] raid6: sse2x1   gen()  7179 MB/s
> >> [    0.290069] raid6: sse2x1   xor()  5636 MB/s
> >> [    0.311068] raid6: sse2x2   gen()  9160 MB/s
> >> [    0.332076] raid6: sse2x2   xor()  6375 MB/s
> >> [    0.353075] raid6: sse2x4   gen() 11164 MB/s
> >> [    0.374064] raid6: sse2x4   xor()  7429 MB/s
> >> [    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
> >> [    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
> >> [    0.391008] raid6: using ssse3x2 recovery algorithm
> >> [    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
> >> [    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> >> [   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
> >> [   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
> >> [   10.743295] aacraid: Comm Interface type3 enabled
> >> $ lspci -nn | grep Adaptec
> >> 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
> >> 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
> >> ```
> >>
> >> But, it still works with a Dell PowerEdge R715 with two eight core AMD
> >> Opteron 6136, the card below.
> >>
> >> ```
> >> $ lspci -nn | grep Adaptec
> >> 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
> >> ```
> >>
> >> Reverting the commit fixes the issue.
> >>
> >> commit ef86f3a72adb8a7931f67335560740a7ad696d1d
> >> Author: Christoph Hellwig <hch@lst.de>
> >> Date:   Fri Jan 12 10:53:05 2018 +0800
> >>
> >>     genirq/affinity: assign vectors to all possible CPUs
> >>     
> >>     commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
> >>     
> >>     Currently we assign managed interrupt vectors to all present CPUs.  This
> >>     works fine for systems were we only online/offline CPUs.  But in case of
> >>     systems that support physical CPU hotplug (or the virtualized version of
> >>     it) this means the additional CPUs covered for in the ACPI tables or on
> >>     the command line are not catered for.  To fix this we'd either need to
> >>     introduce new hotplug CPU states just for this case, or we can start
> >>     assining vectors to possible but not present CPUs.
> >>     
> >>     Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
> >>     Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
> >>     Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
> >>     Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
> >>     Cc: linux-kernel@vger.kernel.org
> >>     Cc: Thomas Gleixner <tglx@linutronix.de>
> >>     Signed-off-by: Christoph Hellwig <hch@lst.de>
> >>     Signed-off-by: Jens Axboe <axboe@kernel.dk>
> >>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> >>
> >> The problem doesn’t happen with Linux 4.17.11, so there are commits in
> >> Linux master fixing this. Unfortunately, my attempts to find out failed.
> >>
> >> I was able to cherry-pick the three commits below on top of 4.14.62,
> >> but the problem persists.
> >>
> >> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
> >> 355d7ecdea35 scsi: hpsa: fix selection of reply queue
> >> e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> >>
> >> Trying to cherry-pick the commits below, referencing the commit
> >> in question, gave conflicts.
> >>
> >> 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
> >> 2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
> >>
> >> To avoid further trial and error with the server with a slow firmware,
> >> do you know what commits should fix the issue?
> > 
> > Look at the email on the stable mailing list:
> > 	Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y
> > it should help you out here.
> 
> Ah, I didn’t see that [1] yet. Also I can’t find the original message, and a
> way to reply to that thread. Therefore, here is my reply.
> 
> > Can you try the patches listed there?
> 
> I tried some of these already without success.
> 
> b5b6e8c8d3b4 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> 2f31115e940c scsi: core: introduce force_blk_mq
> adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
> 
> The commit above is already in v4.14.56.
> 
> 8b834bff1b73 scsi: hpsa: fix selection of reply queue
> 
> The problem persists.
> 
> The problem also persists with the state below.
> 
> 3528f73a4e5d scsi: core: introduce force_blk_mq
> 16dc4d8215f3 scsi: hpsa: fix selection of reply queue
> f0a7ab12232d scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
> 1aa1166eface (tag: v4.14.62, stable/linux-4.14.y) Linux 4.14.62
> 
> So, some more commits are necessary.

Or I revert the original patch here, and the follow-on ones that were
added to "fix" this issue.  I think that might be the better thing
overall here, right?  Have you tried that?

thanks,

greg k-h

Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Paul Menzel]

Dear Greg,


Am 10.08.2018 um 17:55 schrieb Greg Kroah-Hartman:
> On Fri, Aug 10, 2018 at 04:11:23PM +0200, Paul Menzel wrote:

>> On 08/10/18 15:36, Greg Kroah-Hartman wrote:
>>> On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote:
>>>> Dear Greg,
>>>>
>>>>
>>>> Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
>>>> for Linux 4.14.56 causes the aacraid module to not detect the attached devices
>>>> anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.
>>>>
>>>> ```
>>>> $ dmesg | grep raid
>>>> [    0.269768] raid6: sse2x1   gen()  7179 MB/s
>>>> [    0.290069] raid6: sse2x1   xor()  5636 MB/s
>>>> [    0.311068] raid6: sse2x2   gen()  9160 MB/s
>>>> [    0.332076] raid6: sse2x2   xor()  6375 MB/s
>>>> [    0.353075] raid6: sse2x4   gen() 11164 MB/s
>>>> [    0.374064] raid6: sse2x4   xor()  7429 MB/s
>>>> [    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
>>>> [    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
>>>> [    0.391008] raid6: using ssse3x2 recovery algorithm
>>>> [    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
>>>> [    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
>>>> [   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
>>>> [   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
>>>> [   10.743295] aacraid: Comm Interface type3 enabled
>>>> $ lspci -nn | grep Adaptec
>>>> 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
>>>> 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
>>>> ```
>>>>
>>>> But, it still works with a Dell PowerEdge R715 with two eight core AMD
>>>> Opteron 6136, the card below.
>>>>
>>>> ```
>>>> $ lspci -nn | grep Adaptec
>>>> 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
>>>> ```
>>>>
>>>> Reverting the commit fixes the issue.
>>>>
>>>> commit ef86f3a72adb8a7931f67335560740a7ad696d1d
>>>> Author: Christoph Hellwig <hch@lst.de>
>>>> Date:   Fri Jan 12 10:53:05 2018 +0800
>>>>
>>>>      genirq/affinity: assign vectors to all possible CPUs
>>>>      
>>>>      commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
>>>>      
>>>>      Currently we assign managed interrupt vectors to all present CPUs.  This
>>>>      works fine for systems were we only online/offline CPUs.  But in case of
>>>>      systems that support physical CPU hotplug (or the virtualized version of
>>>>      it) this means the additional CPUs covered for in the ACPI tables or on
>>>>      the command line are not catered for.  To fix this we'd either need to
>>>>      introduce new hotplug CPU states just for this case, or we can start
>>>>      assining vectors to possible but not present CPUs.
>>>>      
>>>>      Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>>      Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>>      Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
>>>>      Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
>>>>      Cc: linux-kernel@vger.kernel.org
>>>>      Cc: Thomas Gleixner <tglx@linutronix.de>
>>>>      Signed-off-by: Christoph Hellwig <hch@lst.de>
>>>>      Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>>>      Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>>>>
>>>> The problem doesn’t happen with Linux 4.17.11, so there are commits in
>>>> Linux master fixing this. Unfortunately, my attempts to find out failed.
>>>>
>>>> I was able to cherry-pick the three commits below on top of 4.14.62,
>>>> but the problem persists.
>>>>
>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
>>>> 355d7ecdea35 scsi: hpsa: fix selection of reply queue
>>>> e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>>>
>>>> Trying to cherry-pick the commits below, referencing the commit
>>>> in question, gave conflicts.
>>>>
>>>> 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
>>>> 2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
>>>>
>>>> To avoid further trial and error with the server with a slow firmware,
>>>> do you know what commits should fix the issue?
>>>
>>> Look at the email on the stable mailing list:
>>> 	Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y
>>> it should help you out here.
>>
>> Ah, I didn’t see that [1] yet. Also I can’t find the original message, and a
>> way to reply to that thread. Therefore, here is my reply.
>>
>>> Can you try the patches listed there?
>>
>> I tried some of these already without success.
>>
>> b5b6e8c8d3b4 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>> 2f31115e940c scsi: core: introduce force_blk_mq
>> adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
>>
>> The commit above is already in v4.14.56.
>>
>> 8b834bff1b73 scsi: hpsa: fix selection of reply queue
>>
>> The problem persists.
>>
>> The problem also persists with the state below.
>>
>> 3528f73a4e5d scsi: core: introduce force_blk_mq
>> 16dc4d8215f3 scsi: hpsa: fix selection of reply queue
>> f0a7ab12232d scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
>> 1aa1166eface (tag: v4.14.62, stable/linux-4.14.y) Linux 4.14.62
>>
>> So, some more commits are necessary.
> 
> Or I revert the original patch here, and the follow-on ones that were
> added to "fix" this issue.  I think that might be the better thing
> overall here, right?  Have you tried that?

Yes, reverting the commit fixed the issue for us. If Christoph or Ming 
do not have another suggestion for a commit, that would be the way to go.


Kind regards,

Paul

Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Greg Kroah-Hartman]

On Sat, Aug 11, 2018 at 10:14:18AM +0200, Paul Menzel wrote:
> Dear Greg,
> 
> 
> Am 10.08.2018 um 17:55 schrieb Greg Kroah-Hartman:
> > On Fri, Aug 10, 2018 at 04:11:23PM +0200, Paul Menzel wrote:
> 
> > > On 08/10/18 15:36, Greg Kroah-Hartman wrote:
> > > > On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote:
> > > > > Dear Greg,
> > > > > 
> > > > > 
> > > > > Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
> > > > > for Linux 4.14.56 causes the aacraid module to not detect the attached devices
> > > > > anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.
> > > > > 
> > > > > ```
> > > > > $ dmesg | grep raid
> > > > > [    0.269768] raid6: sse2x1   gen()  7179 MB/s
> > > > > [    0.290069] raid6: sse2x1   xor()  5636 MB/s
> > > > > [    0.311068] raid6: sse2x2   gen()  9160 MB/s
> > > > > [    0.332076] raid6: sse2x2   xor()  6375 MB/s
> > > > > [    0.353075] raid6: sse2x4   gen() 11164 MB/s
> > > > > [    0.374064] raid6: sse2x4   xor()  7429 MB/s
> > > > > [    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
> > > > > [    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
> > > > > [    0.391008] raid6: using ssse3x2 recovery algorithm
> > > > > [    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
> > > > > [    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> > > > > [   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
> > > > > [   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
> > > > > [   10.743295] aacraid: Comm Interface type3 enabled
> > > > > $ lspci -nn | grep Adaptec
> > > > > 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
> > > > > 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
> > > > > ```
> > > > > 
> > > > > But, it still works with a Dell PowerEdge R715 with two eight core AMD
> > > > > Opteron 6136, the card below.
> > > > > 
> > > > > ```
> > > > > $ lspci -nn | grep Adaptec
> > > > > 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
> > > > > ```
> > > > > 
> > > > > Reverting the commit fixes the issue.
> > > > > 
> > > > > commit ef86f3a72adb8a7931f67335560740a7ad696d1d
> > > > > Author: Christoph Hellwig <hch@lst.de>
> > > > > Date:   Fri Jan 12 10:53:05 2018 +0800
> > > > > 
> > > > >      genirq/affinity: assign vectors to all possible CPUs
> > > > >      commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
> > > > >      Currently we assign managed interrupt vectors to all present CPUs.  This
> > > > >      works fine for systems were we only online/offline CPUs.  But in case of
> > > > >      systems that support physical CPU hotplug (or the virtualized version of
> > > > >      it) this means the additional CPUs covered for in the ACPI tables or on
> > > > >      the command line are not catered for.  To fix this we'd either need to
> > > > >      introduce new hotplug CPU states just for this case, or we can start
> > > > >      assining vectors to possible but not present CPUs.
> > > > >      Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
> > > > >      Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
> > > > >      Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
> > > > >      Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
> > > > >      Cc: linux-kernel@vger.kernel.org
> > > > >      Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > >      Signed-off-by: Christoph Hellwig <hch@lst.de>
> > > > >      Signed-off-by: Jens Axboe <axboe@kernel.dk>
> > > > >      Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> > > > > 
> > > > > The problem doesn’t happen with Linux 4.17.11, so there are commits in
> > > > > Linux master fixing this. Unfortunately, my attempts to find out failed.
> > > > > 
> > > > > I was able to cherry-pick the three commits below on top of 4.14.62,
> > > > > but the problem persists.
> > > > > 
> > > > > 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
> > > > > 355d7ecdea35 scsi: hpsa: fix selection of reply queue
> > > > > e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> > > > > 
> > > > > Trying to cherry-pick the commits below, referencing the commit
> > > > > in question, gave conflicts.
> > > > > 
> > > > > 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
> > > > > 2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
> > > > > 
> > > > > To avoid further trial and error with the server with a slow firmware,
> > > > > do you know what commits should fix the issue?
> > > > 
> > > > Look at the email on the stable mailing list:
> > > > 	Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y
> > > > it should help you out here.
> > > 
> > > Ah, I didn’t see that [1] yet. Also I can’t find the original message, and a
> > > way to reply to that thread. Therefore, here is my reply.
> > > 
> > > > Can you try the patches listed there?
> > > 
> > > I tried some of these already without success.
> > > 
> > > b5b6e8c8d3b4 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> > > 2f31115e940c scsi: core: introduce force_blk_mq
> > > adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
> > > 
> > > The commit above is already in v4.14.56.
> > > 
> > > 8b834bff1b73 scsi: hpsa: fix selection of reply queue
> > > 
> > > The problem persists.
> > > 
> > > The problem also persists with the state below.
> > > 
> > > 3528f73a4e5d scsi: core: introduce force_blk_mq
> > > 16dc4d8215f3 scsi: hpsa: fix selection of reply queue
> > > f0a7ab12232d scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> > > 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
> > > 1aa1166eface (tag: v4.14.62, stable/linux-4.14.y) Linux 4.14.62
> > > 
> > > So, some more commits are necessary.
> > 
> > Or I revert the original patch here, and the follow-on ones that were
> > added to "fix" this issue.  I think that might be the better thing
> > overall here, right?  Have you tried that?
> 
> Yes, reverting the commit fixed the issue for us. If Christoph or Ming do
> not have another suggestion for a commit, that would be the way to go.

Christoph or Ming, any ideas here?

In looking at the aacraid code, I don't see anywhere that this is using
a specific cpu number for queues or anything, but I could be wrong.
Ideally this should also be failing in 4.17 or 4.18-rc right now as well
as I don't see anything that would have "fixed" this recently.  Unless
I'm missing something here?

thanks,

greg k-h

Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Paul Menzel]

Dear Greg,


Am 11.08.2018 um 15:50 schrieb Greg Kroah-Hartman:
> On Sat, Aug 11, 2018 at 10:14:18AM +0200, Paul Menzel wrote:

>> Am 10.08.2018 um 17:55 schrieb Greg Kroah-Hartman:
>>> On Fri, Aug 10, 2018 at 04:11:23PM +0200, Paul Menzel wrote:
>>
>>>> On 08/10/18 15:36, Greg Kroah-Hartman wrote:
>>>>> On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote:
>>>>>> Dear Greg,
>>>>>>
>>>>>>
>>>>>> Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
>>>>>> for Linux 4.14.56 causes the aacraid module to not detect the attached devices
>>>>>> anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.
>>>>>>
>>>>>> ```
>>>>>> $ dmesg | grep raid
>>>>>> [    0.269768] raid6: sse2x1   gen()  7179 MB/s
>>>>>> [    0.290069] raid6: sse2x1   xor()  5636 MB/s
>>>>>> [    0.311068] raid6: sse2x2   gen()  9160 MB/s
>>>>>> [    0.332076] raid6: sse2x2   xor()  6375 MB/s
>>>>>> [    0.353075] raid6: sse2x4   gen() 11164 MB/s
>>>>>> [    0.374064] raid6: sse2x4   xor()  7429 MB/s
>>>>>> [    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
>>>>>> [    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
>>>>>> [    0.391008] raid6: using ssse3x2 recovery algorithm
>>>>>> [    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
>>>>>> [    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
>>>>>> [   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
>>>>>> [   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
>>>>>> [   10.743295] aacraid: Comm Interface type3 enabled
>>>>>> $ lspci -nn | grep Adaptec
>>>>>> 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
>>>>>> 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
>>>>>> ```
>>>>>>
>>>>>> But, it still works with a Dell PowerEdge R715 with two eight core AMD
>>>>>> Opteron 6136, the card below.
>>>>>>
>>>>>> ```
>>>>>> $ lspci -nn | grep Adaptec
>>>>>> 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
>>>>>> ```
>>>>>>
>>>>>> Reverting the commit fixes the issue.
>>>>>>
>>>>>> commit ef86f3a72adb8a7931f67335560740a7ad696d1d
>>>>>> Author: Christoph Hellwig <hch@lst.de>
>>>>>> Date:   Fri Jan 12 10:53:05 2018 +0800
>>>>>>
>>>>>>       genirq/affinity: assign vectors to all possible CPUs
>>>>>>       commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
>>>>>>       Currently we assign managed interrupt vectors to all present CPUs.  This
>>>>>>       works fine for systems were we only online/offline CPUs.  But in case of
>>>>>>       systems that support physical CPU hotplug (or the virtualized version of
>>>>>>       it) this means the additional CPUs covered for in the ACPI tables or on
>>>>>>       the command line are not catered for.  To fix this we'd either need to
>>>>>>       introduce new hotplug CPU states just for this case, or we can start
>>>>>>       assining vectors to possible but not present CPUs.
>>>>>>       Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>>>>       Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>>>>       Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
>>>>>>       Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
>>>>>>       Cc: linux-kernel@vger.kernel.org
>>>>>>       Cc: Thomas Gleixner <tglx@linutronix.de>
>>>>>>       Signed-off-by: Christoph Hellwig <hch@lst.de>
>>>>>>       Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>>>>>       Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>>>>>>
>>>>>> The problem doesn’t happen with Linux 4.17.11, so there are commits in
>>>>>> Linux master fixing this. Unfortunately, my attempts to find out failed.
>>>>>>
>>>>>> I was able to cherry-pick the three commits below on top of 4.14.62,
>>>>>> but the problem persists.
>>>>>>
>>>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
>>>>>> 355d7ecdea35 scsi: hpsa: fix selection of reply queue
>>>>>> e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>>>>>
>>>>>> Trying to cherry-pick the commits below, referencing the commit
>>>>>> in question, gave conflicts.
>>>>>>
>>>>>> 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
>>>>>> 2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
>>>>>>
>>>>>> To avoid further trial and error with the server with a slow firmware,
>>>>>> do you know what commits should fix the issue?
>>>>>
>>>>> Look at the email on the stable mailing list:
>>>>> 	Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y
>>>>> it should help you out here.
>>>>
>>>> Ah, I didn’t see that [1] yet. Also I can’t find the original message, and a
>>>> way to reply to that thread. Therefore, here is my reply.
>>>>
>>>>> Can you try the patches listed there?
>>>>
>>>> I tried some of these already without success.
>>>>
>>>> b5b6e8c8d3b4 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>>> 2f31115e940c scsi: core: introduce force_blk_mq
>>>> adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
>>>>
>>>> The commit above is already in v4.14.56.
>>>>
>>>> 8b834bff1b73 scsi: hpsa: fix selection of reply queue
>>>>
>>>> The problem persists.
>>>>
>>>> The problem also persists with the state below.
>>>>
>>>> 3528f73a4e5d scsi: core: introduce force_blk_mq
>>>> 16dc4d8215f3 scsi: hpsa: fix selection of reply queue
>>>> f0a7ab12232d scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
>>>> 1aa1166eface (tag: v4.14.62, stable/linux-4.14.y) Linux 4.14.62
>>>>
>>>> So, some more commits are necessary.
>>>
>>> Or I revert the original patch here, and the follow-on ones that were
>>> added to "fix" this issue.  I think that might be the better thing
>>> overall here, right?  Have you tried that?
>>
>> Yes, reverting the commit fixed the issue for us. If Christoph or Ming do
>> not have another suggestion for a commit, that would be the way to go.
> 
> Christoph or Ming, any ideas here?
> 
> In looking at the aacraid code, I don't see anywhere that this is using
> a specific cpu number for queues or anything, but I could be wrong.
> Ideally this should also be failing in 4.17 or 4.18-rc right now as well
> as I don't see anything that would have "fixed" this recently.  Unless
> I'm missing something here?

As written strangely, it works with Linux 4.17.11.


Kind regards,

Paul

Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Ming Lei]

On Sat, Aug 11, 2018 at 03:50:21PM +0200, Greg Kroah-Hartman wrote:
> On Sat, Aug 11, 2018 at 10:14:18AM +0200, Paul Menzel wrote:
> > Dear Greg,
> > 
> > 
> > Am 10.08.2018 um 17:55 schrieb Greg Kroah-Hartman:
> > > On Fri, Aug 10, 2018 at 04:11:23PM +0200, Paul Menzel wrote:
> > 
> > > > On 08/10/18 15:36, Greg Kroah-Hartman wrote:
> > > > > On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote:
> > > > > > Dear Greg,
> > > > > > 
> > > > > > 
> > > > > > Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
> > > > > > for Linux 4.14.56 causes the aacraid module to not detect the attached devices
> > > > > > anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.
> > > > > > 
> > > > > > ```
> > > > > > $ dmesg | grep raid
> > > > > > [    0.269768] raid6: sse2x1   gen()  7179 MB/s
> > > > > > [    0.290069] raid6: sse2x1   xor()  5636 MB/s
> > > > > > [    0.311068] raid6: sse2x2   gen()  9160 MB/s
> > > > > > [    0.332076] raid6: sse2x2   xor()  6375 MB/s
> > > > > > [    0.353075] raid6: sse2x4   gen() 11164 MB/s
> > > > > > [    0.374064] raid6: sse2x4   xor()  7429 MB/s
> > > > > > [    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
> > > > > > [    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
> > > > > > [    0.391008] raid6: using ssse3x2 recovery algorithm
> > > > > > [    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
> > > > > > [    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> > > > > > [   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
> > > > > > [   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
> > > > > > [   10.743295] aacraid: Comm Interface type3 enabled
> > > > > > $ lspci -nn | grep Adaptec
> > > > > > 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
> > > > > > 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
> > > > > > ```
> > > > > > 
> > > > > > But, it still works with a Dell PowerEdge R715 with two eight core AMD
> > > > > > Opteron 6136, the card below.
> > > > > > 
> > > > > > ```
> > > > > > $ lspci -nn | grep Adaptec
> > > > > > 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
> > > > > > ```
> > > > > > 
> > > > > > Reverting the commit fixes the issue.
> > > > > > 
> > > > > > commit ef86f3a72adb8a7931f67335560740a7ad696d1d
> > > > > > Author: Christoph Hellwig <hch@lst.de>
> > > > > > Date:   Fri Jan 12 10:53:05 2018 +0800
> > > > > > 
> > > > > >      genirq/affinity: assign vectors to all possible CPUs
> > > > > >      commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
> > > > > >      Currently we assign managed interrupt vectors to all present CPUs.  This
> > > > > >      works fine for systems were we only online/offline CPUs.  But in case of
> > > > > >      systems that support physical CPU hotplug (or the virtualized version of
> > > > > >      it) this means the additional CPUs covered for in the ACPI tables or on
> > > > > >      the command line are not catered for.  To fix this we'd either need to
> > > > > >      introduce new hotplug CPU states just for this case, or we can start
> > > > > >      assining vectors to possible but not present CPUs.
> > > > > >      Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
> > > > > >      Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
> > > > > >      Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
> > > > > >      Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
> > > > > >      Cc: linux-kernel@vger.kernel.org
> > > > > >      Cc: Thomas Gleixner <tglx@linutronix.de>
> > > > > >      Signed-off-by: Christoph Hellwig <hch@lst.de>
> > > > > >      Signed-off-by: Jens Axboe <axboe@kernel.dk>
> > > > > >      Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> > > > > > 
> > > > > > The problem doesn’t happen with Linux 4.17.11, so there are commits in
> > > > > > Linux master fixing this. Unfortunately, my attempts to find out failed.
> > > > > > 
> > > > > > I was able to cherry-pick the three commits below on top of 4.14.62,
> > > > > > but the problem persists.
> > > > > > 
> > > > > > 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
> > > > > > 355d7ecdea35 scsi: hpsa: fix selection of reply queue
> > > > > > e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> > > > > > 
> > > > > > Trying to cherry-pick the commits below, referencing the commit
> > > > > > in question, gave conflicts.
> > > > > > 
> > > > > > 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
> > > > > > 2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
> > > > > > 
> > > > > > To avoid further trial and error with the server with a slow firmware,
> > > > > > do you know what commits should fix the issue?
> > > > > 
> > > > > Look at the email on the stable mailing list:
> > > > > 	Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y
> > > > > it should help you out here.
> > > > 
> > > > Ah, I didn’t see that [1] yet. Also I can’t find the original message, and a
> > > > way to reply to that thread. Therefore, here is my reply.
> > > > 
> > > > > Can you try the patches listed there?
> > > > 
> > > > I tried some of these already without success.
> > > > 
> > > > b5b6e8c8d3b4 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> > > > 2f31115e940c scsi: core: introduce force_blk_mq
> > > > adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
> > > > 
> > > > The commit above is already in v4.14.56.
> > > > 
> > > > 8b834bff1b73 scsi: hpsa: fix selection of reply queue
> > > > 
> > > > The problem persists.
> > > > 
> > > > The problem also persists with the state below.
> > > > 
> > > > 3528f73a4e5d scsi: core: introduce force_blk_mq
> > > > 16dc4d8215f3 scsi: hpsa: fix selection of reply queue
> > > > f0a7ab12232d scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> > > > 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
> > > > 1aa1166eface (tag: v4.14.62, stable/linux-4.14.y) Linux 4.14.62
> > > > 
> > > > So, some more commits are necessary.
> > > 
> > > Or I revert the original patch here, and the follow-on ones that were
> > > added to "fix" this issue.  I think that might be the better thing
> > > overall here, right?  Have you tried that?
> > 
> > Yes, reverting the commit fixed the issue for us. If Christoph or Ming do
> > not have another suggestion for a commit, that would be the way to go.
> 
> Christoph or Ming, any ideas here?
> 
> In looking at the aacraid code, I don't see anywhere that this is using
> a specific cpu number for queues or anything, but I could be wrong.
> Ideally this should also be failing in 4.17 or 4.18-rc right now as well
> as I don't see anything that would have "fixed" this recently.  Unless
> I'm missing something here?

From both aac_fib_vector_assign() and aac_src_deliver_message(), looks
every msix vector is used inside driver, so causes this trouble.

From 84676c1f21e8ff54b (genirq/affinity: assign vectors to all possible CPUs),
actually it should have been since 9a0ef98e18 (genirq/affinity: Assign vectors
to all present CPUs), it isn't guaranteed that one vector can be mapped to
at least one online CPU.

So it should be fixed in driver by similar approach taken in the
following patches:
	scsi: hpsa: fix selection of reply queue
	scsi: megaraid_sas: fix selection of reply queue

And the reason this issue won't be observed is that the following
patches(merged to v4.17) may avoid this issue, but in theory, the issue
won't be avoided completely by these patches:

d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
1a2d0914e23a genirq/affinity: Allow irq spreading from a given starting point
b3e6aaa8d94d genirq/affinity: Move actual irq vector spreading into a helper function
47778f33dcba genirq/affinity: Rename *node_to_possible_cpumask as *node_to_cpumask
0211e12dd0a5 genirq/affinity: Don't return with empty affinity masks on error

In short, this issue should have been fixed in upstream aacraid, and the idea
is simple, use the current CPU id to retrieve the msix vector offset, this
way will make sure that the msix vector can be mapped from one online
CPU.

Thanks,
Ming

Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 8636 bytes --]

Dear Ming,


On 08/13/18 05:32, Ming Lei wrote:
> On Sat, Aug 11, 2018 at 03:50:21PM +0200, Greg Kroah-Hartman wrote:
>> On Sat, Aug 11, 2018 at 10:14:18AM +0200, Paul Menzel wrote:

>>> Am 10.08.2018 um 17:55 schrieb Greg Kroah-Hartman:
>>>> On Fri, Aug 10, 2018 at 04:11:23PM +0200, Paul Menzel wrote:
>>>
>>>>> On 08/10/18 15:36, Greg Kroah-Hartman wrote:
>>>>>> On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote:

>>>>>>> Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
>>>>>>> for Linux 4.14.56 causes the aacraid module to not detect the attached devices
>>>>>>> anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.
>>>>>>>
>>>>>>> ```
>>>>>>> $ dmesg | grep raid
>>>>>>> [    0.269768] raid6: sse2x1   gen()  7179 MB/s
>>>>>>> [    0.290069] raid6: sse2x1   xor()  5636 MB/s
>>>>>>> [    0.311068] raid6: sse2x2   gen()  9160 MB/s
>>>>>>> [    0.332076] raid6: sse2x2   xor()  6375 MB/s
>>>>>>> [    0.353075] raid6: sse2x4   gen() 11164 MB/s
>>>>>>> [    0.374064] raid6: sse2x4   xor()  7429 MB/s
>>>>>>> [    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
>>>>>>> [    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
>>>>>>> [    0.391008] raid6: using ssse3x2 recovery algorithm
>>>>>>> [    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
>>>>>>> [    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
>>>>>>> [   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
>>>>>>> [   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
>>>>>>> [   10.743295] aacraid: Comm Interface type3 enabled
>>>>>>> $ lspci -nn | grep Adaptec
>>>>>>> 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
>>>>>>> 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
>>>>>>> ```
>>>>>>>
>>>>>>> But, it still works with a Dell PowerEdge R715 with two eight core AMD
>>>>>>> Opteron 6136, the card below.
>>>>>>>
>>>>>>> ```
>>>>>>> $ lspci -nn | grep Adaptec
>>>>>>> 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
>>>>>>> ```
>>>>>>>
>>>>>>> Reverting the commit fixes the issue.
>>>>>>>
>>>>>>> commit ef86f3a72adb8a7931f67335560740a7ad696d1d
>>>>>>> Author: Christoph Hellwig <hch@lst.de>
>>>>>>> Date:   Fri Jan 12 10:53:05 2018 +0800
>>>>>>>
>>>>>>>      genirq/affinity: assign vectors to all possible CPUs
>>>>>>>      commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
>>>>>>>      Currently we assign managed interrupt vectors to all present CPUs.  This
>>>>>>>      works fine for systems were we only online/offline CPUs.  But in case of
>>>>>>>      systems that support physical CPU hotplug (or the virtualized version of
>>>>>>>      it) this means the additional CPUs covered for in the ACPI tables or on
>>>>>>>      the command line are not catered for.  To fix this we'd either need to
>>>>>>>      introduce new hotplug CPU states just for this case, or we can start
>>>>>>>      assining vectors to possible but not present CPUs.
>>>>>>>      Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>>>>>      Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>>>>>      Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
>>>>>>>      Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
>>>>>>>      Cc: linux-kernel@vger.kernel.org
>>>>>>>      Cc: Thomas Gleixner <tglx@linutronix.de>
>>>>>>>      Signed-off-by: Christoph Hellwig <hch@lst.de>
>>>>>>>      Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>>>>>>      Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>>>>>>>
>>>>>>> The problem doesn’t happen with Linux 4.17.11, so there are commits in
>>>>>>> Linux master fixing this. Unfortunately, my attempts to find out failed.
>>>>>>>
>>>>>>> I was able to cherry-pick the three commits below on top of 4.14.62,
>>>>>>> but the problem persists.
>>>>>>>
>>>>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
>>>>>>> 355d7ecdea35 scsi: hpsa: fix selection of reply queue
>>>>>>> e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>>>>>>
>>>>>>> Trying to cherry-pick the commits below, referencing the commit
>>>>>>> in question, gave conflicts.
>>>>>>>
>>>>>>> 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
>>>>>>> 2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
>>>>>>>
>>>>>>> To avoid further trial and error with the server with a slow firmware,
>>>>>>> do you know what commits should fix the issue?
>>>>>>
>>>>>> Look at the email on the stable mailing list:
>>>>>> 	Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y
>>>>>> it should help you out here.
>>>>>
>>>>> Ah, I didn’t see that [1] yet. Also I can’t find the original message, and a
>>>>> way to reply to that thread. Therefore, here is my reply.
>>>>>
>>>>>> Can you try the patches listed there?
>>>>>
>>>>> I tried some of these already without success.
>>>>>
>>>>> b5b6e8c8d3b4 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>>>> 2f31115e940c scsi: core: introduce force_blk_mq
>>>>> adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
>>>>>
>>>>> The commit above is already in v4.14.56.
>>>>>
>>>>> 8b834bff1b73 scsi: hpsa: fix selection of reply queue
>>>>>
>>>>> The problem persists.
>>>>>
>>>>> The problem also persists with the state below.
>>>>>
>>>>> 3528f73a4e5d scsi: core: introduce force_blk_mq
>>>>> 16dc4d8215f3 scsi: hpsa: fix selection of reply queue
>>>>> f0a7ab12232d scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
>>>>> 1aa1166eface (tag: v4.14.62, stable/linux-4.14.y) Linux 4.14.62
>>>>>
>>>>> So, some more commits are necessary.
>>>>
>>>> Or I revert the original patch here, and the follow-on ones that were
>>>> added to "fix" this issue.  I think that might be the better thing
>>>> overall here, right?  Have you tried that?
>>>
>>> Yes, reverting the commit fixed the issue for us. If Christoph or Ming do
>>> not have another suggestion for a commit, that would be the way to go.
>>
>> Christoph or Ming, any ideas here?
>>
>> In looking at the aacraid code, I don't see anywhere that this is using
>> a specific cpu number for queues or anything, but I could be wrong.
>> Ideally this should also be failing in 4.17 or 4.18-rc right now as well
>> as I don't see anything that would have "fixed" this recently.  Unless
>> I'm missing something here?
> 
> From both aac_fib_vector_assign() and aac_src_deliver_message(), looks
> every msix vector is used inside driver, so causes this trouble.
> 
> From 84676c1f21e8ff54b (genirq/affinity: assign vectors to all possible CPUs),
> actually it should have been since 9a0ef98e18 (genirq/affinity: Assign vectors
> to all present CPUs), it isn't guaranteed that one vector can be mapped to
> at least one online CPU.
> 
> So it should be fixed in driver by similar approach taken in the
> following patches:
> 	scsi: hpsa: fix selection of reply queue
> 	scsi: megaraid_sas: fix selection of reply queue
> 
> And the reason this issue won't be observed is that the following
> patches(merged to v4.17) may avoid this issue, but in theory, the issue
> won't be avoided completely by these patches:
> 
> d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
> 1a2d0914e23a genirq/affinity: Allow irq spreading from a given starting point
> b3e6aaa8d94d genirq/affinity: Move actual irq vector spreading into a helper function
> 47778f33dcba genirq/affinity: Rename *node_to_possible_cpumask as *node_to_cpumask
> 0211e12dd0a5 genirq/affinity: Don't return with empty affinity masks on error
> 
> In short, this issue should have been fixed in upstream aacraid, and the idea
> is simple, use the current CPU id to retrieve the msix vector offset, this
> way will make sure that the msix vector can be mapped from one online
> CPU.

AACRAID folks, are you going to work on that? I looked at the code a bit,
and it’ll take me some days to understand it and to fix it.

Just out of curiosity, shouldn’t the subsystems(?) communicate better to
avoid such breakage, or was that unexpected?


Kind regards,

Paul


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 9106 bytes --]

Dear Ming, dear Raghava, dear Dave,


On 08/16/18 19:09, Paul Menzel wrote:

> On 08/13/18 05:32, Ming Lei wrote:
>> On Sat, Aug 11, 2018 at 03:50:21PM +0200, Greg Kroah-Hartman wrote:
>>> On Sat, Aug 11, 2018 at 10:14:18AM +0200, Paul Menzel wrote:
> 
>>>> Am 10.08.2018 um 17:55 schrieb Greg Kroah-Hartman:
>>>>> On Fri, Aug 10, 2018 at 04:11:23PM +0200, Paul Menzel wrote:
>>>>
>>>>>> On 08/10/18 15:36, Greg Kroah-Hartman wrote:
>>>>>>> On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote:
> 
>>>>>>>> Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
>>>>>>>> for Linux 4.14.56 causes the aacraid module to not detect the attached devices
>>>>>>>> anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.
>>>>>>>>
>>>>>>>> ```
>>>>>>>> $ dmesg | grep raid
>>>>>>>> [    0.269768] raid6: sse2x1   gen()  7179 MB/s
>>>>>>>> [    0.290069] raid6: sse2x1   xor()  5636 MB/s
>>>>>>>> [    0.311068] raid6: sse2x2   gen()  9160 MB/s
>>>>>>>> [    0.332076] raid6: sse2x2   xor()  6375 MB/s
>>>>>>>> [    0.353075] raid6: sse2x4   gen() 11164 MB/s
>>>>>>>> [    0.374064] raid6: sse2x4   xor()  7429 MB/s
>>>>>>>> [    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
>>>>>>>> [    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
>>>>>>>> [    0.391008] raid6: using ssse3x2 recovery algorithm
>>>>>>>> [    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
>>>>>>>> [    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
>>>>>>>> [   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
>>>>>>>> [   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
>>>>>>>> [   10.743295] aacraid: Comm Interface type3 enabled
>>>>>>>> $ lspci -nn | grep Adaptec
>>>>>>>> 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
>>>>>>>> 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
>>>>>>>> ```
>>>>>>>>
>>>>>>>> But, it still works with a Dell PowerEdge R715 with two eight core AMD
>>>>>>>> Opteron 6136, the card below.
>>>>>>>>
>>>>>>>> ```
>>>>>>>> $ lspci -nn | grep Adaptec
>>>>>>>> 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
>>>>>>>> ```
>>>>>>>>
>>>>>>>> Reverting the commit fixes the issue.
>>>>>>>>
>>>>>>>> commit ef86f3a72adb8a7931f67335560740a7ad696d1d
>>>>>>>> Author: Christoph Hellwig <hch@lst.de>
>>>>>>>> Date:   Fri Jan 12 10:53:05 2018 +0800
>>>>>>>>
>>>>>>>>      genirq/affinity: assign vectors to all possible CPUs
>>>>>>>>      commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
>>>>>>>>      Currently we assign managed interrupt vectors to all present CPUs.  This
>>>>>>>>      works fine for systems were we only online/offline CPUs.  But in case of
>>>>>>>>      systems that support physical CPU hotplug (or the virtualized version of
>>>>>>>>      it) this means the additional CPUs covered for in the ACPI tables or on
>>>>>>>>      the command line are not catered for.  To fix this we'd either need to
>>>>>>>>      introduce new hotplug CPU states just for this case, or we can start
>>>>>>>>      assining vectors to possible but not present CPUs.
>>>>>>>>      Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>>>>>>      Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
>>>>>>>>      Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
>>>>>>>>      Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
>>>>>>>>      Cc: linux-kernel@vger.kernel.org
>>>>>>>>      Cc: Thomas Gleixner <tglx@linutronix.de>
>>>>>>>>      Signed-off-by: Christoph Hellwig <hch@lst.de>
>>>>>>>>      Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>>>>>>>      Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>>>>>>>>
>>>>>>>> The problem doesn’t happen with Linux 4.17.11, so there are commits in
>>>>>>>> Linux master fixing this. Unfortunately, my attempts to find out failed.
>>>>>>>>
>>>>>>>> I was able to cherry-pick the three commits below on top of 4.14.62,
>>>>>>>> but the problem persists.
>>>>>>>>
>>>>>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
>>>>>>>> 355d7ecdea35 scsi: hpsa: fix selection of reply queue
>>>>>>>> e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>>>>>>>
>>>>>>>> Trying to cherry-pick the commits below, referencing the commit
>>>>>>>> in question, gave conflicts.
>>>>>>>>
>>>>>>>> 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
>>>>>>>> 2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
>>>>>>>>
>>>>>>>> To avoid further trial and error with the server with a slow firmware,
>>>>>>>> do you know what commits should fix the issue?
>>>>>>>
>>>>>>> Look at the email on the stable mailing list:
>>>>>>> 	Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y
>>>>>>> it should help you out here.
>>>>>>
>>>>>> Ah, I didn’t see that [1] yet. Also I can’t find the original message, and a
>>>>>> way to reply to that thread. Therefore, here is my reply.
>>>>>>
>>>>>>> Can you try the patches listed there?
>>>>>>
>>>>>> I tried some of these already without success.
>>>>>>
>>>>>> b5b6e8c8d3b4 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>>>>> 2f31115e940c scsi: core: introduce force_blk_mq
>>>>>> adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
>>>>>>
>>>>>> The commit above is already in v4.14.56.
>>>>>>
>>>>>> 8b834bff1b73 scsi: hpsa: fix selection of reply queue
>>>>>>
>>>>>> The problem persists.
>>>>>>
>>>>>> The problem also persists with the state below.
>>>>>>
>>>>>> 3528f73a4e5d scsi: core: introduce force_blk_mq
>>>>>> 16dc4d8215f3 scsi: hpsa: fix selection of reply queue
>>>>>> f0a7ab12232d scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
>>>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
>>>>>> 1aa1166eface (tag: v4.14.62, stable/linux-4.14.y) Linux 4.14.62
>>>>>>
>>>>>> So, some more commits are necessary.
>>>>>
>>>>> Or I revert the original patch here, and the follow-on ones that were
>>>>> added to "fix" this issue.  I think that might be the better thing
>>>>> overall here, right?  Have you tried that?
>>>>
>>>> Yes, reverting the commit fixed the issue for us. If Christoph or Ming do
>>>> not have another suggestion for a commit, that would be the way to go.
>>>
>>> Christoph or Ming, any ideas here?
>>>
>>> In looking at the aacraid code, I don't see anywhere that this is using
>>> a specific cpu number for queues or anything, but I could be wrong.
>>> Ideally this should also be failing in 4.17 or 4.18-rc right now as well
>>> as I don't see anything that would have "fixed" this recently.  Unless
>>> I'm missing something here?
>>
>> From both aac_fib_vector_assign() and aac_src_deliver_message(), looks
>> every msix vector is used inside driver, so causes this trouble.
>>
>> From 84676c1f21e8ff54b (genirq/affinity: assign vectors to all possible CPUs),
>> actually it should have been since 9a0ef98e18 (genirq/affinity: Assign vectors
>> to all present CPUs), it isn't guaranteed that one vector can be mapped to
>> at least one online CPU.
>>
>> So it should be fixed in driver by similar approach taken in the
>> following patches:
>> 	scsi: hpsa: fix selection of reply queue
>> 	scsi: megaraid_sas: fix selection of reply queue
>>
>> And the reason this issue won't be observed is that the following
>> patches(merged to v4.17) may avoid this issue, but in theory, the issue
>> won't be avoided completely by these patches:
>>
>> d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
>> 1a2d0914e23a genirq/affinity: Allow irq spreading from a given starting point
>> b3e6aaa8d94d genirq/affinity: Move actual irq vector spreading into a helper function
>> 47778f33dcba genirq/affinity: Rename *node_to_possible_cpumask as *node_to_cpumask
>> 0211e12dd0a5 genirq/affinity: Don't return with empty affinity masks on error
>>
>> In short, this issue should have been fixed in upstream aacraid, and the idea
>> is simple, use the current CPU id to retrieve the msix vector offset, this
>> way will make sure that the msix vector can be mapped from one online
>> CPU.
> 
> AACRAID folks, are you going to work on that? I looked at the code a bit,
> and it’ll take me some days to understand it and to fix it.

Unfortunately, there was no reply yet and I won’t be able to fix it any
time soon. The aacraid driver is still broken in 4.14.68. Reverting the
*one* mentioned commit fixes the issue.

Raghava, Dave, are you aware of the issue.

> Just out of curiosity, shouldn’t the subsystems(?) communicate better to
> avoid such breakage, or was that unexpected?


Kind regards,

Paul


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

[PATCH] Revert "genirq/affinity: assign vectors to all possible CPUs"

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 5073 bytes --]

Date: Wed, 29 Aug 2018 17:28:45 +0200

This reverts commit ef86f3a72adb8a7931f67335560740a7ad696d1d.

See the discussion in the thread *aacraid: Regression in 4.14.56
with *genirq/affinity: assign vectors to all possible CPUs** on
the linux-scsi list.

Reverting the commit, the aacraid driver loads correctly, and the
drives are visible. So revert the commit to adhere to Linux’ no
regression policy.

Strangely, the issue is not reproducible with Linux 4.18.x, but
Ming Lei wrote, that this might only be by chance.

Link: https://www.spinics.net/lists/linux-scsi/msg122732.html
Signed-off-by: Paul Menzel <pmenzel@molgen.mpg.de>

---
 kernel/irq/affinity.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/kernel/irq/affinity.c b/kernel/irq/affinity.c
index a37a3b4b6342..e12d35108225 100644
--- a/kernel/irq/affinity.c
+++ b/kernel/irq/affinity.c
@@ -39,7 +39,7 @@ static void irq_spread_init_one(struct cpumask *irqmsk, struct cpumask *nmsk,
 	}
 }
 
-static cpumask_var_t *alloc_node_to_possible_cpumask(void)
+static cpumask_var_t *alloc_node_to_present_cpumask(void)
 {
 	cpumask_var_t *masks;
 	int node;
@@ -62,7 +62,7 @@ static cpumask_var_t *alloc_node_to_possible_cpumask(void)
 	return NULL;
 }
 
-static void free_node_to_possible_cpumask(cpumask_var_t *masks)
+static void free_node_to_present_cpumask(cpumask_var_t *masks)
 {
 	int node;
 
@@ -71,22 +71,22 @@ static void free_node_to_possible_cpumask(cpumask_var_t *masks)
 	kfree(masks);
 }
 
-static void build_node_to_possible_cpumask(cpumask_var_t *masks)
+static void build_node_to_present_cpumask(cpumask_var_t *masks)
 {
 	int cpu;
 
-	for_each_possible_cpu(cpu)
+	for_each_present_cpu(cpu)
 		cpumask_set_cpu(cpu, masks[cpu_to_node(cpu)]);
 }
 
-static int get_nodes_in_cpumask(cpumask_var_t *node_to_possible_cpumask,
+static int get_nodes_in_cpumask(cpumask_var_t *node_to_present_cpumask,
 				const struct cpumask *mask, nodemask_t *nodemsk)
 {
 	int n, nodes = 0;
 
 	/* Calculate the number of nodes in the supplied affinity mask */
 	for_each_node(n) {
-		if (cpumask_intersects(mask, node_to_possible_cpumask[n])) {
+		if (cpumask_intersects(mask, node_to_present_cpumask[n])) {
 			node_set(n, *nodemsk);
 			nodes++;
 		}
@@ -109,7 +109,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 	int last_affv = affv + affd->pre_vectors;
 	nodemask_t nodemsk = NODE_MASK_NONE;
 	struct cpumask *masks;
-	cpumask_var_t nmsk, *node_to_possible_cpumask;
+	cpumask_var_t nmsk, *node_to_present_cpumask;
 
 	/*
 	 * If there aren't any vectors left after applying the pre/post
@@ -125,8 +125,8 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 	if (!masks)
 		goto out;
 
-	node_to_possible_cpumask = alloc_node_to_possible_cpumask();
-	if (!node_to_possible_cpumask)
+	node_to_present_cpumask = alloc_node_to_present_cpumask();
+	if (!node_to_present_cpumask)
 		goto out;
 
 	/* Fill out vectors at the beginning that don't need affinity */
@@ -135,8 +135,8 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 
 	/* Stabilize the cpumasks */
 	get_online_cpus();
-	build_node_to_possible_cpumask(node_to_possible_cpumask);
-	nodes = get_nodes_in_cpumask(node_to_possible_cpumask, cpu_possible_mask,
+	build_node_to_present_cpumask(node_to_present_cpumask);
+	nodes = get_nodes_in_cpumask(node_to_present_cpumask, cpu_present_mask,
 				     &nodemsk);
 
 	/*
@@ -146,7 +146,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 	if (affv <= nodes) {
 		for_each_node_mask(n, nodemsk) {
 			cpumask_copy(masks + curvec,
-				     node_to_possible_cpumask[n]);
+				     node_to_present_cpumask[n]);
 			if (++curvec == last_affv)
 				break;
 		}
@@ -160,7 +160,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 		vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;
 
 		/* Get the cpus on this node which are in the mask */
-		cpumask_and(nmsk, cpu_possible_mask, node_to_possible_cpumask[n]);
+		cpumask_and(nmsk, cpu_present_mask, node_to_present_cpumask[n]);
 
 		/* Calculate the number of cpus per vector */
 		ncpus = cpumask_weight(nmsk);
@@ -192,7 +192,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 	/* Fill out vectors at the end that don't need affinity */
 	for (; curvec < nvecs; curvec++)
 		cpumask_copy(masks + curvec, irq_default_affinity);
-	free_node_to_possible_cpumask(node_to_possible_cpumask);
+	free_node_to_present_cpumask(node_to_present_cpumask);
 out:
 	free_cpumask_var(nmsk);
 	return masks;
@@ -214,7 +214,7 @@ int irq_calc_affinity_vectors(int minvec, int maxvec, const struct irq_affinity
 		return 0;
 
 	get_online_cpus();
-	ret = min_t(int, cpumask_weight(cpu_possible_mask), vecs) + resv;
+	ret = min_t(int, cpumask_weight(cpu_present_mask), vecs) + resv;
 	put_online_cpus();
 	return ret;
 }
-- 
2.17.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH] Revert "genirq/affinity: assign vectors to all possible CPUs"

[Christoph Hellwig]

On Mon, Oct 01, 2018 at 02:33:07PM +0200, Paul Menzel wrote:
> Date: Wed, 29 Aug 2018 17:28:45 +0200
> 
> This reverts commit ef86f3a72adb8a7931f67335560740a7ad696d1d.

This seems rather odd.  If add all you'd revert the patch adding the
PCI_IRQ_AFFINITY to aacraid, not core infrastructure.

Re: [PATCH] Revert "genirq/affinity: assign vectors to all possible CPUs"

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]

Dear Christoph,


On 10/01/18 14:35, Christoph Hellwig wrote:
> On Mon, Oct 01, 2018 at 02:33:07PM +0200, Paul Menzel wrote:
>> Date: Wed, 29 Aug 2018 17:28:45 +0200
>>
>> This reverts commit ef86f3a72adb8a7931f67335560740a7ad696d1d.
> 
> This seems rather odd.  If at all you'd revert the patch adding the
> PCI_IRQ_AFFINITY to aacraid, not core infrastructure.

Thank you for the suggestion, but that flag was added in 2016
to the aacraid driver.

> commit 0910d8bbdd99856af1394d3d8830955abdefee4a
> Author: Hannes Reinecke <hare@suse.de>
> Date:   Tue Nov 8 08:11:30 2016 +0100
> 
>     scsi: aacraid: switch to pci_alloc_irq_vectors
>     
>     Use pci_alloc_irq_vectors and drop the hand-crafted interrupt affinity
>     routines.

So what would happen, if `PCI_IRQ_AFFINITY` was removed? Will the
system still work with the same performance?

As far as I understood, the no regression policy is there for
exactly that reason, and it shouldn’t matter if it’s core
infrastructure or not. As written, I have no idea, and just know
reverting the commit in question fixes the problem here. So I’ll
gladly test other solutions to fix this issue.


Kind regards,

Paul


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH] Revert "genirq/affinity: assign vectors to all possible CPUs"

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 1493 bytes --]

Dear Christoph,


On 10/01/18 14:43, Paul Menzel wrote:

> On 10/01/18 14:35, Christoph Hellwig wrote:
>> On Mon, Oct 01, 2018 at 02:33:07PM +0200, Paul Menzel wrote:
>>> Date: Wed, 29 Aug 2018 17:28:45 +0200
>>>
>>> This reverts commit ef86f3a72adb8a7931f67335560740a7ad696d1d.
>>
>> This seems rather odd.  If at all you'd revert the patch adding the
>> PCI_IRQ_AFFINITY to aacraid, not core infrastructure.
> 
> Thank you for the suggestion, but that flag was added in 2016
> to the aacraid driver.
> 
>> commit 0910d8bbdd99856af1394d3d8830955abdefee4a
>> Author: Hannes Reinecke <hare@suse.de>
>> Date:   Tue Nov 8 08:11:30 2016 +0100
>>
>>     scsi: aacraid: switch to pci_alloc_irq_vectors
>>     
>>     Use pci_alloc_irq_vectors and drop the hand-crafted interrupt affinity
>>     routines.
> 
> So what would happen, if `PCI_IRQ_AFFINITY` was removed? Will the
> system still work with the same performance?
> 
> As far as I understood, the no regression policy is there for
> exactly that reason, and it shouldn’t matter if it’s core
> infrastructure or not. As written, I have no idea, and just know
> reverting the commit in question fixes the problem here. So I’ll
> gladly test other solutions to fix this issue.

Just as another datapoint, with `PCI_IRQ_AFFINITY` removed from
`drivers/scsi/aacraid/comminit.c` in Linux 4.14.73, the driver
initializes correctly. I have no idea regarding the performance.


Kind regards,

Paul


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH] Revert "genirq/affinity: assign vectors to all possible CPUs"

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 1973 bytes --]

Dear Greg, dear Linux folks,


On 10/01/18 17:59, Paul Menzel wrote:

> On 10/01/18 14:43, Paul Menzel wrote:
> 
>> On 10/01/18 14:35, Christoph Hellwig wrote:
>>> On Mon, Oct 01, 2018 at 02:33:07PM +0200, Paul Menzel wrote:
>>>> Date: Wed, 29 Aug 2018 17:28:45 +0200
>>>>
>>>> This reverts commit ef86f3a72adb8a7931f67335560740a7ad696d1d.
>>>
>>> This seems rather odd.  If at all you'd revert the patch adding the
>>> PCI_IRQ_AFFINITY to aacraid, not core infrastructure.
>>
>> Thank you for the suggestion, but that flag was added in 2016
>> to the aacraid driver.
>>
>>> commit 0910d8bbdd99856af1394d3d8830955abdefee4a
>>> Author: Hannes Reinecke <hare@suse.de>
>>> Date:   Tue Nov 8 08:11:30 2016 +0100
>>>
>>>     scsi: aacraid: switch to pci_alloc_irq_vectors
>>>     
>>>     Use pci_alloc_irq_vectors and drop the hand-crafted interrupt affinity
>>>     routines.
>>
>> So what would happen, if `PCI_IRQ_AFFINITY` was removed? Will the
>> system still work with the same performance?
>>
>> As far as I understood, the no regression policy is there for
>> exactly that reason, and it shouldn’t matter if it’s core
>> infrastructure or not. As written, I have no idea, and just know
>> reverting the commit in question fixes the problem here. So I’ll
>> gladly test other solutions to fix this issue.
> 
> Just as another datapoint, with `PCI_IRQ_AFFINITY` removed from
> `drivers/scsi/aacraid/comminit.c` in Linux 4.14.73, the driver
> initializes correctly. I have no idea regarding the performance.

This commit has not been picked up yet. I guess, you are busy, but
in case there are still objections, it’d be great if the two
questions below were answered.

1.  What bug is fixed in the LTS series by backporting the commit
    causing the regression?

2.  Why does the *no regression* policy *not* apply in this case?


Kind regards,

Paul


[1]: Documentation/process/stable-kernel-rules.rst


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH] Revert "genirq/affinity: assign vectors to all possible CPUs"

[Greg Kroah-Hartman]

On Mon, Oct 15, 2018 at 02:17:11PM +0200, Paul Menzel wrote:
> Dear Greg, dear Linux folks,
> 
> 
> On 10/01/18 17:59, Paul Menzel wrote:
> 
> > On 10/01/18 14:43, Paul Menzel wrote:
> > 
> >> On 10/01/18 14:35, Christoph Hellwig wrote:
> >>> On Mon, Oct 01, 2018 at 02:33:07PM +0200, Paul Menzel wrote:
> >>>> Date: Wed, 29 Aug 2018 17:28:45 +0200
> >>>>
> >>>> This reverts commit ef86f3a72adb8a7931f67335560740a7ad696d1d.
> >>>
> >>> This seems rather odd.  If at all you'd revert the patch adding the
> >>> PCI_IRQ_AFFINITY to aacraid, not core infrastructure.
> >>
> >> Thank you for the suggestion, but that flag was added in 2016
> >> to the aacraid driver.
> >>
> >>> commit 0910d8bbdd99856af1394d3d8830955abdefee4a
> >>> Author: Hannes Reinecke <hare@suse.de>
> >>> Date:   Tue Nov 8 08:11:30 2016 +0100
> >>>
> >>>     scsi: aacraid: switch to pci_alloc_irq_vectors
> >>>     
> >>>     Use pci_alloc_irq_vectors and drop the hand-crafted interrupt affinity
> >>>     routines.
> >>
> >> So what would happen, if `PCI_IRQ_AFFINITY` was removed? Will the
> >> system still work with the same performance?
> >>
> >> As far as I understood, the no regression policy is there for
> >> exactly that reason, and it shouldn’t matter if it’s core
> >> infrastructure or not. As written, I have no idea, and just know
> >> reverting the commit in question fixes the problem here. So I’ll
> >> gladly test other solutions to fix this issue.
> > 
> > Just as another datapoint, with `PCI_IRQ_AFFINITY` removed from
> > `drivers/scsi/aacraid/comminit.c` in Linux 4.14.73, the driver
> > initializes correctly. I have no idea regarding the performance.
> 
> This commit has not been picked up yet. I guess, you are busy, but
> in case there are still objections, it’d be great if the two
> questions below were answered.
> 
> 1.  What bug is fixed in the LTS series by backporting the commit
>     causing the regression?

I can't remember anymore, but unwinding this mess is going to be a pain :(

> 2.  Why does the *no regression* policy *not* apply in this case?

It does, but also we are following the "stick to what mainline does",
and the fact that this is not showing up in mainline seems just to be a
lucky accident at the moment.  My real worry is that suddenly you are
going to have problems there and that this is just the early-warning
system happening...

thanks,

greg k-h

Re: [PATCH] Revert "genirq/affinity: assign vectors to all possible CPUs"

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 3157 bytes --]

Dear Greg,


On 10/15/18 15:21, Greg Kroah-Hartman wrote:
> On Mon, Oct 15, 2018 at 02:17:11PM +0200, Paul Menzel wrote:

>> On 10/01/18 17:59, Paul Menzel wrote:
>>
>>> On 10/01/18 14:43, Paul Menzel wrote:
>>>
>>>> On 10/01/18 14:35, Christoph Hellwig wrote:
>>>>> On Mon, Oct 01, 2018 at 02:33:07PM +0200, Paul Menzel wrote:
>>>>>> Date: Wed, 29 Aug 2018 17:28:45 +0200
>>>>>>
>>>>>> This reverts commit ef86f3a72adb8a7931f67335560740a7ad696d1d.
>>>>>
>>>>> This seems rather odd.  If at all you'd revert the patch adding the
>>>>> PCI_IRQ_AFFINITY to aacraid, not core infrastructure.
>>>>
>>>> Thank you for the suggestion, but that flag was added in 2016
>>>> to the aacraid driver.
>>>>
>>>>> commit 0910d8bbdd99856af1394d3d8830955abdefee4a
>>>>> Author: Hannes Reinecke <hare@suse.de>
>>>>> Date:   Tue Nov 8 08:11:30 2016 +0100
>>>>>
>>>>>     scsi: aacraid: switch to pci_alloc_irq_vectors
>>>>>     
>>>>>     Use pci_alloc_irq_vectors and drop the hand-crafted interrupt affinity
>>>>>     routines.
>>>>
>>>> So what would happen, if `PCI_IRQ_AFFINITY` was removed? Will the
>>>> system still work with the same performance?
>>>>
>>>> As far as I understood, the no regression policy is there for
>>>> exactly that reason, and it shouldn’t matter if it’s core
>>>> infrastructure or not. As written, I have no idea, and just know
>>>> reverting the commit in question fixes the problem here. So I’ll
>>>> gladly test other solutions to fix this issue.
>>>
>>> Just as another datapoint, with `PCI_IRQ_AFFINITY` removed from
>>> `drivers/scsi/aacraid/comminit.c` in Linux 4.14.73, the driver
>>> initializes correctly. I have no idea regarding the performance.
>>
>> This commit has not been picked up yet. I guess, you are busy, but
>> in case there are still objections, it’d be great if the two
>> questions below were answered.
>>
>> 1.  What bug is fixed in the LTS series by backporting the commit
>>     causing the regression?
> 
> I can't remember anymore, but unwinding this mess is going to be a
> pain :(

Agreed.

>> 2.  Why does the *no regression* policy *not* apply in this case?
> 
> It does, but also we are following the "stick to what mainline does",

Hmm, but I thought only for bug fixes.

> and the fact that this is not showing up in mainline seems just to be a
> lucky accident at the moment.  My real worry is that suddenly you are
> going to have problems there and that this is just the early-warning
> system happening...

It is still a mystery for me, why it doesn’t happen in master.

In the current situation, where the SCSI/AACRAID subsystem folks haven’t
joined the discussion, I still think, the best way for the Linux 4.14
series is to revert.

Additionally, there are other reports about errors with the aacraid
driver [1]. I heard they develop against the Linux kernel version in
the enterprise distributions, and then port that to master. Maybe
that is one of the reasons for the current state. (But also off-topic.)


Kind regards,

Paul


[1]: https://www.spinics.net/lists/linux-scsi/threads.html#123414


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH] Revert "genirq/affinity: assign vectors to all possible CPUs"

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 3573 bytes --]

Dear Greg,


Hopefully, you enjoyed the maintainers summit, and didn’t have too
many emails to catch up on.

On 10/17/18 17:00, Paul Menzel wrote:

> On 10/15/18 15:21, Greg Kroah-Hartman wrote:
>> On Mon, Oct 15, 2018 at 02:17:11PM +0200, Paul Menzel wrote:
> 
>>> On 10/01/18 17:59, Paul Menzel wrote:
>>>
>>>> On 10/01/18 14:43, Paul Menzel wrote:
>>>>
>>>>> On 10/01/18 14:35, Christoph Hellwig wrote:
>>>>>> On Mon, Oct 01, 2018 at 02:33:07PM +0200, Paul Menzel wrote:
>>>>>>> Date: Wed, 29 Aug 2018 17:28:45 +0200
>>>>>>>
>>>>>>> This reverts commit ef86f3a72adb8a7931f67335560740a7ad696d1d.
>>>>>>
>>>>>> This seems rather odd.  If at all you'd revert the patch adding the
>>>>>> PCI_IRQ_AFFINITY to aacraid, not core infrastructure.
>>>>>
>>>>> Thank you for the suggestion, but that flag was added in 2016
>>>>> to the aacraid driver.
>>>>>
>>>>>> commit 0910d8bbdd99856af1394d3d8830955abdefee4a
>>>>>> Author: Hannes Reinecke <hare@suse.de>
>>>>>> Date:   Tue Nov 8 08:11:30 2016 +0100
>>>>>>
>>>>>>     scsi: aacraid: switch to pci_alloc_irq_vectors
>>>>>>     
>>>>>>     Use pci_alloc_irq_vectors and drop the hand-crafted interrupt affinity
>>>>>>     routines.
>>>>>
>>>>> So what would happen, if `PCI_IRQ_AFFINITY` was removed? Will the
>>>>> system still work with the same performance?
>>>>>
>>>>> As far as I understood, the no regression policy is there for
>>>>> exactly that reason, and it shouldn’t matter if it’s core
>>>>> infrastructure or not. As written, I have no idea, and just know
>>>>> reverting the commit in question fixes the problem here. So I’ll
>>>>> gladly test other solutions to fix this issue.
>>>>
>>>> Just as another datapoint, with `PCI_IRQ_AFFINITY` removed from
>>>> `drivers/scsi/aacraid/comminit.c` in Linux 4.14.73, the driver
>>>> initializes correctly. I have no idea regarding the performance.
>>>
>>> This commit has not been picked up yet. I guess, you are busy, but
>>> in case there are still objections, it’d be great if the two
>>> questions below were answered.
>>>
>>> 1.  What bug is fixed in the LTS series by backporting the commit
>>>     causing the regression?
>>
>> I can't remember anymore, but unwinding this mess is going to be a
>> pain :(
> 
> Agreed.
> 
>>> 2.  Why does the *no regression* policy *not* apply in this case?
>>
>> It does, but also we are following the "stick to what mainline does",
> 
> Hmm, but I thought only for bug fixes.
> 
>> and the fact that this is not showing up in mainline seems just to be a
>> lucky accident at the moment.  My real worry is that suddenly you are
>> going to have problems there and that this is just the early-warning
>> system happening...
> 
> It is still a mystery for me, why it doesn’t happen in master.
> 
> In the current situation, where the SCSI/AACRAID subsystem folks
> haven’t joined the discussion, I still think, the best way for the
> Linux 4.14 series is to revert.
> 
> Additionally, there are other reports about errors with the aacraid 
> driver [1]. I heard they develop against the Linux kernel version in 
> the enterprise distributions, and then port that to master. Maybe 
> that is one of the reasons for the current state. (But also
> off-topic.)

Do you have an update, how to deal with this situation in the LTS
series? The longer we wait, the more likely it is, that there will be
conflicts in the future releases.


Kind regards,

Paul


> [1]: https://www.spinics.net/lists/linux-scsi/threads.html#123414


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH] Revert "genirq/affinity: assign vectors to all possible CPUs"

[Ming Lei]

On Mon, Oct 01, 2018 at 02:33:07PM +0200, Paul Menzel wrote:
> Date: Wed, 29 Aug 2018 17:28:45 +0200
> 
> This reverts commit ef86f3a72adb8a7931f67335560740a7ad696d1d.
> 
> See the discussion in the thread *aacraid: Regression in 4.14.56
> with *genirq/affinity: assign vectors to all possible CPUs** on
> the linux-scsi list.
> 
> Reverting the commit, the aacraid driver loads correctly, and the
> drives are visible. So revert the commit to adhere to Linux’ no
> regression policy.
> 
> Strangely, the issue is not reproducible with Linux 4.18.x, but
> Ming Lei wrote, that this might only be by chance.
> 
> Link: https://www.spinics.net/lists/linux-scsi/msg122732.html
> Signed-off-by: Paul Menzel <pmenzel@molgen.mpg.de>
> 
> ---
>  kernel/irq/affinity.c | 30 +++++++++++++++---------------
>  1 file changed, 15 insertions(+), 15 deletions(-)
> 
> diff --git a/kernel/irq/affinity.c b/kernel/irq/affinity.c
> index a37a3b4b6342..e12d35108225 100644
> --- a/kernel/irq/affinity.c
> +++ b/kernel/irq/affinity.c
> @@ -39,7 +39,7 @@ static void irq_spread_init_one(struct cpumask *irqmsk, struct cpumask *nmsk,
>  	}
>  }
>  
> -static cpumask_var_t *alloc_node_to_possible_cpumask(void)
> +static cpumask_var_t *alloc_node_to_present_cpumask(void)
>  {
>  	cpumask_var_t *masks;
>  	int node;
> @@ -62,7 +62,7 @@ static cpumask_var_t *alloc_node_to_possible_cpumask(void)
>  	return NULL;
>  }
>  
> -static void free_node_to_possible_cpumask(cpumask_var_t *masks)
> +static void free_node_to_present_cpumask(cpumask_var_t *masks)
>  {
>  	int node;
>  
> @@ -71,22 +71,22 @@ static void free_node_to_possible_cpumask(cpumask_var_t *masks)
>  	kfree(masks);
>  }
>  
> -static void build_node_to_possible_cpumask(cpumask_var_t *masks)
> +static void build_node_to_present_cpumask(cpumask_var_t *masks)
>  {
>  	int cpu;
>  
> -	for_each_possible_cpu(cpu)
> +	for_each_present_cpu(cpu)
>  		cpumask_set_cpu(cpu, masks[cpu_to_node(cpu)]);
>  }
>  
> -static int get_nodes_in_cpumask(cpumask_var_t *node_to_possible_cpumask,
> +static int get_nodes_in_cpumask(cpumask_var_t *node_to_present_cpumask,
>  				const struct cpumask *mask, nodemask_t *nodemsk)
>  {
>  	int n, nodes = 0;
>  
>  	/* Calculate the number of nodes in the supplied affinity mask */
>  	for_each_node(n) {
> -		if (cpumask_intersects(mask, node_to_possible_cpumask[n])) {
> +		if (cpumask_intersects(mask, node_to_present_cpumask[n])) {
>  			node_set(n, *nodemsk);
>  			nodes++;
>  		}
> @@ -109,7 +109,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>  	int last_affv = affv + affd->pre_vectors;
>  	nodemask_t nodemsk = NODE_MASK_NONE;
>  	struct cpumask *masks;
> -	cpumask_var_t nmsk, *node_to_possible_cpumask;
> +	cpumask_var_t nmsk, *node_to_present_cpumask;
>  
>  	/*
>  	 * If there aren't any vectors left after applying the pre/post
> @@ -125,8 +125,8 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>  	if (!masks)
>  		goto out;
>  
> -	node_to_possible_cpumask = alloc_node_to_possible_cpumask();
> -	if (!node_to_possible_cpumask)
> +	node_to_present_cpumask = alloc_node_to_present_cpumask();
> +	if (!node_to_present_cpumask)
>  		goto out;
>  
>  	/* Fill out vectors at the beginning that don't need affinity */
> @@ -135,8 +135,8 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>  
>  	/* Stabilize the cpumasks */
>  	get_online_cpus();
> -	build_node_to_possible_cpumask(node_to_possible_cpumask);
> -	nodes = get_nodes_in_cpumask(node_to_possible_cpumask, cpu_possible_mask,
> +	build_node_to_present_cpumask(node_to_present_cpumask);
> +	nodes = get_nodes_in_cpumask(node_to_present_cpumask, cpu_present_mask,
>  				     &nodemsk);
>  
>  	/*
> @@ -146,7 +146,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>  	if (affv <= nodes) {
>  		for_each_node_mask(n, nodemsk) {
>  			cpumask_copy(masks + curvec,
> -				     node_to_possible_cpumask[n]);
> +				     node_to_present_cpumask[n]);
>  			if (++curvec == last_affv)
>  				break;
>  		}
> @@ -160,7 +160,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>  		vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;
>  
>  		/* Get the cpus on this node which are in the mask */
> -		cpumask_and(nmsk, cpu_possible_mask, node_to_possible_cpumask[n]);
> +		cpumask_and(nmsk, cpu_present_mask, node_to_present_cpumask[n]);
>  
>  		/* Calculate the number of cpus per vector */
>  		ncpus = cpumask_weight(nmsk);
> @@ -192,7 +192,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>  	/* Fill out vectors at the end that don't need affinity */
>  	for (; curvec < nvecs; curvec++)
>  		cpumask_copy(masks + curvec, irq_default_affinity);
> -	free_node_to_possible_cpumask(node_to_possible_cpumask);
> +	free_node_to_present_cpumask(node_to_present_cpumask);
>  out:
>  	free_cpumask_var(nmsk);
>  	return masks;
> @@ -214,7 +214,7 @@ int irq_calc_affinity_vectors(int minvec, int maxvec, const struct irq_affinity
>  		return 0;
>  
>  	get_online_cpus();
> -	ret = min_t(int, cpumask_weight(cpu_possible_mask), vecs) + resv;
> +	ret = min_t(int, cpumask_weight(cpu_present_mask), vecs) + resv;
>  	put_online_cpus();
>  	return ret;
>  }
> -- 
> 2.17.1
> 

Hello Paul,

I have explained that this is one aacraid issue in the following link:

https://marc.info/?l=linux-kernel&m=153413115909580&w=2

So this issue should have been fixed in the driver instead of genirq
core code, otherwise regression on other drivers can be caused too.

So I suggest you to ask aacraid guys to take a look at this issue.

Thanks,
Ming

Re: [PATCH] Revert "genirq/affinity: assign vectors to all possible CPUs"

[IT (Donald Buczek)]

On 10/08/18 04:11, Ming Lei wrote:
> On Mon, Oct 01, 2018 at 02:33:07PM +0200, Paul Menzel wrote:
>> Date: Wed, 29 Aug 2018 17:28:45 +0200
>>
>> This reverts commit ef86f3a72adb8a7931f67335560740a7ad696d1d.
>>
>> See the discussion in the thread *aacraid: Regression in 4.14.56
>> with *genirq/affinity: assign vectors to all possible CPUs** on
>> the linux-scsi list.
>>
>> Reverting the commit, the aacraid driver loads correctly, and the
>> drives are visible. So revert the commit to adhere to Linux’ no
>> regression policy.
>>
>> Strangely, the issue is not reproducible with Linux 4.18.x, but
>> Ming Lei wrote, that this might only be by chance.
>>
>> Link: https://www.spinics.net/lists/linux-scsi/msg122732.html
>> Signed-off-by: Paul Menzel <pmenzel@molgen.mpg.de>
>>
>> ---
>>   kernel/irq/affinity.c | 30 +++++++++++++++---------------
>>   1 file changed, 15 insertions(+), 15 deletions(-)
>>
>> diff --git a/kernel/irq/affinity.c b/kernel/irq/affinity.c
>> index a37a3b4b6342..e12d35108225 100644
>> --- a/kernel/irq/affinity.c
>> +++ b/kernel/irq/affinity.c
>> @@ -39,7 +39,7 @@ static void irq_spread_init_one(struct cpumask *irqmsk, struct cpumask *nmsk,
>>   	}
>>   }
>>   
>> -static cpumask_var_t *alloc_node_to_possible_cpumask(void)
>> +static cpumask_var_t *alloc_node_to_present_cpumask(void)
>>   {
>>   	cpumask_var_t *masks;
>>   	int node;
>> @@ -62,7 +62,7 @@ static cpumask_var_t *alloc_node_to_possible_cpumask(void)
>>   	return NULL;
>>   }
>>   
>> -static void free_node_to_possible_cpumask(cpumask_var_t *masks)
>> +static void free_node_to_present_cpumask(cpumask_var_t *masks)
>>   {
>>   	int node;
>>   
>> @@ -71,22 +71,22 @@ static void free_node_to_possible_cpumask(cpumask_var_t *masks)
>>   	kfree(masks);
>>   }
>>   
>> -static void build_node_to_possible_cpumask(cpumask_var_t *masks)
>> +static void build_node_to_present_cpumask(cpumask_var_t *masks)
>>   {
>>   	int cpu;
>>   
>> -	for_each_possible_cpu(cpu)
>> +	for_each_present_cpu(cpu)
>>   		cpumask_set_cpu(cpu, masks[cpu_to_node(cpu)]);
>>   }
>>   
>> -static int get_nodes_in_cpumask(cpumask_var_t *node_to_possible_cpumask,
>> +static int get_nodes_in_cpumask(cpumask_var_t *node_to_present_cpumask,
>>   				const struct cpumask *mask, nodemask_t *nodemsk)
>>   {
>>   	int n, nodes = 0;
>>   
>>   	/* Calculate the number of nodes in the supplied affinity mask */
>>   	for_each_node(n) {
>> -		if (cpumask_intersects(mask, node_to_possible_cpumask[n])) {
>> +		if (cpumask_intersects(mask, node_to_present_cpumask[n])) {
>>   			node_set(n, *nodemsk);
>>   			nodes++;
>>   		}
>> @@ -109,7 +109,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>>   	int last_affv = affv + affd->pre_vectors;
>>   	nodemask_t nodemsk = NODE_MASK_NONE;
>>   	struct cpumask *masks;
>> -	cpumask_var_t nmsk, *node_to_possible_cpumask;
>> +	cpumask_var_t nmsk, *node_to_present_cpumask;
>>   
>>   	/*
>>   	 * If there aren't any vectors left after applying the pre/post
>> @@ -125,8 +125,8 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>>   	if (!masks)
>>   		goto out;
>>   
>> -	node_to_possible_cpumask = alloc_node_to_possible_cpumask();
>> -	if (!node_to_possible_cpumask)
>> +	node_to_present_cpumask = alloc_node_to_present_cpumask();
>> +	if (!node_to_present_cpumask)
>>   		goto out;
>>   
>>   	/* Fill out vectors at the beginning that don't need affinity */
>> @@ -135,8 +135,8 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>>   
>>   	/* Stabilize the cpumasks */
>>   	get_online_cpus();
>> -	build_node_to_possible_cpumask(node_to_possible_cpumask);
>> -	nodes = get_nodes_in_cpumask(node_to_possible_cpumask, cpu_possible_mask,
>> +	build_node_to_present_cpumask(node_to_present_cpumask);
>> +	nodes = get_nodes_in_cpumask(node_to_present_cpumask, cpu_present_mask,
>>   				     &nodemsk);
>>   
>>   	/*
>> @@ -146,7 +146,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>>   	if (affv <= nodes) {
>>   		for_each_node_mask(n, nodemsk) {
>>   			cpumask_copy(masks + curvec,
>> -				     node_to_possible_cpumask[n]);
>> +				     node_to_present_cpumask[n]);
>>   			if (++curvec == last_affv)
>>   				break;
>>   		}
>> @@ -160,7 +160,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>>   		vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;
>>   
>>   		/* Get the cpus on this node which are in the mask */
>> -		cpumask_and(nmsk, cpu_possible_mask, node_to_possible_cpumask[n]);
>> +		cpumask_and(nmsk, cpu_present_mask, node_to_present_cpumask[n]);
>>   
>>   		/* Calculate the number of cpus per vector */
>>   		ncpus = cpumask_weight(nmsk);
>> @@ -192,7 +192,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
>>   	/* Fill out vectors at the end that don't need affinity */
>>   	for (; curvec < nvecs; curvec++)
>>   		cpumask_copy(masks + curvec, irq_default_affinity);
>> -	free_node_to_possible_cpumask(node_to_possible_cpumask);
>> +	free_node_to_present_cpumask(node_to_present_cpumask);
>>   out:
>>   	free_cpumask_var(nmsk);
>>   	return masks;
>> @@ -214,7 +214,7 @@ int irq_calc_affinity_vectors(int minvec, int maxvec, const struct irq_affinity
>>   		return 0;
>>   
>>   	get_online_cpus();
>> -	ret = min_t(int, cpumask_weight(cpu_possible_mask), vecs) + resv;
>> +	ret = min_t(int, cpumask_weight(cpu_present_mask), vecs) + resv;
>>   	put_online_cpus();
>>   	return ret;
>>   }
>> -- 
>> 2.17.1
>>
> 
> Hello Paul,
> 
> I have explained that this is one aacraid issue in the following link:
> 
> https://marc.info/?l=linux-kernel&m=153413115909580&w=2
> 
> So this issue should have been fixed in the driver instead of genirq
> core code, otherwise regression on other drivers can be caused too.

Dear Ming,

is your argument, that the bug existed before, because even with the IRQs spread over the present CPUs only, a driver was wrong to assume that every IRQ would be served, because of offline CPUs? So that the change to spread over all possible CPUs did not fundamentally change the API of pci_alloc_irq_vectors() but just increased the already existing chance of a failure?

When you say, reverting this can cause a regression, do you refer to performance or functional regression? I don't see the later yet, because even updated drivers have no guarantee, that all possible CPUs are used for the IRQs, so limiting this to the present CPUs again should do no harm?

Thank you

   Donald

> 
> So I suggest you to ask aacraid guys to take a look at this issue.> 
> Thanks,
> Ming
> 


-- 
Donald Buczek
it@molgen.mpg.de
Tel: +49 30 8413 1433

Re: aacraid: Regression in 4.14.56 with *genirq/affinity: assign vectors to all possible CPUs*

[Greg Kroah-Hartman]

On Tue, Sep 11, 2018 at 12:53:54PM +0200, Paul Menzel wrote:
> Dear Ming, dear Raghava, dear Dave,
> 
> 
> On 08/16/18 19:09, Paul Menzel wrote:
> 
> > On 08/13/18 05:32, Ming Lei wrote:
> >> On Sat, Aug 11, 2018 at 03:50:21PM +0200, Greg Kroah-Hartman wrote:
> >>> On Sat, Aug 11, 2018 at 10:14:18AM +0200, Paul Menzel wrote:
> > 
> >>>> Am 10.08.2018 um 17:55 schrieb Greg Kroah-Hartman:
> >>>>> On Fri, Aug 10, 2018 at 04:11:23PM +0200, Paul Menzel wrote:
> >>>>
> >>>>>> On 08/10/18 15:36, Greg Kroah-Hartman wrote:
> >>>>>>> On Fri, Aug 10, 2018 at 03:21:52PM +0200, Paul Menzel wrote:
> > 
> >>>>>>>> Commit ef86f3a7 (genirq/affinity: assign vectors to all possible CPUs) added
> >>>>>>>> for Linux 4.14.56 causes the aacraid module to not detect the attached devices
> >>>>>>>> anymore on a Dell PowerEdge R720 with two six core 24x E5-2630 @ 2.30GHz.
> >>>>>>>>
> >>>>>>>> ```
> >>>>>>>> $ dmesg | grep raid
> >>>>>>>> [    0.269768] raid6: sse2x1   gen()  7179 MB/s
> >>>>>>>> [    0.290069] raid6: sse2x1   xor()  5636 MB/s
> >>>>>>>> [    0.311068] raid6: sse2x2   gen()  9160 MB/s
> >>>>>>>> [    0.332076] raid6: sse2x2   xor()  6375 MB/s
> >>>>>>>> [    0.353075] raid6: sse2x4   gen() 11164 MB/s
> >>>>>>>> [    0.374064] raid6: sse2x4   xor()  7429 MB/s
> >>>>>>>> [    0.379001] raid6: using algorithm sse2x4 gen() 11164 MB/s
> >>>>>>>> [    0.386001] raid6: .... xor() 7429 MB/s, rmw enabled
> >>>>>>>> [    0.391008] raid6: using ssse3x2 recovery algorithm
> >>>>>>>> [    3.559682] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
> >>>>>>>> [    3.570061] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
> >>>>>>>> [   10.725767] Adaptec aacraid driver 1.2.1[50834]-custom
> >>>>>>>> [   10.731724] aacraid 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control
> >>>>>>>> [   10.743295] aacraid: Comm Interface type3 enabled
> >>>>>>>> $ lspci -nn | grep Adaptec
> >>>>>>>> 04:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
> >>>>>>>> 42:00.0 Serial Attached SCSI controller [0107]: Adaptec Smart Storage PQI 12G SAS/PCIe 3 [9005:028f] (rev 01)
> >>>>>>>> ```
> >>>>>>>>
> >>>>>>>> But, it still works with a Dell PowerEdge R715 with two eight core AMD
> >>>>>>>> Opteron 6136, the card below.
> >>>>>>>>
> >>>>>>>> ```
> >>>>>>>> $ lspci -nn | grep Adaptec
> >>>>>>>> 22:00.0 Serial Attached SCSI controller [0107]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)
> >>>>>>>> ```
> >>>>>>>>
> >>>>>>>> Reverting the commit fixes the issue.
> >>>>>>>>
> >>>>>>>> commit ef86f3a72adb8a7931f67335560740a7ad696d1d
> >>>>>>>> Author: Christoph Hellwig <hch@lst.de>
> >>>>>>>> Date:   Fri Jan 12 10:53:05 2018 +0800
> >>>>>>>>
> >>>>>>>>      genirq/affinity: assign vectors to all possible CPUs
> >>>>>>>>      commit 84676c1f21e8ff54befe985f4f14dc1edc10046b upstream.
> >>>>>>>>      Currently we assign managed interrupt vectors to all present CPUs.  This
> >>>>>>>>      works fine for systems were we only online/offline CPUs.  But in case of
> >>>>>>>>      systems that support physical CPU hotplug (or the virtualized version of
> >>>>>>>>      it) this means the additional CPUs covered for in the ACPI tables or on
> >>>>>>>>      the command line are not catered for.  To fix this we'd either need to
> >>>>>>>>      introduce new hotplug CPU states just for this case, or we can start
> >>>>>>>>      assining vectors to possible but not present CPUs.
> >>>>>>>>      Reported-by: Christian Borntraeger <borntraeger@de.ibm.com>
> >>>>>>>>      Tested-by: Christian Borntraeger <borntraeger@de.ibm.com>
> >>>>>>>>      Tested-by: Stefan Haberland <sth@linux.vnet.ibm.com>
> >>>>>>>>      Fixes: 4b855ad37194 ("blk-mq: Create hctx for each present CPU")
> >>>>>>>>      Cc: linux-kernel@vger.kernel.org
> >>>>>>>>      Cc: Thomas Gleixner <tglx@linutronix.de>
> >>>>>>>>      Signed-off-by: Christoph Hellwig <hch@lst.de>
> >>>>>>>>      Signed-off-by: Jens Axboe <axboe@kernel.dk>
> >>>>>>>>      Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> >>>>>>>>
> >>>>>>>> The problem doesn’t happen with Linux 4.17.11, so there are commits in
> >>>>>>>> Linux master fixing this. Unfortunately, my attempts to find out failed.
> >>>>>>>>
> >>>>>>>> I was able to cherry-pick the three commits below on top of 4.14.62,
> >>>>>>>> but the problem persists.
> >>>>>>>>
> >>>>>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
> >>>>>>>> 355d7ecdea35 scsi: hpsa: fix selection of reply queue
> >>>>>>>> e944e9615741 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> >>>>>>>>
> >>>>>>>> Trying to cherry-pick the commits below, referencing the commit
> >>>>>>>> in question, gave conflicts.
> >>>>>>>>
> >>>>>>>> 1. adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
> >>>>>>>> 2. d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
> >>>>>>>>
> >>>>>>>> To avoid further trial and error with the server with a slow firmware,
> >>>>>>>> do you know what commits should fix the issue?
> >>>>>>>
> >>>>>>> Look at the email on the stable mailing list:
> >>>>>>> 	Subject: Re: Fix for 84676c1f (b5b6e8c8) missing in 4.14.y
> >>>>>>> it should help you out here.
> >>>>>>
> >>>>>> Ah, I didn’t see that [1] yet. Also I can’t find the original message, and a
> >>>>>> way to reply to that thread. Therefore, here is my reply.
> >>>>>>
> >>>>>>> Can you try the patches listed there?
> >>>>>>
> >>>>>> I tried some of these already without success.
> >>>>>>
> >>>>>> b5b6e8c8d3b4 scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> >>>>>> 2f31115e940c scsi: core: introduce force_blk_mq
> >>>>>> adbe552349f2 scsi: megaraid_sas: fix selection of reply queue
> >>>>>>
> >>>>>> The commit above is already in v4.14.56.
> >>>>>>
> >>>>>> 8b834bff1b73 scsi: hpsa: fix selection of reply queue
> >>>>>>
> >>>>>> The problem persists.
> >>>>>>
> >>>>>> The problem also persists with the state below.
> >>>>>>
> >>>>>> 3528f73a4e5d scsi: core: introduce force_blk_mq
> >>>>>> 16dc4d8215f3 scsi: hpsa: fix selection of reply queue
> >>>>>> f0a7ab12232d scsi: virtio_scsi: fix IO hang caused by automatic irq vector affinity
> >>>>>> 6aba81b5a2f5 genirq/affinity: Don't return with empty affinity masks on error
> >>>>>> 1aa1166eface (tag: v4.14.62, stable/linux-4.14.y) Linux 4.14.62
> >>>>>>
> >>>>>> So, some more commits are necessary.
> >>>>>
> >>>>> Or I revert the original patch here, and the follow-on ones that were
> >>>>> added to "fix" this issue.  I think that might be the better thing
> >>>>> overall here, right?  Have you tried that?
> >>>>
> >>>> Yes, reverting the commit fixed the issue for us. If Christoph or Ming do
> >>>> not have another suggestion for a commit, that would be the way to go.
> >>>
> >>> Christoph or Ming, any ideas here?
> >>>
> >>> In looking at the aacraid code, I don't see anywhere that this is using
> >>> a specific cpu number for queues or anything, but I could be wrong.
> >>> Ideally this should also be failing in 4.17 or 4.18-rc right now as well
> >>> as I don't see anything that would have "fixed" this recently.  Unless
> >>> I'm missing something here?
> >>
> >> From both aac_fib_vector_assign() and aac_src_deliver_message(), looks
> >> every msix vector is used inside driver, so causes this trouble.
> >>
> >> From 84676c1f21e8ff54b (genirq/affinity: assign vectors to all possible CPUs),
> >> actually it should have been since 9a0ef98e18 (genirq/affinity: Assign vectors
> >> to all present CPUs), it isn't guaranteed that one vector can be mapped to
> >> at least one online CPU.
> >>
> >> So it should be fixed in driver by similar approach taken in the
> >> following patches:
> >> 	scsi: hpsa: fix selection of reply queue
> >> 	scsi: megaraid_sas: fix selection of reply queue
> >>
> >> And the reason this issue won't be observed is that the following
> >> patches(merged to v4.17) may avoid this issue, but in theory, the issue
> >> won't be avoided completely by these patches:
> >>
> >> d3056812e7df genirq/affinity: Spread irq vectors among present CPUs as far as possible
> >> 1a2d0914e23a genirq/affinity: Allow irq spreading from a given starting point
> >> b3e6aaa8d94d genirq/affinity: Move actual irq vector spreading into a helper function
> >> 47778f33dcba genirq/affinity: Rename *node_to_possible_cpumask as *node_to_cpumask
> >> 0211e12dd0a5 genirq/affinity: Don't return with empty affinity masks on error
> >>
> >> In short, this issue should have been fixed in upstream aacraid, and the idea
> >> is simple, use the current CPU id to retrieve the msix vector offset, this
> >> way will make sure that the msix vector can be mapped from one online
> >> CPU.
> > 
> > AACRAID folks, are you going to work on that? I looked at the code a bit,
> > and it’ll take me some days to understand it and to fix it.
> 
> Unfortunately, there was no reply yet and I won’t be able to fix it any
> time soon. The aacraid driver is still broken in 4.14.68. Reverting the
> *one* mentioned commit fixes the issue.
> 
> Raghava, Dave, are you aware of the issue.

I'm pretty sure this should now be fixed, have you tested this recently?

thanks,

greg k-h