[regression 5.4.97 → 5.10.24]: raid6 avx2x4 speed drops from 18429 MB/s to 6155 MB/s

[Paul Menzel <pmenzel@molgen.mpg.de>]

Dear Linux folks,


On an two socket AMD EPYC 7601, we noticed a decrease in raid6 avx2x4 
speed shown at the beginning of the boot.

                        5.4.95        5.10.24
----------------------------------------------
raid6: avx2x4 gen()   18429 MB/s     6155 MB/s
raid6: avx2x4 xor()    6644 MB/s     4274 MB/s
raid6: avx2x2 gen()   17894 MB/s    18744 MB/s
raid6: avx2x2 xor()   11642 MB/s    11950 MB/s
raid6: avx2x1 gen()   13992 MB/s    17112 MB/s
raid6: avx2x1 xor()   10855 MB/s    11143 MB/s

We are able to reproduce this with different models: Supermicro 
AS-2023US-TR4/H11DSU-iN and Dell PowerEdge R7425 (with different 
microcode versions).

Can you reproduce this on your systems?

Bisecting is going to be hard, so the systems are in production and also 
take a while to boot. (Maybe kexec would help here.)


Kind regards,

Paul


PS: Some more information:

```
[    0.000000] Linux version 5.4.97.mx64.368 
(root@theinternet.molgen.mpg.de) (gcc version 7.5.0 (GCC
)) #1 SMP Wed Feb 10 18:22:50 CET 2021
[…]
[    0.000000] DMI: Supermicro AS -2023US-TR4/H11DSU-iN, BIOS 1.1 02/07/2018
[…]
[    0.630603] raid6: avx2x4   gen() 18429 MB/s
[    0.651607] raid6: avx2x4   xor()  6644 MB/s
[    0.672605] raid6: avx2x2   gen() 17894 MB/s
[    0.693603] raid6: avx2x2   xor() 11642 MB/s
[    0.714605] raid6: avx2x1   gen() 13992 MB/s
[    0.735604] raid6: avx2x1   xor() 10855 MB/s
[    0.756607] raid6: sse2x4   gen() 12246 MB/s
[    0.777605] raid6: sse2x4   xor()  5724 MB/s
[    0.798605] raid6: sse2x2   gen() 10945 MB/s
[    0.819603] raid6: sse2x2   xor()  8097 MB/s
[    0.840606] raid6: sse2x1   gen()  5941 MB/s
[    0.861606] raid6: sse2x1   xor()  5894 MB/s
[    0.866565] raid6: using algorithm avx2x4 gen() 18429 MB/s
[    0.871567] raid6: .... xor() 6644 MB/s, rmw enabled
[    0.877566] raid6: using avx2x2 recovery algorithm
[…]
```


```
[    0.000000] Linux version 5.10.24.mx64.375 
(root@theinternet.molgen.mpg.de) (gcc (GCC) 7.5.0, GNU ld (GNU Binutils) 
2.32) #1 SMP Fri Mar 19 12:29:21 CET 2021
[…]
[    0.000000] DMI: Supermicro AS -2023US-TR4/H11DSU-iN, BIOS 1.1 02/07/2018
[…]
[    0.655382] raid6: avx2x4   gen()  6155 MB/s
[    0.676382] raid6: avx2x4   xor()  4274 MB/s
[    0.697380] raid6: avx2x2   gen() 18744 MB/s
[    0.718380] raid6: avx2x2   xor() 11950 MB/s
[    0.739380] raid6: avx2x1   gen() 17112 MB/s
[    0.760380] raid6: avx2x1   xor() 11143 MB/s
[    0.781381] raid6: sse2x4   gen() 11062 MB/s
[    0.802380] raid6: sse2x4   xor()  5180 MB/s
[    0.823380] raid6: sse2x2   gen() 12467 MB/s
[    0.844380] raid6: sse2x2   xor()  7672 MB/s
[    0.865381] raid6: sse2x1   gen()  9733 MB/s
[    0.886380] raid6: sse2x1   xor()  5717 MB/s
[    0.890674] raid6: using algorithm avx2x2 gen() 18744 MB/s
[    0.895673] raid6: .... xor() 11950 MB/s, rmw enabled
[    0.901673] raid6: using avx2x2 recovery algorithm
```

```
$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          128
On-line CPU(s) list:             0-127
Thread(s) per core:              2
Core(s) per socket:              32
Socket(s):                       2
NUMA node(s):                    8
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           1
Model name:                      AMD EPYC 7601 32-Core Processor
Stepping:                        2
Frequency boost:                 enabled
CPU MHz:                         3100.798
CPU max MHz:                     2200.0000
CPU min MHz:                     1200.0000
BogoMIPS:                        4399.53
Virtualization:                  AMD-V
L1d cache:                       2 MiB
L1i cache:                       4 MiB
L2 cache:                        32 MiB
L3 cache:                        128 MiB
NUMA node0 CPU(s):               0-7,64-71
NUMA node1 CPU(s):               8-15,72-79
NUMA node2 CPU(s):               16-23,80-87
NUMA node3 CPU(s):               24-31,88-95
NUMA node4 CPU(s):               32-39,96-103
NUMA node5 CPU(s):               40-47,104-111
NUMA node6 CPU(s):               48-55,112-119
NUMA node7 CPU(s):               56-63,120-127
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass 
disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers 
and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB 
conditional, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic 
sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx 
mmxex
                                  t fxsr_opt pdpe1gb rdtscp lm 
constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm 
aperfmperf pni pclmulqd
                                  q monitor ssse3 fma cx16 sse4_1 sse4_2 
movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic 
cr8_lega
                                  cy abm sse4a misalignsse 3dnowprefetch 
osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc 
mwaitx c
                                  pb hw_pstate ssbd ibpb vmmcall 
fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt 
xsavec xgetbv1
                                   xsaves clzero irperf xsaveerptr arat 
npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid 
decodeassists paus
                                  efilter pfthreshold avic 
v_vmsave_vmload vgif overflow_recov succor smca
```