Re: Severe performance regression in "net: macsec: preserve ingress frame ordering"

[Ryan Cox <ryan_cox@byu.edu>]

On 8/6/20 9:48 PM, Scott Dial wrote:
> The aes-aesni driver is smart enough to use the FPU if it's not busy and
> fallback to the CPU otherwise. Unfortunately, the ghash-clmulni driver
> does not have that kind of logic in it and only provides an async version,
> so we are forced to use the ghash-generic implementation, which is a pure
> CPU implementation. The ideal would be for aesni_intel to provide a
> synchronous version of gcm(aes) that fell back to the CPU if the FPU is
> busy.

I don't know how the AES-NI support works, but I did see your specific 
mention of aesni_intel and figured I should mention that this does also 
affect AMD. I just got access to AMD nodes (2 x EPYC 7302) with a 
Mellanox 10 GbE NIC.  I did the same test and it had a similar 
performance pattern.  I doubt this means much but I figured I should 
mention it.

> I don't know if the crypto maintainers would be open to such a change, but
> if the choice was between reverting and patching the crypto code, then I
> would work on patching the crypto code.

I can't opine on anything crypto-related since it is extremely way 
outside of my area of expertise, though it is helpful to hear what is 
going on.

> In any case, you didn't report how many packets arrived out of order, which
> was the issue being addressed by my change. It would be helpful to get
> the output of "ip -s macsec show" and specifically the InPktsDelayed
> counter. Did iperf3 report out-of-order packets with the patch reverted?
> Otherwise, if this is the only process running on your test servers,
> then you may not be generating any contention for the FPU, which is the
> source of the out-of-order issue. Maybe you could run prime95 to busy
> the FPU to see the issue that I was seeing.

I ran some tests again on the same servers as before with the Intel 
NICs.  I tested with prime95 running on 27 of the 28 cores in *each* 
server simultaneously (allowing iperf3 to use a core on each) throughout 
the entire test.  This was using 5.7.11 with 
ab046a5d4be4c90a3952a0eae75617b49c0cb01b reverted, so pre-5.7 performance.

MACsec interfaces are deleted and recreated before each test, so 
counters are always fresh.

== MACSEC WITHOUT ENCRYPTION ==

* Server1:
18: ms1: protect on validate strict sc off sa off encrypt off send_sci 
on end_station off scb off replay off
     cipher suite: GCM-AES-128, using ICV length 16
     TXSC: 0000000000001234 on SA 0
     stats: OutPktsUntagged InPktsUntagged OutPktsTooLong InPktsNoTag 
InPktsBadTag InPktsUnknownSCI InPktsNoSCI InPktsOverrun
                          0              0              0 
1123            0                0           1             0
     stats: OutPktsProtected OutPktsEncrypted OutOctetsProtected 
OutOctetsEncrypted
                     3798421                0 30889802591                  0
         0: PN 3799655, state on, key 01000000000000000000000000000000
     stats: OutPktsProtected OutPktsEncrypted
                     3798421                0
     RXSC: 0000000000001234, state on
     stats: InOctetsValidated InOctetsDecrypted InPktsUnchecked 
InPktsDelayed InPktsOK InPktsInvalid InPktsLate InPktsNotValid 
InPktsNotUsingSA InPktsUnusedSA
                  30042694872                 0 0           218  
3675170             0          0 0                0              0
         0: PN 3676633, state on, key 01000000000000000000000000000000
     stats: InPktsOK InPktsInvalid InPktsNotValid InPktsNotUsingSA 
InPktsUnusedSA
             3675170             0              0 0              0

*Server2:
18: ms1: protect on validate strict sc off sa off encrypt off send_sci 
on end_station off scb off replay off
     cipher suite: GCM-AES-128, using ICV length 16
     TXSC: 0000000000001234 on SA 0
     stats: OutPktsUntagged InPktsUntagged OutPktsTooLong InPktsNoTag 
InPktsBadTag InPktsUnknownSCI InPktsNoSCI InPktsOverrun
                          0              0              0 
1227            0                0           1             0
     stats: OutPktsProtected OutPktsEncrypted OutOctetsProtected 
OutOctetsEncrypted
                     3675399                0 30042696158                  0
         0: PN 3676633, state on, key 01000000000000000000000000000000
     stats: OutPktsProtected OutPktsEncrypted
                     3675399                0
     RXSC: 0000000000001234, state on
     stats: InOctetsValidated InOctetsDecrypted InPktsUnchecked 
InPktsDelayed InPktsOK InPktsInvalid InPktsLate InPktsNotValid 
InPktsNotUsingSA InPktsUnusedSA
                  30889801305                 0 0             0  
3798410             0          0 0                0              0
         0: PN 3799655, state on, key 01000000000000000000000000000000
     stats: InPktsOK InPktsInvalid InPktsNotValid InPktsNotUsingSA 
InPktsUnusedSA
             3798410             0              0 0              0


InPktsDelayed was 218 for Server1 and 0 for Server2.

== MACSEC WITH ENCRYPTION ==

I got the following *with* encryption (macsec interface deleted and 
recreated before the test, so counters are fresh):
*Server1:
19: ms1: protect on validate strict sc off sa off encrypt on send_sci on 
end_station off scb off replay off
     cipher suite: GCM-AES-128, using ICV length 16
     TXSC: 0000000000001234 on SA 0
     stats: OutPktsUntagged InPktsUntagged OutPktsTooLong InPktsNoTag 
InPktsBadTag InPktsUnknownSCI InPktsNoSCI InPktsOverrun
                          0              0              0 
1397            0                0           0             0
     stats: OutPktsProtected OutPktsEncrypted OutOctetsProtected 
OutOctetsEncrypted
                           0          5560714 0        46931594623
         0: PN 5561948, state on, key 01000000000000000000000000000000
     stats: OutPktsProtected OutPktsEncrypted
                           0          5560714
     RXSC: 0000000000001234, state on
     stats: InOctetsValidated InOctetsDecrypted InPktsUnchecked 
InPktsDelayed InPktsOK InPktsInvalid InPktsLate InPktsNotValid 
InPktsNotUsingSA InPktsUnusedSA
                            0       45977049585 0          3771  
5417843             0          0 0                0              0
         0: PN 5422860, state on, key 01000000000000000000000000000000
     stats: InPktsOK InPktsInvalid InPktsNotValid InPktsNotUsingSA 
InPktsUnusedSA
             5417843             0              0 0              0

*Server2:
19: ms1: protect on validate strict sc off sa off encrypt on send_sci on 
end_station off scb off replay off
     cipher suite: GCM-AES-128, using ICV length 16
     TXSC: 0000000000001234 on SA 0
     stats: OutPktsUntagged InPktsUntagged OutPktsTooLong InPktsNoTag 
InPktsBadTag InPktsUnknownSCI InPktsNoSCI InPktsOverrun
                          0              0              0 
1490            0                0           0             0
     stats: OutPktsProtected OutPktsEncrypted OutOctetsProtected 
OutOctetsEncrypted
                           0          5421626 0        45977059885
         0: PN 5422860, state on, key 01000000000000000000000000000000
     stats: OutPktsProtected OutPktsEncrypted
                           0          5421626
     RXSC: 0000000000001234, state on
     stats: InOctetsValidated InOctetsDecrypted InPktsUnchecked 
InPktsDelayed InPktsOK InPktsInvalid InPktsLate InPktsNotValid 
InPktsNotUsingSA InPktsUnusedSA
                            0       46931106683 0           109  
5560541             0          0 0                0              0
         0: PN 5561948, state on, key 01000000000000000000000000000000
     stats: InPktsOK InPktsInvalid InPktsNotValid InPktsNotUsingSA 
InPktsUnusedSA
             5560541             0              0 0              0


InPktsDelayed was 3771 for Server1 and 109 for Server2.


The performance numbers were:
* 9.87 Gb/s without macsec
* 6.00 Gb/s with macsec WITHOUT encryption
* 9.19 Gb/s with macsec WITH encryption

iperf3 retransmits were:
* 27 without macsec
* 1211 with macsec WITHOUT encryption
* 721 with macsec WITH encryption


Thanks for the reply and for the background on this.

Ryan