Netdev Archive on lore.kernel.org
 help / color / Atom feed
* High (200+) XFRM interface count performance problem (throughput)
@ 2021-02-23 13:33 Vinš Karel
  2021-02-24  8:14 ` Eyal Birger
  0 siblings, 1 reply; 4+ messages in thread
From: Vinš Karel @ 2021-02-23 13:33 UTC (permalink / raw)
  To: 'netdev@vger.kernel.org'

Hello,

I would like to ask you for help or advise.

I'm testing setup with higher number of XFRM interfaces and I'm facing throughput degradation with a growing number of created XFRM interfaces - not concurrent tunnels established but only XFRM interfaces created - even in DOWN state.
Issue is only unidirectional - from "client" to "vpn hub". Throughput for traffic from hub to client is not affected.

XFRM interface created with:
for i in {1..500}; do link add ipsec$i type xfrm dev ens224 if_id $i  ; done

I'm testing with iperf3 with 1 client connected - from client to hub:
2 interfaces - 1.36 Gbps
100 interfaces - 1.35 Gbps
200 interfaces - 1.19 Gbps
300 interfaces - 0.98 Gbps
500 interfaces - 0.71 Gbps

Throughput from hub to client is around 1.4 Gbps in all cases.

1 CPU core is 100%

Linux v-hub 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Thank you.

Regards
Karel Vins

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: High (200+) XFRM interface count performance problem (throughput)
  2021-02-23 13:33 High (200+) XFRM interface count performance problem (throughput) Vinš Karel
@ 2021-02-24  8:14 ` Eyal Birger
  2021-02-24 11:02   ` [External] " Vinš Karel
  0 siblings, 1 reply; 4+ messages in thread
From: Eyal Birger @ 2021-02-24  8:14 UTC (permalink / raw)
  To: Vinš Karel; +Cc: netdev

Hi Vinš,

On Tue, Feb 23, 2021 at 9:52 PM Vinš Karel <karel.vins@skoda.cz> wrote:
>
> Hello,
>
> I would like to ask you for help or advise.
>
> I'm testing setup with higher number of XFRM interfaces and I'm facing throughput degradation with a growing number of created XFRM interfaces - not concurrent tunnels established but only XFRM interfaces created - even in DOWN state.
> Issue is only unidirectional - from "client" to "vpn hub". Throughput for traffic from hub to client is not affected.
>
> XFRM interface created with:
> for i in {1..500}; do link add ipsec$i type xfrm dev ens224 if_id $i  ; done
>
> I'm testing with iperf3 with 1 client connected - from client to hub:
> 2 interfaces - 1.36 Gbps
> 100 interfaces - 1.35 Gbps
> 200 interfaces - 1.19 Gbps
> 300 interfaces - 0.98 Gbps
> 500 interfaces - 0.71 Gbps
>
> Throughput from hub to client is around 1.4 Gbps in all cases.
>
> 1 CPU core is 100%
>
> Linux v-hub 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Can you please try with a higher kernel version (>= 5.9)?
We've done some work to improve xfrm interface scaling
specifically e98e44562ba2
("xfrm interface: store xfrmi contexts in a hash by if_id").

Thanks,
Eyal.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* RE: [External] Re: High (200+) XFRM interface count performance problem (throughput)
  2021-02-24  8:14 ` Eyal Birger
@ 2021-02-24 11:02   ` Vinš Karel
  2021-02-25 16:28     ` Vinš Karel
  0 siblings, 1 reply; 4+ messages in thread
From: Vinš Karel @ 2021-02-24 11:02 UTC (permalink / raw)
  To: 'Eyal Birger'; +Cc: netdev

Hi Eyal, thank you for response. I found that commit with your comment during the night. I will test it.
Do you think that there is a chance to backport this to 5.4 as it is LTS kernel?

Regards,

Karel

-----Original Message-----
From: Eyal Birger <eyal.birger@gmail.com> 
Sent: Wednesday, February 24, 2021 9:15 AM
To: Vinš Karel <karel.vins@skoda.cz>
Cc: netdev@vger.kernel.org
Subject: [External] Re: High (200+) XFRM interface count performance problem (throughput)

.

Hi Vinš,

On Tue, Feb 23, 2021 at 9:52 PM Vinš Karel <karel.vins@skoda.cz> wrote:
>
> Hello,
>
> I would like to ask you for help or advise.
>
> I'm testing setup with higher number of XFRM interfaces and I'm facing throughput degradation with a growing number of created XFRM interfaces - not concurrent tunnels established but only XFRM interfaces created - even in DOWN state.
> Issue is only unidirectional - from "client" to "vpn hub". Throughput for traffic from hub to client is not affected.
>
> XFRM interface created with:
> for i in {1..500}; do link add ipsec$i type xfrm dev ens224 if_id $i  
> ; done
>
> I'm testing with iperf3 with 1 client connected - from client to hub:
> 2 interfaces - 1.36 Gbps
> 100 interfaces - 1.35 Gbps
> 200 interfaces - 1.19 Gbps
> 300 interfaces - 0.98 Gbps
> 500 interfaces - 0.71 Gbps
>
> Throughput from hub to client is around 1.4 Gbps in all cases.
>
> 1 CPU core is 100%
>
> Linux v-hub 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 
> 2021 x86_64 x86_64 x86_64 GNU/Linux

Can you please try with a higher kernel version (>= 5.9)?
We've done some work to improve xfrm interface scaling specifically e98e44562ba2 ("xfrm interface: store xfrmi contexts in a hash by if_id").

Thanks,
Eyal.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* RE: [External] Re: High (200+) XFRM interface count performance problem (throughput)
  2021-02-24 11:02   ` [External] " Vinš Karel
@ 2021-02-25 16:28     ` Vinš Karel
  0 siblings, 0 replies; 4+ messages in thread
From: Vinš Karel @ 2021-02-25 16:28 UTC (permalink / raw)
  To: 'Eyal Birger'; +Cc: netdev

Hi Eyal,
with kernel 5.10 it work very well. Tested with 10 000 interfaces. Thank you once more.

Regards, Karel

-----Original Message-----
From: Vinš Karel <karel.vins@skoda.cz> 
Sent: Wednesday, February 24, 2021 12:02 PM
To: 'Eyal Birger' <eyal.birger@gmail.com>
Cc: netdev@vger.kernel.org
Subject: [External] RE: [External] Re: High (200+) XFRM interface count performance problem (throughput)

.

Hi Eyal, thank you for response. I found that commit with your comment during the night. I will test it.
Do you think that there is a chance to backport this to 5.4 as it is LTS kernel?

Regards,

Karel

-----Original Message-----
From: Eyal Birger <eyal.birger@gmail.com>
Sent: Wednesday, February 24, 2021 9:15 AM
To: Vinš Karel <karel.vins@skoda.cz>
Cc: netdev@vger.kernel.org
Subject: [External] Re: High (200+) XFRM interface count performance problem (throughput)

.

Hi Vinš,

On Tue, Feb 23, 2021 at 9:52 PM Vinš Karel <karel.vins@skoda.cz> wrote:
>
> Hello,
>
> I would like to ask you for help or advise.
>
> I'm testing setup with higher number of XFRM interfaces and I'm facing throughput degradation with a growing number of created XFRM interfaces - not concurrent tunnels established but only XFRM interfaces created - even in DOWN state.
> Issue is only unidirectional - from "client" to "vpn hub". Throughput for traffic from hub to client is not affected.
>
> XFRM interface created with:
> for i in {1..500}; do link add ipsec$i type xfrm dev ens224 if_id $i ; 
> done
>
> I'm testing with iperf3 with 1 client connected - from client to hub:
> 2 interfaces - 1.36 Gbps
> 100 interfaces - 1.35 Gbps
> 200 interfaces - 1.19 Gbps
> 300 interfaces - 0.98 Gbps
> 500 interfaces - 0.71 Gbps
>
> Throughput from hub to client is around 1.4 Gbps in all cases.
>
> 1 CPU core is 100%
>
> Linux v-hub 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC
> 2021 x86_64 x86_64 x86_64 GNU/Linux

Can you please try with a higher kernel version (>= 5.9)?
We've done some work to improve xfrm interface scaling specifically e98e44562ba2 ("xfrm interface: store xfrmi contexts in a hash by if_id").

Thanks,
Eyal.

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-23 13:33 High (200+) XFRM interface count performance problem (throughput) Vinš Karel
2021-02-24  8:14 ` Eyal Birger
2021-02-24 11:02   ` [External] " Vinš Karel
2021-02-25 16:28     ` Vinš Karel

Netdev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/netdev/0 netdev/git/0.git
	git clone --mirror https://lore.kernel.org/netdev/1 netdev/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 netdev netdev/ https://lore.kernel.org/netdev \
		netdev@vger.kernel.org
	public-inbox-index netdev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.netdev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git