Re: [PATCH RFC xfrm-next v3 0/8] Extend XFRM core to allow full offload configuration

[Leon Romanovsky <leon@kernel.org>]

On Wed, Sep 21, 2022 at 07:59:27AM -0700, Jakub Kicinski wrote:
> On Thu, 8 Sep 2022 12:56:16 +0300 Leon Romanovsky wrote:
> > I have TX traces too and can add if RX are not sufficient. 
> 
> The perf trace is good, but for those of us not intimately familiar
> with xfrm, could you provide some analysis here?

The perf trace presented is for RX path of IPsec crypto offload mode. In that
mode, decrypted packet enters the netdev stack to perform various XFRM specific
checks.

The trace presents "the cost" of these checks, which is 25% according to the
line "--25.80%--xfrm_input".

The xfrm_input has number of "slow" places (other places are not fast either),
which are handled by HW in parallel without any locks in IPsec full offload
mode.

The avoided checks include:
 * XFRM state lookup. It is linked list iteration.
 * Lock of whole xfrm_state. It means that parallel traffic will be
   congested on this lock.
 * Double calculation of replay window protection.
 * Update of replay window.

https://elixir.bootlin.com/linux/v6.0-rc6/source/net/xfrm/xfrm_input.c#L459
int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
{
...
		x = xfrm_state_lookup(net, mark, daddr, spi, nexthdr, family);
...
		spin_lock(&x->lock);
...
		if (xfrm_replay_check(x, skb, seq)) {
...
		spin_unlock(&x->lock);
...
		spin_lock(&x->lock);
...
		if (xfrm_replay_recheck(x, skb, seq)) {
...
		xfrm_replay_advance(x, seq);
.....