From: Jakub Sitnicki <jakub@cloudflare.com>
To: John Fastabend <john.fastabend@gmail.com>
Cc: daniel@iogearbox.net, bpf@vger.kernel.org,
netdev@vger.kernel.org, edumazet@google.com, ast@kernel.org,
andrii@kernel.org, will@isovalent.com
Subject: Re: [PATCH bpf v10 04/14] bpf: sockmap, improved check for empty queue
Date: Tue, 23 May 2023 09:35:42 +0200 [thread overview]
Message-ID: <87h6s37b3f.fsf@cloudflare.com> (raw)
In-Reply-To: <20230523025618.113937-5-john.fastabend@gmail.com>
On Mon, May 22, 2023 at 07:56 PM -07, John Fastabend wrote:
> We noticed some rare sk_buffs were stepping past the queue when system was
> under memory pressure. The general theory is to skip enqueueing
> sk_buffs when its not necessary which is the normal case with a system
> that is properly provisioned for the task, no memory pressure and enough
> cpu assigned.
>
> But, if we can't allocate memory due to an ENOMEM error when enqueueing
> the sk_buff into the sockmap receive queue we push it onto a delayed
> workqueue to retry later. When a new sk_buff is received we then check
> if that queue is empty. However, there is a problem with simply checking
> the queue length. When a sk_buff is being processed from the ingress queue
> but not yet on the sockmap msg receive queue its possible to also recv
> a sk_buff through normal path. It will check the ingress queue which is
> zero and then skip ahead of the pkt being processed.
>
> Previously we used sock lock from both contexts which made the problem
> harder to hit, but not impossible.
>
> To fix instead of popping the skb from the queue entirely we peek the
> skb from the queue and do the copy there. This ensures checks to the
> queue length are non-zero while skb is being processed. Then finally
> when the entire skb has been copied to user space queue or another
> socket we pop it off the queue. This way the queue length check allows
> bypassing the queue only after the list has been completely processed.
>
> To reproduce issue we run NGINX compliance test with sockmap running and
> observe some flakes in our testing that we attributed to this issue.
>
> Fixes: 04919bed948dc ("tcp: Introduce tcp_read_skb()")
> Tested-by: William Findlay <will@isovalent.com>
> Suggested-by: Jakub Sitnicki <jakub@cloudflare.com>
> Signed-off-by: John Fastabend <john.fastabend@gmail.com>
> ---
Reviewed-by: Jakub Sitnicki <jakub@cloudflare.com>
next prev parent reply other threads:[~2023-05-23 7:36 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-23 2:56 [PATCH bpf v10 00/14] bpf sockmap fixes John Fastabend
2023-05-23 2:56 ` [PATCH bpf v10 01/14] bpf: sockmap, pass skb ownership through read_skb John Fastabend
2023-05-23 2:56 ` [PATCH bpf v10 02/14] bpf: sockmap, convert schedule_work into delayed_work John Fastabend
2023-05-23 2:56 ` [PATCH bpf v10 03/14] bpf: sockmap, reschedule is now done through backlog John Fastabend
2023-05-23 2:56 ` [PATCH bpf v10 04/14] bpf: sockmap, improved check for empty queue John Fastabend
2023-05-23 7:35 ` Jakub Sitnicki [this message]
2023-05-23 2:56 ` [PATCH bpf v10 05/14] bpf: sockmap, handle fin correctly John Fastabend
2023-05-23 2:56 ` [PATCH bpf v10 06/14] bpf: sockmap, TCP data stall on recv before accept John Fastabend
2023-05-23 2:56 ` [PATCH bpf v10 07/14] bpf: sockmap, wake up polling after data copy John Fastabend
2023-05-30 6:30 ` Eric Dumazet
2023-05-30 18:34 ` John Fastabend
2023-05-30 18:43 ` John Fastabend
2023-05-30 18:51 ` Eric Dumazet
2023-05-23 2:56 ` [PATCH bpf v10 08/14] bpf: sockmap, incorrectly handling copied_seq John Fastabend
2023-05-23 9:09 ` Jakub Sitnicki
2023-05-23 2:56 ` [PATCH bpf v10 09/14] bpf: sockmap, pull socket helpers out of listen test for general use John Fastabend
2023-05-23 2:56 ` [PATCH bpf v10 10/14] bpf: sockmap, build helper to create connected socket pair John Fastabend
2023-05-23 9:23 ` Jakub Sitnicki
2023-05-23 2:56 ` [PATCH bpf v10 11/14] bpf: sockmap, test shutdown() correctly exits epoll and recv()=0 John Fastabend
2023-05-23 9:41 ` Jakub Sitnicki
2023-05-23 2:56 ` [PATCH bpf v10 12/14] bpf: sockmap, test FIONREAD returns correct bytes in rx buffer John Fastabend
2023-05-23 2:56 ` [PATCH bpf v10 13/14] bpf: sockmap, test FIONREAD returns correct bytes in rx buffer with drops John Fastabend
2023-05-23 2:56 ` [PATCH bpf v10 14/14] bpf: sockmap, test progs verifier error with latest clang John Fastabend
2023-05-23 10:00 ` Jakub Sitnicki
2023-05-23 14:31 ` [PATCH bpf v10 00/14] bpf sockmap fixes Daniel Borkmann
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87h6s37b3f.fsf@cloudflare.com \
--to=jakub@cloudflare.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=edumazet@google.com \
--cc=john.fastabend@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=will@isovalent.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.