[PATCH net] net: bpf: fix request_sock leak in filter.c

[PATCH net] net: bpf: fix request_sock leak in filter.c

[Jon Maxwell]

A customer reported a request_socket leak in a Calico cloud environment. We 
found that a BPF program was doing a socket lookup with takes a refcnt on 
the socket and that it was finding the request_socket but returning the parent 
LISTEN socket via sk_to_full_sk() without decrementing the child request socket 
1st, resulting in request_sock slab object leak. This patch retains the 
existing behaviour of returning full socks to the caller but it also decrements
the child request_socket if one is present before doing so to prevent the leak.

Thanks to Curtis Taylor for all the help in diagnosing and testing this. And 
thanks to Antoine Tenart for the reproducer and patch input.

Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
Co-developed-by: Antoine Tenart <atenart@kernel.org>
Signed-off-by:: Antoine Tenart <atenart@kernel.org>
Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
---
 net/core/filter.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/net/core/filter.c b/net/core/filter.c
index 2e32cee2c469..e3c04ae7381f 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
 {
 	struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
 					   ifindex, proto, netns_id, flags);
+	struct sock *sk1 = sk;
 
 	if (sk) {
 		sk = sk_to_full_sk(sk);
-		if (!sk_fullsock(sk)) {
-			sock_gen_put(sk);
+		/* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
+		 * sock refcnt is decremented to prevent a request_sock leak.
+		 */
+		if (!sk_fullsock(sk1))
+			sock_gen_put(sk1);
+		if (!sk_fullsock(sk))
 			return NULL;
-		}
 	}
 
 	return sk;
@@ -6239,13 +6243,17 @@ bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
 {
 	struct sock *sk = bpf_skc_lookup(skb, tuple, len, proto, netns_id,
 					 flags);
+	struct sock *sk1 = sk;
 
 	if (sk) {
 		sk = sk_to_full_sk(sk);
-		if (!sk_fullsock(sk)) {
-			sock_gen_put(sk);
+		/* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
+		 * sock refcnt is decremented to prevent a request_sock leak.
+		 */
+		if (!sk_fullsock(sk1))
+			sock_gen_put(sk1);
+		if (!sk_fullsock(sk))
 			return NULL;
-		}
 	}
 
 	return sk;
-- 
2.31.1

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Antoine Tenart]

Hi Jon,

Quoting Jon Maxwell (2022-06-09 03:18:44)
> A customer reported a request_socket leak in a Calico cloud environment. We 
> found that a BPF program was doing a socket lookup with takes a refcnt on 
> the socket and that it was finding the request_socket but returning the parent 
> LISTEN socket via sk_to_full_sk() without decrementing the child request socket 
> 1st, resulting in request_sock slab object leak. This patch retains the 
> existing behaviour of returning full socks to the caller but it also decrements
> the child request_socket if one is present before doing so to prevent the leak.
> 
> Thanks to Curtis Taylor for all the help in diagnosing and testing this. And 
> thanks to Antoine Tenart for the reproducer and patch input.
> 
> Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")

"bpf:" should be inside the parenthesis in the two above lines.

Isn't the issue from before edbf8c01de5a for bpf_sk_lookup? Looking at a
5.1 kernel[1], __bpf_sk_lookup was called and also did the full socket
translation[2]. bpf_sk_release would not be called on the original
socket when that happens.

[1] https://elixir.bootlin.com/linux/v5.1/source/net/core/filter.c#L5204
[2] https://elixir.bootlin.com/linux/v5.1/source/net/core/filter.c#L5198

> Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
> Co-developed-by: Antoine Tenart <atenart@kernel.org>
> Signed-off-by:: Antoine Tenart <atenart@kernel.org>

Please remove the extra ':'.

Thanks!
Antoine

> Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
> ---
>  net/core/filter.c | 20 ++++++++++++++------
>  1 file changed, 14 insertions(+), 6 deletions(-)
> 
> diff --git a/net/core/filter.c b/net/core/filter.c
> index 2e32cee2c469..e3c04ae7381f 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
>  {
>         struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
>                                            ifindex, proto, netns_id, flags);
> +       struct sock *sk1 = sk;
>  
>         if (sk) {
>                 sk = sk_to_full_sk(sk);
> -               if (!sk_fullsock(sk)) {
> -                       sock_gen_put(sk);
> +               /* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> +                * sock refcnt is decremented to prevent a request_sock leak.
> +                */
> +               if (!sk_fullsock(sk1))
> +                       sock_gen_put(sk1);
> +               if (!sk_fullsock(sk))
>                         return NULL;
> -               }
>         }
>  
>         return sk;
> @@ -6239,13 +6243,17 @@ bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
>  {
>         struct sock *sk = bpf_skc_lookup(skb, tuple, len, proto, netns_id,
>                                          flags);
> +       struct sock *sk1 = sk;
>  
>         if (sk) {
>                 sk = sk_to_full_sk(sk);
> -               if (!sk_fullsock(sk)) {
> -                       sock_gen_put(sk);
> +               /* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> +                * sock refcnt is decremented to prevent a request_sock leak.
> +                */
> +               if (!sk_fullsock(sk1))
> +                       sock_gen_put(sk1);
> +               if (!sk_fullsock(sk))
>                         return NULL;
> -               }
>         }
>  
>         return sk;
> -- 
> 2.31.1
> 

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Daniel Borkmann]

On 6/9/22 3:18 AM, Jon Maxwell wrote:
> A customer reported a request_socket leak in a Calico cloud environment. We
> found that a BPF program was doing a socket lookup with takes a refcnt on
> the socket and that it was finding the request_socket but returning the parent
> LISTEN socket via sk_to_full_sk() without decrementing the child request socket
> 1st, resulting in request_sock slab object leak. This patch retains the
> existing behaviour of returning full socks to the caller but it also decrements
> the child request_socket if one is present before doing so to prevent the leak.
> 
> Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
> thanks to Antoine Tenart for the reproducer and patch input.
> 
> Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
> Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
> Co-developed-by: Antoine Tenart <atenart@kernel.org>
> Signed-off-by:: Antoine Tenart <atenart@kernel.org>
> Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
> ---
>   net/core/filter.c | 20 ++++++++++++++------
>   1 file changed, 14 insertions(+), 6 deletions(-)
> 
> diff --git a/net/core/filter.c b/net/core/filter.c
> index 2e32cee2c469..e3c04ae7381f 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
>   {
>   	struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
>   					   ifindex, proto, netns_id, flags);
> +	struct sock *sk1 = sk;
>   
>   	if (sk) {
>   		sk = sk_to_full_sk(sk);
> -		if (!sk_fullsock(sk)) {
> -			sock_gen_put(sk);
> +		/* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> +		 * sock refcnt is decremented to prevent a request_sock leak.
> +		 */
> +		if (!sk_fullsock(sk1))
> +			sock_gen_put(sk1);
> +		if (!sk_fullsock(sk))
>   			return NULL;

[ +Martin/Joe/Lorenz ]

I wonder, should we also add some asserts in here to ensure we don't get an unbalance for the
bpf_sk_release() case later on? Rough pseudocode could be something like below:

static struct sock *
__bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
                 struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id,
                 u64 flags)
{
         struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
                                            ifindex, proto, netns_id, flags);
         if (sk) {
                 struct sock *sk2 = sk_to_full_sk(sk);

                 if (!sk_fullsock(sk2))
                         sk2 = NULL;
                 if (sk2 != sk) {
                         sock_gen_put(sk);
                         if (unlikely(sk2 && !sock_flag(sk2, SOCK_RCU_FREE))) {
                                 WARN_ONCE(1, "Found non-RCU, unreferenced socket!");
                                 sk2 = NULL;
                         }
                 }
                 sk = sk2;
         }
         return sk;
}

Thanks,
Daniel

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Joe Stringer]

On Wed, Jun 8, 2022 at 6:21 PM Jon Maxwell <jmaxwell37@gmail.com> wrote:
>
> A customer reported a request_socket leak in a Calico cloud environment. We
> found that a BPF program was doing a socket lookup with takes a refcnt on
> the socket and that it was finding the request_socket but returning the parent
> LISTEN socket via sk_to_full_sk() without decrementing the child request socket
> 1st, resulting in request_sock slab object leak. This patch retains the
> existing behaviour of returning full socks to the caller but it also decrements
> the child request_socket if one is present before doing so to prevent the leak.
>
> Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
> thanks to Antoine Tenart for the reproducer and patch input.
>
> Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
> Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
> Co-developed-by: Antoine Tenart <atenart@kernel.org>
> Signed-off-by:: Antoine Tenart <atenart@kernel.org>
> Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
> ---
>  net/core/filter.c | 20 ++++++++++++++------
>  1 file changed, 14 insertions(+), 6 deletions(-)
>
> diff --git a/net/core/filter.c b/net/core/filter.c
> index 2e32cee2c469..e3c04ae7381f 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
>  {
>         struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
>                                            ifindex, proto, netns_id, flags);
> +       struct sock *sk1 = sk;
>
>         if (sk) {
>                 sk = sk_to_full_sk(sk);
> -               if (!sk_fullsock(sk)) {
> -                       sock_gen_put(sk);
> +               /* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> +                * sock refcnt is decremented to prevent a request_sock leak.
> +                */
> +               if (!sk_fullsock(sk1))
> +                       sock_gen_put(sk1);
> +               if (!sk_fullsock(sk))
>                         return NULL;
> -               }
>         }
>
>         return sk;

Thinking through the constraints of this function:
1. If the return value is NULL, then all references taken during the
processing must be released.
2. If the return value is non-NULL, then the socket must either have
gained one reference OR it must have the SOCK_RCU_FREE flag set.
3. It also shouldn't return TIME_WAIT / request sockets (!sk_fullsock(sk)).

__bpf_skc_lookup() will give us the properties of (1)/(2) in a socket
that may or may not be `sk_is_refcounted()` at the start of the
function, so then we just need to consider the logic being changed
here.

Digging further, are these statements accurate?
* sk_to_full_sk() can either return the argument or a different listen socket.
* Iff sk1 and sk are the same, then we only need to consider (3),
hence the fullsock check, then depending on what type of socket it is,
we satisfy either (1) (current sock_gen_put() call + NULL) or (2)
(just return).
* Iff sk1 and sk are different, then we should release the reference
on sk1 and then do something with sk following the constraints above.
* Iff sk1 and sk are different, then sk must be a LISTEN socket.
* LISTEN sockets always have SOCK_RCU_FREE.
* Therefore, if sk1 and sk are different, we must release the
reference on sk1 and we do not need to take a reference on sk, and we
can just return sk.

Following the above, the implementation looks concise and follows the
logic for each case. I can't help but think that it would be easier to
read with an sk_is_refcounted() call in there though since the concern
is how the references for sk vs sk1 are tracked in this function.

Thanks,
Joe

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Joe Stringer]

On Thu, Jun 9, 2022 at 1:30 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On 6/9/22 3:18 AM, Jon Maxwell wrote:
> > A customer reported a request_socket leak in a Calico cloud environment. We
> > found that a BPF program was doing a socket lookup with takes a refcnt on
> > the socket and that it was finding the request_socket but returning the parent
> > LISTEN socket via sk_to_full_sk() without decrementing the child request socket
> > 1st, resulting in request_sock slab object leak. This patch retains the
> > existing behaviour of returning full socks to the caller but it also decrements
> > the child request_socket if one is present before doing so to prevent the leak.
> >
> > Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
> > thanks to Antoine Tenart for the reproducer and patch input.
> >
> > Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> > Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
> > Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
> > Co-developed-by: Antoine Tenart <atenart@kernel.org>
> > Signed-off-by:: Antoine Tenart <atenart@kernel.org>
> > Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
> > ---
> >   net/core/filter.c | 20 ++++++++++++++------
> >   1 file changed, 14 insertions(+), 6 deletions(-)
> >
> > diff --git a/net/core/filter.c b/net/core/filter.c
> > index 2e32cee2c469..e3c04ae7381f 100644
> > --- a/net/core/filter.c
> > +++ b/net/core/filter.c
> > @@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
> >   {
> >       struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
> >                                          ifindex, proto, netns_id, flags);
> > +     struct sock *sk1 = sk;
> >
> >       if (sk) {
> >               sk = sk_to_full_sk(sk);
> > -             if (!sk_fullsock(sk)) {
> > -                     sock_gen_put(sk);
> > +             /* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> > +              * sock refcnt is decremented to prevent a request_sock leak.
> > +              */
> > +             if (!sk_fullsock(sk1))
> > +                     sock_gen_put(sk1);
> > +             if (!sk_fullsock(sk))
> >                       return NULL;
>
> [ +Martin/Joe/Lorenz ]
>
> I wonder, should we also add some asserts in here to ensure we don't get an unbalance for the
> bpf_sk_release() case later on? Rough pseudocode could be something like below:
>
> static struct sock *
> __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
>                  struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id,
>                  u64 flags)
> {
>          struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
>                                             ifindex, proto, netns_id, flags);
>          if (sk) {
>                  struct sock *sk2 = sk_to_full_sk(sk);
>
>                  if (!sk_fullsock(sk2))
>                          sk2 = NULL;
>                  if (sk2 != sk) {
>                          sock_gen_put(sk);
>                          if (unlikely(sk2 && !sock_flag(sk2, SOCK_RCU_FREE))) {
>                                  WARN_ONCE(1, "Found non-RCU, unreferenced socket!");
>                                  sk2 = NULL;
>                          }
>                  }
>                  sk = sk2;
>          }
>          return sk;
> }

This seems a bit more readable to me from the perspective of
understanding the way that the socket references are tracked & freed.

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Jonathan Maxwell]

On Fri, Jun 10, 2022 at 8:22 AM Joe Stringer <joe@cilium.io> wrote:
>
> On Thu, Jun 9, 2022 at 1:30 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
> >
> > On 6/9/22 3:18 AM, Jon Maxwell wrote:
> > > A customer reported a request_socket leak in a Calico cloud environment. We
> > > found that a BPF program was doing a socket lookup with takes a refcnt on
> > > the socket and that it was finding the request_socket but returning the parent
> > > LISTEN socket via sk_to_full_sk() without decrementing the child request socket
> > > 1st, resulting in request_sock slab object leak. This patch retains the
> > > existing behaviour of returning full socks to the caller but it also decrements
> > > the child request_socket if one is present before doing so to prevent the leak.
> > >
> > > Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
> > > thanks to Antoine Tenart for the reproducer and patch input.
> > >
> > > Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> > > Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
> > > Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
> > > Co-developed-by: Antoine Tenart <atenart@kernel.org>
> > > Signed-off-by:: Antoine Tenart <atenart@kernel.org>
> > > Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
> > > ---
> > >   net/core/filter.c | 20 ++++++++++++++------
> > >   1 file changed, 14 insertions(+), 6 deletions(-)
> > >
> > > diff --git a/net/core/filter.c b/net/core/filter.c
> > > index 2e32cee2c469..e3c04ae7381f 100644
> > > --- a/net/core/filter.c
> > > +++ b/net/core/filter.c
> > > @@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
> > >   {
> > >       struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
> > >                                          ifindex, proto, netns_id, flags);
> > > +     struct sock *sk1 = sk;
> > >
> > >       if (sk) {
> > >               sk = sk_to_full_sk(sk);
> > > -             if (!sk_fullsock(sk)) {
> > > -                     sock_gen_put(sk);
> > > +             /* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> > > +              * sock refcnt is decremented to prevent a request_sock leak.
> > > +              */
> > > +             if (!sk_fullsock(sk1))
> > > +                     sock_gen_put(sk1);
> > > +             if (!sk_fullsock(sk))
> > >                       return NULL;
> >
> > [ +Martin/Joe/Lorenz ]
> >
> > I wonder, should we also add some asserts in here to ensure we don't get an unbalance for the
> > bpf_sk_release() case later on? Rough pseudocode could be something like below:
> >
> > static struct sock *
> > __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
> >                  struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id,
> >                  u64 flags)
> > {
> >          struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
> >                                             ifindex, proto, netns_id, flags);
> >          if (sk) {
> >                  struct sock *sk2 = sk_to_full_sk(sk);
> >
> >                  if (!sk_fullsock(sk2))
> >                          sk2 = NULL;
> >                  if (sk2 != sk) {
> >                          sock_gen_put(sk);
> >                          if (unlikely(sk2 && !sock_flag(sk2, SOCK_RCU_FREE))) {
> >                                  WARN_ONCE(1, "Found non-RCU, unreferenced socket!");
> >                                  sk2 = NULL;
> >                          }
> >                  }
> >                  sk = sk2;
> >          }
> >          return sk;
> > }
>
> This seems a bit more readable to me from the perspective of
> understanding the way that the socket references are tracked & freed.

Thanks for the suggestion Daniel and Joe, looks good to me, we will run some
tests with that implemented in our reproducer.

Regards

Jon

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Martin KaFai Lau]

On Thu, Jun 09, 2022 at 10:29:15PM +0200, Daniel Borkmann wrote:
> On 6/9/22 3:18 AM, Jon Maxwell wrote:
> > A customer reported a request_socket leak in a Calico cloud environment. We
> > found that a BPF program was doing a socket lookup with takes a refcnt on
> > the socket and that it was finding the request_socket but returning the parent
> > LISTEN socket via sk_to_full_sk() without decrementing the child request socket
> > 1st, resulting in request_sock slab object leak. This patch retains the
Great catch and debug indeed!

> > existing behaviour of returning full socks to the caller but it also decrements
> > the child request_socket if one is present before doing so to prevent the leak.
> > 
> > Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
> > thanks to Antoine Tenart for the reproducer and patch input.
> > 
> > Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> > Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
Instead of the above commits, I think this dated back to
6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")

> > Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
> > Co-developed-by: Antoine Tenart <atenart@kernel.org>
> > Signed-off-by:: Antoine Tenart <atenart@kernel.org>
> > Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
> > ---
> >   net/core/filter.c | 20 ++++++++++++++------
> >   1 file changed, 14 insertions(+), 6 deletions(-)
> > 
> > diff --git a/net/core/filter.c b/net/core/filter.c
> > index 2e32cee2c469..e3c04ae7381f 100644
> > --- a/net/core/filter.c
> > +++ b/net/core/filter.c
> > @@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
> >   {
> >   	struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
> >   					   ifindex, proto, netns_id, flags);
> > +	struct sock *sk1 = sk;
> >   	if (sk) {
> >   		sk = sk_to_full_sk(sk);
> > -		if (!sk_fullsock(sk)) {
> > -			sock_gen_put(sk);
> > +		/* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> > +		 * sock refcnt is decremented to prevent a request_sock leak.
> > +		 */
> > +		if (!sk_fullsock(sk1))
> > +			sock_gen_put(sk1);
> > +		if (!sk_fullsock(sk))
In this case, sk1 == sk (timewait).  It is a bit worrying to pass
sk to sk_fullsock(sk) after the above sock_gen_put().
I think Daniel's 'if (sk2 != sk) { sock_gen_put(sk); }' check is better.

> 
> [ +Martin/Joe/Lorenz ]
> 
> I wonder, should we also add some asserts in here to ensure we don't get an unbalance for the
> bpf_sk_release() case later on? Rough pseudocode could be something like below:
> 
> static struct sock *
> __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
>                 struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id,
>                 u64 flags)
> {
>         struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
>                                            ifindex, proto, netns_id, flags);
>         if (sk) {
>                 struct sock *sk2 = sk_to_full_sk(sk);
> 
>                 if (!sk_fullsock(sk2))
>                         sk2 = NULL;
>                 if (sk2 != sk) {
>                         sock_gen_put(sk);
>                         if (unlikely(sk2 && !sock_flag(sk2, SOCK_RCU_FREE))) {
I don't think it matters if the helper-returned sk2 is refcounted or not (SOCK_RCU_FREE).
The verifier has ensured the bpf_sk_lookup() and bpf_sk_release() are
always balanced regardless of the type of sk2.

bpf_sk_release() will do the right thing to check the sk2 is refcounted or not
before calling sock_gen_put().

The bug here is the helper forgot to call sock_gen_put(sk) while
the verifier only tracks the sk2, so I think the 'if (unlikely...) { WARN_ONCE(...); }'
can be saved.

>                                 WARN_ONCE(1, "Found non-RCU, unreferenced socket!");
>                                 sk2 = NULL;
>                         }
>                 }
>                 sk = sk2;
>         }
>         return sk;
> }

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Martin KaFai Lau]

On Thu, Jun 09, 2022 at 05:17:47PM -0700, Martin KaFai Lau wrote:
> On Thu, Jun 09, 2022 at 10:29:15PM +0200, Daniel Borkmann wrote:
> > On 6/9/22 3:18 AM, Jon Maxwell wrote:
> > > A customer reported a request_socket leak in a Calico cloud environment. We
> > > found that a BPF program was doing a socket lookup with takes a refcnt on
> > > the socket and that it was finding the request_socket but returning the parent
> > > LISTEN socket via sk_to_full_sk() without decrementing the child request socket
> > > 1st, resulting in request_sock slab object leak. This patch retains the
> Great catch and debug indeed!
> 
> > > existing behaviour of returning full socks to the caller but it also decrements
> > > the child request_socket if one is present before doing so to prevent the leak.
> > > 
> > > Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
> > > thanks to Antoine Tenart for the reproducer and patch input.
> > > 
> > > Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> > > Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
> Instead of the above commits, I think this dated back to
> 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")

Since this is more bpf specific, I think it could go to the bpf tree.
In v2, please cc bpf@vger.kernel.org and tag it with 'PATCH v2 bpf'.

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Jonathan Maxwell]

On Fri, Jun 10, 2022 at 10:36 AM Martin KaFai Lau <kafai@fb.com> wrote:
>
> On Thu, Jun 09, 2022 at 05:17:47PM -0700, Martin KaFai Lau wrote:
> > On Thu, Jun 09, 2022 at 10:29:15PM +0200, Daniel Borkmann wrote:
> > > On 6/9/22 3:18 AM, Jon Maxwell wrote:
> > > > A customer reported a request_socket leak in a Calico cloud environment. We
> > > > found that a BPF program was doing a socket lookup with takes a refcnt on
> > > > the socket and that it was finding the request_socket but returning the parent
> > > > LISTEN socket via sk_to_full_sk() without decrementing the child request socket
> > > > 1st, resulting in request_sock slab object leak. This patch retains the
> > Great catch and debug indeed!
> >
> > > > existing behaviour of returning full socks to the caller but it also decrements
> > > > the child request_socket if one is present before doing so to prevent the leak.
> > > >
> > > > Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
> > > > thanks to Antoine Tenart for the reproducer and patch input.
> > > >
> > > > Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> > > > Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
> > Instead of the above commits, I think this dated back to
> > 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
>
> Since this is more bpf specific, I think it could go to the bpf tree.
> In v2, please cc bpf@vger.kernel.org and tag it with 'PATCH v2 bpf'.

Okay thanks will do.

Daniel, are you okay with omitting 'if (unlikely...) { WARN_ONCE(...); }'?

If so I'll stick to the rest of the logic of your suggestion and omit that
check in v1.

Regards

Jon

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Jonathan Maxwell]

On Thu, Jun 9, 2022 at 11:35 PM Antoine Tenart <atenart@kernel.org> wrote:
>
> Hi Jon,
>
> Quoting Jon Maxwell (2022-06-09 03:18:44)
> > A customer reported a request_socket leak in a Calico cloud environment. We
> > found that a BPF program was doing a socket lookup with takes a refcnt on
> > the socket and that it was finding the request_socket but returning the parent
> > LISTEN socket via sk_to_full_sk() without decrementing the child request socket
> > 1st, resulting in request_sock slab object leak. This patch retains the
> > existing behaviour of returning full socks to the caller but it also decrements
> > the child request_socket if one is present before doing so to prevent the leak.
> >
> > Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
> > thanks to Antoine Tenart for the reproducer and patch input.
> >
> > Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> > Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
>
> "bpf:" should be inside the parenthesis in the two above lines.
>
> Isn't the issue from before edbf8c01de5a for bpf_sk_lookup? Looking at a
> 5.1 kernel[1], __bpf_sk_lookup was called and also did the full socket
> translation[2]. bpf_sk_release would not be called on the original
> socket when that happens.
>
> [1] https://elixir.bootlin.com/linux/v5.1/source/net/core/filter.c#L5204
> [2] https://elixir.bootlin.com/linux/v5.1/source/net/core/filter.c#L5198
>
> > Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
> > Co-developed-by: Antoine Tenart <atenart@kernel.org>
> > Signed-off-by:: Antoine Tenart <atenart@kernel.org>
>
> Please remove the extra ':'.
>

Sure will correct those typos in v1.

Regards

Jon

> Thanks!
> Antoine
>
> > Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
> > ---
> >  net/core/filter.c | 20 ++++++++++++++------
> >  1 file changed, 14 insertions(+), 6 deletions(-)
> >
> > diff --git a/net/core/filter.c b/net/core/filter.c
> > index 2e32cee2c469..e3c04ae7381f 100644
> > --- a/net/core/filter.c
> > +++ b/net/core/filter.c
> > @@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
> >  {
> >         struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
> >                                            ifindex, proto, netns_id, flags);
> > +       struct sock *sk1 = sk;
> >
> >         if (sk) {
> >                 sk = sk_to_full_sk(sk);
> > -               if (!sk_fullsock(sk)) {
> > -                       sock_gen_put(sk);
> > +               /* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> > +                * sock refcnt is decremented to prevent a request_sock leak.
> > +                */
> > +               if (!sk_fullsock(sk1))
> > +                       sock_gen_put(sk1);
> > +               if (!sk_fullsock(sk))
> >                         return NULL;
> > -               }
> >         }
> >
> >         return sk;
> > @@ -6239,13 +6243,17 @@ bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
> >  {
> >         struct sock *sk = bpf_skc_lookup(skb, tuple, len, proto, netns_id,
> >                                          flags);
> > +       struct sock *sk1 = sk;
> >
> >         if (sk) {
> >                 sk = sk_to_full_sk(sk);
> > -               if (!sk_fullsock(sk)) {
> > -                       sock_gen_put(sk);
> > +               /* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> > +                * sock refcnt is decremented to prevent a request_sock leak.
> > +                */
> > +               if (!sk_fullsock(sk1))
> > +                       sock_gen_put(sk1);
> > +               if (!sk_fullsock(sk))
> >                         return NULL;
> > -               }
> >         }
> >
> >         return sk;
> > --
> > 2.31.1
> >

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Daniel Borkmann]

On 6/10/22 2:17 AM, Martin KaFai Lau wrote:
> On Thu, Jun 09, 2022 at 10:29:15PM +0200, Daniel Borkmann wrote:
>> On 6/9/22 3:18 AM, Jon Maxwell wrote:
>>> A customer reported a request_socket leak in a Calico cloud environment. We
>>> found that a BPF program was doing a socket lookup with takes a refcnt on
>>> the socket and that it was finding the request_socket but returning the parent
>>> LISTEN socket via sk_to_full_sk() without decrementing the child request socket
>>> 1st, resulting in request_sock slab object leak. This patch retains the
> Great catch and debug indeed!
> 
>>> existing behaviour of returning full socks to the caller but it also decrements
>>> the child request_socket if one is present before doing so to prevent the leak.
>>>
>>> Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
>>> thanks to Antoine Tenart for the reproducer and patch input.
>>>
>>> Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
>>> Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
> Instead of the above commits, I think this dated back to
> 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
> 
>>> Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
>>> Co-developed-by: Antoine Tenart <atenart@kernel.org>
>>> Signed-off-by:: Antoine Tenart <atenart@kernel.org>
>>> Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
>>> ---
>>>    net/core/filter.c | 20 ++++++++++++++------
>>>    1 file changed, 14 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/net/core/filter.c b/net/core/filter.c
>>> index 2e32cee2c469..e3c04ae7381f 100644
>>> --- a/net/core/filter.c
>>> +++ b/net/core/filter.c
>>> @@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
>>>    {
>>>    	struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
>>>    					   ifindex, proto, netns_id, flags);
>>> +	struct sock *sk1 = sk;
>>>    	if (sk) {
>>>    		sk = sk_to_full_sk(sk);
>>> -		if (!sk_fullsock(sk)) {
>>> -			sock_gen_put(sk);
>>> +		/* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
>>> +		 * sock refcnt is decremented to prevent a request_sock leak.
>>> +		 */
>>> +		if (!sk_fullsock(sk1))
>>> +			sock_gen_put(sk1);
>>> +		if (!sk_fullsock(sk))
> In this case, sk1 == sk (timewait).  It is a bit worrying to pass
> sk to sk_fullsock(sk) after the above sock_gen_put().
> I think Daniel's 'if (sk2 != sk) { sock_gen_put(sk); }' check is better.
> 
>> [ +Martin/Joe/Lorenz ]
>>
>> I wonder, should we also add some asserts in here to ensure we don't get an unbalance for the
>> bpf_sk_release() case later on? Rough pseudocode could be something like below:
>>
>> static struct sock *
>> __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
>>                  struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id,
>>                  u64 flags)
>> {
>>          struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
>>                                             ifindex, proto, netns_id, flags);
>>          if (sk) {
>>                  struct sock *sk2 = sk_to_full_sk(sk);
>>
>>                  if (!sk_fullsock(sk2))
>>                          sk2 = NULL;
>>                  if (sk2 != sk) {
>>                          sock_gen_put(sk);
>>                          if (unlikely(sk2 && !sock_flag(sk2, SOCK_RCU_FREE))) {
> I don't think it matters if the helper-returned sk2 is refcounted or not (SOCK_RCU_FREE).
> The verifier has ensured the bpf_sk_lookup() and bpf_sk_release() are
> always balanced regardless of the type of sk2.
> 
> bpf_sk_release() will do the right thing to check the sk2 is refcounted or not
> before calling sock_gen_put().
> 
> The bug here is the helper forgot to call sock_gen_put(sk) while
> the verifier only tracks the sk2, so I think the 'if (unlikely...) { WARN_ONCE(...); }'
> can be saved.

I was mainly thinking given in sk_lookup() we have the check around `sk && !refcounted &&
!sock_flag(sk, SOCK_RCU_FREE)` to check for unreferenced non-SOCK_RCU_FREE socket, and
given sk_to_full_sk() can return inet_reqsk(sk)->rsk_listener we don't have a similar
assertion there. Given we don't bump any ref on the latter, it must be SOCK_RCU_FREE then
as otherwise latter call to bpf_sk_release() will unbalance sk2. @Jon: maybe lets just
manually verify that such sk2 has SOCK_RCU_FREE and state it in the commit message for
future reference then, either is fine with me. Thanks!

>>                                  WARN_ONCE(1, "Found non-RCU, unreferenced socket!");
>>                                  sk2 = NULL;
>>                          }
>>                  }
>>                  sk = sk2;
>>          }
>>          return sk;
>> }

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Daniel Borkmann]

On 6/10/22 2:45 AM, Jonathan Maxwell wrote:
> On Fri, Jun 10, 2022 at 10:36 AM Martin KaFai Lau <kafai@fb.com> wrote:
>>
>> On Thu, Jun 09, 2022 at 05:17:47PM -0700, Martin KaFai Lau wrote:
>>> On Thu, Jun 09, 2022 at 10:29:15PM +0200, Daniel Borkmann wrote:
>>>> On 6/9/22 3:18 AM, Jon Maxwell wrote:
>>>>> A customer reported a request_socket leak in a Calico cloud environment. We
>>>>> found that a BPF program was doing a socket lookup with takes a refcnt on
>>>>> the socket and that it was finding the request_socket but returning the parent
>>>>> LISTEN socket via sk_to_full_sk() without decrementing the child request socket
>>>>> 1st, resulting in request_sock slab object leak. This patch retains the
>>> Great catch and debug indeed!
>>>
>>>>> existing behaviour of returning full socks to the caller but it also decrements
>>>>> the child request_socket if one is present before doing so to prevent the leak.
>>>>>
>>>>> Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
>>>>> thanks to Antoine Tenart for the reproducer and patch input.
>>>>>
>>>>> Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
>>>>> Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
>>> Instead of the above commits, I think this dated back to
>>> 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
>>
>> Since this is more bpf specific, I think it could go to the bpf tree.
>> In v2, please cc bpf@vger.kernel.org and tag it with 'PATCH v2 bpf'.
> 
> Okay thanks will do.
> 
> Daniel, are you okay with omitting 'if (unlikely...) { WARN_ONCE(...); }'?
> 
> If so I'll stick to the rest of the logic of your suggestion and omit that
> check in v1.

Ok, works for me, see also my other reply that we should at least mention it in
the commit log.

Thanks!
Daniel

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Martin KaFai Lau]

On Fri, Jun 10, 2022 at 09:08:41AM +0200, Daniel Borkmann wrote:
> On 6/10/22 2:17 AM, Martin KaFai Lau wrote:
> > On Thu, Jun 09, 2022 at 10:29:15PM +0200, Daniel Borkmann wrote:
> > > On 6/9/22 3:18 AM, Jon Maxwell wrote:
> > > > A customer reported a request_socket leak in a Calico cloud environment. We
> > > > found that a BPF program was doing a socket lookup with takes a refcnt on
> > > > the socket and that it was finding the request_socket but returning the parent
> > > > LISTEN socket via sk_to_full_sk() without decrementing the child request socket
> > > > 1st, resulting in request_sock slab object leak. This patch retains the
> > Great catch and debug indeed!
> > 
> > > > existing behaviour of returning full socks to the caller but it also decrements
> > > > the child request_socket if one is present before doing so to prevent the leak.
> > > > 
> > > > Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
> > > > thanks to Antoine Tenart for the reproducer and patch input.
> > > > 
> > > > Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> > > > Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
> > Instead of the above commits, I think this dated back to
> > 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
> > 
> > > > Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
> > > > Co-developed-by: Antoine Tenart <atenart@kernel.org>
> > > > Signed-off-by:: Antoine Tenart <atenart@kernel.org>
> > > > Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
> > > > ---
> > > >    net/core/filter.c | 20 ++++++++++++++------
> > > >    1 file changed, 14 insertions(+), 6 deletions(-)
> > > > 
> > > > diff --git a/net/core/filter.c b/net/core/filter.c
> > > > index 2e32cee2c469..e3c04ae7381f 100644
> > > > --- a/net/core/filter.c
> > > > +++ b/net/core/filter.c
> > > > @@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
> > > >    {
> > > >    	struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
> > > >    					   ifindex, proto, netns_id, flags);
> > > > +	struct sock *sk1 = sk;
> > > >    	if (sk) {
> > > >    		sk = sk_to_full_sk(sk);
> > > > -		if (!sk_fullsock(sk)) {
> > > > -			sock_gen_put(sk);
> > > > +		/* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> > > > +		 * sock refcnt is decremented to prevent a request_sock leak.
> > > > +		 */
> > > > +		if (!sk_fullsock(sk1))
> > > > +			sock_gen_put(sk1);
> > > > +		if (!sk_fullsock(sk))
> > In this case, sk1 == sk (timewait).  It is a bit worrying to pass
> > sk to sk_fullsock(sk) after the above sock_gen_put().
> > I think Daniel's 'if (sk2 != sk) { sock_gen_put(sk); }' check is better.
> > 
> > > [ +Martin/Joe/Lorenz ]
> > > 
> > > I wonder, should we also add some asserts in here to ensure we don't get an unbalance for the
> > > bpf_sk_release() case later on? Rough pseudocode could be something like below:
> > > 
> > > static struct sock *
> > > __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
> > >                  struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id,
> > >                  u64 flags)
> > > {
> > >          struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
> > >                                             ifindex, proto, netns_id, flags);
> > >          if (sk) {
> > >                  struct sock *sk2 = sk_to_full_sk(sk);
> > > 
> > >                  if (!sk_fullsock(sk2))
> > >                          sk2 = NULL;
> > >                  if (sk2 != sk) {
> > >                          sock_gen_put(sk);
> > >                          if (unlikely(sk2 && !sock_flag(sk2, SOCK_RCU_FREE))) {
> > I don't think it matters if the helper-returned sk2 is refcounted or not (SOCK_RCU_FREE).
> > The verifier has ensured the bpf_sk_lookup() and bpf_sk_release() are
> > always balanced regardless of the type of sk2.
> > 
> > bpf_sk_release() will do the right thing to check the sk2 is refcounted or not
> > before calling sock_gen_put().
> > 
> > The bug here is the helper forgot to call sock_gen_put(sk) while
> > the verifier only tracks the sk2, so I think the 'if (unlikely...) { WARN_ONCE(...); }'
> > can be saved.
> 
> I was mainly thinking given in sk_lookup() we have the check around `sk && !refcounted &&
> !sock_flag(sk, SOCK_RCU_FREE)` to check for unreferenced non-SOCK_RCU_FREE socket, and
> given sk_to_full_sk() can return inet_reqsk(sk)->rsk_listener we don't have a similar
> assertion there. Given we don't bump any ref on the latter, it must be SOCK_RCU_FREE then
Ah. got it.  Thanks for the explanation.

Yep, agree.  It is useful to have this check here to ensure
no need to bump the sk2 refcnt.  A comment may be useful
here also, /* Ensure there is no need to bump sk2 refcnt */

Thanks!

Re: [PATCH net] net: bpf: fix request_sock leak in filter.c

[Jonathan Maxwell]

On Sat, Jun 11, 2022 at 3:58 AM Martin KaFai Lau <kafai@fb.com> wrote:
>
> On Fri, Jun 10, 2022 at 09:08:41AM +0200, Daniel Borkmann wrote:
> > On 6/10/22 2:17 AM, Martin KaFai Lau wrote:
> > > On Thu, Jun 09, 2022 at 10:29:15PM +0200, Daniel Borkmann wrote:
> > > > On 6/9/22 3:18 AM, Jon Maxwell wrote:
> > > > > A customer reported a request_socket leak in a Calico cloud environment. We
> > > > > found that a BPF program was doing a socket lookup with takes a refcnt on
> > > > > the socket and that it was finding the request_socket but returning the parent
> > > > > LISTEN socket via sk_to_full_sk() without decrementing the child request socket
> > > > > 1st, resulting in request_sock slab object leak. This patch retains the
> > > Great catch and debug indeed!
> > >
> > > > > existing behaviour of returning full socks to the caller but it also decrements
> > > > > the child request_socket if one is present before doing so to prevent the leak.
> > > > >
> > > > > Thanks to Curtis Taylor for all the help in diagnosing and testing this. And
> > > > > thanks to Antoine Tenart for the reproducer and patch input.
> > > > >
> > > > > Fixes: f7355a6c0497 bpf: ("Check sk_fullsock() before returning from bpf_sk_lookup()")
> > > > > Fixes: edbf8c01de5a bpf: ("add skc_lookup_tcp helper")
> > > Instead of the above commits, I think this dated back to
> > > 6acc9b432e67 ("bpf: Add helper to retrieve socket in BPF")
> > >
> > > > > Tested-by: Curtis Taylor <cutaylor-pub@yahoo.com>
> > > > > Co-developed-by: Antoine Tenart <atenart@kernel.org>
> > > > > Signed-off-by:: Antoine Tenart <atenart@kernel.org>
> > > > > Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com>
> > > > > ---
> > > > >    net/core/filter.c | 20 ++++++++++++++------
> > > > >    1 file changed, 14 insertions(+), 6 deletions(-)
> > > > >
> > > > > diff --git a/net/core/filter.c b/net/core/filter.c
> > > > > index 2e32cee2c469..e3c04ae7381f 100644
> > > > > --- a/net/core/filter.c
> > > > > +++ b/net/core/filter.c
> > > > > @@ -6202,13 +6202,17 @@ __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
> > > > >    {
> > > > >         struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
> > > > >                                            ifindex, proto, netns_id, flags);
> > > > > +       struct sock *sk1 = sk;
> > > > >         if (sk) {
> > > > >                 sk = sk_to_full_sk(sk);
> > > > > -               if (!sk_fullsock(sk)) {
> > > > > -                       sock_gen_put(sk);
> > > > > +               /* sk_to_full_sk() may return (sk)->rsk_listener, so make sure the original sk1
> > > > > +                * sock refcnt is decremented to prevent a request_sock leak.
> > > > > +                */
> > > > > +               if (!sk_fullsock(sk1))
> > > > > +                       sock_gen_put(sk1);
> > > > > +               if (!sk_fullsock(sk))
> > > In this case, sk1 == sk (timewait).  It is a bit worrying to pass
> > > sk to sk_fullsock(sk) after the above sock_gen_put().
> > > I think Daniel's 'if (sk2 != sk) { sock_gen_put(sk); }' check is better.
> > >
> > > > [ +Martin/Joe/Lorenz ]
> > > >
> > > > I wonder, should we also add some asserts in here to ensure we don't get an unbalance for the
> > > > bpf_sk_release() case later on? Rough pseudocode could be something like below:
> > > >
> > > > static struct sock *
> > > > __bpf_sk_lookup(struct sk_buff *skb, struct bpf_sock_tuple *tuple, u32 len,
> > > >                  struct net *caller_net, u32 ifindex, u8 proto, u64 netns_id,
> > > >                  u64 flags)
> > > > {
> > > >          struct sock *sk = __bpf_skc_lookup(skb, tuple, len, caller_net,
> > > >                                             ifindex, proto, netns_id, flags);
> > > >          if (sk) {
> > > >                  struct sock *sk2 = sk_to_full_sk(sk);
> > > >
> > > >                  if (!sk_fullsock(sk2))
> > > >                          sk2 = NULL;
> > > >                  if (sk2 != sk) {
> > > >                          sock_gen_put(sk);
> > > >                          if (unlikely(sk2 && !sock_flag(sk2, SOCK_RCU_FREE))) {
> > > I don't think it matters if the helper-returned sk2 is refcounted or not (SOCK_RCU_FREE).
> > > The verifier has ensured the bpf_sk_lookup() and bpf_sk_release() are
> > > always balanced regardless of the type of sk2.
> > >
> > > bpf_sk_release() will do the right thing to check the sk2 is refcounted or not
> > > before calling sock_gen_put().
> > >
> > > The bug here is the helper forgot to call sock_gen_put(sk) while
> > > the verifier only tracks the sk2, so I think the 'if (unlikely...) { WARN_ONCE(...); }'
> > > can be saved.
> >
> > I was mainly thinking given in sk_lookup() we have the check around `sk && !refcounted &&
> > !sock_flag(sk, SOCK_RCU_FREE)` to check for unreferenced non-SOCK_RCU_FREE socket, and
> > given sk_to_full_sk() can return inet_reqsk(sk)->rsk_listener we don't have a similar
> > assertion there. Given we don't bump any ref on the latter, it must be SOCK_RCU_FREE then
> Ah. got it.  Thanks for the explanation.
>
> Yep, agree.  It is useful to have this check here to ensure
> no need to bump the sk2 refcnt.  A comment may be useful
> here also, /* Ensure there is no need to bump sk2 refcnt */
>

I'll add that comment.

I'll add the SOCK_RCU_FREE check. We are currently testing the new patch
based on Daniels recommendation. When that is complete I'll resubmit the next
version of the patch including that. It'll probably be a few days.

Regards

Jon

> Thanks!
>