kernel panic TPROXY , vanilla 4.7.1

kernel panic TPROXY , vanilla 4.7.1

[Denys Fedoryshchenko]

Hi!

Tried to run squid on latest kernel, and hit a panic
Sometimes it just shows warning in dmesg (but doesnt work properly)
[   75.701666] IPv4: Attempt to release TCP socket in state 10 
ffff88102d430780
[   83.866974] squid (2700) used greatest stack depth: 12912 bytes left
[   87.506644] IPv4: Attempt to release TCP socket in state 10 
ffff880078a48780
[  114.704295] IPv4: Attempt to release TCP socket in state 10 
ffff881029f8ad00

I cannot catch yet oops/panic message, netconsole not working.

After triggering warning message 3 times, i am unable to run squid 
anymore (without reboot), and in netstat it doesnt show port running.

firewall is:
*mangle
-A PREROUTING -p tcp -m socket -j DIVERT
-A PREROUTING -p tcp -m tcp --dport 80 -i eno1 -j TPROXY --on-port 3129 
--on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
-A DIVERT -j MARK --set-xmark 0x1/0xffffffff
-A DIVERT -j ACCEPT

routing
ip rule add fwmark 1 lookup 100
ip route add local default dev eno1 table 100


squid config is default with tproxy option
http_port 3129 tproxy

Re: kernel panic TPROXY , vanilla 4.7.1

[Eric Dumazet]

On Wed, 2016-08-17 at 17:31 +0300, Denys Fedoryshchenko wrote:
> Hi!
> 
> Tried to run squid on latest kernel, and hit a panic
> Sometimes it just shows warning in dmesg (but doesnt work properly)
> [   75.701666] IPv4: Attempt to release TCP socket in state 10 
> ffff88102d430780
> [   83.866974] squid (2700) used greatest stack depth: 12912 bytes left
> [   87.506644] IPv4: Attempt to release TCP socket in state 10 
> ffff880078a48780
> [  114.704295] IPv4: Attempt to release TCP socket in state 10 
> ffff881029f8ad00
> 
> I cannot catch yet oops/panic message, netconsole not working.
> 
> After triggering warning message 3 times, i am unable to run squid 
> anymore (without reboot), and in netstat it doesnt show port running.
> 
> firewall is:
> *mangle
> -A PREROUTING -p tcp -m socket -j DIVERT
> -A PREROUTING -p tcp -m tcp --dport 80 -i eno1 -j TPROXY --on-port 3129 
> --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
> -A DIVERT -j MARK --set-xmark 0x1/0xffffffff
> -A DIVERT -j ACCEPT
> 
> routing
> ip rule add fwmark 1 lookup 100
> ip route add local default dev eno1 table 100
> 
> 
> squid config is default with tproxy option
> http_port 3129 tproxy
> 

Hmppff... sorry for this, I will send a fix.

Thanks for the report !

Re: kernel panic TPROXY , vanilla 4.7.1

[Eric Dumazet]

On Wed, 2016-08-17 at 08:42 -0700, Eric Dumazet wrote:
> On Wed, 2016-08-17 at 17:31 +0300, Denys Fedoryshchenko wrote:
> > Hi!
> > 
> > Tried to run squid on latest kernel, and hit a panic
> > Sometimes it just shows warning in dmesg (but doesnt work properly)
> > [   75.701666] IPv4: Attempt to release TCP socket in state 10 
> > ffff88102d430780
> > [   83.866974] squid (2700) used greatest stack depth: 12912 bytes left
> > [   87.506644] IPv4: Attempt to release TCP socket in state 10 
> > ffff880078a48780
> > [  114.704295] IPv4: Attempt to release TCP socket in state 10 
> > ffff881029f8ad00
> > 
> > I cannot catch yet oops/panic message, netconsole not working.
> > 
> > After triggering warning message 3 times, i am unable to run squid 
> > anymore (without reboot), and in netstat it doesnt show port running.
> > 
> > firewall is:
> > *mangle
> > -A PREROUTING -p tcp -m socket -j DIVERT
> > -A PREROUTING -p tcp -m tcp --dport 80 -i eno1 -j TPROXY --on-port 3129 
> > --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
> > -A DIVERT -j MARK --set-xmark 0x1/0xffffffff
> > -A DIVERT -j ACCEPT
> > 
> > routing
> > ip rule add fwmark 1 lookup 100
> > ip route add local default dev eno1 table 100
> > 
> > 
> > squid config is default with tproxy option
> > http_port 3129 tproxy
> > 
> 
> Hmppff... sorry for this, I will send a fix.
> 
> Thanks for the report !
> 


Could you try the following ?

Thanks !

 net/netfilter/xt_TPROXY.c |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/net/netfilter/xt_TPROXY.c b/net/netfilter/xt_TPROXY.c
index 7f4414d26a66..663c4c3c9072 100644
--- a/net/netfilter/xt_TPROXY.c
+++ b/net/netfilter/xt_TPROXY.c
@@ -127,6 +127,8 @@ nf_tproxy_get_sock_v4(struct net *net, struct sk_buff *skb, void *hp,
 						    daddr, dport,
 						    in->ifindex);
 
+			if (sk && !atomic_inc_not_zero(&sk->sk_refcnt))
+				sk = NULL;
 			/* NOTE: we return listeners even if bound to
 			 * 0.0.0.0, those are filtered out in
 			 * xt_socket, since xt_TPROXY needs 0 bound
@@ -195,6 +197,8 @@ nf_tproxy_get_sock_v6(struct net *net, struct sk_buff *skb, int thoff, void *hp,
 						   daddr, ntohs(dport),
 						   in->ifindex);
 
+			if (sk && !atomic_inc_not_zero(&sk->sk_refcnt))
+				sk = NULL;
 			/* NOTE: we return listeners even if bound to
 			 * 0.0.0.0, those are filtered out in
 			 * xt_socket, since xt_TPROXY needs 0 bound

Re: kernel panic TPROXY , vanilla 4.7.1

[Denys Fedoryshchenko]

On 2016-08-17 19:04, Eric Dumazet wrote:
> On Wed, 2016-08-17 at 08:42 -0700, Eric Dumazet wrote:
>> On Wed, 2016-08-17 at 17:31 +0300, Denys Fedoryshchenko wrote:
>> > Hi!
>> >
>> > Tried to run squid on latest kernel, and hit a panic
>> > Sometimes it just shows warning in dmesg (but doesnt work properly)
>> > [   75.701666] IPv4: Attempt to release TCP socket in state 10
>> > ffff88102d430780
>> > [   83.866974] squid (2700) used greatest stack depth: 12912 bytes left
>> > [   87.506644] IPv4: Attempt to release TCP socket in state 10
>> > ffff880078a48780
>> > [  114.704295] IPv4: Attempt to release TCP socket in state 10
>> > ffff881029f8ad00
>> >
>> > I cannot catch yet oops/panic message, netconsole not working.
>> >
>> > After triggering warning message 3 times, i am unable to run squid
>> > anymore (without reboot), and in netstat it doesnt show port running.
>> >
>> > firewall is:
>> > *mangle
>> > -A PREROUTING -p tcp -m socket -j DIVERT
>> > -A PREROUTING -p tcp -m tcp --dport 80 -i eno1 -j TPROXY --on-port 3129
>> > --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
>> > -A DIVERT -j MARK --set-xmark 0x1/0xffffffff
>> > -A DIVERT -j ACCEPT
>> >
>> > routing
>> > ip rule add fwmark 1 lookup 100
>> > ip route add local default dev eno1 table 100
>> >
>> >
>> > squid config is default with tproxy option
>> > http_port 3129 tproxy
>> >
>> 
>> Hmppff... sorry for this, I will send a fix.
>> 
>> Thanks for the report !
>> 
> 
> 
> Could you try the following ?
> 
> Thanks !
> 
>  net/netfilter/xt_TPROXY.c |    4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/net/netfilter/xt_TPROXY.c b/net/netfilter/xt_TPROXY.c
> index 7f4414d26a66..663c4c3c9072 100644
> --- a/net/netfilter/xt_TPROXY.c
> +++ b/net/netfilter/xt_TPROXY.c
> @@ -127,6 +127,8 @@ nf_tproxy_get_sock_v4(struct net *net, struct
> sk_buff *skb, void *hp,
>  						    daddr, dport,
>  						    in->ifindex);
> 
> +			if (sk && !atomic_inc_not_zero(&sk->sk_refcnt))
> +				sk = NULL;
>  			/* NOTE: we return listeners even if bound to
>  			 * 0.0.0.0, those are filtered out in
>  			 * xt_socket, since xt_TPROXY needs 0 bound
> @@ -195,6 +197,8 @@ nf_tproxy_get_sock_v6(struct net *net, struct
> sk_buff *skb, int thoff, void *hp,
>  						   daddr, ntohs(dport),
>  						   in->ifindex);
> 
> +			if (sk && !atomic_inc_not_zero(&sk->sk_refcnt))
> +				sk = NULL;
>  			/* NOTE: we return listeners even if bound to
>  			 * 0.0.0.0, those are filtered out in
>  			 * xt_socket, since xt_TPROXY needs 0 bound
Yes, everything fine after patch!
Thanks a lot

Re: kernel panic TPROXY , vanilla 4.7.1

[Eric Dumazet]

On Wed, 2016-08-17 at 19:44 +0300, Denys Fedoryshchenko wrote:

> Yes, everything fine after patch!
> Thanks a lot

Perfect, thanks for testing, I am sending the official patch.

[PATCH net] netfilter: tproxy: properly refcount tcp listeners

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

inet_lookup_listener() and inet6_lookup_listener() no longer
take a reference on the found listener.

This minimal patch adds back the refcounting, but we might do
this differently in net-next later.

Fixes: 3b24d854cb35 ("tcp/dccp: do not touch listener sk_refcnt under synflood")
Reported-and-tested-by: Denys Fedoryshchenko <nuclearcat@nuclearcat.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
Note: bug added in 4.7, stable candidate.

 net/netfilter/xt_TPROXY.c |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/net/netfilter/xt_TPROXY.c b/net/netfilter/xt_TPROXY.c
index 7f4414d26a66..663c4c3c9072 100644
--- a/net/netfilter/xt_TPROXY.c
+++ b/net/netfilter/xt_TPROXY.c
@@ -127,6 +127,8 @@ nf_tproxy_get_sock_v4(struct net *net, struct sk_buff *skb, void *hp,
 						    daddr, dport,
 						    in->ifindex);
 
+			if (sk && !atomic_inc_not_zero(&sk->sk_refcnt))
+				sk = NULL;
 			/* NOTE: we return listeners even if bound to
 			 * 0.0.0.0, those are filtered out in
 			 * xt_socket, since xt_TPROXY needs 0 bound
@@ -195,6 +197,8 @@ nf_tproxy_get_sock_v6(struct net *net, struct sk_buff *skb, int thoff, void *hp,
 						   daddr, ntohs(dport),
 						   in->ifindex);
 
+			if (sk && !atomic_inc_not_zero(&sk->sk_refcnt))
+				sk = NULL;
 			/* NOTE: we return listeners even if bound to
 			 * 0.0.0.0, those are filtered out in
 			 * xt_socket, since xt_TPROXY needs 0 bound

Re: [PATCH net] netfilter: tproxy: properly refcount tcp listeners

[Pablo Neira Ayuso]

On Wed, Aug 17, 2016 at 09:56:46AM -0700, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> inet_lookup_listener() and inet6_lookup_listener() no longer
> take a reference on the found listener.
> 
> This minimal patch adds back the refcounting, but we might do
> this differently in net-next later.

Applied, thanks Eric!