Subject: [PATCH netfilter] ipvs: Fix crash when ipv6 route unreach

Subject: [PATCH netfilter] ipvs: Fix crash when ipv6 route unreach

[hujunwei]

From: Junwei Hu <hujunwei4@huawei.com>

When Tcp send RST packet in ipvs, crash occurs with the following
stack trace:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
PID: 0 COMMAND: "swapper/2"
TASK: ffff9ec83889bf40  (1 of 4)  [THREAD_INFO: ffff9ec8388b0000]
CPU: 2  STATE: TASK_RUNNING (PANIC)
 [exception RIP: __ip_vs_get_out_rt_v6+1250]
RIP: ffffffffc0d566f2  RSP: ffff9ec83ed03c68  RFLAGS: 00010246
RAX: 0000000000000000  RBX: ffff9ec835e85000  RCX: 000000000005e1f9
RDX: 000000000005e1f8  RSI: 0000000000000200  RDI: ffff9ec83e801b00
RBP: ffff9ec83ed03cd8   R8: 000000000001bb40   R9: ffffffffc0d5673f
R10: ffff9ec83ed1bb40  R11: ffffe2d384d4fdc0  R12: ffff9ec7b7ad5900
R13: 0000000000000000  R14: 0000000000000007  R15: ffff9ec8353f7580
ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 [ffff9ec83ed03ce0] ip_vs_fnat_xmit_v6 at ffffffffc0d5b42c [ip_vs]
 [ffff9ec83ed03d70] tcp_send_rst_ipv6 at ffffffffc0d6542a [ip_vs]
 [ffff9ec83ed03df8] tcp_conn_expire_handler at ffffffffc0d65823 [ip_vs]
 [ffff9ec83ed03e20] ip_vs_conn_expire at ffffffffc0d42373 [ip_vs]
 [ffff9ec83ed03e70] call_timer_fn at ffffffffae0a6b58
 [ffff9ec83ed03ea8] run_timer_softirq at ffffffffae0a904d
 [ffff9ec83ed03f20] __do_softirq at ffffffffae09fa85
 [ffff9ec83ed03f90] call_softirq at ffffffffae739dac
 [ffff9ec83ed03fa8] do_softirq at ffffffffae02e62b
 [ffff9ec83ed03fc0] irq_exit at ffffffffae09fe25
 [ffff9ec83ed03fd8] smp_apic_timer_interrupt at ffffffffae73b158
 [ffff9ec83ed03ff0] apic_timer_interrupt at ffffffffae737872

TCP connection timeout and send a RST packet, the skb is alloc
by alloc_skb, the pointer skb->dev and skb_dst(skb) is NULL,
however, ipv6 route unreach at that time, so go into err_unreach.
In err_unreach, crash occurs when skb->dev and skb_dst(skb) is NULL.

The code is added by the following patch:
commit 326bf17ea5d4 ("ipvs: fix ipv6 route unreach panic")
because the ip6_link_failure function requires the skb->dev
in icmp6_send with that version.

This patch only fix the problem in specific scene, and icmp6_send in
current version is robust against null skb->dev by adding the
following patch.
commit 8d9336704521
("ipv6: make icmp6_send() robust against null skb->dev")

So I delete the code, make __ip_vs_get_out_rt_v6() robust, when
skb->dev and skb_dst(skb) is NULL.

Fixes: 326bf17ea5d4 ("ipvs: fix ipv6 route unreach panic")
Signed-off-by: Junwei Hu <hujunwei4@huawei.com>
Reported-by: Wenhao Zhang <zhangwenhao8@huawei.com>
---
 net/netfilter/ipvs/ip_vs_xmit.c | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
index 175349fcf91f..e2bb6c223396 100644
--- a/net/netfilter/ipvs/ip_vs_xmit.c
+++ b/net/netfilter/ipvs/ip_vs_xmit.c
@@ -561,13 +561,6 @@ __ip_vs_get_out_rt_v6(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb,
 	return -1;

 err_unreach:
-	/* The ip6_link_failure function requires the dev field to be set
-	 * in order to get the net (further for the sake of fwmark
-	 * reflection).
-	 */
-	if (!skb->dev)
-		skb->dev = skb_dst(skb)->dev;
-
 	dst_link_failure(skb);
 	return -1;
 }
-- 
2.21.GIT

Re: Subject: [PATCH netfilter] ipvs: Fix crash when ipv6 route unreach

[Julian Anastasov]

	Hello,

On Tue, 7 May 2019, hujunwei wrote:

> From: Junwei Hu <hujunwei4@huawei.com>
> 
> When Tcp send RST packet in ipvs, crash occurs with the following
> stack trace:
> 
> BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
> PID: 0 COMMAND: "swapper/2"
> TASK: ffff9ec83889bf40  (1 of 4)  [THREAD_INFO: ffff9ec8388b0000]
> CPU: 2  STATE: TASK_RUNNING (PANIC)
>  [exception RIP: __ip_vs_get_out_rt_v6+1250]
> RIP: ffffffffc0d566f2  RSP: ffff9ec83ed03c68  RFLAGS: 00010246
> RAX: 0000000000000000  RBX: ffff9ec835e85000  RCX: 000000000005e1f9
> RDX: 000000000005e1f8  RSI: 0000000000000200  RDI: ffff9ec83e801b00
> RBP: ffff9ec83ed03cd8   R8: 000000000001bb40   R9: ffffffffc0d5673f
> R10: ffff9ec83ed1bb40  R11: ffffe2d384d4fdc0  R12: ffff9ec7b7ad5900
> R13: 0000000000000000  R14: 0000000000000007  R15: ffff9ec8353f7580
> ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
>  [ffff9ec83ed03ce0] ip_vs_fnat_xmit_v6 at ffffffffc0d5b42c [ip_vs]
>  [ffff9ec83ed03d70] tcp_send_rst_ipv6 at ffffffffc0d6542a [ip_vs]
>  [ffff9ec83ed03df8] tcp_conn_expire_handler at ffffffffc0d65823 [ip_vs]
>  [ffff9ec83ed03e20] ip_vs_conn_expire at ffffffffc0d42373 [ip_vs]
>  [ffff9ec83ed03e70] call_timer_fn at ffffffffae0a6b58
>  [ffff9ec83ed03ea8] run_timer_softirq at ffffffffae0a904d
>  [ffff9ec83ed03f20] __do_softirq at ffffffffae09fa85
>  [ffff9ec83ed03f90] call_softirq at ffffffffae739dac
>  [ffff9ec83ed03fa8] do_softirq at ffffffffae02e62b
>  [ffff9ec83ed03fc0] irq_exit at ffffffffae09fe25
>  [ffff9ec83ed03fd8] smp_apic_timer_interrupt at ffffffffae73b158
>  [ffff9ec83ed03ff0] apic_timer_interrupt at ffffffffae737872
> 
> TCP connection timeout and send a RST packet, the skb is alloc
> by alloc_skb, the pointer skb->dev and skb_dst(skb) is NULL,
> however, ipv6 route unreach at that time, so go into err_unreach.
> In err_unreach, crash occurs when skb->dev and skb_dst(skb) is NULL.

	I guess, this is a modified IPVS module and the problem
can not occur in mainline kernel. ip_vs_in() and ip_vs_out() already 
have check for skb_dst(). May be you generate skb without attached
route, so skb_dst is NULL. Also, note that decrement_ttl() has similar
code.

> The code is added by the following patch:
> commit 326bf17ea5d4 ("ipvs: fix ipv6 route unreach panic")
> because the ip6_link_failure function requires the skb->dev
> in icmp6_send with that version.
> 
> This patch only fix the problem in specific scene, and icmp6_send in
> current version is robust against null skb->dev by adding the
> following patch.
> commit 8d9336704521
> ("ipv6: make icmp6_send() robust against null skb->dev")
> 
> So I delete the code, make __ip_vs_get_out_rt_v6() robust, when
> skb->dev and skb_dst(skb) is NULL.
> 
> Fixes: 326bf17ea5d4 ("ipvs: fix ipv6 route unreach panic")
> Signed-off-by: Junwei Hu <hujunwei4@huawei.com>
> Reported-by: Wenhao Zhang <zhangwenhao8@huawei.com>
> ---
>  net/netfilter/ipvs/ip_vs_xmit.c | 7 -------
>  1 file changed, 7 deletions(-)
> 
> diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
> index 175349fcf91f..e2bb6c223396 100644
> --- a/net/netfilter/ipvs/ip_vs_xmit.c
> +++ b/net/netfilter/ipvs/ip_vs_xmit.c
> @@ -561,13 +561,6 @@ __ip_vs_get_out_rt_v6(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb,
>  	return -1;
> 
>  err_unreach:
> -	/* The ip6_link_failure function requires the dev field to be set
> -	 * in order to get the net (further for the sake of fwmark
> -	 * reflection).
> -	 */
> -	if (!skb->dev)
> -		skb->dev = skb_dst(skb)->dev;
> -
>  	dst_link_failure(skb);
>  	return -1;
>  }
> -- 
> 2.21.GIT

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: Subject: [PATCH netfilter] ipvs: Fix crash when ipv6 route unreach

[hujunwei]

On 2019/5/7 15:45, Julian Anastasov wrote:
> 
> 	Hello,
> 
> On Tue, 7 May 2019, hujunwei wrote:
> 
>> From: Junwei Hu <hujunwei4@huawei.com>
>>
>> When Tcp send RST packet in ipvs, crash occurs with the following
>> stack trace:
>>
>> BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
>> PID: 0 COMMAND: "swapper/2"
>> TASK: ffff9ec83889bf40  (1 of 4)  [THREAD_INFO: ffff9ec8388b0000]
>> CPU: 2  STATE: TASK_RUNNING (PANIC)
>>  [exception RIP: __ip_vs_get_out_rt_v6+1250]
>> RIP: ffffffffc0d566f2  RSP: ffff9ec83ed03c68  RFLAGS: 00010246
>> RAX: 0000000000000000  RBX: ffff9ec835e85000  RCX: 000000000005e1f9
>> RDX: 000000000005e1f8  RSI: 0000000000000200  RDI: ffff9ec83e801b00
>> RBP: ffff9ec83ed03cd8   R8: 000000000001bb40   R9: ffffffffc0d5673f
>> R10: ffff9ec83ed1bb40  R11: ffffe2d384d4fdc0  R12: ffff9ec7b7ad5900
>> R13: 0000000000000000  R14: 0000000000000007  R15: ffff9ec8353f7580
>> ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
>>  [ffff9ec83ed03ce0] ip_vs_fnat_xmit_v6 at ffffffffc0d5b42c [ip_vs]
>>  [ffff9ec83ed03d70] tcp_send_rst_ipv6 at ffffffffc0d6542a [ip_vs]
>>  [ffff9ec83ed03df8] tcp_conn_expire_handler at ffffffffc0d65823 [ip_vs]
>>  [ffff9ec83ed03e20] ip_vs_conn_expire at ffffffffc0d42373 [ip_vs]
>>  [ffff9ec83ed03e70] call_timer_fn at ffffffffae0a6b58
>>  [ffff9ec83ed03ea8] run_timer_softirq at ffffffffae0a904d
>>  [ffff9ec83ed03f20] __do_softirq at ffffffffae09fa85
>>  [ffff9ec83ed03f90] call_softirq at ffffffffae739dac
>>  [ffff9ec83ed03fa8] do_softirq at ffffffffae02e62b
>>  [ffff9ec83ed03fc0] irq_exit at ffffffffae09fe25
>>  [ffff9ec83ed03fd8] smp_apic_timer_interrupt at ffffffffae73b158
>>  [ffff9ec83ed03ff0] apic_timer_interrupt at ffffffffae737872
>>
>> TCP connection timeout and send a RST packet, the skb is alloc
>> by alloc_skb, the pointer skb->dev and skb_dst(skb) is NULL,
>> however, ipv6 route unreach at that time, so go into err_unreach.
>> In err_unreach, crash occurs when skb->dev and skb_dst(skb) is NULL.
> 
> 	I guess, this is a modified IPVS module and the problem
> can not occur in mainline kernel. ip_vs_in() and ip_vs_out() already 
> have check for skb_dst(). May be you generate skb without attached
> route, so skb_dst is NULL. Also, note that decrement_ttl() has similar
> code.
> 

Hi Julian, thank you for replay.
We are developing based on IPVS module. That code seems less compatible,
i think it is meaningful to make the code robust.
All above just my advice.

Regards,
Junwei