Re: flow cache removed = xfrm doesnt work

Re: flow cache removed = xfrm doesnt work

[Florian Westphal]

Tomas Charvat <tc@excello.cz> wrote:

[ CC stable, Steffen ]

> Hi Florian and David, I'm running several servers that use XFRM ipsec.
> It do work well on all kernels bellow 4.14.0.
>
> It doesnt work on 4.14.0-2. There is no any error in dmesg or in
> userspace when I do configure policies.
> 
> Since there is not much info about XFRM in dmesg I have no clue, where
> to start when I want to debug this issue.

David, please consider picking up
94802151894d482e82c324edf2c658f8e6b96508
("Revert "xfrm: Fix stack-out-of-bounds read in xfrm_state_find.")

for the 4.14.y stable queue.

I think its a pretty safe bet that this fixes the problem, it broke
transport mode wildcard policy lookup.

> I have seen that you have removed flow-cache that we have fixed 2 time.
> Do you have clue where to start with debug of this issue ?

If the revert doesn't help, please do a bug report to
netdev@vger.kernel.org  and provide /proc/net/xfrm_stat content
and the list of policies/SAs.

Re: flow cache removed = xfrm doesnt work

[David Miller]

From: Florian Westphal <fw@strlen.de>
Date: Fri, 24 Nov 2017 20:32:12 +0100

> Tomas Charvat <tc@excello.cz> wrote:
> 
> [ CC stable, Steffen ]
> 
>> Hi Florian and David, I'm running several servers that use XFRM ipsec.
>> It do work well on all kernels bellow 4.14.0.
>>
>> It doesnt work on 4.14.0-2. There is no any error in dmesg or in
>> userspace when I do configure policies.
>> 
>> Since there is not much info about XFRM in dmesg I have no clue, where
>> to start when I want to debug this issue.
> 
> David, please consider picking up
> 94802151894d482e82c324edf2c658f8e6b96508
> ("Revert "xfrm: Fix stack-out-of-bounds read in xfrm_state_find.")
> 
> for the 4.14.y stable queue.
> 
> I think its a pretty safe bet that this fixes the problem, it broke
> transport mode wildcard policy lookup.

Ok, once we have confirmation that this fixes it I also need to pair
it up with Steffen's alternative fix for the bug that commit was
trying to fix.

Re: flow cache removed = xfrm doesnt work

[Tomas Charvat]

[-- Attachment #1: Type: text/plain, Size: 1232 bytes --]

Thanks I will give it a try. Any change that patch will make it into
4.14.3?

br
Tomas

On Fri, 2017-11-24 at 20:32 +0100, Florian Westphal wrote:
> Tomas Charvat <tc@excello.cz> wrote:
> 
> [ CC stable, Steffen ]
> 
> > Hi Florian and David, I'm running several servers that use XFRM
> > ipsec.
> > It do work well on all kernels bellow 4.14.0.
> > 
> > It doesnt work on 4.14.0-2. There is no any error in dmesg or in
> > userspace when I do configure policies.
> > 
> > Since there is not much info about XFRM in dmesg I have no clue,
> > where
> > to start when I want to debug this issue.
> 
> David, please consider picking up
> 94802151894d482e82c324edf2c658f8e6b96508
> ("Revert "xfrm: Fix stack-out-of-bounds read in xfrm_state_find.")
> 
> for the 4.14.y stable queue.
> 
> I think its a pretty safe bet that this fixes the problem, it broke
> transport mode wildcard policy lookup.
> 
> > I have seen that you have removed flow-cache that we have fixed 2
> > time.
> > Do you have clue where to start with debug of this issue ?
> 
> If the revert doesn't help, please do a bug report to
> netdev@vger.kernel.org  and provide /proc/net/xfrm_stat content
> and the list of policies/SAs.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 4919 bytes --]

Re: flow cache removed = xfrm doesnt work

[Steffen Klassert]

On Sat, Nov 25, 2017 at 04:50:31AM +0900, David Miller wrote:
> From: Florian Westphal <fw@strlen.de>
> Date: Fri, 24 Nov 2017 20:32:12 +0100
> 
> > Tomas Charvat <tc@excello.cz> wrote:
> > 
> > [ CC stable, Steffen ]
> > 
> >> Hi Florian and David, I'm running several servers that use XFRM ipsec.
> >> It do work well on all kernels bellow 4.14.0.
> >>
> >> It doesnt work on 4.14.0-2. There is no any error in dmesg or in
> >> userspace when I do configure policies.
> >> 
> >> Since there is not much info about XFRM in dmesg I have no clue, where
> >> to start when I want to debug this issue.
> > 
> > David, please consider picking up
> > 94802151894d482e82c324edf2c658f8e6b96508
> > ("Revert "xfrm: Fix stack-out-of-bounds read in xfrm_state_find.")
> > 
> > for the 4.14.y stable queue.
> > 
> > I think its a pretty safe bet that this fixes the problem, it broke
> > transport mode wildcard policy lookup.
> 
> Ok, once we have confirmation that this fixes it I also need to pair
> it up with Steffen's alternative fix for the bug that commit was
> trying to fix.

We need this revert in the 4.14.y stable tree anyway as it broke
transport mode IPsec.

I thought quite a lot about the original problem that I tried
to fix. It is a rather subtile thing, like almost all bugs
reported from syzcaller I have seen.

In between I think our template validation is not strict enough.
It is possible to configure policies with transport mode template
where the selector address family does not match the templates
address family. The address family can not change on a transport
mode transformation, so this configuration does not make much
sense but lead to problems because we use the assumption that
the address family can not change on thransport mode later on.

Unfortunately the reproducer provided by syzcaller does not
trigger anything on my test setup, so I don't even know if
this fixes this exact problem.

Florian, could you please give the patch blelow a try?


Subject: [PATCH] xfrm: Fix stack-out-of-bounds with misconfigured transport
 mode policies.

On policies with a transport mode template, we pass the addresses
from the flowi to xfrm_state_find(), assuming that the IP addresses
(and address family) don't change during transformation.

Unfortunately our policy template validation is not strict enough.
It is possible to configure policies with transport mode template
where the address family of the template does not match the selectors
address family. This lead to stack-out-of-bound reads because
we compare arddesses of the wrong family. Fix this by refusing
such a configuration, address family can not change on transport
mode.

We use the assumption that, on transport mode, the first templates
address family must match the address family of the policy selector.
Subsequent transport mode templates must match the address family of
the previous template.

Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
---
 net/xfrm/xfrm_user.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c
index 983b0233767b..57ad016ae675 100644
--- a/net/xfrm/xfrm_user.c
+++ b/net/xfrm/xfrm_user.c
@@ -1419,11 +1419,14 @@ static void copy_templates(struct xfrm_policy *xp, struct xfrm_user_tmpl *ut,
 
 static int validate_tmpl(int nr, struct xfrm_user_tmpl *ut, u16 family)
 {
+	u16 prev_family;
 	int i;
 
 	if (nr > XFRM_MAX_DEPTH)
 		return -EINVAL;
 
+	prev_family = family;
+
 	for (i = 0; i < nr; i++) {
 		/* We never validated the ut->family value, so many
 		 * applications simply leave it at zero.  The check was
@@ -1435,6 +1438,12 @@ static int validate_tmpl(int nr, struct xfrm_user_tmpl *ut, u16 family)
 		if (!ut[i].family)
 			ut[i].family = family;
 
+		if ((ut[i].mode == XFRM_MODE_TRANSPORT) &&
+		    (ut[i].family != prev_family))
+			return -EINVAL;
+
+		prev_family = ut[i].family;
+
 		switch (ut[i].family) {
 		case AF_INET:
 			break;
-- 
2.14.1

Re: flow cache removed = xfrm doesnt work

[Tomas Charvat]

[-- Attachment #1: Type: text/plain, Size: 9294 bytes --]

Ok under heavy load  patch
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/?id=94802151894d482e82c324edf2c658f8e6b96508

cause following error once in few minutes, it doesn't happen instantly

Nov 27 09:00:06 [kernel] [ 5530.333416] BUG: unable to handle kernel
NULL pointer dereference at 0000000000000018
Nov 27 09:00:06 [kernel] [ 5530.333669] IP: xfrm_output_resume+0x211/0x460
Nov 27 09:00:06 [kernel] [ 5530.333817] PGD 0 P4D 0
Nov 27 09:00:06 [kernel] [ 5530.333959] Oops: 0000 [#3] SMP
Nov 27 09:00:06 [kernel] [ 5530.334104] CPU: 0 PID: 10893 Comm: stunnel
Tainted: G      D         4.14.2-gentoo #2
Nov 27 09:00:06 [kernel] [ 5530.334343] Hardware name: Supermicro
H8DGU/H8DGU, BIOS 3.5c       03/18/2016
Nov 27 09:00:06 [kernel] [ 5530.334498] task: ffff918aa11ce580
task.stack: ffffad9ca678c000
Nov 27 09:00:06 [kernel] [ 5530.334667] RIP:
0010:xfrm_output_resume+0x211/0x460
Nov 27 09:00:06 [kernel] [ 5530.334815] RSP: 0018:ffffad9ca678f8d8
EFLAGS: 00010246
Nov 27 09:00:06 [kernel] [ 5530.334966] RAX: 0000000000000000 RBX:
ffff918a786e32e0 RCX: 0000000000000000
Nov 27 09:00:06 [kernel] [ 5530.335122] RDX: 0000000000000020 RSI:
0000000000000000 RDI: 0000000000000000
Nov 27 09:00:06 [kernel] [ 5530.335279] RBP: ffffad9ca678f938 R08:
ffffad9ca678f750 R09: ffff918acd9ccc50
Nov 27 09:00:06 [kernel] [ 5530.335436] R10: ffff918a711018d4 R11:
0000000000000000 R12: ffffffff8ab10200
Nov 27 09:00:06 [kernel] [ 5530.335591] R13: ffff918acd188c00 R14:
ffff918acd188c3c R15: ffffad9ca678fb10
Nov 27 09:00:06 [kernel] [ 5530.335747] FS:  00007e2611a0b700(0000)
GS:ffff918acfc00000(0000) knlGS:0000000000000000
Nov 27 09:00:06 [kernel] [ 5530.336009] CS:  0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Nov 27 09:00:06 [kernel] [ 5530.336160] CR2: 0000000000000018 CR3:
000000080b90e000 CR4: 00000000000406f0
Nov 27 09:00:06 [kernel] [ 5530.336315] Call Trace:
Nov 27 09:00:06 [kernel] [ 5530.336461]  ? skb_checksum+0x3f/0x60
Nov 27 09:00:06 [kernel] [ 5530.336607]  ? reqsk_fastopen_remove+0x160/0x160
Nov 27 09:00:06 [kernel] [ 5530.336754]  ? skb_panic+0x70/0x70
Nov 27 09:00:06 [kernel] [ 5530.336898]  xfrm_output+0x91/0x210
Nov 27 09:00:06 [kernel] [ 5530.337070]  ? ipv6_confirm+0xcb/0x140
Nov 27 09:00:06 [kernel] [ 5530.337215]  xfrm6_output_finish+0x38/0x40
Nov 27 09:00:06 [kernel] [ 5530.337361]  __xfrm6_output+0x57/0x1e0
Nov 27 09:00:06 [kernel] [ 5530.337507]  xfrm6_output+0x9e/0x110
Nov 27 09:00:06 [kernel] [ 5530.337651]  ? xfrm6_local_rxpmtu+0x90/0x90
Nov 27 09:00:06 [kernel] [ 5530.337798]  ip6_xmit+0x29a/0x560
Nov 27 09:00:06 [kernel] [ 5530.337943]  ?
__ip6_append_data.isra.42+0xc10/0xc10
Nov 27 09:00:06 [kernel] [ 5530.338092]  inet6_csk_xmit+0xaa/0x100
Nov 27 09:00:06 [kernel] [ 5530.338263]  tcp_transmit_skb+0x57f/0xa10
Nov 27 09:00:06 [kernel] [ 5530.338411]  tcp_write_xmit+0x1cd/0xf90
Nov 27 09:00:06 [kernel] [ 5530.338557]  __tcp_push_pending_frames+0x3f/0xb0
Nov 27 09:00:06 [kernel] [ 5530.338704]  tcp_push+0xf9/0x120
Nov 27 09:00:06 [kernel] [ 5530.338848]  tcp_sendmsg_locked+0x66f/0xe30
Nov 27 09:00:06 [kernel] [ 5530.338995]  tcp_sendmsg+0x3a/0x60
Nov 27 09:00:06 [kernel] [ 5530.339140]  inet_sendmsg+0x3e/0xc0
Nov 27 09:00:06 [kernel] [ 5530.339285]  sock_sendmsg+0x48/0x60
Nov 27 09:00:06 [kernel] [ 5530.339429]  sock_write_iter+0x8f/0x100
Nov 27 09:00:06 [kernel] [ 5530.339602]  new_sync_write+0x18b/0x1d0
Nov 27 09:00:06 [kernel] [ 5530.339747]  __vfs_write+0x37/0x50
Nov 27 09:00:06 [kernel] [ 5530.339890]  vfs_write+0xc7/0x1c0
Nov 27 09:00:06 [kernel] [ 5530.340034]  SyS_write+0x5f/0xd0
Nov 27 09:00:06 [kernel] [ 5530.340178]  entry_SYSCALL_64_fastpath+0x13/0x94
Nov 27 09:00:06 [kernel] [ 5530.340327] RIP: 0033:0x7e2610b6b36d
Nov 27 09:00:06 [kernel] [ 5530.340480] RSP: 002b:00007e2611a0ad40
EFLAGS: 00000293 ORIG_RAX: 0000000000000001
Nov 27 09:00:06 [kernel] [ 5530.340741] RAX: ffffffffffffffda RBX:
00007e260000efd0 RCX: 00007e2610b6b36d
Nov 27 09:00:06 [kernel] [ 5530.340896] RDX: 0000000000000026 RSI:
00007e2611a46a30 RDI: 0000000000000013
Nov 27 09:00:06 [kernel] [ 5530.341051] RBP: 0000000000000003 R08:
0000000000000000 R09: 0000000000000000
Nov 27 09:00:06 [kernel] [ 5530.341207] R10: 0000000000000000 R11:
0000000000000293 R12: 0000000000000003
Nov 27 09:00:06 [kernel] [ 5530.341361] R13: 000055e303aaadf8 R14:
00007e2611a0accc R15: 00007e2611a42060
Nov 27 09:00:06 [kernel] [ 5530.341515] Code: f6 0f 8f b9 fe ff ff 31 f6
85 d2 0f 8f af fe ff ff e9 bb fe ff ff 85 f6 89 f0 0f 85 d6 fe ff ff 48
8b 7b 58 48 89 f8 48 83 e0 fe <4c> 8b 70 18 4d 85 f6 0f 84 a3 01 00 00
41 8b 86 80 00 00 00 85
Nov 27 09:00:06 [kernel] [ 5530.341929] RIP:
xfrm_output_resume+0x211/0x460 RSP: ffffad9ca678f8d8
Nov 27 09:00:06 [kernel] [ 5530.342083] CR2: 0000000000000018
Nov 27 09:00:06 [kernel] [ 5530.342532] ---[ end trace a5ae59251632552d ]---


Tomas Charvat
EXCELLO | Virusfree
w: www.virusfree.cz
e: tc@excello.cz

On 11/27/2017 10:02 AM, Steffen Klassert wrote:
> On Sat, Nov 25, 2017 at 04:50:31AM +0900, David Miller wrote:
>> From: Florian Westphal <fw@strlen.de>
>> Date: Fri, 24 Nov 2017 20:32:12 +0100
>>
>>> Tomas Charvat <tc@excello.cz> wrote:
>>>
>>> [ CC stable, Steffen ]
>>>
>>>> Hi Florian and David, I'm running several servers that use XFRM ipsec.
>>>> It do work well on all kernels bellow 4.14.0.
>>>>
>>>> It doesnt work on 4.14.0-2. There is no any error in dmesg or in
>>>> userspace when I do configure policies.
>>>>
>>>> Since there is not much info about XFRM in dmesg I have no clue, where
>>>> to start when I want to debug this issue.
>>> David, please consider picking up
>>> 94802151894d482e82c324edf2c658f8e6b96508
>>> ("Revert "xfrm: Fix stack-out-of-bounds read in xfrm_state_find.")
>>>
>>> for the 4.14.y stable queue.
>>>
>>> I think its a pretty safe bet that this fixes the problem, it broke
>>> transport mode wildcard policy lookup.
>> Ok, once we have confirmation that this fixes it I also need to pair
>> it up with Steffen's alternative fix for the bug that commit was
>> trying to fix.
> We need this revert in the 4.14.y stable tree anyway as it broke
> transport mode IPsec.
>
> I thought quite a lot about the original problem that I tried
> to fix. It is a rather subtile thing, like almost all bugs
> reported from syzcaller I have seen.
>
> In between I think our template validation is not strict enough.
> It is possible to configure policies with transport mode template
> where the selector address family does not match the templates
> address family. The address family can not change on a transport
> mode transformation, so this configuration does not make much
> sense but lead to problems because we use the assumption that
> the address family can not change on thransport mode later on.
>
> Unfortunately the reproducer provided by syzcaller does not
> trigger anything on my test setup, so I don't even know if
> this fixes this exact problem.
>
> Florian, could you please give the patch blelow a try?
>
>
> Subject: [PATCH] xfrm: Fix stack-out-of-bounds with misconfigured transport
>  mode policies.
>
> On policies with a transport mode template, we pass the addresses
> from the flowi to xfrm_state_find(), assuming that the IP addresses
> (and address family) don't change during transformation.
>
> Unfortunately our policy template validation is not strict enough.
> It is possible to configure policies with transport mode template
> where the address family of the template does not match the selectors
> address family. This lead to stack-out-of-bound reads because
> we compare arddesses of the wrong family. Fix this by refusing
> such a configuration, address family can not change on transport
> mode.
>
> We use the assumption that, on transport mode, the first templates
> address family must match the address family of the policy selector.
> Subsequent transport mode templates must match the address family of
> the previous template.
>
> Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
> ---
>  net/xfrm/xfrm_user.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c
> index 983b0233767b..57ad016ae675 100644
> --- a/net/xfrm/xfrm_user.c
> +++ b/net/xfrm/xfrm_user.c
> @@ -1419,11 +1419,14 @@ static void copy_templates(struct xfrm_policy *xp, struct xfrm_user_tmpl *ut,
>  
>  static int validate_tmpl(int nr, struct xfrm_user_tmpl *ut, u16 family)
>  {
> +	u16 prev_family;
>  	int i;
>  
>  	if (nr > XFRM_MAX_DEPTH)
>  		return -EINVAL;
>  
> +	prev_family = family;
> +
>  	for (i = 0; i < nr; i++) {
>  		/* We never validated the ut->family value, so many
>  		 * applications simply leave it at zero.  The check was
> @@ -1435,6 +1438,12 @@ static int validate_tmpl(int nr, struct xfrm_user_tmpl *ut, u16 family)
>  		if (!ut[i].family)
>  			ut[i].family = family;
>  
> +		if ((ut[i].mode == XFRM_MODE_TRANSPORT) &&
> +		    (ut[i].family != prev_family))
> +			return -EINVAL;
> +
> +		prev_family = ut[i].family;
> +
>  		switch (ut[i].family) {
>  		case AF_INET:
>  			break;



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3695 bytes --]

Re: flow cache removed = xfrm doesnt work

[Florian Westphal]

Steffen Klassert <steffen.klassert@secunet.com> wrote:
> On Sat, Nov 25, 2017 at 04:50:31AM +0900, David Miller wrote:
> > From: Florian Westphal <fw@strlen.de>
> > Date: Fri, 24 Nov 2017 20:32:12 +0100
> > 
> > > Tomas Charvat <tc@excello.cz> wrote:
> > > 
> > > [ CC stable, Steffen ]
> > > 
> > >> Hi Florian and David, I'm running several servers that use XFRM ipsec.
> > >> It do work well on all kernels bellow 4.14.0.
> > >>
> > >> It doesnt work on 4.14.0-2. There is no any error in dmesg or in
> > >> userspace when I do configure policies.
> > >> 
> > >> Since there is not much info about XFRM in dmesg I have no clue, where
> > >> to start when I want to debug this issue.
> > > 
> > > David, please consider picking up
> > > 94802151894d482e82c324edf2c658f8e6b96508
> > > ("Revert "xfrm: Fix stack-out-of-bounds read in xfrm_state_find.")
> > > 
> > > for the 4.14.y stable queue.
> > > 
> > > I think its a pretty safe bet that this fixes the problem, it broke
> > > transport mode wildcard policy lookup.
> > 
> > Ok, once we have confirmation that this fixes it I also need to pair
> > it up with Steffen's alternative fix for the bug that commit was
> > trying to fix.
> 
> We need this revert in the 4.14.y stable tree anyway as it broke
> transport mode IPsec.
> 
> I thought quite a lot about the original problem that I tried
> to fix. It is a rather subtile thing, like almost all bugs
> reported from syzcaller I have seen.
> 
> In between I think our template validation is not strict enough.
> It is possible to configure policies with transport mode template
> where the selector address family does not match the templates
> address family. The address family can not change on a transport
> mode transformation, so this configuration does not make much
> sense but lead to problems because we use the assumption that
> the address family can not change on thransport mode later on.
> 
> Unfortunately the reproducer provided by syzcaller does not
> trigger anything on my test setup, so I don't even know if
> this fixes this exact problem.
> 
> Florian, could you please give the patch blelow a try?

Doesn't help.

>  static int validate_tmpl(int nr, struct xfrm_user_tmpl *ut, u16 family)
>  {
> +	u16 prev_family;
>  	int i;
>  
>  	if (nr > XFRM_MAX_DEPTH)
>  		return -EINVAL;
>  
> +	prev_family = family;

family is AF_INET6 (10), nr is 1.

>  	for (i = 0; i < nr; i++) {
>  		/* We never validated the ut->family value, so many
>  		 * applications simply leave it at zero.  The check was
> @@ -1435,6 +1438,12 @@ static int validate_tmpl(int nr, struct xfrm_user_tmpl *ut, u16 family)
>  		if (!ut[i].family)
>  			ut[i].family = family;
>  
> +		if ((ut[i].mode == XFRM_MODE_TRANSPORT) &&
> +		    (ut[i].family != prev_family))
> +			return -EINVAL;

mode is XFRM_MODE_TRANSPORT, family == prev_family (10), so no
-EINVAL here.

Socket is AF_INET6.
setsockopt(3, SOL_IPV6, 0x23 /* IPV6_??? */, ...
sendto(3, "", 0, MSG_EOR|MSG_NOSIGNAL|0x10, {sa_family=AF_INET,
	sin_port=htons(20002), sin_addr=inet_addr("0.0.0.0")}, 16) = -1
	EAGAIN (Resource temporarily unavailable)

and then used with AF_INET address, which causes the KASAN warning
as we feed xfrm_state_find with on-stack ipv4 addresses.

Re: flow cache removed = xfrm doesnt work

[Steffen Klassert]

On Mon, Nov 27, 2017 at 11:39:40AM +0100, Florian Westphal wrote:
> Steffen Klassert <steffen.klassert@secunet.com> wrote:
> > 
> > Florian, could you please give the patch blelow a try?
> 
> Doesn't help.

Thanks for testing!

> 
> >  static int validate_tmpl(int nr, struct xfrm_user_tmpl *ut, u16 family)
> >  {
> > +	u16 prev_family;
> >  	int i;
> >  
> >  	if (nr > XFRM_MAX_DEPTH)
> >  		return -EINVAL;
> >  
> > +	prev_family = family;
> 
> family is AF_INET6 (10), nr is 1.
> 
> >  	for (i = 0; i < nr; i++) {
> >  		/* We never validated the ut->family value, so many
> >  		 * applications simply leave it at zero.  The check was
> > @@ -1435,6 +1438,12 @@ static int validate_tmpl(int nr, struct xfrm_user_tmpl *ut, u16 family)
> >  		if (!ut[i].family)
> >  			ut[i].family = family;
> >  
> > +		if ((ut[i].mode == XFRM_MODE_TRANSPORT) &&
> > +		    (ut[i].family != prev_family))
> > +			return -EINVAL;
> 
> mode is XFRM_MODE_TRANSPORT, family == prev_family (10), so no
> -EINVAL here.
> 
> Socket is AF_INET6.
> setsockopt(3, SOL_IPV6, 0x23 /* IPV6_??? */, ...
> sendto(3, "", 0, MSG_EOR|MSG_NOSIGNAL|0x10, {sa_family=AF_INET,
> 	sin_port=htons(20002), sin_addr=inet_addr("0.0.0.0")}, 16) = -1
> 	EAGAIN (Resource temporarily unavailable)
> 
> and then used with AF_INET address, which causes the KASAN warning
> as we feed xfrm_state_find with on-stack ipv4 addresses.

This means that a policy with IPv6 selector matched an IPv4 flow, hmm.
This is probably because most selector fields are wildcard. We could
catch this by checking the address families of selector and flow
in xfrm_sk_policy_lookup() before the policy match().

I'll try to get the reproducer to work here so that I don't need
to bother you with this, thanks again!

Re: flow cache removed = xfrm doesnt work

[Steffen Klassert]

On Mon, Nov 27, 2017 at 10:39:25AM +0100, Tomas Charvat wrote:
> Ok under heavy load� patch
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/?id=94802151894d482e82c324edf2c658f8e6b96508
> 
> cause following error once in few minutes, it doesn't happen instantly

This patch simply reverts to the old behaviour, this bug does not seem
to be related to the revert.

Is the commit that you referring the head commit of your branch,
or did you picked this one to some other branch?

Can you give some details on your setup and send your .config?

Re: flow cache removed = xfrm doesnt work

[Tomas Charvat]

[-- Attachment #1.1.1: Type: text/plain, Size: 1068 bytes --]

It was on gentoo-sources-4.14.2 (almost vanila), config is attached.

I just took referred patch, applied to gentoo-sources-4.14.2 and result
was that:

- XFRM started to work (it didn't work at all before that patch)

- Errors that I have sent you did happen

Setup is, that we do use XFRM among multiple servers with AES-GCM.

Tomas Charvat
EXCELLO | Virusfree
w: www.virusfree.cz
e: tc@excello.cz

On 11/27/2017 12:41 PM, Steffen Klassert wrote:
> On Mon, Nov 27, 2017 at 10:39:25AM +0100, Tomas Charvat wrote:
>> Ok under heavy load  patch
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/?id=94802151894d482e82c324edf2c658f8e6b96508
>>
>> cause following error once in few minutes, it doesn't happen instantly
> This patch simply reverts to the old behaviour, this bug does not seem
> to be related to the revert.
>
> Is the commit that you referring the head commit of your branch,
> or did you picked this one to some other branch?
>
> Can you give some details on your setup and send your .config?
>


[-- Attachment #1.1.2: config_4.14.2 --]
[-- Type: application/x-troff-man, Size: 85773 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: flow cache removed = xfrm doesnt work

[Steffen Klassert]

Cc netdev@vger.kernel.org, remove stable@vger.kernel.org from Cc.

On Mon, Nov 27, 2017 at 01:36:50PM +0100, Tomas Charvat wrote:
> It was on gentoo-sources-4.14.2 (almost vanila), config is attached.

Could you please test with a vanilla v4.14.2 from kernel.org with
the referred patch?

If the problem is still there, please try to disable:

CONFIG_INET_ESP_OFFLOAD
CONFIG_INET6_ESP_OFFLOAD

These are quite new, so might cause problems.

Re: flow cache removed = xfrm doesnt work

[Tomas Charvat]

[-- Attachment #1.1: Type: text/plain, Size: 4003 bytes --]

Gentoo-sources has no change vs vanilla in ipsec. However here is result
from Vanila 4.14.2 with OFFLOAD=N

[ 2338.440735] BUG: unable to handle kernel NULL pointer dereference at
0000000000000018
[ 2338.440830] IP: xfrm_output_resume+0x211/0x460
[ 2338.440897] PGD 0 P4D 0
[ 2338.440960] Oops: 0000 [#1] SMP
[ 2338.441025] CPU: 0 PID: 31977 Comm: stunnel Not tainted 4.14.2 #2
[ 2338.441098] Hardware name: Supermicro H8DGU/H8DGU, BIOS 3.5c      
03/18/2016
[ 2338.441174] task: ffff98c4dc0ee4c0 task.stack: ffffb41460d84000
[ 2338.441248] RIP: 0010:xfrm_output_resume+0x211/0x460
[ 2338.441343] RSP: 0018:ffffb41460d87900 EFLAGS: 00010246
[ 2338.441413] RAX: 0000000000000000 RBX: ffff98c54090e2e0 RCX:
0000000000000000
[ 2338.441489] RDX: 000000000000001b RSI: 0000000000000000 RDI:
0000000000000000
[ 2338.441565] RBP: ffffb41460d87960 R08: ffffb41460d87780 R09:
ffff98c54b94ec50
[ 2338.441640] R10: ffff98c4d6bd38cc R11: 0000000000000000 R12:
ffffffff8cb0f740
[ 2338.441716] R13: ffff98c54b94e400 R14: ffff98c54b94e43c R15:
ffffb41460d87b28
[ 2338.441792] FS:  00007ec52f91c700(0000) GS:ffff98c54fc00000(0000)
knlGS:0000000000000000
[ 2338.441871] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2338.441942] CR2: 0000000000000018 CR3: 000000100a5e6000 CR4:
00000000000406f0
[ 2338.442018] Call Trace:
[ 2338.442083]  ? skb_checksum+0x3f/0x60
[ 2338.442227]  ? reqsk_fastopen_remove+0x160/0x160
[ 2338.442374]  ? skb_panic+0x70/0x70
[ 2338.442546]  xfrm_output+0x6a/0x110
[ 2338.442689]  xfrm6_output_finish+0x38/0x40
[ 2338.442834]  __xfrm6_output+0x57/0x1e0
[ 2338.442978]  xfrm6_output+0x9e/0x110
[ 2338.443121]  ? xfrm6_local_rxpmtu+0x90/0x90
[ 2338.443267]  ip6_xmit+0x297/0x550
[ 2338.443409]  ? ip6_dst_check+0xd7/0xf0
[ 2338.443555]  ? __ip6_append_data.isra.42+0xc00/0xc00
[ 2338.443705]  inet6_csk_xmit+0xaa/0x100
[ 2338.443880]  tcp_transmit_skb+0x57f/0xa00
[ 2338.444038]  tcp_write_xmit+0x1cd/0xf90
[ 2338.444183]  __tcp_push_pending_frames+0x3f/0xb0
[ 2338.444330]  tcp_push+0xf6/0x120
[ 2338.444471]  tcp_sendmsg_locked+0x672/0xe30
[ 2338.444617]  tcp_sendmsg+0x3a/0x60
[ 2338.444760]  inet_sendmsg+0x3b/0xb0
[ 2338.444904]  sock_sendmsg+0x48/0x60
[ 2338.445070]  sock_write_iter+0x8d/0x100
[ 2338.445216]  __vfs_write+0x154/0x1b0
[ 2338.445359]  vfs_write+0xcd/0x1c0
[ 2338.445501]  SyS_write+0x62/0xd0
[ 2338.445643]  entry_SYSCALL_64_fastpath+0x13/0x94
[ 2338.445789] RIP: 0033:0x7ec52eb3336d
[ 2338.445930] RSP: 002b:00007ec52f91bd40 EFLAGS: 00000293 ORIG_RAX:
0000000000000001
[ 2338.446189] RAX: ffffffffffffffda RBX: 00007ec520011010 RCX:
00007ec52eb3336d
[ 2338.446341] RDX: 000000000000001c RSI: 000056f01370b5a0 RDI:
0000000000000003
[ 2338.446495] RBP: 0000000000000000 R08: 0000000000000000 R09:
0000000000000000
[ 2338.446647] R10: 0000000000000000 R11: 0000000000000293 R12:
000056f011c8010c
[ 2338.446800] R13: 00007ec520010fc0 R14: 000056f013706da0 R15:
0000000000000047
[ 2338.446952] Code: f6 0f 8f b9 fe ff ff 31 f6 85 d2 0f 8f af fe ff ff
e9 bb fe ff ff 85 f6 89 f0 0f 85 d6 fe ff ff 48 8b 7b 58 48 89 f8 48 83
e0 fe <4c> 8b 70 18 4d 85 f6 0f 84 a3 01 00 00 41 8b 86 80 00 00 00 85
[ 2338.447330] RIP: xfrm_output_resume+0x211/0x460 RSP: ffffb41460d87900
[ 2338.447505] CR2: 0000000000000018
[ 2338.447748] ---[ end trace ed1d8c14abc8e6d6 ]---

Tomas Charvat
EXCELLO | Virusfree
w: www.virusfree.cz
e: tc@excello.cz

On 11/27/2017 01:49 PM, Steffen Klassert wrote:
> Cc netdev@vger.kernel.org, remove stable@vger.kernel.org from Cc.
>
> On Mon, Nov 27, 2017 at 01:36:50PM +0100, Tomas Charvat wrote:
>> It was on gentoo-sources-4.14.2 (almost vanila), config is attached.
> Could you please test with a vanilla v4.14.2 from kernel.org with
> the referred patch?
>
> If the problem is still there, please try to disable:
>
> CONFIG_INET_ESP_OFFLOAD
> CONFIG_INET6_ESP_OFFLOAD
>
> These are quite new, so might cause problems.
>



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: flow cache removed = xfrm doesnt work

[Steffen Klassert]

On Mon, Nov 27, 2017 at 11:39:40AM +0100, Florian Westphal wrote:
> Steffen Klassert <steffen.klassert@secunet.com> wrote:
> > 
> > In between I think our template validation is not strict enough.
> > It is possible to configure policies with transport mode template
> > where the selector address family does not match the templates
> > address family. The address family can not change on a transport
> > mode transformation, so this configuration does not make much
> > sense but lead to problems because we use the assumption that
> > the address family can not change on thransport mode later on.
> > 
> > Unfortunately the reproducer provided by syzcaller does not
> > trigger anything on my test setup, so I don't even know if
> > this fixes this exact problem.

I had to update my compliler, now I can reproduce the problem.

> > 
> > Florian, could you please give the patch blelow a try?
> 
> Doesn't help.

It does not fix this problem, but the one referenced here:

https://lkml.org/lkml/2017/11/22/414

I knew it fixes something :-)

The bug we are talking about here should be fixed
with the patch below.

Subject: [PATCH] xfrm: Fix stack-out-of-bounds read on socket policy lookup.

When we do tunnel or beet mode, we pass saddr and daddr from the
template to xfrm_state_find(), this is ok. On transport mode,
we pass the addresses from the flowi, assuming that the IP
addresses (and address family) don't change during transformation.
This assumption is wrong in the IPv4 mapped IPv6 case, packet
is IPv4 and template is IPv6.

Fix this by catching address family missmatches of the policy
and the flow already before we do the lookup.

Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
---
 net/xfrm/xfrm_policy.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
index 9542975eb2f9..038ec68f6901 100644
--- a/net/xfrm/xfrm_policy.c
+++ b/net/xfrm/xfrm_policy.c
@@ -1168,9 +1168,15 @@ static struct xfrm_policy *xfrm_sk_policy_lookup(const struct sock *sk, int dir,
  again:
 	pol = rcu_dereference(sk->sk_policy[dir]);
 	if (pol != NULL) {
-		bool match = xfrm_selector_match(&pol->selector, fl, family);
+		bool match;
 		int err = 0;
 
+		if (pol->family != family) {
+			pol = NULL;
+			goto out;
+		}
+
+		match = xfrm_selector_match(&pol->selector, fl, family);
 		if (match) {
 			if ((sk->sk_mark & pol->mark.m) != pol->mark.v) {
 				pol = NULL;
-- 
2.14.1

Re: flow cache removed = xfrm doesnt work

[Steffen Klassert]

On Mon, Nov 27, 2017 at 05:46:28PM +0100, Tomas Charvat wrote:
> Gentoo-sources has no change vs vanilla in ipsec. However here is result
> from Vanila 4.14.2 with OFFLOAD=N
> 
> [ 2338.440735] BUG: unable to handle kernel NULL pointer dereference at
> 0000000000000018
> [ 2338.440830] IP: xfrm_output_resume+0x211/0x460
> [ 2338.440897] PGD 0 P4D 0
> [ 2338.440960] Oops: 0000 [#1] SMP
> [ 2338.441025] CPU: 0 PID: 31977 Comm: stunnel Not tainted 4.14.2 #2
> [ 2338.441098] Hardware name: Supermicro H8DGU/H8DGU, BIOS 3.5c      
> 03/18/2016
> [ 2338.441174] task: ffff98c4dc0ee4c0 task.stack: ffffb41460d84000
> [ 2338.441248] RIP: 0010:xfrm_output_resume+0x211/0x460
> [ 2338.441343] RSP: 0018:ffffb41460d87900 EFLAGS: 00010246
> [ 2338.441413] RAX: 0000000000000000 RBX: ffff98c54090e2e0 RCX:
> 0000000000000000
> [ 2338.441489] RDX: 000000000000001b RSI: 0000000000000000 RDI:
> 0000000000000000
> [ 2338.441565] RBP: ffffb41460d87960 R08: ffffb41460d87780 R09:
> ffff98c54b94ec50
> [ 2338.441640] R10: ffff98c4d6bd38cc R11: 0000000000000000 R12:
> ffffffff8cb0f740
> [ 2338.441716] R13: ffff98c54b94e400 R14: ffff98c54b94e43c R15:
> ffffb41460d87b28
> [ 2338.441792] FS:  00007ec52f91c700(0000) GS:ffff98c54fc00000(0000)
> knlGS:0000000000000000
> [ 2338.441871] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2338.441942] CR2: 0000000000000018 CR3: 000000100a5e6000 CR4:
> 00000000000406f0
> [ 2338.442018] Call Trace:
> [ 2338.442083]  ? skb_checksum+0x3f/0x60
> [ 2338.442227]  ? reqsk_fastopen_remove+0x160/0x160
> [ 2338.442374]  ? skb_panic+0x70/0x70
> [ 2338.442546]  xfrm_output+0x6a/0x110
> [ 2338.442689]  xfrm6_output_finish+0x38/0x40
> [ 2338.442834]  __xfrm6_output+0x57/0x1e0
> [ 2338.442978]  xfrm6_output+0x9e/0x110
> [ 2338.443121]  ? xfrm6_local_rxpmtu+0x90/0x90
> [ 2338.443267]  ip6_xmit+0x297/0x550
> [ 2338.443409]  ? ip6_dst_check+0xd7/0xf0
> [ 2338.443555]  ? __ip6_append_data.isra.42+0xc00/0xc00
> [ 2338.443705]  inet6_csk_xmit+0xaa/0x100
> [ 2338.443880]  tcp_transmit_skb+0x57f/0xa00
> [ 2338.444038]  tcp_write_xmit+0x1cd/0xf90
> [ 2338.444183]  __tcp_push_pending_frames+0x3f/0xb0
> [ 2338.444330]  tcp_push+0xf6/0x120
> [ 2338.444471]  tcp_sendmsg_locked+0x672/0xe30
> [ 2338.444617]  tcp_sendmsg+0x3a/0x60
> [ 2338.444760]  inet_sendmsg+0x3b/0xb0
> [ 2338.444904]  sock_sendmsg+0x48/0x60
> [ 2338.445070]  sock_write_iter+0x8d/0x100
> [ 2338.445216]  __vfs_write+0x154/0x1b0
> [ 2338.445359]  vfs_write+0xcd/0x1c0
> [ 2338.445501]  SyS_write+0x62/0xd0
> [ 2338.445643]  entry_SYSCALL_64_fastpath+0x13/0x94
> [ 2338.445789] RIP: 0033:0x7ec52eb3336d
> [ 2338.445930] RSP: 002b:00007ec52f91bd40 EFLAGS: 00000293 ORIG_RAX:
> 0000000000000001
> [ 2338.446189] RAX: ffffffffffffffda RBX: 00007ec520011010 RCX:
> 00007ec52eb3336d
> [ 2338.446341] RDX: 000000000000001c RSI: 000056f01370b5a0 RDI:
> 0000000000000003
> [ 2338.446495] RBP: 0000000000000000 R08: 0000000000000000 R09:
> 0000000000000000
> [ 2338.446647] R10: 0000000000000000 R11: 0000000000000293 R12:
> 000056f011c8010c
> [ 2338.446800] R13: 00007ec520010fc0 R14: 000056f013706da0 R15:
> 0000000000000047
> [ 2338.446952] Code: f6 0f 8f b9 fe ff ff 31 f6 85 d2 0f 8f af fe ff ff
> e9 bb fe ff ff 85 f6 89 f0 0f 85 d6 fe ff ff 48 8b 7b 58 48 89 f8 48 83
> e0 fe <4c> 8b 70 18 4d 85 f6 0f 84 a3 01 00 00 41 8b 86 80 00 00 00 85

When I look at this code, I'd say it tries to dereference dst->child
and dst is a NULL pointer. Unfortunately I don't see how this can
happen. I need to find a way to reproduce this.

Can you show me your policy and SA database, i.e.
'ip x s' and 'ip x p'?

Also, please show /proc/crypto

Re: flow cache removed = xfrm doesnt work

[Tomas Charvat]

[-- Attachment #1: Type: text/plain, Size: 2567 bytes --]

> Ok I have tried 4.14.14 and got following after 1 hour or so. It was
> not fatal, system kept going.



[ 2833.840452] BUG: unable to handle kernel NULL pointer dereference at
0000000000000018
[ 2833.840549] IP: xfrm_output_resume+0x20a/0x460
[ 2833.840618] PGD 0 P4D 0 
[ 2833.840682] Oops: 0000 [#1] SMP NOPTI
[ 2833.840752] CPU: 81 PID: 121868 Comm: spamd child Not tainted
4.14.14-gentoo #2
[ 2833.840831] Hardware name: Supermicro AS -1123US-TR4/H11DSU-iN, BIOS
1.0a 09/25/2017
[ 2833.840911] task: ffff891d1d5b8600 task.stack: ffff980cde96c000
[ 2833.840987] RIP: 0010:xfrm_output_resume+0x20a/0x460
[ 2833.841058] RSP: 0018:ffff980cde96fb70 EFLAGS: 00010246
[ 2833.841129] RAX: 0000000000000000 RBX: ffff891b1e0a48e0 RCX:
0000000000000000
[ 2833.841207] RDX: 0000000000000005 RSI: 0000000000000000 RDI:
0000000000000000
[ 2833.841285] RBP: ffffffffb1f11300 R08: ffff980cde96fa20 R09:
ffff891b199ba450
[ 2833.841363] R10: ffff891b1d5b30c4 R11: 0000000000000000 R12:
ffff891b1e0f7400
[ 2833.841441] R13: ffff891b1e0f743c R14: 0000000000000020 R15:
000000000001f000
[ 2833.841519] FS:  00007fe90028e700(0000) GS:ffff891b1fc40000(0000)
knlGS:0000000000000000
[ 2833.841601] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2833.841673] CR2: 0000000000000018 CR3: 0000800dcc944000 CR4:
00000000001406e0
[ 2833.841908] Call Trace:
[ 2833.842141]  ? skb_checksum_help+0x83/0x1b0
[ 2833.842382]  ? xfrm_output+0x1c9/0x210
[ 2833.842626]  xfrm6_output+0x91/0x110
[ 2833.842904]  ? xfrm6_local_rxpmtu+0x80/0x80
[ 2833.843200]  tcp_transmit_skb+0x541/0x990
[ 2833.843488]  tcp_write_xmit+0x1bc/0xf90
[ 2833.843743]  ? _copy_from_iter_full+0x93/0x230
[ 2833.843955]  __tcp_push_pending_frames+0x28/0x90
[ 2833.844115]  tcp_sendmsg_locked+0x65c/0xe40
[ 2833.844351]  tcp_sendmsg+0x2e/0x50
[ 2833.844565]  sock_sendmsg+0x3e/0x50
[ 2833.844784]  sock_write_iter+0x82/0xf0
[ 2833.845011]  __vfs_write+0x129/0x190
[ 2833.845233]  vfs_write+0xc3/0x1c0
[ 2833.845500]  SyS_write+0x5f/0xd0
[ 2833.845709]  entry_SYSCALL_64_fastpath+0x19/0x72
[ 2833.845932] RIP: 0033:0x7fe8ff9b9b80
[ 2833.846148] RSP: 002b:00007ffd441632c8 EFLAGS: 00000246
[ 2833.846148] Code: b0 fe ff ff 31 f6 85 d2 0f 8f a6 fe ff ff 0f 1f 00
e9 af fe ff ff 85 f6 89 f0 0f 85 cc fe ff ff 48 8b 7b 58 48 89 f8 48 83
e0 fe <4c> 8b 68 18 4d 85 ed 0f 84 ad 01 00 00 41 8b 85 80 00 00 00 85 
[ 2833.846987] RIP: xfrm_output_resume+0x20a/0x460 RSP:
ffff980cde96fb70
[ 2833.847280] CR2: 0000000000000018
[ 2833.847532] ---[ end trace 504f6c380fc3f734 ]---

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]