[PATCH net] net/smc: postpone sk_refcnt increment in connect()

[PATCH net] net/smc: postpone sk_refcnt increment in connect()

[liuyacan]

From: liuyacan <liuyacan@corp.netease.com>

Same trigger condition as commit 86434744. When setsockopt runs
in parallel to a connect(), and switch the socket into fallback
mode. Then the sk_refcnt is incremented in smc_connect(), but
its state stay in SMC_INIT (NOT SMC_ACTIVE). This cause the
corresponding sk_refcnt decrement in __smc_release() will not be
performed.

Signed-off-by: liuyacan <liuyacan@corp.netease.com>
Fixes: 86434744 ("net/smc: add fallback check to connect()")
---
 net/smc/af_smc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
index fce16b9d6..45a24d242 100644
--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c
@@ -1564,9 +1564,9 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr,
 	if (rc && rc != -EINPROGRESS)
 		goto out;
 
-	sock_hold(&smc->sk); /* sock put in passive closing */
 	if (smc->use_fallback)
 		goto out;
+	sock_hold(&smc->sk); /* sock put in passive closing */
 	if (flags & O_NONBLOCK) {
 		if (queue_work(smc_hs_wq, &smc->connect_work))
 			smc->connect_nonblock = 1;
-- 
2.20.1

[PATCH v2 net] net/smc: postpone sk_refcnt increment in connect()

[liuyacan]

From: liuyacan <liuyacan@corp.netease.com>

Same trigger condition as commit 86434744. When setsockopt runs
in parallel to a connect(), and switch the socket into fallback
mode. Then the sk_refcnt is incremented in smc_connect(), but
its state stay in SMC_INIT (NOT SMC_ACTIVE). This cause the
corresponding sk_refcnt decrement in __smc_release() will not be
performed.

Fixes: 86434744fedf ("net/smc: add fallback check to connect()")
Signed-off-by: liuyacan <liuyacan@corp.netease.com>
---
 net/smc/af_smc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
index fce16b9d6..45a24d242 100644
--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c
@@ -1564,9 +1564,9 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr,
 	if (rc && rc != -EINPROGRESS)
 		goto out;
 
-	sock_hold(&smc->sk); /* sock put in passive closing */
 	if (smc->use_fallback)
 		goto out;
+	sock_hold(&smc->sk); /* sock put in passive closing */
 	if (flags & O_NONBLOCK) {
 		if (queue_work(smc_hs_wq, &smc->connect_work))
 			smc->connect_nonblock = 1;
-- 
2.20.1

Re: [PATCH v2 net] net/smc: postpone sk_refcnt increment in connect()

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net.git (master)
by David S. Miller <davem@davemloft.net>:

On Mon, 23 May 2022 12:57:07 +0800 you wrote:
> From: liuyacan <liuyacan@corp.netease.com>
> 
> Same trigger condition as commit 86434744. When setsockopt runs
> in parallel to a connect(), and switch the socket into fallback
> mode. Then the sk_refcnt is incremented in smc_connect(), but
> its state stay in SMC_INIT (NOT SMC_ACTIVE). This cause the
> corresponding sk_refcnt decrement in __smc_release() will not be
> performed.
> 
> [...]

Here is the summary with links:
  - [v2,net] net/smc: postpone sk_refcnt increment in connect()
    https://git.kernel.org/netdev/net/c/75c1edf23b95

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: [PATCH v2 net] net/smc: postpone sk_refcnt increment in connect()

[Karsten Graul]

On 23/05/2022 06:57, liuyacan@corp.netease.com wrote:
> From: liuyacan <liuyacan@corp.netease.com>
> 
> Same trigger condition as commit 86434744. When setsockopt runs
> in parallel to a connect(), and switch the socket into fallback
> mode. Then the sk_refcnt is incremented in smc_connect(), but
> its state stay in SMC_INIT (NOT SMC_ACTIVE). This cause the
> corresponding sk_refcnt decrement in __smc_release() will not be
> performed.
> 
> Fixes: 86434744fedf ("net/smc: add fallback check to connect()")
> Signed-off-by: liuyacan <liuyacan@corp.netease.com>
> ---
>  net/smc/af_smc.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
> index fce16b9d6..45a24d242 100644
> --- a/net/smc/af_smc.c
> +++ b/net/smc/af_smc.c
> @@ -1564,9 +1564,9 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr,
>  	if (rc && rc != -EINPROGRESS)
>  		goto out;
>  
> -	sock_hold(&smc->sk); /* sock put in passive closing */
>  	if (smc->use_fallback)
>  		goto out;
> +	sock_hold(&smc->sk); /* sock put in passive closing */
>  	if (flags & O_NONBLOCK) {
>  		if (queue_work(smc_hs_wq, &smc->connect_work))
>  			smc->connect_nonblock = 1;

This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect()
is called. But nevertheless, it is a problem.

Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback
processing. This change also conflicts with a patch that is already on net-next (3aba1030).

With the new patch on net-next it would also be possible to detect in __smc_release() that
the socket is in state sk->sk_state == SMC_INIT but the sock->state is SS_CONNECTING or 
SS_CONNECTED and call sock_put() in this case.
What do you think?

Re: [PATCH v2 net] net/smc: postpone sk_refcnt increment in connect()

[liuyacan]

> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect()
> is called. But nevertheless, it is a problem.
> 
> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback
> processing. This change also conflicts with a patch that is already on net-next (3aba1030).

Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove 
the fallback branch sock_put() in __smc_release()?

> With the new patch on net-next it would also be possible to detect in __smc_release() that
> the socket is in state sk->sk_state == SMC_INIT but the sock->state is SS_CONNECTING or 
> SS_CONNECTED and call sock_put() in this case.
> What do you think?

Oh, I didn't notice this patch on net-next. Emm, I think I need to do some testing with this 
patch.

Thank you.

Re: [PATCH v2 net] net/smc: postpone sk_refcnt increment in connect()

[Karsten Graul]

On 23/05/2022 16:19, liuyacan@corp.netease.com wrote:
>> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect()
>> is called. But nevertheless, it is a problem.
>>
>> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback
>> processing. This change also conflicts with a patch that is already on net-next (3aba1030).
> 
> Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove 
> the fallback branch sock_put() in __smc_release()?

What I had in mind was to eventually call sock_put() in __smc_release() even if sk->sk_state == SMC_INIT
(currently the extra check in the if() for sk->sk_state != SMC_INIT prevents the sock_put()), but only
when it is sure that we actually reached the sock_hold() in smc_connect() before.

But maybe we find out that the sock_hold() is not needed for fallback sockets, I don't know...

Re: [PATCH v2 net] net/smc: postpone sk_refcnt increment in connect()

[liuyacan]

> >> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect()
> >> is called. But nevertheless, it is a problem.
> >>
> >> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback
> >> processing. This change also conflicts with a patch that is already on net-next (3aba1030).
> > 
> > Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove 
> > the fallback branch sock_put() in __smc_release()?
> 
> What I had in mind was to eventually call sock_put() in __smc_release() even if sk->sk_state == SMC_INIT
> (currently the extra check in the if() for sk->sk_state != SMC_INIT prevents the sock_put()), but only
> when it is sure that we actually reached the sock_hold() in smc_connect() before.
> 
> But maybe we find out that the sock_hold() is not needed for fallback sockets, I don't know...

I do think the sock_hold()/sock_put() for smc->sk is a bit complicated, Emm, I'm not sure if it 
can be simplified..

In fact, I'm sure there must be another ref count issue in my environment,but I haven't caught it yet.

Re: [PATCH v2 net] net/smc: postpone sk_refcnt increment in connect()

[Tony Lu]

On Mon, May 23, 2022 at 11:21:19PM +0800, liuyacan@corp.netease.com wrote:
> > >> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect()
> > >> is called. But nevertheless, it is a problem.
> > >>
> > >> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback
> > >> processing. This change also conflicts with a patch that is already on net-next (3aba1030).
> > > 
> > > Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove 
> > > the fallback branch sock_put() in __smc_release()?
> > 
> > What I had in mind was to eventually call sock_put() in __smc_release() even if sk->sk_state == SMC_INIT
> > (currently the extra check in the if() for sk->sk_state != SMC_INIT prevents the sock_put()), but only
> > when it is sure that we actually reached the sock_hold() in smc_connect() before.
> > 
> > But maybe we find out that the sock_hold() is not needed for fallback sockets, I don't know...
> 
> I do think the sock_hold()/sock_put() for smc->sk is a bit complicated, Emm, I'm not sure if it 
> can be simplified..
> 
> In fact, I'm sure there must be another ref count issue in my environment,but I haven't caught it yet.

I am wondering the issue of this ref count. If it is convenient, would
you like to provide some more details?

syzkaller has reported some issues about ref count, but syzkaller and
others' bot don't have RDMA devices, they cannot cover most of the code
routines in SMC. We are working on it to provide SMC fuzz test with RDMA
environment. So it's very nice to have real world issues.

Thanks,
Tony Lu

Re: [PATCH v2 net] net/smc: postpone sk_refcnt increment in connect()

[Karsten Graul]

On 23/05/2022 17:21, liuyacan@corp.netease.com wrote:
>>>> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect()
>>>> is called. But nevertheless, it is a problem.
>>>>
>>>> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback
>>>> processing. This change also conflicts with a patch that is already on net-next (3aba1030).
>>>
>>> Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove 
>>> the fallback branch sock_put() in __smc_release()?
>>
>> What I had in mind was to eventually call sock_put() in __smc_release() even if sk->sk_state == SMC_INIT
>> (currently the extra check in the if() for sk->sk_state != SMC_INIT prevents the sock_put()), but only
>> when it is sure that we actually reached the sock_hold() in smc_connect() before.
>>
>> But maybe we find out that the sock_hold() is not needed for fallback sockets, I don't know...
> 
> I do think the sock_hold()/sock_put() for smc->sk is a bit complicated, Emm, I'm not sure if it 
> can be simplified..
> 
> In fact, I'm sure there must be another ref count issue in my environment,but I haven't caught it yet.
> 

Can you check my latest mail from a minute ago in thread
"Re: [PATCH net-next v2] net/smc: align the connect behaviour with TCP"
I think this answer also affects our discussion.

Re: [PATCH v2 net] net/smc: postpone sk_refcnt increment in connect()

[liuyacan]

> > > >> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect()
> > > >> is called. But nevertheless, it is a problem.
> > > >>
> > > >> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback
> > > >> processing. This change also conflicts with a patch that is already on net-next (3aba1030).
> > > > 
> > > > Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove 
> > > > the fallback branch sock_put() in __smc_release()?
> > > 
> > > What I had in mind was to eventually call sock_put() in __smc_release() even if sk->sk_state == SMC_INIT
> > > (currently the extra check in the if() for sk->sk_state != SMC_INIT prevents the sock_put()), but only
> > > when it is sure that we actually reached the sock_hold() in smc_connect() before.
> > > 
> > > But maybe we find out that the sock_hold() is not needed for fallback sockets, I don't know...
> > 
> > I do think the sock_hold()/sock_put() for smc->sk is a bit complicated, Emm, I'm not sure if it 
> > can be simplified..
> > 
> > In fact, I'm sure there must be another ref count issue in my environment,but I haven't caught it yet.
> 
> I am wondering the issue of this ref count. If it is convenient, would
> you like to provide some more details?
> 
> syzkaller has reported some issues about ref count, but syzkaller and
> others' bot don't have RDMA devices, they cannot cover most of the code
> routines in SMC. We are working on it to provide SMC fuzz test with RDMA
> environment. So it's very nice to have real world issues.
> 
> Thanks,
> Tony Lu

I have encountered two types of problems. However, I cannot reproduce it stably.

case 1. After closing the app (>> TIME_WAIT), 'lsmod' shows that the smc module ref count is still greater than 0.
case 2 [rare]. 'lsmod' shows smc module ref count is less than 0.

Some clues of case 2 are as follows:

kernel: [67166.688386] ------------[ cut here ]------------
  kernel: [67166.693658] cache_from_obj: Wrong slab cache. SMC but object is from SMC
  kernel: [67166.701136] WARNING: CPU: 47 PID: 176961 at mm/slab.h:469 kmem_cache_free+0x329/0x410
  ......
  kernel: [67166.846819] CPU: 47 PID: 176961 Comm: redis-server Kdump: loaded Tainted: G  R B      OE     5.10.0-0.bpo.9-amd64 #1 Debian 5.10.70-1~bpo10+1
  kernel: [67166.860915] Hardware name: Inspur SA5280M6/SA5280M6, BIOS 06.00.01 10/09/2021
  kernel: [67166.868747] RIP: 0010:kmem_cache_free+0x329/0x410
  kernel: [67166.874168] Code: ff 0f 0b 48 8d b8 f0 9d 02 00 e9 e4 fe ff ff 48 8b 57 60 49 8b 4f 60 48 c7 c6 30 86 63 a4 48 c7 c7 f8 e6 8f a4 e8 89 63 5c 00 <0f> 0b 48 89 de 4c
89 ff e8 1a ad ff ff 48 8b 0d 63 34 ef 00 e9 49
  kernel: [67166.894360] RSP: 0018:ffffbd450f527e18 EFLAGS: 00010286
  kernel: [67166.900306] RAX: 0000000000000000 RBX: ffffa00fa4548d00 RCX: 0000000000000000
  kernel: [67166.908169] RDX: ffffa04c7f7e8760 RSI: ffffa04c7f7d8a00 RDI: ffffa04c7f7d8a00
  kernel: [67166.916027] RBP: ffffa01024548d00 R08: 0000000000000000 R09: c0000000ffffbfff
  kernel: [67166.923860] R10: 0000000000000001 R11: ffffbd450f527c20 R12: 0000000000000000
  kernel: [67166.931713] R13: 0000000000000000 R14: ffffa00fa4548f28 R15: ffffa02d3366bf00
  kernel: [67166.939564] FS:  00007fe131c80f40(0000) GS:ffffa04c7f7c0000(0000) knlGS:0000000000000000
  kernel: [67166.948361] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: [67166.954817] CR2: 00007fe12f477000 CR3: 00000004874be003 CR4: 0000000000770ee0
  kernel: [67166.962662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  kernel: [67166.970498] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  kernel: [67166.978306] PKRU: 55555554
  kernel: [67166.981695] Call Trace:
  kernel: [67166.985017]  __sk_destruct+0x12c/0x1e0
  kernel: [67166.989449]  smc_release+0x19a/0x230 [smc]
  kernel: [67166.994325]  __sock_release+0x3d/0xa0
  kernel: [67166.998656]  sock_close+0x11/0x20
  kernel: [67167.002637]  __fput+0x93/0x240
  kernel: [67167.006347]  task_work_run+0x76/0xb0
  kernel: [67167.010569]  exit_to_user_mode_prepare+0x129/0x130
  kernel: [67167.016000]  syscall_exit_to_user_mode+0x28/0x140
  kernel: [67167.021339]  entry_SYSCALL_64_after_hwframe+0x44/0xa9