* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) @ 2018-02-02 21:31 Daniel Reichelt 2018-02-06 0:24 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Daniel Reichelt @ 2018-02-02 21:31 UTC (permalink / raw) To: Trond Myklebust, Steven Rostedt; +Cc: Linux NFS Mailing List, LKML [-- Attachment #1.1: Type: text/plain, Size: 672 bytes --] Hi Trond, Steven, eversince I switched from Debian Jessie to Stretch last summer, I've been seeing the very same hidden ports on an NFS server as described in [1], which is a follow-up to [2]. Your patch ([3], [4]) solved the issue back then. Later on, you changed that fix again in [5], which lead to the situation we're seeing today. Reverting 0b0ab51 fixes the issue for me. Let me know if you need more info. Thanks Daniel [1] https://lkml.org/lkml/2016/6/30/341 [2] https://lkml.org/lkml/2015/6/11/803 [3] https://lkml.org/lkml/2015/6/19/759 [4] 4876cc779ff525b9c2376d8076edf47815e71f2c [5] 4b0ab51db32eba0f48b7618254742f143364a28d [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 866 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) 2018-02-02 21:31 It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) Daniel Reichelt @ 2018-02-06 0:24 ` Trond Myklebust 2018-02-06 9:20 ` Daniel Reichelt 0 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2018-02-06 0:24 UTC (permalink / raw) To: rostedt, hacking; +Cc: linux-kernel, linux-nfs [-- Attachment #1: Type: text/plain, Size: 3592 bytes --] On Fri, 2018-02-02 at 22:31 +0100, Daniel Reichelt wrote: > Hi Trond, Steven, > > eversince I switched from Debian Jessie to Stretch last summer, I've > been seeing the very same hidden ports on an NFS server as described > in > [1], which is a follow-up to [2]. > > Your patch ([3], [4]) solved the issue back then. Later on, you > changed > that fix again in [5], which lead to the situation we're seeing > today. > > Reverting 0b0ab51 fixes the issue for me. > > Let me know if you need more info. > > > > Thanks > Daniel > > > [1] https://lkml.org/lkml/2016/6/30/341 > [2] https://lkml.org/lkml/2015/6/11/803 > [3] https://lkml.org/lkml/2015/6/19/759 > [4] 4876cc779ff525b9c2376d8076edf47815e71f2c > [5] 4b0ab51db32eba0f48b7618254742f143364a28d Does the following fix the issue? 8<----------------------------------------------- From 9b30889c548a4d45bfe6226e58de32504c1d682f Mon Sep 17 00:00:00 2001 From: Trond Myklebust <trond.myklebust@primarydata.com> Date: Mon, 5 Feb 2018 10:20:06 -0500 Subject: [PATCH] SUNRPC: Ensure we always close the socket after a connection shuts down Ensure that we release the TCP socket once it is in the TCP_CLOSE or TCP_TIME_WAIT state (and only then) so that we don't confuse rkhunter and its ilk. Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> --- net/sunrpc/xprtsock.c | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c index 18803021f242..5d0108172ed3 100644 --- a/net/sunrpc/xprtsock.c +++ b/net/sunrpc/xprtsock.c @@ -807,13 +807,6 @@ static void xs_sock_reset_connection_flags(struct rpc_xprt *xprt) smp_mb__after_atomic(); } -static void xs_sock_mark_closed(struct rpc_xprt *xprt) -{ - xs_sock_reset_connection_flags(xprt); - /* Mark transport as closed and wake up all pending tasks */ - xprt_disconnect_done(xprt); -} - /** * xs_error_report - callback to handle TCP socket state errors * @sk: socket @@ -833,9 +826,6 @@ static void xs_error_report(struct sock *sk) err = -sk->sk_err; if (err == 0) goto out; - /* Is this a reset event? */ - if (sk->sk_state == TCP_CLOSE) - xs_sock_mark_closed(xprt); dprintk("RPC: xs_error_report client %p, error=%d...\n", xprt, -err); trace_rpc_socket_error(xprt, sk->sk_socket, err); @@ -1655,9 +1645,11 @@ static void xs_tcp_state_change(struct sock *sk) if (test_and_clear_bit(XPRT_SOCK_CONNECTING, &transport->sock_state)) xprt_clear_connecting(xprt); + clear_bit(XPRT_CLOSING, &xprt->state); if (sk->sk_err) xprt_wake_pending_tasks(xprt, -sk->sk_err); - xs_sock_mark_closed(xprt); + /* Trigger the socket release */ + xs_tcp_force_close(xprt); } out: read_unlock_bh(&sk->sk_callback_lock); @@ -2265,14 +2257,19 @@ static void xs_tcp_shutdown(struct rpc_xprt *xprt) { struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); struct socket *sock = transport->sock; + int skst = transport->inet ? transport->inet->sk_state : TCP_CLOSE; if (sock == NULL) return; - if (xprt_connected(xprt)) { + switch (skst) { + default: kernel_sock_shutdown(sock, SHUT_RDWR); trace_rpc_socket_shutdown(xprt, sock); - } else + break; + case TCP_CLOSE: + case TCP_TIME_WAIT: xs_reset_transport(transport); + } } static void xs_tcp_set_socket_timeouts(struct rpc_xprt *xprt, -- 2.14.3 -- Trond Myklebust Linux NFS client maintainer, PrimaryData trond.myklebust@primarydata.com [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) 2018-02-06 0:24 ` Trond Myklebust @ 2018-02-06 9:20 ` Daniel Reichelt 2018-02-06 19:26 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Daniel Reichelt @ 2018-02-06 9:20 UTC (permalink / raw) To: Trond Myklebust, rostedt; +Cc: linux-kernel, linux-nfs [-- Attachment #1.1: Type: text/plain, Size: 3159 bytes --] On 02/06/2018 01:24 AM, Trond Myklebust wrote: > Does the following fix the issue? > > 8<----------------------------------------------- > From 9b30889c548a4d45bfe6226e58de32504c1d682f Mon Sep 17 00:00:00 2001 > From: Trond Myklebust <trond.myklebust@primarydata.com> > Date: Mon, 5 Feb 2018 10:20:06 -0500 > Subject: [PATCH] SUNRPC: Ensure we always close the socket after a connection > shuts down > > Ensure that we release the TCP socket once it is in the TCP_CLOSE or > TCP_TIME_WAIT state (and only then) so that we don't confuse rkhunter > and its ilk. > > Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> > --- > net/sunrpc/xprtsock.c | 23 ++++++++++------------- > 1 file changed, 10 insertions(+), 13 deletions(-) > > diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c > index 18803021f242..5d0108172ed3 100644 > --- a/net/sunrpc/xprtsock.c > +++ b/net/sunrpc/xprtsock.c > @@ -807,13 +807,6 @@ static void xs_sock_reset_connection_flags(struct rpc_xprt *xprt) > smp_mb__after_atomic(); > } > > -static void xs_sock_mark_closed(struct rpc_xprt *xprt) > -{ > - xs_sock_reset_connection_flags(xprt); > - /* Mark transport as closed and wake up all pending tasks */ > - xprt_disconnect_done(xprt); > -} > - > /** > * xs_error_report - callback to handle TCP socket state errors > * @sk: socket > @@ -833,9 +826,6 @@ static void xs_error_report(struct sock *sk) > err = -sk->sk_err; > if (err == 0) > goto out; > - /* Is this a reset event? */ > - if (sk->sk_state == TCP_CLOSE) > - xs_sock_mark_closed(xprt); > dprintk("RPC: xs_error_report client %p, error=%d...\n", > xprt, -err); > trace_rpc_socket_error(xprt, sk->sk_socket, err); > @@ -1655,9 +1645,11 @@ static void xs_tcp_state_change(struct sock *sk) > if (test_and_clear_bit(XPRT_SOCK_CONNECTING, > &transport->sock_state)) > xprt_clear_connecting(xprt); > + clear_bit(XPRT_CLOSING, &xprt->state); > if (sk->sk_err) > xprt_wake_pending_tasks(xprt, -sk->sk_err); > - xs_sock_mark_closed(xprt); > + /* Trigger the socket release */ > + xs_tcp_force_close(xprt); > } > out: > read_unlock_bh(&sk->sk_callback_lock); > @@ -2265,14 +2257,19 @@ static void xs_tcp_shutdown(struct rpc_xprt *xprt) > { > struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); > struct socket *sock = transport->sock; > + int skst = transport->inet ? transport->inet->sk_state : TCP_CLOSE; > > if (sock == NULL) > return; > - if (xprt_connected(xprt)) { > + switch (skst) { > + default: > kernel_sock_shutdown(sock, SHUT_RDWR); > trace_rpc_socket_shutdown(xprt, sock); > - } else > + break; > + case TCP_CLOSE: > + case TCP_TIME_WAIT: > xs_reset_transport(transport); > + } > } > > static void xs_tcp_set_socket_timeouts(struct rpc_xprt *xprt, > Previously, I've seen hidden ports within 5-6 minutes after re-starting the nfsd and re-mounting nfs-exports on clients. With this patch applied, I don't see any hidden ports after 15mins. I guess it's a valid fix. Thank you! Daniel [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 866 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) 2018-02-06 9:20 ` Daniel Reichelt @ 2018-02-06 19:26 ` Trond Myklebust 0 siblings, 0 replies; 15+ messages in thread From: Trond Myklebust @ 2018-02-06 19:26 UTC (permalink / raw) To: rostedt, hacking; +Cc: linux-kernel, linux-nfs [-- Attachment #1: Type: text/plain, Size: 3799 bytes --] On Tue, 2018-02-06 at 10:20 +0100, Daniel Reichelt wrote: > On 02/06/2018 01:24 AM, Trond Myklebust wrote: > > Does the following fix the issue? > > > > 8<----------------------------------------------- > > From 9b30889c548a4d45bfe6226e58de32504c1d682f Mon Sep 17 00:00:00 > > 2001 > > From: Trond Myklebust <trond.myklebust@primarydata.com> > > Date: Mon, 5 Feb 2018 10:20:06 -0500 > > Subject: [PATCH] SUNRPC: Ensure we always close the socket after a > > connection > > shuts down > > > > Ensure that we release the TCP socket once it is in the TCP_CLOSE > > or > > TCP_TIME_WAIT state (and only then) so that we don't confuse > > rkhunter > > and its ilk. > > > > Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> > > --- > > net/sunrpc/xprtsock.c | 23 ++++++++++------------- > > 1 file changed, 10 insertions(+), 13 deletions(-) > > > > diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c > > index 18803021f242..5d0108172ed3 100644 > > --- a/net/sunrpc/xprtsock.c > > +++ b/net/sunrpc/xprtsock.c > > @@ -807,13 +807,6 @@ static void > > xs_sock_reset_connection_flags(struct rpc_xprt *xprt) > > smp_mb__after_atomic(); > > } > > > > -static void xs_sock_mark_closed(struct rpc_xprt *xprt) > > -{ > > - xs_sock_reset_connection_flags(xprt); > > - /* Mark transport as closed and wake up all pending tasks > > */ > > - xprt_disconnect_done(xprt); > > -} > > - > > /** > > * xs_error_report - callback to handle TCP socket state errors > > * @sk: socket > > @@ -833,9 +826,6 @@ static void xs_error_report(struct sock *sk) > > err = -sk->sk_err; > > if (err == 0) > > goto out; > > - /* Is this a reset event? */ > > - if (sk->sk_state == TCP_CLOSE) > > - xs_sock_mark_closed(xprt); > > dprintk("RPC: xs_error_report client %p, > > error=%d...\n", > > xprt, -err); > > trace_rpc_socket_error(xprt, sk->sk_socket, err); > > @@ -1655,9 +1645,11 @@ static void xs_tcp_state_change(struct sock > > *sk) > > if (test_and_clear_bit(XPRT_SOCK_CONNECTING, > > &transport->sock_state)) > > xprt_clear_connecting(xprt); > > + clear_bit(XPRT_CLOSING, &xprt->state); > > if (sk->sk_err) > > xprt_wake_pending_tasks(xprt, -sk- > > >sk_err); > > - xs_sock_mark_closed(xprt); > > + /* Trigger the socket release */ > > + xs_tcp_force_close(xprt); > > } > > out: > > read_unlock_bh(&sk->sk_callback_lock); > > @@ -2265,14 +2257,19 @@ static void xs_tcp_shutdown(struct rpc_xprt > > *xprt) > > { > > struct sock_xprt *transport = container_of(xprt, struct > > sock_xprt, xprt); > > struct socket *sock = transport->sock; > > + int skst = transport->inet ? transport->inet->sk_state : > > TCP_CLOSE; > > > > if (sock == NULL) > > return; > > - if (xprt_connected(xprt)) { > > + switch (skst) { > > + default: > > kernel_sock_shutdown(sock, SHUT_RDWR); > > trace_rpc_socket_shutdown(xprt, sock); > > - } else > > + break; > > + case TCP_CLOSE: > > + case TCP_TIME_WAIT: > > xs_reset_transport(transport); > > + } > > } > > > > static void xs_tcp_set_socket_timeouts(struct rpc_xprt *xprt, > > > > > Previously, I've seen hidden ports within 5-6 minutes after re- > starting > the nfsd and re-mounting nfs-exports on clients. > > With this patch applied, I don't see any hidden ports after 15mins. I > guess it's a valid fix. For the record, the intention of the patch is not to adjust or correct any connection timeout values. Merely to ensure that once the connection breakage is detected by the socket layer, so that is it no longer usable by the RPC client, we release the socket. -- Trond Myklebust Linux NFS client maintainer, PrimaryData trond.myklebust@primarydata.com [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ))
@ 2016-06-30 12:59 ` Steven Rostedt
0 siblings, 0 replies; 15+ messages in thread
From: Steven Rostedt @ 2016-06-30 12:59 UTC (permalink / raw)
To: Trond Myklebust
Cc: Jeff Layton, Eric Dumazet, Anna Schumaker,
Linux NFS Mailing List, Linux Network Devel Mailing List, LKML,
Andrew Morton, Bruce James Fields
[-- Attachment #1: Type: text/plain, Size: 4137 bytes --]
[ resending as a new email, as I'm assuming people do not sort their
INBOX via last email on thread, thus my last email is sitting in the
bottom of everyone's INBOX ]
I've hit this again. Not sure when it started, but I applied my old
debug trace_printk() patch (attached) and rebooted (4.5.7). I just
tested the latest kernel from Linus's tree (from last nights pull), and
it still gives me the problem.
Here's the trace I have:
kworker/3:1H-134 [003] ..s. 61.036129: inet_csk_get_port: snum 805
kworker/3:1H-134 [003] ..s. 61.036135: <stack trace>
=> sched_clock
=> inet_addr_type_table
=> security_capable
=> inet_bind
=> xs_bind
=> release_sock
=> sock_setsockopt
=> __sock_create
=> xs_create_sock.isra.19
=> xs_tcp_setup_socket
=> process_one_work
=> worker_thread
=> worker_thread
=> kthread
=> ret_from_fork
=> kthread
kworker/3:1H-134 [003] ..s. 61.036136: inet_bind_hash: add 805
kworker/3:1H-134 [003] ..s. 61.036138: <stack trace>
=> inet_csk_get_port
=> sched_clock
=> inet_addr_type_table
=> security_capable
=> inet_bind
=> xs_bind
=> release_sock
=> sock_setsockopt
=> __sock_create
=> xs_create_sock.isra.19
=> xs_tcp_setup_socket
=> process_one_work
=> worker_thread
=> worker_thread
=> kthread
=> ret_from_fork
=> kthread
kworker/3:1H-134 [003] .... 61.036139: xs_bind: RPC: xs_bind 4.136.255.255:805: ok (0)
kworker/3:1H-134 [003] .... 61.036140: xs_tcp_setup_socket: RPC: worker connecting xprt ffff880407eca800 via tcp to 192.168.23.22 (port 43651)
kworker/3:1H-134 [003] .... 61.036162: xs_tcp_setup_socket: RPC: ffff880407eca800 connect status 115 connected 0 sock state 2
<idle>-0 [001] ..s. 61.036450: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff880407eca800...
<idle>-0 [001] ..s. 61.036452: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0
kworker/1:1H-136 [001] .... 61.036476: xprt_connect_status: RPC: 43 xprt_connect_status: retrying
kworker/1:1H-136 [001] .... 61.036478: xprt_prepare_transmit: RPC: 43 xprt_prepare_transmit
kworker/1:1H-136 [001] .... 61.036479: xprt_transmit: RPC: 43 xprt_transmit(72)
kworker/1:1H-136 [001] .... 61.036486: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0
kworker/1:1H-136 [001] .... 61.036487: xprt_transmit: RPC: 43 xmit complete
<idle>-0 [001] ..s. 61.036789: xs_tcp_data_ready: RPC: xs_tcp_data_ready...
kworker/1:1H-136 [001] .... 61.036798: xs_tcp_data_recv: RPC: xs_tcp_data_recv started
kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24
kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading XID (4 bytes)
kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading request with XID 2f4c3f88
kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes)
kworker/1:1H-136 [001] .... 61.036801: xs_tcp_data_recv: RPC: read reply XID 2f4c3f88
kworker/1:1H-136 [001] ..s. 61.036801: xs_tcp_data_recv: RPC: XID 2f4c3f88 read 16 bytes
kworker/1:1H-136 [001] ..s. 61.036802: xs_tcp_data_recv: RPC: xprt = ffff880407eca800, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24
kworker/1:1H-136 [001] ..s. 61.036802: xprt_complete_rqst: RPC: 43 xid 2f4c3f88 complete (24 bytes received)
kworker/1:1H-136 [001] .... 61.036803: xs_tcp_data_recv: RPC: xs_tcp_data_recv done
kworker/1:1H-136 [001] .... 61.036812: xprt_release: RPC: 43 release request ffff88040b270800
# unhide-tcp
Unhide-tcp 20130526
Copyright © 2013 Yago Jesus & Patrick Gouin
License GPLv3+ : GNU GPL version 3 or later
http://www.unhide-forensics.info
Used options:
[*]Starting TCP checking
Found Hidden port that not appears in ss: 805
-- Steve
[-- Attachment #2: debug-hidden-port-4.7.patch --]
[-- Type: text/x-patch, Size: 2378 bytes --]
---
net/ipv4/inet_connection_sock.c | 4 ++++
net/ipv4/inet_hashtables.c | 5 +++++
net/sunrpc/xprt.c | 5 +++++
net/sunrpc/xprtsock.c | 5 +++++
5 files changed, 22 insertions(+)
Index: linux-build.git/net/ipv4/inet_connection_sock.c
===================================================================
--- linux-build.git.orig/net/ipv4/inet_connection_sock.c 2016-06-22 11:55:05.952267493 -0400
+++ linux-build.git/net/ipv4/inet_connection_sock.c 2016-06-22 11:56:20.002662092 -0400
@@ -232,6 +232,10 @@ tb_found:
}
}
success:
+ if (!current->mm) {
+ trace_printk("snum %d\n", snum);
+ trace_dump_stack(1);
+ }
if (!inet_csk(sk)->icsk_bind_hash)
inet_bind_hash(sk, tb, port);
WARN_ON(inet_csk(sk)->icsk_bind_hash != tb);
Index: linux-build.git/net/ipv4/inet_hashtables.c
===================================================================
--- linux-build.git.orig/net/ipv4/inet_hashtables.c 2016-06-22 11:55:05.952267493 -0400
+++ linux-build.git/net/ipv4/inet_hashtables.c 2016-06-22 11:55:05.948267360 -0400
@@ -93,6 +93,11 @@ void inet_bind_bucket_destroy(struct kme
void inet_bind_hash(struct sock *sk, struct inet_bind_bucket *tb,
const unsigned short snum)
{
+ if (!current->mm) {
+ trace_printk("add %d\n", snum);
+ trace_dump_stack(1);
+ }
+
inet_sk(sk)->inet_num = snum;
sk_add_bind_node(sk, &tb->owners);
tb->num_owners++;
Index: linux-build.git/net/sunrpc/xprt.c
===================================================================
--- linux-build.git.orig/net/sunrpc/xprt.c 2016-06-22 11:55:05.952267493 -0400
+++ linux-build.git/net/sunrpc/xprt.c 2016-06-22 11:55:05.948267360 -0400
@@ -54,6 +54,11 @@
#include "sunrpc.h"
+#undef dprintk
+#undef dprintk_rcu
+#define dprintk(args...) trace_printk(args)
+#define dprintk_rcu(args...) trace_printk(args)
+
/*
* Local variables
*/
Index: linux-build.git/net/sunrpc/xprtsock.c
===================================================================
--- linux-build.git.orig/net/sunrpc/xprtsock.c 2016-06-22 11:55:05.952267493 -0400
+++ linux-build.git/net/sunrpc/xprtsock.c 2016-06-22 11:55:05.948267360 -0400
@@ -51,6 +51,11 @@
#include "sunrpc.h"
+#undef dprintk
+#undef dprintk_rcu
+#define dprintk(args...) trace_printk(args)
+#define dprintk_rcu(args...) trace_printk(args)
+
static void xs_close(struct rpc_xprt *xprt);
/*
^ permalink raw reply [flat|nested] 15+ messages in thread* It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) @ 2016-06-30 12:59 ` Steven Rostedt 0 siblings, 0 replies; 15+ messages in thread From: Steven Rostedt @ 2016-06-30 12:59 UTC (permalink / raw) To: Trond Myklebust Cc: Jeff Layton, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Bruce James Fields [-- Attachment #1: Type: text/plain, Size: 4137 bytes --] [ resending as a new email, as I'm assuming people do not sort their INBOX via last email on thread, thus my last email is sitting in the bottom of everyone's INBOX ] I've hit this again. Not sure when it started, but I applied my old debug trace_printk() patch (attached) and rebooted (4.5.7). I just tested the latest kernel from Linus's tree (from last nights pull), and it still gives me the problem. Here's the trace I have: kworker/3:1H-134 [003] ..s. 61.036129: inet_csk_get_port: snum 805 kworker/3:1H-134 [003] ..s. 61.036135: <stack trace> => sched_clock => inet_addr_type_table => security_capable => inet_bind => xs_bind => release_sock => sock_setsockopt => __sock_create => xs_create_sock.isra.19 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => ret_from_fork => kthread kworker/3:1H-134 [003] ..s. 61.036136: inet_bind_hash: add 805 kworker/3:1H-134 [003] ..s. 61.036138: <stack trace> => inet_csk_get_port => sched_clock => inet_addr_type_table => security_capable => inet_bind => xs_bind => release_sock => sock_setsockopt => __sock_create => xs_create_sock.isra.19 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => ret_from_fork => kthread kworker/3:1H-134 [003] .... 61.036139: xs_bind: RPC: xs_bind 4.136.255.255:805: ok (0) kworker/3:1H-134 [003] .... 61.036140: xs_tcp_setup_socket: RPC: worker connecting xprt ffff880407eca800 via tcp to 192.168.23.22 (port 43651) kworker/3:1H-134 [003] .... 61.036162: xs_tcp_setup_socket: RPC: ffff880407eca800 connect status 115 connected 0 sock state 2 <idle>-0 [001] ..s. 61.036450: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff880407eca800... <idle>-0 [001] ..s. 61.036452: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 kworker/1:1H-136 [001] .... 61.036476: xprt_connect_status: RPC: 43 xprt_connect_status: retrying kworker/1:1H-136 [001] .... 61.036478: xprt_prepare_transmit: RPC: 43 xprt_prepare_transmit kworker/1:1H-136 [001] .... 61.036479: xprt_transmit: RPC: 43 xprt_transmit(72) kworker/1:1H-136 [001] .... 61.036486: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 kworker/1:1H-136 [001] .... 61.036487: xprt_transmit: RPC: 43 xmit complete <idle>-0 [001] ..s. 61.036789: xs_tcp_data_ready: RPC: xs_tcp_data_ready... kworker/1:1H-136 [001] .... 61.036798: xs_tcp_data_recv: RPC: xs_tcp_data_recv started kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading XID (4 bytes) kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading request with XID 2f4c3f88 kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) kworker/1:1H-136 [001] .... 61.036801: xs_tcp_data_recv: RPC: read reply XID 2f4c3f88 kworker/1:1H-136 [001] ..s. 61.036801: xs_tcp_data_recv: RPC: XID 2f4c3f88 read 16 bytes kworker/1:1H-136 [001] ..s. 61.036802: xs_tcp_data_recv: RPC: xprt = ffff880407eca800, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 kworker/1:1H-136 [001] ..s. 61.036802: xprt_complete_rqst: RPC: 43 xid 2f4c3f88 complete (24 bytes received) kworker/1:1H-136 [001] .... 61.036803: xs_tcp_data_recv: RPC: xs_tcp_data_recv done kworker/1:1H-136 [001] .... 61.036812: xprt_release: RPC: 43 release request ffff88040b270800 # unhide-tcp Unhide-tcp 20130526 Copyright © 2013 Yago Jesus & Patrick Gouin License GPLv3+ : GNU GPL version 3 or later http://www.unhide-forensics.info Used options: [*]Starting TCP checking Found Hidden port that not appears in ss: 805 -- Steve [-- Attachment #2: debug-hidden-port-4.7.patch --] [-- Type: text/x-patch, Size: 2378 bytes --] --- net/ipv4/inet_connection_sock.c | 4 ++++ net/ipv4/inet_hashtables.c | 5 +++++ net/sunrpc/xprt.c | 5 +++++ net/sunrpc/xprtsock.c | 5 +++++ 5 files changed, 22 insertions(+) Index: linux-build.git/net/ipv4/inet_connection_sock.c =================================================================== --- linux-build.git.orig/net/ipv4/inet_connection_sock.c 2016-06-22 11:55:05.952267493 -0400 +++ linux-build.git/net/ipv4/inet_connection_sock.c 2016-06-22 11:56:20.002662092 -0400 @@ -232,6 +232,10 @@ tb_found: } } success: + if (!current->mm) { + trace_printk("snum %d\n", snum); + trace_dump_stack(1); + } if (!inet_csk(sk)->icsk_bind_hash) inet_bind_hash(sk, tb, port); WARN_ON(inet_csk(sk)->icsk_bind_hash != tb); Index: linux-build.git/net/ipv4/inet_hashtables.c =================================================================== --- linux-build.git.orig/net/ipv4/inet_hashtables.c 2016-06-22 11:55:05.952267493 -0400 +++ linux-build.git/net/ipv4/inet_hashtables.c 2016-06-22 11:55:05.948267360 -0400 @@ -93,6 +93,11 @@ void inet_bind_bucket_destroy(struct kme void inet_bind_hash(struct sock *sk, struct inet_bind_bucket *tb, const unsigned short snum) { + if (!current->mm) { + trace_printk("add %d\n", snum); + trace_dump_stack(1); + } + inet_sk(sk)->inet_num = snum; sk_add_bind_node(sk, &tb->owners); tb->num_owners++; Index: linux-build.git/net/sunrpc/xprt.c =================================================================== --- linux-build.git.orig/net/sunrpc/xprt.c 2016-06-22 11:55:05.952267493 -0400 +++ linux-build.git/net/sunrpc/xprt.c 2016-06-22 11:55:05.948267360 -0400 @@ -54,6 +54,11 @@ #include "sunrpc.h" +#undef dprintk +#undef dprintk_rcu +#define dprintk(args...) trace_printk(args) +#define dprintk_rcu(args...) trace_printk(args) + /* * Local variables */ Index: linux-build.git/net/sunrpc/xprtsock.c =================================================================== --- linux-build.git.orig/net/sunrpc/xprtsock.c 2016-06-22 11:55:05.952267493 -0400 +++ linux-build.git/net/sunrpc/xprtsock.c 2016-06-22 11:55:05.948267360 -0400 @@ -51,6 +51,11 @@ #include "sunrpc.h" +#undef dprintk +#undef dprintk_rcu +#define dprintk(args...) trace_printk(args) +#define dprintk_rcu(args...) trace_printk(args) + static void xs_close(struct rpc_xprt *xprt); /* ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) 2016-06-30 12:59 ` Steven Rostedt (?) @ 2016-06-30 13:17 ` Trond Myklebust 2016-06-30 15:23 ` Steven Rostedt -1 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2016-06-30 13:17 UTC (permalink / raw) To: Rostedt Steven Cc: Jeff Layton, Eric Dumazet, Schumaker Anna, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Fields Bruce > On Jun 30, 2016, at 08:59, Steven Rostedt <rostedt@goodmis.org> wrote: > > [ resending as a new email, as I'm assuming people do not sort their > INBOX via last email on thread, thus my last email is sitting in the > bottom of everyone's INBOX ] > > I've hit this again. Not sure when it started, but I applied my old > debug trace_printk() patch (attached) and rebooted (4.5.7). I just > tested the latest kernel from Linus's tree (from last nights pull), and > it still gives me the problem. > > Here's the trace I have: > > kworker/3:1H-134 [003] ..s. 61.036129: inet_csk_get_port: snum 805 > kworker/3:1H-134 [003] ..s. 61.036135: <stack trace> > => sched_clock > => inet_addr_type_table > => security_capable > => inet_bind > => xs_bind > => release_sock > => sock_setsockopt > => __sock_create > => xs_create_sock.isra.19 > => xs_tcp_setup_socket > => process_one_work > => worker_thread > => worker_thread > => kthread > => ret_from_fork > => kthread > kworker/3:1H-134 [003] ..s. 61.036136: inet_bind_hash: add 805 > kworker/3:1H-134 [003] ..s. 61.036138: <stack trace> > => inet_csk_get_port > => sched_clock > => inet_addr_type_table > => security_capable > => inet_bind > => xs_bind > => release_sock > => sock_setsockopt > => __sock_create > => xs_create_sock.isra.19 > => xs_tcp_setup_socket > => process_one_work > => worker_thread > => worker_thread > => kthread > => ret_from_fork > => kthread > kworker/3:1H-134 [003] .... 61.036139: xs_bind: RPC: xs_bind 4.136.255.255:805: ok (0) > kworker/3:1H-134 [003] .... 61.036140: xs_tcp_setup_socket: RPC: worker connecting xprt ffff880407eca800 via tcp to 192.168.23.22 (port 43651) > kworker/3:1H-134 [003] .... 61.036162: xs_tcp_setup_socket: RPC: ffff880407eca800 connect status 115 connected 0 sock state 2 > <idle>-0 [001] ..s. 61.036450: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff880407eca800... > <idle>-0 [001] ..s. 61.036452: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 > kworker/1:1H-136 [001] .... 61.036476: xprt_connect_status: RPC: 43 xprt_connect_status: retrying > kworker/1:1H-136 [001] .... 61.036478: xprt_prepare_transmit: RPC: 43 xprt_prepare_transmit > kworker/1:1H-136 [001] .... 61.036479: xprt_transmit: RPC: 43 xprt_transmit(72) > kworker/1:1H-136 [001] .... 61.036486: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 > kworker/1:1H-136 [001] .... 61.036487: xprt_transmit: RPC: 43 xmit complete > <idle>-0 [001] ..s. 61.036789: xs_tcp_data_ready: RPC: xs_tcp_data_ready... > kworker/1:1H-136 [001] .... 61.036798: xs_tcp_data_recv: RPC: xs_tcp_data_recv started > kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 > kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading XID (4 bytes) > kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading request with XID 2f4c3f88 > kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) > kworker/1:1H-136 [001] .... 61.036801: xs_tcp_data_recv: RPC: read reply XID 2f4c3f88 > kworker/1:1H-136 [001] ..s. 61.036801: xs_tcp_data_recv: RPC: XID 2f4c3f88 read 16 bytes > kworker/1:1H-136 [001] ..s. 61.036802: xs_tcp_data_recv: RPC: xprt = ffff880407eca800, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 > kworker/1:1H-136 [001] ..s. 61.036802: xprt_complete_rqst: RPC: 43 xid 2f4c3f88 complete (24 bytes received) > kworker/1:1H-136 [001] .... 61.036803: xs_tcp_data_recv: RPC: xs_tcp_data_recv done > kworker/1:1H-136 [001] .... 61.036812: xprt_release: RPC: 43 release request ffff88040b270800 > > > # unhide-tcp > Unhide-tcp 20130526 > Copyright © 2013 Yago Jesus & Patrick Gouin > License GPLv3+ : GNU GPL version 3 or later > http://www.unhide-forensics.info > Used options: > [*]Starting TCP checking > > Found Hidden port that not appears in ss: 805 > What is a “Hidden port that not appears in ss: 805”, and what does this report mean? Are we failing to close a socket? Cheers Trond ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) 2016-06-30 13:17 ` Trond Myklebust @ 2016-06-30 15:23 ` Steven Rostedt 2016-06-30 16:24 ` Steven Rostedt 2016-06-30 18:30 ` Trond Myklebust 0 siblings, 2 replies; 15+ messages in thread From: Steven Rostedt @ 2016-06-30 15:23 UTC (permalink / raw) To: Trond Myklebust Cc: Jeff Layton, Eric Dumazet, Schumaker Anna, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Fields Bruce On Thu, 30 Jun 2016 13:17:47 +0000 Trond Myklebust <trondmy@primarydata.com> wrote: > > On Jun 30, 2016, at 08:59, Steven Rostedt <rostedt@goodmis.org> wrote: > > > > [ resending as a new email, as I'm assuming people do not sort their > > INBOX via last email on thread, thus my last email is sitting in the > > bottom of everyone's INBOX ] > > > > I've hit this again. Not sure when it started, but I applied my old > > debug trace_printk() patch (attached) and rebooted (4.5.7). I just > > tested the latest kernel from Linus's tree (from last nights pull), and > > it still gives me the problem. > > > > Here's the trace I have: > > > > kworker/3:1H-134 [003] ..s. 61.036129: inet_csk_get_port: snum 805 Here's were the port is taken > > kworker/3:1H-134 [003] ..s. 61.036135: <stack trace> > > => sched_clock > > => inet_addr_type_table > > => security_capable > > => inet_bind > > => xs_bind > > => release_sock > > => sock_setsockopt > > => __sock_create > > => xs_create_sock.isra.19 > > => xs_tcp_setup_socket > > => process_one_work > > => worker_thread > > => worker_thread > > => kthread > > => ret_from_fork > > => kthread > > kworker/3:1H-134 [003] ..s. 61.036136: inet_bind_hash: add 805 > > kworker/3:1H-134 [003] ..s. 61.036138: <stack trace> > > => inet_csk_get_port > > => sched_clock > > => inet_addr_type_table > > => security_capable > > => inet_bind > > => xs_bind > > => release_sock > > => sock_setsockopt > > => __sock_create > > => xs_create_sock.isra.19 > > => xs_tcp_setup_socket > > => process_one_work > > => worker_thread > > => worker_thread > > => kthread > > => ret_from_fork > > => kthread > > kworker/3:1H-134 [003] .... 61.036139: xs_bind: RPC: xs_bind 4.136.255.255:805: ok (0) Here's where it is bounded. > > kworker/3:1H-134 [003] .... 61.036140: xs_tcp_setup_socket: RPC: worker connecting xprt ffff880407eca800 via tcp to 192.168.23.22 (port 43651) > > kworker/3:1H-134 [003] .... 61.036162: xs_tcp_setup_socket: RPC: ffff880407eca800 connect status 115 connected 0 sock state 2 > > <idle>-0 [001] ..s. 61.036450: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff880407eca800... > > <idle>-0 [001] ..s. 61.036452: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 > > kworker/1:1H-136 [001] .... 61.036476: xprt_connect_status: RPC: 43 xprt_connect_status: retrying > > kworker/1:1H-136 [001] .... 61.036478: xprt_prepare_transmit: RPC: 43 xprt_prepare_transmit > > kworker/1:1H-136 [001] .... 61.036479: xprt_transmit: RPC: 43 xprt_transmit(72) > > kworker/1:1H-136 [001] .... 61.036486: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 > > kworker/1:1H-136 [001] .... 61.036487: xprt_transmit: RPC: 43 xmit complete > > <idle>-0 [001] ..s. 61.036789: xs_tcp_data_ready: RPC: xs_tcp_data_ready... > > kworker/1:1H-136 [001] .... 61.036798: xs_tcp_data_recv: RPC: xs_tcp_data_recv started > > kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 > > kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading XID (4 bytes) > > kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading request with XID 2f4c3f88 > > kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) > > kworker/1:1H-136 [001] .... 61.036801: xs_tcp_data_recv: RPC: read reply XID 2f4c3f88 > > kworker/1:1H-136 [001] ..s. 61.036801: xs_tcp_data_recv: RPC: XID 2f4c3f88 read 16 bytes > > kworker/1:1H-136 [001] ..s. 61.036802: xs_tcp_data_recv: RPC: xprt = ffff880407eca800, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 > > kworker/1:1H-136 [001] ..s. 61.036802: xprt_complete_rqst: RPC: 43 xid 2f4c3f88 complete (24 bytes received) > > kworker/1:1H-136 [001] .... 61.036803: xs_tcp_data_recv: RPC: xs_tcp_data_recv done > > kworker/1:1H-136 [001] .... 61.036812: xprt_release: RPC: 43 release request ffff88040b270800 > > > > > > # unhide-tcp > > Unhide-tcp 20130526 > > Copyright © 2013 Yago Jesus & Patrick Gouin > > License GPLv3+ : GNU GPL version 3 or later > > http://www.unhide-forensics.info > > Used options: > > [*]Starting TCP checking > > > > Found Hidden port that not appears in ss: 805 > > > > What is a “Hidden port that not appears in ss: 805”, and what does this report mean? Are we failing to close a socket? I believe hidden ports are ports that are bound to no socket. Basically, a "port leak". Where they are in limbo and can never be reused. I looked at my past report, and everthing is exactly like the issue before. When I first boot my box, the port is there, I have the above trace. I run netstat -tapn and grep for the port. And it shows that it is an established socket between my box and my wife's box (I have a nfs mounted file system for her to copy her pictures to my server). After a couple of minutes, the port turns from ESTABLISHED to TIME_WAIT, and after another minute it disappears. At that moment, the unhide-tcp shows the port as hidden. When the socket goes away (without releasing the port) I see this in my trace: kworker/1:1H-131 [001] ..s. 364.762537: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040ad68800... kworker/1:1H-131 [001] ..s. 364.762539: xs_tcp_state_change: RPC: state 4 conn 1 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [001] ..s. 364.762715: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040ad68800... <idle>-0 [001] ..s. 364.762716: xs_tcp_state_change: RPC: state 5 conn 0 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [001] ..s. 364.762728: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040ad68800... <idle>-0 [001] ..s. 364.762728: xs_tcp_state_change: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [001] ..s. 364.762729: xprt_disconnect_done: RPC: disconnected transport ffff88040ad68800 <idle>-0 [001] ..s. 364.762730: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040ad68800... <idle>-0 [001] ..s. 364.762730: xs_tcp_state_change: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [001] ..s. 364.762730: xprt_disconnect_done: RPC: disconnected transport ffff88040ad68800 I can add more trace_printk()s if it would help. -- Steve ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) 2016-06-30 15:23 ` Steven Rostedt @ 2016-06-30 16:24 ` Steven Rostedt 2016-06-30 18:30 ` Trond Myklebust 1 sibling, 0 replies; 15+ messages in thread From: Steven Rostedt @ 2016-06-30 16:24 UTC (permalink / raw) To: Trond Myklebust Cc: Jeff Layton, Eric Dumazet, Schumaker Anna, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Fields Bruce On Thu, 30 Jun 2016 11:23:41 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > I can add more trace_printk()s if it would help. I added a trace_printk() in inet_bind_bucket_destroy() to print out some information on the socket used by xs_bind(), and it shows that the bind destroy is called, but the list is not empty. /* * Caller must hold hashbucket lock for this tb with local BH disabled */ void inet_bind_bucket_destroy(struct kmem_cache *cachep, struct inet_bind_bucket *tb) { if (!current->mm && xs_port == tb->port) { trace_printk("destroy %d empty=%d %p\n", tb->port, hlist_empty(&tb->owners), tb); trace_dump_stack(1); } if (hlist_empty(&tb->owners)) { __hlist_del(&tb->node); kmem_cache_free(cachep, tb); } } I created "xs_port" to hold the port of the variable used by xs_bind, and when it is called, the hlist_empty(&tb->owners) returns false. I'll add more trace_printks to find out where those owners are being added. -- Steve ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) 2016-06-30 15:23 ` Steven Rostedt @ 2016-06-30 18:30 ` Trond Myklebust 2016-06-30 18:30 ` Trond Myklebust 1 sibling, 0 replies; 15+ messages in thread From: Trond Myklebust @ 2016-06-30 18:30 UTC (permalink / raw) To: Rostedt Steven Cc: Jeff Layton, Eric Dumazet, Schumaker Anna, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Fields Bruce > On Jun 30, 2016, at 11:23, Steven Rostedt <rostedt@goodmis.org> wrote: > > On Thu, 30 Jun 2016 13:17:47 +0000 > Trond Myklebust <trondmy@primarydata.com> wrote: > >>> On Jun 30, 2016, at 08:59, Steven Rostedt <rostedt@goodmis.org> wrote: >>> >>> [ resending as a new email, as I'm assuming people do not sort their >>> INBOX via last email on thread, thus my last email is sitting in the >>> bottom of everyone's INBOX ] >>> >>> I've hit this again. Not sure when it started, but I applied my old >>> debug trace_printk() patch (attached) and rebooted (4.5.7). I just >>> tested the latest kernel from Linus's tree (from last nights pull), and >>> it still gives me the problem. >>> >>> Here's the trace I have: >>> >>> kworker/3:1H-134 [003] ..s. 61.036129: inet_csk_get_port: snum 805 > > Here's were the port is taken > >>> kworker/3:1H-134 [003] ..s. 61.036135: <stack trace> >>> => sched_clock >>> => inet_addr_type_table >>> => security_capable >>> => inet_bind >>> => xs_bind >>> => release_sock >>> => sock_setsockopt >>> => __sock_create >>> => xs_create_sock.isra.19 >>> => xs_tcp_setup_socket >>> => process_one_work >>> => worker_thread >>> => worker_thread >>> => kthread >>> => ret_from_fork >>> => kthread >>> kworker/3:1H-134 [003] ..s. 61.036136: inet_bind_hash: add 805 >>> kworker/3:1H-134 [003] ..s. 61.036138: <stack trace> >>> => inet_csk_get_port >>> => sched_clock >>> => inet_addr_type_table >>> => security_capable >>> => inet_bind >>> => xs_bind >>> => release_sock >>> => sock_setsockopt >>> => __sock_create >>> => xs_create_sock.isra.19 >>> => xs_tcp_setup_socket >>> => process_one_work >>> => worker_thread >>> => worker_thread >>> => kthread >>> => ret_from_fork >>> => kthread >>> kworker/3:1H-134 [003] .... 61.036139: xs_bind: RPC: xs_bind 4.136.255.255:805: ok (0) > > Here's where it is bounded. > >>> kworker/3:1H-134 [003] .... 61.036140: xs_tcp_setup_socket: RPC: worker connecting xprt ffff880407eca800 via tcp to 192.168.23.22 (port 43651) >>> kworker/3:1H-134 [003] .... 61.036162: xs_tcp_setup_socket: RPC: ffff880407eca800 connect status 115 connected 0 sock state 2 >>> <idle>-0 [001] ..s. 61.036450: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff880407eca800... >>> <idle>-0 [001] ..s. 61.036452: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 >>> kworker/1:1H-136 [001] .... 61.036476: xprt_connect_status: RPC: 43 xprt_connect_status: retrying >>> kworker/1:1H-136 [001] .... 61.036478: xprt_prepare_transmit: RPC: 43 xprt_prepare_transmit >>> kworker/1:1H-136 [001] .... 61.036479: xprt_transmit: RPC: 43 xprt_transmit(72) >>> kworker/1:1H-136 [001] .... 61.036486: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 >>> kworker/1:1H-136 [001] .... 61.036487: xprt_transmit: RPC: 43 xmit complete >>> <idle>-0 [001] ..s. 61.036789: xs_tcp_data_ready: RPC: xs_tcp_data_ready... >>> kworker/1:1H-136 [001] .... 61.036798: xs_tcp_data_recv: RPC: xs_tcp_data_recv started >>> kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 >>> kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading XID (4 bytes) >>> kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading request with XID 2f4c3f88 >>> kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) >>> kworker/1:1H-136 [001] .... 61.036801: xs_tcp_data_recv: RPC: read reply XID 2f4c3f88 >>> kworker/1:1H-136 [001] ..s. 61.036801: xs_tcp_data_recv: RPC: XID 2f4c3f88 read 16 bytes >>> kworker/1:1H-136 [001] ..s. 61.036802: xs_tcp_data_recv: RPC: xprt = ffff880407eca800, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 >>> kworker/1:1H-136 [001] ..s. 61.036802: xprt_complete_rqst: RPC: 43 xid 2f4c3f88 complete (24 bytes received) >>> kworker/1:1H-136 [001] .... 61.036803: xs_tcp_data_recv: RPC: xs_tcp_data_recv done >>> kworker/1:1H-136 [001] .... 61.036812: xprt_release: RPC: 43 release request ffff88040b270800 >>> >>> >>> # unhide-tcp >>> Unhide-tcp 20130526 >>> Copyright © 2013 Yago Jesus & Patrick Gouin >>> License GPLv3+ : GNU GPL version 3 or later >>> http://www.unhide-forensics.info >>> Used options: >>> [*]Starting TCP checking >>> >>> Found Hidden port that not appears in ss: 805 >>> >> >> What is a “Hidden port that not appears in ss: 805”, and what does this report mean? Are we failing to close a socket? > > I believe hidden ports are ports that are bound to no socket. > Basically, a "port leak". Where they are in limbo and can never be > reused. > > I looked at my past report, and everthing is exactly like the issue > before. When I first boot my box, the port is there, I have the above > trace. I run netstat -tapn and grep for the port. And it shows that it > is an established socket between my box and my wife's box (I have a nfs > mounted file system for her to copy her pictures to my server). After a > couple of minutes, the port turns from ESTABLISHED to TIME_WAIT, and > after another minute it disappears. At that moment, the unhide-tcp > shows the port as hidden. > > When the socket goes away (without releasing the port) I see this in my > trace: > > kworker/1:1H-131 [001] ..s. 364.762537: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040ad68800... > kworker/1:1H-131 [001] ..s. 364.762539: xs_tcp_state_change: RPC: state 4 conn 1 dead 0 zapped 1 sk_shutdown 3 > <idle>-0 [001] ..s. 364.762715: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040ad68800... > <idle>-0 [001] ..s. 364.762716: xs_tcp_state_change: RPC: state 5 conn 0 dead 0 zapped 1 sk_shutdown 3 > <idle>-0 [001] ..s. 364.762728: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040ad68800... > <idle>-0 [001] ..s. 364.762728: xs_tcp_state_change: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 > <idle>-0 [001] ..s. 364.762729: xprt_disconnect_done: RPC: disconnected transport ffff88040ad68800 > <idle>-0 [001] ..s. 364.762730: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040ad68800... > <idle>-0 [001] ..s. 364.762730: xs_tcp_state_change: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 > <idle>-0 [001] ..s. 364.762730: xprt_disconnect_done: RPC: disconnected transport ffff88040ad68800 > > I can add more trace_printk()s if it would help. Wait. So the NFS mount is still active, it’s just that the socket disconnected due to no traffic? That should be OK. Granted that the port can’t be reused by another process, but you really don’t want that: what if there are no other ports available and you start writing to a file on the NFS partition? Cheers Trond ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) @ 2016-06-30 18:30 ` Trond Myklebust 0 siblings, 0 replies; 15+ messages in thread From: Trond Myklebust @ 2016-06-30 18:30 UTC (permalink / raw) To: Rostedt Steven Cc: Jeff Layton, Eric Dumazet, Schumaker Anna, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Fields Bruce DQo+IE9uIEp1biAzMCwgMjAxNiwgYXQgMTE6MjMsIFN0ZXZlbiBSb3N0ZWR0IDxyb3N0ZWR0QGdv b2RtaXMub3JnPiB3cm90ZToNCj4gDQo+IE9uIFRodSwgMzAgSnVuIDIwMTYgMTM6MTc6NDcgKzAw MDANCj4gVHJvbmQgTXlrbGVidXN0IDx0cm9uZG15QHByaW1hcnlkYXRhLmNvbT4gd3JvdGU6DQo+ IA0KPj4+IE9uIEp1biAzMCwgMjAxNiwgYXQgMDg6NTksIFN0ZXZlbiBSb3N0ZWR0IDxyb3N0ZWR0 QGdvb2RtaXMub3JnPiB3cm90ZToNCj4+PiANCj4+PiBbIHJlc2VuZGluZyBhcyBhIG5ldyBlbWFp bCwgYXMgSSdtIGFzc3VtaW5nIHBlb3BsZSBkbyBub3Qgc29ydCB0aGVpcg0KPj4+IElOQk9YIHZp YSBsYXN0IGVtYWlsIG9uIHRocmVhZCwgdGh1cyBteSBsYXN0IGVtYWlsIGlzIHNpdHRpbmcgaW4g dGhlDQo+Pj4gYm90dG9tIG9mIGV2ZXJ5b25lJ3MgSU5CT1ggXQ0KPj4+IA0KPj4+IEkndmUgaGl0 IHRoaXMgYWdhaW4uIE5vdCBzdXJlIHdoZW4gaXQgc3RhcnRlZCwgYnV0IEkgYXBwbGllZCBteSBv bGQNCj4+PiBkZWJ1ZyB0cmFjZV9wcmludGsoKSBwYXRjaCAoYXR0YWNoZWQpIGFuZCByZWJvb3Rl ZCAoNC41LjcpLiBJIGp1c3QNCj4+PiB0ZXN0ZWQgdGhlIGxhdGVzdCBrZXJuZWwgZnJvbSBMaW51 cydzIHRyZWUgKGZyb20gbGFzdCBuaWdodHMgcHVsbCksIGFuZA0KPj4+IGl0IHN0aWxsIGdpdmVz IG1lIHRoZSBwcm9ibGVtLg0KPj4+IA0KPj4+IEhlcmUncyB0aGUgdHJhY2UgSSBoYXZlOg0KPj4+ IA0KPj4+ICAga3dvcmtlci8zOjFILTEzNCAgIFswMDNdIC4ucy4gICAgNjEuMDM2MTI5OiBpbmV0 X2Nza19nZXRfcG9ydDogc251bSA4MDUNCj4gDQo+IEhlcmUncyB3ZXJlIHRoZSBwb3J0IGlzIHRh a2VuDQo+IA0KPj4+ICAga3dvcmtlci8zOjFILTEzNCAgIFswMDNdIC4ucy4gICAgNjEuMDM2MTM1 OiA8c3RhY2sgdHJhY2U+ICANCj4+PiA9PiBzY2hlZF9jbG9jaw0KPj4+ID0+IGluZXRfYWRkcl90 eXBlX3RhYmxlDQo+Pj4gPT4gc2VjdXJpdHlfY2FwYWJsZQ0KPj4+ID0+IGluZXRfYmluZA0KPj4+ ID0+IHhzX2JpbmQNCj4+PiA9PiByZWxlYXNlX3NvY2sNCj4+PiA9PiBzb2NrX3NldHNvY2tvcHQN Cj4+PiA9PiBfX3NvY2tfY3JlYXRlDQo+Pj4gPT4geHNfY3JlYXRlX3NvY2suaXNyYS4xOQ0KPj4+ ID0+IHhzX3RjcF9zZXR1cF9zb2NrZXQNCj4+PiA9PiBwcm9jZXNzX29uZV93b3JrDQo+Pj4gPT4g d29ya2VyX3RocmVhZA0KPj4+ID0+IHdvcmtlcl90aHJlYWQNCj4+PiA9PiBrdGhyZWFkDQo+Pj4g PT4gcmV0X2Zyb21fZm9yaw0KPj4+ID0+IGt0aHJlYWQgICAgDQo+Pj4gICBrd29ya2VyLzM6MUgt MTM0ICAgWzAwM10gLi5zLiAgICA2MS4wMzYxMzY6IGluZXRfYmluZF9oYXNoOiBhZGQgODA1DQo+ Pj4gICBrd29ya2VyLzM6MUgtMTM0ICAgWzAwM10gLi5zLiAgICA2MS4wMzYxMzg6IDxzdGFjayB0 cmFjZT4gIA0KPj4+ID0+IGluZXRfY3NrX2dldF9wb3J0DQo+Pj4gPT4gc2NoZWRfY2xvY2sNCj4+ PiA9PiBpbmV0X2FkZHJfdHlwZV90YWJsZQ0KPj4+ID0+IHNlY3VyaXR5X2NhcGFibGUNCj4+PiA9 PiBpbmV0X2JpbmQNCj4+PiA9PiB4c19iaW5kDQo+Pj4gPT4gcmVsZWFzZV9zb2NrDQo+Pj4gPT4g c29ja19zZXRzb2Nrb3B0DQo+Pj4gPT4gX19zb2NrX2NyZWF0ZQ0KPj4+ID0+IHhzX2NyZWF0ZV9z b2NrLmlzcmEuMTkNCj4+PiA9PiB4c190Y3Bfc2V0dXBfc29ja2V0DQo+Pj4gPT4gcHJvY2Vzc19v bmVfd29yaw0KPj4+ID0+IHdvcmtlcl90aHJlYWQNCj4+PiA9PiB3b3JrZXJfdGhyZWFkDQo+Pj4g PT4ga3RocmVhZA0KPj4+ID0+IHJldF9mcm9tX2ZvcmsNCj4+PiA9PiBrdGhyZWFkICAgIA0KPj4+ ICAga3dvcmtlci8zOjFILTEzNCAgIFswMDNdIC4uLi4gICAgNjEuMDM2MTM5OiB4c19iaW5kOiBS UEM6ICAgICAgIHhzX2JpbmQgNC4xMzYuMjU1LjI1NTo4MDU6IG9rICgwKQ0KPiANCj4gSGVyZSdz IHdoZXJlIGl0IGlzIGJvdW5kZWQuDQo+IA0KPj4+ICAga3dvcmtlci8zOjFILTEzNCAgIFswMDNd IC4uLi4gICAgNjEuMDM2MTQwOiB4c190Y3Bfc2V0dXBfc29ja2V0OiBSUEM6ICAgICAgIHdvcmtl ciBjb25uZWN0aW5nIHhwcnQgZmZmZjg4MDQwN2VjYTgwMCB2aWEgdGNwIHRvIDE5Mi4xNjguMjMu MjIgKHBvcnQgNDM2NTEpDQo+Pj4gICBrd29ya2VyLzM6MUgtMTM0ICAgWzAwM10gLi4uLiAgICA2 MS4wMzYxNjI6IHhzX3RjcF9zZXR1cF9zb2NrZXQ6IFJQQzogICAgICAgZmZmZjg4MDQwN2VjYTgw MCBjb25uZWN0IHN0YXR1cyAxMTUgY29ubmVjdGVkIDAgc29jayBzdGF0ZSAyDQo+Pj4gICAgICAg ICA8aWRsZT4tMCAgICAgWzAwMV0gLi5zLiAgICA2MS4wMzY0NTA6IHhzX3RjcF9zdGF0ZV9jaGFu Z2U6IFJQQzogICAgICAgeHNfdGNwX3N0YXRlX2NoYW5nZSBjbGllbnQgZmZmZjg4MDQwN2VjYTgw MC4uLg0KPj4+ICAgICAgICAgPGlkbGU+LTAgICAgIFswMDFdIC4ucy4gICAgNjEuMDM2NDUyOiB4 c190Y3Bfc3RhdGVfY2hhbmdlOiBSUEM6ICAgICAgIHN0YXRlIDEgY29ubiAwIGRlYWQgMCB6YXBw ZWQgMSBza19zaHV0ZG93biAwDQo+Pj4gICBrd29ya2VyLzE6MUgtMTM2ICAgWzAwMV0gLi4uLiAg ICA2MS4wMzY0NzY6IHhwcnRfY29ubmVjdF9zdGF0dXM6IFJQQzogICAgNDMgeHBydF9jb25uZWN0 X3N0YXR1czogcmV0cnlpbmcNCj4+PiAgIGt3b3JrZXIvMToxSC0xMzYgICBbMDAxXSAuLi4uICAg IDYxLjAzNjQ3ODogeHBydF9wcmVwYXJlX3RyYW5zbWl0OiBSUEM6ICAgIDQzIHhwcnRfcHJlcGFy ZV90cmFuc21pdA0KPj4+ICAga3dvcmtlci8xOjFILTEzNiAgIFswMDFdIC4uLi4gICAgNjEuMDM2 NDc5OiB4cHJ0X3RyYW5zbWl0OiBSUEM6ICAgIDQzIHhwcnRfdHJhbnNtaXQoNzIpDQo+Pj4gICBr d29ya2VyLzE6MUgtMTM2ICAgWzAwMV0gLi4uLiAgICA2MS4wMzY0ODY6IHhzX3RjcF9zZW5kX3Jl cXVlc3Q6IFJQQzogICAgICAgeHNfdGNwX3NlbmRfcmVxdWVzdCg3MikgPSAwDQo+Pj4gICBrd29y a2VyLzE6MUgtMTM2ICAgWzAwMV0gLi4uLiAgICA2MS4wMzY0ODc6IHhwcnRfdHJhbnNtaXQ6IFJQ QzogICAgNDMgeG1pdCBjb21wbGV0ZQ0KPj4+ICAgICAgICAgPGlkbGU+LTAgICAgIFswMDFdIC4u cy4gICAgNjEuMDM2Nzg5OiB4c190Y3BfZGF0YV9yZWFkeTogUlBDOiAgICAgICB4c190Y3BfZGF0 YV9yZWFkeS4uLg0KPj4+ICAga3dvcmtlci8xOjFILTEzNiAgIFswMDFdIC4uLi4gICAgNjEuMDM2 Nzk4OiB4c190Y3BfZGF0YV9yZWN2OiBSUEM6ICAgICAgIHhzX3RjcF9kYXRhX3JlY3Ygc3RhcnRl ZA0KPj4+ICAga3dvcmtlci8xOjFILTEzNiAgIFswMDFdIC4uLi4gICAgNjEuMDM2Nzk5OiB4c190 Y3BfZGF0YV9yZWN2OiBSUEM6ICAgICAgIHJlYWRpbmcgVENQIHJlY29yZCBmcmFnbWVudCBvZiBs ZW5ndGggMjQNCj4+PiAgIGt3b3JrZXIvMToxSC0xMzYgICBbMDAxXSAuLi4uICAgIDYxLjAzNjc5 OTogeHNfdGNwX2RhdGFfcmVjdjogUlBDOiAgICAgICByZWFkaW5nIFhJRCAoNCBieXRlcykNCj4+ PiAgIGt3b3JrZXIvMToxSC0xMzYgICBbMDAxXSAuLi4uICAgIDYxLjAzNjgwMDogeHNfdGNwX2Rh dGFfcmVjdjogUlBDOiAgICAgICByZWFkaW5nIHJlcXVlc3Qgd2l0aCBYSUQgMmY0YzNmODgNCj4+ PiAgIGt3b3JrZXIvMToxSC0xMzYgICBbMDAxXSAuLi4uICAgIDYxLjAzNjgwMDogeHNfdGNwX2Rh dGFfcmVjdjogUlBDOiAgICAgICByZWFkaW5nIENBTEwvUkVQTFkgZmxhZyAoNCBieXRlcykNCj4+ PiAgIGt3b3JrZXIvMToxSC0xMzYgICBbMDAxXSAuLi4uICAgIDYxLjAzNjgwMTogeHNfdGNwX2Rh dGFfcmVjdjogUlBDOiAgICAgICByZWFkIHJlcGx5IFhJRCAyZjRjM2Y4OA0KPj4+ICAga3dvcmtl ci8xOjFILTEzNiAgIFswMDFdIC4ucy4gICAgNjEuMDM2ODAxOiB4c190Y3BfZGF0YV9yZWN2OiBS UEM6ICAgICAgIFhJRCAyZjRjM2Y4OCByZWFkIDE2IGJ5dGVzDQo+Pj4gICBrd29ya2VyLzE6MUgt MTM2ICAgWzAwMV0gLi5zLiAgICA2MS4wMzY4MDI6IHhzX3RjcF9kYXRhX3JlY3Y6IFJQQzogICAg ICAgeHBydCA9IGZmZmY4ODA0MDdlY2E4MDAsIHRjcF9jb3BpZWQgPSAyNCwgdGNwX29mZnNldCA9 IDI0LCB0Y3BfcmVjbGVuID0gMjQNCj4+PiAgIGt3b3JrZXIvMToxSC0xMzYgICBbMDAxXSAuLnMu ICAgIDYxLjAzNjgwMjogeHBydF9jb21wbGV0ZV9ycXN0OiBSUEM6ICAgIDQzIHhpZCAyZjRjM2Y4 OCBjb21wbGV0ZSAoMjQgYnl0ZXMgcmVjZWl2ZWQpDQo+Pj4gICBrd29ya2VyLzE6MUgtMTM2ICAg WzAwMV0gLi4uLiAgICA2MS4wMzY4MDM6IHhzX3RjcF9kYXRhX3JlY3Y6IFJQQzogICAgICAgeHNf dGNwX2RhdGFfcmVjdiBkb25lDQo+Pj4gICBrd29ya2VyLzE6MUgtMTM2ICAgWzAwMV0gLi4uLiAg ICA2MS4wMzY4MTI6IHhwcnRfcmVsZWFzZTogUlBDOiAgICA0MyByZWxlYXNlIHJlcXVlc3QgZmZm Zjg4MDQwYjI3MDgwMA0KPj4+IA0KPj4+IA0KPj4+ICMgdW5oaWRlLXRjcCANCj4+PiBVbmhpZGUt dGNwIDIwMTMwNTI2DQo+Pj4gQ29weXJpZ2h0IMKpIDIwMTMgWWFnbyBKZXN1cyAmIFBhdHJpY2sg R291aW4NCj4+PiBMaWNlbnNlIEdQTHYzKyA6IEdOVSBHUEwgdmVyc2lvbiAzIG9yIGxhdGVyDQo+ Pj4gaHR0cDovL3d3dy51bmhpZGUtZm9yZW5zaWNzLmluZm8NCj4+PiBVc2VkIG9wdGlvbnM6IA0K Pj4+IFsqXVN0YXJ0aW5nIFRDUCBjaGVja2luZw0KPj4+IA0KPj4+IEZvdW5kIEhpZGRlbiBwb3J0 IHRoYXQgbm90IGFwcGVhcnMgaW4gc3M6IDgwNQ0KPj4+IA0KPj4gDQo+PiBXaGF0IGlzIGEg4oCc SGlkZGVuIHBvcnQgdGhhdCBub3QgYXBwZWFycyBpbiBzczogODA14oCdLCBhbmQgd2hhdCBkb2Vz IHRoaXMgcmVwb3J0IG1lYW4/IEFyZSB3ZSBmYWlsaW5nIHRvIGNsb3NlIGEgc29ja2V0Pw0KPiAN Cj4gSSBiZWxpZXZlIGhpZGRlbiBwb3J0cyBhcmUgcG9ydHMgdGhhdCBhcmUgYm91bmQgdG8gbm8g c29ja2V0Lg0KPiBCYXNpY2FsbHksIGEgInBvcnQgbGVhayIuIFdoZXJlIHRoZXkgYXJlIGluIGxp bWJvIGFuZCBjYW4gbmV2ZXIgYmUNCj4gcmV1c2VkLg0KPiANCj4gSSBsb29rZWQgYXQgbXkgcGFz dCByZXBvcnQsIGFuZCBldmVydGhpbmcgaXMgZXhhY3RseSBsaWtlIHRoZSBpc3N1ZQ0KPiBiZWZv cmUuIFdoZW4gSSBmaXJzdCBib290IG15IGJveCwgdGhlIHBvcnQgaXMgdGhlcmUsIEkgaGF2ZSB0 aGUgYWJvdmUNCj4gdHJhY2UuIEkgcnVuIG5ldHN0YXQgLXRhcG4gYW5kIGdyZXAgZm9yIHRoZSBw b3J0LiBBbmQgaXQgc2hvd3MgdGhhdCBpdA0KPiBpcyBhbiBlc3RhYmxpc2hlZCBzb2NrZXQgYmV0 d2VlbiBteSBib3ggYW5kIG15IHdpZmUncyBib3ggKEkgaGF2ZSBhIG5mcw0KPiBtb3VudGVkIGZp bGUgc3lzdGVtIGZvciBoZXIgdG8gY29weSBoZXIgcGljdHVyZXMgdG8gbXkgc2VydmVyKS4gQWZ0 ZXIgYQ0KPiBjb3VwbGUgb2YgbWludXRlcywgdGhlIHBvcnQgdHVybnMgZnJvbSBFU1RBQkxJU0hF RCB0byBUSU1FX1dBSVQsIGFuZA0KPiBhZnRlciBhbm90aGVyIG1pbnV0ZSBpdCBkaXNhcHBlYXJz LiBBdCB0aGF0IG1vbWVudCwgdGhlIHVuaGlkZS10Y3ANCj4gc2hvd3MgdGhlIHBvcnQgYXMgaGlk ZGVuLg0KPiANCj4gV2hlbiB0aGUgc29ja2V0IGdvZXMgYXdheSAod2l0aG91dCByZWxlYXNpbmcg dGhlIHBvcnQpIEkgc2VlIHRoaXMgaW4gbXkNCj4gdHJhY2U6DQo+IA0KPiAgICBrd29ya2VyLzE6 MUgtMTMxICAgWzAwMV0gLi5zLiAgIDM2NC43NjI1Mzc6IHhzX3RjcF9zdGF0ZV9jaGFuZ2U6IFJQ QzogICAgICAgeHNfdGNwX3N0YXRlX2NoYW5nZSBjbGllbnQgZmZmZjg4MDQwYWQ2ODgwMC4uLg0K PiAgICBrd29ya2VyLzE6MUgtMTMxICAgWzAwMV0gLi5zLiAgIDM2NC43NjI1Mzk6IHhzX3RjcF9z dGF0ZV9jaGFuZ2U6IFJQQzogICAgICAgc3RhdGUgNCBjb25uIDEgZGVhZCAwIHphcHBlZCAxIHNr X3NodXRkb3duIDMNCj4gICAgICAgICAgPGlkbGU+LTAgICAgIFswMDFdIC4ucy4gICAzNjQuNzYy NzE1OiB4c190Y3Bfc3RhdGVfY2hhbmdlOiBSUEM6ICAgICAgIHhzX3RjcF9zdGF0ZV9jaGFuZ2Ug Y2xpZW50IGZmZmY4ODA0MGFkNjg4MDAuLi4NCj4gICAgICAgICAgPGlkbGU+LTAgICAgIFswMDFd IC4ucy4gICAzNjQuNzYyNzE2OiB4c190Y3Bfc3RhdGVfY2hhbmdlOiBSUEM6ICAgICAgIHN0YXRl IDUgY29ubiAwIGRlYWQgMCB6YXBwZWQgMSBza19zaHV0ZG93biAzDQo+ICAgICAgICAgIDxpZGxl Pi0wICAgICBbMDAxXSAuLnMuICAgMzY0Ljc2MjcyODogeHNfdGNwX3N0YXRlX2NoYW5nZTogUlBD OiAgICAgICB4c190Y3Bfc3RhdGVfY2hhbmdlIGNsaWVudCBmZmZmODgwNDBhZDY4ODAwLi4uDQo+ ICAgICAgICAgIDxpZGxlPi0wICAgICBbMDAxXSAuLnMuICAgMzY0Ljc2MjcyODogeHNfdGNwX3N0 YXRlX2NoYW5nZTogUlBDOiAgICAgICBzdGF0ZSA3IGNvbm4gMCBkZWFkIDAgemFwcGVkIDEgc2tf c2h1dGRvd24gMw0KPiAgICAgICAgICA8aWRsZT4tMCAgICAgWzAwMV0gLi5zLiAgIDM2NC43NjI3 Mjk6IHhwcnRfZGlzY29ubmVjdF9kb25lOiBSUEM6ICAgICAgIGRpc2Nvbm5lY3RlZCB0cmFuc3Bv cnQgZmZmZjg4MDQwYWQ2ODgwMA0KPiAgICAgICAgICA8aWRsZT4tMCAgICAgWzAwMV0gLi5zLiAg IDM2NC43NjI3MzA6IHhzX3RjcF9zdGF0ZV9jaGFuZ2U6IFJQQzogICAgICAgeHNfdGNwX3N0YXRl X2NoYW5nZSBjbGllbnQgZmZmZjg4MDQwYWQ2ODgwMC4uLg0KPiAgICAgICAgICA8aWRsZT4tMCAg ICAgWzAwMV0gLi5zLiAgIDM2NC43NjI3MzA6IHhzX3RjcF9zdGF0ZV9jaGFuZ2U6IFJQQzogICAg ICAgc3RhdGUgNyBjb25uIDAgZGVhZCAwIHphcHBlZCAxIHNrX3NodXRkb3duIDMNCj4gICAgICAg ICAgPGlkbGU+LTAgICAgIFswMDFdIC4ucy4gICAzNjQuNzYyNzMwOiB4cHJ0X2Rpc2Nvbm5lY3Rf ZG9uZTogUlBDOiAgICAgICBkaXNjb25uZWN0ZWQgdHJhbnNwb3J0IGZmZmY4ODA0MGFkNjg4MDAN Cj4gDQo+IEkgY2FuIGFkZCBtb3JlIHRyYWNlX3ByaW50aygpcyBpZiBpdCB3b3VsZCBoZWxwLg0K DQoNCldhaXQuIFNvIHRoZSBORlMgbW91bnQgaXMgc3RpbGwgYWN0aXZlLCBpdOKAmXMganVzdCB0 aGF0IHRoZSBzb2NrZXQgZGlzY29ubmVjdGVkIGR1ZSB0byBubyB0cmFmZmljPyBUaGF0IHNob3Vs ZCBiZSBPSy4gR3JhbnRlZCB0aGF0IHRoZSBwb3J0IGNhbuKAmXQgYmUgcmV1c2VkIGJ5IGFub3Ro ZXIgcHJvY2VzcywgYnV0IHlvdSByZWFsbHkgZG9u4oCZdCB3YW50IHRoYXQ6IHdoYXQgaWYgdGhl cmUgYXJlIG5vIG90aGVyIHBvcnRzIGF2YWlsYWJsZSBhbmQgeW91IHN0YXJ0IHdyaXRpbmcgdG8g YSBmaWxlIG9uIHRoZSBORlMgcGFydGl0aW9uPw0KDQpDaGVlcnMNCiAgVHJvbmQ= ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) @ 2016-06-30 20:07 ` Steven Rostedt 0 siblings, 0 replies; 15+ messages in thread From: Steven Rostedt @ 2016-06-30 20:07 UTC (permalink / raw) To: Trond Myklebust Cc: Jeff Layton, Eric Dumazet, Schumaker Anna, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Fields Bruce On Thu, 30 Jun 2016 18:30:42 +0000 Trond Myklebust <trondmy@primarydata.com> wrote: > Wait. So the NFS mount is still active, it’s just that the socket > disconnected due to no traffic? That should be OK. Granted that the > port can’t be reused by another process, but you really don’t want > that: what if there are no other ports available and you start > writing to a file on the NFS partition? What would cause the port to be connected to a socket again? I copied a large file to the nfs mount, and the hidden port is still there? Remember, this wasn't always the case, the hidden port is a recent issue. I ran wireshark on this and it appears to create two ports for NFS. One of them is canceled by the client (sends a FIN/ACK) and this port is what lays around never to be used again, and uses the other port for all connections after that. When I unmount the NFS directory, the port is finally freed (but has no socket attached to it). What is the purpose of keeping this port around? I can reproduce this by having the client unmount and remount the directory. -- Steve ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) @ 2016-06-30 20:07 ` Steven Rostedt 0 siblings, 0 replies; 15+ messages in thread From: Steven Rostedt @ 2016-06-30 20:07 UTC (permalink / raw) To: Trond Myklebust Cc: Jeff Layton, Eric Dumazet, Schumaker Anna, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Fields Bruce On Thu, 30 Jun 2016 18:30:42 +0000 Trond Myklebust <trondmy-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org> wrote: > Wait. So the NFS mount is still active, it’s just that the socket > disconnected due to no traffic? That should be OK. Granted that the > port can’t be reused by another process, but you really don’t want > that: what if there are no other ports available and you start > writing to a file on the NFS partition? What would cause the port to be connected to a socket again? I copied a large file to the nfs mount, and the hidden port is still there? Remember, this wasn't always the case, the hidden port is a recent issue. I ran wireshark on this and it appears to create two ports for NFS. One of them is canceled by the client (sends a FIN/ACK) and this port is what lays around never to be used again, and uses the other port for all connections after that. When I unmount the NFS directory, the port is finally freed (but has no socket attached to it). What is the purpose of keeping this port around? I can reproduce this by having the client unmount and remount the directory. -- Steve -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) 2016-06-30 20:07 ` Steven Rostedt (?) @ 2016-06-30 21:56 ` Steven Rostedt -1 siblings, 0 replies; 15+ messages in thread From: Steven Rostedt @ 2016-06-30 21:56 UTC (permalink / raw) To: Trond Myklebust Cc: Jeff Layton, Eric Dumazet, Schumaker Anna, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Fields Bruce On Thu, 30 Jun 2016 16:07:26 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > I can reproduce this by having the client unmount and remount the > directory. It gets even more interesting. When I unmount the directory, the hidden port does not go away. It is still there. But if I mount it again, it goes away (until it times out again). Even more info: When I first mount it, it creates 3 sockets, where one immediately is closed: tcp 0 0 192.168.23.9:892 192.168.23.22:44672 TIME_WAIT - tcp 0 0 192.168.23.9:2049 192.168.23.22:815 ESTABLISHED - tcp 0 0 192.168.23.9:754 192.168.23.22:44672 ESTABLISHED - (192.168.23.22 is the machine remotely mounting a directory from the server 192.168.23.9) The trace of port 892 is this: kworker/u32:1-13473 [000] .... 4093.915114: xs_setup_tcp: RPC: set up xprt to 192.168.23.22 (port 44672) via tcp kworker/u32:1-13473 [000] .... 4093.915122: xprt_create_transport: RPC: created transport ffff8803b1c38000 with 65536 slots kworker/0:1H-129 [000] .... 4093.915152: xprt_alloc_slot: RPC: 47 reserved req ffff88040b27ca00 xid c50ccaff kworker/0:1H-129 [000] .... 4093.915157: xprt_connect: RPC: 47 xprt_connect xprt ffff8803b1c38000 is not connected kworker/0:1H-129 [000] .... 4093.915159: xs_connect: RPC: xs_connect scheduled xprt ffff8803b1c38000 kworker/0:1H-129 [000] ..s. 4093.915170: inet_csk_get_port: snum 892 kworker/0:1H-129 [000] ..s. 4093.915177: <stack trace> => sched_clock => inet_addr_type_table => security_capable => inet_bind => xs_bind => release_sock => sock_setsockopt => __sock_create => xs_create_sock.isra.19 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => ret_from_fork => kthread kworker/0:1H-129 [000] ..s. 4093.915178: inet_bind_hash: add 892 ffff8803bb9b5cc0 kworker/0:1H-129 [000] ..s. 4093.915184: <stack trace> => inet_csk_get_port => sched_clock => inet_addr_type_table => security_capable => inet_bind => xs_bind => release_sock => sock_setsockopt => __sock_create => xs_create_sock.isra.19 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => ret_from_fork => kthread kworker/0:1H-129 [000] .... 4093.915185: xs_bind: RPC: xs_bind 4.136.255.255:892: ok (0) kworker/0:1H-129 [000] .... 4093.915186: xs_tcp_setup_socket: RPC: worker connecting xprt ffff8803b1c38000 via tcp to 192.168.23.22 (port 44672) kworker/0:1H-129 [000] .... 4093.915221: xs_tcp_setup_socket: RPC: ffff8803b1c38000 connect status 115 connected 0 sock state 2 <idle>-0 [003] ..s. 4093.915434: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8803b1c38000... <idle>-0 [003] ..s. 4093.915435: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 kworker/3:1H-145 [003] .... 4093.915558: xprt_connect_status: RPC: 47 xprt_connect_status: retrying kworker/3:1H-145 [003] .... 4093.915560: xprt_prepare_transmit: RPC: 47 xprt_prepare_transmit kworker/3:1H-145 [003] .... 4093.915562: xprt_transmit: RPC: 47 xprt_transmit(72) kworker/3:1H-145 [003] .... 4093.915588: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 kworker/3:1H-145 [003] .... 4093.915589: xprt_transmit: RPC: 47 xmit complete <idle>-0 [003] ..s. 4093.915969: xs_tcp_data_ready: RPC: xs_tcp_data_ready... kworker/3:1H-145 [003] .... 4093.916081: xs_tcp_data_recv: RPC: xs_tcp_data_recv started kworker/3:1H-145 [003] .... 4093.916083: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 kworker/3:1H-145 [003] .... 4093.916084: xs_tcp_data_recv: RPC: reading XID (4 bytes) kworker/3:1H-145 [003] .... 4093.916085: xs_tcp_data_recv: RPC: reading request with XID c50ccaff kworker/3:1H-145 [003] .... 4093.916086: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) kworker/3:1H-145 [003] .... 4093.916087: xs_tcp_data_recv: RPC: read reply XID c50ccaff kworker/3:1H-145 [003] ..s. 4093.916088: xs_tcp_data_recv: RPC: XID c50ccaff read 16 bytes kworker/3:1H-145 [003] ..s. 4093.916089: xs_tcp_data_recv: RPC: xprt = ffff8803b1c38000, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 kworker/3:1H-145 [003] ..s. 4093.916090: xprt_complete_rqst: RPC: 47 xid c50ccaff complete (24 bytes received) kworker/3:1H-145 [003] .... 4093.916091: xs_tcp_data_recv: RPC: xs_tcp_data_recv done kworker/3:1H-145 [003] .... 4093.916098: xprt_release: RPC: 47 release request ffff88040b27ca00 kworker/u32:1-13473 [002] .... 4093.976056: xprt_destroy: RPC: destroying transport ffff8803b1c38000 kworker/u32:1-13473 [002] .... 4093.976068: xs_destroy: RPC: xs_destroy xprt ffff8803b1c38000 kworker/u32:1-13473 [002] .... 4093.976069: xs_close: RPC: xs_close xprt ffff8803b1c38000 kworker/u32:1-13473 [002] ..s. 4093.976096: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8803b1c38000... kworker/u32:1-13473 [002] ..s. 4093.976098: xs_tcp_state_change: RPC: state 4 conn 1 dead 0 zapped 1 sk_shutdown 3 kworker/u32:1-13473 [002] .... 4093.976103: xprt_disconnect_done: RPC: disconnected transport ffff8803b1c38000 kworker/u32:1-13473 [002] .... 4093.976104: xprt_disconnect_done: disconnect transport! kworker/u32:1-13473 [002] .... 4093.976113: <stack trace> => xs_destroy => xprt_switch_free => rpc_free_client => rpc_release_client => rpc_shutdown_client => load_balance => ttwu_do_wakeup => nfsd4_process_cb_update.isra.14 => __switch_to => pick_next_task_fair => nfsd4_run_cb_work => process_one_work => worker_thread => worker_thread => kthread => ret_from_fork => kthread Immediately followed by setting up of the port that will eventually turn into the hidden port: kworker/u32:1-13473 [002] .... 4093.976128: xs_setup_tcp: RPC: set up xprt to 192.168.23.22 (port 44672) via tcp kworker/u32:1-13473 [002] .... 4093.976136: xprt_create_transport: RPC: created transport ffff8803b8c22000 with 65536 slots kworker/2:1H-144 [002] .... 4093.976209: xprt_alloc_slot: RPC: 48 reserved req ffff8803bfe89c00 xid 10c028fe kworker/2:1H-144 [002] .... 4093.976213: xprt_connect: RPC: 48 xprt_connect xprt ffff8803b8c22000 is not connected kworker/2:1H-144 [002] .... 4093.976215: xs_connect: RPC: xs_connect scheduled xprt ffff8803b8c22000 kworker/2:1H-144 [002] ..s. 4093.976231: inet_csk_get_port: snum 754 kworker/2:1H-144 [002] ..s. 4093.976239: <stack trace> => sched_clock => inet_addr_type_table => security_capable => inet_bind => xs_bind => release_sock => sock_setsockopt => __sock_create => xs_create_sock.isra.19 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => ret_from_fork => kthread kworker/2:1H-144 [002] ..s. 4093.976239: inet_bind_hash: add 754 ffff8803afc20e40 kworker/2:1H-144 [002] ..s. 4093.976247: <stack trace> => inet_csk_get_port => sched_clock => inet_addr_type_table => security_capable => inet_bind => xs_bind => release_sock => sock_setsockopt => __sock_create => xs_create_sock.isra.19 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => ret_from_fork => kthread kworker/2:1H-144 [002] .... 4093.976248: xs_bind: RPC: xs_bind 4.136.255.255:754: ok (0) kworker/2:1H-144 [002] .... 4093.976250: xs_tcp_setup_socket: RPC: worker connecting xprt ffff8803b8c22000 via tcp to 192.168.23.22 (port 44672) kworker/2:1H-144 [002] .... 4093.976284: xs_tcp_setup_socket: RPC: ffff8803b8c22000 connect status 115 connected 0 sock state 2 <idle>-0 [003] ..s. 4093.976456: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8803b8c22000... <idle>-0 [003] ..s. 4093.976458: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 kworker/3:1H-145 [003] .... 4093.976588: xprt_connect_status: RPC: 48 xprt_connect_status: retrying kworker/3:1H-145 [003] .... 4093.976590: xprt_prepare_transmit: RPC: 48 xprt_prepare_transmit kworker/3:1H-145 [003] .... 4093.976604: xprt_transmit: RPC: 48 xprt_transmit(72) kworker/3:1H-145 [003] .... 4093.976622: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 kworker/3:1H-145 [003] .... 4093.976623: xprt_transmit: RPC: 48 xmit complete <idle>-0 [003] ..s. 4093.977040: xs_tcp_data_ready: RPC: xs_tcp_data_ready... kworker/3:1H-145 [003] .... 4093.977151: xs_tcp_data_recv: RPC: xs_tcp_data_recv started kworker/3:1H-145 [003] .... 4093.977153: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 kworker/3:1H-145 [003] .... 4093.977154: xs_tcp_data_recv: RPC: reading XID (4 bytes) kworker/3:1H-145 [003] .... 4093.977155: xs_tcp_data_recv: RPC: reading request with XID 10c028fe kworker/3:1H-145 [003] .... 4093.977156: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) kworker/3:1H-145 [003] .... 4093.977157: xs_tcp_data_recv: RPC: read reply XID 10c028fe kworker/3:1H-145 [003] ..s. 4093.977158: xs_tcp_data_recv: RPC: XID 10c028fe read 16 bytes kworker/3:1H-145 [003] ..s. 4093.977159: xs_tcp_data_recv: RPC: xprt = ffff8803b8c22000, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 kworker/3:1H-145 [003] ..s. 4093.977160: xprt_complete_rqst: RPC: 48 xid 10c028fe complete (24 bytes received) kworker/3:1H-145 [003] .... 4093.977161: xs_tcp_data_recv: RPC: xs_tcp_data_recv done kworker/3:1H-145 [003] .... 4093.977167: xprt_release: RPC: 48 release request ffff8803bfe89c00 That "2049" port is what is used for all transferring of data. When the FIN/ACK is sent by the client, the socket is destroyed (no more connections can be used) but the port is not freed up, because it appears there's still an owner attached to it. That means this port will *never* be used again. Even if the client unmount the directory. That port is still in limbo. When the FIN/ACK comes in, it goes into the TIME_WAIT state here: kworker/3:1H-145 [003] ..s. 4394.370019: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8803b8c22000... kworker/3:1H-145 [003] ..s. 4394.370022: xs_tcp_state_change: RPC: state 4 conn 1 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [003] ..s. 4394.370352: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8803b8c22000... <idle>-0 [003] ..s. 4394.370354: xs_tcp_state_change: RPC: state 5 conn 0 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [003] ..s. 4394.370375: tcp_time_wait: moving xs_bind sock to time wait <idle>-0 [003] ..s. 4394.370396: <stack trace> => tcp_data_queue => tcp_rcv_state_process => tcp_v4_inbound_md5_hash => tcp_v4_do_rcv => tcp_v4_rcv => ipv4_confirm => nf_iterate => ip_local_deliver_finish => ip_local_deliver => ip_local_deliver_finish => ip_rcv => packet_rcv => ip_rcv_finish => __netif_receive_skb_core => kmem_cache_alloc => netif_receive_skb_internal => br_pass_frame_up => br_flood => br_handle_frame => br_handle_frame_finish => enqueue_task_fair => br_handle_frame => br_handle_frame => find_busiest_group => __netif_receive_skb_core => inet_gro_receive => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => net_rx_action => __do_softirq => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary Then eventually, the socket is freed, and 'netstat' no longer shows it. <idle>-0 [003] .Ns. 4454.371802: inet_bind_bucket_destroy: destroy 754 empty=0 ffff8803afc20e40 <idle>-0 [003] .Ns. 4454.371813: <stack trace> => inet_twsk_bind_unhash => inet_twsk_kill => tw_timer_handler => call_timer_fn => tw_timer_handler => run_timer_softirq => __do_softirq => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary "empty=0" means the tb->owners associated with the port is not empty, and the freeing of the port is skipped. Now when I go and remount the directory from the client, the code finally cleans up the port: kworker/u32:0-25031 [000] .... 4544.674603: xprt_destroy: RPC: destroying transport ffff8803b8c22000 kworker/u32:0-25031 [000] .... 4544.674616: xs_destroy: RPC: xs_destroy xprt ffff8803b8c22000 kworker/u32:0-25031 [000] .... 4544.674617: xs_close: RPC: xs_close xprt ffff8803b8c22000 kworker/u32:0-25031 [000] ..s. 4544.674619: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8803b8c22000... kworker/u32:0-25031 [000] ..s. 4544.674620: xs_tcp_state_change: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 kworker/u32:0-25031 [000] ..s. 4544.674621: xprt_disconnect_done: RPC: disconnected transport ffff8803b8c22000 kworker/u32:0-25031 [000] ..s. 4544.674621: xprt_disconnect_done: disconnect transport! kworker/u32:0-25031 [000] ..s. 4544.674647: inet_bind_bucket_destroy: destroy 754 empty=1 ffff8803afc20e40 kworker/u32:0-25031 [000] ..s. 4544.674655: <stack trace> => inet_put_port => tcp_v4_destroy_sock => inet_csk_destroy_sock => tcp_close => inet_release => sock_release => xs_close => xs_destroy => xprt_switch_free => rpc_free_client => rpc_release_client => rpc_shutdown_client => nfsd4_process_cb_update.isra.14 => update_curr => dequeue_task_fair => __switch_to => pick_next_task_fair => nfsd4_run_cb_work => process_one_work => worker_thread => worker_thread => kthread => ret_from_fork => kthread kworker/u32:0-25031 [000] .... 4544.674660: xprt_disconnect_done: RPC: disconnected transport ffff8803b8c22000 Notice that "empty=1" now, and the port is freed. Then it goes back doing everything all over again: kworker/3:1H-145 [003] .... 4558.442458: xs_bind: RPC: xs_bind 4.136.255.255:973: ok (0) kworker/3:1H-145 [003] .... 4558.442460: xs_tcp_setup_socket: RPC: worker connecting xprt ffff8803d7fd3800 via tcp to 192.168.23.22 (port 45075) kworker/3:1H-145 [003] .... 4558.442496: xs_tcp_setup_socket: RPC: ffff8803d7fd3800 connect status 115 connected 0 sock state 2 <idle>-0 [002] ..s. 4558.442691: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8803d7fd3800... <idle>-0 [002] ..s. 4558.442693: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 kworker/2:1H-144 [002] .... 4558.442732: xprt_connect_status: RPC: 49 xprt_connect_status: retrying kworker/2:1H-144 [002] .... 4558.442734: xprt_prepare_transmit: RPC: 49 xprt_prepare_transmit kworker/2:1H-144 [002] .... 4558.442737: xprt_transmit: RPC: 49 xprt_transmit(72) kworker/2:1H-144 [002] .... 4558.442753: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 kworker/2:1H-144 [002] .... 4558.442754: xprt_transmit: RPC: 49 xmit complete nfsd-4382 [002] ..s. 4558.443203: xs_tcp_data_ready: RPC: xs_tcp_data_ready... kworker/2:1H-144 [002] .... 4558.443227: xs_tcp_data_recv: RPC: xs_tcp_data_recv started kworker/2:1H-144 [002] .... 4558.443229: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 kworker/2:1H-144 [002] .... 4558.443230: xs_tcp_data_recv: RPC: reading XID (4 bytes) kworker/2:1H-144 [002] .... 4558.443231: xs_tcp_data_recv: RPC: reading request with XID e2e1dc21 kworker/2:1H-144 [002] .... 4558.443232: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) kworker/2:1H-144 [002] .... 4558.443233: xs_tcp_data_recv: RPC: read reply XID e2e1dc21 kworker/2:1H-144 [002] ..s. 4558.443235: xs_tcp_data_recv: RPC: XID e2e1dc21 read 16 bytes kworker/2:1H-144 [002] ..s. 4558.443236: xs_tcp_data_recv: RPC: xprt = ffff8803d7fd3800, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 kworker/2:1H-144 [002] ..s. 4558.443237: xprt_complete_rqst: RPC: 49 xid e2e1dc21 complete (24 bytes received) kworker/2:1H-144 [002] .... 4558.443238: xs_tcp_data_recv: RPC: xs_tcp_data_recv done kworker/2:1H-144 [002] .... 4558.443246: xprt_release: RPC: 49 release request ffff8800dba14800 kworker/u32:1-13473 [003] .... 4558.496850: xprt_destroy: RPC: destroying transport ffff8803d7fd3800 kworker/u32:1-13473 [003] .... 4558.496860: xs_destroy: RPC: xs_destroy xprt ffff8803d7fd3800 kworker/u32:1-13473 [003] .... 4558.496861: xs_close: RPC: xs_close xprt ffff8803d7fd3800 kworker/u32:1-13473 [003] ..s. 4558.496888: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8803d7fd3800... kworker/u32:1-13473 [003] ..s. 4558.496889: xs_tcp_state_change: RPC: state 4 conn 1 dead 0 zapped 1 sk_shutdown 3 kworker/u32:1-13473 [003] .... 4558.496894: xprt_disconnect_done: RPC: disconnected transport ffff8803d7fd3800 kworker/u32:1-13473 [003] .... 4558.496895: xprt_disconnect_done: disconnect transport! Where 973 is the port that gets added and disconnected right away (and freed) kworker/3:1H-145 [003] .... 4558.496991: xs_bind: RPC: xs_bind 4.136.255.255:688: ok (0) kworker/3:1H-145 [003] .... 4558.496993: xs_tcp_setup_socket: RPC: worker connecting xprt ffff8803bb889000 via tcp to 192.168.23.22 (port 45075) kworker/3:1H-145 [003] .... 4558.497024: xs_tcp_setup_socket: RPC: ffff8803bb889000 connect status 115 connected 0 sock state 2 <idle>-0 [002] .Ns. 4558.497171: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8803bb889000... <idle>-0 [002] .Ns. 4558.497173: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 kworker/2:1H-144 [002] .... 4558.497196: xprt_connect_status: RPC: 50 xprt_connect_status: retrying kworker/2:1H-144 [002] .... 4558.497197: xprt_prepare_transmit: RPC: 50 xprt_prepare_transmit kworker/2:1H-144 [002] .... 4558.497199: xprt_transmit: RPC: 50 xprt_transmit(72) kworker/2:1H-144 [002] .... 4558.497210: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 kworker/2:1H-144 [002] .... 4558.497210: xprt_transmit: RPC: 50 xmit complete <idle>-0 [002] ..s. 4558.497475: xs_tcp_data_ready: RPC: xs_tcp_data_ready... kworker/2:1H-144 [002] .... 4558.497569: xs_tcp_data_recv: RPC: xs_tcp_data_recv started kworker/2:1H-144 [002] .... 4558.497571: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 kworker/2:1H-144 [002] .... 4558.497571: xs_tcp_data_recv: RPC: reading XID (4 bytes) kworker/2:1H-144 [002] .... 4558.497572: xs_tcp_data_recv: RPC: reading request with XID a4418f34 kworker/2:1H-144 [002] .... 4558.497573: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) kworker/2:1H-144 [002] .... 4558.497573: xs_tcp_data_recv: RPC: read reply XID a4418f34 kworker/2:1H-144 [002] ..s. 4558.497574: xs_tcp_data_recv: RPC: XID a4418f34 read 16 bytes kworker/2:1H-144 [002] ..s. 4558.497575: xs_tcp_data_recv: RPC: xprt = ffff8803bb889000, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 kworker/2:1H-144 [002] ..s. 4558.497575: xprt_complete_rqst: RPC: 50 xid a4418f34 complete (24 bytes received) kworker/2:1H-144 [002] .... 4558.497577: xs_tcp_data_recv: RPC: xs_tcp_data_recv done kworker/2:1H-144 [002] .... 4558.497581: xprt_release: RPC: 50 release request ffff8800db28d800 And 688 will be the port that becomes the new hidden port. Thus it looks like there's something that is not cleaning up ports properly. There's some timing issues here. Any thoughts why. Because this is obviously wrong, and not only is it wasting a port and memory to store it, but it's causing rkhunter to report it, because it's one of the things that rootkits do. -- Steve ^ permalink raw reply [flat|nested] 15+ messages in thread
* [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )
@ 2015-06-12 3:49 Steven Rostedt
2015-06-12 14:10 ` Trond Myklebust
0 siblings, 1 reply; 15+ messages in thread
From: Steven Rostedt @ 2015-06-12 3:49 UTC (permalink / raw)
To: Trond Myklebust; +Cc: Anna Schumaker, linux-nfs, netdev, LKML, Andrew Morton
I recently upgraded my main server to 4.0.4 from 3.19.5 and rkhunter
started reporting a hidden port on my box.
Running unhide-tcp I see this:
# unhide-tcp
Unhide-tcp 20121229
Copyright © 2012 Yago Jesus & Patrick Gouin
License GPLv3+ : GNU GPL version 3 or later
http://www.unhide-forensics.info
Used options:
[*]Starting TCP checking
Found Hidden port that not appears in ss: 946
[*]Starting UDP checking
This scared the hell out of me as I'm thinking that I have got some kind
of NSA backdoor hooked into my server and it is monitoring my plans to
smuggle Kinder Überraschung into the USA from Germany. I panicked!
Well, I wasted the day writing modules to first look at all the sockets
opened by all processes (via their file descriptors) and posted their
port numbers.
http://rostedt.homelinux.com/private/tasklist.c
But this port wasn't there either.
Then I decided to look at the ports in tcp_hashinfo.
http://rostedt.homelinux.com/private/portlist.c
This found the port but no file was connected to it, and worse yet,
when I first ran it without using probe_kernel_read(), it crashed my
kernel, because sk->sk_socket pointed to a freed socket!
Note, each boot, the hidden port is different.
Finally, I decided to bring in the big guns, and inserted a
trace_printk() into the bind logic, to see if I could find the culprit.
After fiddling with it a few times, I found a suspect:
kworker/3:1H-123 [003] ..s. 96.696213: inet_bind_hash: add 946
Bah, it's a kernel thread doing it, via a work queue. I then added a
trace_dump_stack() to find what was calling this, and here it is:
kworker/3:1H-123 [003] ..s. 96.696222: <stack trace>
=> inet_csk_get_port
=> inet_addr_type
=> inet_bind
=> xs_bind
=> sock_setsockopt
=> __sock_create
=> xs_create_sock.isra.18
=> xs_tcp_setup_socket
=> process_one_work
=> worker_thread
=> worker_thread
=> kthread
=> kthread
=> ret_from_fork
=> kthread
I rebooted, and examined what happens. I see the kworker binding that
port, and all seems well:
# netstat -tapn |grep 946
tcp 0 0 192.168.23.9:946 192.168.23.22:55201 ESTABLISHED -
But waiting for a bit, the connection goes into a TIME_WAIT, and then
it just disappears. But the bind to the port does not get released, and
that port is from then on, taken.
This never happened with my 3.19 kernels. I would bisect it but this is
happening on my main server box which I usually only reboot every other
month doing upgrades. It causes too much disturbance for myself (and my
family) as when this box is offline, basically the rest of my machines
are too.
I figured this may be enough information to see if you can fix it.
Otherwise I can try to do the bisect, but that's not going to happen
any time soon. I may just go back to 3.19 for now, such that rkhunter
stops complaining about the hidden port.
If you need anymore information, let me know.
-- Steve
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-12 3:49 [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) Steven Rostedt @ 2015-06-12 14:10 ` Trond Myklebust 2015-06-12 14:40 ` Eric Dumazet 0 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2015-06-12 14:10 UTC (permalink / raw) To: Steven Rostedt Cc: Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton On Thu, Jun 11, 2015 at 11:49 PM, Steven Rostedt <rostedt@goodmis.org> wrote: > > I recently upgraded my main server to 4.0.4 from 3.19.5 and rkhunter > started reporting a hidden port on my box. > > Running unhide-tcp I see this: > > # unhide-tcp > Unhide-tcp 20121229 > Copyright © 2012 Yago Jesus & Patrick Gouin > License GPLv3+ : GNU GPL version 3 or later > http://www.unhide-forensics.info > Used options: > [*]Starting TCP checking > > Found Hidden port that not appears in ss: 946 > [*]Starting UDP checking > > This scared the hell out of me as I'm thinking that I have got some kind > of NSA backdoor hooked into my server and it is monitoring my plans to > smuggle Kinder Überraschung into the USA from Germany. I panicked! > > Well, I wasted the day writing modules to first look at all the sockets > opened by all processes (via their file descriptors) and posted their > port numbers. > > http://rostedt.homelinux.com/private/tasklist.c > > But this port wasn't there either. > > Then I decided to look at the ports in tcp_hashinfo. > > http://rostedt.homelinux.com/private/portlist.c > > This found the port but no file was connected to it, and worse yet, > when I first ran it without using probe_kernel_read(), it crashed my > kernel, because sk->sk_socket pointed to a freed socket! > > Note, each boot, the hidden port is different. > > Finally, I decided to bring in the big guns, and inserted a > trace_printk() into the bind logic, to see if I could find the culprit. > After fiddling with it a few times, I found a suspect: > > kworker/3:1H-123 [003] ..s. 96.696213: inet_bind_hash: add 946 > > Bah, it's a kernel thread doing it, via a work queue. I then added a > trace_dump_stack() to find what was calling this, and here it is: > > kworker/3:1H-123 [003] ..s. 96.696222: <stack trace> > => inet_csk_get_port > => inet_addr_type > => inet_bind > => xs_bind > => sock_setsockopt > => __sock_create > => xs_create_sock.isra.18 > => xs_tcp_setup_socket > => process_one_work > => worker_thread > => worker_thread > => kthread > => kthread > => ret_from_fork > => kthread > > I rebooted, and examined what happens. I see the kworker binding that > port, and all seems well: > > # netstat -tapn |grep 946 > tcp 0 0 192.168.23.9:946 192.168.23.22:55201 ESTABLISHED - > > But waiting for a bit, the connection goes into a TIME_WAIT, and then > it just disappears. But the bind to the port does not get released, and > that port is from then on, taken. > > This never happened with my 3.19 kernels. I would bisect it but this is > happening on my main server box which I usually only reboot every other > month doing upgrades. It causes too much disturbance for myself (and my > family) as when this box is offline, basically the rest of my machines > are too. > > I figured this may be enough information to see if you can fix it. > Otherwise I can try to do the bisect, but that's not going to happen > any time soon. I may just go back to 3.19 for now, such that rkhunter > stops complaining about the hidden port. > The only new thing that we're doing with 4.0 is to set SO_REUSEPORT on the socket before binding the port (commit 4dda9c8a5e34: "SUNRPC: Set SO_REUSEPORT socket option for TCP connections"). Perhaps there is an issue with that? Cheers Trond ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-12 14:10 ` Trond Myklebust @ 2015-06-12 14:40 ` Eric Dumazet 2015-06-12 15:34 ` Steven Rostedt 0 siblings, 1 reply; 15+ messages in thread From: Eric Dumazet @ 2015-06-12 14:40 UTC (permalink / raw) To: Trond Myklebust Cc: Steven Rostedt, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton On Fri, 2015-06-12 at 10:10 -0400, Trond Myklebust wrote: > On Thu, Jun 11, 2015 at 11:49 PM, Steven Rostedt <rostedt@goodmis.org> wrote: > > > > I recently upgraded my main server to 4.0.4 from 3.19.5 and rkhunter > > started reporting a hidden port on my box. > > > > Running unhide-tcp I see this: > > > > # unhide-tcp > > Unhide-tcp 20121229 > > Copyright © 2012 Yago Jesus & Patrick Gouin > > License GPLv3+ : GNU GPL version 3 or later > > http://www.unhide-forensics.info > > Used options: > > [*]Starting TCP checking > > > > Found Hidden port that not appears in ss: 946 > > [*]Starting UDP checking > > > > This scared the hell out of me as I'm thinking that I have got some kind > > of NSA backdoor hooked into my server and it is monitoring my plans to > > smuggle Kinder Überraschung into the USA from Germany. I panicked! > > > > Well, I wasted the day writing modules to first look at all the sockets > > opened by all processes (via their file descriptors) and posted their > > port numbers. > > > > http://rostedt.homelinux.com/private/tasklist.c > > > > But this port wasn't there either. > > > > Then I decided to look at the ports in tcp_hashinfo. > > > > http://rostedt.homelinux.com/private/portlist.c > > > > This found the port but no file was connected to it, and worse yet, > > when I first ran it without using probe_kernel_read(), it crashed my > > kernel, because sk->sk_socket pointed to a freed socket! > > > > Note, each boot, the hidden port is different. > > > > Finally, I decided to bring in the big guns, and inserted a > > trace_printk() into the bind logic, to see if I could find the culprit. > > After fiddling with it a few times, I found a suspect: > > > > kworker/3:1H-123 [003] ..s. 96.696213: inet_bind_hash: add 946 > > > > Bah, it's a kernel thread doing it, via a work queue. I then added a > > trace_dump_stack() to find what was calling this, and here it is: > > > > kworker/3:1H-123 [003] ..s. 96.696222: <stack trace> > > => inet_csk_get_port > > => inet_addr_type > > => inet_bind > > => xs_bind > > => sock_setsockopt > > => __sock_create > > => xs_create_sock.isra.18 > > => xs_tcp_setup_socket > > => process_one_work > > => worker_thread > > => worker_thread > > => kthread > > => kthread > > => ret_from_fork > > => kthread > > > > I rebooted, and examined what happens. I see the kworker binding that > > port, and all seems well: > > > > # netstat -tapn |grep 946 > > tcp 0 0 192.168.23.9:946 192.168.23.22:55201 ESTABLISHED - > > > > But waiting for a bit, the connection goes into a TIME_WAIT, and then > > it just disappears. But the bind to the port does not get released, and > > that port is from then on, taken. > > > > This never happened with my 3.19 kernels. I would bisect it but this is > > happening on my main server box which I usually only reboot every other > > month doing upgrades. It causes too much disturbance for myself (and my > > family) as when this box is offline, basically the rest of my machines > > are too. > > > > I figured this may be enough information to see if you can fix it. > > Otherwise I can try to do the bisect, but that's not going to happen > > any time soon. I may just go back to 3.19 for now, such that rkhunter > > stops complaining about the hidden port. > > > > The only new thing that we're doing with 4.0 is to set SO_REUSEPORT on > the socket before binding the port (commit 4dda9c8a5e34: "SUNRPC: Set > SO_REUSEPORT socket option for TCP connections"). Perhaps there is an > issue with that? Strange, because the usual way to not have time-wait is to use SO_LINGER with linger=0 And apparently xs_tcp_finish_connecting() has this : sock_reset_flag(sk, SOCK_LINGER); tcp_sk(sk)->linger2 = 0; Are you sure SO_REUSEADDR was not the thing you wanted ? Steven, have you tried kmemleak ? ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-12 14:40 ` Eric Dumazet @ 2015-06-12 15:34 ` Steven Rostedt 2015-06-12 15:50 ` Steven Rostedt 0 siblings, 1 reply; 15+ messages in thread From: Steven Rostedt @ 2015-06-12 15:34 UTC (permalink / raw) To: Eric Dumazet Cc: Trond Myklebust, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton On Fri, 12 Jun 2015 07:40:35 -0700 Eric Dumazet <eric.dumazet@gmail.com> wrote: > Strange, because the usual way to not have time-wait is to use SO_LINGER > with linger=0 > > And apparently xs_tcp_finish_connecting() has this : > > sock_reset_flag(sk, SOCK_LINGER); > tcp_sk(sk)->linger2 = 0; > > Are you sure SO_REUSEADDR was not the thing you wanted ? > > Steven, have you tried kmemleak ? Nope, and again, I'm hesitant on adding too much debug. This is my main server (build server, ssh server, web server, mail server, proxy server, irc server, etc). Although, I made dprintk() into trace_printk() in xprtsock.c and xprt.c, and reran it. Here's the output: (port 684 was the bad one this time) # tracer: nop # # entries-in-buffer/entries-written: 396/396 #P:4 # # _-----=> irqs-off # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | rpc.nfsd-4710 [002] .... 48.615382: xs_local_setup_socket: RPC: worker connecting xprt ffff8800d9018000 via AF_LOCAL to /var/run/rpcbind.sock rpc.nfsd-4710 [002] .... 48.615393: xs_local_setup_socket: RPC: xprt ffff8800d9018000 connected to /var/run/rpcbind.sock rpc.nfsd-4710 [002] .... 48.615394: xs_setup_local: RPC: set up xprt to /var/run/rpcbind.sock via AF_LOCAL rpc.nfsd-4710 [002] .... 48.615399: xprt_create_transport: RPC: created transport ffff8800d9018000 with 65536 slots rpc.nfsd-4710 [002] .... 48.615416: xprt_alloc_slot: RPC: 1 reserved req ffff8800db829600 xid cb06d5e8 rpc.nfsd-4710 [002] .... 48.615419: xprt_prepare_transmit: RPC: 1 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.615420: xprt_transmit: RPC: 1 xprt_transmit(44) rpc.nfsd-4710 [002] .... 48.615424: xs_local_send_request: RPC: xs_local_send_request(44) = 0 rpc.nfsd-4710 [002] .... 48.615425: xprt_transmit: RPC: 1 xmit complete rpcbind-1829 [003] ..s. 48.615503: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [003] ..s. 48.615506: xprt_complete_rqst: RPC: 1 xid cb06d5e8 complete (24 bytes received) rpc.nfsd-4710 [002] .... 48.615556: xprt_release: RPC: 1 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.615568: xprt_alloc_slot: RPC: 2 reserved req ffff8800db829600 xid cc06d5e8 rpc.nfsd-4710 [002] .... 48.615569: xprt_prepare_transmit: RPC: 2 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.615569: xprt_transmit: RPC: 2 xprt_transmit(44) rpc.nfsd-4710 [002] .... 48.615578: xs_local_send_request: RPC: xs_local_send_request(44) = 0 rpc.nfsd-4710 [002] .... 48.615578: xprt_transmit: RPC: 2 xmit complete rpcbind-1829 [003] ..s. 48.615643: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [003] ..s. 48.615645: xprt_complete_rqst: RPC: 2 xid cc06d5e8 complete (24 bytes received) rpc.nfsd-4710 [002] .... 48.615695: xprt_release: RPC: 2 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.615698: xprt_alloc_slot: RPC: 3 reserved req ffff8800db829600 xid cd06d5e8 rpc.nfsd-4710 [002] .... 48.615699: xprt_prepare_transmit: RPC: 3 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.615700: xprt_transmit: RPC: 3 xprt_transmit(68) rpc.nfsd-4710 [002] .... 48.615706: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4710 [002] .... 48.615707: xprt_transmit: RPC: 3 xmit complete rpcbind-1829 [003] ..s. 48.615784: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [003] ..s. 48.615785: xprt_complete_rqst: RPC: 3 xid cd06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.615830: xprt_release: RPC: 3 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.615833: xprt_alloc_slot: RPC: 4 reserved req ffff8800db829600 xid ce06d5e8 rpc.nfsd-4710 [002] .... 48.615834: xprt_prepare_transmit: RPC: 4 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.615835: xprt_transmit: RPC: 4 xprt_transmit(68) rpc.nfsd-4710 [002] .... 48.615841: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4710 [002] .... 48.615841: xprt_transmit: RPC: 4 xmit complete rpcbind-1829 [003] ..s. 48.615892: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [003] ..s. 48.615894: xprt_complete_rqst: RPC: 4 xid ce06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.615958: xprt_release: RPC: 4 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.615961: xprt_alloc_slot: RPC: 5 reserved req ffff8800db829600 xid cf06d5e8 rpc.nfsd-4710 [002] .... 48.615962: xprt_prepare_transmit: RPC: 5 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.615966: xprt_transmit: RPC: 5 xprt_transmit(68) rpc.nfsd-4710 [002] .... 48.615971: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4710 [002] .... 48.615972: xprt_transmit: RPC: 5 xmit complete rpcbind-1829 [003] ..s. 48.616011: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [003] ..s. 48.616012: xprt_complete_rqst: RPC: 5 xid cf06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616057: xprt_release: RPC: 5 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616059: xprt_alloc_slot: RPC: 6 reserved req ffff8800db829600 xid d006d5e8 rpc.nfsd-4710 [002] .... 48.616060: xprt_prepare_transmit: RPC: 6 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616061: xprt_transmit: RPC: 6 xprt_transmit(68) rpc.nfsd-4710 [002] .... 48.616065: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4710 [002] .... 48.616066: xprt_transmit: RPC: 6 xmit complete rpcbind-1829 [003] ..s. 48.616117: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [003] ..s. 48.616119: xprt_complete_rqst: RPC: 6 xid d006d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616163: xprt_release: RPC: 6 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616165: xprt_alloc_slot: RPC: 7 reserved req ffff8800db829600 xid d106d5e8 rpc.nfsd-4710 [002] .... 48.616166: xprt_prepare_transmit: RPC: 7 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616167: xprt_transmit: RPC: 7 xprt_transmit(68) rpc.nfsd-4710 [002] .... 48.616172: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4710 [002] .... 48.616172: xprt_transmit: RPC: 7 xmit complete rpcbind-1829 [000] ..s. 48.616247: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616249: xprt_complete_rqst: RPC: 7 xid d106d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616289: xprt_release: RPC: 7 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616296: xprt_alloc_slot: RPC: 8 reserved req ffff8800db829600 xid d206d5e8 rpc.nfsd-4710 [002] .... 48.616297: xprt_prepare_transmit: RPC: 8 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616298: xprt_transmit: RPC: 8 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.616302: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.616302: xprt_transmit: RPC: 8 xmit complete rpcbind-1829 [000] ..s. 48.616324: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616326: xprt_complete_rqst: RPC: 8 xid d206d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616340: xprt_release: RPC: 8 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616346: xprt_alloc_slot: RPC: 9 reserved req ffff8800db829600 xid d306d5e8 rpc.nfsd-4710 [002] .... 48.616347: xprt_prepare_transmit: RPC: 9 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616348: xprt_transmit: RPC: 9 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.616355: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.616355: xprt_transmit: RPC: 9 xmit complete rpcbind-1829 [000] ..s. 48.616380: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616381: xprt_complete_rqst: RPC: 9 xid d306d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616392: xprt_release: RPC: 9 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616396: xprt_alloc_slot: RPC: 10 reserved req ffff8800db829600 xid d406d5e8 rpc.nfsd-4710 [002] .... 48.616396: xprt_prepare_transmit: RPC: 10 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616397: xprt_transmit: RPC: 10 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.616401: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.616402: xprt_transmit: RPC: 10 xmit complete rpcbind-1829 [000] ..s. 48.616421: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616422: xprt_complete_rqst: RPC: 10 xid d406d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616433: xprt_release: RPC: 10 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616436: xprt_alloc_slot: RPC: 11 reserved req ffff8800db829600 xid d506d5e8 rpc.nfsd-4710 [002] .... 48.616437: xprt_prepare_transmit: RPC: 11 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616438: xprt_transmit: RPC: 11 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.616442: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.616442: xprt_transmit: RPC: 11 xmit complete rpcbind-1829 [000] ..s. 48.616461: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616462: xprt_complete_rqst: RPC: 11 xid d506d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616473: xprt_release: RPC: 11 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616476: xprt_alloc_slot: RPC: 12 reserved req ffff8800db829600 xid d606d5e8 rpc.nfsd-4710 [002] .... 48.616477: xprt_prepare_transmit: RPC: 12 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616478: xprt_transmit: RPC: 12 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.616482: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.616482: xprt_transmit: RPC: 12 xmit complete rpcbind-1829 [000] ..s. 48.616501: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616502: xprt_complete_rqst: RPC: 12 xid d606d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616511: xprt_release: RPC: 12 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616535: xprt_alloc_slot: RPC: 13 reserved req ffff8800db829600 xid d706d5e8 rpc.nfsd-4710 [002] .... 48.616536: xprt_prepare_transmit: RPC: 13 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616537: xprt_transmit: RPC: 13 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.616541: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.616541: xprt_transmit: RPC: 13 xmit complete rpcbind-1829 [000] ..s. 48.616580: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616581: xprt_complete_rqst: RPC: 13 xid d706d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616589: xprt_release: RPC: 13 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616591: xprt_alloc_slot: RPC: 14 reserved req ffff8800db829600 xid d806d5e8 rpc.nfsd-4710 [002] .... 48.616591: xprt_prepare_transmit: RPC: 14 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616592: xprt_transmit: RPC: 14 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.616594: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.616595: xprt_transmit: RPC: 14 xmit complete rpcbind-1829 [000] ..s. 48.616610: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616611: xprt_complete_rqst: RPC: 14 xid d806d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616618: xprt_release: RPC: 14 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616619: xprt_alloc_slot: RPC: 15 reserved req ffff8800db829600 xid d906d5e8 rpc.nfsd-4710 [002] .... 48.616620: xprt_prepare_transmit: RPC: 15 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616620: xprt_transmit: RPC: 15 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.616623: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.616623: xprt_transmit: RPC: 15 xmit complete rpcbind-1829 [000] ..s. 48.616635: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616637: xprt_complete_rqst: RPC: 15 xid d906d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616643: xprt_release: RPC: 15 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616644: xprt_alloc_slot: RPC: 16 reserved req ffff8800db829600 xid da06d5e8 rpc.nfsd-4710 [002] .... 48.616645: xprt_prepare_transmit: RPC: 16 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616645: xprt_transmit: RPC: 16 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.616648: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.616648: xprt_transmit: RPC: 16 xmit complete rpcbind-1829 [000] ..s. 48.616658: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616659: xprt_complete_rqst: RPC: 16 xid da06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616665: xprt_release: RPC: 16 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.616666: xprt_alloc_slot: RPC: 17 reserved req ffff8800db829600 xid db06d5e8 rpc.nfsd-4710 [002] .... 48.616667: xprt_prepare_transmit: RPC: 17 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.616667: xprt_transmit: RPC: 17 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.616670: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.616670: xprt_transmit: RPC: 17 xmit complete rpcbind-1829 [000] ..s. 48.616680: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.616681: xprt_complete_rqst: RPC: 17 xid db06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.616687: xprt_release: RPC: 17 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617251: xprt_alloc_slot: RPC: 18 reserved req ffff8800db829600 xid dc06d5e8 rpc.nfsd-4710 [002] .... 48.617252: xprt_prepare_transmit: RPC: 18 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617252: xprt_transmit: RPC: 18 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617256: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617257: xprt_transmit: RPC: 18 xmit complete rpcbind-1829 [000] ..s. 48.617265: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617265: xprt_complete_rqst: RPC: 18 xid dc06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617272: xprt_release: RPC: 18 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617274: xprt_alloc_slot: RPC: 19 reserved req ffff8800db829600 xid dd06d5e8 rpc.nfsd-4710 [002] .... 48.617274: xprt_prepare_transmit: RPC: 19 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617275: xprt_transmit: RPC: 19 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617277: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617277: xprt_transmit: RPC: 19 xmit complete rpcbind-1829 [000] ..s. 48.617287: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617288: xprt_complete_rqst: RPC: 19 xid dd06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617293: xprt_release: RPC: 19 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617295: xprt_alloc_slot: RPC: 20 reserved req ffff8800db829600 xid de06d5e8 rpc.nfsd-4710 [002] .... 48.617295: xprt_prepare_transmit: RPC: 20 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617296: xprt_transmit: RPC: 20 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617298: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617298: xprt_transmit: RPC: 20 xmit complete rpcbind-1829 [000] ..s. 48.617307: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617308: xprt_complete_rqst: RPC: 20 xid de06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617314: xprt_release: RPC: 20 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617315: xprt_alloc_slot: RPC: 21 reserved req ffff8800db829600 xid df06d5e8 rpc.nfsd-4710 [002] .... 48.617316: xprt_prepare_transmit: RPC: 21 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617316: xprt_transmit: RPC: 21 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617318: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617319: xprt_transmit: RPC: 21 xmit complete rpcbind-1829 [000] ..s. 48.617328: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617329: xprt_complete_rqst: RPC: 21 xid df06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617334: xprt_release: RPC: 21 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617336: xprt_alloc_slot: RPC: 22 reserved req ffff8800db829600 xid e006d5e8 rpc.nfsd-4710 [002] .... 48.617336: xprt_prepare_transmit: RPC: 22 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617336: xprt_transmit: RPC: 22 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617339: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617339: xprt_transmit: RPC: 22 xmit complete rpcbind-1829 [000] ..s. 48.617348: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617349: xprt_complete_rqst: RPC: 22 xid e006d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617354: xprt_release: RPC: 22 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617370: xprt_alloc_slot: RPC: 23 reserved req ffff8800db829600 xid e106d5e8 rpc.nfsd-4710 [002] .... 48.617371: xprt_prepare_transmit: RPC: 23 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617371: xprt_transmit: RPC: 23 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617374: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617374: xprt_transmit: RPC: 23 xmit complete rpcbind-1829 [000] ..s. 48.617382: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617383: xprt_complete_rqst: RPC: 23 xid e106d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617389: xprt_release: RPC: 23 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617390: xprt_alloc_slot: RPC: 24 reserved req ffff8800db829600 xid e206d5e8 rpc.nfsd-4710 [002] .... 48.617391: xprt_prepare_transmit: RPC: 24 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617391: xprt_transmit: RPC: 24 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617394: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617394: xprt_transmit: RPC: 24 xmit complete rpcbind-1829 [000] ..s. 48.617403: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617404: xprt_complete_rqst: RPC: 24 xid e206d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617410: xprt_release: RPC: 24 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617411: xprt_alloc_slot: RPC: 25 reserved req ffff8800db829600 xid e306d5e8 rpc.nfsd-4710 [002] .... 48.617412: xprt_prepare_transmit: RPC: 25 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617412: xprt_transmit: RPC: 25 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617414: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617414: xprt_transmit: RPC: 25 xmit complete rpcbind-1829 [000] ..s. 48.617424: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617424: xprt_complete_rqst: RPC: 25 xid e306d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617430: xprt_release: RPC: 25 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617431: xprt_alloc_slot: RPC: 26 reserved req ffff8800db829600 xid e406d5e8 rpc.nfsd-4710 [002] .... 48.617432: xprt_prepare_transmit: RPC: 26 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617432: xprt_transmit: RPC: 26 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617434: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617434: xprt_transmit: RPC: 26 xmit complete rpcbind-1829 [000] ..s. 48.617444: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617444: xprt_complete_rqst: RPC: 26 xid e406d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617450: xprt_release: RPC: 26 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617451: xprt_alloc_slot: RPC: 27 reserved req ffff8800db829600 xid e506d5e8 rpc.nfsd-4710 [002] .... 48.617452: xprt_prepare_transmit: RPC: 27 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617452: xprt_transmit: RPC: 27 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617454: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617455: xprt_transmit: RPC: 27 xmit complete rpcbind-1829 [000] ..s. 48.617464: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617464: xprt_complete_rqst: RPC: 27 xid e506d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617470: xprt_release: RPC: 27 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617575: xprt_alloc_slot: RPC: 28 reserved req ffff8800db829600 xid e606d5e8 rpc.nfsd-4710 [002] .... 48.617576: xprt_prepare_transmit: RPC: 28 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617577: xprt_transmit: RPC: 28 xprt_transmit(68) rpc.nfsd-4710 [002] .... 48.617580: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4710 [002] .... 48.617580: xprt_transmit: RPC: 28 xmit complete rpcbind-1829 [000] ..s. 48.617590: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617591: xprt_complete_rqst: RPC: 28 xid e606d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617598: xprt_release: RPC: 28 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617599: xprt_alloc_slot: RPC: 29 reserved req ffff8800db829600 xid e706d5e8 rpc.nfsd-4710 [002] .... 48.617599: xprt_prepare_transmit: RPC: 29 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617600: xprt_transmit: RPC: 29 xprt_transmit(68) rpc.nfsd-4710 [002] .... 48.617602: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4710 [002] .... 48.617602: xprt_transmit: RPC: 29 xmit complete rpcbind-1829 [000] ..s. 48.617614: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617615: xprt_complete_rqst: RPC: 29 xid e706d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617621: xprt_release: RPC: 29 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617622: xprt_alloc_slot: RPC: 30 reserved req ffff8800db829600 xid e806d5e8 rpc.nfsd-4710 [002] .... 48.617622: xprt_prepare_transmit: RPC: 30 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617623: xprt_transmit: RPC: 30 xprt_transmit(68) rpc.nfsd-4710 [002] .... 48.617625: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4710 [002] .... 48.617625: xprt_transmit: RPC: 30 xmit complete rpcbind-1829 [000] ..s. 48.617634: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617635: xprt_complete_rqst: RPC: 30 xid e806d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617640: xprt_release: RPC: 30 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617647: xprt_alloc_slot: RPC: 31 reserved req ffff8800db829600 xid e906d5e8 rpc.nfsd-4710 [002] .... 48.617647: xprt_prepare_transmit: RPC: 31 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617648: xprt_transmit: RPC: 31 xprt_transmit(88) rpc.nfsd-4710 [002] .... 48.617650: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4710 [002] .... 48.617650: xprt_transmit: RPC: 31 xmit complete rpcbind-1829 [000] ..s. 48.617659: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617660: xprt_complete_rqst: RPC: 31 xid e906d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617666: xprt_release: RPC: 31 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617668: xprt_alloc_slot: RPC: 32 reserved req ffff8800db829600 xid ea06d5e8 rpc.nfsd-4710 [002] .... 48.617668: xprt_prepare_transmit: RPC: 32 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617669: xprt_transmit: RPC: 32 xprt_transmit(88) rpc.nfsd-4710 [002] .... 48.617671: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4710 [002] .... 48.617671: xprt_transmit: RPC: 32 xmit complete rpcbind-1829 [000] ..s. 48.617681: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617681: xprt_complete_rqst: RPC: 32 xid ea06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617687: xprt_release: RPC: 32 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617688: xprt_alloc_slot: RPC: 33 reserved req ffff8800db829600 xid eb06d5e8 rpc.nfsd-4710 [002] .... 48.617689: xprt_prepare_transmit: RPC: 33 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617689: xprt_transmit: RPC: 33 xprt_transmit(88) rpc.nfsd-4710 [002] .... 48.617692: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4710 [002] .... 48.617692: xprt_transmit: RPC: 33 xmit complete rpcbind-1829 [000] ..s. 48.617701: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617702: xprt_complete_rqst: RPC: 33 xid eb06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617707: xprt_release: RPC: 33 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617716: xprt_alloc_slot: RPC: 34 reserved req ffff8800db829600 xid ec06d5e8 rpc.nfsd-4710 [002] .... 48.617716: xprt_prepare_transmit: RPC: 34 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617717: xprt_transmit: RPC: 34 xprt_transmit(88) rpc.nfsd-4710 [002] .... 48.617719: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4710 [002] .... 48.617719: xprt_transmit: RPC: 34 xmit complete rpcbind-1829 [000] ..s. 48.617728: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617729: xprt_complete_rqst: RPC: 34 xid ec06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617735: xprt_release: RPC: 34 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617737: xprt_alloc_slot: RPC: 35 reserved req ffff8800db829600 xid ed06d5e8 rpc.nfsd-4710 [002] .... 48.617737: xprt_prepare_transmit: RPC: 35 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617738: xprt_transmit: RPC: 35 xprt_transmit(88) rpc.nfsd-4710 [002] .... 48.617740: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4710 [002] .... 48.617740: xprt_transmit: RPC: 35 xmit complete rpcbind-1829 [000] ..s. 48.617749: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617750: xprt_complete_rqst: RPC: 35 xid ed06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617756: xprt_release: RPC: 35 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617757: xprt_alloc_slot: RPC: 36 reserved req ffff8800db829600 xid ee06d5e8 rpc.nfsd-4710 [002] .... 48.617758: xprt_prepare_transmit: RPC: 36 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617758: xprt_transmit: RPC: 36 xprt_transmit(88) rpc.nfsd-4710 [002] .... 48.617760: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4710 [002] .... 48.617760: xprt_transmit: RPC: 36 xmit complete rpcbind-1829 [000] ..s. 48.617770: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617770: xprt_complete_rqst: RPC: 36 xid ee06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617776: xprt_release: RPC: 36 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617782: xprt_alloc_slot: RPC: 37 reserved req ffff8800db829600 xid ef06d5e8 rpc.nfsd-4710 [002] .... 48.617782: xprt_prepare_transmit: RPC: 37 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617783: xprt_transmit: RPC: 37 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.617785: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.617785: xprt_transmit: RPC: 37 xmit complete rpcbind-1829 [000] ..s. 48.617794: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617795: xprt_complete_rqst: RPC: 37 xid ef06d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617800: xprt_release: RPC: 37 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617802: xprt_alloc_slot: RPC: 38 reserved req ffff8800db829600 xid f006d5e8 rpc.nfsd-4710 [002] .... 48.617802: xprt_prepare_transmit: RPC: 38 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617803: xprt_transmit: RPC: 38 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.617805: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.617805: xprt_transmit: RPC: 38 xmit complete rpcbind-1829 [000] ..s. 48.617814: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617815: xprt_complete_rqst: RPC: 38 xid f006d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617821: xprt_release: RPC: 38 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617822: xprt_alloc_slot: RPC: 39 reserved req ffff8800db829600 xid f106d5e8 rpc.nfsd-4710 [002] .... 48.617822: xprt_prepare_transmit: RPC: 39 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617823: xprt_transmit: RPC: 39 xprt_transmit(84) rpc.nfsd-4710 [002] .... 48.617825: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4710 [002] .... 48.617825: xprt_transmit: RPC: 39 xmit complete rpcbind-1829 [000] ..s. 48.617834: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617835: xprt_complete_rqst: RPC: 39 xid f106d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617840: xprt_release: RPC: 39 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617848: xprt_alloc_slot: RPC: 40 reserved req ffff8800db829600 xid f206d5e8 rpc.nfsd-4710 [002] .... 48.617849: xprt_prepare_transmit: RPC: 40 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617849: xprt_transmit: RPC: 40 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617851: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617851: xprt_transmit: RPC: 40 xmit complete rpcbind-1829 [000] ..s. 48.617860: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617861: xprt_complete_rqst: RPC: 40 xid f206d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617867: xprt_release: RPC: 40 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617868: xprt_alloc_slot: RPC: 41 reserved req ffff8800db829600 xid f306d5e8 rpc.nfsd-4710 [002] .... 48.617869: xprt_prepare_transmit: RPC: 41 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617869: xprt_transmit: RPC: 41 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617871: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617871: xprt_transmit: RPC: 41 xmit complete rpcbind-1829 [000] ..s. 48.617881: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617881: xprt_complete_rqst: RPC: 41 xid f306d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617887: xprt_release: RPC: 41 release request ffff8800db829600 rpc.nfsd-4710 [002] .... 48.617888: xprt_alloc_slot: RPC: 42 reserved req ffff8800db829600 xid f406d5e8 rpc.nfsd-4710 [002] .... 48.617889: xprt_prepare_transmit: RPC: 42 xprt_prepare_transmit rpc.nfsd-4710 [002] .... 48.617889: xprt_transmit: RPC: 42 xprt_transmit(80) rpc.nfsd-4710 [002] .... 48.617891: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4710 [002] .... 48.617891: xprt_transmit: RPC: 42 xmit complete rpcbind-1829 [000] ..s. 48.617901: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [000] ..s. 48.617902: xprt_complete_rqst: RPC: 42 xid f406d5e8 complete (28 bytes received) rpc.nfsd-4710 [002] .... 48.617907: xprt_release: RPC: 42 release request ffff8800db829600 <idle>-0 [003] ..s. 57.765235: inet_bind_hash: add 2049 <idle>-0 [003] ..s. 57.765278: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => csum_block_add_ext => __skb_gro_checksum_complete => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => try_to_wake_up => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary kworker/u32:7-118 [000] .... 57.767716: xs_setup_tcp: RPC: set up xprt to 192.168.23.22 (port 55201) via tcp kworker/u32:7-118 [000] .... 57.767726: xprt_create_transport: RPC: created transport ffff88040b251000 with 65536 slots kworker/0:1H-128 [000] .... 57.767758: xprt_alloc_slot: RPC: 43 reserved req ffff8804033c3800 xid f4185658 kworker/0:1H-128 [000] .... 57.767764: xprt_connect: RPC: 43 xprt_connect xprt ffff88040b251000 is not connected kworker/0:1H-128 [000] .... 57.767767: xs_connect: RPC: xs_connect scheduled xprt ffff88040b251000 kworker/0:1H-128 [000] ..s. 57.767780: inet_csk_get_port: snum 684 kworker/0:1H-128 [000] ..s. 57.767792: <stack trace> => inet_addr_type => inet_bind => xs_bind => sock_setsockopt => __sock_create => xs_create_sock.isra.18 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => kthread => ret_from_fork => kthread kworker/0:1H-128 [000] ..s. 57.767793: inet_bind_hash: add 684 kworker/0:1H-128 [000] ..s. 57.767801: <stack trace> => inet_csk_get_port => inet_addr_type => inet_bind => xs_bind => sock_setsockopt => __sock_create => xs_create_sock.isra.18 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => kthread => ret_from_fork => kthread kworker/0:1H-128 [000] .... 57.767803: xs_bind: RPC: xs_bind 4.136.255.255:684: ok (0) kworker/0:1H-128 [000] .... 57.767805: xs_tcp_setup_socket: RPC: worker connecting xprt ffff88040b251000 via tcp to 192.168.23.22 (port 55201) kworker/0:1H-128 [000] .... 57.767841: xs_tcp_setup_socket: RPC: ffff88040b251000 connect status 115 connected 0 sock state 2 <idle>-0 [003] ..s. 57.768178: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040b251000... <idle>-0 [003] ..s. 57.768180: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 kworker/3:1H-127 [003] .... 57.768216: xprt_connect_status: RPC: 43 xprt_connect_status: retrying kworker/3:1H-127 [003] .... 57.768218: xprt_prepare_transmit: RPC: 43 xprt_prepare_transmit kworker/3:1H-127 [003] .... 57.768229: xprt_transmit: RPC: 43 xprt_transmit(72) kworker/3:1H-127 [003] .... 57.768245: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 kworker/3:1H-127 [003] .... 57.768246: xprt_transmit: RPC: 43 xmit complete <idle>-0 [003] ..s. 57.768621: xs_tcp_data_ready: RPC: xs_tcp_data_ready... <idle>-0 [003] ..s. 57.768622: xs_tcp_data_recv: RPC: xs_tcp_data_recv started <idle>-0 [003] ..s. 57.768624: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 <idle>-0 [003] ..s. 57.768625: xs_tcp_data_recv: RPC: reading XID (4 bytes) <idle>-0 [003] ..s. 57.768626: xs_tcp_data_recv: RPC: reading request with XID f4185658 <idle>-0 [003] ..s. 57.768627: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) <idle>-0 [003] ..s. 57.768628: xs_tcp_data_recv: RPC: read reply XID f4185658 <idle>-0 [003] ..s. 57.768630: xs_tcp_data_recv: RPC: XID f4185658 read 16 bytes <idle>-0 [003] ..s. 57.768631: xs_tcp_data_recv: RPC: xprt = ffff88040b251000, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 <idle>-0 [003] ..s. 57.768632: xprt_complete_rqst: RPC: 43 xid f4185658 complete (24 bytes received) <idle>-0 [003] .Ns. 57.768637: xs_tcp_data_recv: RPC: xs_tcp_data_recv done kworker/3:1H-127 [003] .... 57.768656: xprt_release: RPC: 43 release request ffff8804033c3800 <idle>-0 [003] ..s. 96.518571: inet_bind_hash: add 10993 <idle>-0 [003] ..s. 96.518612: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => try_to_wake_up => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 138.868936: inet_bind_hash: add 22 <idle>-0 [003] ..s. 138.868978: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 144.103035: inet_bind_hash: add 22 <idle>-0 [003] ..s. 144.103078: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => try_to_wake_up => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 174.758123: inet_bind_hash: add 10993 <idle>-0 [003] ..s. 174.758151: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => try_to_wake_up => ipt_do_table => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 216.551651: inet_bind_hash: add 10993 <idle>-0 [003] ..s. 216.551689: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => try_to_wake_up => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary kworker/3:1H-127 [003] ..s. 358.800834: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040b251000... kworker/3:1H-127 [003] ..s. 358.800837: xs_tcp_state_change: RPC: state 4 conn 1 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [003] ..s. 358.801180: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040b251000... <idle>-0 [003] ..s. 358.801182: xs_tcp_state_change: RPC: state 5 conn 0 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [003] ..s. 358.801204: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040b251000... <idle>-0 [003] ..s. 358.801205: xs_tcp_state_change: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [003] ..s. 358.801206: xprt_disconnect_done: RPC: disconnected transport ffff88040b251000 <idle>-0 [003] ..s. 358.801207: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff88040b251000... <idle>-0 [003] ..s. 358.801208: xs_tcp_state_change: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [003] ..s. 358.801208: xprt_disconnect_done: RPC: disconnected transport ffff88040b251000 <idle>-0 [003] ..s. 358.801209: xs_tcp_data_ready: RPC: xs_tcp_data_ready... <idle>-0 [003] ..s. 476.855136: inet_bind_hash: add 10993 <idle>-0 [003] ..s. 476.855172: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary -- Steve ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-12 15:34 ` Steven Rostedt @ 2015-06-12 15:50 ` Steven Rostedt 2015-06-18 3:08 ` Steven Rostedt 0 siblings, 1 reply; 15+ messages in thread From: Steven Rostedt @ 2015-06-12 15:50 UTC (permalink / raw) To: Eric Dumazet Cc: Trond Myklebust, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton On Fri, 12 Jun 2015 11:34:20 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > On Fri, 12 Jun 2015 07:40:35 -0700 > Eric Dumazet <eric.dumazet@gmail.com> wrote: > > > Strange, because the usual way to not have time-wait is to use SO_LINGER > > with linger=0 > > > > And apparently xs_tcp_finish_connecting() has this : > > > > sock_reset_flag(sk, SOCK_LINGER); > > tcp_sk(sk)->linger2 = 0; > > > > Are you sure SO_REUSEADDR was not the thing you wanted ? > > > > Steven, have you tried kmemleak ? > > Nope, and again, I'm hesitant on adding too much debug. This is my main > server (build server, ssh server, web server, mail server, proxy > server, irc server, etc). > > Although, I made dprintk() into trace_printk() in xprtsock.c and > xprt.c, and reran it. Here's the output: > I reverted the following commits: c627d31ba0696cbd829437af2be2f2dee3546b1e 9e2b9f37760e129cee053cc7b6e7288acc2a7134 caf4ccd4e88cf2795c927834bc488c8321437586 And the issue goes away. That is, I watched the port go from ESTABLISHED to TIME_WAIT, and then gone, and theirs no hidden port. In fact, I watched the port with my portlist.c module, and it disappeared there too when it entered the TIME_WAIT state. Here's the trace of that run: # tracer: nop # # entries-in-buffer/entries-written: 397/397 #P:4 # # _-----=> irqs-off # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | rpc.nfsd-3932 [002] .... 44.098689: xs_local_setup_socket: RPC: worker connecting xprt ffff88040b6f5800 via AF_LOCAL to /var/run/rpcbind.sock rpc.nfsd-3932 [002] .... 44.098699: xs_local_setup_socket: RPC: xprt ffff88040b6f5800 connected to /var/run/rpcbind.sock rpc.nfsd-3932 [002] .... 44.098700: xs_setup_local: RPC: set up xprt to /var/run/rpcbind.sock via AF_LOCAL rpc.nfsd-3932 [002] .... 44.098704: xprt_create_transport: RPC: created transport ffff88040b6f5800 with 65536 slots rpc.nfsd-3932 [002] .... 44.098717: xprt_alloc_slot: RPC: 1 reserved req ffff8800d8cc6800 xid 0850084b rpc.nfsd-3932 [002] .... 44.098720: xprt_prepare_transmit: RPC: 1 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.098721: xprt_transmit: RPC: 1 xprt_transmit(44) rpc.nfsd-3932 [002] .... 44.098724: xs_local_send_request: RPC: xs_local_send_request(44) = 0 rpc.nfsd-3932 [002] .... 44.098724: xprt_transmit: RPC: 1 xmit complete rpcbind-1829 [001] ..s. 44.098812: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.098815: xprt_complete_rqst: RPC: 1 xid 0850084b complete (24 bytes received) rpc.nfsd-3932 [002] .... 44.098854: xprt_release: RPC: 1 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.098864: xprt_alloc_slot: RPC: 2 reserved req ffff8800d8cc6800 xid 0950084b rpc.nfsd-3932 [002] .... 44.098865: xprt_prepare_transmit: RPC: 2 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.098865: xprt_transmit: RPC: 2 xprt_transmit(44) rpc.nfsd-3932 [002] .... 44.098870: xs_local_send_request: RPC: xs_local_send_request(44) = 0 rpc.nfsd-3932 [002] .... 44.098870: xprt_transmit: RPC: 2 xmit complete rpcbind-1829 [001] ..s. 44.098915: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.098917: xprt_complete_rqst: RPC: 2 xid 0950084b complete (24 bytes received) rpc.nfsd-3932 [002] .... 44.098968: xprt_release: RPC: 2 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.098971: xprt_alloc_slot: RPC: 3 reserved req ffff8800d8cc6800 xid 0a50084b rpc.nfsd-3932 [002] .... 44.098972: xprt_prepare_transmit: RPC: 3 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.098973: xprt_transmit: RPC: 3 xprt_transmit(68) rpc.nfsd-3932 [002] .... 44.098978: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-3932 [002] .... 44.098978: xprt_transmit: RPC: 3 xmit complete rpcbind-1829 [001] ..s. 44.099029: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099031: xprt_complete_rqst: RPC: 3 xid 0a50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099083: xprt_release: RPC: 3 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099086: xprt_alloc_slot: RPC: 4 reserved req ffff8800d8cc6800 xid 0b50084b rpc.nfsd-3932 [002] .... 44.099086: xprt_prepare_transmit: RPC: 4 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099087: xprt_transmit: RPC: 4 xprt_transmit(68) rpc.nfsd-3932 [002] .... 44.099091: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-3932 [002] .... 44.099092: xprt_transmit: RPC: 4 xmit complete rpcbind-1829 [001] ..s. 44.099145: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099147: xprt_complete_rqst: RPC: 4 xid 0b50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099155: xprt_release: RPC: 4 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099157: xprt_alloc_slot: RPC: 5 reserved req ffff8800d8cc6800 xid 0c50084b rpc.nfsd-3932 [002] .... 44.099157: xprt_prepare_transmit: RPC: 5 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099158: xprt_transmit: RPC: 5 xprt_transmit(68) rpc.nfsd-3932 [002] .... 44.099161: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-3932 [002] .... 44.099162: xprt_transmit: RPC: 5 xmit complete rpcbind-1829 [001] ..s. 44.099172: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099173: xprt_complete_rqst: RPC: 5 xid 0c50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099180: xprt_release: RPC: 5 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099181: xprt_alloc_slot: RPC: 6 reserved req ffff8800d8cc6800 xid 0d50084b rpc.nfsd-3932 [002] .... 44.099181: xprt_prepare_transmit: RPC: 6 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099182: xprt_transmit: RPC: 6 xprt_transmit(68) rpc.nfsd-3932 [002] .... 44.099184: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-3932 [002] .... 44.099184: xprt_transmit: RPC: 6 xmit complete rpcbind-1829 [001] ..s. 44.099204: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099205: xprt_complete_rqst: RPC: 6 xid 0d50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099212: xprt_release: RPC: 6 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099213: xprt_alloc_slot: RPC: 7 reserved req ffff8800d8cc6800 xid 0e50084b rpc.nfsd-3932 [002] .... 44.099214: xprt_prepare_transmit: RPC: 7 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099214: xprt_transmit: RPC: 7 xprt_transmit(68) rpc.nfsd-3932 [002] .... 44.099217: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-3932 [002] .... 44.099217: xprt_transmit: RPC: 7 xmit complete rpcbind-1829 [001] ..s. 44.099228: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099229: xprt_complete_rqst: RPC: 7 xid 0e50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099236: xprt_release: RPC: 7 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099241: xprt_alloc_slot: RPC: 8 reserved req ffff8800d8cc6800 xid 0f50084b rpc.nfsd-3932 [002] .... 44.099241: xprt_prepare_transmit: RPC: 8 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099242: xprt_transmit: RPC: 8 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.099244: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.099244: xprt_transmit: RPC: 8 xmit complete rpcbind-1829 [001] ..s. 44.099261: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099263: xprt_complete_rqst: RPC: 8 xid 0f50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099270: xprt_release: RPC: 8 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099271: xprt_alloc_slot: RPC: 9 reserved req ffff8800d8cc6800 xid 1050084b rpc.nfsd-3932 [002] .... 44.099272: xprt_prepare_transmit: RPC: 9 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099272: xprt_transmit: RPC: 9 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.099275: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.099275: xprt_transmit: RPC: 9 xmit complete rpcbind-1829 [001] ..s. 44.099290: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099291: xprt_complete_rqst: RPC: 9 xid 1050084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099298: xprt_release: RPC: 9 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099300: xprt_alloc_slot: RPC: 10 reserved req ffff8800d8cc6800 xid 1150084b rpc.nfsd-3932 [002] .... 44.099301: xprt_prepare_transmit: RPC: 10 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099301: xprt_transmit: RPC: 10 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.099303: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.099304: xprt_transmit: RPC: 10 xmit complete rpcbind-1829 [001] ..s. 44.099318: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099320: xprt_complete_rqst: RPC: 10 xid 1150084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099327: xprt_release: RPC: 10 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099329: xprt_alloc_slot: RPC: 11 reserved req ffff8800d8cc6800 xid 1250084b rpc.nfsd-3932 [002] .... 44.099329: xprt_prepare_transmit: RPC: 11 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099330: xprt_transmit: RPC: 11 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.099332: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.099332: xprt_transmit: RPC: 11 xmit complete rpcbind-1829 [001] ..s. 44.099344: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099345: xprt_complete_rqst: RPC: 11 xid 1250084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099352: xprt_release: RPC: 11 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099354: xprt_alloc_slot: RPC: 12 reserved req ffff8800d8cc6800 xid 1350084b rpc.nfsd-3932 [002] .... 44.099354: xprt_prepare_transmit: RPC: 12 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099355: xprt_transmit: RPC: 12 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.099357: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.099357: xprt_transmit: RPC: 12 xmit complete rpcbind-1829 [001] ..s. 44.099368: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099369: xprt_complete_rqst: RPC: 12 xid 1350084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099376: xprt_release: RPC: 12 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099394: xprt_alloc_slot: RPC: 13 reserved req ffff8800d8cc6800 xid 1450084b rpc.nfsd-3932 [002] .... 44.099395: xprt_prepare_transmit: RPC: 13 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099395: xprt_transmit: RPC: 13 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.099399: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.099399: xprt_transmit: RPC: 13 xmit complete rpcbind-1829 [001] ..s. 44.099405: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099406: xprt_complete_rqst: RPC: 13 xid 1450084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099413: xprt_release: RPC: 13 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099414: xprt_alloc_slot: RPC: 14 reserved req ffff8800d8cc6800 xid 1550084b rpc.nfsd-3932 [002] .... 44.099415: xprt_prepare_transmit: RPC: 14 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099415: xprt_transmit: RPC: 14 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.099418: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.099418: xprt_transmit: RPC: 14 xmit complete rpcbind-1829 [001] ..s. 44.099424: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099425: xprt_complete_rqst: RPC: 14 xid 1550084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099431: xprt_release: RPC: 14 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099433: xprt_alloc_slot: RPC: 15 reserved req ffff8800d8cc6800 xid 1650084b rpc.nfsd-3932 [002] .... 44.099433: xprt_prepare_transmit: RPC: 15 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099434: xprt_transmit: RPC: 15 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.099436: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.099436: xprt_transmit: RPC: 15 xmit complete rpcbind-1829 [001] ..s. 44.099443: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099443: xprt_complete_rqst: RPC: 15 xid 1650084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099449: xprt_release: RPC: 15 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099451: xprt_alloc_slot: RPC: 16 reserved req ffff8800d8cc6800 xid 1750084b rpc.nfsd-3932 [002] .... 44.099451: xprt_prepare_transmit: RPC: 16 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099452: xprt_transmit: RPC: 16 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.099454: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.099455: xprt_transmit: RPC: 16 xmit complete rpcbind-1829 [001] ..s. 44.099461: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099461: xprt_complete_rqst: RPC: 16 xid 1750084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099467: xprt_release: RPC: 16 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.099469: xprt_alloc_slot: RPC: 17 reserved req ffff8800d8cc6800 xid 1850084b rpc.nfsd-3932 [002] .... 44.099469: xprt_prepare_transmit: RPC: 17 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.099470: xprt_transmit: RPC: 17 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.099472: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.099472: xprt_transmit: RPC: 17 xmit complete rpcbind-1829 [001] ..s. 44.099479: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.099479: xprt_complete_rqst: RPC: 17 xid 1850084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.099485: xprt_release: RPC: 17 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100111: xprt_alloc_slot: RPC: 18 reserved req ffff8800d8cc6800 xid 1950084b rpc.nfsd-3932 [002] .... 44.100112: xprt_prepare_transmit: RPC: 18 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100113: xprt_transmit: RPC: 18 xprt_transmit(80) rpc.nfsd-3932 [002] .... 44.100118: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-3932 [002] .... 44.100118: xprt_transmit: RPC: 18 xmit complete rpcbind-1829 [001] ..s. 44.100124: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100125: xprt_complete_rqst: RPC: 18 xid 1950084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100132: xprt_release: RPC: 18 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100134: xprt_alloc_slot: RPC: 19 reserved req ffff8800d8cc6800 xid 1a50084b rpc.nfsd-3932 [002] .... 44.100135: xprt_prepare_transmit: RPC: 19 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100135: xprt_transmit: RPC: 19 xprt_transmit(80) rpc.nfsd-3932 [002] .... 44.100138: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-3932 [002] .... 44.100138: xprt_transmit: RPC: 19 xmit complete rpcbind-1829 [001] ..s. 44.100144: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100145: xprt_complete_rqst: RPC: 19 xid 1a50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100151: xprt_release: RPC: 19 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100153: xprt_alloc_slot: RPC: 20 reserved req ffff8800d8cc6800 xid 1b50084b rpc.nfsd-3932 [002] .... 44.100153: xprt_prepare_transmit: RPC: 20 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100154: xprt_transmit: RPC: 20 xprt_transmit(80) rpc.nfsd-3932 [002] .... 44.100156: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-3932 [002] .... 44.100156: xprt_transmit: RPC: 20 xmit complete rpcbind-1829 [001] ..s. 44.100162: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100163: xprt_complete_rqst: RPC: 20 xid 1b50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100169: xprt_release: RPC: 20 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100171: xprt_alloc_slot: RPC: 21 reserved req ffff8800d8cc6800 xid 1c50084b rpc.nfsd-3932 [002] .... 44.100171: xprt_prepare_transmit: RPC: 21 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100172: xprt_transmit: RPC: 21 xprt_transmit(80) rpc.nfsd-3932 [002] .... 44.100174: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-3932 [002] .... 44.100174: xprt_transmit: RPC: 21 xmit complete rpcbind-1829 [001] ..s. 44.100180: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100181: xprt_complete_rqst: RPC: 21 xid 1c50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100187: xprt_release: RPC: 21 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100189: xprt_alloc_slot: RPC: 22 reserved req ffff8800d8cc6800 xid 1d50084b rpc.nfsd-3932 [002] .... 44.100189: xprt_prepare_transmit: RPC: 22 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100190: xprt_transmit: RPC: 22 xprt_transmit(80) rpc.nfsd-3932 [002] .... 44.100192: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-3932 [002] .... 44.100192: xprt_transmit: RPC: 22 xmit complete rpcbind-1829 [001] ..s. 44.100198: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100199: xprt_complete_rqst: RPC: 22 xid 1d50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100205: xprt_release: RPC: 22 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100223: xprt_alloc_slot: RPC: 23 reserved req ffff8800d8cc6800 xid 1e50084b rpc.nfsd-3932 [002] .... 44.100223: xprt_prepare_transmit: RPC: 23 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100224: xprt_transmit: RPC: 23 xprt_transmit(80) rpc.nfsd-3932 [002] .... 44.100227: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-3932 [002] .... 44.100227: xprt_transmit: RPC: 23 xmit complete rpcbind-1829 [001] ..s. 44.100233: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100234: xprt_complete_rqst: RPC: 23 xid 1e50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100241: xprt_release: RPC: 23 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100242: xprt_alloc_slot: RPC: 24 reserved req ffff8800d8cc6800 xid 1f50084b rpc.nfsd-3932 [002] .... 44.100243: xprt_prepare_transmit: RPC: 24 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100243: xprt_transmit: RPC: 24 xprt_transmit(80) rpc.nfsd-3932 [002] .... 44.100246: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-3932 [002] .... 44.100246: xprt_transmit: RPC: 24 xmit complete rpcbind-1829 [001] ..s. 44.100252: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100253: xprt_complete_rqst: RPC: 24 xid 1f50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100259: xprt_release: RPC: 24 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100260: xprt_alloc_slot: RPC: 25 reserved req ffff8800d8cc6800 xid 2050084b rpc.nfsd-3932 [002] .... 44.100261: xprt_prepare_transmit: RPC: 25 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100261: xprt_transmit: RPC: 25 xprt_transmit(80) rpc.nfsd-3932 [002] .... 44.100263: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-3932 [002] .... 44.100264: xprt_transmit: RPC: 25 xmit complete rpcbind-1829 [001] ..s. 44.100270: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100270: xprt_complete_rqst: RPC: 25 xid 2050084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100276: xprt_release: RPC: 25 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100278: xprt_alloc_slot: RPC: 26 reserved req ffff8800d8cc6800 xid 2150084b rpc.nfsd-3932 [002] .... 44.100278: xprt_prepare_transmit: RPC: 26 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100279: xprt_transmit: RPC: 26 xprt_transmit(80) rpc.nfsd-3932 [002] .... 44.100281: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-3932 [002] .... 44.100281: xprt_transmit: RPC: 26 xmit complete rpcbind-1829 [001] ..s. 44.100287: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100288: xprt_complete_rqst: RPC: 26 xid 2150084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100294: xprt_release: RPC: 26 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100296: xprt_alloc_slot: RPC: 27 reserved req ffff8800d8cc6800 xid 2250084b rpc.nfsd-3932 [002] .... 44.100296: xprt_prepare_transmit: RPC: 27 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100297: xprt_transmit: RPC: 27 xprt_transmit(80) rpc.nfsd-3932 [002] .... 44.100299: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-3932 [002] .... 44.100299: xprt_transmit: RPC: 27 xmit complete rpcbind-1829 [001] ..s. 44.100305: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100306: xprt_complete_rqst: RPC: 27 xid 2250084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100312: xprt_release: RPC: 27 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100403: xprt_alloc_slot: RPC: 28 reserved req ffff8800d8cc6800 xid 2350084b rpc.nfsd-3932 [002] .... 44.100404: xprt_prepare_transmit: RPC: 28 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100405: xprt_transmit: RPC: 28 xprt_transmit(68) rpc.nfsd-3932 [002] .... 44.100409: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-3932 [002] .... 44.100409: xprt_transmit: RPC: 28 xmit complete rpcbind-1829 [001] ..s. 44.100415: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100416: xprt_complete_rqst: RPC: 28 xid 2350084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100422: xprt_release: RPC: 28 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100423: xprt_alloc_slot: RPC: 29 reserved req ffff8800d8cc6800 xid 2450084b rpc.nfsd-3932 [002] .... 44.100424: xprt_prepare_transmit: RPC: 29 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100424: xprt_transmit: RPC: 29 xprt_transmit(68) rpc.nfsd-3932 [002] .... 44.100427: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-3932 [002] .... 44.100427: xprt_transmit: RPC: 29 xmit complete rpcbind-1829 [001] ..s. 44.100432: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100433: xprt_complete_rqst: RPC: 29 xid 2450084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100439: xprt_release: RPC: 29 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100440: xprt_alloc_slot: RPC: 30 reserved req ffff8800d8cc6800 xid 2550084b rpc.nfsd-3932 [002] .... 44.100441: xprt_prepare_transmit: RPC: 30 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100441: xprt_transmit: RPC: 30 xprt_transmit(68) rpc.nfsd-3932 [002] .... 44.100443: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-3932 [002] .... 44.100444: xprt_transmit: RPC: 30 xmit complete rpcbind-1829 [001] ..s. 44.100450: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100450: xprt_complete_rqst: RPC: 30 xid 2550084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100456: xprt_release: RPC: 30 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100463: xprt_alloc_slot: RPC: 31 reserved req ffff8800d8cc6800 xid 2650084b rpc.nfsd-3932 [002] .... 44.100463: xprt_prepare_transmit: RPC: 31 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100464: xprt_transmit: RPC: 31 xprt_transmit(88) rpc.nfsd-3932 [002] .... 44.100467: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-3932 [002] .... 44.100467: xprt_transmit: RPC: 31 xmit complete rpcbind-1829 [001] ..s. 44.100473: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100474: xprt_complete_rqst: RPC: 31 xid 2650084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100480: xprt_release: RPC: 31 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100482: xprt_alloc_slot: RPC: 32 reserved req ffff8800d8cc6800 xid 2750084b rpc.nfsd-3932 [002] .... 44.100482: xprt_prepare_transmit: RPC: 32 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100483: xprt_transmit: RPC: 32 xprt_transmit(88) rpc.nfsd-3932 [002] .... 44.100485: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-3932 [002] .... 44.100485: xprt_transmit: RPC: 32 xmit complete rpcbind-1829 [001] ..s. 44.100492: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100493: xprt_complete_rqst: RPC: 32 xid 2750084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100498: xprt_release: RPC: 32 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100500: xprt_alloc_slot: RPC: 33 reserved req ffff8800d8cc6800 xid 2850084b rpc.nfsd-3932 [002] .... 44.100501: xprt_prepare_transmit: RPC: 33 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100501: xprt_transmit: RPC: 33 xprt_transmit(88) rpc.nfsd-3932 [002] .... 44.100504: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-3932 [002] .... 44.100504: xprt_transmit: RPC: 33 xmit complete rpcbind-1829 [001] ..s. 44.100510: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100511: xprt_complete_rqst: RPC: 33 xid 2850084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100517: xprt_release: RPC: 33 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100526: xprt_alloc_slot: RPC: 34 reserved req ffff8800d8cc6800 xid 2950084b rpc.nfsd-3932 [002] .... 44.100527: xprt_prepare_transmit: RPC: 34 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100528: xprt_transmit: RPC: 34 xprt_transmit(88) rpc.nfsd-3932 [002] .... 44.100530: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-3932 [002] .... 44.100531: xprt_transmit: RPC: 34 xmit complete rpcbind-1829 [001] ..s. 44.100537: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100538: xprt_complete_rqst: RPC: 34 xid 2950084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100544: xprt_release: RPC: 34 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100546: xprt_alloc_slot: RPC: 35 reserved req ffff8800d8cc6800 xid 2a50084b rpc.nfsd-3932 [002] .... 44.100546: xprt_prepare_transmit: RPC: 35 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100547: xprt_transmit: RPC: 35 xprt_transmit(88) rpc.nfsd-3932 [002] .... 44.100549: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-3932 [002] .... 44.100549: xprt_transmit: RPC: 35 xmit complete rpcbind-1829 [001] ..s. 44.100556: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100556: xprt_complete_rqst: RPC: 35 xid 2a50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100562: xprt_release: RPC: 35 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100564: xprt_alloc_slot: RPC: 36 reserved req ffff8800d8cc6800 xid 2b50084b rpc.nfsd-3932 [002] .... 44.100565: xprt_prepare_transmit: RPC: 36 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100565: xprt_transmit: RPC: 36 xprt_transmit(88) rpc.nfsd-3932 [002] .... 44.100567: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-3932 [002] .... 44.100568: xprt_transmit: RPC: 36 xmit complete rpcbind-1829 [001] ..s. 44.100574: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100575: xprt_complete_rqst: RPC: 36 xid 2b50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100581: xprt_release: RPC: 36 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100587: xprt_alloc_slot: RPC: 37 reserved req ffff8800d8cc6800 xid 2c50084b rpc.nfsd-3932 [002] .... 44.100587: xprt_prepare_transmit: RPC: 37 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100588: xprt_transmit: RPC: 37 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.100590: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.100590: xprt_transmit: RPC: 37 xmit complete rpcbind-1829 [001] ..s. 44.100597: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100597: xprt_complete_rqst: RPC: 37 xid 2c50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100604: xprt_release: RPC: 37 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100605: xprt_alloc_slot: RPC: 38 reserved req ffff8800d8cc6800 xid 2d50084b rpc.nfsd-3932 [002] .... 44.100606: xprt_prepare_transmit: RPC: 38 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100606: xprt_transmit: RPC: 38 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.100608: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.100609: xprt_transmit: RPC: 38 xmit complete rpcbind-1829 [001] ..s. 44.100615: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100616: xprt_complete_rqst: RPC: 38 xid 2d50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100622: xprt_release: RPC: 38 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100623: xprt_alloc_slot: RPC: 39 reserved req ffff8800d8cc6800 xid 2e50084b rpc.nfsd-3932 [002] .... 44.100624: xprt_prepare_transmit: RPC: 39 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100624: xprt_transmit: RPC: 39 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.100626: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.100627: xprt_transmit: RPC: 39 xmit complete rpcbind-1829 [001] ..s. 44.100633: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100634: xprt_complete_rqst: RPC: 39 xid 2e50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100640: xprt_release: RPC: 39 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100647: xprt_alloc_slot: RPC: 40 reserved req ffff8800d8cc6800 xid 2f50084b rpc.nfsd-3932 [002] .... 44.100648: xprt_prepare_transmit: RPC: 40 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100648: xprt_transmit: RPC: 40 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.100651: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.100651: xprt_transmit: RPC: 40 xmit complete rpcbind-1829 [001] ..s. 44.100657: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100658: xprt_complete_rqst: RPC: 40 xid 2f50084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100664: xprt_release: RPC: 40 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100666: xprt_alloc_slot: RPC: 41 reserved req ffff8800d8cc6800 xid 3050084b rpc.nfsd-3932 [002] .... 44.100666: xprt_prepare_transmit: RPC: 41 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100667: xprt_transmit: RPC: 41 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.100669: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.100669: xprt_transmit: RPC: 41 xmit complete rpcbind-1829 [001] ..s. 44.100675: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100676: xprt_complete_rqst: RPC: 41 xid 3050084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100682: xprt_release: RPC: 41 release request ffff8800d8cc6800 rpc.nfsd-3932 [002] .... 44.100684: xprt_alloc_slot: RPC: 42 reserved req ffff8800d8cc6800 xid 3150084b rpc.nfsd-3932 [002] .... 44.100684: xprt_prepare_transmit: RPC: 42 xprt_prepare_transmit rpc.nfsd-3932 [002] .... 44.100685: xprt_transmit: RPC: 42 xprt_transmit(84) rpc.nfsd-3932 [002] .... 44.100687: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-3932 [002] .... 44.100687: xprt_transmit: RPC: 42 xmit complete rpcbind-1829 [001] ..s. 44.100693: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1829 [001] ..s. 44.100694: xprt_complete_rqst: RPC: 42 xid 3150084b complete (28 bytes received) rpc.nfsd-3932 [002] .... 44.100700: xprt_release: RPC: 42 release request ffff8800d8cc6800 <idle>-0 [003] ..s. 52.302416: inet_bind_hash: add 22 <idle>-0 [003] ..s. 52.302456: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => delay_tsc => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => ack_ioapic_level => do_IRQ => net_rx_action => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary kworker/u32:2-105 [001] .... 77.750302: xs_setup_tcp: RPC: set up xprt to 192.168.23.22 (port 55201) via tcp kworker/u32:2-105 [001] .... 77.750310: xprt_create_transport: RPC: created transport ffff8804082fb000 with 65536 slots kworker/1:1H-133 [001] .... 77.750352: xprt_alloc_slot: RPC: 43 reserved req ffff88040ab08200 xid 83da2dc3 kworker/1:1H-133 [001] .... 77.750356: xprt_connect: RPC: 43 xprt_connect xprt ffff8804082fb000 is not connected kworker/1:1H-133 [001] .... 77.750358: xs_connect: RPC: xs_connect scheduled xprt ffff8804082fb000 kworker/1:1H-133 [001] ..s. 77.750365: inet_csk_get_port: snum 737 kworker/1:1H-133 [001] ..s. 77.750374: <stack trace> => inet_addr_type => inet_bind => xs_bind => sock_setsockopt => __sock_create => xs_create_sock.isra.19 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => kthread => ret_from_fork => kthread kworker/1:1H-133 [001] ..s. 77.750374: inet_bind_hash: add 737 kworker/1:1H-133 [001] ..s. 77.750377: <stack trace> => inet_csk_get_port => inet_addr_type => inet_bind => xs_bind => sock_setsockopt => __sock_create => xs_create_sock.isra.19 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => kthread => ret_from_fork => kthread kworker/1:1H-133 [001] .... 77.750378: xs_bind: RPC: xs_bind 4.136.255.255:737: ok (0) kworker/1:1H-133 [001] .... 77.750379: xs_tcp_setup_socket: RPC: worker connecting xprt ffff8804082fb000 via tcp to 192.168.23.22 (port 55201) kworker/1:1H-133 [001] .... 77.750397: xs_tcp_setup_socket: xprt=ffff8804082fb000 sock=ffff880408a47d40 status=-115 kworker/1:1H-133 [001] .... 77.750397: xs_tcp_setup_socket: RPC: ffff8804082fb000 connect status 115 connected 0 sock state 2 fail2ban-server-4683 [002] ..s. 77.750554: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8804082fb000... fail2ban-server-4683 [002] ..s. 77.750555: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 kworker/2:1H-126 [002] .... 77.750571: xprt_connect_status: RPC: 43 xprt_connect_status: retrying kworker/2:1H-126 [002] .... 77.750572: xprt_prepare_transmit: RPC: 43 xprt_prepare_transmit kworker/2:1H-126 [002] .... 77.750573: xprt_transmit: RPC: 43 xprt_transmit(72) kworker/2:1H-126 [002] .... 77.750581: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 kworker/2:1H-126 [002] .... 77.750581: xprt_transmit: RPC: 43 xmit complete fail2ban-server-4683 [002] ..s. 77.750798: xs_tcp_data_ready: RPC: xs_tcp_data_ready... fail2ban-server-4683 [002] ..s. 77.750799: xs_tcp_data_recv: RPC: xs_tcp_data_recv started fail2ban-server-4683 [002] ..s. 77.750800: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 fail2ban-server-4683 [002] ..s. 77.750800: xs_tcp_data_recv: RPC: reading XID (4 bytes) fail2ban-server-4683 [002] ..s. 77.750801: xs_tcp_data_recv: RPC: reading request with XID 83da2dc3 fail2ban-server-4683 [002] ..s. 77.750801: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) fail2ban-server-4683 [002] ..s. 77.750801: xs_tcp_data_recv: RPC: read reply XID 83da2dc3 fail2ban-server-4683 [002] ..s. 77.750802: xs_tcp_data_recv: RPC: XID 83da2dc3 read 16 bytes fail2ban-server-4683 [002] ..s. 77.750803: xs_tcp_data_recv: RPC: xprt = ffff8804082fb000, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 fail2ban-server-4683 [002] ..s. 77.750803: xprt_complete_rqst: RPC: 43 xid 83da2dc3 complete (24 bytes received) fail2ban-server-4683 [002] .Ns. 77.750805: xs_tcp_data_recv: RPC: xs_tcp_data_recv done kworker/2:1H-126 [002] .... 77.750813: xprt_release: RPC: 43 release request ffff88040ab08200 <idle>-0 [003] ..s. 94.613312: inet_bind_hash: add 22 <idle>-0 [003] ..s. 94.613354: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 98.776868: inet_bind_hash: add 10993 <idle>-0 [003] ..s. 98.776910: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 151.179778: inet_bind_hash: add 80 <idle>-0 [003] ..s. 151.179822: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_v6_syn_recv_sock => ipt_do_table => nf_conntrack_in => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => try_to_wake_up => ktime_get => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 172.217453: inet_bind_hash: add 10993 <idle>-0 [003] ..s. 172.217496: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] .Ns. 181.603150: inet_bind_hash: add 80 <idle>-0 [003] .Ns. 181.603194: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_v6_syn_recv_sock => ipt_do_table => nf_conntrack_in => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => try_to_wake_up => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 234.638237: inet_bind_hash: add 10993 <idle>-0 [003] ..s. 234.638281: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => try_to_wake_up => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 241.694872: inet_bind_hash: add 57000 <idle>-0 [003] ..s. 241.694915: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => try_to_wake_up => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 242.308627: inet_bind_hash: add 10993 <idle>-0 [003] ..s. 242.308670: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_check_req => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => try_to_wake_up => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 296.125499: inet_bind_hash: add 80 <idle>-0 [003] ..s. 296.125543: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_v6_syn_recv_sock => ipt_do_table => nf_conntrack_in => tcp_check_req => fib_validate_source => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary <idle>-0 [003] ..s. 304.196576: inet_bind_hash: add 80 <idle>-0 [003] ..s. 304.196618: <stack trace> => __inet_inherit_port => tcp_v4_syn_recv_sock => tcp_v6_syn_recv_sock => ipt_do_table => nf_conntrack_in => tcp_check_req => fib_validate_source => tcp_v4_do_rcv => tcp_v4_rcv => ip_local_deliver_finish => __netif_receive_skb_core => netif_receive_skb => netif_receive_skb_internal => br_handle_frame_finish => br_handle_frame => br_handle_frame => __netif_receive_skb_core => read_tsc => netif_receive_skb_internal => napi_gro_receive => e1000_clean_rx_irq => e1000_clean => raise_softirq_irqoff => netif_schedule_queue => dev_watchdog => net_rx_action => add_interrupt_randomness => __do_softirq => ack_ioapic_level => irq_exit => do_IRQ => ret_from_intr => cpuidle_enter_state => cpuidle_enter_state => cpu_startup_entry => start_secondary kworker/2:1H-126 [002] .... 378.264745: xs_tcp_close: close %p kworker/2:1H-126 [002] .... 378.264748: xs_close: RPC: xs_close xprt ffff8804082fb000 kworker/2:1H-126 [002] .... 378.264786: xprt_disconnect_done: RPC: disconnected transport ffff8804082fb000 -- Steve ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-12 15:50 ` Steven Rostedt @ 2015-06-18 3:08 ` Steven Rostedt 2015-06-18 19:24 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Steven Rostedt @ 2015-06-18 3:08 UTC (permalink / raw) To: Eric Dumazet Cc: Trond Myklebust, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton On Fri, 12 Jun 2015 11:50:38 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > I reverted the following commits: > > c627d31ba0696cbd829437af2be2f2dee3546b1e > 9e2b9f37760e129cee053cc7b6e7288acc2a7134 > caf4ccd4e88cf2795c927834bc488c8321437586 > > And the issue goes away. That is, I watched the port go from > ESTABLISHED to TIME_WAIT, and then gone, and theirs no hidden port. > > In fact, I watched the port with my portlist.c module, and it > disappeared there too when it entered the TIME_WAIT state. > I've been running v4.0.5 with the above commits reverted for 5 days now, and there's still no hidden port appearing. What's the status on this? Should those commits be reverted or is there another solution to this bug? -- Steve ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-18 3:08 ` Steven Rostedt @ 2015-06-18 19:24 ` Trond Myklebust 2015-06-18 19:49 ` Steven Rostedt 0 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2015-06-18 19:24 UTC (permalink / raw) To: Steven Rostedt Cc: Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton On Wed, Jun 17, 2015 at 11:08 PM, Steven Rostedt <rostedt@goodmis.org> wrote: > On Fri, 12 Jun 2015 11:50:38 -0400 > Steven Rostedt <rostedt@goodmis.org> wrote: > >> I reverted the following commits: >> >> c627d31ba0696cbd829437af2be2f2dee3546b1e >> 9e2b9f37760e129cee053cc7b6e7288acc2a7134 >> caf4ccd4e88cf2795c927834bc488c8321437586 >> >> And the issue goes away. That is, I watched the port go from >> ESTABLISHED to TIME_WAIT, and then gone, and theirs no hidden port. >> >> In fact, I watched the port with my portlist.c module, and it >> disappeared there too when it entered the TIME_WAIT state. >> I've scanned those commits again and again, and I'm not seeing how we could be introducing a socket leak there. The only suspect I can see would be the NFS swap bugs that Jeff fixed a few weeks ago. Are you using NFS swap? > I've been running v4.0.5 with the above commits reverted for 5 days > now, and there's still no hidden port appearing. > > What's the status on this? Should those commits be reverted or is there > another solution to this bug? > I'm trying to reproduce, but I've had no luck yet. Cheers Trond ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-18 19:24 ` Trond Myklebust @ 2015-06-18 19:49 ` Steven Rostedt 2015-06-18 22:50 ` Jeff Layton 0 siblings, 1 reply; 15+ messages in thread From: Steven Rostedt @ 2015-06-18 19:49 UTC (permalink / raw) To: Trond Myklebust Cc: Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton [-- Attachment #1: Type: text/plain, Size: 1845 bytes --] On Thu, 18 Jun 2015 15:24:52 -0400 Trond Myklebust <trond.myklebust@primarydata.com> wrote: > On Wed, Jun 17, 2015 at 11:08 PM, Steven Rostedt <rostedt@goodmis.org> wrote: > > On Fri, 12 Jun 2015 11:50:38 -0400 > > Steven Rostedt <rostedt@goodmis.org> wrote: > > > >> I reverted the following commits: > >> > >> c627d31ba0696cbd829437af2be2f2dee3546b1e > >> 9e2b9f37760e129cee053cc7b6e7288acc2a7134 > >> caf4ccd4e88cf2795c927834bc488c8321437586 > >> > >> And the issue goes away. That is, I watched the port go from > >> ESTABLISHED to TIME_WAIT, and then gone, and theirs no hidden port. > >> > >> In fact, I watched the port with my portlist.c module, and it > >> disappeared there too when it entered the TIME_WAIT state. > >> > > I've scanned those commits again and again, and I'm not seeing how we > could be introducing a socket leak there. The only suspect I can see > would be the NFS swap bugs that Jeff fixed a few weeks ago. Are you > using NFS swap? Not that I'm aware of. > > > I've been running v4.0.5 with the above commits reverted for 5 days > > now, and there's still no hidden port appearing. > > > > What's the status on this? Should those commits be reverted or is there > > another solution to this bug? > > > > I'm trying to reproduce, but I've had no luck yet. It seems to happen with the connection to my wife's machine, and that is where my wife's box connects two directories via nfs: This is what's in my wife's /etc/fstab directory goliath:/home/upload /upload nfs auto,rw,intr,soft 0 0 goliath:/home/gallery /gallery nfs auto,ro,intr,soft 0 0 And here's what's in my /etc/exports directory /home/upload wife(no_root_squash,no_all_squash,rw,sync,no_subtree_check) /home/gallery 192.168.23.0/24(ro,sync,no_subtree_check) Attached is my config. -- Steve [-- Attachment #2: config.gz --] [-- Type: application/gzip, Size: 40609 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-18 19:49 ` Steven Rostedt @ 2015-06-18 22:50 ` Jeff Layton 2015-06-19 1:08 ` Steven Rostedt 0 siblings, 1 reply; 15+ messages in thread From: Jeff Layton @ 2015-06-18 22:50 UTC (permalink / raw) To: Steven Rostedt Cc: Trond Myklebust, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton On Thu, 18 Jun 2015 15:49:14 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > On Thu, 18 Jun 2015 15:24:52 -0400 > Trond Myklebust <trond.myklebust@primarydata.com> wrote: > > > On Wed, Jun 17, 2015 at 11:08 PM, Steven Rostedt <rostedt@goodmis.org> wrote: > > > On Fri, 12 Jun 2015 11:50:38 -0400 > > > Steven Rostedt <rostedt@goodmis.org> wrote: > > > > > >> I reverted the following commits: > > >> > > >> c627d31ba0696cbd829437af2be2f2dee3546b1e > > >> 9e2b9f37760e129cee053cc7b6e7288acc2a7134 > > >> caf4ccd4e88cf2795c927834bc488c8321437586 > > >> > > >> And the issue goes away. That is, I watched the port go from > > >> ESTABLISHED to TIME_WAIT, and then gone, and theirs no hidden port. > > >> > > >> In fact, I watched the port with my portlist.c module, and it > > >> disappeared there too when it entered the TIME_WAIT state. > > >> > > > > I've scanned those commits again and again, and I'm not seeing how we > > could be introducing a socket leak there. The only suspect I can see > > would be the NFS swap bugs that Jeff fixed a few weeks ago. Are you > > using NFS swap? > > Not that I'm aware of. > > > > > > I've been running v4.0.5 with the above commits reverted for 5 days > > > now, and there's still no hidden port appearing. > > > > > > What's the status on this? Should those commits be reverted or is there > > > another solution to this bug? > > > > > > > I'm trying to reproduce, but I've had no luck yet. > > It seems to happen with the connection to my wife's machine, and that > is where my wife's box connects two directories via nfs: > > This is what's in my wife's /etc/fstab directory > > goliath:/home/upload /upload nfs auto,rw,intr,soft 0 0 > goliath:/home/gallery /gallery nfs auto,ro,intr,soft 0 0 > > And here's what's in my /etc/exports directory > > /home/upload wife(no_root_squash,no_all_squash,rw,sync,no_subtree_check) > /home/gallery 192.168.23.0/24(ro,sync,no_subtree_check) > > Attached is my config. > The interesting bit here is that the sockets all seem to connect to port 55201 on the remote host, if I'm reading these traces correctly. What's listening on that port on the server? This might give some helpful info: $ rpcinfo -p <NFS servername> Also, what NFS version are you using to mount here? Your fstab entries suggest that you're using the default version (for whatever distro this is), but have you (e.g.) set up nfsmount.conf to default to v3 on this box? -- Jeff Layton <jlayton@poochiereds.net> ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-18 22:50 ` Jeff Layton @ 2015-06-19 1:08 ` Steven Rostedt 2015-06-19 1:37 ` Jeff Layton 0 siblings, 1 reply; 15+ messages in thread From: Steven Rostedt @ 2015-06-19 1:08 UTC (permalink / raw) To: Jeff Layton Cc: Trond Myklebust, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton On Thu, 18 Jun 2015 18:50:51 -0400 Jeff Layton <jlayton@poochiereds.net> wrote: > The interesting bit here is that the sockets all seem to connect to port > 55201 on the remote host, if I'm reading these traces correctly. What's > listening on that port on the server? > > This might give some helpful info: > > $ rpcinfo -p <NFS servername> # rpcinfo -p wife program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 34243 status 100024 1 tcp 34498 status # rpcinfo -p localhost program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 38332 status 100024 1 tcp 52684 status 100003 2 tcp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 tcp 2049 nfs 100227 2 tcp 2049 100227 3 tcp 2049 100003 2 udp 2049 nfs 100003 3 udp 2049 nfs 100003 4 udp 2049 nfs 100227 2 udp 2049 100227 3 udp 2049 100021 1 udp 53218 nlockmgr 100021 3 udp 53218 nlockmgr 100021 4 udp 53218 nlockmgr 100021 1 tcp 49825 nlockmgr 100021 3 tcp 49825 nlockmgr 100021 4 tcp 49825 nlockmgr 100005 1 udp 49166 mountd 100005 1 tcp 48797 mountd 100005 2 udp 47856 mountd 100005 2 tcp 53839 mountd 100005 3 udp 36090 mountd 100005 3 tcp 46390 mountd Note, the box has been rebooted since I posted my last trace. > > Also, what NFS version are you using to mount here? Your fstab entries > suggest that you're using the default version (for whatever distro this > is), but have you (e.g.) set up nfsmount.conf to default to v3 on this > box? > My box is Debian testing (recently updated). # dpkg -l nfs-* ii nfs-common 1:1.2.8-9 amd64 NFS support files common to clien ii nfs-kernel-ser 1:1.2.8-9 amd64 support for NFS kernel server same for both boxes. nfsmount.conf doesn't exist on either box. I'm assuming it is using nfs4. Anything else I can provide? -- Steve ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-19 1:08 ` Steven Rostedt @ 2015-06-19 1:37 ` Jeff Layton 2015-06-19 16:25 ` Steven Rostedt 0 siblings, 1 reply; 15+ messages in thread From: Jeff Layton @ 2015-06-19 1:37 UTC (permalink / raw) To: Steven Rostedt Cc: Trond Myklebust, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, bfields On Thu, 18 Jun 2015 21:08:43 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > On Thu, 18 Jun 2015 18:50:51 -0400 > Jeff Layton <jlayton@poochiereds.net> wrote: > > > The interesting bit here is that the sockets all seem to connect to port > > 55201 on the remote host, if I'm reading these traces correctly. What's > > listening on that port on the server? > > > > This might give some helpful info: > > > > $ rpcinfo -p <NFS servername> > > # rpcinfo -p wife > program vers proto port service > 100000 4 tcp 111 portmapper > 100000 3 tcp 111 portmapper > 100000 2 tcp 111 portmapper > 100000 4 udp 111 portmapper > 100000 3 udp 111 portmapper > 100000 2 udp 111 portmapper > 100024 1 udp 34243 status > 100024 1 tcp 34498 status > > # rpcinfo -p localhost > program vers proto port service > 100000 4 tcp 111 portmapper > 100000 3 tcp 111 portmapper > 100000 2 tcp 111 portmapper > 100000 4 udp 111 portmapper > 100000 3 udp 111 portmapper > 100000 2 udp 111 portmapper > 100024 1 udp 38332 status > 100024 1 tcp 52684 status > 100003 2 tcp 2049 nfs > 100003 3 tcp 2049 nfs > 100003 4 tcp 2049 nfs > 100227 2 tcp 2049 > 100227 3 tcp 2049 > 100003 2 udp 2049 nfs > 100003 3 udp 2049 nfs > 100003 4 udp 2049 nfs > 100227 2 udp 2049 > 100227 3 udp 2049 > 100021 1 udp 53218 nlockmgr > 100021 3 udp 53218 nlockmgr > 100021 4 udp 53218 nlockmgr > 100021 1 tcp 49825 nlockmgr > 100021 3 tcp 49825 nlockmgr > 100021 4 tcp 49825 nlockmgr > 100005 1 udp 49166 mountd > 100005 1 tcp 48797 mountd > 100005 2 udp 47856 mountd > 100005 2 tcp 53839 mountd > 100005 3 udp 36090 mountd > 100005 3 tcp 46390 mountd > > Note, the box has been rebooted since I posted my last trace. > Ahh pity. The port has probably changed...if you trace it again maybe try to figure out what it's talking to before rebooting the server? > > > > Also, what NFS version are you using to mount here? Your fstab entries > > suggest that you're using the default version (for whatever distro this > > is), but have you (e.g.) set up nfsmount.conf to default to v3 on this > > box? > > > > My box is Debian testing (recently updated). > > # dpkg -l nfs-* > > ii nfs-common 1:1.2.8-9 amd64 NFS support files common to clien > ii nfs-kernel-ser 1:1.2.8-9 amd64 support for NFS kernel server > > > same for both boxes. > > nfsmount.conf doesn't exist on either box. > > I'm assuming it is using nfs4. > (cc'ing Bruce) Oh! I was thinking that you were seeing this extra port on the _client_, but now rereading your original mail I see that it's appearing up on the NFS server. Is that correct? So, assuming that this is NFSv4.0, then this port is probably bound when the server is establishing the callback channel to the client. So we may need to look at how those xprts are being created and whether there are differences from a standard client xprt. -- Jeff Layton <jlayton@poochiereds.net> ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-19 1:37 ` Jeff Layton @ 2015-06-19 16:25 ` Steven Rostedt 2015-06-19 17:17 ` Steven Rostedt 0 siblings, 1 reply; 15+ messages in thread From: Steven Rostedt @ 2015-06-19 16:25 UTC (permalink / raw) To: Jeff Layton Cc: Trond Myklebust, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, bfields [-- Attachment #1: Type: text/plain, Size: 218400 bytes --] On Thu, 18 Jun 2015 21:37:02 -0400 Jeff Layton <jlayton@poochiereds.net> wrote: > > Note, the box has been rebooted since I posted my last trace. > > > > Ahh pity. The port has probably changed...if you trace it again maybe > try to figure out what it's talking to before rebooting the server? > OK, I ran it again. Here's exactly what I did: I reverted my revert and applied the attached patch. I built and rebooted the box (with the same config) an then I waited till I saw the kworker in my trace: # grep kworker /debug/tracing/trace Once I found it, I noted the port that it bounded to. kworker/1:1H-131 [001] ..s. 149.230212: inet_csk_get_port: kworker/1:1H:131 got port 947 Note, unhide-tcp didn't show any issues. I saved the rpcinfo of both my box and my wifes box: # rpcinfo -p localhost program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 33043 status 100024 1 tcp 53880 status 100003 2 tcp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 tcp 2049 nfs 100227 2 tcp 2049 100227 3 tcp 2049 100003 2 udp 2049 nfs 100003 3 udp 2049 nfs 100003 4 udp 2049 nfs 100227 2 udp 2049 100227 3 udp 2049 100021 1 udp 39455 nlockmgr 100021 3 udp 39455 nlockmgr 100021 4 udp 39455 nlockmgr 100021 1 tcp 48916 nlockmgr 100021 3 tcp 48916 nlockmgr 100021 4 tcp 48916 nlockmgr 100005 1 udp 58465 mountd 100005 1 tcp 56391 mountd 100005 2 udp 35741 mountd 100005 2 tcp 40520 mountd 100005 3 udp 56522 mountd 100005 3 tcp 33464 mountd # rpcinfo -p wife program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 34243 status 100024 1 tcp 34498 status and ran: while :; do netstat -tapn |grep 947; sleep 1; done Waited for the state to turn from ESTABLISHED to TIME_WAIT, and then I ran the rpcinfo again, but they didn't change. I checked for hidden ports, but none were listed (yet). I then waited for the port to disappear. I ran the rpcinfo again, but it still didn't change. But unhide-tpc reports: Found Hidden port that not appears in ss: 947 Here's that trace: # tracer: nop # # entries-in-buffer/entries-written: 1978/1978 #P:4 # # _-----=> irqs-off # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | modprobe-1904 [002] .... 22.119972: rpc_init_mempool: RPC: creating workqueue rpciod mount-1912 [002] .... 22.305981: rpc_fill_super: RPC: sending pipefs MOUNT notification for net ffffffff818b9780 (init_net) rpc.nfsd-4720 [001] .... 50.855600: xs_local_setup_socket: RPC: worker connecting xprt ffff880407939800 via AF_LOCAL to /var/run/rpcbind.sock rpc.nfsd-4720 [001] .... 50.855609: xs_local_setup_socket: RPC: xprt ffff880407939800 connected to /var/run/rpcbind.sock rpc.nfsd-4720 [001] .... 50.855610: xs_setup_local: RPC: set up xprt to /var/run/rpcbind.sock via AF_LOCAL rpc.nfsd-4720 [001] .... 50.855614: xprt_create_transport: RPC: created transport ffff880407939800 with 65536 slots rpc.nfsd-4720 [001] .... 50.855614: rpc_new_client: RPC: creating rpcbind client for localhost (xprt ffff880407939800) rpc.nfsd-4720 [001] .... 50.855625: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.855626: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.855627: __rpc_execute: RPC: 1 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.855628: call_start: RPC: 1 call_start rpcbind2 proc NULL (sync) rpc.nfsd-4720 [001] .... 50.855628: call_reserve: RPC: 1 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.855629: xprt_alloc_slot: RPC: 1 reserved req ffff880403542200 xid 3a45b0ec rpc.nfsd-4720 [001] .... 50.855629: call_reserveresult: RPC: 1 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.855630: call_refresh: RPC: 1 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.855631: call_refreshresult: RPC: 1 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.855631: call_allocate: RPC: 1 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.855632: rpc_malloc: RPC: 1 allocated buffer of size 96 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.855633: call_bind: RPC: 1 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.855633: call_connect: RPC: 1 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.855634: call_transmit: RPC: 1 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.855634: xprt_prepare_transmit: RPC: 1 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.855635: call_transmit: RPC: 1 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.855635: xprt_transmit: RPC: 1 xprt_transmit(44) rpc.nfsd-4720 [001] .... 50.855638: xs_local_send_request: RPC: xs_local_send_request(44) = 0 rpc.nfsd-4720 [001] .... 50.855638: xprt_transmit: RPC: 1 xmit complete rpc.nfsd-4720 [001] ..s. 50.855639: __rpc_sleep_on_priority: RPC: 1 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.855640: __rpc_sleep_on_priority: RPC: 1 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.855640: __rpc_sleep_on_priority: RPC: 1 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.855641: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.855642: __rpc_execute: RPC: 1 sync task going to sleep rpcbind-1871 [003] ..s. 50.855723: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.855725: xprt_complete_rqst: RPC: 1 xid 3a45b0ec complete (24 bytes received) rpcbind-1871 [003] ..s. 50.855726: rpc_wake_up_task_queue_locked: RPC: 1 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.855726: rpc_wake_up_task_queue_locked: RPC: 1 disabling timer rpcbind-1871 [003] ..s. 50.855727: rpc_wake_up_task_queue_locked: RPC: 1 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.855729: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.855768: __rpc_execute: RPC: 1 sync task resuming rpc.nfsd-4720 [001] .... 50.855770: call_status: RPC: 1 call_status (status 24) rpc.nfsd-4720 [001] .... 50.855771: call_decode: RPC: 1 call_decode (status 24) rpc.nfsd-4720 [001] .... 50.855773: call_decode: RPC: 1 call_decode result 0 rpc.nfsd-4720 [001] .... 50.855774: __rpc_execute: RPC: 1 return 0, status 0 rpc.nfsd-4720 [001] .... 50.855774: __rpc_execute: RPC: 1 release task rpc.nfsd-4720 [001] .... 50.855776: rpc_free: RPC: freeing buffer of size 96 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.855777: xprt_release: RPC: 1 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.855778: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.855779: rpc_release_client: RPC: rpc_release_client(ffff8800d8e62c00) rpc.nfsd-4720 [001] .... 50.855781: rpc_free_task: RPC: 1 freeing task rpc.nfsd-4720 [001] .... 50.855782: rpc_new_client: RPC: creating rpcbind client for localhost (xprt ffff880407939800) rpc.nfsd-4720 [001] .... 50.855795: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.855796: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.855797: __rpc_execute: RPC: 2 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.855798: call_start: RPC: 2 call_start rpcbind4 proc NULL (sync) rpc.nfsd-4720 [001] .... 50.855799: call_reserve: RPC: 2 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.855800: xprt_alloc_slot: RPC: 2 reserved req ffff880403542200 xid 3b45b0ec rpc.nfsd-4720 [001] .... 50.855801: call_reserveresult: RPC: 2 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.855801: call_refresh: RPC: 2 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.855802: call_refreshresult: RPC: 2 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.855803: call_allocate: RPC: 2 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.855804: rpc_malloc: RPC: 2 allocated buffer of size 96 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.855805: call_bind: RPC: 2 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.855806: call_connect: RPC: 2 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.855806: call_transmit: RPC: 2 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.855807: xprt_prepare_transmit: RPC: 2 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.855808: call_transmit: RPC: 2 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.855808: xprt_transmit: RPC: 2 xprt_transmit(44) rpc.nfsd-4720 [001] .... 50.855817: xs_local_send_request: RPC: xs_local_send_request(44) = 0 rpc.nfsd-4720 [001] .... 50.855817: xprt_transmit: RPC: 2 xmit complete rpc.nfsd-4720 [001] ..s. 50.855818: __rpc_sleep_on_priority: RPC: 2 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.855819: __rpc_sleep_on_priority: RPC: 2 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.855820: __rpc_sleep_on_priority: RPC: 2 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.855822: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.855822: __rpc_execute: RPC: 2 sync task going to sleep rpcbind-1871 [003] ..s. 50.855866: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.855868: xprt_complete_rqst: RPC: 2 xid 3b45b0ec complete (24 bytes received) rpcbind-1871 [003] ..s. 50.855869: rpc_wake_up_task_queue_locked: RPC: 2 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.855870: rpc_wake_up_task_queue_locked: RPC: 2 disabling timer rpcbind-1871 [003] ..s. 50.855871: rpc_wake_up_task_queue_locked: RPC: 2 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.855875: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.855910: __rpc_execute: RPC: 2 sync task resuming rpc.nfsd-4720 [001] .... 50.855911: call_status: RPC: 2 call_status (status 24) rpc.nfsd-4720 [001] .... 50.855912: call_decode: RPC: 2 call_decode (status 24) rpc.nfsd-4720 [001] .... 50.855914: call_decode: RPC: 2 call_decode result 0 rpc.nfsd-4720 [001] .... 50.855914: __rpc_execute: RPC: 2 return 0, status 0 rpc.nfsd-4720 [001] .... 50.855915: __rpc_execute: RPC: 2 release task rpc.nfsd-4720 [001] .... 50.855916: rpc_free: RPC: freeing buffer of size 96 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.855917: xprt_release: RPC: 2 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.855918: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.855919: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.855920: rpc_free_task: RPC: 2 freeing task rpc.nfsd-4720 [001] .... 50.855922: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.855923: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.855924: __rpc_execute: RPC: 3 __rpc_execute flags=0x2280 rpc.nfsd-4720 [001] .... 50.855925: call_start: RPC: 3 call_start rpcbind4 proc UNSET (sync) rpc.nfsd-4720 [001] .... 50.855926: call_reserve: RPC: 3 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.855927: xprt_alloc_slot: RPC: 3 reserved req ffff880403542200 xid 3c45b0ec rpc.nfsd-4720 [001] .... 50.855927: call_reserveresult: RPC: 3 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.855928: call_refresh: RPC: 3 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.855929: call_refreshresult: RPC: 3 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.855930: call_allocate: RPC: 3 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.855931: rpc_malloc: RPC: 3 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.855932: call_bind: RPC: 3 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.855933: call_connect: RPC: 3 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.855934: call_transmit: RPC: 3 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.855934: xprt_prepare_transmit: RPC: 3 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.855935: call_transmit: RPC: 3 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.855938: xprt_transmit: RPC: 3 xprt_transmit(68) rpc.nfsd-4720 [001] .... 50.855945: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4720 [001] .... 50.855946: xprt_transmit: RPC: 3 xmit complete rpc.nfsd-4720 [001] ..s. 50.855947: __rpc_sleep_on_priority: RPC: 3 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.855948: __rpc_sleep_on_priority: RPC: 3 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.855948: __rpc_sleep_on_priority: RPC: 3 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.855950: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.855951: __rpc_execute: RPC: 3 sync task going to sleep rpcbind-1871 [003] ..s. 50.856003: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856005: xprt_complete_rqst: RPC: 3 xid 3c45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856006: rpc_wake_up_task_queue_locked: RPC: 3 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856007: rpc_wake_up_task_queue_locked: RPC: 3 disabling timer rpcbind-1871 [003] ..s. 50.856008: rpc_wake_up_task_queue_locked: RPC: 3 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856011: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856046: __rpc_execute: RPC: 3 sync task resuming rpc.nfsd-4720 [001] .... 50.856048: call_status: RPC: 3 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856049: call_decode: RPC: 3 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856050: call_decode: RPC: 3 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856051: __rpc_execute: RPC: 3 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856052: __rpc_execute: RPC: 3 release task rpc.nfsd-4720 [001] .... 50.856053: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856054: xprt_release: RPC: 3 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856055: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856055: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856056: rpc_free_task: RPC: 3 freeing task rpc.nfsd-4720 [001] .... 50.856058: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856059: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.856060: __rpc_execute: RPC: 4 __rpc_execute flags=0x2280 rpc.nfsd-4720 [001] .... 50.856061: call_start: RPC: 4 call_start rpcbind4 proc UNSET (sync) rpc.nfsd-4720 [001] .... 50.856062: call_reserve: RPC: 4 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856063: xprt_alloc_slot: RPC: 4 reserved req ffff880403542200 xid 3d45b0ec rpc.nfsd-4720 [001] .... 50.856063: call_reserveresult: RPC: 4 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856064: call_refresh: RPC: 4 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856065: call_refreshresult: RPC: 4 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856066: call_allocate: RPC: 4 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856067: rpc_malloc: RPC: 4 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856067: call_bind: RPC: 4 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856068: call_connect: RPC: 4 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856069: call_transmit: RPC: 4 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856069: xprt_prepare_transmit: RPC: 4 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856070: call_transmit: RPC: 4 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856072: xprt_transmit: RPC: 4 xprt_transmit(68) rpc.nfsd-4720 [001] .... 50.856079: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4720 [001] .... 50.856080: xprt_transmit: RPC: 4 xmit complete rpc.nfsd-4720 [001] ..s. 50.856081: __rpc_sleep_on_priority: RPC: 4 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856081: __rpc_sleep_on_priority: RPC: 4 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856082: __rpc_sleep_on_priority: RPC: 4 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856084: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856084: __rpc_execute: RPC: 4 sync task going to sleep rpcbind-1871 [003] ..s. 50.856139: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856142: xprt_complete_rqst: RPC: 4 xid 3d45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856143: rpc_wake_up_task_queue_locked: RPC: 4 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856143: rpc_wake_up_task_queue_locked: RPC: 4 disabling timer rpcbind-1871 [003] ..s. 50.856145: rpc_wake_up_task_queue_locked: RPC: 4 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856148: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856186: __rpc_execute: RPC: 4 sync task resuming rpc.nfsd-4720 [001] .... 50.856188: call_status: RPC: 4 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856189: call_decode: RPC: 4 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856190: call_decode: RPC: 4 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856191: __rpc_execute: RPC: 4 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856192: __rpc_execute: RPC: 4 release task rpc.nfsd-4720 [001] .... 50.856193: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856194: xprt_release: RPC: 4 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856195: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856196: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856196: rpc_free_task: RPC: 4 freeing task rpc.nfsd-4720 [001] .... 50.856198: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856199: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.856200: __rpc_execute: RPC: 5 __rpc_execute flags=0x2280 rpc.nfsd-4720 [001] .... 50.856201: call_start: RPC: 5 call_start rpcbind4 proc UNSET (sync) rpc.nfsd-4720 [001] .... 50.856202: call_reserve: RPC: 5 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856202: xprt_alloc_slot: RPC: 5 reserved req ffff880403542200 xid 3e45b0ec rpc.nfsd-4720 [001] .... 50.856203: call_reserveresult: RPC: 5 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856204: call_refresh: RPC: 5 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856205: call_refreshresult: RPC: 5 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856206: call_allocate: RPC: 5 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856207: rpc_malloc: RPC: 5 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856207: call_bind: RPC: 5 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856208: call_connect: RPC: 5 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856209: call_transmit: RPC: 5 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856209: xprt_prepare_transmit: RPC: 5 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856210: call_transmit: RPC: 5 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856212: xprt_transmit: RPC: 5 xprt_transmit(68) rpc.nfsd-4720 [001] .... 50.856219: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4720 [001] .... 50.856219: xprt_transmit: RPC: 5 xmit complete rpc.nfsd-4720 [001] ..s. 50.856221: __rpc_sleep_on_priority: RPC: 5 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856221: __rpc_sleep_on_priority: RPC: 5 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856222: __rpc_sleep_on_priority: RPC: 5 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856224: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856224: __rpc_execute: RPC: 5 sync task going to sleep rpcbind-1871 [003] ..s. 50.856277: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856279: xprt_complete_rqst: RPC: 5 xid 3e45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856280: rpc_wake_up_task_queue_locked: RPC: 5 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856281: rpc_wake_up_task_queue_locked: RPC: 5 disabling timer rpcbind-1871 [003] ..s. 50.856282: rpc_wake_up_task_queue_locked: RPC: 5 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856285: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856321: __rpc_execute: RPC: 5 sync task resuming rpc.nfsd-4720 [001] .... 50.856322: call_status: RPC: 5 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856323: call_decode: RPC: 5 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856325: call_decode: RPC: 5 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856326: __rpc_execute: RPC: 5 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856326: __rpc_execute: RPC: 5 release task rpc.nfsd-4720 [001] .... 50.856327: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856328: xprt_release: RPC: 5 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856329: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856330: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856331: rpc_free_task: RPC: 5 freeing task rpc.nfsd-4720 [001] .... 50.856333: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856334: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.856335: __rpc_execute: RPC: 6 __rpc_execute flags=0x2280 rpc.nfsd-4720 [001] .... 50.856336: call_start: RPC: 6 call_start rpcbind4 proc UNSET (sync) rpc.nfsd-4720 [001] .... 50.856337: call_reserve: RPC: 6 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856337: xprt_alloc_slot: RPC: 6 reserved req ffff880403542200 xid 3f45b0ec rpc.nfsd-4720 [001] .... 50.856338: call_reserveresult: RPC: 6 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856339: call_refresh: RPC: 6 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856340: call_refreshresult: RPC: 6 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856341: call_allocate: RPC: 6 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856342: rpc_malloc: RPC: 6 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856342: call_bind: RPC: 6 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856343: call_connect: RPC: 6 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856344: call_transmit: RPC: 6 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856344: xprt_prepare_transmit: RPC: 6 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856345: call_transmit: RPC: 6 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856346: xprt_transmit: RPC: 6 xprt_transmit(68) rpc.nfsd-4720 [001] .... 50.856354: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4720 [001] .... 50.856354: xprt_transmit: RPC: 6 xmit complete rpc.nfsd-4720 [001] ..s. 50.856355: __rpc_sleep_on_priority: RPC: 6 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856356: __rpc_sleep_on_priority: RPC: 6 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856357: __rpc_sleep_on_priority: RPC: 6 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856359: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856359: __rpc_execute: RPC: 6 sync task going to sleep rpcbind-1871 [003] ..s. 50.856409: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856411: xprt_complete_rqst: RPC: 6 xid 3f45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856412: rpc_wake_up_task_queue_locked: RPC: 6 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856413: rpc_wake_up_task_queue_locked: RPC: 6 disabling timer rpcbind-1871 [003] ..s. 50.856414: rpc_wake_up_task_queue_locked: RPC: 6 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856417: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856453: __rpc_execute: RPC: 6 sync task resuming rpc.nfsd-4720 [001] .... 50.856455: call_status: RPC: 6 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856456: call_decode: RPC: 6 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856457: call_decode: RPC: 6 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856458: __rpc_execute: RPC: 6 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856459: __rpc_execute: RPC: 6 release task rpc.nfsd-4720 [001] .... 50.856460: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856461: xprt_release: RPC: 6 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856462: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856463: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856464: rpc_free_task: RPC: 6 freeing task rpc.nfsd-4720 [001] .... 50.856465: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856466: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.856467: __rpc_execute: RPC: 7 __rpc_execute flags=0x2280 rpc.nfsd-4720 [001] .... 50.856468: call_start: RPC: 7 call_start rpcbind4 proc UNSET (sync) rpc.nfsd-4720 [001] .... 50.856469: call_reserve: RPC: 7 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856470: xprt_alloc_slot: RPC: 7 reserved req ffff880403542200 xid 4045b0ec rpc.nfsd-4720 [001] .... 50.856470: call_reserveresult: RPC: 7 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856471: call_refresh: RPC: 7 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856472: call_refreshresult: RPC: 7 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856473: call_allocate: RPC: 7 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856474: rpc_malloc: RPC: 7 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856474: call_bind: RPC: 7 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856475: call_connect: RPC: 7 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856476: call_transmit: RPC: 7 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856476: xprt_prepare_transmit: RPC: 7 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856477: call_transmit: RPC: 7 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856479: xprt_transmit: RPC: 7 xprt_transmit(68) rpc.nfsd-4720 [001] .... 50.856486: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4720 [001] .... 50.856487: xprt_transmit: RPC: 7 xmit complete rpc.nfsd-4720 [001] ..s. 50.856488: __rpc_sleep_on_priority: RPC: 7 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856489: __rpc_sleep_on_priority: RPC: 7 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856489: __rpc_sleep_on_priority: RPC: 7 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856491: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856492: __rpc_execute: RPC: 7 sync task going to sleep rpcbind-1871 [003] ..s. 50.856507: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856509: xprt_complete_rqst: RPC: 7 xid 4045b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856510: rpc_wake_up_task_queue_locked: RPC: 7 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856510: rpc_wake_up_task_queue_locked: RPC: 7 disabling timer rpcbind-1871 [003] ..s. 50.856511: rpc_wake_up_task_queue_locked: RPC: 7 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856514: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856517: __rpc_execute: RPC: 7 sync task resuming rpc.nfsd-4720 [001] .... 50.856518: call_status: RPC: 7 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856519: call_decode: RPC: 7 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856521: call_decode: RPC: 7 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856522: __rpc_execute: RPC: 7 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856522: __rpc_execute: RPC: 7 release task rpc.nfsd-4720 [001] .... 50.856523: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856524: xprt_release: RPC: 7 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856525: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856526: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856527: rpc_free_task: RPC: 7 freeing task rpc.nfsd-4720 [001] .... 50.856530: svc_setup_socket: svc: svc_setup_socket ffff8800db68bac0 rpc.nfsd-4720 [001] .... 50.856535: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856536: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.856536: __rpc_execute: RPC: 8 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.856538: call_start: RPC: 8 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.856538: call_reserve: RPC: 8 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856539: xprt_alloc_slot: RPC: 8 reserved req ffff880403542200 xid 4145b0ec rpc.nfsd-4720 [001] .... 50.856540: call_reserveresult: RPC: 8 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856541: call_refresh: RPC: 8 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856542: call_refreshresult: RPC: 8 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856542: call_allocate: RPC: 8 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856543: rpc_malloc: RPC: 8 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856544: call_bind: RPC: 8 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856545: call_connect: RPC: 8 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856545: call_transmit: RPC: 8 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856546: xprt_prepare_transmit: RPC: 8 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856547: call_transmit: RPC: 8 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856548: xprt_transmit: RPC: 8 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.856555: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.856555: xprt_transmit: RPC: 8 xmit complete rpc.nfsd-4720 [001] ..s. 50.856556: __rpc_sleep_on_priority: RPC: 8 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856558: __rpc_sleep_on_priority: RPC: 8 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856558: __rpc_sleep_on_priority: RPC: 8 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856560: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856561: __rpc_execute: RPC: 8 sync task going to sleep rpcbind-1871 [003] ..s. 50.856580: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856580: xprt_complete_rqst: RPC: 8 xid 4145b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856581: rpc_wake_up_task_queue_locked: RPC: 8 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856581: rpc_wake_up_task_queue_locked: RPC: 8 disabling timer rpcbind-1871 [003] ..s. 50.856581: rpc_wake_up_task_queue_locked: RPC: 8 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856582: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856584: __rpc_execute: RPC: 8 sync task resuming rpc.nfsd-4720 [001] .... 50.856584: call_status: RPC: 8 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856584: call_decode: RPC: 8 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856585: call_decode: RPC: 8 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856585: __rpc_execute: RPC: 8 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856585: __rpc_execute: RPC: 8 release task rpc.nfsd-4720 [001] .... 50.856586: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856586: xprt_release: RPC: 8 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856586: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856586: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856587: rpc_free_task: RPC: 8 freeing task rpc.nfsd-4720 [001] .... 50.856588: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856588: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.856588: __rpc_execute: RPC: 9 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.856589: call_start: RPC: 9 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.856589: call_reserve: RPC: 9 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856589: xprt_alloc_slot: RPC: 9 reserved req ffff880403542200 xid 4245b0ec rpc.nfsd-4720 [001] .... 50.856590: call_reserveresult: RPC: 9 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856590: call_refresh: RPC: 9 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856590: call_refreshresult: RPC: 9 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856590: call_allocate: RPC: 9 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856591: rpc_malloc: RPC: 9 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856591: call_bind: RPC: 9 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856591: call_connect: RPC: 9 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856592: call_transmit: RPC: 9 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856592: xprt_prepare_transmit: RPC: 9 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856592: call_transmit: RPC: 9 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856593: xprt_transmit: RPC: 9 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.856595: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.856595: xprt_transmit: RPC: 9 xmit complete rpc.nfsd-4720 [001] ..s. 50.856595: __rpc_sleep_on_priority: RPC: 9 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856596: __rpc_sleep_on_priority: RPC: 9 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856596: __rpc_sleep_on_priority: RPC: 9 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856596: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856596: __rpc_execute: RPC: 9 sync task going to sleep rpcbind-1871 [003] ..s. 50.856604: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856604: xprt_complete_rqst: RPC: 9 xid 4245b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856604: rpc_wake_up_task_queue_locked: RPC: 9 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856605: rpc_wake_up_task_queue_locked: RPC: 9 disabling timer rpcbind-1871 [003] ..s. 50.856605: rpc_wake_up_task_queue_locked: RPC: 9 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856606: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856609: __rpc_execute: RPC: 9 sync task resuming rpc.nfsd-4720 [001] .... 50.856610: call_status: RPC: 9 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856611: call_decode: RPC: 9 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856611: call_decode: RPC: 9 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856612: __rpc_execute: RPC: 9 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856612: __rpc_execute: RPC: 9 release task rpc.nfsd-4720 [001] .... 50.856612: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856613: xprt_release: RPC: 9 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856613: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856614: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856614: rpc_free_task: RPC: 9 freeing task rpc.nfsd-4720 [001] .... 50.856616: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856616: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.856617: __rpc_execute: RPC: 10 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.856617: call_start: RPC: 10 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.856617: call_reserve: RPC: 10 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856618: xprt_alloc_slot: RPC: 10 reserved req ffff880403542200 xid 4345b0ec rpc.nfsd-4720 [001] .... 50.856618: call_reserveresult: RPC: 10 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856618: call_refresh: RPC: 10 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856619: call_refreshresult: RPC: 10 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856619: call_allocate: RPC: 10 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856619: rpc_malloc: RPC: 10 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856620: call_bind: RPC: 10 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856620: call_connect: RPC: 10 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856620: call_transmit: RPC: 10 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856620: xprt_prepare_transmit: RPC: 10 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856621: call_transmit: RPC: 10 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856622: xprt_transmit: RPC: 10 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.856631: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.856631: xprt_transmit: RPC: 10 xmit complete rpc.nfsd-4720 [001] ..s. 50.856633: __rpc_sleep_on_priority: RPC: 10 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856633: __rpc_sleep_on_priority: RPC: 10 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856634: __rpc_sleep_on_priority: RPC: 10 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856636: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856637: __rpc_execute: RPC: 10 sync task going to sleep rpcbind-1871 [003] ..s. 50.856661: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856664: xprt_complete_rqst: RPC: 10 xid 4345b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856665: rpc_wake_up_task_queue_locked: RPC: 10 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856665: rpc_wake_up_task_queue_locked: RPC: 10 disabling timer rpcbind-1871 [003] ..s. 50.856666: rpc_wake_up_task_queue_locked: RPC: 10 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856670: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856690: __rpc_execute: RPC: 10 sync task resuming rpc.nfsd-4720 [001] .... 50.856692: call_status: RPC: 10 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856692: call_decode: RPC: 10 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856694: call_decode: RPC: 10 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856695: __rpc_execute: RPC: 10 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856695: __rpc_execute: RPC: 10 release task rpc.nfsd-4720 [001] .... 50.856697: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856697: xprt_release: RPC: 10 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856698: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856699: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856700: rpc_free_task: RPC: 10 freeing task rpc.nfsd-4720 [001] .... 50.856704: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856705: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.856706: __rpc_execute: RPC: 11 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.856707: call_start: RPC: 11 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.856708: call_reserve: RPC: 11 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856709: xprt_alloc_slot: RPC: 11 reserved req ffff880403542200 xid 4445b0ec rpc.nfsd-4720 [001] .... 50.856709: call_reserveresult: RPC: 11 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856710: call_refresh: RPC: 11 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856711: call_refreshresult: RPC: 11 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856712: call_allocate: RPC: 11 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856713: rpc_malloc: RPC: 11 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856714: call_bind: RPC: 11 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856715: call_connect: RPC: 11 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856715: call_transmit: RPC: 11 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856716: xprt_prepare_transmit: RPC: 11 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856717: call_transmit: RPC: 11 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856718: xprt_transmit: RPC: 11 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.856726: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.856726: xprt_transmit: RPC: 11 xmit complete rpc.nfsd-4720 [001] ..s. 50.856727: __rpc_sleep_on_priority: RPC: 11 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856728: __rpc_sleep_on_priority: RPC: 11 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856729: __rpc_sleep_on_priority: RPC: 11 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856731: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856731: __rpc_execute: RPC: 11 sync task going to sleep rpcbind-1871 [003] ..s. 50.856771: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856773: xprt_complete_rqst: RPC: 11 xid 4445b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856774: rpc_wake_up_task_queue_locked: RPC: 11 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856775: rpc_wake_up_task_queue_locked: RPC: 11 disabling timer rpcbind-1871 [003] ..s. 50.856776: rpc_wake_up_task_queue_locked: RPC: 11 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856779: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856782: __rpc_execute: RPC: 11 sync task resuming rpc.nfsd-4720 [001] .... 50.856783: call_status: RPC: 11 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856784: call_decode: RPC: 11 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856786: call_decode: RPC: 11 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856787: __rpc_execute: RPC: 11 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856787: __rpc_execute: RPC: 11 release task rpc.nfsd-4720 [001] .... 50.856788: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856789: xprt_release: RPC: 11 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856790: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856791: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856792: rpc_free_task: RPC: 11 freeing task rpc.nfsd-4720 [001] .... 50.856795: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856795: rpc_new_task: RPC: allocated task ffff88040a645e00 rpc.nfsd-4720 [001] .... 50.856796: __rpc_execute: RPC: 12 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.856797: call_start: RPC: 12 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.856798: call_reserve: RPC: 12 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856799: xprt_alloc_slot: RPC: 12 reserved req ffff880403542200 xid 4545b0ec rpc.nfsd-4720 [001] .... 50.856800: call_reserveresult: RPC: 12 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856801: call_refresh: RPC: 12 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856801: call_refreshresult: RPC: 12 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856802: call_allocate: RPC: 12 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856803: rpc_malloc: RPC: 12 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856804: call_bind: RPC: 12 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856805: call_connect: RPC: 12 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856805: call_transmit: RPC: 12 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856806: xprt_prepare_transmit: RPC: 12 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856807: call_transmit: RPC: 12 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856808: xprt_transmit: RPC: 12 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.856814: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.856815: xprt_transmit: RPC: 12 xmit complete rpc.nfsd-4720 [001] ..s. 50.856816: __rpc_sleep_on_priority: RPC: 12 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856817: __rpc_sleep_on_priority: RPC: 12 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856818: __rpc_sleep_on_priority: RPC: 12 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856819: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856820: __rpc_execute: RPC: 12 sync task going to sleep rpcbind-1871 [003] ..s. 50.856838: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856840: xprt_complete_rqst: RPC: 12 xid 4545b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856841: rpc_wake_up_task_queue_locked: RPC: 12 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856842: rpc_wake_up_task_queue_locked: RPC: 12 disabling timer rpcbind-1871 [003] ..s. 50.856843: rpc_wake_up_task_queue_locked: RPC: 12 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856846: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856848: __rpc_execute: RPC: 12 sync task resuming rpc.nfsd-4720 [001] .... 50.856849: call_status: RPC: 12 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856850: call_decode: RPC: 12 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856851: call_decode: RPC: 12 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856852: __rpc_execute: RPC: 12 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856853: __rpc_execute: RPC: 12 release task rpc.nfsd-4720 [001] .... 50.856854: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856854: xprt_release: RPC: 12 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856855: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856856: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856857: rpc_free_task: RPC: 12 freeing task rpc.nfsd-4720 [001] .... 50.856859: svc_setup_socket: setting up TCP socket for listening rpc.nfsd-4720 [001] .... 50.856860: svc_setup_socket: svc: svc_setup_socket created ffff880402bd4000 (inet ffff88040a708780) rpc.nfsd-4720 [001] .... 50.856882: svc_setup_socket: svc: svc_setup_socket ffff88040c7cacc0 rpc.nfsd-4720 [001] .... 50.856884: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856884: rpc_new_task: RPC: allocated task ffff88040b326c00 rpc.nfsd-4720 [001] .... 50.856884: __rpc_execute: RPC: 13 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.856885: call_start: RPC: 13 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.856885: call_reserve: RPC: 13 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856886: xprt_alloc_slot: RPC: 13 reserved req ffff880403542200 xid 4645b0ec rpc.nfsd-4720 [001] .... 50.856886: call_reserveresult: RPC: 13 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856886: call_refresh: RPC: 13 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856886: call_refreshresult: RPC: 13 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856887: call_allocate: RPC: 13 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856887: rpc_malloc: RPC: 13 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856887: call_bind: RPC: 13 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856888: call_connect: RPC: 13 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856888: call_transmit: RPC: 13 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856888: xprt_prepare_transmit: RPC: 13 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856889: call_transmit: RPC: 13 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856889: xprt_transmit: RPC: 13 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.856892: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.856892: xprt_transmit: RPC: 13 xmit complete rpc.nfsd-4720 [001] ..s. 50.856893: __rpc_sleep_on_priority: RPC: 13 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856893: __rpc_sleep_on_priority: RPC: 13 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856893: __rpc_sleep_on_priority: RPC: 13 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856894: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856894: __rpc_execute: RPC: 13 sync task going to sleep rpcbind-1871 [003] ..s. 50.856901: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856901: xprt_complete_rqst: RPC: 13 xid 4645b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856902: rpc_wake_up_task_queue_locked: RPC: 13 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856902: rpc_wake_up_task_queue_locked: RPC: 13 disabling timer rpcbind-1871 [003] ..s. 50.856902: rpc_wake_up_task_queue_locked: RPC: 13 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856903: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856905: __rpc_execute: RPC: 13 sync task resuming rpc.nfsd-4720 [001] .... 50.856905: call_status: RPC: 13 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856905: call_decode: RPC: 13 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856906: call_decode: RPC: 13 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856906: __rpc_execute: RPC: 13 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856907: __rpc_execute: RPC: 13 release task rpc.nfsd-4720 [001] .... 50.856907: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856907: xprt_release: RPC: 13 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856908: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856908: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856908: rpc_free_task: RPC: 13 freeing task rpc.nfsd-4720 [001] .... 50.856909: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856909: rpc_new_task: RPC: allocated task ffff88040b326c00 rpc.nfsd-4720 [001] .... 50.856910: __rpc_execute: RPC: 14 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.856910: call_start: RPC: 14 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.856910: call_reserve: RPC: 14 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856911: xprt_alloc_slot: RPC: 14 reserved req ffff880403542200 xid 4745b0ec rpc.nfsd-4720 [001] .... 50.856911: call_reserveresult: RPC: 14 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856911: call_refresh: RPC: 14 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856912: call_refreshresult: RPC: 14 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856912: call_allocate: RPC: 14 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856912: rpc_malloc: RPC: 14 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856912: call_bind: RPC: 14 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856913: call_connect: RPC: 14 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856913: call_transmit: RPC: 14 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856913: xprt_prepare_transmit: RPC: 14 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856913: call_transmit: RPC: 14 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856914: xprt_transmit: RPC: 14 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.856916: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.856916: xprt_transmit: RPC: 14 xmit complete rpc.nfsd-4720 [001] ..s. 50.856917: __rpc_sleep_on_priority: RPC: 14 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856917: __rpc_sleep_on_priority: RPC: 14 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856917: __rpc_sleep_on_priority: RPC: 14 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856918: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856918: __rpc_execute: RPC: 14 sync task going to sleep rpcbind-1871 [003] ..s. 50.856924: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856925: xprt_complete_rqst: RPC: 14 xid 4745b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856926: rpc_wake_up_task_queue_locked: RPC: 14 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856926: rpc_wake_up_task_queue_locked: RPC: 14 disabling timer rpcbind-1871 [003] ..s. 50.856926: rpc_wake_up_task_queue_locked: RPC: 14 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856927: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856928: __rpc_execute: RPC: 14 sync task resuming rpc.nfsd-4720 [001] .... 50.856929: call_status: RPC: 14 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856929: call_decode: RPC: 14 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856929: call_decode: RPC: 14 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856930: __rpc_execute: RPC: 14 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856930: __rpc_execute: RPC: 14 release task rpc.nfsd-4720 [001] .... 50.856930: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856930: xprt_release: RPC: 14 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856931: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856931: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856931: rpc_free_task: RPC: 14 freeing task rpc.nfsd-4720 [001] .... 50.856932: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856933: rpc_new_task: RPC: allocated task ffff88040b326c00 rpc.nfsd-4720 [001] .... 50.856933: __rpc_execute: RPC: 15 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.856933: call_start: RPC: 15 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.856934: call_reserve: RPC: 15 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856934: xprt_alloc_slot: RPC: 15 reserved req ffff880403542200 xid 4845b0ec rpc.nfsd-4720 [001] .... 50.856934: call_reserveresult: RPC: 15 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856934: call_refresh: RPC: 15 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856935: call_refreshresult: RPC: 15 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856935: call_allocate: RPC: 15 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856935: rpc_malloc: RPC: 15 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856935: call_bind: RPC: 15 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856936: call_connect: RPC: 15 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856936: call_transmit: RPC: 15 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856936: xprt_prepare_transmit: RPC: 15 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856936: call_transmit: RPC: 15 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856937: xprt_transmit: RPC: 15 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.856939: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.856939: xprt_transmit: RPC: 15 xmit complete rpc.nfsd-4720 [001] ..s. 50.856939: __rpc_sleep_on_priority: RPC: 15 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856940: __rpc_sleep_on_priority: RPC: 15 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856940: __rpc_sleep_on_priority: RPC: 15 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856941: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856941: __rpc_execute: RPC: 15 sync task going to sleep rpcbind-1871 [003] ..s. 50.856947: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856948: xprt_complete_rqst: RPC: 15 xid 4845b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856949: rpc_wake_up_task_queue_locked: RPC: 15 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856949: rpc_wake_up_task_queue_locked: RPC: 15 disabling timer rpcbind-1871 [003] ..s. 50.856949: rpc_wake_up_task_queue_locked: RPC: 15 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856950: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856951: __rpc_execute: RPC: 15 sync task resuming rpc.nfsd-4720 [001] .... 50.856952: call_status: RPC: 15 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856952: call_decode: RPC: 15 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856952: call_decode: RPC: 15 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856953: __rpc_execute: RPC: 15 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856953: __rpc_execute: RPC: 15 release task rpc.nfsd-4720 [001] .... 50.856953: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856953: xprt_release: RPC: 15 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856954: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856954: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856954: rpc_free_task: RPC: 15 freeing task rpc.nfsd-4720 [001] .... 50.856955: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856955: rpc_new_task: RPC: allocated task ffff88040b326c00 rpc.nfsd-4720 [001] .... 50.856956: __rpc_execute: RPC: 16 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.856956: call_start: RPC: 16 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.856956: call_reserve: RPC: 16 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856957: xprt_alloc_slot: RPC: 16 reserved req ffff880403542200 xid 4945b0ec rpc.nfsd-4720 [001] .... 50.856957: call_reserveresult: RPC: 16 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856957: call_refresh: RPC: 16 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856958: call_refreshresult: RPC: 16 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856958: call_allocate: RPC: 16 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856958: rpc_malloc: RPC: 16 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856959: call_bind: RPC: 16 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856959: call_connect: RPC: 16 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856959: call_transmit: RPC: 16 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856959: xprt_prepare_transmit: RPC: 16 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856960: call_transmit: RPC: 16 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856960: xprt_transmit: RPC: 16 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.856962: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.856962: xprt_transmit: RPC: 16 xmit complete rpc.nfsd-4720 [001] ..s. 50.856963: __rpc_sleep_on_priority: RPC: 16 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856963: __rpc_sleep_on_priority: RPC: 16 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856963: __rpc_sleep_on_priority: RPC: 16 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856964: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856964: __rpc_execute: RPC: 16 sync task going to sleep rpcbind-1871 [003] ..s. 50.856971: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856971: xprt_complete_rqst: RPC: 16 xid 4945b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856972: rpc_wake_up_task_queue_locked: RPC: 16 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856972: rpc_wake_up_task_queue_locked: RPC: 16 disabling timer rpcbind-1871 [003] ..s. 50.856972: rpc_wake_up_task_queue_locked: RPC: 16 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856973: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856974: __rpc_execute: RPC: 16 sync task resuming rpc.nfsd-4720 [001] .... 50.856975: call_status: RPC: 16 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856975: call_decode: RPC: 16 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856976: call_decode: RPC: 16 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856976: __rpc_execute: RPC: 16 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856976: __rpc_execute: RPC: 16 release task rpc.nfsd-4720 [001] .... 50.856976: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856977: xprt_release: RPC: 16 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.856977: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.856977: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.856978: rpc_free_task: RPC: 16 freeing task rpc.nfsd-4720 [001] .... 50.856979: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.856979: rpc_new_task: RPC: allocated task ffff88040b326c00 rpc.nfsd-4720 [001] .... 50.856979: __rpc_execute: RPC: 17 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.856979: call_start: RPC: 17 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.856980: call_reserve: RPC: 17 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.856980: xprt_alloc_slot: RPC: 17 reserved req ffff880403542200 xid 4a45b0ec rpc.nfsd-4720 [001] .... 50.856980: call_reserveresult: RPC: 17 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.856980: call_refresh: RPC: 17 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.856981: call_refreshresult: RPC: 17 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.856981: call_allocate: RPC: 17 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.856981: rpc_malloc: RPC: 17 allocated buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.856982: call_bind: RPC: 17 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.856982: call_connect: RPC: 17 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.856982: call_transmit: RPC: 17 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.856982: xprt_prepare_transmit: RPC: 17 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.856983: call_transmit: RPC: 17 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.856983: xprt_transmit: RPC: 17 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.856985: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.856985: xprt_transmit: RPC: 17 xmit complete rpc.nfsd-4720 [001] ..s. 50.856986: __rpc_sleep_on_priority: RPC: 17 sleep_on(queue "xprt_pending" time 4294904942) rpc.nfsd-4720 [001] ..s. 50.856986: __rpc_sleep_on_priority: RPC: 17 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.856986: __rpc_sleep_on_priority: RPC: 17 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.856987: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.856987: __rpc_execute: RPC: 17 sync task going to sleep rpcbind-1871 [003] ..s. 50.856994: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.856994: xprt_complete_rqst: RPC: 17 xid 4a45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.856995: rpc_wake_up_task_queue_locked: RPC: 17 __rpc_wake_up_task (now 4294904942) rpcbind-1871 [003] ..s. 50.856995: rpc_wake_up_task_queue_locked: RPC: 17 disabling timer rpcbind-1871 [003] ..s. 50.856996: rpc_wake_up_task_queue_locked: RPC: 17 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.856997: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.856998: __rpc_execute: RPC: 17 sync task resuming rpc.nfsd-4720 [001] .... 50.856998: call_status: RPC: 17 call_status (status 28) rpc.nfsd-4720 [001] .... 50.856998: call_decode: RPC: 17 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.856999: call_decode: RPC: 17 call_decode result 0 rpc.nfsd-4720 [001] .... 50.856999: __rpc_execute: RPC: 17 return 0, status 0 rpc.nfsd-4720 [001] .... 50.856999: __rpc_execute: RPC: 17 release task rpc.nfsd-4720 [001] .... 50.857000: rpc_free: RPC: freeing buffer of size 188 at ffff8804045aa000 rpc.nfsd-4720 [001] .... 50.857000: xprt_release: RPC: 17 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857000: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857001: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857001: rpc_free_task: RPC: 17 freeing task rpc.nfsd-4720 [001] .... 50.857002: svc_write_space: svc: socket ffff880402ba1000(inet ffff880407b2cc00), write_space busy=1 rpc.nfsd-4720 [001] .... 50.857003: svc_setup_socket: svc: kernel_setsockopt returned 0 rpc.nfsd-4720 [001] .... 50.857003: svc_setup_socket: svc: svc_setup_socket created ffff880402ba1000 (inet ffff880407b2cc00) rpc.nfsd-4720 [001] .... 50.857568: svc_setup_socket: svc: svc_setup_socket ffff88040ec230c0 rpc.nfsd-4720 [001] .... 50.857571: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857571: rpc_new_task: RPC: allocated task ffff88040a14b900 rpc.nfsd-4720 [001] .... 50.857572: __rpc_execute: RPC: 18 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.857572: call_start: RPC: 18 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.857573: call_reserve: RPC: 18 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857573: xprt_alloc_slot: RPC: 18 reserved req ffff880403542200 xid 4b45b0ec rpc.nfsd-4720 [001] .... 50.857573: call_reserveresult: RPC: 18 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857574: call_refresh: RPC: 18 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857574: call_refreshresult: RPC: 18 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857574: call_allocate: RPC: 18 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857575: rpc_malloc: RPC: 18 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857575: call_bind: RPC: 18 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857575: call_connect: RPC: 18 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857576: call_transmit: RPC: 18 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857576: xprt_prepare_transmit: RPC: 18 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857576: call_transmit: RPC: 18 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857577: xprt_transmit: RPC: 18 xprt_transmit(80) rpc.nfsd-4720 [001] .... 50.857581: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4720 [001] .... 50.857581: xprt_transmit: RPC: 18 xmit complete rpc.nfsd-4720 [001] ..s. 50.857581: __rpc_sleep_on_priority: RPC: 18 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857582: __rpc_sleep_on_priority: RPC: 18 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857582: __rpc_sleep_on_priority: RPC: 18 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857583: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857583: __rpc_execute: RPC: 18 sync task going to sleep rpcbind-1871 [003] ..s. 50.857592: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857593: xprt_complete_rqst: RPC: 18 xid 4b45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857593: rpc_wake_up_task_queue_locked: RPC: 18 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857594: rpc_wake_up_task_queue_locked: RPC: 18 disabling timer rpcbind-1871 [003] ..s. 50.857594: rpc_wake_up_task_queue_locked: RPC: 18 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857595: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857596: __rpc_execute: RPC: 18 sync task resuming rpc.nfsd-4720 [001] .... 50.857597: call_status: RPC: 18 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857597: call_decode: RPC: 18 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857598: call_decode: RPC: 18 call_decode result 0 rpc.nfsd-4720 [001] .... 50.857598: __rpc_execute: RPC: 18 return 0, status 0 rpc.nfsd-4720 [001] .... 50.857598: __rpc_execute: RPC: 18 release task rpc.nfsd-4720 [001] .... 50.857599: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857599: xprt_release: RPC: 18 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857600: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857600: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857601: rpc_free_task: RPC: 18 freeing task rpc.nfsd-4720 [001] .... 50.857602: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857602: rpc_new_task: RPC: allocated task ffff88040a14b900 rpc.nfsd-4720 [001] .... 50.857602: __rpc_execute: RPC: 19 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.857602: call_start: RPC: 19 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.857603: call_reserve: RPC: 19 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857603: xprt_alloc_slot: RPC: 19 reserved req ffff880403542200 xid 4c45b0ec rpc.nfsd-4720 [001] .... 50.857603: call_reserveresult: RPC: 19 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857604: call_refresh: RPC: 19 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857604: call_refreshresult: RPC: 19 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857604: call_allocate: RPC: 19 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857605: rpc_malloc: RPC: 19 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857605: call_bind: RPC: 19 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857605: call_connect: RPC: 19 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857605: call_transmit: RPC: 19 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857606: xprt_prepare_transmit: RPC: 19 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857606: call_transmit: RPC: 19 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857606: xprt_transmit: RPC: 19 xprt_transmit(80) rpc.nfsd-4720 [001] .... 50.857608: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4720 [001] .... 50.857609: xprt_transmit: RPC: 19 xmit complete rpc.nfsd-4720 [001] ..s. 50.857609: __rpc_sleep_on_priority: RPC: 19 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857609: __rpc_sleep_on_priority: RPC: 19 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857610: __rpc_sleep_on_priority: RPC: 19 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857610: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857610: __rpc_execute: RPC: 19 sync task going to sleep rpcbind-1871 [003] ..s. 50.857617: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857618: xprt_complete_rqst: RPC: 19 xid 4c45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857618: rpc_wake_up_task_queue_locked: RPC: 19 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857618: rpc_wake_up_task_queue_locked: RPC: 19 disabling timer rpcbind-1871 [003] ..s. 50.857618: rpc_wake_up_task_queue_locked: RPC: 19 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857619: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857620: __rpc_execute: RPC: 19 sync task resuming rpc.nfsd-4720 [001] .... 50.857621: call_status: RPC: 19 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857621: call_decode: RPC: 19 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857622: call_decode: RPC: 19 call_decode result 0 rpc.nfsd-4720 [001] .... 50.857622: __rpc_execute: RPC: 19 return 0, status 0 rpc.nfsd-4720 [001] .... 50.857622: __rpc_execute: RPC: 19 release task rpc.nfsd-4720 [001] .... 50.857623: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857623: xprt_release: RPC: 19 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857623: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857623: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857624: rpc_free_task: RPC: 19 freeing task rpc.nfsd-4720 [001] .... 50.857625: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857625: rpc_new_task: RPC: allocated task ffff88040a14b900 rpc.nfsd-4720 [001] .... 50.857625: __rpc_execute: RPC: 20 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.857625: call_start: RPC: 20 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.857626: call_reserve: RPC: 20 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857626: xprt_alloc_slot: RPC: 20 reserved req ffff880403542200 xid 4d45b0ec rpc.nfsd-4720 [001] .... 50.857626: call_reserveresult: RPC: 20 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857627: call_refresh: RPC: 20 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857627: call_refreshresult: RPC: 20 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857627: call_allocate: RPC: 20 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857628: rpc_malloc: RPC: 20 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857628: call_bind: RPC: 20 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857628: call_connect: RPC: 20 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857628: call_transmit: RPC: 20 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857628: xprt_prepare_transmit: RPC: 20 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857629: call_transmit: RPC: 20 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857629: xprt_transmit: RPC: 20 xprt_transmit(80) rpc.nfsd-4720 [001] .... 50.857631: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4720 [001] .... 50.857632: xprt_transmit: RPC: 20 xmit complete rpc.nfsd-4720 [001] ..s. 50.857632: __rpc_sleep_on_priority: RPC: 20 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857632: __rpc_sleep_on_priority: RPC: 20 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857632: __rpc_sleep_on_priority: RPC: 20 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857633: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857633: __rpc_execute: RPC: 20 sync task going to sleep rpcbind-1871 [003] ..s. 50.857640: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857640: xprt_complete_rqst: RPC: 20 xid 4d45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857641: rpc_wake_up_task_queue_locked: RPC: 20 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857641: rpc_wake_up_task_queue_locked: RPC: 20 disabling timer rpcbind-1871 [003] ..s. 50.857641: rpc_wake_up_task_queue_locked: RPC: 20 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857642: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857643: __rpc_execute: RPC: 20 sync task resuming rpc.nfsd-4720 [001] .... 50.857644: call_status: RPC: 20 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857644: call_decode: RPC: 20 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857645: call_decode: RPC: 20 call_decode result 0 rpc.nfsd-4720 [001] .... 50.857645: __rpc_execute: RPC: 20 return 0, status 0 rpc.nfsd-4720 [001] .... 50.857645: __rpc_execute: RPC: 20 release task rpc.nfsd-4720 [001] .... 50.857645: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857646: xprt_release: RPC: 20 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857646: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857646: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857646: rpc_free_task: RPC: 20 freeing task rpc.nfsd-4720 [001] .... 50.857647: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857648: rpc_new_task: RPC: allocated task ffff88040a14b900 rpc.nfsd-4720 [001] .... 50.857648: __rpc_execute: RPC: 21 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.857648: call_start: RPC: 21 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.857649: call_reserve: RPC: 21 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857649: xprt_alloc_slot: RPC: 21 reserved req ffff880403542200 xid 4e45b0ec rpc.nfsd-4720 [001] .... 50.857649: call_reserveresult: RPC: 21 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857649: call_refresh: RPC: 21 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857650: call_refreshresult: RPC: 21 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857650: call_allocate: RPC: 21 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857650: rpc_malloc: RPC: 21 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857651: call_bind: RPC: 21 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857651: call_connect: RPC: 21 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857651: call_transmit: RPC: 21 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857651: xprt_prepare_transmit: RPC: 21 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857652: call_transmit: RPC: 21 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857652: xprt_transmit: RPC: 21 xprt_transmit(80) rpc.nfsd-4720 [001] .... 50.857654: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4720 [001] .... 50.857654: xprt_transmit: RPC: 21 xmit complete rpc.nfsd-4720 [001] ..s. 50.857655: __rpc_sleep_on_priority: RPC: 21 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857655: __rpc_sleep_on_priority: RPC: 21 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857655: __rpc_sleep_on_priority: RPC: 21 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857656: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857656: __rpc_execute: RPC: 21 sync task going to sleep rpcbind-1871 [003] ..s. 50.857662: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857663: xprt_complete_rqst: RPC: 21 xid 4e45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857664: rpc_wake_up_task_queue_locked: RPC: 21 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857664: rpc_wake_up_task_queue_locked: RPC: 21 disabling timer rpcbind-1871 [003] ..s. 50.857664: rpc_wake_up_task_queue_locked: RPC: 21 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857665: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857666: __rpc_execute: RPC: 21 sync task resuming rpc.nfsd-4720 [001] .... 50.857667: call_status: RPC: 21 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857667: call_decode: RPC: 21 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857667: call_decode: RPC: 21 call_decode result 0 rpc.nfsd-4720 [001] .... 50.857668: __rpc_execute: RPC: 21 return 0, status 0 rpc.nfsd-4720 [001] .... 50.857668: __rpc_execute: RPC: 21 release task rpc.nfsd-4720 [001] .... 50.857668: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857669: xprt_release: RPC: 21 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857669: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857669: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857669: rpc_free_task: RPC: 21 freeing task rpc.nfsd-4720 [001] .... 50.857670: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857671: rpc_new_task: RPC: allocated task ffff88040a14b900 rpc.nfsd-4720 [001] .... 50.857671: __rpc_execute: RPC: 22 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.857671: call_start: RPC: 22 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.857672: call_reserve: RPC: 22 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857672: xprt_alloc_slot: RPC: 22 reserved req ffff880403542200 xid 4f45b0ec rpc.nfsd-4720 [001] .... 50.857672: call_reserveresult: RPC: 22 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857672: call_refresh: RPC: 22 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857673: call_refreshresult: RPC: 22 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857673: call_allocate: RPC: 22 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857673: rpc_malloc: RPC: 22 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857674: call_bind: RPC: 22 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857674: call_connect: RPC: 22 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857674: call_transmit: RPC: 22 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857674: xprt_prepare_transmit: RPC: 22 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857675: call_transmit: RPC: 22 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857675: xprt_transmit: RPC: 22 xprt_transmit(80) rpc.nfsd-4720 [001] .... 50.857677: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4720 [001] .... 50.857677: xprt_transmit: RPC: 22 xmit complete rpc.nfsd-4720 [001] ..s. 50.857678: __rpc_sleep_on_priority: RPC: 22 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857678: __rpc_sleep_on_priority: RPC: 22 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857678: __rpc_sleep_on_priority: RPC: 22 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857679: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857679: __rpc_execute: RPC: 22 sync task going to sleep rpcbind-1871 [003] ..s. 50.857685: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857686: xprt_complete_rqst: RPC: 22 xid 4f45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857687: rpc_wake_up_task_queue_locked: RPC: 22 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857687: rpc_wake_up_task_queue_locked: RPC: 22 disabling timer rpcbind-1871 [003] ..s. 50.857687: rpc_wake_up_task_queue_locked: RPC: 22 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857688: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857689: __rpc_execute: RPC: 22 sync task resuming rpc.nfsd-4720 [001] .... 50.857690: call_status: RPC: 22 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857690: call_decode: RPC: 22 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857690: call_decode: RPC: 22 call_decode result 0 rpc.nfsd-4720 [001] .... 50.857691: __rpc_execute: RPC: 22 return 0, status 0 rpc.nfsd-4720 [001] .... 50.857691: __rpc_execute: RPC: 22 release task rpc.nfsd-4720 [001] .... 50.857691: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857691: xprt_release: RPC: 22 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857692: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857692: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857692: rpc_free_task: RPC: 22 freeing task rpc.nfsd-4720 [001] .... 50.857693: svc_setup_socket: setting up TCP socket for listening rpc.nfsd-4720 [001] .... 50.857693: svc_setup_socket: svc: svc_setup_socket created ffff88040cf94000 (inet ffff88040a152800) rpc.nfsd-4720 [001] .... 50.857706: svc_setup_socket: svc: svc_setup_socket ffff880402295340 rpc.nfsd-4720 [001] .... 50.857708: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857708: rpc_new_task: RPC: allocated task ffff88040b0c0e00 rpc.nfsd-4720 [001] .... 50.857708: __rpc_execute: RPC: 23 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.857709: call_start: RPC: 23 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.857709: call_reserve: RPC: 23 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857709: xprt_alloc_slot: RPC: 23 reserved req ffff880403542200 xid 5045b0ec rpc.nfsd-4720 [001] .... 50.857710: call_reserveresult: RPC: 23 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857710: call_refresh: RPC: 23 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857710: call_refreshresult: RPC: 23 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857711: call_allocate: RPC: 23 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857711: rpc_malloc: RPC: 23 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857711: call_bind: RPC: 23 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857712: call_connect: RPC: 23 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857712: call_transmit: RPC: 23 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857712: xprt_prepare_transmit: RPC: 23 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857712: call_transmit: RPC: 23 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857713: xprt_transmit: RPC: 23 xprt_transmit(80) rpc.nfsd-4720 [001] .... 50.857716: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4720 [001] .... 50.857716: xprt_transmit: RPC: 23 xmit complete rpc.nfsd-4720 [001] ..s. 50.857716: __rpc_sleep_on_priority: RPC: 23 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857717: __rpc_sleep_on_priority: RPC: 23 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857717: __rpc_sleep_on_priority: RPC: 23 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857718: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857718: __rpc_execute: RPC: 23 sync task going to sleep rpcbind-1871 [003] ..s. 50.857724: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857725: xprt_complete_rqst: RPC: 23 xid 5045b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857725: rpc_wake_up_task_queue_locked: RPC: 23 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857725: rpc_wake_up_task_queue_locked: RPC: 23 disabling timer rpcbind-1871 [003] ..s. 50.857726: rpc_wake_up_task_queue_locked: RPC: 23 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857727: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857728: __rpc_execute: RPC: 23 sync task resuming rpc.nfsd-4720 [001] .... 50.857729: call_status: RPC: 23 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857729: call_decode: RPC: 23 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857729: call_decode: RPC: 23 call_decode result 0 rpc.nfsd-4720 [001] .... 50.857730: __rpc_execute: RPC: 23 return 0, status 0 rpc.nfsd-4720 [001] .... 50.857730: __rpc_execute: RPC: 23 release task rpc.nfsd-4720 [001] .... 50.857730: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857731: xprt_release: RPC: 23 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857731: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857731: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857732: rpc_free_task: RPC: 23 freeing task rpc.nfsd-4720 [001] .... 50.857732: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857733: rpc_new_task: RPC: allocated task ffff88040b0c0e00 rpc.nfsd-4720 [001] .... 50.857733: __rpc_execute: RPC: 24 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.857734: call_start: RPC: 24 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.857734: call_reserve: RPC: 24 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857734: xprt_alloc_slot: RPC: 24 reserved req ffff880403542200 xid 5145b0ec rpc.nfsd-4720 [001] .... 50.857734: call_reserveresult: RPC: 24 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857735: call_refresh: RPC: 24 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857735: call_refreshresult: RPC: 24 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857735: call_allocate: RPC: 24 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857736: rpc_malloc: RPC: 24 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857736: call_bind: RPC: 24 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857736: call_connect: RPC: 24 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857736: call_transmit: RPC: 24 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857737: xprt_prepare_transmit: RPC: 24 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857737: call_transmit: RPC: 24 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857737: xprt_transmit: RPC: 24 xprt_transmit(80) rpc.nfsd-4720 [001] .... 50.857740: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4720 [001] .... 50.857740: xprt_transmit: RPC: 24 xmit complete rpc.nfsd-4720 [001] ..s. 50.857740: __rpc_sleep_on_priority: RPC: 24 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857741: __rpc_sleep_on_priority: RPC: 24 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857741: __rpc_sleep_on_priority: RPC: 24 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857741: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857742: __rpc_execute: RPC: 24 sync task going to sleep rpcbind-1871 [003] ..s. 50.857748: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857749: xprt_complete_rqst: RPC: 24 xid 5145b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857749: rpc_wake_up_task_queue_locked: RPC: 24 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857749: rpc_wake_up_task_queue_locked: RPC: 24 disabling timer rpcbind-1871 [003] ..s. 50.857750: rpc_wake_up_task_queue_locked: RPC: 24 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857751: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857752: __rpc_execute: RPC: 24 sync task resuming rpc.nfsd-4720 [001] .... 50.857752: call_status: RPC: 24 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857753: call_decode: RPC: 24 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857753: call_decode: RPC: 24 call_decode result 0 rpc.nfsd-4720 [001] .... 50.857753: __rpc_execute: RPC: 24 return 0, status 0 rpc.nfsd-4720 [001] .... 50.857753: __rpc_execute: RPC: 24 release task rpc.nfsd-4720 [001] .... 50.857754: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857754: xprt_release: RPC: 24 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857754: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857755: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857755: rpc_free_task: RPC: 24 freeing task rpc.nfsd-4720 [001] .... 50.857756: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857756: rpc_new_task: RPC: allocated task ffff88040b0c0e00 rpc.nfsd-4720 [001] .... 50.857756: __rpc_execute: RPC: 25 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.857757: call_start: RPC: 25 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.857757: call_reserve: RPC: 25 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857757: xprt_alloc_slot: RPC: 25 reserved req ffff880403542200 xid 5245b0ec rpc.nfsd-4720 [001] .... 50.857758: call_reserveresult: RPC: 25 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857758: call_refresh: RPC: 25 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857758: call_refreshresult: RPC: 25 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857759: call_allocate: RPC: 25 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857759: rpc_malloc: RPC: 25 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857759: call_bind: RPC: 25 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857759: call_connect: RPC: 25 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857760: call_transmit: RPC: 25 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857760: xprt_prepare_transmit: RPC: 25 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857760: call_transmit: RPC: 25 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857761: xprt_transmit: RPC: 25 xprt_transmit(80) rpc.nfsd-4720 [001] .... 50.857763: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4720 [001] .... 50.857763: xprt_transmit: RPC: 25 xmit complete rpc.nfsd-4720 [001] ..s. 50.857763: __rpc_sleep_on_priority: RPC: 25 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857764: __rpc_sleep_on_priority: RPC: 25 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857764: __rpc_sleep_on_priority: RPC: 25 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857764: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857765: __rpc_execute: RPC: 25 sync task going to sleep rpcbind-1871 [003] ..s. 50.857771: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857772: xprt_complete_rqst: RPC: 25 xid 5245b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857772: rpc_wake_up_task_queue_locked: RPC: 25 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857772: rpc_wake_up_task_queue_locked: RPC: 25 disabling timer rpcbind-1871 [003] ..s. 50.857773: rpc_wake_up_task_queue_locked: RPC: 25 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857774: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857775: __rpc_execute: RPC: 25 sync task resuming rpc.nfsd-4720 [001] .... 50.857775: call_status: RPC: 25 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857776: call_decode: RPC: 25 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857776: call_decode: RPC: 25 call_decode result 0 rpc.nfsd-4720 [001] .... 50.857776: __rpc_execute: RPC: 25 return 0, status 0 rpc.nfsd-4720 [001] .... 50.857777: __rpc_execute: RPC: 25 release task rpc.nfsd-4720 [001] .... 50.857777: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857777: xprt_release: RPC: 25 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857778: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857778: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857778: rpc_free_task: RPC: 25 freeing task rpc.nfsd-4720 [001] .... 50.857779: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857779: rpc_new_task: RPC: allocated task ffff88040b0c0e00 rpc.nfsd-4720 [001] .... 50.857780: __rpc_execute: RPC: 26 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.857780: call_start: RPC: 26 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.857780: call_reserve: RPC: 26 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857781: xprt_alloc_slot: RPC: 26 reserved req ffff880403542200 xid 5345b0ec rpc.nfsd-4720 [001] .... 50.857781: call_reserveresult: RPC: 26 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857781: call_refresh: RPC: 26 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857781: call_refreshresult: RPC: 26 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857782: call_allocate: RPC: 26 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857782: rpc_malloc: RPC: 26 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857782: call_bind: RPC: 26 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857782: call_connect: RPC: 26 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857783: call_transmit: RPC: 26 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857783: xprt_prepare_transmit: RPC: 26 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857783: call_transmit: RPC: 26 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857784: xprt_transmit: RPC: 26 xprt_transmit(80) rpc.nfsd-4720 [001] .... 50.857786: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4720 [001] .... 50.857786: xprt_transmit: RPC: 26 xmit complete rpc.nfsd-4720 [001] ..s. 50.857786: __rpc_sleep_on_priority: RPC: 26 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857787: __rpc_sleep_on_priority: RPC: 26 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857787: __rpc_sleep_on_priority: RPC: 26 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857787: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857788: __rpc_execute: RPC: 26 sync task going to sleep rpcbind-1871 [003] ..s. 50.857794: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857795: xprt_complete_rqst: RPC: 26 xid 5345b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857795: rpc_wake_up_task_queue_locked: RPC: 26 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857795: rpc_wake_up_task_queue_locked: RPC: 26 disabling timer rpcbind-1871 [003] ..s. 50.857796: rpc_wake_up_task_queue_locked: RPC: 26 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857797: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857798: __rpc_execute: RPC: 26 sync task resuming rpc.nfsd-4720 [001] .... 50.857798: call_status: RPC: 26 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857798: call_decode: RPC: 26 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857799: call_decode: RPC: 26 call_decode result 0 rpc.nfsd-4720 [001] .... 50.857799: __rpc_execute: RPC: 26 return 0, status 0 rpc.nfsd-4720 [001] .... 50.857799: __rpc_execute: RPC: 26 release task rpc.nfsd-4720 [001] .... 50.857800: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857800: xprt_release: RPC: 26 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857800: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857801: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857801: rpc_free_task: RPC: 26 freeing task rpc.nfsd-4720 [001] .... 50.857802: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857802: rpc_new_task: RPC: allocated task ffff88040b0c0e00 rpc.nfsd-4720 [001] .... 50.857802: __rpc_execute: RPC: 27 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.857803: call_start: RPC: 27 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.857803: call_reserve: RPC: 27 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857803: xprt_alloc_slot: RPC: 27 reserved req ffff880403542200 xid 5445b0ec rpc.nfsd-4720 [001] .... 50.857804: call_reserveresult: RPC: 27 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857804: call_refresh: RPC: 27 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857804: call_refreshresult: RPC: 27 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857804: call_allocate: RPC: 27 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857805: rpc_malloc: RPC: 27 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857805: call_bind: RPC: 27 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857805: call_connect: RPC: 27 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857806: call_transmit: RPC: 27 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857806: xprt_prepare_transmit: RPC: 27 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857806: call_transmit: RPC: 27 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857806: xprt_transmit: RPC: 27 xprt_transmit(80) rpc.nfsd-4720 [001] .... 50.857809: xs_local_send_request: RPC: xs_local_send_request(80) = 0 rpc.nfsd-4720 [001] .... 50.857809: xprt_transmit: RPC: 27 xmit complete rpc.nfsd-4720 [001] ..s. 50.857809: __rpc_sleep_on_priority: RPC: 27 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857809: __rpc_sleep_on_priority: RPC: 27 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857810: __rpc_sleep_on_priority: RPC: 27 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857810: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857810: __rpc_execute: RPC: 27 sync task going to sleep rpcbind-1871 [003] ..s. 50.857817: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857818: xprt_complete_rqst: RPC: 27 xid 5445b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857818: rpc_wake_up_task_queue_locked: RPC: 27 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857818: rpc_wake_up_task_queue_locked: RPC: 27 disabling timer rpcbind-1871 [003] ..s. 50.857819: rpc_wake_up_task_queue_locked: RPC: 27 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857820: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857821: __rpc_execute: RPC: 27 sync task resuming rpc.nfsd-4720 [001] .... 50.857821: call_status: RPC: 27 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857821: call_decode: RPC: 27 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857822: call_decode: RPC: 27 call_decode result 0 rpc.nfsd-4720 [001] .... 50.857822: __rpc_execute: RPC: 27 return 0, status 0 rpc.nfsd-4720 [001] .... 50.857822: __rpc_execute: RPC: 27 release task rpc.nfsd-4720 [001] .... 50.857823: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857823: xprt_release: RPC: 27 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.857823: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.857824: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.857824: rpc_free_task: RPC: 27 freeing task rpc.nfsd-4720 [001] .... 50.857824: svc_write_space: svc: socket ffff880402966000(inet ffff8800db854180), write_space busy=1 rpc.nfsd-4720 [001] .... 50.857825: svc_setup_socket: svc: kernel_setsockopt returned 0 rpc.nfsd-4720 [001] .... 50.857825: svc_setup_socket: svc: svc_setup_socket created ffff880402966000 (inet ffff8800db854180) rpc.nfsd-4720 [001] .... 50.857974: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.857974: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.857975: __rpc_execute: RPC: 28 __rpc_execute flags=0x2280 rpc.nfsd-4720 [001] .... 50.857975: call_start: RPC: 28 call_start rpcbind4 proc UNSET (sync) rpc.nfsd-4720 [001] .... 50.857976: call_reserve: RPC: 28 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.857976: xprt_alloc_slot: RPC: 28 reserved req ffff880403542200 xid 5545b0ec rpc.nfsd-4720 [001] .... 50.857977: call_reserveresult: RPC: 28 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.857977: call_refresh: RPC: 28 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.857977: call_refreshresult: RPC: 28 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.857978: call_allocate: RPC: 28 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.857978: rpc_malloc: RPC: 28 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.857979: call_bind: RPC: 28 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.857979: call_connect: RPC: 28 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.857979: call_transmit: RPC: 28 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.857979: xprt_prepare_transmit: RPC: 28 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.857980: call_transmit: RPC: 28 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.857981: xprt_transmit: RPC: 28 xprt_transmit(68) rpc.nfsd-4720 [001] .... 50.857984: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4720 [001] .... 50.857984: xprt_transmit: RPC: 28 xmit complete rpc.nfsd-4720 [001] ..s. 50.857985: __rpc_sleep_on_priority: RPC: 28 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.857985: __rpc_sleep_on_priority: RPC: 28 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.857985: __rpc_sleep_on_priority: RPC: 28 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.857986: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.857986: __rpc_execute: RPC: 28 sync task going to sleep rpcbind-1871 [003] ..s. 50.857994: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.857994: xprt_complete_rqst: RPC: 28 xid 5545b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.857995: rpc_wake_up_task_queue_locked: RPC: 28 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.857995: rpc_wake_up_task_queue_locked: RPC: 28 disabling timer rpcbind-1871 [003] ..s. 50.857995: rpc_wake_up_task_queue_locked: RPC: 28 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.857996: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.857998: __rpc_execute: RPC: 28 sync task resuming rpc.nfsd-4720 [001] .... 50.857998: call_status: RPC: 28 call_status (status 28) rpc.nfsd-4720 [001] .... 50.857999: call_decode: RPC: 28 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.857999: call_decode: RPC: 28 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858000: __rpc_execute: RPC: 28 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858000: __rpc_execute: RPC: 28 release task rpc.nfsd-4720 [001] .... 50.858000: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858001: xprt_release: RPC: 28 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858001: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858002: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858002: rpc_free_task: RPC: 28 freeing task rpc.nfsd-4720 [001] .... 50.858003: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858003: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858003: __rpc_execute: RPC: 29 __rpc_execute flags=0x2280 rpc.nfsd-4720 [001] .... 50.858004: call_start: RPC: 29 call_start rpcbind4 proc UNSET (sync) rpc.nfsd-4720 [001] .... 50.858004: call_reserve: RPC: 29 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858004: xprt_alloc_slot: RPC: 29 reserved req ffff880403542200 xid 5645b0ec rpc.nfsd-4720 [001] .... 50.858005: call_reserveresult: RPC: 29 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858005: call_refresh: RPC: 29 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858005: call_refreshresult: RPC: 29 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858006: call_allocate: RPC: 29 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858006: rpc_malloc: RPC: 29 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858006: call_bind: RPC: 29 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858007: call_connect: RPC: 29 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858007: call_transmit: RPC: 29 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858007: xprt_prepare_transmit: RPC: 29 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858008: call_transmit: RPC: 29 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858008: xprt_transmit: RPC: 29 xprt_transmit(68) rpc.nfsd-4720 [001] .... 50.858010: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4720 [001] .... 50.858011: xprt_transmit: RPC: 29 xmit complete rpc.nfsd-4720 [001] ..s. 50.858011: __rpc_sleep_on_priority: RPC: 29 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858011: __rpc_sleep_on_priority: RPC: 29 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858012: __rpc_sleep_on_priority: RPC: 29 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858012: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858012: __rpc_execute: RPC: 29 sync task going to sleep rpcbind-1871 [003] ..s. 50.858019: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858020: xprt_complete_rqst: RPC: 29 xid 5645b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858020: rpc_wake_up_task_queue_locked: RPC: 29 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858020: rpc_wake_up_task_queue_locked: RPC: 29 disabling timer rpcbind-1871 [003] ..s. 50.858021: rpc_wake_up_task_queue_locked: RPC: 29 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858022: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858023: __rpc_execute: RPC: 29 sync task resuming rpc.nfsd-4720 [001] .... 50.858023: call_status: RPC: 29 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858024: call_decode: RPC: 29 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858024: call_decode: RPC: 29 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858025: __rpc_execute: RPC: 29 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858025: __rpc_execute: RPC: 29 release task rpc.nfsd-4720 [001] .... 50.858025: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858026: xprt_release: RPC: 29 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858026: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858026: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858027: rpc_free_task: RPC: 29 freeing task rpc.nfsd-4720 [001] .... 50.858027: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858027: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858028: __rpc_execute: RPC: 30 __rpc_execute flags=0x2280 rpc.nfsd-4720 [001] .... 50.858028: call_start: RPC: 30 call_start rpcbind4 proc UNSET (sync) rpc.nfsd-4720 [001] .... 50.858029: call_reserve: RPC: 30 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858029: xprt_alloc_slot: RPC: 30 reserved req ffff880403542200 xid 5745b0ec rpc.nfsd-4720 [001] .... 50.858029: call_reserveresult: RPC: 30 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858030: call_refresh: RPC: 30 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858030: call_refreshresult: RPC: 30 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858030: call_allocate: RPC: 30 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858030: rpc_malloc: RPC: 30 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858031: call_bind: RPC: 30 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858031: call_connect: RPC: 30 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858031: call_transmit: RPC: 30 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858032: xprt_prepare_transmit: RPC: 30 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858032: call_transmit: RPC: 30 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858032: xprt_transmit: RPC: 30 xprt_transmit(68) rpc.nfsd-4720 [001] .... 50.858035: xs_local_send_request: RPC: xs_local_send_request(68) = 0 rpc.nfsd-4720 [001] .... 50.858035: xprt_transmit: RPC: 30 xmit complete rpc.nfsd-4720 [001] ..s. 50.858035: __rpc_sleep_on_priority: RPC: 30 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858036: __rpc_sleep_on_priority: RPC: 30 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858036: __rpc_sleep_on_priority: RPC: 30 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858036: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858037: __rpc_execute: RPC: 30 sync task going to sleep rpcbind-1871 [003] ..s. 50.858043: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858044: xprt_complete_rqst: RPC: 30 xid 5745b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858044: rpc_wake_up_task_queue_locked: RPC: 30 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858045: rpc_wake_up_task_queue_locked: RPC: 30 disabling timer rpcbind-1871 [003] ..s. 50.858045: rpc_wake_up_task_queue_locked: RPC: 30 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858046: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858047: __rpc_execute: RPC: 30 sync task resuming rpc.nfsd-4720 [001] .... 50.858048: call_status: RPC: 30 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858048: call_decode: RPC: 30 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858049: call_decode: RPC: 30 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858049: __rpc_execute: RPC: 30 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858049: __rpc_execute: RPC: 30 release task rpc.nfsd-4720 [001] .... 50.858050: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858050: xprt_release: RPC: 30 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858050: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858051: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858051: rpc_free_task: RPC: 30 freeing task rpc.nfsd-4720 [001] .... 50.858053: svc_create_socket: svc: svc_create_socket(lockd, 17, 0.0.0.0, port=0) rpc.nfsd-4720 [001] .... 50.858058: svc_setup_socket: svc: svc_setup_socket ffff8804022955c0 rpc.nfsd-4720 [001] .... 50.858059: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858060: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858060: __rpc_execute: RPC: 31 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858060: call_start: RPC: 31 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858061: call_reserve: RPC: 31 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858061: xprt_alloc_slot: RPC: 31 reserved req ffff880403542200 xid 5845b0ec rpc.nfsd-4720 [001] .... 50.858061: call_reserveresult: RPC: 31 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858062: call_refresh: RPC: 31 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858062: call_refreshresult: RPC: 31 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858062: call_allocate: RPC: 31 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858063: rpc_malloc: RPC: 31 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858063: call_bind: RPC: 31 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858063: call_connect: RPC: 31 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858064: call_transmit: RPC: 31 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858064: xprt_prepare_transmit: RPC: 31 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858064: call_transmit: RPC: 31 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858065: xprt_transmit: RPC: 31 xprt_transmit(88) rpc.nfsd-4720 [001] .... 50.858067: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4720 [001] .... 50.858068: xprt_transmit: RPC: 31 xmit complete rpc.nfsd-4720 [001] ..s. 50.858068: __rpc_sleep_on_priority: RPC: 31 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858068: __rpc_sleep_on_priority: RPC: 31 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858069: __rpc_sleep_on_priority: RPC: 31 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858069: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858070: __rpc_execute: RPC: 31 sync task going to sleep rpcbind-1871 [003] ..s. 50.858080: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858081: xprt_complete_rqst: RPC: 31 xid 5845b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858081: rpc_wake_up_task_queue_locked: RPC: 31 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858081: rpc_wake_up_task_queue_locked: RPC: 31 disabling timer rpcbind-1871 [003] ..s. 50.858081: rpc_wake_up_task_queue_locked: RPC: 31 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858082: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858084: __rpc_execute: RPC: 31 sync task resuming rpc.nfsd-4720 [001] .... 50.858084: call_status: RPC: 31 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858085: call_decode: RPC: 31 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858085: call_decode: RPC: 31 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858085: __rpc_execute: RPC: 31 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858086: __rpc_execute: RPC: 31 release task rpc.nfsd-4720 [001] .... 50.858086: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858086: xprt_release: RPC: 31 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858087: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858087: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858087: rpc_free_task: RPC: 31 freeing task rpc.nfsd-4720 [001] .... 50.858088: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858089: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858089: __rpc_execute: RPC: 32 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858089: call_start: RPC: 32 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858090: call_reserve: RPC: 32 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858090: xprt_alloc_slot: RPC: 32 reserved req ffff880403542200 xid 5945b0ec rpc.nfsd-4720 [001] .... 50.858090: call_reserveresult: RPC: 32 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858090: call_refresh: RPC: 32 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858091: call_refreshresult: RPC: 32 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858091: call_allocate: RPC: 32 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858091: rpc_malloc: RPC: 32 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858091: call_bind: RPC: 32 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858092: call_connect: RPC: 32 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858092: call_transmit: RPC: 32 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858092: xprt_prepare_transmit: RPC: 32 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858092: call_transmit: RPC: 32 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858093: xprt_transmit: RPC: 32 xprt_transmit(88) rpc.nfsd-4720 [001] .... 50.858095: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4720 [001] .... 50.858095: xprt_transmit: RPC: 32 xmit complete rpc.nfsd-4720 [001] ..s. 50.858096: __rpc_sleep_on_priority: RPC: 32 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858096: __rpc_sleep_on_priority: RPC: 32 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858096: __rpc_sleep_on_priority: RPC: 32 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858097: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858097: __rpc_execute: RPC: 32 sync task going to sleep rpcbind-1871 [003] ..s. 50.858104: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858104: xprt_complete_rqst: RPC: 32 xid 5945b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858105: rpc_wake_up_task_queue_locked: RPC: 32 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858105: rpc_wake_up_task_queue_locked: RPC: 32 disabling timer rpcbind-1871 [003] ..s. 50.858105: rpc_wake_up_task_queue_locked: RPC: 32 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858106: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858107: __rpc_execute: RPC: 32 sync task resuming rpc.nfsd-4720 [001] .... 50.858108: call_status: RPC: 32 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858108: call_decode: RPC: 32 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858109: call_decode: RPC: 32 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858109: __rpc_execute: RPC: 32 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858109: __rpc_execute: RPC: 32 release task rpc.nfsd-4720 [001] .... 50.858109: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858110: xprt_release: RPC: 32 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858110: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858110: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858111: rpc_free_task: RPC: 32 freeing task rpc.nfsd-4720 [001] .... 50.858112: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858112: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858112: __rpc_execute: RPC: 33 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858112: call_start: RPC: 33 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858113: call_reserve: RPC: 33 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858113: xprt_alloc_slot: RPC: 33 reserved req ffff880403542200 xid 5a45b0ec rpc.nfsd-4720 [001] .... 50.858113: call_reserveresult: RPC: 33 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858114: call_refresh: RPC: 33 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858114: call_refreshresult: RPC: 33 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858114: call_allocate: RPC: 33 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858114: rpc_malloc: RPC: 33 allocated buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858115: call_bind: RPC: 33 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858115: call_connect: RPC: 33 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858115: call_transmit: RPC: 33 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858115: xprt_prepare_transmit: RPC: 33 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858116: call_transmit: RPC: 33 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858116: xprt_transmit: RPC: 33 xprt_transmit(88) rpc.nfsd-4720 [001] .... 50.858118: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4720 [001] .... 50.858118: xprt_transmit: RPC: 33 xmit complete rpc.nfsd-4720 [001] ..s. 50.858119: __rpc_sleep_on_priority: RPC: 33 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858119: __rpc_sleep_on_priority: RPC: 33 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858119: __rpc_sleep_on_priority: RPC: 33 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858120: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858120: __rpc_execute: RPC: 33 sync task going to sleep rpcbind-1871 [003] ..s. 50.858127: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858128: xprt_complete_rqst: RPC: 33 xid 5a45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858128: rpc_wake_up_task_queue_locked: RPC: 33 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858128: rpc_wake_up_task_queue_locked: RPC: 33 disabling timer rpcbind-1871 [003] ..s. 50.858128: rpc_wake_up_task_queue_locked: RPC: 33 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858129: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858131: __rpc_execute: RPC: 33 sync task resuming rpc.nfsd-4720 [001] .... 50.858131: call_status: RPC: 33 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858131: call_decode: RPC: 33 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858132: call_decode: RPC: 33 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858132: __rpc_execute: RPC: 33 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858132: __rpc_execute: RPC: 33 release task rpc.nfsd-4720 [001] .... 50.858133: rpc_free: RPC: freeing buffer of size 188 at ffff8800d8dc0000 rpc.nfsd-4720 [001] .... 50.858133: xprt_release: RPC: 33 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858133: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858133: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858134: rpc_free_task: RPC: 33 freeing task rpc.nfsd-4720 [001] .... 50.858134: svc_write_space: svc: socket ffff8800db83a000(inet ffff88040a14eb80), write_space busy=1 rpc.nfsd-4720 [001] .... 50.858135: svc_setup_socket: svc: kernel_setsockopt returned 0 rpc.nfsd-4720 [001] .... 50.858135: svc_setup_socket: svc: svc_setup_socket created ffff8800db83a000 (inet ffff88040a14eb80) rpc.nfsd-4720 [001] .... 50.858137: svc_create_socket: svc: svc_create_socket(lockd, 6, 0.0.0.0, port=0) rpc.nfsd-4720 [001] .... 50.858141: svc_setup_socket: svc: svc_setup_socket ffff88040acedd40 rpc.nfsd-4720 [001] .... 50.858142: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858143: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858143: __rpc_execute: RPC: 34 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858143: call_start: RPC: 34 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858144: call_reserve: RPC: 34 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858144: xprt_alloc_slot: RPC: 34 reserved req ffff880403542200 xid 5b45b0ec rpc.nfsd-4720 [001] .... 50.858144: call_reserveresult: RPC: 34 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858145: call_refresh: RPC: 34 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858145: call_refreshresult: RPC: 34 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858145: call_allocate: RPC: 34 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858146: rpc_malloc: RPC: 34 allocated buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858146: call_bind: RPC: 34 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858146: call_connect: RPC: 34 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858146: call_transmit: RPC: 34 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858147: xprt_prepare_transmit: RPC: 34 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858147: call_transmit: RPC: 34 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858147: xprt_transmit: RPC: 34 xprt_transmit(88) rpc.nfsd-4720 [001] .... 50.858150: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4720 [001] .... 50.858150: xprt_transmit: RPC: 34 xmit complete rpc.nfsd-4720 [001] ..s. 50.858150: __rpc_sleep_on_priority: RPC: 34 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858150: __rpc_sleep_on_priority: RPC: 34 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858151: __rpc_sleep_on_priority: RPC: 34 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858151: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858152: __rpc_execute: RPC: 34 sync task going to sleep rpcbind-1871 [003] ..s. 50.858158: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858159: xprt_complete_rqst: RPC: 34 xid 5b45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858159: rpc_wake_up_task_queue_locked: RPC: 34 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858160: rpc_wake_up_task_queue_locked: RPC: 34 disabling timer rpcbind-1871 [003] ..s. 50.858160: rpc_wake_up_task_queue_locked: RPC: 34 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858161: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858162: __rpc_execute: RPC: 34 sync task resuming rpc.nfsd-4720 [001] .... 50.858163: call_status: RPC: 34 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858163: call_decode: RPC: 34 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858164: call_decode: RPC: 34 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858164: __rpc_execute: RPC: 34 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858164: __rpc_execute: RPC: 34 release task rpc.nfsd-4720 [001] .... 50.858164: rpc_free: RPC: freeing buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858165: xprt_release: RPC: 34 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858165: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858165: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858166: rpc_free_task: RPC: 34 freeing task rpc.nfsd-4720 [001] .... 50.858167: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858167: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858167: __rpc_execute: RPC: 35 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858168: call_start: RPC: 35 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858168: call_reserve: RPC: 35 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858168: xprt_alloc_slot: RPC: 35 reserved req ffff880403542200 xid 5c45b0ec rpc.nfsd-4720 [001] .... 50.858168: call_reserveresult: RPC: 35 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858169: call_refresh: RPC: 35 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858169: call_refreshresult: RPC: 35 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858169: call_allocate: RPC: 35 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858170: rpc_malloc: RPC: 35 allocated buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858170: call_bind: RPC: 35 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858170: call_connect: RPC: 35 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858170: call_transmit: RPC: 35 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858171: xprt_prepare_transmit: RPC: 35 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858171: call_transmit: RPC: 35 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858171: xprt_transmit: RPC: 35 xprt_transmit(88) rpc.nfsd-4720 [001] .... 50.858173: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4720 [001] .... 50.858173: xprt_transmit: RPC: 35 xmit complete rpc.nfsd-4720 [001] ..s. 50.858174: __rpc_sleep_on_priority: RPC: 35 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858174: __rpc_sleep_on_priority: RPC: 35 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858174: __rpc_sleep_on_priority: RPC: 35 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858175: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858175: __rpc_execute: RPC: 35 sync task going to sleep rpcbind-1871 [003] ..s. 50.858182: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858183: xprt_complete_rqst: RPC: 35 xid 5c45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858183: rpc_wake_up_task_queue_locked: RPC: 35 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858184: rpc_wake_up_task_queue_locked: RPC: 35 disabling timer rpcbind-1871 [003] ..s. 50.858184: rpc_wake_up_task_queue_locked: RPC: 35 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858185: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858186: __rpc_execute: RPC: 35 sync task resuming rpc.nfsd-4720 [001] .... 50.858186: call_status: RPC: 35 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858187: call_decode: RPC: 35 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858187: call_decode: RPC: 35 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858187: __rpc_execute: RPC: 35 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858188: __rpc_execute: RPC: 35 release task rpc.nfsd-4720 [001] .... 50.858188: rpc_free: RPC: freeing buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858188: xprt_release: RPC: 35 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858189: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858189: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858189: rpc_free_task: RPC: 35 freeing task rpc.nfsd-4720 [001] .... 50.858190: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858190: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858191: __rpc_execute: RPC: 36 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858191: call_start: RPC: 36 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858191: call_reserve: RPC: 36 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858192: xprt_alloc_slot: RPC: 36 reserved req ffff880403542200 xid 5d45b0ec rpc.nfsd-4720 [001] .... 50.858192: call_reserveresult: RPC: 36 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858192: call_refresh: RPC: 36 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858192: call_refreshresult: RPC: 36 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858193: call_allocate: RPC: 36 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858193: rpc_malloc: RPC: 36 allocated buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858193: call_bind: RPC: 36 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858194: call_connect: RPC: 36 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858194: call_transmit: RPC: 36 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858194: xprt_prepare_transmit: RPC: 36 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858194: call_transmit: RPC: 36 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858195: xprt_transmit: RPC: 36 xprt_transmit(88) rpc.nfsd-4720 [001] .... 50.858197: xs_local_send_request: RPC: xs_local_send_request(88) = 0 rpc.nfsd-4720 [001] .... 50.858197: xprt_transmit: RPC: 36 xmit complete rpc.nfsd-4720 [001] ..s. 50.858198: __rpc_sleep_on_priority: RPC: 36 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858198: __rpc_sleep_on_priority: RPC: 36 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858198: __rpc_sleep_on_priority: RPC: 36 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858199: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858199: __rpc_execute: RPC: 36 sync task going to sleep rpcbind-1871 [003] ..s. 50.858206: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858206: xprt_complete_rqst: RPC: 36 xid 5d45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858207: rpc_wake_up_task_queue_locked: RPC: 36 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858207: rpc_wake_up_task_queue_locked: RPC: 36 disabling timer rpcbind-1871 [003] ..s. 50.858207: rpc_wake_up_task_queue_locked: RPC: 36 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858208: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858209: __rpc_execute: RPC: 36 sync task resuming rpc.nfsd-4720 [001] .... 50.858210: call_status: RPC: 36 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858210: call_decode: RPC: 36 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858210: call_decode: RPC: 36 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858211: __rpc_execute: RPC: 36 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858211: __rpc_execute: RPC: 36 release task rpc.nfsd-4720 [001] .... 50.858211: rpc_free: RPC: freeing buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858212: xprt_release: RPC: 36 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858212: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858212: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858212: rpc_free_task: RPC: 36 freeing task rpc.nfsd-4720 [001] .... 50.858213: svc_setup_socket: setting up TCP socket for listening rpc.nfsd-4720 [001] .... 50.858213: svc_setup_socket: svc: svc_setup_socket created ffff88040298e000 (inet ffff88040a708040) rpc.nfsd-4720 [001] .... 50.858215: svc_create_socket: svc: svc_create_socket(lockd, 17, 0000:0000:0000:0000:0000:0000:0000:0000, port=0) rpc.nfsd-4720 [001] .... 50.858217: svc_setup_socket: svc: svc_setup_socket ffff88040ba2c840 rpc.nfsd-4720 [001] .... 50.858218: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858218: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858219: __rpc_execute: RPC: 37 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858219: call_start: RPC: 37 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858220: call_reserve: RPC: 37 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858220: xprt_alloc_slot: RPC: 37 reserved req ffff880403542200 xid 5e45b0ec rpc.nfsd-4720 [001] .... 50.858220: call_reserveresult: RPC: 37 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858220: call_refresh: RPC: 37 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858221: call_refreshresult: RPC: 37 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858221: call_allocate: RPC: 37 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858221: rpc_malloc: RPC: 37 allocated buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858222: call_bind: RPC: 37 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858222: call_connect: RPC: 37 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858222: call_transmit: RPC: 37 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858222: xprt_prepare_transmit: RPC: 37 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858223: call_transmit: RPC: 37 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858223: xprt_transmit: RPC: 37 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.858225: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.858225: xprt_transmit: RPC: 37 xmit complete rpc.nfsd-4720 [001] ..s. 50.858226: __rpc_sleep_on_priority: RPC: 37 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858226: __rpc_sleep_on_priority: RPC: 37 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858226: __rpc_sleep_on_priority: RPC: 37 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858227: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858227: __rpc_execute: RPC: 37 sync task going to sleep rpcbind-1871 [003] ..s. 50.858234: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858234: xprt_complete_rqst: RPC: 37 xid 5e45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858235: rpc_wake_up_task_queue_locked: RPC: 37 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858235: rpc_wake_up_task_queue_locked: RPC: 37 disabling timer rpcbind-1871 [003] ..s. 50.858235: rpc_wake_up_task_queue_locked: RPC: 37 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858236: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858237: __rpc_execute: RPC: 37 sync task resuming rpc.nfsd-4720 [001] .... 50.858238: call_status: RPC: 37 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858238: call_decode: RPC: 37 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858239: call_decode: RPC: 37 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858239: __rpc_execute: RPC: 37 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858239: __rpc_execute: RPC: 37 release task rpc.nfsd-4720 [001] .... 50.858239: rpc_free: RPC: freeing buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858240: xprt_release: RPC: 37 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858240: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858240: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858241: rpc_free_task: RPC: 37 freeing task rpc.nfsd-4720 [001] .... 50.858242: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858242: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858242: __rpc_execute: RPC: 38 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858243: call_start: RPC: 38 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858243: call_reserve: RPC: 38 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858243: xprt_alloc_slot: RPC: 38 reserved req ffff880403542200 xid 5f45b0ec rpc.nfsd-4720 [001] .... 50.858243: call_reserveresult: RPC: 38 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858244: call_refresh: RPC: 38 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858244: call_refreshresult: RPC: 38 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858244: call_allocate: RPC: 38 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858245: rpc_malloc: RPC: 38 allocated buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858245: call_bind: RPC: 38 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858245: call_connect: RPC: 38 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858245: call_transmit: RPC: 38 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858245: xprt_prepare_transmit: RPC: 38 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858246: call_transmit: RPC: 38 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858246: xprt_transmit: RPC: 38 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.858248: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.858248: xprt_transmit: RPC: 38 xmit complete rpc.nfsd-4720 [001] ..s. 50.858249: __rpc_sleep_on_priority: RPC: 38 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858249: __rpc_sleep_on_priority: RPC: 38 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858249: __rpc_sleep_on_priority: RPC: 38 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858250: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858250: __rpc_execute: RPC: 38 sync task going to sleep rpcbind-1871 [003] ..s. 50.858257: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858257: xprt_complete_rqst: RPC: 38 xid 5f45b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858258: rpc_wake_up_task_queue_locked: RPC: 38 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858258: rpc_wake_up_task_queue_locked: RPC: 38 disabling timer rpcbind-1871 [003] ..s. 50.858258: rpc_wake_up_task_queue_locked: RPC: 38 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858259: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858260: __rpc_execute: RPC: 38 sync task resuming rpc.nfsd-4720 [001] .... 50.858261: call_status: RPC: 38 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858261: call_decode: RPC: 38 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858262: call_decode: RPC: 38 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858262: __rpc_execute: RPC: 38 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858262: __rpc_execute: RPC: 38 release task rpc.nfsd-4720 [001] .... 50.858262: rpc_free: RPC: freeing buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858263: xprt_release: RPC: 38 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858263: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858263: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858264: rpc_free_task: RPC: 38 freeing task rpc.nfsd-4720 [001] .... 50.858265: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858265: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858265: __rpc_execute: RPC: 39 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858265: call_start: RPC: 39 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858266: call_reserve: RPC: 39 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858266: xprt_alloc_slot: RPC: 39 reserved req ffff880403542200 xid 6045b0ec rpc.nfsd-4720 [001] .... 50.858266: call_reserveresult: RPC: 39 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858266: call_refresh: RPC: 39 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858267: call_refreshresult: RPC: 39 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858267: call_allocate: RPC: 39 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858268: rpc_malloc: RPC: 39 allocated buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858268: call_bind: RPC: 39 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858268: call_connect: RPC: 39 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858268: call_transmit: RPC: 39 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858269: xprt_prepare_transmit: RPC: 39 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858269: call_transmit: RPC: 39 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858269: xprt_transmit: RPC: 39 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.858271: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.858272: xprt_transmit: RPC: 39 xmit complete rpc.nfsd-4720 [001] ..s. 50.858272: __rpc_sleep_on_priority: RPC: 39 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858272: __rpc_sleep_on_priority: RPC: 39 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858272: __rpc_sleep_on_priority: RPC: 39 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858273: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858273: __rpc_execute: RPC: 39 sync task going to sleep rpcbind-1871 [003] ..s. 50.858280: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858280: xprt_complete_rqst: RPC: 39 xid 6045b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858281: rpc_wake_up_task_queue_locked: RPC: 39 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858281: rpc_wake_up_task_queue_locked: RPC: 39 disabling timer rpcbind-1871 [003] ..s. 50.858281: rpc_wake_up_task_queue_locked: RPC: 39 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858282: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858283: __rpc_execute: RPC: 39 sync task resuming rpc.nfsd-4720 [001] .... 50.858284: call_status: RPC: 39 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858284: call_decode: RPC: 39 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858285: call_decode: RPC: 39 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858285: __rpc_execute: RPC: 39 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858285: __rpc_execute: RPC: 39 release task rpc.nfsd-4720 [001] .... 50.858285: rpc_free: RPC: freeing buffer of size 188 at ffff88040a84a800 rpc.nfsd-4720 [001] .... 50.858286: xprt_release: RPC: 39 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858286: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858286: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858287: rpc_free_task: RPC: 39 freeing task rpc.nfsd-4720 [001] .... 50.858287: svc_write_space: svc: socket ffff8800db81e000(inet ffff8800db8545c0), write_space busy=1 rpc.nfsd-4720 [001] .... 50.858288: svc_setup_socket: svc: kernel_setsockopt returned 0 rpc.nfsd-4720 [001] .... 50.858288: svc_setup_socket: svc: svc_setup_socket created ffff8800db81e000 (inet ffff8800db8545c0) rpc.nfsd-4720 [001] .... 50.858289: svc_create_socket: svc: svc_create_socket(lockd, 6, 0000:0000:0000:0000:0000:0000:0000:0000, port=0) rpc.nfsd-4720 [001] .... 50.858293: svc_setup_socket: svc: svc_setup_socket ffff88040b9d12c0 rpc.nfsd-4720 [001] .... 50.858295: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858295: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858295: __rpc_execute: RPC: 40 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858296: call_start: RPC: 40 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858296: call_reserve: RPC: 40 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858296: xprt_alloc_slot: RPC: 40 reserved req ffff880403542200 xid 6145b0ec rpc.nfsd-4720 [001] .... 50.858296: call_reserveresult: RPC: 40 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858297: call_refresh: RPC: 40 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858297: call_refreshresult: RPC: 40 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858297: call_allocate: RPC: 40 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858298: rpc_malloc: RPC: 40 allocated buffer of size 188 at ffff88040a848000 rpc.nfsd-4720 [001] .... 50.858298: call_bind: RPC: 40 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858298: call_connect: RPC: 40 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858299: call_transmit: RPC: 40 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858299: xprt_prepare_transmit: RPC: 40 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858299: call_transmit: RPC: 40 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858300: xprt_transmit: RPC: 40 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.858302: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.858302: xprt_transmit: RPC: 40 xmit complete rpc.nfsd-4720 [001] ..s. 50.858302: __rpc_sleep_on_priority: RPC: 40 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858303: __rpc_sleep_on_priority: RPC: 40 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858303: __rpc_sleep_on_priority: RPC: 40 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858303: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858304: __rpc_execute: RPC: 40 sync task going to sleep rpcbind-1871 [003] ..s. 50.858310: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858311: xprt_complete_rqst: RPC: 40 xid 6145b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858311: rpc_wake_up_task_queue_locked: RPC: 40 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858311: rpc_wake_up_task_queue_locked: RPC: 40 disabling timer rpcbind-1871 [003] ..s. 50.858312: rpc_wake_up_task_queue_locked: RPC: 40 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858313: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858314: __rpc_execute: RPC: 40 sync task resuming rpc.nfsd-4720 [001] .... 50.858314: call_status: RPC: 40 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858315: call_decode: RPC: 40 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858315: call_decode: RPC: 40 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858315: __rpc_execute: RPC: 40 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858316: __rpc_execute: RPC: 40 release task rpc.nfsd-4720 [001] .... 50.858316: rpc_free: RPC: freeing buffer of size 188 at ffff88040a848000 rpc.nfsd-4720 [001] .... 50.858316: xprt_release: RPC: 40 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858317: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858317: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858317: rpc_free_task: RPC: 40 freeing task rpc.nfsd-4720 [001] .... 50.858318: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858318: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858319: __rpc_execute: RPC: 41 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858319: call_start: RPC: 41 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858319: call_reserve: RPC: 41 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858320: xprt_alloc_slot: RPC: 41 reserved req ffff880403542200 xid 6245b0ec rpc.nfsd-4720 [001] .... 50.858320: call_reserveresult: RPC: 41 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858320: call_refresh: RPC: 41 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858320: call_refreshresult: RPC: 41 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858321: call_allocate: RPC: 41 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858321: rpc_malloc: RPC: 41 allocated buffer of size 188 at ffff88040a848000 rpc.nfsd-4720 [001] .... 50.858321: call_bind: RPC: 41 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858322: call_connect: RPC: 41 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858322: call_transmit: RPC: 41 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858322: xprt_prepare_transmit: RPC: 41 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858322: call_transmit: RPC: 41 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858323: xprt_transmit: RPC: 41 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.858325: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.858325: xprt_transmit: RPC: 41 xmit complete rpc.nfsd-4720 [001] ..s. 50.858326: __rpc_sleep_on_priority: RPC: 41 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858326: __rpc_sleep_on_priority: RPC: 41 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858326: __rpc_sleep_on_priority: RPC: 41 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858327: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858327: __rpc_execute: RPC: 41 sync task going to sleep rpcbind-1871 [003] ..s. 50.858333: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858334: xprt_complete_rqst: RPC: 41 xid 6245b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858334: rpc_wake_up_task_queue_locked: RPC: 41 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858334: rpc_wake_up_task_queue_locked: RPC: 41 disabling timer rpcbind-1871 [003] ..s. 50.858335: rpc_wake_up_task_queue_locked: RPC: 41 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858336: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858337: __rpc_execute: RPC: 41 sync task resuming rpc.nfsd-4720 [001] .... 50.858337: call_status: RPC: 41 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858338: call_decode: RPC: 41 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858338: call_decode: RPC: 41 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858339: __rpc_execute: RPC: 41 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858339: __rpc_execute: RPC: 41 release task rpc.nfsd-4720 [001] .... 50.858339: rpc_free: RPC: freeing buffer of size 188 at ffff88040a848000 rpc.nfsd-4720 [001] .... 50.858339: xprt_release: RPC: 41 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858340: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858340: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858340: rpc_free_task: RPC: 41 freeing task rpc.nfsd-4720 [001] .... 50.858341: rpc_new_task: RPC: new task initialized, procpid 4720 rpc.nfsd-4720 [001] .... 50.858341: rpc_new_task: RPC: allocated task ffff88040b718200 rpc.nfsd-4720 [001] .... 50.858342: __rpc_execute: RPC: 42 __rpc_execute flags=0x680 rpc.nfsd-4720 [001] .... 50.858342: call_start: RPC: 42 call_start rpcbind4 proc SET (sync) rpc.nfsd-4720 [001] .... 50.858343: call_reserve: RPC: 42 call_reserve (status 0) rpc.nfsd-4720 [001] .... 50.858343: xprt_alloc_slot: RPC: 42 reserved req ffff880403542200 xid 6345b0ec rpc.nfsd-4720 [001] .... 50.858343: call_reserveresult: RPC: 42 call_reserveresult (status 0) rpc.nfsd-4720 [001] .... 50.858343: call_refresh: RPC: 42 call_refresh (status 0) rpc.nfsd-4720 [001] .... 50.858344: call_refreshresult: RPC: 42 call_refreshresult (status 0) rpc.nfsd-4720 [001] .... 50.858344: call_allocate: RPC: 42 call_allocate (status 0) rpc.nfsd-4720 [001] .... 50.858344: rpc_malloc: RPC: 42 allocated buffer of size 188 at ffff88040a848000 rpc.nfsd-4720 [001] .... 50.858344: call_bind: RPC: 42 call_bind (status 0) rpc.nfsd-4720 [001] .... 50.858345: call_connect: RPC: 42 call_connect xprt ffff880407939800 is connected rpc.nfsd-4720 [001] .... 50.858345: call_transmit: RPC: 42 call_transmit (status 0) rpc.nfsd-4720 [001] .... 50.858345: xprt_prepare_transmit: RPC: 42 xprt_prepare_transmit rpc.nfsd-4720 [001] .... 50.858345: call_transmit: RPC: 42 rpc_xdr_encode (status 0) rpc.nfsd-4720 [001] .... 50.858346: xprt_transmit: RPC: 42 xprt_transmit(84) rpc.nfsd-4720 [001] .... 50.858348: xs_local_send_request: RPC: xs_local_send_request(84) = 0 rpc.nfsd-4720 [001] .... 50.858348: xprt_transmit: RPC: 42 xmit complete rpc.nfsd-4720 [001] ..s. 50.858348: __rpc_sleep_on_priority: RPC: 42 sleep_on(queue "xprt_pending" time 4294904943) rpc.nfsd-4720 [001] ..s. 50.858349: __rpc_sleep_on_priority: RPC: 42 added to queue ffff880407939a58 "xprt_pending" rpc.nfsd-4720 [001] ..s. 50.858349: __rpc_sleep_on_priority: RPC: 42 setting alarm for 10000 ms rpc.nfsd-4720 [001] ..s. 50.858349: rpc_wake_up_first: RPC: wake_up_first(ffff880407939990 "xprt_sending") rpc.nfsd-4720 [001] .... 50.858350: __rpc_execute: RPC: 42 sync task going to sleep rpcbind-1871 [003] ..s. 50.858356: xs_local_data_ready: RPC: xs_local_data_ready... rpcbind-1871 [003] ..s. 50.858357: xprt_complete_rqst: RPC: 42 xid 6345b0ec complete (28 bytes received) rpcbind-1871 [003] ..s. 50.858357: rpc_wake_up_task_queue_locked: RPC: 42 __rpc_wake_up_task (now 4294904943) rpcbind-1871 [003] ..s. 50.858357: rpc_wake_up_task_queue_locked: RPC: 42 disabling timer rpcbind-1871 [003] ..s. 50.858358: rpc_wake_up_task_queue_locked: RPC: 42 removed from queue ffff880407939a58 "xprt_pending" rpcbind-1871 [003] ..s. 50.858359: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done rpc.nfsd-4720 [001] .... 50.858360: __rpc_execute: RPC: 42 sync task resuming rpc.nfsd-4720 [001] .... 50.858360: call_status: RPC: 42 call_status (status 28) rpc.nfsd-4720 [001] .... 50.858361: call_decode: RPC: 42 call_decode (status 28) rpc.nfsd-4720 [001] .... 50.858361: call_decode: RPC: 42 call_decode result 0 rpc.nfsd-4720 [001] .... 50.858361: __rpc_execute: RPC: 42 return 0, status 0 rpc.nfsd-4720 [001] .... 50.858362: __rpc_execute: RPC: 42 release task rpc.nfsd-4720 [001] .... 50.858362: rpc_free: RPC: freeing buffer of size 188 at ffff88040a848000 rpc.nfsd-4720 [001] .... 50.858362: xprt_release: RPC: 42 release request ffff880403542200 rpc.nfsd-4720 [001] .... 50.858363: rpc_wake_up_first: RPC: wake_up_first(ffff880407939b20 "xprt_backlog") rpc.nfsd-4720 [001] .... 50.858363: rpc_release_client: RPC: rpc_release_client(ffff88040a8b3600) rpc.nfsd-4720 [001] .... 50.858363: rpc_free_task: RPC: 42 freeing task rpc.nfsd-4720 [001] .... 50.858364: svc_setup_socket: setting up TCP socket for listening rpc.nfsd-4720 [001] .... 50.858364: svc_setup_socket: svc: svc_setup_socket created ffff880409d69000 (inet ffff88040a152040) lockd-4750 [003] .... 50.858433: svc_write_space: svc: socket ffff8800db83a000(inet ffff88040a14eb80), write_space busy=1 lockd-4750 [003] .... 50.858435: svc_tcp_accept: svc: tcp_accept ffff88040298e000 sock ffff88040acedd40 lockd-4750 [003] .... 50.858438: svc_write_space: svc: socket ffff8800db81e000(inet ffff8800db8545c0), write_space busy=1 lockd-4750 [003] .... 50.858439: svc_tcp_accept: svc: tcp_accept ffff880409d69000 sock ffff88040b9d12c0 nfsd-4771 [001] .... 50.956508: svc_tcp_accept: svc: tcp_accept ffff880402bd4000 sock ffff8800db68bac0 nfsd-4772 [003] .... 50.956510: svc_write_space: svc: socket ffff880402ba1000(inet ffff880407b2cc00), write_space busy=1 nfsd-4772 [003] .... 50.956516: svc_tcp_accept: svc: tcp_accept ffff88040cf94000 sock ffff88040ec230c0 nfsd-4771 [001] .... 50.956517: svc_write_space: svc: socket ffff880402966000(inet ffff8800db854180), write_space busy=1 <idle>-0 [003] ..s. 149.227677: svc_tcp_listen_data_ready: svc: socket ffff88040a708780 TCP (listen) state change 10 nfsd-4779 [003] .... 149.227697: svc_tcp_accept: svc: tcp_accept ffff880402bd4000 sock ffff8800db68bac0 nfsd-4779 [003] .... 149.227705: svc_tcp_accept: nfsd: connect from 192.168.23.22, port=867 nfsd-4779 [003] .... 149.227706: svc_setup_socket: svc: svc_setup_socket ffff8804081ad580 nfsd-4779 [003] .... 149.227708: svc_setup_socket: setting up TCP socket for reading nfsd-4779 [003] .... 149.227709: svc_setup_socket: svc: svc_setup_socket created ffff8800daaa1000 (inet ffff8800d8c817c0) nfsd-4777 [002] .... 149.227792: svc_tcp_accept: svc: tcp_accept ffff880402bd4000 sock ffff8800db68bac0 nfsd-4778 [000] .... 149.227792: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [000] .... 149.227797: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [000] .... 149.227798: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [000] .... 149.227798: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN <idle>-0 [003] ..s. 149.227933: svc_tcp_data_ready: svc: socket ffff8800d8c817c0 TCP data ready (svsk ffff8800daaa1000) nfsd-4779 [003] .... 149.227966: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4779 [003] .... 149.227970: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = 4 nfsd-4779 [003] .... 149.227971: svc_tcp_recvfrom: svc: TCP record, 92 bytes nfsd-4779 [003] .... 149.227973: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800dbb8a000, 4096) = 92 nfsd-4779 [003] .... 149.227974: svc_tcp_fragment_received: svc: TCP final record (92 bytes) nfsd-4778 [000] .... 149.228077: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [000] .... 149.228081: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [000] .... 149.228081: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [000] .... 149.228082: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN nfsd-4779 [003] .... 149.229024: svc_sendto: svc: socket ffff8800daaa1000 sendto([ffff8800dbb8b000 48... ], 48) = 48 (addr 192.168.23.22, port=867) <idle>-0 [003] ..s. 149.229463: svc_tcp_data_ready: svc: socket ffff8800d8c817c0 TCP data ready (svsk ffff8800daaa1000) nfsd-4779 [003] .... 149.229481: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4779 [003] .... 149.229484: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = 4 nfsd-4779 [003] .... 149.229485: svc_tcp_recvfrom: svc: TCP record, 184 bytes nfsd-4779 [003] .... 149.229487: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800dbb8a000, 4096) = 184 nfsd-4779 [003] .... 149.229488: svc_tcp_fragment_received: svc: TCP final record (184 bytes) nfsd-4779 [003] .... 149.229536: svc_sendto: svc: socket ffff8800daaa1000 sendto([ffff8800d9324000 64... ], 64) = 64 (addr 192.168.23.22, port=867) nfsd-4778 [000] .... 149.229558: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [000] .... 149.229562: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [000] .... 149.229562: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [000] .... 149.229563: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN <idle>-0 [003] ..s. 149.230009: svc_tcp_data_ready: svc: socket ffff8800d8c817c0 TCP data ready (svsk ffff8800daaa1000) nfsd-4779 [003] .... 149.230027: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4779 [003] .... 149.230031: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = 4 nfsd-4779 [003] .... 149.230032: svc_tcp_recvfrom: svc: TCP record, 100 bytes nfsd-4779 [003] .... 149.230034: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800dbb8a000, 4096) = 100 nfsd-4779 [003] .... 149.230035: svc_tcp_fragment_received: svc: TCP final record (100 bytes) nfsd-4779 [003] .... 149.230076: svc_sendto: svc: socket ffff8800daaa1000 sendto([ffff88003787b000 48... ], 48) = 48 (addr 192.168.23.22, port=867) nfsd-4778 [000] .... 149.230122: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [000] .... 149.230126: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [000] .... 149.230127: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [000] .... 149.230127: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN kworker/u32:3-105 [001] .... 149.230132: xs_setup_tcp: RPC: set up xprt to 192.168.23.22 (port 55201) via tcp kworker/u32:3-105 [001] .... 149.230143: xprt_create_transport: RPC: created transport ffff8800daa83800 with 65536 slots kworker/u32:3-105 [001] .... 149.230145: rpc_new_client: RPC: creating nfs4_cb client for (null) (xprt ffff8800daa83800) kworker/u32:3-105 [001] .... 149.230167: rpc_new_task: RPC: new task initialized, procpid 105 kworker/u32:3-105 [001] .... 149.230168: rpc_new_task: RPC: allocated task ffff880408652900 kworker/1:1H-131 [001] .... 149.230177: __rpc_execute: RPC: 43 __rpc_execute flags=0x681 kworker/1:1H-131 [001] .... 149.230179: call_start: RPC: 43 call_start nfs4_cb1 proc CB_NULL (async) kworker/1:1H-131 [001] .... 149.230180: call_reserve: RPC: 43 call_reserve (status 0) kworker/1:1H-131 [001] .... 149.230183: xprt_alloc_slot: RPC: 43 reserved req ffff8800daa7f000 xid 2f21bdb9 kworker/1:1H-131 [001] ..s. 149.230184: rpc_wake_up_first: RPC: wake_up_first(ffff8800daa83990 "xprt_sending") kworker/1:1H-131 [001] .... 149.230185: call_reserveresult: RPC: 43 call_reserveresult (status 0) kworker/1:1H-131 [001] .... 149.230186: call_refresh: RPC: 43 call_refresh (status 0) kworker/1:1H-131 [001] .... 149.230189: call_refreshresult: RPC: 43 call_refreshresult (status 0) kworker/1:1H-131 [001] .... 149.230190: call_allocate: RPC: 43 call_allocate (status 0) kworker/1:1H-131 [001] .... 149.230192: rpc_malloc: RPC: 43 allocated buffer of size 396 at ffff8800daafc000 kworker/1:1H-131 [001] .... 149.230192: call_bind: RPC: 43 call_bind (status 0) kworker/1:1H-131 [001] .... 149.230194: call_connect: RPC: 43 call_connect xprt ffff8800daa83800 is not connected kworker/1:1H-131 [001] .... 149.230195: xprt_connect: RPC: 43 xprt_connect xprt ffff8800daa83800 is not connected kworker/1:1H-131 [001] ..s. 149.230196: __rpc_sleep_on_priority: RPC: 43 sleep_on(queue "xprt_pending" time 4294929524) kworker/1:1H-131 [001] ..s. 149.230197: __rpc_sleep_on_priority: RPC: 43 added to queue ffff8800daa83a58 "xprt_pending" kworker/1:1H-131 [001] ..s. 149.230198: __rpc_sleep_on_priority: RPC: 43 setting alarm for 9000 ms kworker/1:1H-131 [001] .... 149.230201: xs_connect: RPC: xs_connect scheduled xprt ffff8800daa83800 kworker/1:1H-131 [001] ..s. 149.230212: inet_csk_get_port: kworker/1:1H:131 got port 947 kworker/1:1H-131 [001] .... 149.230274: xs_bind: RPC: xs_bind 0.0.0.0:947: ok (0) kworker/1:1H-131 [001] .... 149.230276: xs_tcp_setup_socket: RPC: worker connecting xprt ffff8800daa83800 via tcp to 192.168.23.22 (port 55201) kworker/1:1H-131 [001] .... 149.230310: xs_tcp_setup_socket: RPC: ffff8800daa83800 connect status 115 connected 0 sock state 2 kworker/1:1H-131 [001] ..s. 149.230312: rpc_wake_up_first: RPC: wake_up_first(ffff8800daa83990 "xprt_sending") <idle>-0 [003] ..s. 149.230583: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8800daa83800... <idle>-0 [003] ..s. 149.230585: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 <idle>-0 [003] ..s. 149.230587: rpc_wake_up_task_queue_locked: RPC: 43 __rpc_wake_up_task (now 4294929525) <idle>-0 [003] ..s. 149.230587: rpc_wake_up_task_queue_locked: RPC: 43 disabling timer <idle>-0 [003] ..s. 149.230588: rpc_wake_up_task_queue_locked: RPC: 43 removed from queue ffff8800daa83a58 "xprt_pending" <idle>-0 [003] .Ns. 149.230592: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done kworker/3:1H-124 [003] .... 149.230618: __rpc_execute: RPC: 43 __rpc_execute flags=0x681 kworker/3:1H-124 [003] .... 149.230619: xprt_connect_status: RPC: 43 xprt_connect_status: retrying kworker/3:1H-124 [003] .... 149.230621: call_connect_status: RPC: 43 call_connect_status (status -11) kworker/3:1H-124 [003] .... 149.230622: call_timeout: RPC: 43 call_timeout (minor) kworker/3:1H-124 [003] .... 149.230623: call_bind: RPC: 43 call_bind (status 0) kworker/3:1H-124 [003] .... 149.230624: call_connect: RPC: 43 call_connect xprt ffff8800daa83800 is connected kworker/3:1H-124 [003] .... 149.230625: call_transmit: RPC: 43 call_transmit (status 0) kworker/3:1H-124 [003] .... 149.230625: xprt_prepare_transmit: RPC: 43 xprt_prepare_transmit kworker/3:1H-124 [003] .... 149.230626: call_transmit: RPC: 43 rpc_xdr_encode (status 0) kworker/3:1H-124 [003] .... 149.230629: xprt_transmit: RPC: 43 xprt_transmit(72) kworker/3:1H-124 [003] .... 149.230644: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 kworker/3:1H-124 [003] .... 149.230644: xprt_transmit: RPC: 43 xmit complete kworker/3:1H-124 [003] ..s. 149.230645: __rpc_sleep_on_priority: RPC: 43 sleep_on(queue "xprt_pending" time 4294929525) kworker/3:1H-124 [003] ..s. 149.230646: __rpc_sleep_on_priority: RPC: 43 added to queue ffff8800daa83a58 "xprt_pending" kworker/3:1H-124 [003] ..s. 149.230647: __rpc_sleep_on_priority: RPC: 43 setting alarm for 9000 ms kworker/3:1H-124 [003] ..s. 149.230649: rpc_wake_up_first: RPC: wake_up_first(ffff8800daa83990 "xprt_sending") <idle>-0 [003] ..s. 149.230989: xs_tcp_data_ready: RPC: xs_tcp_data_ready... <idle>-0 [003] ..s. 149.230990: xs_tcp_data_recv: RPC: xs_tcp_data_recv started <idle>-0 [003] ..s. 149.230992: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 <idle>-0 [003] ..s. 149.230992: xs_tcp_data_recv: RPC: reading XID (4 bytes) <idle>-0 [003] ..s. 149.230994: xs_tcp_data_recv: RPC: reading request with XID 2f21bdb9 <idle>-0 [003] ..s. 149.230995: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) <idle>-0 [003] ..s. 149.230995: xs_tcp_data_recv: RPC: read reply XID 2f21bdb9 <idle>-0 [003] ..s. 149.230997: xs_tcp_data_recv: RPC: XID 2f21bdb9 read 16 bytes <idle>-0 [003] ..s. 149.230998: xs_tcp_data_recv: RPC: xprt = ffff8800daa83800, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 <idle>-0 [003] ..s. 149.230999: xprt_complete_rqst: RPC: 43 xid 2f21bdb9 complete (24 bytes received) <idle>-0 [003] ..s. 149.231000: rpc_wake_up_task_queue_locked: RPC: 43 __rpc_wake_up_task (now 4294929525) <idle>-0 [003] ..s. 149.231000: rpc_wake_up_task_queue_locked: RPC: 43 disabling timer <idle>-0 [003] ..s. 149.231002: rpc_wake_up_task_queue_locked: RPC: 43 removed from queue ffff8800daa83a58 "xprt_pending" <idle>-0 [003] .Ns. 149.231004: rpc_wake_up_task_queue_locked: RPC: __rpc_wake_up_task done <idle>-0 [003] .Ns. 149.231005: xs_tcp_data_recv: RPC: xs_tcp_data_recv done kworker/3:1H-124 [003] .... 149.231016: __rpc_execute: RPC: 43 __rpc_execute flags=0xe81 kworker/3:1H-124 [003] .... 149.231017: call_status: RPC: 43 call_status (status 24) kworker/3:1H-124 [003] .... 149.231018: call_decode: RPC: 43 call_decode (status 24) kworker/3:1H-124 [003] .... 149.231020: call_decode: RPC: 43 call_decode result 0 kworker/3:1H-124 [003] .... 149.231021: __rpc_execute: RPC: 43 return 0, status 0 kworker/3:1H-124 [003] .... 149.231022: __rpc_execute: RPC: 43 release task kworker/3:1H-124 [003] .... 149.231024: rpc_free: RPC: freeing buffer of size 396 at ffff8800daafc000 kworker/3:1H-124 [003] .... 149.231025: xprt_release: RPC: 43 release request ffff8800daa7f000 kworker/3:1H-124 [003] .... 149.231026: rpc_wake_up_first: RPC: wake_up_first(ffff8800daa83b20 "xprt_backlog") kworker/3:1H-124 [003] .... 149.231027: rpc_release_client: RPC: rpc_release_client(ffff8800daa7f800) kworker/3:1H-124 [003] .... 149.231028: rpc_free_task: RPC: 43 freeing task <idle>-0 [000] ..s. 154.237735: svc_tcp_data_ready: svc: socket ffff8800d8c817c0 TCP data ready (svsk ffff8800daaa1000) nfsd-4779 [003] .... 154.237808: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4779 [003] .... 154.237814: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = 4 nfsd-4779 [003] .... 154.237815: svc_tcp_recvfrom: svc: TCP record, 92 bytes nfsd-4779 [003] .... 154.237818: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800dbb8a000, 4096) = 92 nfsd-4779 [003] .... 154.237819: svc_tcp_fragment_received: svc: TCP final record (92 bytes) nfsd-4779 [003] .... 154.237872: svc_sendto: svc: socket ffff8800daaa1000 sendto([ffff8800d9324000 48... ], 48) = 48 (addr 192.168.23.22, port=867) nfsd-4778 [000] .... 154.237892: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [000] .... 154.237896: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [000] .... 154.237897: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [000] .... 154.237897: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN <idle>-0 [001] ..s. 214.282477: svc_tcp_data_ready: svc: socket ffff8800d8c817c0 TCP data ready (svsk ffff8800daaa1000) nfsd-4779 [003] .... 214.282554: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4779 [003] .... 214.282559: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = 4 nfsd-4779 [003] .... 214.282560: svc_tcp_recvfrom: svc: TCP record, 92 bytes nfsd-4779 [003] .... 214.282563: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800dbb8a000, 4096) = 92 nfsd-4779 [003] .... 214.282564: svc_tcp_fragment_received: svc: TCP final record (92 bytes) nfsd-4779 [003] .... 214.282617: svc_sendto: svc: socket ffff8800daaa1000 sendto([ffff8804086ca000 48... ], 48) = 48 (addr 192.168.23.22, port=867) nfsd-4778 [000] .... 214.282637: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [000] .... 214.282642: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [000] .... 214.282643: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [000] .... 214.282643: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN spamassassin-5827 [000] ..s. 274.471016: svc_tcp_data_ready: svc: socket ffff8800d8c817c0 TCP data ready (svsk ffff8800daaa1000) nfsd-4779 [003] .... 274.471080: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4779 [003] .... 274.471085: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = 4 nfsd-4779 [003] .... 274.471085: svc_tcp_recvfrom: svc: TCP record, 92 bytes nfsd-4779 [003] .... 274.471087: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800dbb8a000, 4096) = 92 nfsd-4779 [003] .... 274.471088: svc_tcp_fragment_received: svc: TCP final record (92 bytes) nfsd-4779 [003] .... 274.471130: svc_sendto: svc: socket ffff8800daaa1000 sendto([ffff88040a6dd000 48... ], 48) = 48 (addr 192.168.23.22, port=867) nfsd-4778 [001] .... 274.471134: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [001] .... 274.471136: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [001] .... 274.471136: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [001] .... 274.471137: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN <idle>-0 [000] .Ns. 334.659832: svc_tcp_data_ready: svc: socket ffff8800d8c817c0 TCP data ready (svsk ffff8800daaa1000) nfsd-4779 [003] .... 334.659927: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4779 [003] .... 334.659933: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = 4 nfsd-4779 [003] .... 334.659934: svc_tcp_recvfrom: svc: TCP record, 92 bytes nfsd-4779 [003] .... 334.659937: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800dbb8a000, 4096) = 92 nfsd-4779 [003] .... 334.659938: svc_tcp_fragment_received: svc: TCP final record (92 bytes) nfsd-4778 [001] .... 334.659952: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [001] .... 334.659957: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [001] .... 334.659957: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [001] .... 334.659958: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN nfsd-4779 [003] .... 334.659991: svc_sendto: svc: socket ffff8800daaa1000 sendto([ffff88040a043000 48... ], 48) = 48 (addr 192.168.23.22, port=867) <idle>-0 [003] ..s. 394.848497: svc_tcp_data_ready: svc: socket ffff8800d8c817c0 TCP data ready (svsk ffff8800daaa1000) nfsd-4779 [003] .... 394.848520: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4779 [003] .... 394.848524: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = 4 nfsd-4779 [003] .... 394.848525: svc_tcp_recvfrom: svc: TCP record, 92 bytes nfsd-4779 [003] .... 394.848527: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800dbb8a000, 4096) = 92 nfsd-4779 [003] .... 394.848528: svc_tcp_fragment_received: svc: TCP final record (92 bytes) nfsd-4779 [003] .... 394.848579: svc_sendto: svc: socket ffff8800daaa1000 sendto([ffff880402bc9000 48... ], 48) = 48 (addr 192.168.23.22, port=867) nfsd-4778 [001] .... 394.848599: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [001] .... 394.848603: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [001] .... 394.848604: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [001] .... 394.848604: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN kworker/3:1H-124 [003] ..s. 449.959363: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8800daa83800... kworker/3:1H-124 [003] ..s. 449.959366: xs_tcp_state_change: RPC: state 4 conn 1 dead 0 zapped 1 sk_shutdown 3 kworker/3:1H-124 [003] ..s. 449.959369: rpc_wake_up_first: RPC: wake_up_first(ffff8800daa83990 "xprt_sending") <idle>-0 [000] ..s. 449.959669: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8800daa83800... <idle>-0 [000] ..s. 449.959671: xs_tcp_state_change: RPC: state 5 conn 0 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [000] ..s. 449.959693: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8800daa83800... <idle>-0 [000] ..s. 449.959694: xs_tcp_state_change: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [000] ..s. 449.959695: xprt_disconnect_done: RPC: disconnected transport ffff8800daa83800 <idle>-0 [000] ..s. 449.959696: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff8800daa83800... <idle>-0 [000] ..s. 449.959696: xs_tcp_state_change: RPC: state 7 conn 0 dead 0 zapped 1 sk_shutdown 3 <idle>-0 [000] ..s. 449.959697: xprt_disconnect_done: RPC: disconnected transport ffff8800daa83800 <idle>-0 [000] ..s. 449.959698: xs_tcp_data_ready: RPC: xs_tcp_data_ready... <idle>-0 [003] ..s. 455.037231: svc_tcp_data_ready: svc: socket ffff8800d8c817c0 TCP data ready (svsk ffff8800daaa1000) nfsd-4779 [003] .... 455.037253: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4779 [003] .... 455.037267: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = 4 nfsd-4779 [003] .... 455.037268: svc_tcp_recvfrom: svc: TCP record, 92 bytes nfsd-4779 [003] .... 455.037270: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800dbb8a000, 4096) = 92 nfsd-4779 [003] .... 455.037272: svc_tcp_fragment_received: svc: TCP final record (92 bytes) nfsd-4779 [003] .... 455.037313: svc_sendto: svc: socket ffff8800daaa1000 sendto([ffff880408502000 48... ], 48) = 48 (addr 192.168.23.22, port=867) nfsd-4778 [001] .... 455.037340: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [001] .... 455.037345: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [001] .... 455.037346: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [001] .... 455.037347: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN <idle>-0 [002] ..s. 515.225890: svc_tcp_data_ready: svc: socket ffff8800d8c817c0 TCP data ready (svsk ffff8800daaa1000) nfsd-4779 [002] .... 515.225914: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4779 [002] .... 515.225918: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = 4 nfsd-4779 [002] .... 515.225919: svc_tcp_recvfrom: svc: TCP record, 92 bytes nfsd-4779 [002] .... 515.225921: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800dbb8a000, 4096) = 92 nfsd-4779 [002] .... 515.225922: svc_tcp_fragment_received: svc: TCP final record (92 bytes) nfsd-4779 [002] .... 515.225972: svc_sendto: svc: socket ffff8800daaa1000 sendto([ffff8800d8cab000 48... ], 48) = 48 (addr 192.168.23.22, port=867) nfsd-4778 [001] .... 515.225990: svc_tcp_recvfrom: svc: tcp_recv ffff8800daaa1000 data 1 conn 0 close 0 nfsd-4778 [001] .... 515.225995: svc_recvfrom.isra.10: svc: socket ffff8800daaa1000 recvfrom(ffff8800daaa12b8, 4) = -11 nfsd-4778 [001] .... 515.225995: svc_tcp_recvfrom: RPC: TCP recv_record got -11 nfsd-4778 [001] .... 515.225996: svc_tcp_recvfrom: RPC: TCP recvfrom got EAGAIN I don't see that 55201 anywhere. But then again, I didn't look for it before the port disappeared. I could reboot and look for it again. I should have saved the full netstat -tapn as well :-/ Oh well, I'll do this again (saving all the info and also netstat) -- Steve [-- Attachment #2: debug-nfs.patch --] [-- Type: text/x-patch, Size: 939 bytes --] diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index 3e44b9b0b78e..90cc377388b4 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -225,6 +225,9 @@ tb_not_found: tb->fastreuseport = 0; } success: + if (!current->mm) + trace_printk("%s:%d got port %d\n", current->comm, current->pid, + snum); if (!inet_csk(sk)->icsk_bind_hash) inet_bind_hash(sk, tb, snum); WARN_ON(inet_csk(sk)->icsk_bind_hash != tb); diff --git a/net/sunrpc/sunrpc.h b/net/sunrpc/sunrpc.h index f2b7cb540e61..8ea4ddaed8b3 100644 --- a/net/sunrpc/sunrpc.h +++ b/net/sunrpc/sunrpc.h @@ -29,6 +29,12 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. #include <linux/net.h> +#undef dprintk +#undef dprintk_rcu + +#define dprintk(args...) trace_printk(args) +#define dprintk_rcu(args...) trace_printk(args) + /* * Header for dynamically allocated rpc buffers. */ ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-19 16:25 ` Steven Rostedt @ 2015-06-19 17:17 ` Steven Rostedt 2015-06-19 17:39 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Steven Rostedt @ 2015-06-19 17:17 UTC (permalink / raw) To: Jeff Layton Cc: Trond Myklebust, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, bfields On Fri, 19 Jun 2015 12:25:53 -0400 Steven Rostedt <rostedt@goodmis.org> wrote: > I don't see that 55201 anywhere. But then again, I didn't look for it > before the port disappeared. I could reboot and look for it again. I > should have saved the full netstat -tapn as well :-/ Of course I didn't find it anywhere, that's the port on my wife's box that port 947 was connected to. Now I even went over to my wife's box and ran # rpcinfo -p localhost program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 34243 status 100024 1 tcp 34498 status which doesn't show anything. but something is listening to that port... # netstat -ntap |grep 55201 tcp 0 0 0.0.0.0:55201 0.0.0.0:* LISTEN I rebooted again, but this time I ran this on my wife's box: # trace-cmd record -e nfs -e nfs4 -e net -e skb -e sock -e udp -e workqueue -e sunrpc I started it when my server started booting the kernel, and kept it running till the port vanished. The full trace can be downloaded from http://rostedt.homelinux.com/private/wife-trace.txt Here's some interesting output from that trace: ksoftirq-13 1..s. 12272627.681760: netif_receive_skb: dev=lo skbaddr=0xffff88020944c600 len=88 ksoftirq-13 1..s. 12272627.681776: net_dev_queue: dev=eth0 skbaddr=0xffff880234e5b100 len=42 ksoftirq-13 1..s. 12272627.681777: net_dev_start_xmit: dev=eth0 queue_mapping=0 skbaddr=0xffff880234e5b100 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0806 ip_ summed=0 len=42 data_len=0 network_offset=14 transport_offset_valid=0 transport_offset=65533 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0 ksoftirq-13 1..s. 12272627.681779: net_dev_xmit: dev=eth0 skbaddr=0xffff880234e5b100 len=42 rc=0 ksoftirq-13 1..s. 12272627.681780: kfree_skb: skbaddr=0xffff88023444cf00 protocol=2048 location=0xffffffff81422a72 ksoftirq-13 1..s. 12272627.681783: rpc_socket_error: error=-113 socket:[11886206] dstaddr=192.168.23.9/2049 state=2 () sk_state=2 () ksoftirq-13 1..s. 12272627.681785: rpc_task_wakeup: task:18128@0 flags=5281 state=0006 status=-113 timeout=45000 queue=xprt_pending ksoftirq-13 1d.s. 12272627.681786: workqueue_queue_work: work struct=0xffff8800b5a94588 function=rpc_async_schedule workqueue=0xffff880234666800 req_cpu=512 cpu=1 ksoftirq-13 1d.s. 12272627.681787: workqueue_activate_work: work struct 0xffff8800b5a94588 ksoftirq-13 1..s. 12272627.681791: rpc_socket_state_change: socket:[11886206] dstaddr=192.168.23.9/2049 state=2 () sk_state=7 () ksoftirq-13 1..s. 12272627.681792: kfree_skb: skbaddr=0xffff88020944c600 protocol=2048 location=0xffffffff81482c05 kworker/-20111 1.... 12272627.681796: workqueue_execute_start: work struct 0xffff8800b5a94588: function rpc_async_schedule kworker/-20111 1.... 12272627.681797: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=-113 action=call_connect_status kworker/-20111 1.... 12272627.681798: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=-113 action=call_connect_status kworker/-20111 1.... 12272627.681798: rpc_connect_status: task:18128@0, status -113 kworker/-20111 1..s. 12272627.681799: rpc_task_sleep: task:18128@0 flags=5281 state=0005 status=0 timeout=750 queue=delayq kworker/-20111 1.... 12272627.681800: workqueue_execute_end: work struct 0xffff8800b5a94588 <idle>-0 1..s. 12272630.688741: rpc_task_wakeup: task:18128@0 flags=5281 state=0006 status=-110 timeout=750 queue=delayq <idle>-0 1dNs. 12272630.688749: workqueue_queue_work: work struct=0xffff8800b5a94588 function=rpc_async_schedule workqueue=0xffff880234666800 req_cpu=512 cpu=1 <idle>-0 1dNs. 12272630.688749: workqueue_activate_work: work struct 0xffff8800b5a94588 kworker/-20111 1.... 12272630.688758: workqueue_execute_start: work struct 0xffff8800b5a94588: function rpc_async_schedule kworker/-20111 1.... 12272630.688759: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=-110 action=call_timeout kworker/-20111 1.... 12272630.688760: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=0 action=call_timeout kworker/-20111 1.... 12272630.688760: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=0 action=call_bind kworker/-20111 1.... 12272630.688761: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=0 action=call_connect kworker/-20111 1..s. 12272630.688762: rpc_task_sleep: task:18128@0 flags=5281 state=0005 status=0 timeout=45000 queue=xprt_pending kworker/-20111 1.... 12272630.688765: workqueue_execute_end: work struct 0xffff8800b5a94588 <idle>-0 3d.s. 12272630.696742: workqueue_queue_work: work struct=0xffff880234ac9060 function=disk_events_workfn workqueue=0xffff8802370d9000 req_cpu=512 cpu=3 <idle>-0 3d.s. 12272630.696744: workqueue_activate_work: work struct 0xffff880234ac9060 kworker/-7491 3.... 12272630.696760: workqueue_execute_start: work struct 0xffff880234ac9060: function disk_events_workfn kworker/-7491 3d... 12272630.696827: workqueue_queue_work: work struct=0xffff8802347440b8 function=ata_sff_pio_task workqueue=0xffff880234491c00 req_cpu=512 cpu=3 kworker/-7491 3d... 12272630.696828: workqueue_activate_work: work struct 0xffff8802347440b8 kworker/-16140 3.... 12272630.696837: workqueue_execute_start: work struct 0xffff8802347440b8: function ata_sff_pio_task kworker/-16140 3.... 12272630.696853: workqueue_execute_end: work struct 0xffff8802347440b8 kworker/-7491 3.... 12272630.697383: workqueue_execute_end: work struct 0xffff880234ac9060 <idle>-0 1d.s. 12272654.753029: workqueue_queue_work: work struct=0xffff8802361f4de0 function=xs_tcp_setup_socket workqueue=0xffff880234666800 req_cpu=512 cpu=1 <idle>-0 1d.s. 12272654.753031: workqueue_activate_work: work struct 0xffff8802361f4de0 kworker/-20111 1.... 12272654.753049: workqueue_execute_start: work struct 0xffff8802361f4de0: function xs_tcp_setup_socket kworker/-20111 1..s. 12272654.753054: rpc_socket_error: error=-113 socket:[11886206] dstaddr=192.168.23.9/2049 state=2 () sk_state=7 () kworker/-20111 1.... 12272654.753055: rpc_socket_reset_connection: error=0 socket:[11886206] dstaddr=192.168.23.9/2049 state=1 () sk_state=7 () kworker/-20111 1..s. 12272654.753075: net_dev_queue: dev=eth0 skbaddr=0xffff880082117ae8 len=74 kworker/-20111 1..s. 12272654.753083: net_dev_start_xmit: dev=eth0 queue_mapping=0 skbaddr=0xffff880082117ae8 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=74 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=1 gso_type=0 kworker/-20111 1..s. 12272654.753088: net_dev_xmit: dev=eth0 skbaddr=0xffff880082117ae8 len=74 rc=0 kworker/-20111 1.... 12272654.753090: rpc_socket_connect: error=-115 socket:[11886206] dstaddr=192.168.23.9/2049 state=2 () sk_state=2 () kworker/-20111 1.... 12272654.753093: workqueue_execute_end: work struct 0xffff8802361f4de0 <idle>-0 1..s. 12272654.753320: consume_skb: skbaddr=0xffff880082117ae8 <idle>-0 1..s. 12272654.753503: napi_gro_receive_entry: dev=eth0 napi_id=0 queue_mapping=0 skbaddr=0xffff8801f647d100 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0806 ip_summed=0 hash=0x00000000 l4_hash=0 len=46 data_len=0 truesize=704 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0 <idle>-0 1.Ns. 12272654.753508: netif_receive_skb: dev=eth0 skbaddr=0xffff8801f647d100 len=46 <idle>-0 1.Ns. 12272654.753519: consume_skb: skbaddr=0xffff8800a9aa2d00 <idle>-0 1.Ns. 12272654.753522: net_dev_queue: dev=eth0 skbaddr=0xffff8800a9aa2d00 len=42 <idle>-0 1.Ns. 12272654.753523: net_dev_start_xmit: dev=eth0 queue_mapping=0 skbaddr=0xffff8800a9aa2d00 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0806 ip_summed=0 len=42 data_len=0 network_offset=14 transport_offset_valid=0 transport_offset=65533 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0 <idle>-0 1.Ns. 12272654.753525: net_dev_xmit: dev=eth0 skbaddr=0xffff8800a9aa2d00 len=42 rc=0 <idle>-0 1.Ns. 12272654.753526: consume_skb: skbaddr=0xffff8801f647d100 <idle>-0 1..s. 12272654.753585: napi_gro_receive_entry: dev=eth0 napi_id=0 queue_mapping=0 skbaddr=0xffff8801f647d100 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=1 hash=0x00000000 l4_hash=0 len=46 data_len=0 truesize=704 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0 <idle>-0 1.Ns. 12272654.753589: netif_receive_skb: dev=eth0 skbaddr=0xffff8801f647d100 len=46 <idle>-0 1.Ns. 12272654.753595: rpc_socket_error: error=-111 socket:[11886206] dstaddr=192.168.23.9/2049 state=2 () sk_state=2 () <idle>-0 1.Ns. 12272654.753597: rpc_task_wakeup: task:18128@0 flags=5281 state=0006 status=-111 timeout=45000 queue=xprt_pending <idle>-0 1dNs. 12272654.753598: workqueue_queue_work: work struct=0xffff8800b5a94588 function=rpc_async_schedule workqueue=0xffff880234666800 req_cpu=512 cpu=1 <idle>-0 1dNs. 12272654.753599: workqueue_activate_work: work struct 0xffff8800b5a94588 <idle>-0 1.Ns. 12272654.753601: rpc_socket_state_change: socket:[11886206] dstaddr=192.168.23.9/2049 state=2 () sk_state=7 () kworker/-20111 1.... 12272654.753607: workqueue_execute_start: work struct 0xffff8800b5a94588: function rpc_async_schedule kworker/-20111 1.... 12272654.753608: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=-111 action=call_connect_status kworker/-20111 1.... 12272654.753609: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=-111 action=call_connect_status kworker/-20111 1.... 12272654.753609: rpc_connect_status: task:18128@0, status -111 kworker/-20111 1..s. 12272654.753610: rpc_task_sleep: task:18128@0 flags=5281 state=0005 status=0 timeout=750 queue=delayq kworker/-20111 1.... 12272654.753612: workqueue_execute_end: work struct 0xffff8800b5a94588 <idle>-0 1..s. 12272654.753692: consume_skb: skbaddr=0xffff8800a9aa2d00 <idle>-0 1.Ns. 12272657.605105: netif_receive_skb: dev=eth0 skbaddr=0xffff8802223d1e00 len=345 <idle>-0 1.Ns. 12272657.605108: kfree_skb: skbaddr=0xffff8802223d1e00 protocol=2048 location=0xffffffff8147e361 <idle>-0 1..s. 12272657.760044: rpc_task_wakeup: task:18128@0 flags=5281 state=0006 status=-110 timeout=750 queue=delayq <idle>-0 1dNs. 12272657.760051: workqueue_queue_work: work struct=0xffff8800b5a94588 function=rpc_async_schedule workqueue=0xffff880234666800 req_cpu=512 cpu=1 <idle>-0 1dNs. 12272657.760052: workqueue_activate_work: work struct 0xffff8800b5a94588 kworker/-20111 1.... 12272657.760063: workqueue_execute_start: work struct 0xffff8800b5a94588: function rpc_async_schedule kworker/-20111 1.... 12272657.760064: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=-110 action=call_timeout kworker/-20111 1.... 12272657.760065: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=0 action=call_timeout kworker/-20111 1.... 12272657.760066: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=0 action=call_bind kworker/-20111 1.... 12272657.760066: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=0 action=call_connect kworker/-20111 1..s. 12272657.760068: rpc_task_sleep: task:18128@0 flags=5281 state=0005 status=0 timeout=45000 queue=xprt_pending kworker/-20111 1.... 12272657.760070: workqueue_execute_end: work struct 0xffff8800b5a94588 <idle>-0 3d.s. 12272657.824024: workqueue_queue_work: work struct=0xffff8800b5551760 function=disk_events_workfn workqueue=0xffff8802370d9000 req_cpu=512 cpu=3 <idle>-0 3d.s. 12272657.824025: workqueue_activate_work: work struct 0xffff8800b5551760 kworker/-7491 3.... 12272657.824041: workqueue_execute_start: work struct 0xffff8800b5551760: function disk_events_workfn kworker/-7491 3.... 12272657.824807: workqueue_execute_end: work struct 0xffff8800b5551760 <idle>-0 1d.s. 12272705.808564: workqueue_queue_work: work struct=0xffff8802361f4de0 function=xs_tcp_setup_socket workqueue=0xffff880234666800 req_cpu=512 cpu=1 <idle>-0 1d.s. 12272705.808565: workqueue_activate_work: work struct 0xffff8802361f4de0 kworker/-20111 1.... 12272705.808574: workqueue_execute_start: work struct 0xffff8802361f4de0: function xs_tcp_setup_socket kworker/-20111 1..s. 12272705.808580: rpc_socket_error: error=-111 socket:[11886206] dstaddr=192.168.23.9/2049 state=2 () sk_state=7 () kworker/-20111 1.... 12272705.808581: rpc_socket_reset_connection: error=0 socket:[11886206] dstaddr=192.168.23.9/2049 state=1 () sk_state=7 () kworker/-20111 1..s. 12272705.808599: net_dev_queue: dev=eth0 skbaddr=0xffff880082117ae8 len=74 kworker/-20111 1..s. 12272705.808602: net_dev_start_xmit: dev=eth0 queue_mapping=0 skbaddr=0xffff880082117ae8 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=74 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=1 gso_type=0 kworker/-20111 1..s. 12272705.808605: net_dev_xmit: dev=eth0 skbaddr=0xffff880082117ae8 len=74 rc=0 kworker/-20111 1.... 12272705.808614: rpc_socket_connect: error=-115 socket:[11886206] dstaddr=192.168.23.9/2049 state=2 () sk_state=2 () kworker/-20111 1.... 12272705.808615: workqueue_execute_end: work struct 0xffff8802361f4de0 <idle>-0 1..s. 12272705.808841: napi_gro_receive_entry: dev=eth0 napi_id=0 queue_mapping=0 skbaddr=0xffff8800a60e8900 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=1 hash=0x00000000 l4_hash=0 len=60 data_len=0 truesize=768 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0 <idle>-0 1.Ns. 12272705.808849: netif_receive_skb: dev=eth0 skbaddr=0xffff8800a60e8900 len=60 <idle>-0 1.Ns. 12272705.808872: rpc_socket_state_change: socket:[11886206] dstaddr=192.168.23.9/2049 state=2 () sk_state=1 () <idle>-0 1.Ns. 12272705.808874: rpc_task_wakeup: task:18128@0 flags=5281 state=0006 status=-11 timeout=45000 queue=xprt_pending <idle>-0 1dNs. 12272705.808875: workqueue_queue_work: work struct=0xffff8800b5a94588 function=rpc_async_schedule workqueue=0xffff880234666800 req_cpu=512 cpu=1 <idle>-0 1dNs. 12272705.808876: workqueue_activate_work: work struct 0xffff8800b5a94588 <idle>-0 1.Ns. 12272705.808881: net_dev_queue: dev=eth0 skbaddr=0xffff8800a60e8500 len=66 <idle>-0 1.Ns. 12272705.808883: net_dev_start_xmit: dev=eth0 queue_mapping=0 skbaddr=0xffff8800a60e8500 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=66 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=1 gso_type=0 <idle>-0 1.Ns. 12272705.808885: net_dev_xmit: dev=eth0 skbaddr=0xffff8800a60e8500 len=66 rc=0 <idle>-0 1.Ns. 12272705.808887: consume_skb: skbaddr=0xffff880082117ae8 kworker/-20111 1.... 12272705.808895: workqueue_execute_start: work struct 0xffff8800b5a94588: function rpc_async_schedule kworker/-20111 1.... 12272705.808896: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=-11 action=call_connect_status kworker/-20111 1.... 12272705.808897: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=-11 action=call_connect_status kworker/-20111 1.... 12272705.808897: rpc_connect_status: task:18128@0, status -11 kworker/-20111 1.... 12272705.808897: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=0 action=call_timeout kworker/-20111 1.... 12272705.808898: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=0 action=call_bind kworker/-20111 1.... 12272705.808899: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=0 action=call_connect kworker/-20111 1.... 12272705.808899: rpc_task_run_action: task:18128@0 flags=5281 state=0005 status=0 action=call_transmit kworker/-20111 1..s. 12272705.808912: net_dev_queue: dev=eth0 skbaddr=0xffff880082117ae8 len=162 kworker/-20111 1..s. 12272705.808913: net_dev_start_xmit: dev=eth0 queue_mapping=0 skbaddr=0xffff880082117ae8 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=162 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=1 gso_type=0 kworker/-20111 1..s. 12272705.808913: net_dev_xmit: dev=eth0 skbaddr=0xffff880082117ae8 len=162 rc=0 kworker/-20111 1..s. 12272705.808916: rpc_task_sleep: task:18128@0 flags=5a81 state=0005 status=0 timeout=45000 queue=xprt_pending kworker/-20111 1.... 12272705.808917: workqueue_execute_end: work struct 0xffff8800b5a94588 <idle>-0 1..s. 12272705.809098: consume_skb: skbaddr=0xffff8800a60e8500 <idle>-0 1.Ns. 12272705.809840: rpc_task_wakeup: task:18128@0 flags=5a81 state=0006 status=0 timeout=45000 queue=xprt_pending <idle>-0 1dNs. 12272705.809842: workqueue_queue_work: work struct=0xffff8800b5a94588 function=rpc_async_schedule workqueue=0xffff880234666800 req_cpu=512 cpu=1 <idle>-0 1dNs. 12272705.809842: workqueue_activate_work: work struct 0xffff8800b5a94588 kworker/-20111 1.... 12272705.809853: workqueue_execute_start: work struct 0xffff8800b5a94588: function rpc_async_schedule kworker/-20111 1.... 12272705.809853: rpc_task_run_action: task:18128@0 flags=5a81 state=0005 status=0 action=call_status kworker/-20111 1.... 12272705.809854: rpc_task_run_action: task:18128@0 flags=5a81 state=0005 status=0 action=call_status kworker/-20111 1.... 12272705.809854: rpc_task_run_action: task:18128@0 flags=5a81 state=0005 status=44 action=call_decode kworker/-20111 1.... 12272705.809856: rpc_task_run_action: task:18128@0 flags=5a81 state=0005 status=-10022 action=rpc_exit_task kworker/-20111 1.... 12272705.809858: nfs4_renew_async: error=-10022 () dstaddr=192.168.23.9 <idle>-0 1.Ns. 12272705.810000: consume_skb: skbaddr=0xffff8800a60e8900 kworker/-20111 1.... 12272705.810033: workqueue_execute_end: work struct 0xffff8800b5a94588 192.168.-16171 3.... 12272705.810062: rpc_task_begin: task:0@0 flags=5280 state=0000 status=0 action=irq_stack_union 192.168.-16171 3.... 12272705.810068: rpc_task_run_action: task:18129@0 flags=5280 state=0005 status=0 action=call_start 192.168.-16171 3.... 12272705.810069: rpc_task_run_action: task:18129@0 flags=5280 state=0005 status=0 action=call_reserve 192.168.-16171 3.... 12272705.810071: rpc_task_run_action: task:18129@0 flags=5280 state=0005 status=0 action=call_reserveresult 192.168.-16171 3.... 12272705.810071: rpc_task_run_action: task:18129@0 flags=5280 state=0005 status=0 action=call_refresh 192.168.-16171 3.... 12272705.810073: rpc_task_run_action: task:18129@0 flags=5280 state=0005 status=0 action=call_refreshresult 192.168.-16171 3.... 12272705.810073: rpc_task_run_action: task:18129@0 flags=5280 state=0005 status=0 action=call_allocate 192.168.-16171 3.... 12272705.810074: rpc_task_run_action: task:18129@0 flags=5280 state=0005 status=0 action=call_bind 192.168.-16171 3.... 12272705.810075: rpc_task_run_action: task:18129@0 flags=5280 state=0005 status=0 action=call_connect 192.168.-16171 3.... 12272705.810075: rpc_task_run_action: task:18129@0 flags=5280 state=0005 status=0 action=call_transmit 192.168.-16171 3..s. 12272705.810095: net_dev_queue: dev=eth0 skbaddr=0xffff8800a9b924e8 len=162 192.168.-16171 3..s. 12272705.810097: net_dev_start_xmit: dev=eth0 queue_mapping=0 skbaddr=0xffff8800a9b924e8 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=162 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=1 gso_type=0 192.168.-16171 3..s. 12272705.810098: net_dev_xmit: dev=eth0 skbaddr=0xffff8800a9b924e8 len=162 rc=0 192.168.-16171 3..s. 12272705.810101: rpc_task_sleep: task:18129@0 flags=5a80 state=0005 status=0 timeout=15000 queue=xprt_pending <idle>-0 1..s. 12272705.810318: napi_gro_receive_entry: dev=eth0 napi_id=0 queue_mapping=0 skbaddr=0xffff8800a60e8900 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=1 hash=0x00000000 l4_hash=0 len=100 data_len=0 truesize=768 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0 <idle>-0 1.Ns. 12272705.810326: netif_receive_skb: dev=eth0 skbaddr=0xffff8800a60e8900 len=100 <idle>-0 1.Ns. 12272705.810344: rpc_task_wakeup: task:18129@0 flags=5a80 state=0006 status=0 timeout=15000 queue=xprt_pending <idle>-0 1.Ns. 12272705.810349: consume_skb: skbaddr=0xffff8800a9b924e8 192.168.-16171 3.... 12272705.810379: rpc_task_run_action: task:18129@0 flags=5a80 state=0005 status=0 action=call_status 192.168.-16171 3.... 12272705.810385: rpc_task_run_action: task:18129@0 flags=5a80 state=0005 status=0 action=call_status 192.168.-16171 3.... 12272705.810385: rpc_task_run_action: task:18129@0 flags=5a80 state=0005 status=44 action=call_decode 192.168.-16171 3.... 12272705.810387: rpc_task_run_action: task:18129@0 flags=5a80 state=0005 status=-10022 action=rpc_exit_task 192.168.-16171 3.... 12272705.810397: rpc_task_begin: task:0@0 flags=5280 state=0000 status=0 action=irq_stack_union 192.168.-16171 3.... 12272705.810398: rpc_task_run_action: task:18130@0 flags=5280 state=0005 status=0 action=call_start 192.168.-16171 3.... 12272705.810398: rpc_task_run_action: task:18130@0 flags=5280 state=0005 status=0 action=call_reserve 192.168.-16171 3.... 12272705.810399: rpc_task_run_action: task:18130@0 flags=5280 state=0005 status=0 action=call_reserveresult 192.168.-16171 3.... 12272705.810399: rpc_task_run_action: task:18130@0 flags=5280 state=0005 status=0 action=call_refresh 192.168.-16171 3.... 12272705.810400: rpc_task_run_action: task:18130@0 flags=5280 state=0005 status=0 action=call_refreshresult 192.168.-16171 3.... 12272705.810400: rpc_task_run_action: task:18130@0 flags=5280 state=0005 status=0 action=call_allocate 192.168.-16171 3.... 12272705.810404: rpc_task_run_action: task:18130@0 flags=5280 state=0005 status=0 action=call_bind 192.168.-16171 3.... 12272705.810404: rpc_task_run_action: task:18130@0 flags=5280 state=0005 status=0 action=call_connect 192.168.-16171 3.... 12272705.810404: rpc_task_run_action: task:18130@0 flags=5280 state=0005 status=0 action=call_transmit 192.168.-16171 3..s. 12272705.810417: net_dev_queue: dev=eth0 skbaddr=0xffff8800a9b92ae8 len=254 192.168.-16171 3..s. 12272705.810418: net_dev_start_xmit: dev=eth0 queue_mapping=0 skbaddr=0xffff8800a9b92ae8 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=254 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=1 gso_type=0 192.168.-16171 3..s. 12272705.810426: net_dev_xmit: dev=eth0 skbaddr=0xffff8800a9b92ae8 len=254 rc=0 192.168.-16171 3..s. 12272705.810428: rpc_task_sleep: task:18130@0 flags=5a80 state=0005 status=0 timeout=15000 queue=xprt_pending 192.168.-16171 3.... 12272705.810902: rpc_task_run_action: task:18130@0 flags=5a80 state=0005 status=0 action=call_status 192.168.-16171 3.... 12272705.810908: rpc_task_run_action: task:18130@0 flags=5a80 state=0005 status=0 action=call_status 192.168.-16171 3.... 12272705.810908: rpc_task_run_action: task:18130@0 flags=5a80 state=0005 status=60 action=call_decode 192.168.-16171 3.... 12272705.810910: rpc_task_run_action: task:18130@0 flags=5a80 state=0005 status=0 action=rpc_exit_task 192.168.-16171 3.... 12272705.810914: nfs4_setclientid: error=0 (ACCESS) dstaddr=192.168.23.9 192.168.-16171 3.... 12272705.810915: rpc_task_begin: task:0@0 flags=5280 state=0000 status=0 action=irq_stack_union 192.168.-16171 3.... 12272705.810916: rpc_task_run_action: task:18131@0 flags=5280 state=0005 status=0 action=call_start 192.168.-16171 3.... 12272705.810916: rpc_task_run_action: task:18131@0 flags=5280 state=0005 status=0 action=call_reserve 192.168.-16171 3.... 12272705.810917: rpc_task_run_action: task:18131@0 flags=5280 state=0005 status=0 action=call_reserveresult 192.168.-16171 3.... 12272705.810917: rpc_task_run_action: task:18131@0 flags=5280 state=0005 status=0 action=call_refresh 192.168.-16171 3.... 12272705.810918: rpc_task_run_action: task:18131@0 flags=5280 state=0005 status=0 action=call_refreshresult 192.168.-16171 3.... 12272705.810918: rpc_task_run_action: task:18131@0 flags=5280 state=0005 status=0 action=call_allocate 192.168.-16171 3.... 12272705.810919: rpc_task_run_action: task:18131@0 flags=5280 state=0005 status=0 action=call_bind 192.168.-16171 3.... 12272705.810919: rpc_task_run_action: task:18131@0 flags=5280 state=0005 status=0 action=call_connect 192.168.-16171 3.... 12272705.810919: rpc_task_run_action: task:18131@0 flags=5280 state=0005 status=0 action=call_transmit 192.168.-16171 3..s. 12272705.810931: net_dev_queue: dev=eth0 skbaddr=0xffff8800a9b92ce8 len=170 192.168.-16171 3..s. 12272705.810932: net_dev_start_xmit: dev=eth0 queue_mapping=0 skbaddr=0xffff8800a9b92ce8 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=170 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=1 gso_type=0 192.168.-16171 3..s. 12272705.810933: net_dev_xmit: dev=eth0 skbaddr=0xffff8800a9b92ce8 len=170 rc=0 192.168.-16171 3..s. 12272705.810936: rpc_task_sleep: task:18131@0 flags=5a80 state=0005 status=0 timeout=15000 queue=xprt_pending 192.168.-16171 3.... 12272705.811213: rpc_task_run_action: task:18131@0 flags=5a80 state=0005 status=0 action=call_status 192.168.-16171 3.... 12272705.811220: rpc_task_run_action: task:18131@0 flags=5a80 state=0005 status=0 action=call_status 192.168.-16171 3.... 12272705.811220: rpc_task_run_action: task:18131@0 flags=5a80 state=0005 status=44 action=call_decode 192.168.-16171 3.... 12272705.811222: rpc_task_run_action: task:18131@0 flags=5a80 state=0005 status=0 action=rpc_exit_task 192.168.-16171 3.... 12272705.811227: nfs4_setclientid_confirm: error=0 (ACCESS) dstaddr=192.168.23.9 And it goes on, but you can look at the full trace. I just searched for "rpc". Maybe this will shed more light on the issue. I'll keep this kernel on my server for a little longer, but it's going to start triggering rkhunter warnings about hidden ports again. -- Steve -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-19 17:17 ` Steven Rostedt @ 2015-06-19 17:39 ` Trond Myklebust 2015-06-19 19:52 ` Jeff Layton 0 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2015-06-19 17:39 UTC (permalink / raw) To: Steven Rostedt Cc: Jeff Layton, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Bruce James Fields On Fri, Jun 19, 2015 at 1:17 PM, Steven Rostedt <rostedt@goodmis.org> wrote: > On Fri, 19 Jun 2015 12:25:53 -0400 > Steven Rostedt <rostedt@goodmis.org> wrote: > > >> I don't see that 55201 anywhere. But then again, I didn't look for it >> before the port disappeared. I could reboot and look for it again. I >> should have saved the full netstat -tapn as well :-/ > > Of course I didn't find it anywhere, that's the port on my wife's box > that port 947 was connected to. > > Now I even went over to my wife's box and ran > > # rpcinfo -p localhost > program vers proto port service > 100000 4 tcp 111 portmapper > 100000 3 tcp 111 portmapper > 100000 2 tcp 111 portmapper > 100000 4 udp 111 portmapper > 100000 3 udp 111 portmapper > 100000 2 udp 111 portmapper > 100024 1 udp 34243 status > 100024 1 tcp 34498 status > > which doesn't show anything. > > but something is listening to that port... > > # netstat -ntap |grep 55201 > tcp 0 0 0.0.0.0:55201 0.0.0.0:* LISTEN Hang on. This is on the client box while there is an active NFSv4 mount? Then that's probably the NFSv4 callback channel listening for delegation callbacks. Can you please try: echo "options nfs callback_tcpport=4048" > /etc/modprobe.d/nfs-local.conf and then either reboot the client or unload and then reload the nfs modules before reattempting the mount. If this is indeed the callback channel, then that will move your phantom listener to port 4048... Cheers Trond -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-19 17:39 ` Trond Myklebust @ 2015-06-19 19:52 ` Jeff Layton 2015-06-19 20:30 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Jeff Layton @ 2015-06-19 19:52 UTC (permalink / raw) To: Trond Myklebust Cc: Steven Rostedt, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Bruce James Fields On Fri, 19 Jun 2015 13:39:08 -0400 Trond Myklebust <trond.myklebust@primarydata.com> wrote: > On Fri, Jun 19, 2015 at 1:17 PM, Steven Rostedt <rostedt@goodmis.org> wrote: > > On Fri, 19 Jun 2015 12:25:53 -0400 > > Steven Rostedt <rostedt@goodmis.org> wrote: > > > > > >> I don't see that 55201 anywhere. But then again, I didn't look for it > >> before the port disappeared. I could reboot and look for it again. I > >> should have saved the full netstat -tapn as well :-/ > > > > Of course I didn't find it anywhere, that's the port on my wife's box > > that port 947 was connected to. > > > > Now I even went over to my wife's box and ran > > > > # rpcinfo -p localhost > > program vers proto port service > > 100000 4 tcp 111 portmapper > > 100000 3 tcp 111 portmapper > > 100000 2 tcp 111 portmapper > > 100000 4 udp 111 portmapper > > 100000 3 udp 111 portmapper > > 100000 2 udp 111 portmapper > > 100024 1 udp 34243 status > > 100024 1 tcp 34498 status > > > > which doesn't show anything. > > > > but something is listening to that port... > > > > # netstat -ntap |grep 55201 > > tcp 0 0 0.0.0.0:55201 0.0.0.0:* LISTEN > > > Hang on. This is on the client box while there is an active NFSv4 > mount? Then that's probably the NFSv4 callback channel listening for > delegation callbacks. > > Can you please try: > > echo "options nfs callback_tcpport=4048" > /etc/modprobe.d/nfs-local.conf > > and then either reboot the client or unload and then reload the nfs > modules before reattempting the mount. If this is indeed the callback > channel, then that will move your phantom listener to port 4048... > Right, it was a little unclear to me before, but it now seems clear that the callback socket that the server is opening to the client is the one squatting on the port. ...and that sort of makes sense, doesn't it? That rpc_clnt will stick around for the life of the client's lease, and the rpc_clnt binds to a particular port so that it can reconnect using the same one. Given that Stephen has done the legwork and figured out that reverting those commits fixes the issue, then I suspect that the real culprit is caf4ccd4e88cf2. The client is likely closing down the other end of the callback socket when it goes idle. Before that commit, we probably did an xs_close on it, but now we're doing a xs_tcp_shutdown and that leaves the port bound. I'm travelling this weekend and am not set up to reproduce it to confirm, but that does seem to be a plausible scenario. -- Jeff Layton <jlayton@poochiereds.net> -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-19 19:52 ` Jeff Layton @ 2015-06-19 20:30 ` Trond Myklebust 2015-06-19 22:14 ` Steven Rostedt 0 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2015-06-19 20:30 UTC (permalink / raw) To: Jeff Layton Cc: Steven Rostedt, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Bruce James Fields On Fri, 2015-06-19 at 15:52 -0400, Jeff Layton wrote: > On Fri, 19 Jun 2015 13:39:08 -0400 > Trond Myklebust <trond.myklebust@primarydata.com> wrote: > > > On Fri, Jun 19, 2015 at 1:17 PM, Steven Rostedt < > > rostedt@goodmis.org> wrote: > > > On Fri, 19 Jun 2015 12:25:53 -0400 > > > Steven Rostedt <rostedt@goodmis.org> wrote: > > > > > > > > > > I don't see that 55201 anywhere. But then again, I didn't look > > > > for it > > > > before the port disappeared. I could reboot and look for it > > > > again. I > > > > should have saved the full netstat -tapn as well :-/ > > > > > > Of course I didn't find it anywhere, that's the port on my wife's > > > box > > > that port 947 was connected to. > > > > > > Now I even went over to my wife's box and ran > > > > > > # rpcinfo -p localhost > > > program vers proto port service > > > 100000 4 tcp 111 portmapper > > > 100000 3 tcp 111 portmapper > > > 100000 2 tcp 111 portmapper > > > 100000 4 udp 111 portmapper > > > 100000 3 udp 111 portmapper > > > 100000 2 udp 111 portmapper > > > 100024 1 udp 34243 status > > > 100024 1 tcp 34498 status > > > > > > which doesn't show anything. > > > > > > but something is listening to that port... > > > > > > # netstat -ntap |grep 55201 > > > tcp 0 0 0.0.0.0:55201 0.0.0.0:* > > > LISTEN > > > > > > Hang on. This is on the client box while there is an active NFSv4 > > mount? Then that's probably the NFSv4 callback channel listening > > for > > delegation callbacks. > > > > Can you please try: > > > > echo "options nfs callback_tcpport=4048" > /etc/modprobe.d/nfs > > -local.conf > > > > and then either reboot the client or unload and then reload the nfs > > modules before reattempting the mount. If this is indeed the > > callback > > channel, then that will move your phantom listener to port 4048... > > > > Right, it was a little unclear to me before, but it now seems clear > that the callback socket that the server is opening to the client is > the one squatting on the port. > > ...and that sort of makes sense, doesn't it? That rpc_clnt will stick > around for the life of the client's lease, and the rpc_clnt binds to > a > particular port so that it can reconnect using the same one. > > Given that Stephen has done the legwork and figured out that > reverting > those commits fixes the issue, then I suspect that the real culprit > is > caf4ccd4e88cf2. > > The client is likely closing down the other end of the callback > socket when it goes idle. Before that commit, we probably did an > xs_close on it, but now we're doing a xs_tcp_shutdown and that leaves > the port bound. > Agreed. I've been looking into whether or not there is a simple fix. Reverting those patches is not an option, because the whole point was to ensure that the socket is in the TCP_CLOSED state before we release the socket. Steven, how about something like the following patch? 8<----------------------------------------------------------------- >From 9a0bcfdbdbc793eae1ed6d901a6396b6c66f9513 Mon Sep 17 00:00:00 2001 From: Trond Myklebust <trond.myklebust@primarydata.com> Date: Fri, 19 Jun 2015 16:17:57 -0400 Subject: [PATCH] SUNRPC: Ensure we release the TCP socket once it has been closed This fixes a regression introduced by commit caf4ccd4e88cf2 ("SUNRPC: Make xs_tcp_close() do a socket shutdown rather than a sock_release"). Prior to that commit, the autoclose feature would ensure that an idle connection would result in the socket being both disconnected and released, whereas now only gets disconnected. While the current behaviour is harmless, it does leave the port bound until either RPC traffic resumes or the RPC client is shut down. Reported-by: Steven Rostedt <rostedt@goodmis.org> Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> --- net/sunrpc/xprt.c | 2 +- net/sunrpc/xprtsock.c | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c index 3ca31f20b97c..ab5dd621ae0c 100644 --- a/net/sunrpc/xprt.c +++ b/net/sunrpc/xprt.c @@ -611,8 +611,8 @@ static void xprt_autoclose(struct work_struct *work) struct rpc_xprt *xprt = container_of(work, struct rpc_xprt, task_cleanup); - xprt->ops->close(xprt); clear_bit(XPRT_CLOSE_WAIT, &xprt->state); + xprt->ops->close(xprt); xprt_release_write(xprt, NULL); } diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c index fda8ec8c74c0..75dcdadf0269 100644 --- a/net/sunrpc/xprtsock.c +++ b/net/sunrpc/xprtsock.c @@ -634,10 +634,13 @@ static void xs_tcp_shutdown(struct rpc_xprt *xprt) struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); struct socket *sock = transport->sock; - if (sock != NULL) { + if (sock == NULL) + return; + if (xprt_connected(xprt)) { kernel_sock_shutdown(sock, SHUT_RDWR); trace_rpc_socket_shutdown(xprt, sock); - } + } else + xs_reset_transport(transport); } /** @@ -786,6 +789,7 @@ static void xs_sock_mark_closed(struct rpc_xprt *xprt) xs_sock_reset_connection_flags(xprt); /* Mark transport as closed and wake up all pending tasks */ xprt_disconnect_done(xprt); + xprt_force_disconnect(xprt); } /** -- 2.4.3 -- Trond Myklebust Linux NFS client maintainer, PrimaryData trond.myklebust@primarydata.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-19 20:30 ` Trond Myklebust @ 2015-06-19 22:14 ` Steven Rostedt 2015-06-19 23:25 ` Trond Myklebust 0 siblings, 1 reply; 15+ messages in thread From: Steven Rostedt @ 2015-06-19 22:14 UTC (permalink / raw) To: Trond Myklebust Cc: Jeff Layton, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Bruce James Fields On Fri, 19 Jun 2015 16:30:18 -0400 Trond Myklebust <trond.myklebust@primarydata.com> wrote: > Steven, how about something like the following patch? > OK, the box I'm running this on is using v4.0.5, can you make a patch based on that, as whatever you make needs to go to stable as well. distcc[31554] ERROR: compile /home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c on fedora/8 failed distcc[31554] (dcc_build_somewhere) Warning: remote compilation of '/home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c' failed, retrying locally distcc[31554] Warning: failed to distribute /home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c to fedora/8, running locally instead /home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c: In function 'xs_tcp_shutdown': /home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c:643:3: error: implicit declaration of function 'xs_reset_transport' [-Werror=implicit-function-declaration] /home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c: At top level: /home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c:825:13: warning: conflicting types for 'xs_reset_transport' [enabled by default] /home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c:825:13: error: static declaration of 'xs_reset_transport' follows non-static declaration /home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c:643:3: note: previous implicit declaration of 'xs_reset_transport' was here cc1: some warnings being treated as errors distcc[31554] ERROR: compile /home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c on localhost failed /home/rostedt/work/git/nobackup/linux-build.git/scripts/Makefile.build:258: recipe for target 'net/sunrpc/xprtsock.o' failed make[3]: *** [net/sunrpc/xprtsock.o] Error 1 -- Steve > 8<----------------------------------------------------------------- > >From 9a0bcfdbdbc793eae1ed6d901a6396b6c66f9513 Mon Sep 17 00:00:00 2001 > From: Trond Myklebust <trond.myklebust@primarydata.com> > Date: Fri, 19 Jun 2015 16:17:57 -0400 > Subject: [PATCH] SUNRPC: Ensure we release the TCP socket once it has been > closed > > This fixes a regression introduced by commit caf4ccd4e88cf2 ("SUNRPC: > Make xs_tcp_close() do a socket shutdown rather than a sock_release"). > Prior to that commit, the autoclose feature would ensure that an > idle connection would result in the socket being both disconnected and > released, whereas now only gets disconnected. > > While the current behaviour is harmless, it does leave the port bound > until either RPC traffic resumes or the RPC client is shut down. > > Reported-by: Steven Rostedt <rostedt@goodmis.org> > Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> > --- > net/sunrpc/xprt.c | 2 +- > net/sunrpc/xprtsock.c | 8 ++++++-- > 2 files changed, 7 insertions(+), 3 deletions(-) > > diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c > index 3ca31f20b97c..ab5dd621ae0c 100644 > --- a/net/sunrpc/xprt.c > +++ b/net/sunrpc/xprt.c > @@ -611,8 +611,8 @@ static void xprt_autoclose(struct work_struct *work) > struct rpc_xprt *xprt = > container_of(work, struct rpc_xprt, task_cleanup); > > - xprt->ops->close(xprt); > clear_bit(XPRT_CLOSE_WAIT, &xprt->state); > + xprt->ops->close(xprt); > xprt_release_write(xprt, NULL); > } > > diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c > index fda8ec8c74c0..75dcdadf0269 100644 > --- a/net/sunrpc/xprtsock.c > +++ b/net/sunrpc/xprtsock.c > @@ -634,10 +634,13 @@ static void xs_tcp_shutdown(struct rpc_xprt *xprt) > struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); > struct socket *sock = transport->sock; > > - if (sock != NULL) { > + if (sock == NULL) > + return; > + if (xprt_connected(xprt)) { > kernel_sock_shutdown(sock, SHUT_RDWR); > trace_rpc_socket_shutdown(xprt, sock); > - } > + } else > + xs_reset_transport(transport); > } > > /** > @@ -786,6 +789,7 @@ static void xs_sock_mark_closed(struct rpc_xprt *xprt) > xs_sock_reset_connection_flags(xprt); > /* Mark transport as closed and wake up all pending tasks */ > xprt_disconnect_done(xprt); > + xprt_force_disconnect(xprt); > } > > /** -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-19 22:14 ` Steven Rostedt @ 2015-06-19 23:25 ` Trond Myklebust 2015-06-20 1:27 ` Steven Rostedt 0 siblings, 1 reply; 15+ messages in thread From: Trond Myklebust @ 2015-06-19 23:25 UTC (permalink / raw) To: Steven Rostedt Cc: Jeff Layton, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Bruce James Fields On Fri, 2015-06-19 at 18:14 -0400, Steven Rostedt wrote: > On Fri, 19 Jun 2015 16:30:18 -0400 > Trond Myklebust <trond.myklebust@primarydata.com> wrote: > > > Steven, how about something like the following patch? > > > > OK, the box I'm running this on is using v4.0.5, can you make a patch > based on that, as whatever you make needs to go to stable as well. Is it causing any other damage than the rkhunter warning you reported? > distcc[31554] ERROR: compile /home/rostedt/work/git/nobackup/linux > -build.git/net/sunrpc/xprtsock.c on fedora/8 failed > distcc[31554] (dcc_build_somewhere) Warning: remote compilation of > '/home/rostedt/work/git/nobackup/linux > -build.git/net/sunrpc/xprtsock.c' failed, retrying locally > distcc[31554] Warning: failed to distribute > /home/rostedt/work/git/nobackup/linux-build.git/net/sunrpc/xprtsock.c > to fedora/8, running locally instead > /home/rostedt/work/git/nobackup/linux > -build.git/net/sunrpc/xprtsock.c: In function 'xs_tcp_shutdown': > /home/rostedt/work/git/nobackup/linux > -build.git/net/sunrpc/xprtsock.c:643:3: error: implicit declaration > of function 'xs_reset_transport' [-Werror=implicit-function > -declaration] > /home/rostedt/work/git/nobackup/linux > -build.git/net/sunrpc/xprtsock.c: At top level: > /home/rostedt/work/git/nobackup/linux > -build.git/net/sunrpc/xprtsock.c:825:13: warning: conflicting types > for 'xs_reset_transport' [enabled by default] > /home/rostedt/work/git/nobackup/linux > -build.git/net/sunrpc/xprtsock.c:825:13: error: static declaration of > 'xs_reset_transport' follows non-static declaration > /home/rostedt/work/git/nobackup/linux > -build.git/net/sunrpc/xprtsock.c:643:3: note: previous implicit > declaration of 'xs_reset_transport' was here > cc1: some warnings being treated as errors > distcc[31554] ERROR: compile /home/rostedt/work/git/nobackup/linux > -build.git/net/sunrpc/xprtsock.c on localhost failed > /home/rostedt/work/git/nobackup/linux > -build.git/scripts/Makefile.build:258: recipe for target > 'net/sunrpc/xprtsock.o' failed > make[3]: *** [net/sunrpc/xprtsock.o] Error 1 Sorry. I sent that one off too quickly. Try the following. 8<-------------------------------------------------------------- >From 4876cc779ff525b9c2376d8076edf47815e71f2c Mon Sep 17 00:00:00 2001 From: Trond Myklebust <trond.myklebust@primarydata.com> Date: Fri, 19 Jun 2015 16:17:57 -0400 Subject: [PATCH v2] SUNRPC: Ensure we release the TCP socket once it has been closed This fixes a regression introduced by commit caf4ccd4e88cf2 ("SUNRPC: Make xs_tcp_close() do a socket shutdown rather than a sock_release"). Prior to that commit, the autoclose feature would ensure that an idle connection would result in the socket being both disconnected and released, whereas now only gets disconnected. While the current behaviour is harmless, it does leave the port bound until either RPC traffic resumes or the RPC client is shut down. Reported-by: Steven Rostedt <rostedt@goodmis.org> Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> --- net/sunrpc/xprt.c | 2 +- net/sunrpc/xprtsock.c | 40 ++++++++++++++++++++++------------------ 2 files changed, 23 insertions(+), 19 deletions(-) diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c index 3ca31f20b97c..ab5dd621ae0c 100644 --- a/net/sunrpc/xprt.c +++ b/net/sunrpc/xprt.c @@ -611,8 +611,8 @@ static void xprt_autoclose(struct work_struct *work) struct rpc_xprt *xprt = container_of(work, struct rpc_xprt, task_cleanup); - xprt->ops->close(xprt); clear_bit(XPRT_CLOSE_WAIT, &xprt->state); + xprt->ops->close(xprt); xprt_release_write(xprt, NULL); } diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c index fda8ec8c74c0..ee0715dfc3c7 100644 --- a/net/sunrpc/xprtsock.c +++ b/net/sunrpc/xprtsock.c @@ -623,24 +623,6 @@ process_status: } /** - * xs_tcp_shutdown - gracefully shut down a TCP socket - * @xprt: transport - * - * Initiates a graceful shutdown of the TCP socket by calling the - * equivalent of shutdown(SHUT_RDWR); - */ -static void xs_tcp_shutdown(struct rpc_xprt *xprt) -{ - struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); - struct socket *sock = transport->sock; - - if (sock != NULL) { - kernel_sock_shutdown(sock, SHUT_RDWR); - trace_rpc_socket_shutdown(xprt, sock); - } -} - -/** * xs_tcp_send_request - write an RPC request to a TCP socket * @task: address of RPC task that manages the state of an RPC request * @@ -786,6 +768,7 @@ static void xs_sock_mark_closed(struct rpc_xprt *xprt) xs_sock_reset_connection_flags(xprt); /* Mark transport as closed and wake up all pending tasks */ xprt_disconnect_done(xprt); + xprt_force_disconnect(xprt); } /** @@ -2103,6 +2086,27 @@ out: xprt_wake_pending_tasks(xprt, status); } +/** + * xs_tcp_shutdown - gracefully shut down a TCP socket + * @xprt: transport + * + * Initiates a graceful shutdown of the TCP socket by calling the + * equivalent of shutdown(SHUT_RDWR); + */ +static void xs_tcp_shutdown(struct rpc_xprt *xprt) +{ + struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); + struct socket *sock = transport->sock; + + if (sock == NULL) + return; + if (xprt_connected(xprt)) { + kernel_sock_shutdown(sock, SHUT_RDWR); + trace_rpc_socket_shutdown(xprt, sock); + } else + xs_reset_transport(transport); +} + static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock) { struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); -- 2.4.3 -- Trond Myklebust Linux NFS client maintainer, PrimaryData trond.myklebust@primarydata.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) 2015-06-19 23:25 ` Trond Myklebust @ 2015-06-20 1:27 ` Steven Rostedt 2016-06-22 16:41 ` It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) Steven Rostedt 0 siblings, 1 reply; 15+ messages in thread From: Steven Rostedt @ 2015-06-20 1:27 UTC (permalink / raw) To: Trond Myklebust Cc: Jeff Layton, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Bruce James Fields On Fri, 19 Jun 2015 19:25:59 -0400 Trond Myklebust <trond.myklebust@primarydata.com> wrote: > 8<-------------------------------------------------------------- > >From 4876cc779ff525b9c2376d8076edf47815e71f2c Mon Sep 17 00:00:00 2001 > From: Trond Myklebust <trond.myklebust@primarydata.com> > Date: Fri, 19 Jun 2015 16:17:57 -0400 > Subject: [PATCH v2] SUNRPC: Ensure we release the TCP socket once it has been > closed > > This fixes a regression introduced by commit caf4ccd4e88cf2 ("SUNRPC: > Make xs_tcp_close() do a socket shutdown rather than a sock_release"). > Prior to that commit, the autoclose feature would ensure that an > idle connection would result in the socket being both disconnected and > released, whereas now only gets disconnected. > > While the current behaviour is harmless, it does leave the port bound > until either RPC traffic resumes or the RPC client is shut down. Is there a way to test RPC traffic resuming? I'd like to try that before declaring this bug harmless. > > Reported-by: Steven Rostedt <rostedt@goodmis.org> The problem appears to go away with this patch. Tested-by: Steven Rostedt <rostedt@goodmis.org> Thanks a lot! -- Steve > Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com> > --- > net/sunrpc/xprt.c | 2 +- > net/sunrpc/xprtsock.c | 40 ++++++++++++++++++++++------------------ > 2 files changed, 23 insertions(+), 19 deletions(-) > > diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c > index 3ca31f20b97c..ab5dd621ae0c 100644 > --- a/net/sunrpc/xprt.c > +++ b/net/sunrpc/xprt.c > @@ -611,8 +611,8 @@ static void xprt_autoclose(struct work_struct *work) > struct rpc_xprt *xprt = > container_of(work, struct rpc_xprt, task_cleanup); > > - xprt->ops->close(xprt); > clear_bit(XPRT_CLOSE_WAIT, &xprt->state); > + xprt->ops->close(xprt); > xprt_release_write(xprt, NULL); > } > > diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c > index fda8ec8c74c0..ee0715dfc3c7 100644 > --- a/net/sunrpc/xprtsock.c > +++ b/net/sunrpc/xprtsock.c > @@ -623,24 +623,6 @@ process_status: > } > > /** > - * xs_tcp_shutdown - gracefully shut down a TCP socket > - * @xprt: transport > - * > - * Initiates a graceful shutdown of the TCP socket by calling the > - * equivalent of shutdown(SHUT_RDWR); > - */ > -static void xs_tcp_shutdown(struct rpc_xprt *xprt) > -{ > - struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); > - struct socket *sock = transport->sock; > - > - if (sock != NULL) { > - kernel_sock_shutdown(sock, SHUT_RDWR); > - trace_rpc_socket_shutdown(xprt, sock); > - } > -} > - > -/** > * xs_tcp_send_request - write an RPC request to a TCP socket > * @task: address of RPC task that manages the state of an RPC request > * > @@ -786,6 +768,7 @@ static void xs_sock_mark_closed(struct rpc_xprt *xprt) > xs_sock_reset_connection_flags(xprt); > /* Mark transport as closed and wake up all pending tasks */ > xprt_disconnect_done(xprt); > + xprt_force_disconnect(xprt); > } > > /** > @@ -2103,6 +2086,27 @@ out: > xprt_wake_pending_tasks(xprt, status); > } > > +/** > + * xs_tcp_shutdown - gracefully shut down a TCP socket > + * @xprt: transport > + * > + * Initiates a graceful shutdown of the TCP socket by calling the > + * equivalent of shutdown(SHUT_RDWR); > + */ > +static void xs_tcp_shutdown(struct rpc_xprt *xprt) > +{ > + struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); > + struct socket *sock = transport->sock; > + > + if (sock == NULL) > + return; > + if (xprt_connected(xprt)) { > + kernel_sock_shutdown(sock, SHUT_RDWR); > + trace_rpc_socket_shutdown(xprt, sock); > + } else > + xs_reset_transport(transport); > +} > + > static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock) > { > struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 15+ messages in thread
* It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) 2015-06-20 1:27 ` Steven Rostedt @ 2016-06-22 16:41 ` Steven Rostedt 0 siblings, 0 replies; 15+ messages in thread From: Steven Rostedt @ 2016-06-22 16:41 UTC (permalink / raw) To: Trond Myklebust Cc: Jeff Layton, Eric Dumazet, Anna Schumaker, Linux NFS Mailing List, Linux Network Devel Mailing List, LKML, Andrew Morton, Bruce James Fields [-- Attachment #1: Type: text/plain, Size: 3957 bytes --] I've hit this again. Not sure when it started, but I applied my old debug trace_printk() patch (attached) and rebooted (4.5.7). I just tested the latest kernel from Linus's tree (from last nights pull), and it still gives me the problem. Here's the trace I have: kworker/3:1H-134 [003] ..s. 61.036129: inet_csk_get_port: snum 805 kworker/3:1H-134 [003] ..s. 61.036135: <stack trace> => sched_clock => inet_addr_type_table => security_capable => inet_bind => xs_bind => release_sock => sock_setsockopt => __sock_create => xs_create_sock.isra.19 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => ret_from_fork => kthread kworker/3:1H-134 [003] ..s. 61.036136: inet_bind_hash: add 805 kworker/3:1H-134 [003] ..s. 61.036138: <stack trace> => inet_csk_get_port => sched_clock => inet_addr_type_table => security_capable => inet_bind => xs_bind => release_sock => sock_setsockopt => __sock_create => xs_create_sock.isra.19 => xs_tcp_setup_socket => process_one_work => worker_thread => worker_thread => kthread => ret_from_fork => kthread kworker/3:1H-134 [003] .... 61.036139: xs_bind: RPC: xs_bind 4.136.255.255:805: ok (0) kworker/3:1H-134 [003] .... 61.036140: xs_tcp_setup_socket: RPC: worker connecting xprt ffff880407eca800 via tcp to 192.168.23.22 (port 43651) kworker/3:1H-134 [003] .... 61.036162: xs_tcp_setup_socket: RPC: ffff880407eca800 connect status 115 connected 0 sock state 2 <idle>-0 [001] ..s. 61.036450: xs_tcp_state_change: RPC: xs_tcp_state_change client ffff880407eca800... <idle>-0 [001] ..s. 61.036452: xs_tcp_state_change: RPC: state 1 conn 0 dead 0 zapped 1 sk_shutdown 0 kworker/1:1H-136 [001] .... 61.036476: xprt_connect_status: RPC: 43 xprt_connect_status: retrying kworker/1:1H-136 [001] .... 61.036478: xprt_prepare_transmit: RPC: 43 xprt_prepare_transmit kworker/1:1H-136 [001] .... 61.036479: xprt_transmit: RPC: 43 xprt_transmit(72) kworker/1:1H-136 [001] .... 61.036486: xs_tcp_send_request: RPC: xs_tcp_send_request(72) = 0 kworker/1:1H-136 [001] .... 61.036487: xprt_transmit: RPC: 43 xmit complete <idle>-0 [001] ..s. 61.036789: xs_tcp_data_ready: RPC: xs_tcp_data_ready... kworker/1:1H-136 [001] .... 61.036798: xs_tcp_data_recv: RPC: xs_tcp_data_recv started kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading TCP record fragment of length 24 kworker/1:1H-136 [001] .... 61.036799: xs_tcp_data_recv: RPC: reading XID (4 bytes) kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading request with XID 2f4c3f88 kworker/1:1H-136 [001] .... 61.036800: xs_tcp_data_recv: RPC: reading CALL/REPLY flag (4 bytes) kworker/1:1H-136 [001] .... 61.036801: xs_tcp_data_recv: RPC: read reply XID 2f4c3f88 kworker/1:1H-136 [001] ..s. 61.036801: xs_tcp_data_recv: RPC: XID 2f4c3f88 read 16 bytes kworker/1:1H-136 [001] ..s. 61.036802: xs_tcp_data_recv: RPC: xprt = ffff880407eca800, tcp_copied = 24, tcp_offset = 24, tcp_reclen = 24 kworker/1:1H-136 [001] ..s. 61.036802: xprt_complete_rqst: RPC: 43 xid 2f4c3f88 complete (24 bytes received) kworker/1:1H-136 [001] .... 61.036803: xs_tcp_data_recv: RPC: xs_tcp_data_recv done kworker/1:1H-136 [001] .... 61.036812: xprt_release: RPC: 43 release request ffff88040b270800 # unhide-tcp Unhide-tcp 20130526 Copyright © 2013 Yago Jesus & Patrick Gouin License GPLv3+ : GNU GPL version 3 or later http://www.unhide-forensics.info Used options: [*]Starting TCP checking Found Hidden port that not appears in ss: 805 -- Steve [-- Attachment #2: debug-hidden-port-4.7.patch --] [-- Type: text/x-patch, Size: 2378 bytes --] --- net/ipv4/inet_connection_sock.c | 4 ++++ net/ipv4/inet_hashtables.c | 5 +++++ net/sunrpc/xprt.c | 5 +++++ net/sunrpc/xprtsock.c | 5 +++++ 5 files changed, 22 insertions(+) Index: linux-build.git/net/ipv4/inet_connection_sock.c =================================================================== --- linux-build.git.orig/net/ipv4/inet_connection_sock.c 2016-06-22 11:55:05.952267493 -0400 +++ linux-build.git/net/ipv4/inet_connection_sock.c 2016-06-22 11:56:20.002662092 -0400 @@ -232,6 +232,10 @@ tb_found: } } success: + if (!current->mm) { + trace_printk("snum %d\n", snum); + trace_dump_stack(1); + } if (!inet_csk(sk)->icsk_bind_hash) inet_bind_hash(sk, tb, port); WARN_ON(inet_csk(sk)->icsk_bind_hash != tb); Index: linux-build.git/net/ipv4/inet_hashtables.c =================================================================== --- linux-build.git.orig/net/ipv4/inet_hashtables.c 2016-06-22 11:55:05.952267493 -0400 +++ linux-build.git/net/ipv4/inet_hashtables.c 2016-06-22 11:55:05.948267360 -0400 @@ -93,6 +93,11 @@ void inet_bind_bucket_destroy(struct kme void inet_bind_hash(struct sock *sk, struct inet_bind_bucket *tb, const unsigned short snum) { + if (!current->mm) { + trace_printk("add %d\n", snum); + trace_dump_stack(1); + } + inet_sk(sk)->inet_num = snum; sk_add_bind_node(sk, &tb->owners); tb->num_owners++; Index: linux-build.git/net/sunrpc/xprt.c =================================================================== --- linux-build.git.orig/net/sunrpc/xprt.c 2016-06-22 11:55:05.952267493 -0400 +++ linux-build.git/net/sunrpc/xprt.c 2016-06-22 11:55:05.948267360 -0400 @@ -54,6 +54,11 @@ #include "sunrpc.h" +#undef dprintk +#undef dprintk_rcu +#define dprintk(args...) trace_printk(args) +#define dprintk_rcu(args...) trace_printk(args) + /* * Local variables */ Index: linux-build.git/net/sunrpc/xprtsock.c =================================================================== --- linux-build.git.orig/net/sunrpc/xprtsock.c 2016-06-22 11:55:05.952267493 -0400 +++ linux-build.git/net/sunrpc/xprtsock.c 2016-06-22 11:55:05.948267360 -0400 @@ -51,6 +51,11 @@ #include "sunrpc.h" +#undef dprintk +#undef dprintk_rcu +#define dprintk(args...) trace_printk(args) +#define dprintk_rcu(args...) trace_printk(args) + static void xs_close(struct rpc_xprt *xprt); /* ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2018-02-06 19:26 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-02-02 21:31 It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) Daniel Reichelt 2018-02-06 0:24 ` Trond Myklebust 2018-02-06 9:20 ` Daniel Reichelt 2018-02-06 19:26 ` Trond Myklebust -- strict thread matches above, loose matches on Subject: below -- 2016-06-30 12:59 Steven Rostedt 2016-06-30 12:59 ` Steven Rostedt 2016-06-30 13:17 ` Trond Myklebust 2016-06-30 15:23 ` Steven Rostedt 2016-06-30 16:24 ` Steven Rostedt 2016-06-30 18:30 ` Trond Myklebust 2016-06-30 18:30 ` Trond Myklebust 2016-06-30 20:07 ` Steven Rostedt 2016-06-30 20:07 ` Steven Rostedt 2016-06-30 21:56 ` Steven Rostedt 2015-06-12 3:49 [REGRESSION] NFS is creating a hidden port (left over from xs_bind() ) Steven Rostedt 2015-06-12 14:10 ` Trond Myklebust 2015-06-12 14:40 ` Eric Dumazet 2015-06-12 15:34 ` Steven Rostedt 2015-06-12 15:50 ` Steven Rostedt 2015-06-18 3:08 ` Steven Rostedt 2015-06-18 19:24 ` Trond Myklebust 2015-06-18 19:49 ` Steven Rostedt 2015-06-18 22:50 ` Jeff Layton 2015-06-19 1:08 ` Steven Rostedt 2015-06-19 1:37 ` Jeff Layton 2015-06-19 16:25 ` Steven Rostedt 2015-06-19 17:17 ` Steven Rostedt 2015-06-19 17:39 ` Trond Myklebust 2015-06-19 19:52 ` Jeff Layton 2015-06-19 20:30 ` Trond Myklebust 2015-06-19 22:14 ` Steven Rostedt 2015-06-19 23:25 ` Trond Myklebust 2015-06-20 1:27 ` Steven Rostedt 2016-06-22 16:41 ` It's back! (Re: [REGRESSION] NFS is creating a hidden port (left over from xs_bind() )) Steven Rostedt
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.