* nfs stalls over loopback interface (no sk_data_ready events?)
@ 2014-10-27 19:26 Jeff Layton
0 siblings, 0 replies; 4+ messages in thread
From: Jeff Layton @ 2014-10-27 19:26 UTC (permalink / raw)
To: netdev-DgEjT+Ai2ygdnm+yROfE0A
Cc: Christoph Hellwig, Linux NFS Mailing List, Bruce Fields, Trond Myklebust
Sending this to netdev since I think I've now determined that this is
not a NFS specific problem. Recently Christoph mentioned that he was
seeing stalls when running xfstests generic/075 test on NFS over the
loopback interface with v3.18-rc1-ish kernel.
The configuration in this case is the nfs server and client on same box
communicating over the lo interface.
Here's are tracepoints from a typical request as it's supposed to work:
mount.nfs-906 [002] ...1 22711.996969: xprt_transmit: xprt=0xffff8800ce961000 xid=0xa8a34513 status=0
nfsd-678 [000] ...1 22711.997082: svc_recv: rq_xid=0xa8a34513 status=164
nfsd-678 [000] ..s8 22711.997185: xprt_lookup_rqst: xprt=0xffff8800ce961000 xid=0xa8a34513 status=0
nfsd-678 [000] ..s8 22711.997186: xprt_complete_rqst: xprt=0xffff8800ce961000 xid=0xa8a34513 status=140
nfsd-678 [000] ...1 22711.997236: svc_send: rq_xid=0xa8a34513 dropme=0 status=144
nfsd-678 [000] ...1 22711.997236: svc_process: rq_xid=0xa8a34513 dropme=0 status=144
...basically, we send a request to the server. Server picks it up and
sends the reply, and then the client IDs that reply and processes it.
This runs along just fine for ~ a minute or so. At some point, the
client stops seeing replies come in:
kworker/2:2-107 [002] ...1 22741.696070: xprt_transmit: xprt=0xffff8800ce961000 xid=0xc3a84513 status=0
nfsd-678 [002] .N.1 22741.696917: svc_recv: rq_xid=0xc3a84513 status=208
nfsd-678 [002] ...1 22741.699890: svc_send: rq_xid=0xc3a84513 dropme=0 status=262252
nfsd-678 [002] ...1 22741.699891: svc_process: rq_xid=0xc3a84513 dropme=0 status=262252
...a bit more tracepoint work seems to show that we just stop getting
sk_data_ready callbacks on the socket at all. I'm not terribly familiar
with the lower-level socket code, so I figured I'd email here and ask...
Anyone have insight into why this might be happening?
Thanks,
--
Jeff Layton <jlayton-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: nfs stalls over loopback interface (no sk_data_ready events?)
[not found] ` <20141029102123.58f6c960-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org>
@ 2014-10-30 19:00 ` Christoph Hellwig
0 siblings, 0 replies; 4+ messages in thread
From: Christoph Hellwig @ 2014-10-30 19:00 UTC (permalink / raw)
To: Jeff Layton
Cc: netdev-u79uwXL29TY76Z2rM5mHXA, Christoph Hellwig,
Linux NFS Mailing List, Bruce Fields, Trond Myklebust
On Wed, Oct 29, 2014 at 10:21:23AM -0400, Jeff Layton wrote:
> Looks some change that went into -rc2 has fixed the problem for me.
> Christoph, can you confirm that this no longer occurs with -rc2?
I can't reproduce it anymore on -rc2.
However:
generic/133 fails fairly reliably with:
Buffered writer, buffered reader
+pread64: Device or resource busy
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: nfs stalls over loopback interface (no sk_data_ready events?)
[not found] ` <20141027152900.4e81a9d5-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org>
@ 2014-10-29 14:21 ` Jeff Layton
[not found] ` <20141029102123.58f6c960-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org>
0 siblings, 1 reply; 4+ messages in thread
From: Jeff Layton @ 2014-10-29 14:21 UTC (permalink / raw)
To: netdev-u79uwXL29TY76Z2rM5mHXA
Cc: Christoph Hellwig, Linux NFS Mailing List, Bruce Fields, Trond Myklebust
On Mon, 27 Oct 2014 15:29:00 -0400
Jeff Layton <jlayton-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org> wrote:
> (sorry for resend -- I got the netdev address wrong)
>
> Sending this to netdev since I think I've now determined that this is
> not a NFS specific problem. Recently Christoph mentioned that he was
> seeing stalls when running xfstests generic/075 test on NFS over the
> loopback interface with v3.18-rc1-ish kernel.
>
> The configuration in this case is the nfs server and client on same box
> communicating over the lo interface.
>
> Here's are tracepoints from a typical request as it's supposed to work:
>
> mount.nfs-906 [002] ...1 22711.996969: xprt_transmit: xprt=0xffff8800ce961000 xid=0xa8a34513 status=0
> nfsd-678 [000] ...1 22711.997082: svc_recv: rq_xid=0xa8a34513 status=164
> nfsd-678 [000] ..s8 22711.997185: xprt_lookup_rqst: xprt=0xffff8800ce961000 xid=0xa8a34513 status=0
> nfsd-678 [000] ..s8 22711.997186: xprt_complete_rqst: xprt=0xffff8800ce961000 xid=0xa8a34513 status=140
> nfsd-678 [000] ...1 22711.997236: svc_send: rq_xid=0xa8a34513 dropme=0 status=144
> nfsd-678 [000] ...1 22711.997236: svc_process: rq_xid=0xa8a34513 dropme=0 status=144
>
> ...basically, we send a request to the server. Server picks it up and
> sends the reply, and then the client IDs that reply and processes it.
> This runs along just fine for ~ a minute or so. At some point, the
> client stops seeing replies come in:
>
> kworker/2:2-107 [002] ...1 22741.696070: xprt_transmit: xprt=0xffff8800ce961000 xid=0xc3a84513 status=0
> nfsd-678 [002] .N.1 22741.696917: svc_recv: rq_xid=0xc3a84513 status=208
> nfsd-678 [002] ...1 22741.699890: svc_send: rq_xid=0xc3a84513 dropme=0 status=262252
> nfsd-678 [002] ...1 22741.699891: svc_process: rq_xid=0xc3a84513 dropme=0 status=262252
>
>
> ...a bit more tracepoint work seems to show that we just stop getting
> sk_data_ready callbacks on the socket at all. I'm not terribly familiar
> with the lower-level socket code, so I figured I'd email here and ask...
>
> Anyone have insight into why this might be happening?
>
Looks some change that went into -rc2 has fixed the problem for me.
Christoph, can you confirm that this no longer occurs with -rc2?
--
Jeff Layton <jlayton-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* nfs stalls over loopback interface (no sk_data_ready events?)
@ 2014-10-27 19:29 Jeff Layton
[not found] ` <20141027152900.4e81a9d5-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org>
0 siblings, 1 reply; 4+ messages in thread
From: Jeff Layton @ 2014-10-27 19:29 UTC (permalink / raw)
To: netdev-u79uwXL29TY76Z2rM5mHXA
Cc: Christoph Hellwig, Linux NFS Mailing List, Bruce Fields, Trond Myklebust
(sorry for resend -- I got the netdev address wrong)
Sending this to netdev since I think I've now determined that this is
not a NFS specific problem. Recently Christoph mentioned that he was
seeing stalls when running xfstests generic/075 test on NFS over the
loopback interface with v3.18-rc1-ish kernel.
The configuration in this case is the nfs server and client on same box
communicating over the lo interface.
Here's are tracepoints from a typical request as it's supposed to work:
mount.nfs-906 [002] ...1 22711.996969: xprt_transmit: xprt=0xffff8800ce961000 xid=0xa8a34513 status=0
nfsd-678 [000] ...1 22711.997082: svc_recv: rq_xid=0xa8a34513 status=164
nfsd-678 [000] ..s8 22711.997185: xprt_lookup_rqst: xprt=0xffff8800ce961000 xid=0xa8a34513 status=0
nfsd-678 [000] ..s8 22711.997186: xprt_complete_rqst: xprt=0xffff8800ce961000 xid=0xa8a34513 status=140
nfsd-678 [000] ...1 22711.997236: svc_send: rq_xid=0xa8a34513 dropme=0 status=144
nfsd-678 [000] ...1 22711.997236: svc_process: rq_xid=0xa8a34513 dropme=0 status=144
...basically, we send a request to the server. Server picks it up and
sends the reply, and then the client IDs that reply and processes it.
This runs along just fine for ~ a minute or so. At some point, the
client stops seeing replies come in:
kworker/2:2-107 [002] ...1 22741.696070: xprt_transmit: xprt=0xffff8800ce961000 xid=0xc3a84513 status=0
nfsd-678 [002] .N.1 22741.696917: svc_recv: rq_xid=0xc3a84513 status=208
nfsd-678 [002] ...1 22741.699890: svc_send: rq_xid=0xc3a84513 dropme=0 status=262252
nfsd-678 [002] ...1 22741.699891: svc_process: rq_xid=0xc3a84513 dropme=0 status=262252
...a bit more tracepoint work seems to show that we just stop getting
sk_data_ready callbacks on the socket at all. I'm not terribly familiar
with the lower-level socket code, so I figured I'd email here and ask...
Anyone have insight into why this might be happening?
Thanks,
--
Jeff Layton <jlayton-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-10-30 19:00 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-10-27 19:26 nfs stalls over loopback interface (no sk_data_ready events?) Jeff Layton
2014-10-27 19:29 Jeff Layton
[not found] ` <20141027152900.4e81a9d5-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org>
2014-10-29 14:21 ` Jeff Layton
[not found] ` <20141029102123.58f6c960-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org>
2014-10-30 19:00 ` Christoph Hellwig
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).