All of lore.kernel.org
 help / color / mirror / Atom feed
From: Bruce Fields <bfields@fieldses.org>
To: dai.ngo@oracle.com
Cc: Chuck Lever III <chuck.lever@oracle.com>,
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
	"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
Subject: Re: [PATCH RFC v5 0/2] nfsd: Initial implementation of NFSv4 Courteous Server
Date: Wed, 1 Dec 2021 09:36:30 -0500	[thread overview]
Message-ID: <20211201143630.GB24991@fieldses.org> (raw)
In-Reply-To: <f6a948a7-32d6-da9a-6808-9f2f77d5f792@oracle.com>

On Tue, Nov 30, 2021 at 07:50:13PM -0800, dai.ngo@oracle.com wrote:
> 
> On 11/30/21 7:32 AM, Bruce Fields wrote:
> >On Mon, Nov 29, 2021 at 11:13:34PM -0800, dai.ngo@oracle.com wrote:
> >>Just to be clear, the problem of pynfs with 4.0 is that the server takes
> >>~55 secs to expire 1026 4.0 courteous clients, which comes out to ~50ms
> >>per client. This causes the test to time out in waiting for RPC reply of
> >>the OPEN that triggers the conflicts.
> >>
> >>I don't know exactly where the time spent in the process of expiring a
> >>client. But as Bruce mentioned, it could be related to the time to access
> >>/var/lib/nfs to remove the client's persistent record.
> >Could you try something like
> >
> >	strace -r -$(pidof) -oTRACE

Oops, I mean $(pidof nfsdcld).  But, your system isn't using that:

> 
> Strace does not have any info that shows where the server spent time when
> expiring client state. The client record is removed by nfsd4_umh_cltrack_remove
> doing upcall to user space helper /sbin/nfsdcltrack to do the job.  I used
> the low-tech debug tool, printk, to measure the time spent by
> nfsd4_client_record_remove. Here is a sample of the output, START and END
> are in milliseconds:
> 
> Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: START [0x15d418] clp[ffff888119206040] client_tracking_ops[ffffffffa04bc2e0]
> Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: END [0x15d459] clp[ffff888119206040] client_tracking_ops[ffffffffa04bc2e0]
> Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: START [0x15d461] clp[ffff888119206740] client_tracking_ops[ffffffffa04bc2e0]
> Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: END [0x15d48e] clp[ffff888119206740] client_tracking_ops[ffffffffa04bc2e0]
> Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: START [0x15d49c] clp[ffff88811b54e000] client_tracking_ops[ffffffffa04bc2e0]
> Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: END [0x15d4c5] clp[ffff88811b54e000] client_tracking_ops[ffffffffa04bc2e0]
> 
> The average time spent to remove the client record is about ~50ms, matches
> with the time reported by pynfs test. This confirms what Bruce suspected
> earlier.

OK, good to know.  It'd be interesting to dig into where nfsdcltrack is
spending its time, which we could do by replacing it with a wrapper that
runs the real nfsdcltrack under strace.

Though maybe it'd be better to do this on a system using nfsdcld, since
that's what we're transitioning to.

--b.

  reply	other threads:[~2021-12-01 14:39 UTC|newest]

Thread overview: 53+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-29  0:56 [PATCH RFC v5 0/2] nfsd: Initial implementation of NFSv4 Courteous Server Dai Ngo
2021-09-29  0:56 ` [PATCH RFC v5 1/2] fs/lock: add new callback, lm_expire_lock, to lock_manager_operations Dai Ngo
2021-09-29  0:56 ` [PATCH RFC v5 2/2] nfsd: Initial implementation of NFSv4 Courteous Server Dai Ngo
2021-10-01 20:53 ` [PATCH RFC v5 0/2] " J. Bruce Fields
2021-10-01 21:41   ` dai.ngo
2021-10-01 23:03     ` J. Bruce Fields
2021-11-16 23:06     ` dai.ngo
2021-11-17 14:14       ` J. Bruce Fields
2021-11-17 17:59         ` dai.ngo
2021-11-17 21:46           ` dai.ngo
2021-11-18  0:34             ` J. Bruce Fields
2021-11-22  3:04               ` dai.ngo
2021-11-29 17:13                 ` dai.ngo
2021-11-29 17:30                   ` J. Bruce Fields
2021-11-29 18:32                     ` dai.ngo
2021-11-29 19:03                       ` Chuck Lever III
2021-11-29 19:13                         ` Bruce Fields
2021-11-29 19:39                           ` dai.ngo
2021-11-29 19:36                         ` dai.ngo
2021-11-29 21:01                           ` dai.ngo
2021-11-29 21:10                           ` Chuck Lever III
2021-11-30  0:11                             ` dai.ngo
2021-11-30  1:42                               ` Chuck Lever III
2021-11-30  4:08                                 ` Trond Myklebust
2021-11-30  4:47                                   ` Chuck Lever III
2021-11-30  4:57                                     ` Trond Myklebust
2021-11-30  7:22                                       ` dai.ngo
2021-11-30 13:37                                         ` Trond Myklebust
2021-12-01  3:52                                           ` dai.ngo
2021-12-01 14:19                                             ` bfields
2021-11-30 15:36                                         ` Chuck Lever III
2021-11-30 16:05                                           ` Bruce Fields
2021-11-30 16:14                                             ` Trond Myklebust
2021-11-30 19:01                                               ` bfields
2021-11-30  7:13                                 ` dai.ngo
2021-11-30 15:32                                   ` Bruce Fields
2021-12-01  3:50                                     ` dai.ngo
2021-12-01 14:36                                       ` Bruce Fields [this message]
2021-12-01 14:51                                         ` Bruce Fields
2021-12-01 18:47                                           ` dai.ngo
2021-12-01 19:25                                             ` Bruce Fields
2021-12-02 17:53                                           ` Chuck Lever III
2021-12-01 17:42                                         ` Bruce Fields
2021-12-01 18:03                                           ` Bruce Fields
2021-12-01 19:50                                             ` Bruce Fields
2021-12-03 21:22                                               ` Bruce Fields
2021-12-03 21:55                                                 ` [PATCH] nfsdcld: use WAL journal for faster commits Bruce Fields
2021-12-03 22:07                                                   ` Chuck Lever III
2021-12-03 22:39                                                     ` Bruce Fields
2021-12-04  0:35                                                       ` Chuck Lever III
2021-12-04  1:24                                                         ` Bruce Fields
2021-12-06 15:46                                                           ` Chuck Lever III
2022-01-04 22:24                                                             ` Bruce Fields

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20211201143630.GB24991@fieldses.org \
    --to=bfields@fieldses.org \
    --cc=chuck.lever@oracle.com \
    --cc=dai.ngo@oracle.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.