From: dai.ngo@oracle.com
To: Bruce Fields <bfields@fieldses.org>
Cc: Chuck Lever III <chuck.lever@oracle.com>,
Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
Subject: Re: [PATCH RFC v5 0/2] nfsd: Initial implementation of NFSv4 Courteous Server
Date: Tue, 30 Nov 2021 19:50:13 -0800 [thread overview]
Message-ID: <f6a948a7-32d6-da9a-6808-9f2f77d5f792@oracle.com> (raw)
In-Reply-To: <20211130153211.GB8837@fieldses.org>
On 11/30/21 7:32 AM, Bruce Fields wrote:
> On Mon, Nov 29, 2021 at 11:13:34PM -0800, dai.ngo@oracle.com wrote:
>> Just to be clear, the problem of pynfs with 4.0 is that the server takes
>> ~55 secs to expire 1026 4.0 courteous clients, which comes out to ~50ms
>> per client. This causes the test to time out in waiting for RPC reply of
>> the OPEN that triggers the conflicts.
>>
>> I don't know exactly where the time spent in the process of expiring a
>> client. But as Bruce mentioned, it could be related to the time to access
>> /var/lib/nfs to remove the client's persistent record.
> Could you try something like
>
> strace -r -$(pidof) -oTRACE
Strace does not have any info that shows where the server spent time when
expiring client state. The client record is removed by nfsd4_umh_cltrack_remove
doing upcall to user space helper /sbin/nfsdcltrack to do the job. I used
the low-tech debug tool, printk, to measure the time spent by
nfsd4_client_record_remove. Here is a sample of the output, START and END
are in milliseconds:
Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: START [0x15d418] clp[ffff888119206040] client_tracking_ops[ffffffffa04bc2e0]
Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: END [0x15d459] clp[ffff888119206040] client_tracking_ops[ffffffffa04bc2e0]
Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: START [0x15d461] clp[ffff888119206740] client_tracking_ops[ffffffffa04bc2e0]
Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: END [0x15d48e] clp[ffff888119206740] client_tracking_ops[ffffffffa04bc2e0]
Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: START [0x15d49c] clp[ffff88811b54e000] client_tracking_ops[ffffffffa04bc2e0]
Nov 30 12:31:04 localhost kernel: nfsd4_client_record_remove: END [0x15d4c5] clp[ffff88811b54e000] client_tracking_ops[ffffffffa04bc2e0]
The average time spent to remove the client record is about ~50ms, matches
with the time reported by pynfs test. This confirms what Bruce suspected
earlier.
-Dai
>
> and maybe we could take a look at TRACE? My hope would be that there'd
> be a clear set of syscalls whose time, multiplied by 1026, explains most
> of that 55 seconds. Then it might be worth checking whether there are
> any easy optimizations possible.
>
> --b.
next prev parent reply other threads:[~2021-12-01 3:50 UTC|newest]
Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-29 0:56 [PATCH RFC v5 0/2] nfsd: Initial implementation of NFSv4 Courteous Server Dai Ngo
2021-09-29 0:56 ` [PATCH RFC v5 1/2] fs/lock: add new callback, lm_expire_lock, to lock_manager_operations Dai Ngo
2021-09-29 0:56 ` [PATCH RFC v5 2/2] nfsd: Initial implementation of NFSv4 Courteous Server Dai Ngo
2021-10-01 20:53 ` [PATCH RFC v5 0/2] " J. Bruce Fields
2021-10-01 21:41 ` dai.ngo
2021-10-01 23:03 ` J. Bruce Fields
2021-11-16 23:06 ` dai.ngo
2021-11-17 14:14 ` J. Bruce Fields
2021-11-17 17:59 ` dai.ngo
2021-11-17 21:46 ` dai.ngo
2021-11-18 0:34 ` J. Bruce Fields
2021-11-22 3:04 ` dai.ngo
2021-11-29 17:13 ` dai.ngo
2021-11-29 17:30 ` J. Bruce Fields
2021-11-29 18:32 ` dai.ngo
2021-11-29 19:03 ` Chuck Lever III
2021-11-29 19:13 ` Bruce Fields
2021-11-29 19:39 ` dai.ngo
2021-11-29 19:36 ` dai.ngo
2021-11-29 21:01 ` dai.ngo
2021-11-29 21:10 ` Chuck Lever III
2021-11-30 0:11 ` dai.ngo
2021-11-30 1:42 ` Chuck Lever III
2021-11-30 4:08 ` Trond Myklebust
2021-11-30 4:47 ` Chuck Lever III
2021-11-30 4:57 ` Trond Myklebust
2021-11-30 7:22 ` dai.ngo
2021-11-30 13:37 ` Trond Myklebust
2021-12-01 3:52 ` dai.ngo
2021-12-01 14:19 ` bfields
2021-11-30 15:36 ` Chuck Lever III
2021-11-30 16:05 ` Bruce Fields
2021-11-30 16:14 ` Trond Myklebust
2021-11-30 19:01 ` bfields
2021-11-30 7:13 ` dai.ngo
2021-11-30 15:32 ` Bruce Fields
2021-12-01 3:50 ` dai.ngo [this message]
2021-12-01 14:36 ` Bruce Fields
2021-12-01 14:51 ` Bruce Fields
2021-12-01 18:47 ` dai.ngo
2021-12-01 19:25 ` Bruce Fields
2021-12-02 17:53 ` Chuck Lever III
2021-12-01 17:42 ` Bruce Fields
2021-12-01 18:03 ` Bruce Fields
2021-12-01 19:50 ` Bruce Fields
2021-12-03 21:22 ` Bruce Fields
2021-12-03 21:55 ` [PATCH] nfsdcld: use WAL journal for faster commits Bruce Fields
2021-12-03 22:07 ` Chuck Lever III
2021-12-03 22:39 ` Bruce Fields
2021-12-04 0:35 ` Chuck Lever III
2021-12-04 1:24 ` Bruce Fields
2021-12-06 15:46 ` Chuck Lever III
2022-01-04 22:24 ` Bruce Fields
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f6a948a7-32d6-da9a-6808-9f2f77d5f792@oracle.com \
--to=dai.ngo@oracle.com \
--cc=bfields@fieldses.org \
--cc=chuck.lever@oracle.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.