* apparent scaling problem with delegations
@ 2024-02-08 20:26 Charles Hedrick
2024-02-08 21:10 ` Chuck Lever III
0 siblings, 1 reply; 6+ messages in thread
From: Charles Hedrick @ 2024-02-08 20:26 UTC (permalink / raw)
To: linux-nfs
We just turned delegations on for two big NFS servers. One characteristic of our site is that we have lots of small files and lots of files open.
On one server, CPU in system state went to 30%, and NFS performance ground to a halt. When I disabled delegations it came back. The other server was showing high CPU on nfsd, but not enough to disable the server, so I looked around. The server where delegations are still on is spending most of its time in nfsd_file_lru_cb. That's not the case with the server where we've disabled delegations. Here's a typical perf top
Overhead Shared Object Symbol
44.87% [kernel] [k] __list_lru_walk_one
13.18% [kernel] [k] native_queued_spin_lock_slowpath.part.0
7.24% [kernel] [k] nfsd_file_lru_cb
2.61% [kernel] [k] sha1_transform
0.99% [kernel] [k] __crypto_alg_lookup
0.95% [kernel] [k] _raw_spin_lock
0.89% [kernel] [k] memcpy_erms
0.77% [kernel] [k] mutex_lock
0.65% [kernel] [k] svc_tcp_recvfrom
I looked at the code. I'm not clear whether there's a problem with GC'ing the entries, or it's just being called too often (maybe a table is too small?)
When I disabled delegations, it immediately stopped spending all that time in nfsd_file_lru_cb. The number of delegations starting going down slowly. I suspect our system needs a lot more delegations than the maximum table size, and it's thrashing. The sizes were about 40,000 and
60,000 on the two machines. Systems are 384 G and 768 G, respectively. The maximum number of delegations is smaller than I would have expected based on comments in the code.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: apparent scaling problem with delegations
2024-02-08 20:26 apparent scaling problem with delegations Charles Hedrick
@ 2024-02-08 21:10 ` Chuck Lever III
2024-02-08 21:45 ` Charles Hedrick
0 siblings, 1 reply; 6+ messages in thread
From: Chuck Lever III @ 2024-02-08 21:10 UTC (permalink / raw)
To: Charles Hedrick; +Cc: Linux NFS Mailing List
> On Feb 8, 2024, at 3:26 PM, Charles Hedrick <hedrick@rutgers.edu> wrote:
>
> We just turned delegations on for two big NFS servers. One characteristic of our site is that we have lots of small files and lots of files open.
>
> On one server, CPU in system state went to 30%, and NFS performance ground to a halt. When I disabled delegations it came back. The other server was showing high CPU on nfsd, but not enough to disable the server, so I looked around. The server where delegations are still on is spending most of its time in nfsd_file_lru_cb. That's not the case with the server where we've disabled delegations. Here's a typical perf top
>
> Overhead Shared Object Symbol
> 44.87% [kernel] [k] __list_lru_walk_one
> 13.18% [kernel] [k] native_queued_spin_lock_slowpath.part.0
> 7.24% [kernel] [k] nfsd_file_lru_cb
> 2.61% [kernel] [k] sha1_transform
> 0.99% [kernel] [k] __crypto_alg_lookup
> 0.95% [kernel] [k] _raw_spin_lock
> 0.89% [kernel] [k] memcpy_erms
> 0.77% [kernel] [k] mutex_lock
> 0.65% [kernel] [k] svc_tcp_recvfrom
>
> I looked at the code. I'm not clear whether there's a problem with GC'ing the entries, or it's just being called too often (maybe a table is too small?)
>
> When I disabled delegations, it immediately stopped spending all that time in nfsd_file_lru_cb. The number of delegations starting going down slowly. I suspect our system needs a lot more delegations than the maximum table size, and it's thrashing. The sizes were about 40,000 and
> 60,000 on the two machines. Systems are 384 G and 768 G, respectively. The maximum number of delegations is smaller than I would have expected based on comments in the code.
When reporting such problems, please include the kernel version
on your NFS servers. Some late 5.x kernels have known problems
with the NFSD file cache.
--
Chuck Lever
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: apparent scaling problem with delegations
2024-02-08 21:10 ` Chuck Lever III
@ 2024-02-08 21:45 ` Charles Hedrick
2024-02-08 22:06 ` Chuck Lever III
0 siblings, 1 reply; 6+ messages in thread
From: Charles Hedrick @ 2024-02-08 21:45 UTC (permalink / raw)
To: Chuck Lever III; +Cc: Linux NFS Mailing List
>From: Chuck Lever III <chuck.lever@oracle.com>
>> On Feb 8, 2024, at 3:26 PM, Charles Hedrick <hedrick@rutgers.edu> wrote:
>>
>> We just turned delegations on for two big NFS servers. One characteristic
>> of our site is that we have lots of small files and lots of files open.
>>
>> On one server, CPU in system state went to 30%, and NFS performance ground
>> to a halt. When I disabled delegations it came back. The other server was
>> showing high CPU on nfsd, but not enough to disable the server, so I looked
>> around. The server where delegations are still on is spending most of its time
>> in nfsd_file_lru_cb. That's not the case with the server where we've disabled
>> delegations. Here's a typical perf top
>>
>> Overhead Shared Object Symbol
>> 44.87% [kernel] [k] __list_lru_walk_one
>> 13.18% [kernel] [k] native_queued_spin_lock_slowpath.part.0
>> 7.24% [kernel] [k] nfsd_file_lru_cb
>> 2.61% [kernel] [k] sha1_transform
>> 0.99% [kernel] [k] __crypto_alg_lookup
>> 0.95% [kernel] [k] _raw_spin_lock
>> 0.89% [kernel] [k] memcpy_erms
>> 0.77% [kernel] [k] mutex_lock
>> 0.65% [kernel] [k] svc_tcp_recvfrom
>>
>> I looked at the code. I'm not clear whether there's a problem with GC'ing the
>>entries, or it's just being called too often (maybe a table is too small?)
>>
>> When I disabled delegations, it immediately stopped spending all that time
>> in nfsd_file_lru_cb. The number of delegations starting going down slowly.
>> I suspect our system needs a lot more delegations than the maximum table
>> size, and it's thrashing. The sizes were about 40,000 and
>> 60,000 on the two machines. Systems are 384 G and 768 G, respectively.
>> The maximum number of delegations is smaller than I would have expected
>> based on comments in the code.
>When reporting such problems, please include the kernel version
>on your NFS servers. Some late 5.x kernels have known problems
> with the NFSD file cache.
My apologies.Ubuntu 5.15.0-91-generic , which is 5.15.131.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: apparent scaling problem with delegations
2024-02-08 21:45 ` Charles Hedrick
@ 2024-02-08 22:06 ` Chuck Lever III
2024-02-16 23:31 ` Dan Shelton
0 siblings, 1 reply; 6+ messages in thread
From: Chuck Lever III @ 2024-02-08 22:06 UTC (permalink / raw)
To: Charles Hedrick; +Cc: Linux NFS Mailing List
> On Feb 8, 2024, at 4:45 PM, Charles Hedrick <hedrick@rutgers.edu> wrote:
>
>> From: Chuck Lever III <chuck.lever@oracle.com>
>>> On Feb 8, 2024, at 3:26 PM, Charles Hedrick <hedrick@rutgers.edu> wrote:
>>>
>>> We just turned delegations on for two big NFS servers. One characteristic
>>> of our site is that we have lots of small files and lots of files open.
>>>
>>> On one server, CPU in system state went to 30%, and NFS performance ground
>>> to a halt. When I disabled delegations it came back. The other server was
>>> showing high CPU on nfsd, but not enough to disable the server, so I looked
>>> around. The server where delegations are still on is spending most of its time
>>> in nfsd_file_lru_cb. That's not the case with the server where we've disabled
>>> delegations. Here's a typical perf top
>>>
>>> Overhead Shared Object Symbol
>>> 44.87% [kernel] [k] __list_lru_walk_one
>>> 13.18% [kernel] [k] native_queued_spin_lock_slowpath.part.0
>>> 7.24% [kernel] [k] nfsd_file_lru_cb
>>> 2.61% [kernel] [k] sha1_transform
>>> 0.99% [kernel] [k] __crypto_alg_lookup
>>> 0.95% [kernel] [k] _raw_spin_lock
>>> 0.89% [kernel] [k] memcpy_erms
>>> 0.77% [kernel] [k] mutex_lock
>>> 0.65% [kernel] [k] svc_tcp_recvfrom
>>>
>>> I looked at the code. I'm not clear whether there's a problem with GC'ing the
>>> entries, or it's just being called too often (maybe a table is too small?)
>>>
>>> When I disabled delegations, it immediately stopped spending all that time
>>> in nfsd_file_lru_cb. The number of delegations starting going down slowly.
>>> I suspect our system needs a lot more delegations than the maximum table
>>> size, and it's thrashing. The sizes were about 40,000 and
>>> 60,000 on the two machines. Systems are 384 G and 768 G, respectively.
>>> The maximum number of delegations is smaller than I would have expected
>>> based on comments in the code.
>
>> When reporting such problems, please include the kernel version
>> on your NFS servers. Some late 5.x kernels have known problems
>> with the NFSD file cache.
>
> My apologies.Ubuntu 5.15.0-91-generic , which is 5.15.131.
That kernel is likely to have file cache issues with symptoms
very much as you described above. The issues are thought to
be addressed by kernel release v6.2.
I can't suggest a specific patch, as there were a long series
of fixes that went upstream to address the file cache problems.
Best I can suggest is for you to switch to an Ubuntu kernel
based on a more recent kernel release (6.2.y or later).
--
Chuck Lever
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: apparent scaling problem with delegations
2024-02-08 22:06 ` Chuck Lever III
@ 2024-02-16 23:31 ` Dan Shelton
2024-02-17 1:25 ` Chuck Lever
0 siblings, 1 reply; 6+ messages in thread
From: Dan Shelton @ 2024-02-16 23:31 UTC (permalink / raw)
To: Linux NFS Mailing List
On Thu, 8 Feb 2024 at 23:06, Chuck Lever III <chuck.lever@oracle.com> wrote:
>
>
>
> > On Feb 8, 2024, at 4:45 PM, Charles Hedrick <hedrick@rutgers.edu> wrote:
> >
> >> From: Chuck Lever III <chuck.lever@oracle.com>
> >>> On Feb 8, 2024, at 3:26 PM, Charles Hedrick <hedrick@rutgers.edu> wrote:
> >>>
> >>> We just turned delegations on for two big NFS servers. One characteristic
> >>> of our site is that we have lots of small files and lots of files open.
> >>>
> >>> On one server, CPU in system state went to 30%, and NFS performance ground
> >>> to a halt. When I disabled delegations it came back. The other server was
> >>> showing high CPU on nfsd, but not enough to disable the server, so I looked
> >>> around. The server where delegations are still on is spending most of its time
> >>> in nfsd_file_lru_cb. That's not the case with the server where we've disabled
> >>> delegations. Here's a typical perf top
> >>>
> >>> Overhead Shared Object Symbol
> >>> 44.87% [kernel] [k] __list_lru_walk_one
> >>> 13.18% [kernel] [k] native_queued_spin_lock_slowpath.part.0
> >>> 7.24% [kernel] [k] nfsd_file_lru_cb
> >>> 2.61% [kernel] [k] sha1_transform
> >>> 0.99% [kernel] [k] __crypto_alg_lookup
> >>> 0.95% [kernel] [k] _raw_spin_lock
> >>> 0.89% [kernel] [k] memcpy_erms
> >>> 0.77% [kernel] [k] mutex_lock
> >>> 0.65% [kernel] [k] svc_tcp_recvfrom
> >>>
> >>> I looked at the code. I'm not clear whether there's a problem with GC'ing the
> >>> entries, or it's just being called too often (maybe a table is too small?)
> >>>
> >>> When I disabled delegations, it immediately stopped spending all that time
> >>> in nfsd_file_lru_cb. The number of delegations starting going down slowly.
> >>> I suspect our system needs a lot more delegations than the maximum table
> >>> size, and it's thrashing. The sizes were about 40,000 and
> >>> 60,000 on the two machines. Systems are 384 G and 768 G, respectively.
> >>> The maximum number of delegations is smaller than I would have expected
> >>> based on comments in the code.
> >
> >> When reporting such problems, please include the kernel version
> >> on your NFS servers. Some late 5.x kernels have known problems
> >> with the NFSD file cache.
> >
> > My apologies.Ubuntu 5.15.0-91-generic , which is 5.15.131.
>
> That kernel is likely to have file cache issues with symptoms
> very much as you described above. The issues are thought to
> be addressed by kernel release v6.2.
Is there a way to turn the file cache off for nfsd?
Dan
--
Dan Shelton - Cluster Specialist Win/Lin/Bsd
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: apparent scaling problem with delegations
2024-02-16 23:31 ` Dan Shelton
@ 2024-02-17 1:25 ` Chuck Lever
0 siblings, 0 replies; 6+ messages in thread
From: Chuck Lever @ 2024-02-17 1:25 UTC (permalink / raw)
To: Dan Shelton; +Cc: Linux NFS Mailing List
On Sat, Feb 17, 2024 at 12:31:34AM +0100, Dan Shelton wrote:
> On Thu, 8 Feb 2024 at 23:06, Chuck Lever III <chuck.lever@oracle.com> wrote:
> >
> >
> >
> > > On Feb 8, 2024, at 4:45 PM, Charles Hedrick <hedrick@rutgers.edu> wrote:
> > >
> > >> From: Chuck Lever III <chuck.lever@oracle.com>
> > >>> On Feb 8, 2024, at 3:26 PM, Charles Hedrick <hedrick@rutgers.edu> wrote:
> > >>>
> > >>> We just turned delegations on for two big NFS servers. One characteristic
> > >>> of our site is that we have lots of small files and lots of files open.
> > >>>
> > >>> On one server, CPU in system state went to 30%, and NFS performance ground
> > >>> to a halt. When I disabled delegations it came back. The other server was
> > >>> showing high CPU on nfsd, but not enough to disable the server, so I looked
> > >>> around. The server where delegations are still on is spending most of its time
> > >>> in nfsd_file_lru_cb. That's not the case with the server where we've disabled
> > >>> delegations. Here's a typical perf top
> > >>>
> > >>> Overhead Shared Object Symbol
> > >>> 44.87% [kernel] [k] __list_lru_walk_one
> > >>> 13.18% [kernel] [k] native_queued_spin_lock_slowpath.part.0
> > >>> 7.24% [kernel] [k] nfsd_file_lru_cb
> > >>> 2.61% [kernel] [k] sha1_transform
> > >>> 0.99% [kernel] [k] __crypto_alg_lookup
> > >>> 0.95% [kernel] [k] _raw_spin_lock
> > >>> 0.89% [kernel] [k] memcpy_erms
> > >>> 0.77% [kernel] [k] mutex_lock
> > >>> 0.65% [kernel] [k] svc_tcp_recvfrom
> > >>>
> > >>> I looked at the code. I'm not clear whether there's a problem with GC'ing the
> > >>> entries, or it's just being called too often (maybe a table is too small?)
> > >>>
> > >>> When I disabled delegations, it immediately stopped spending all that time
> > >>> in nfsd_file_lru_cb. The number of delegations starting going down slowly.
> > >>> I suspect our system needs a lot more delegations than the maximum table
> > >>> size, and it's thrashing. The sizes were about 40,000 and
> > >>> 60,000 on the two machines. Systems are 384 G and 768 G, respectively.
> > >>> The maximum number of delegations is smaller than I would have expected
> > >>> based on comments in the code.
> > >
> > >> When reporting such problems, please include the kernel version
> > >> on your NFS servers. Some late 5.x kernels have known problems
> > >> with the NFSD file cache.
> > >
> > > My apologies.Ubuntu 5.15.0-91-generic , which is 5.15.131.
> >
> > That kernel is likely to have file cache issues with symptoms
> > very much as you described above. The issues are thought to
> > be addressed by kernel release v6.2.
>
> Is there a way to turn the file cache off for nfsd?
It is integrated into the operation of NFSD, so it cannot be
disabled.
--
Chuck Lever
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-02-17 1:25 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-08 20:26 apparent scaling problem with delegations Charles Hedrick
2024-02-08 21:10 ` Chuck Lever III
2024-02-08 21:45 ` Charles Hedrick
2024-02-08 22:06 ` Chuck Lever III
2024-02-16 23:31 ` Dan Shelton
2024-02-17 1:25 ` Chuck Lever
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.