long-term stable backports of NFSD patches

long-term stable backports of NFSD patches

[Chuck Lever]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.

To address the filecache and other scalability problems in those
kernels, I'm preparing backported patches of NFSD fixes for several
popular LTS kernels. These backports are destined for the official
LTS kernel branches so that distributions can easily integrate them
into their products.

Once this effort is complete, Greg and Sasha will continue to be
responsible for backporting NFSD-related fixes from upstream into
the LTS kernels.

Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v5.15.y

v5.15.154 has been released, and it includes the NFSD filecache
backports. Oracle's own QA team found a small fcntl regression,
which should be addressed by v5.15.156. I am continuing to watch
for issues.

You can find these patches in the "nfsd-5.15.y" branch in the above
repo.


LTS v5.10.y

No change here this week. It needs to be rebased, and a few extra
commits are needed to match up with what's now in v5.15.154.

Next week is NFS bake-a-thon. I plan to restart work on nfsd-5.10.y
the following week.

You can find these patches in the "nfsd-5.10.y" branch in the above
repo.


-- 
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.

To address the filecache and other scalability problems in those
kernels, I'm preparing backported patches of NFSD fixes for several
popular LTS kernels. These backports are destined for the official
LTS kernel branches so that distributions can easily integrate them
into their products.

Once this effort is complete, Greg and Sasha will continue to be
responsible for backporting NFSD-related fixes from upstream into
the LTS kernels.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v5.10.y

I've found no new issues in nfsd-5.10.y, and have continued to
build out automated NFSD testing for this kernel and
origin/linux-5.10.y. I plan to send this patch series to Greg and
Sasha once the v6.10 merge window dust has settled.

You can find this series in the "nfsd-5.10.y" branch in the above
repo.


-- 
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever III]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.

To address the filecache and other scalability problems in those
kernels, I'm preparing backported patches of NFSD fixes for several
popular LTS kernels. These backports are destined for the official
LTS kernel branches so that distributions can easily integrate them
into their products.

Once this effort is complete, Greg and Sasha will continue to be
responsible for backporting NFSD-related fixes from upstream into
the LTS kernels.

Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v5.10.y

Testing has found two issues that were both due to patches I
applied that are not appropriate for this kernel. The patches have
been removed. You can find this series in the "nfsd-5.10.y" branch
in the above repo. Testing will continue this week.


--
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.

To address the filecache and other scalability problems in those
kernels, I'm preparing backported patches of NFSD fixes for several
popular LTS kernels. These backports are destined for the official
LTS kernel branches so that distributions can easily integrate them
into their products.

Once this effort is complete, Greg and Sasha will continue to be
responsible for backporting NFSD-related fixes from upstream into
the LTS kernels.

Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v5.15.y

Work on backporting the NFSD file cache fixes to v5.15.y is now
complete. Subsequent fixes and changes will go through the usual
-> stable@ process.


LTS v5.10.y

I've updated nfsd-5.10.y to include the patches and fixes that
were included in nfsd-5.15.y. You can find these patches in the
"nfsd-5.10.y" branch in the above repo.

This week I intend to set up CI testing for this branch.


-- 
Chuck Lever

Re: long-term stable backports of NFSD patches

[Amir Goldstein]

On Mon, May 6, 2024 at 4:25 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>
> It's apparent that a number of distributions and their customers
> remain on long-term stable kernels. We are aware of the scalability
> problems and other bugs in NFSD in kernels between v5.4 and v6.1.
>

Chuck,

Are you able to share a partial list of scalability problems that were
fixed by this backport series?

Specifically, my interest is in the list of improvements to 5.15.y.

Thanks,
Amir.

Re: long-term stable backports of NFSD patches

[Chuck Lever]

On Wed, May 08, 2024 at 02:29:05PM +0300, Amir Goldstein wrote:
> On Mon, May 6, 2024 at 4:25 PM Chuck Lever <chuck.lever@oracle.com> wrote:
> >
> > It's apparent that a number of distributions and their customers
> > remain on long-term stable kernels. We are aware of the scalability
> > problems and other bugs in NFSD in kernels between v5.4 and v6.1.
> >
> 
> Chuck,
> 
> Are you able to share a partial list of scalability problems that were
> fixed by this backport series?
> 
> Specifically, my interest is in the list of improvements to 5.15.y.

In broad strokes:

- The garbage collection mechanism was rewritten to keep the LRU
  list short, and to keep sweeps productive. This was the main issue
  as the LRU used to have hundreds of thousands of files on it and
  one sweep would take so much CPU it was reported as a soft lockup.
- NFSv4 OPEN files are no longer garbage collected so that an NFSv4
  CLOSE means local accessors have immediate access to a file.
- The filecache hash table is converted to an rhltable so that it
  can efficiently manage many more open files.
- Various fixes prevent writeback of garbage-collected files from
  bogging down.

There are a handful of important bugfixes before this commit, but
starting with commit 0369b53886ec ("NFSD: Report filecache LRU
size"), have a look at the commits that touch fs/nfsd/filecache.c.

-- 
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.

To address the filecache and other scalability problems in those
kernels, I'm preparing backported patches of NFSD fixes for several
popular LTS kernels. These backports are destined for the official
LTS kernel branches so that distributions can easily integrate them
into their products.

Once this effort is complete, Greg and Sasha will continue to be
responsible for backporting NFSD-related fixes from upstream into
the LTS kernels.

Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v5.15.y

There was one new NFSD-related regression reported this week:
"killall -9 nfsd" triggers an oops. There was one missing patch
in the backport which should now be queued for v5.15.158.

I am continuing to watch for issues.

You can find these patches in the "nfsd-5.15.y" branch in the above
repo. This branch has been rebased on v5.15.157.


LTS v5.10.y

No change here this week. I hope to restart work on nfsd-5.10.y
this week by backporting the handful of NFSD patches that are in
v5.15.y but not yet in v5.10.y.

You can find these patches in the "nfsd-5.10.y" branch in the above
repo.


-- 
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.

To address the filecache and other scalability problems in those
kernels, I'm preparing backported patches of NFSD fixes for several
popular LTS kernels. The backported commits are destined for the
official LTS kernel branches so that distributions can easily
integrate them into their products.

Once this effort is complete, Greg and Sasha will continue to be
responsible for backporting NFSD-related fixes from upstream into
the LTS kernels.

Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v5.15.y

Greg is currently working on releasing 5.15.154, which should
contain the full set of NFSD filecache backports.

You can find these patches in the "nfsd-5.15.y" branch in the above
repo.


LTS v5.10.y

This branch has been rebased on 5.10.214. No change here this week.
Once v5.15.y is complete, I will start work on this branch again in
earnest.

You can find these patches in the "nfsd-5.10.y" branch in the above
repo.


-- 
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.

To address the filecache and other scalability problems in those
kernels, I'm preparing backported patches of NFSD fixes for several
popular LTS kernels. The backported commits are destined for the
official LTS kernel branches so that distributions can easily
integrate them into their products.

Once this effort is complete, Greg and Sasha will continue to be
responsible for backporting NFSD-related fixes from upstream into
the LTS kernels.

Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v6.1.y

This part of the project is complete, and this branch has been
removed from my repo.


LTS v5.15.y

This branch has been rebased on 5.15.153 and resubmitted to the
stable maintainers. Once it is merged, there are several fanotify-
related user documentation changes that are needed.

You can find these patches in the "nfsd-5.15.y" branch in the above
repo.


LTS v5.10.y

This branch has been rebased on 5.10.214. I've done some reworking
on this one, but it is not yet complete. Once v5.15.y is complete,
I will start work on this branch again in earnest.

You can find these patches in the "nfsd-5.10.y" branch in the above
repo.


-- 
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever III]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.

To address the filecache and other scalability problems in those
kernels, I'm preparing backported patches of NFSD fixes for several
popular LTS kernels. The backported commits are destined for the
official LTS kernel branches so that distributions can easily
integrate them into their products.

Once this effort is complete, Greg and Sasha will continue to be
responsible for backporting NFSD-related fixes from upstream into
the LTS kernels.

Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v6.1.y

The v6.1.82 kernel has been updated. This part of the project
is complete.


LTS v5.15.y

A first attempt at merging these patches into v5.15.y was NAK'd
due to some SCSI changes made on behalf of patches to NFSD's pNFS
block layout implementation. The series has been reworked and
retested.

You can find these patches in the "nfsd-5.15.y" branch in the above
repo.


LTS v5.10.y

This branch will need the same reworking as nfsd-5.15.y. I will
start work on that once the v5.15.y work is complete.

You can find these patches in the "nfsd-5.10.y" branch in the above
repo.


--
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.

To address the filecache and other scalability problems in those
kernels, I'm preparing backported patches of NFSD fixes for several
popular LTS kernels. The backported commits are destined for the
official LTS kernel branches so that distributions can easily
integrate them into their products.

Once this effort is complete, Greg and Sasha will continue to be
responsible for backporting NFSD-related fixes from upstream into
the LTS kernels.

Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v6.1.y

Additional patches were tested and sent to Greg and Sasha this
week. Sasha has merged them into linux-6.1.y for release soon.

Meanwhile, you can find these patches in the "nfsd-6.1.y" branch in
the above repo.


LTS v5.15.y

This branch was rebased on linux-5.15.152 and has completed
testing. I'm investigating some failures in the ltp net.nfs group
but it looks like these are all related to NFSv4.2.

You can find these patches in the "nfsd-5.15.y" branch in the above
repo.


LTS v5.10.y

This branch was rebased on linux-5.10.213. Testing will commence
once nfsd-5.15.y has been sent to Greg and Sasha.

You can find these patches in the "nfsd-5.10.y" branch in the above
repo.

-- 
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.

To address the filecache and other scalability problems in those
kernels, I'm preparing backported patches of NFSD fixes for several
popular LTS kernels. The backported commits are destined for the
official LTS kernel branches so that distributions can easily
integrate them into their products.

Once this effort is complete, Greg and Sasha will continue to be
responsible for backporting NFSD-related fixes from upstream into
the LTS kernels.

Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v6.1.y

Patches that bring NFSD in linux-6.1.y up to v6.1.16 have been
merged into v6.1.81. I've identified a few additional filecache
related fixes that I will forward to Greg and Sasha soon.


LTS v5.15.y

I have updated this branch to include commits from v6.2.16 through
v5.16, all applied to v5.15.151. Testing has stopped while I work
on the v6.9 merge window.

You can find these patches in the "nfsd-5.15.y" branch in the above
repo.


LTS v5.10.y

I have updated this branch to include commits from v6.2.16 through
v5.11, all applied to v5.10.212. Testing has stopped while I work
on the v6.9 merge window.

You can find these patches in the "nfsd-5.10.y" branch in the above
repo.

-- 
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in those kernels.

Therefore I've started an effort to backport NFSD-related fixes to
long-term stable kernels.

I consulted with GregKH and Sasha to ask their preferences about how
this should be done. They said a full subsystem backport is
preferred. Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes.


LTS v6.1.y

I've backported NFSD patches from v6.2 to v6.1.79 and am just now
completing our usual CI testing on this. You can find these patches
in the "nfsd-6.1.y" branch in the above repo.


LTS v5.15.y

I've backported NFSD patches from v6.2 through v5.16 to v5.15.149.
I have not begun testing this work. You can find these patches in
the "nfsd-5.15.y" branch in the above repo.


LTS v5.10.y

I've started backporting NFSD patches from v6.2 through v5.11 to
v5.10.210. I've gotten up to v5.15 so far. You can find these
patches in the "nfsd-5.10.y" branch in the above repo.


LTS v5.4.y

I'm not certain whether to pursue a backport for v5.4.y, as
backporting from v6.latest becomes increasingly difficult on older
kernels, and the user base for these is dwindling.

-- 
Chuck Lever

Re: long-term stable backports of NFSD patches

[Dan Shelton]

On Mon, 26 Feb 2024 at 15:21, Chuck Lever <chuck.lever@oracle.com> wrote:
>
> It's apparent that a number of distributions and their customers
> remain on long-term stable kernels. We are aware of the scalability
> problems and other bugs in NFSD in those kernels.
>
> Therefore I've started an effort to backport NFSD-related fixes to
> long-term stable kernels.
>
> I consulted with GregKH and Sasha to ask their preferences about how
> this should be done. They said a full subsystem backport is
> preferred. Here's a status update.
>
> ---
>
> I've pushed the NFSD backports to branches in this repo:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git
>
> If you are able, I encourage you to pull these, review them or try
> them out, and report any issues or successes.
>
>
> LTS v6.1.y

Where is LTS v6.6.y?

Dan
-- 
Dan Shelton - Cluster Specialist Win/Lin/Bsd

Re: long-term stable backports of NFSD patches

[Chuck Lever]

On Tue, Feb 27, 2024 at 06:57:43AM +0100, Dan Shelton wrote:
> On Mon, 26 Feb 2024 at 15:21, Chuck Lever <chuck.lever@oracle.com> wrote:
> >
> > It's apparent that a number of distributions and their customers
> > remain on long-term stable kernels. We are aware of the scalability
> > problems and other bugs in NFSD in those kernels.
> >
> > Therefore I've started an effort to backport NFSD-related fixes to
> > long-term stable kernels.
> >
> > I consulted with GregKH and Sasha to ask their preferences about how
> > this should be done. They said a full subsystem backport is
> > preferred. Here's a status update.
> >
> > ---
> >
> > I've pushed the NFSD backports to branches in this repo:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git
> >
> > If you are able, I encourage you to pull these, review them or try
> > them out, and report any issues or successes.
> >
> >
> > LTS v6.1.y
> 
> Where is LTS v6.6.y?

The purpose of this effort is to address NFSD filecache issues
present until v6.2.  v6.6.y, having been released after v6.2,
already has all these fixes, thus there is no need for this effort
to modify it.

Once each of v6.1.y, v5.15.y, and v5.10.y is completed and tested, I
will pass the backported patch series to Greg and Sasha to apply to
the public LTS kernels they maintain.

That will make these backports available to the distributions that
continue to use the public LTS kernels, and enable Greg and Sasha
to continue applying upstream fixes to these kernels automatically.

-- 
Chuck Lever

long-term stable backports of NFSD patches

[Chuck Lever]

It's apparent that a number of distributions and their customers
remain on long-term stable kernels. We are aware of the scalability
problems and other bugs in NFSD in kernels between v5.4 and v6.1.
 
Therefore I've started an effort to backport all NFSD commits to
those kernels. The backported commits are destined for the official
LTS kernel branches so that distributions can easily integrate them
into their products.

Here's a status update.

---

I've pushed the NFSD backports to branches in this repo:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

If you are able, I encourage you to pull these, review them or try
them out, and report any issues or successes. I'm currently using
the NFS workflows in kdevops as the testing platform, but am
planning to include other tests.


LTS v6.1.y

Greg told me this morning that these are now queued for the next
LTS 6.1.y release.


LTS v5.15.y

I have updated this branch to include commits from v6.2.16 through
v5.16, all applied to v5.15.150. The fixes include most NFSD and
fanotify patches in that range. I've begun testing this series, and
so far the backport looks successful.

You can find these patches in the "nfsd-5.15.y" branch in the above
repo.


LTS v5.10.y

Backporting is still under way. I've gotten up to v6.0, and plan
to continue to v6.2.16. The fixes include most NFSD and fanotify
patches in that range, but exclude some because v5.10.y does not
include the VFS ID mapping patches that were merged into v5.12.

You can find these patches in the "nfsd-5.10.y" branch in the above
repo.


LTS v5.4.y

At this time I am planning not to backport to 5.4.y because of
missing features in this kernel.


-- 
Chuck Lever