* odd NFS v4.2 regression 4.4..5.15
@ 2021-10-12 13:54 Mike Galbraith
2021-10-13 6:35 ` [bisected] " Mike Galbraith
0 siblings, 1 reply; 3+ messages in thread
From: Mike Galbraith @ 2021-10-12 13:54 UTC (permalink / raw)
To: netdev
Greetings network wizards,
I seem to have stumbled across a kind of odd looking NFS regression
while looking at something unrelated.
I have a few VMs that are mirrors of my desktop box, including some
data, but they usually over-mount directories with tons of data via
NFS, thus hang off the host like a mini-me Siamese twin for testing.
In the data below, I run a script in git-daemon's home to extract RT
patch sets with git-format-patch, on the host for reference, on the
host in an NFS mount of itself, and the mini-me VM, where I started.
Note: bonnie throughput looked fine for v4.2.
host
time su - git -c ./format-rt-patches.sh
real 1m5.923s
user 0m54.692s
sys 0m11.550s
NFS mount my own box, and do the same from the same spot
time su git -c 'cd /homer/home/git;./format-rt-patches.sh'
4.4.231 v4.2 v3
1 real 2m27.046s 2m2.059s
user 0m59.190s 0m58.701s
sys 0m41.448s 0m32.541s
5.15-rc5 v4.2 v3
real 3m14.954s 2m8.366s
user 0m59.901s 0m58.317s
sys 0m52.708s 0m32.313s
vs 1 0.754 0.951
repeats v4.2 v3
real 3m16.313s 2m7.940s
real 3m10.905s 2m8.029s
----
guest
time ./format-rt-patches.sh
host=4.4, mounted v4.2
guest=5.15 guest=4.4 guest vs guest
1 real 3m38.334s 3m11.383s .876
user 1m7.747s 1m2.666s
sys 0m53.721s 0m48.127s
host=4.4, mounted v3
guest=5.15 guest=4.4
2 real 2m58.458s 2m46.857s .934
user 1m6.660s 1m3.685s
sys 0m43.026s 0m42.657s
vs 1 1.223 1.146
host=5.15,mounted v4.2
guest=5.15 guest=4.4
3 real 4m49.320s 4m29.317s .930
user 1m5.740s 1m4.724s
sys 1m12.240s 1m6.544s hmm..
vs 1 0.754 0.710
real 4m48.886s
user 1m6.503s
sys 1m14.910s *
real 4m46.432s
user 1m5.877s
sys 1m15.251s * odd, only in guest
host=5.15,mounted v3
guest=5.15 guest=4.4
4 real 3m6.965s 2m56.527s .944
user 1m5.192s 1m3.353s
sys 0m45.775s 0m43.361s
vs 2 0.954 0.945
^ permalink raw reply [flat|nested] 3+ messages in thread
* [bisected] Re: odd NFS v4.2 regression 4.4..5.15
2021-10-12 13:54 odd NFS v4.2 regression 4.4..5.15 Mike Galbraith
@ 2021-10-13 6:35 ` Mike Galbraith
2021-10-13 6:39 ` Mike Galbraith
0 siblings, 1 reply; 3+ messages in thread
From: Mike Galbraith @ 2021-10-13 6:35 UTC (permalink / raw)
To: netdev; +Cc: Jeff Layton, Trond Myklebust
On Tue, 2021-10-12 at 15:54 +0200, Mike Galbraith wrote:
> Greetings network wizards,
>
> I seem to have stumbled across a kind of odd looking NFS regression
> while looking at something unrelated.
>
> I have a few VMs that are mirrors of my desktop box, including some
> data, but they usually over-mount directories with tons of data via
> NFS, thus hang off the host like a mini-me Siamese twin for testing.
>
> In the data below, I run a script in git-daemon's home to extract RT
> patch sets with git-format-patch, on the host for reference, on the
> host in an NFS mount of itself, and the mini-me VM, where I started.
>
> Note: bonnie throughput looked fine for v4.2.
>
> host
> time su - git -c ./format-rt-patches.sh
> real 1m5.923s
> user 0m54.692s
> sys 0m11.550s
>
> NFS mount my own box, and do the same from the same spot
> time su git -c 'cd /homer/home/git;./format-rt-patches.sh'
>
> 4.4.231 v4.2 v3
> 1 real 2m27.046s 2m2.059s
> user 0m59.190s 0m58.701s
> sys 0m41.448s 0m32.541s
>
> 5.15-rc5 v4.2 v3
> real 3m14.954s 2m8.366s
> user 0m59.901s 0m58.317s
> sys 0m52.708s 0m32.313s
> vs 1 0.754 0.951
>
> repeats v4.2 v3
> real 3m16.313s 2m7.940s
> real 3m10.905s 2m8.029s
Found the v4.2 regression (for this load anyway) to have landed in 5.4.
It then bisected cleanly to either the build bug or its neighbor.
There are only 'skip'ped commits left to test.
The first bad commit could be any of:
fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189
eb82dd393744107ebc365a53e7813c7c67cb203b
We cannot bisect more!
git bisect start
# good: [4d856f72c10ecb060868ed10ff1b1453943fc6c8] Linux 5.3
git bisect good 4d856f72c10ecb060868ed10ff1b1453943fc6c8
# bad: [219d54332a09e8d8741c1e1982f5eae56099de85] Linux 5.4
git bisect bad 219d54332a09e8d8741c1e1982f5eae56099de85
# good: [a9f8b38a071b468276a243ea3ea5a0636e848cf2] Merge tag 'for-linus-5.4-1' of git://github.com/cminyard/linux-ipmi
git bisect good a9f8b38a071b468276a243ea3ea5a0636e848cf2
# good: [5825a95fe92566ada2292a65de030850b5cff1da] Merge tag 'selinux-pr-20190917' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/selinux
git bisect good 5825a95fe92566ada2292a65de030850b5cff1da
# bad: [48acba989ed5d8707500193048d6c4c5945d5f43] Merge tag 'riscv/for-v5.4-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux
git bisect bad 48acba989ed5d8707500193048d6c4c5945d5f43
# good: [7bccb9f10c8f36ee791769b531ed4d28f6379aae] Merge tag 'linux-watchdog-5.4-rc1' of git://www.linux-watchdog.org/linux-watchdog
git bisect good 7bccb9f10c8f36ee791769b531ed4d28f6379aae
# bad: [ef129d34149ea23d0d442844fc25ae26a85589fc] selftests/net: add nettest to .gitignore
git bisect bad ef129d34149ea23d0d442844fc25ae26a85589fc
# bad: [9c5efe9ae7df78600c0ee7bcce27516eb687fa6e] Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 9c5efe9ae7df78600c0ee7bcce27516eb687fa6e
# good: [8bbe0dec38e147a50e9dd5f585295f7e68e0f2d0] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
git bisect good 8bbe0dec38e147a50e9dd5f585295f7e68e0f2d0
# bad: [298fb76a5583900a155d387efaf37a8b39e5dea2] Merge tag 'nfsd-5.4' of git://linux-nfs.org/~bfields/linux
git bisect bad 298fb76a5583900a155d387efaf37a8b39e5dea2
# bad: [bbf2f098838aa86cee240a7a11486e0601d56a3f] nfsd: Reset the boot verifier on all write I/O errors
git bisect bad bbf2f098838aa86cee240a7a11486e0601d56a3f
# skip: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
git bisect skip fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189
# bad: [5e113224c17e2fb156b785ddbbc48a0209fddb0c] nfsd: nfsd_file cache entries should be per net namespace
git bisect bad 5e113224c17e2fb156b785ddbbc48a0209fddb0c
# good: [48cd7b51258c1a158293cefb97dda988080f5e13] nfsd: hook up nfsd_read to the nfsd_file cache
git bisect good 48cd7b51258c1a158293cefb97dda988080f5e13
# bad: [501cb1849f865960501d19d54e6a5af306f9b6fd] nfsd: rip out the raparms cache
git bisect bad 501cb1849f865960501d19d54e6a5af306f9b6fd
# bad: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
git bisect bad eb82dd393744107ebc365a53e7813c7c67cb203b
# good: [5920afa3c85ff38642f652b6e3880e79392fcc89] nfsd: hook nfsd_commit up to the nfsd_file cache
git bisect good 5920afa3c85ff38642f652b6e3880e79392fcc89
# only skipped commits left to test
# possible first bad commit: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
# possible first bad commit: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
# good: [5920afa3c85ff38642f652b6e3880e79392fcc89] nfsd: hook nfsd_commit up to the nfsd_file cache
git bisect good 5920afa3c85ff38642f652b6e3880e79392fcc89
# only skipped commits left to test
# possible first bad commit: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
# possible first bad commit: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [bisected] Re: odd NFS v4.2 regression 4.4..5.15
2021-10-13 6:35 ` [bisected] " Mike Galbraith
@ 2021-10-13 6:39 ` Mike Galbraith
0 siblings, 0 replies; 3+ messages in thread
From: Mike Galbraith @ 2021-10-13 6:39 UTC (permalink / raw)
To: netdev; +Cc: Jeff Layton, Trond Myklebust
[replaces bouncy address]
On Wed, 2021-10-13 at 08:35 +0200, Mike Galbraith wrote:
> On Tue, 2021-10-12 at 15:54 +0200, Mike Galbraith wrote:
> > Greetings network wizards,
> >
> > I seem to have stumbled across a kind of odd looking NFS regression
> > while looking at something unrelated.
> >
> > I have a few VMs that are mirrors of my desktop box, including some
> > data, but they usually over-mount directories with tons of data via
> > NFS, thus hang off the host like a mini-me Siamese twin for testing.
> >
> > In the data below, I run a script in git-daemon's home to extract RT
> > patch sets with git-format-patch, on the host for reference, on the
> > host in an NFS mount of itself, and the mini-me VM, where I started.
> >
> > Note: bonnie throughput looked fine for v4.2.
> >
> > host
> > time su - git -c ./format-rt-patches.sh
> > real 1m5.923s
> > user 0m54.692s
> > sys 0m11.550s
> >
> > NFS mount my own box, and do the same from the same spot
> > time su git -c 'cd /homer/home/git;./format-rt-patches.sh'
> >
> > 4.4.231 v4.2 v3
> > 1 real 2m27.046s 2m2.059s
> > user 0m59.190s 0m58.701s
> > sys 0m41.448s 0m32.541s
> >
> > 5.15-rc5 v4.2 v3
> > real 3m14.954s 2m8.366s
> > user 0m59.901s 0m58.317s
> > sys 0m52.708s 0m32.313s
> > vs 1 0.754 0.951
> >
> > repeats v4.2 v3
> > real 3m16.313s 2m7.940s
> > real 3m10.905s 2m8.029s
>
> Found the v4.2 regression (for this load anyway) to have landed in 5.4.
> It then bisected cleanly to either the build bug or its neighbor.
>
> There are only 'skip'ped commits left to test.
> The first bad commit could be any of:
> fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189
> eb82dd393744107ebc365a53e7813c7c67cb203b
> We cannot bisect more!
>
> git bisect start
> # good: [4d856f72c10ecb060868ed10ff1b1453943fc6c8] Linux 5.3
> git bisect good 4d856f72c10ecb060868ed10ff1b1453943fc6c8
> # bad: [219d54332a09e8d8741c1e1982f5eae56099de85] Linux 5.4
> git bisect bad 219d54332a09e8d8741c1e1982f5eae56099de85
> # good: [a9f8b38a071b468276a243ea3ea5a0636e848cf2] Merge tag 'for-linus-5.4-1' of git://github.com/cminyard/linux-ipmi
> git bisect good a9f8b38a071b468276a243ea3ea5a0636e848cf2
> # good: [5825a95fe92566ada2292a65de030850b5cff1da] Merge tag 'selinux-pr-20190917' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/selinux
> git bisect good 5825a95fe92566ada2292a65de030850b5cff1da
> # bad: [48acba989ed5d8707500193048d6c4c5945d5f43] Merge tag 'riscv/for-v5.4-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux
> git bisect bad 48acba989ed5d8707500193048d6c4c5945d5f43
> # good: [7bccb9f10c8f36ee791769b531ed4d28f6379aae] Merge tag 'linux-watchdog-5.4-rc1' of git://www.linux-watchdog.org/linux-watchdog
> git bisect good 7bccb9f10c8f36ee791769b531ed4d28f6379aae
> # bad: [ef129d34149ea23d0d442844fc25ae26a85589fc] selftests/net: add nettest to .gitignore
> git bisect bad ef129d34149ea23d0d442844fc25ae26a85589fc
> # bad: [9c5efe9ae7df78600c0ee7bcce27516eb687fa6e] Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> git bisect bad 9c5efe9ae7df78600c0ee7bcce27516eb687fa6e
> # good: [8bbe0dec38e147a50e9dd5f585295f7e68e0f2d0] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
> git bisect good 8bbe0dec38e147a50e9dd5f585295f7e68e0f2d0
> # bad: [298fb76a5583900a155d387efaf37a8b39e5dea2] Merge tag 'nfsd-5.4' of git://linux-nfs.org/~bfields/linux
> git bisect bad 298fb76a5583900a155d387efaf37a8b39e5dea2
> # bad: [bbf2f098838aa86cee240a7a11486e0601d56a3f] nfsd: Reset the boot verifier on all write I/O errors
> git bisect bad bbf2f098838aa86cee240a7a11486e0601d56a3f
> # skip: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
> git bisect skip fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189
> # bad: [5e113224c17e2fb156b785ddbbc48a0209fddb0c] nfsd: nfsd_file cache entries should be per net namespace
> git bisect bad 5e113224c17e2fb156b785ddbbc48a0209fddb0c
> # good: [48cd7b51258c1a158293cefb97dda988080f5e13] nfsd: hook up nfsd_read to the nfsd_file cache
> git bisect good 48cd7b51258c1a158293cefb97dda988080f5e13
> # bad: [501cb1849f865960501d19d54e6a5af306f9b6fd] nfsd: rip out the raparms cache
> git bisect bad 501cb1849f865960501d19d54e6a5af306f9b6fd
> # bad: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
> git bisect bad eb82dd393744107ebc365a53e7813c7c67cb203b
> # good: [5920afa3c85ff38642f652b6e3880e79392fcc89] nfsd: hook nfsd_commit up to the nfsd_file cache
> git bisect good 5920afa3c85ff38642f652b6e3880e79392fcc89
> # only skipped commits left to test
> # possible first bad commit: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
> # possible first bad commit: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
> # good: [5920afa3c85ff38642f652b6e3880e79392fcc89] nfsd: hook nfsd_commit up to the nfsd_file cache
> git bisect good 5920afa3c85ff38642f652b6e3880e79392fcc89
> # only skipped commits left to test
> # possible first bad commit: [eb82dd393744107ebc365a53e7813c7c67cb203b] nfsd: convert fi_deleg_file and ls_file fields to nfsd_file
> # possible first bad commit: [fd4f83fd7dfb1bce2f1af51fcbaf6575f4b9d189] nfsd: convert nfs4_file->fi_fds array to use nfsd_files
>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-10-13 6:40 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-12 13:54 odd NFS v4.2 regression 4.4..5.15 Mike Galbraith
2021-10-13 6:35 ` [bisected] " Mike Galbraith
2021-10-13 6:39 ` Mike Galbraith
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.