Re: NFS server regression in kernel 5.13 (tested w/ 5.13.9)

[Mike Javorski <mike.javorski@gmail.com>]

5.12.15-arch1:
===============
14723 137 0.00921938

5.13.12-arch1: (no freezes)
===============
15333 206 0.013257

5.13.12-arch1: (with freezes)
===============
9230 299 0.0313779

So a decent bump w/ 5.13, and even more with the freezes.
This machine is an older Opteron server w/ nforce networking. Not sure
how much offloading it actually does.

- mike

On Sat, Aug 21, 2021 at 8:42 PM NeilBrown <neilb@suse.de> wrote:
>
> On Sun, 22 Aug 2021, Mike Javorski wrote:
> > OK, so new/fresh captures, reading the same set of files via NFS in
> > roughly the same timing/sequence (client unchanged between runs)
> >
> > 5.12.15-arch1:
> > ===============
> > 0.042320 124082
> > 0.042594 45837
> > 0.043176 19598
> > 0.044092 63667
> > 0.044613 28192
> > 0.045045 131268
> > 0.045982 116572
> > 0.058507 162444
> > 0.369620 153520
> > 0.825167 164682
> >
> > 5.13.12-arch1: (no freezes)
> > ===============
> > 0.040766 12679
> > 0.041565 64532
> > 0.041799 55440
> > 0.042091 159640
> > 0.042105 75075
> > 0.042134 177776
> > 0.042706 40
> > 0.043334 35322
> > 0.045480 183618
> > 0.204246 83997
> >
> > Since I didn't get any freezes, I waited a bit, tried again and got a
> > capture with several freezes...
> >
> > 5.13.12-arch1: (with freezes)
> > ===============
> > 0.042143 55425
> > 0.042252 64787
> > 0.042411 57362
> > 0.042441 34461
> > 0.042503 67041
> > 0.042553 64812
> > 0.042592 55179
> > 0.042715 67002
> > 0.042835 66977
> > 0.043308 64849
> >
> > Not sure what to make of this, but to my (admittedly untrainted) eye,
> > the two 5.13.12 sets are very similar to each other as well as to the
> > 5.12.15 sample, I am not sure if this is giving any indication to what
> > is causing the freezes.
>
> In the trace that I have, most times (242 of 245) were 0.000360 or less.
> Only 3 were greater.
> As your traces are much bigger you naturally have more that are great -
> all of the last 10 and probably more.
>
> I has hoping that 5.12 wouldn't show any large delays, but clearly it
> does.  However it is still possible that there are substantially fewer.
>
> Rather than using tail, please pipe the list of times into
>
>  awk '$1 < 0.001 { fast +=1 } $1 >= 0.001 {slow += 1} END { print fast, slow, slow / (fast + slow) }'
>
> and report the results.  If the final number (the fraction) is reliably
> significantly smaller for 5.12 than for 5.13 (whether it freezes or
> not), then this metric may still be useful.
>
> By the way, disabling the various offload options didn't appear to make
> a different for the other person who has reported this problem.
>
> Thanks,
> NeilBrown