Hi

On 05/02/2019 12:14, Benjamin Coddington wrote:
> On 5 Feb 2019, at 7:10, Cristian Marussi wrote:
> 
>> Hi Ben
>>
>> On 05/02/2019 11:53, Benjamin Coddington wrote:
>>> Hello Cristian and Punit,
>>>
>>> Did you ever get to the bottom of this one?  We just saw this on one 
>>> run
>>> of our 4.18.0-era ppc64le, and I'm wondering if we ever found the 
>>> root
>>> cause.
>>
>> unfortunately I stopped working actively on finding the root cause, 
>> since I've
>> found a viable workaround that let us unblock our broken LTP runs.
>>
>> Setting wsize=65536 in NFS bootparams completely solves the issue with 
>> 64k pages
>> (and does NOT break 4k either :D): this confirmed my hyp that there is 
>> some sort
>> of race when accounting refcounts during the lifetime of nfs_page 
>> structs which
>> leads to a misscounted refcount...but as I said I never looked back 
>> into that
>> again (but never say never...)
>>
>> Hope this helps...
> 
> Hmm, interesting..
> 
> Will you share your reproducer with me?  That will save me some time.

Sure.

My reproducer is the attached nfs_stress.sh script; when invoked with the
following params:

./nfs_stress.sh -w 10 -s 160000 -t 10

it leads to a crash within 10secs BUT ONLY with 64KB page Kconfig AND ONLY if
the above wsize workaround is NOT applied. (or the cleanup-code trick mentioned
in the emails) (the choice of the -s size parameter seemed sensible in determine
how quick it will die...)

BUT UNFORTUNATELY this was true ONLY when running on an AEMv8 FastModel (1-cpu
A53) (whose timings are much different from a real board); I've never been able
to reproduce reliably on real ARM64 silicon instead. (or on x86)
So all my debug and triage was made on the model once I was able to quickly
reproduce the same crash (and in fact the workaround worked then fine also on
silicon...)

On real silicon instead the only reproducer was a full LTP run: we had
consistent failures every night with the same exact refcount stacktrace (but
every time on a different LTP test as a trigger...being related to NFS activity
I suppose it's normal); since we applied the wsize workaround we saw no more
crashes.


Thanks

Regards

Cristian

> 
> Ben
>