Re: Many processes end up in uninterruptible sleep accessing cifs mounts

[Steve French <smfrench@gmail.com>]

Normally dynamic tracing is much easier and less noisy since you can
selectively turn on and off tracing with it (especially easier now
that good tutorials on how to use eBPF to make it even easier).

"trace-cmd record -e cifs"
then repro the problem
then in another process "trace-cmd show"

There are 70 events that can be individually turned on or off in
         /sys/kernel/debug/tracing/events/cifs

e.g. "trace-cmd record -e smb3_cmd_err -e smb3_enter -e smb3_exit_err"

Also can look at /proc/fs/cifs/Stats which shows how many of each
command and how many errors and if any reconnects

On Wed, Jul 3, 2019 at 2:18 PM Aurélien Aptel <aaptel@suse.com> wrote:
>
> Hi Martijn,
>
> Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> writes:
> > One of the symptoms is that our monitoring system complains about not
> > being able to stat() every now and then, the next scraping cycle, stat()
> > works again. Even when the mounts are not accesses at all.
> >
> > Also, lot of applications get stuck on either accessing data on the
> > mounts, or performing stat() like operations on the mounts.
> >
> > For us, the worst part is that applications end up in 'D'. The number of
> > 'D' processes pile up really quickly, blocking users from performing
> > their work.
>
> Gah, sorry to hear. Thanks for the report.
>
> If you could pin down a specific way to reproduce some of those hangs
> that would be of great help.
>
> > We are running Linux 4.20.17 SMP PREEMPT on all machines. We tried
> > upgrading to > 5.x, but caused even more problems and kernel hangs.
>
> 5.0 and 5.1 really fixed a lot of issues with credits and reconnection.
>
> > I do not really have a clue where to start debugging. I enabled kernel
> > debug options suggested on the wiki, but the amount of logging is
> > immense now.
>
> Yes that is normal.
>
> > Can you provide any pointers where to look or start debugging?
> > Or any help on how to kill those D processes and get our Linux servers
> > stable again?
>
> Are there any kernel oops/panic with stack traces and register dumps in
> the log?
>
> You can inspect the kernel stack trace of the hung processes (to see where
> they are stuck) by printing the file /proc/<pid>/stack.
>
> Cheers,
> --
> Aurélien Aptel / SUSE Labs Samba Team
> GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
> SUSE Linux GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
> GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)



-- 
Thanks,

Steve