Re: Many processes end up in uninterruptible sleep accessing cifs mounts

["Aurélien Aptel" <aaptel@suse.com>]

Hi Martijn,

Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> writes:
> One of the symptoms is that our monitoring system complains about not 
> being able to stat() every now and then, the next scraping cycle, stat() 
> works again. Even when the mounts are not accesses at all.
>
> Also, lot of applications get stuck on either accessing data on the 
> mounts, or performing stat() like operations on the mounts.
>
> For us, the worst part is that applications end up in 'D'. The number of 
> 'D' processes pile up really quickly, blocking users from performing 
> their work.

Gah, sorry to hear. Thanks for the report.

If you could pin down a specific way to reproduce some of those hangs
that would be of great help.

> We are running Linux 4.20.17 SMP PREEMPT on all machines. We tried 
> upgrading to > 5.x, but caused even more problems and kernel hangs.

5.0 and 5.1 really fixed a lot of issues with credits and reconnection.

> I do not really have a clue where to start debugging. I enabled kernel 
> debug options suggested on the wiki, but the amount of logging is 
> immense now.

Yes that is normal.

> Can you provide any pointers where to look or start debugging?
> Or any help on how to kill those D processes and get our Linux servers 
> stable again?

Are there any kernel oops/panic with stack traces and register dumps in
the log?

You can inspect the kernel stack trace of the hung processes (to see where
they are stuck) by printing the file /proc/<pid>/stack.

Cheers,
-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Linux GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)