[linux-lvm] dmsetup hangs forever

[linux-lvm] dmsetup hangs forever

[Zhangyanfei (YF)]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

Hello

I find an issue when use  dmsetup in the situation udev event timeout.

Dmsetup use the dm_udev_wait function sync with udev event.When use the dmsetup generate a new dm-disk, if the raw disk is abnormal(for example ,a ipsan disk hung IO request), the udevd daemon handle the dm-disk udev event maybe timeout, and will not notify the dmsetup  by semaphore. And because the  dm_udev_wait use the semop to sync with udevd, if udevd event timeout, the dmsetup will hung forever even when the raw disk be recovery.

I wonder if we could use the semtimedop instead semop to add the timeout in function  dm_udev_wait. If the udevd daemon timeout when handle the dm event, the dm_udev_wait could timeout too, and the dmsetup could return error.

This is my patch base lvm2-2.02.115-3:

Re: [linux-lvm] [dm-devel] dmsetup hangs forever

[Zdenek Kabelac]

Dne 26.10.2017 v 10:07 Zhangyanfei (YF) napsal(a):
> Hello
> 
> I find an issue when use �dmsetup in the situation udev event timeout.
> 
> Dmsetup use the dm_udev_wait function sync with udev event.When use the 
> dmsetup generate a new dm-disk, if the raw disk is abnormal(for example ,a 
> ipsan disk hung IO request), the udevd daemon handle the dm-disk udev event 
> maybe timeout, and will not notify the dmsetup �by semaphore. And because the 
>  �dm_udev_wait use the semop to sync with udevd, if udevd event timeout, the 
> dmsetup will hung forever even when the raw disk be recovery.
> 
> I wonder if we could use the semtimedop instead semop to add the timeout in 
> function �dm_udev_wait. If the udevd daemon timeout when handle the dm event, 
> the dm_udev_wait could timeout too, and the dmsetup could return error.
> 
> This is my patch base lvm2-2.02.115-3:


Hi


Unfortunately the same argument why this can't really work still applies.

If the  dm will start to timeout on it's own - without coordination with udev,
your system's logic will end-up with one big mess.

So if the dm would handle timeout - you would also need to provide mechanism 
to correct associated services around it.

The main case here is - it's mandatory it's udev finalizing any timeouts so 
it's in sync with db content.

Moreover if you start to timeout - you typically mask some system failure. In 
majority of cases I've ever seen - it's been always a bug from this category 
(buggy udev rule, or service). So it's always better to fix the bug then keep 
it masked.

AFAIK I'd like to see the semaphore to go away - but it needs wider cooperation.


Regards

Zdenek

[linux-lvm] 答复: [dm-devel] dmsetup hangs forever

[Zhangyanfei (YF)]

Hello

If the udevd daemon would not timeout, I think dmsetup mandatory wait udev finalizing any timeouts is good idea.
But udevd would timeout in 180 sencond and kill the event process( systemd-udevd[39029]: timeout: killing). In this situation, I think mandatory wait udev finalizing is useless, because udev has been killed and can't coordination dmsetup forever. So I think it's better to tell the one who call the dmsetup, the process error return, than let the process wait forever. 

If not add the dmsetup timeout mechanism, which strategy to solve this issue better?
1、guarantee the udev never timeout.(but I think it is difficult to make sure any udev event will finish in 180 sencond in any abnormal situation)
2、modify the udev daemon, if udev event timeout，also notify the dmsetup it's done.
3、the one who call the command dmsetup needed timeout itself.

Thanks

-----邮件原件-----
发件人: Zdenek Kabelac [mailto:zkabelac@redhat.com] 
发送时间: 2017年10月26日 16:39
收件人: Zhangyanfei (YF) <yanfei.zhang@huawei.com>; agk@redhat.com; christophe.varoqui@opensvc.com
抄送: dm-devel@redhat.com; guijianfeng <guijianfeng@huawei.com>; Fengtiantian <fengtiantian@huawei.com>; linux-lvm@redhat.com
主题: Re: [dm-devel] dmsetup hangs forever

Dne 26.10.2017 v 10:07 Zhangyanfei (YF) napsal(a):
> Hello
> 
> I find an issue when use  dmsetup in the situation udev event timeout.
> 
> Dmsetup use the dm_udev_wait function sync with udev event.When use 
> the dmsetup generate a new dm-disk, if the raw disk is abnormal(for 
> example ,a ipsan disk hung IO request), the udevd daemon handle the 
> dm-disk udev event maybe timeout, and will not notify the dmsetup  by 
> semaphore. And because the
>   dm_udev_wait use the semop to sync with udevd, if udevd event 
> timeout, the dmsetup will hung forever even when the raw disk be recovery.
> 
> I wonder if we could use the semtimedop instead semop to add the 
> timeout in function  dm_udev_wait. If the udevd daemon timeout when 
> handle the dm event, the dm_udev_wait could timeout too, and the dmsetup could return error.
> 
> This is my patch base lvm2-2.02.115-3:


Hi


Unfortunately the same argument why this can't really work still applies.

If the  dm will start to timeout on it's own - without coordination with udev, your system's logic will end-up with one big mess.

So if the dm would handle timeout - you would also need to provide mechanism to correct associated services around it.

The main case here is - it's mandatory it's udev finalizing any timeouts so it's in sync with db content.

Moreover if you start to timeout - you typically mask some system failure. In majority of cases I've ever seen - it's been always a bug from this category (buggy udev rule, or service). So it's always better to fix the bug then keep it masked.

AFAIK I'd like to see the semaphore to go away - but it needs wider cooperation.


Regards

Zdenek

Re: [linux-lvm] 答复: [dm-devel] dmsetup hangs forever

[Zdenek Kabelac]

Dne 27.10.2017 v 10:00 Zhangyanfei (YF) napsal(a):
> Hello
> 
> If the udevd daemon would not timeout, I think dmsetup mandatory wait udev finalizing any timeouts is good idea.
> But udevd would timeout in 180 sencond and kill the event process( systemd-udevd[39029]: timeout: killing). In this situation, I think 

The point here is - udev should not be 'randomly' killing workers - and if it 
needs so - it should provide a 'recovery' path mechanism, where such path
after killing worker is supported.

If this mechanism is missing - you end-up with a device which is NOT present 
in udev DB - yet - there is some 'esoteric' dm device which is there
and you don't know if it's usable or not.

mandatory wait udev finalizing is useless, because udev has been killed and 
can't coordination dmsetup forever. So I think it's better to tell the one who 
call the dmsetup, the process error return, than let the process wait forever.
> 
> If not add the dmsetup timeout mechanism, which strategy to solve this issue better?

You can always use i.e. cron job and every 15 minutes run:

'dmsetup udevcomplete_all 15'

To complete any blocked dmsetup/lvm command on a cookie...

The keypoint here is - you have system with broken device list - so any next 
i.e. lvm2 command may 'freeze'  while it would try to read a device that can 
freeze the reading task - this is certainly bad case.

> 1、guarantee the udev never timeout.(but I think it is difficult to make sure any udev event will finish in 180 sencond in any abnormal situation)
> 2、modify the udev daemon, if udev event timeout，also notify the dmsetup it's done.
> 3、the one who call the command dmsetup needed timeout itself.

In principle timeouts are BAD when we talk about storage.
There needs to be some clear 'state-machine' mechanism.

It doesn't make much sense to kill worker which actually might not even be 
possible if the worker freezes inside kernel.


And then there is even 2nd. category of 'kills' of workers - this happens on 
overloaded machines running gazillion 'virtual guest' instances without any 
resource check mechanism with assumption "OOM" killer is beautiful resource 
manager in kernel - however such systems are destined for a near-time reboot - 
  anyone who tries to 'recover' from  an OOM doesn't realize how complex that 
would be - so this case is not worth dmsetup timeout either.


Regards

Zdenek