Possible bug in mirror target

Possible bug in mirror target

[Drew Hastings]

[-- Attachment #1.1: Type: text/plain, Size: 2032 bytes --]

Hi,

I'm assuming all user space code is expected to use the handle_errors
feature, so this isn't that big of a deal. I'm also using 4.19.13, which I
think is more recent than the latest update to dm-raid1.c

That said, there may be a bug that causes the entire mirror to crash if
there is an error on the first leg of the mirror, but not on the 2nd leg.

Works fine if you do cause IO errors on the 2nd leg:

root@kernel-dev:~# dmsetup create zerodev --table "0 1073741824 zero"
root@kernel-dev:~# dmsetup create errordev --table "0 1073741824 error"


root@kernel-dev:~# dmsetup create mirror-table --table "0 1073741824 mirror
core 2 524288 nosync 2 /dev/mapper/zerodev 0 /dev/mapper/errordev 0 0"
root@kernel-dev:~# dd if=/dev/zero of=/dev/mapper/mirror-table bs=512
count=1
1+0 records in
1+0 records out
512 bytes copied, 0.000417845 s, 1.2 MB/s
root@kernel-dev:~# dd if=/dev/zero of=/dev/mapper/mirror-table bs=512
count=1
1+0 records in
1+0 records out
512 bytes copied, 0.00036228 s, 1.4 MB/s
root@kernel-dev:~# dmsetup status mirror-table
0 1073741824 mirror 2 253:1 253:2 2047/2048 1 AD 1 core
root@kernel-dev:~# dmsetup remove mirror-table


The whole mirror fails if you do the error device on the first leg:

root@kernel-dev:~# dmsetup create mirror-table --table "0 1073741824 mirror
core 2 524288 nosync 2 /dev/mapper/errordev 0 /dev/mapper/zerodev 0 0"
root@kernel-dev:~# dd if=/dev/zero of=/dev/mapper/mirror-table bs=512
count=1
1+0 records in
1+0 records out
512 bytes copied, 0.000274188 s, 1.9 MB/s
root@kernel-dev:~# dd if=/dev/zero of=/dev/mapper/mirror-table bs=512
count=1
dd: error writing '/dev/mapper/mirror-table': Input/output error
1+0 records in
0+0 records out
0 bytes copied, 0.000857634 s, 0.0 kB/s

root@kernel-dev:~# dmsetup status mirror-table
0 1073741824 mirror 2 253:2 253:1 2047/2048 1 DA 1 core

dmesg would be:

[ 1935.876401] device-mapper: raid1: Mirror read failed from 253:2. Trying
alternative device.
[ 1939.091919] device-mapper: raid1: Mirror read failed.

Thanks for your time!

[-- Attachment #1.2: Type: text/html, Size: 2812 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Possible bug in mirror target

[Zdenek Kabelac]

Dne 05. 02. 19 v 1:47 Drew Hastings napsal(a):
> Hi,
> 
> I'm assuming all user space code is expected to use the handle_errors feature, 
> so this isn't that big of a deal. I'm also using 4.19.13, which I think is 
> more recent than the latest update to dm-raid1.c
> 
> That said, there may be a bug that causes the entire mirror to crash if there 
> is an error on the first leg of the mirror, but not on the 2nd leg.
> 
> Works fine if you do cause IO errors on the 2nd leg:
> 
> root@kernel-dev:~# dmsetup create zerodev --table "0 1073741824 zero"
> root@kernel-dev:~# dmsetup create errordev --table "0 1073741824 error"
> 
> 
> root@kernel-dev:~# dmsetup create mirror-table --table "0 1073741824 mirror 
> core 2 524288 nosync 2 /dev/mapper/zerodev 0 /dev/mapper/errordev 0 0"

Hi

There are 2 operational modes for old dm mirror target.
One requires to handle errors externally. Please check i.e. the following 
guide for mirror target:

https://wiki.gentoo.org/wiki/Device-mapper

Lvm2 is using 'dmeventd' to service this error handling (i.e. dropping
the mirror leg or allocating replacement one).

Without this option you will start to receive disc failures.
Also not -  old mirror has been obsoleted by dm-raid target, that
is using small log device for each leg (and using same kernel
internals as mdraid)


Regards

Zdenek

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: Possible bug in mirror target

[John Stoffel]

>>>>> "Zdenek" == Zdenek Kabelac <zkabelac@redhat.com> writes:

Zdenek> Dne 05. 02. 19 v 1:47 Drew Hastings napsal(a):
>> Hi,
>> 
>> I'm assuming all user space code is expected to use the handle_errors feature, 
>> so this isn't that big of a deal. I'm also using 4.19.13, which I think is 
>> more recent than the latest update to dm-raid1.c
>> 
>> That said, there may be a bug that causes the entire mirror to crash if there 
>> is an error on the first leg of the mirror, but not on the 2nd leg.
>> 
>> Works fine if you do cause IO errors on the 2nd leg:
>> 
>> root@kernel-dev:~# dmsetup create zerodev --table "0 1073741824 zero"
>> root@kernel-dev:~# dmsetup create errordev --table "0 1073741824 error"
>> 
>> 
>> root@kernel-dev:~# dmsetup create mirror-table --table "0 1073741824 mirror 
>> core 2 524288 nosync 2 /dev/mapper/zerodev 0 /dev/mapper/errordev 0 0"

Zdenek> There are 2 operational modes for old dm mirror target.
Zdenek> One requires to handle errors externally. Please check i.e. the following 
Zdenek> guide for mirror target:

Zdenek> https://wiki.gentoo.org/wiki/Device-mapper

Zdenek> Lvm2 is using 'dmeventd' to service this error handling
Zdenek> (i.e. dropping the mirror leg or allocating replacement one).

Is it time to remove the dm-mirror target then?  Or to deprecate it?
Or to just silently replace it with something that does the right
thing when errors happen?  I can't think of why *anyone* would want to
use the dm-mirror target as it now seems to work.

John

Re: Possible bug in mirror target

[Zdenek Kabelac]

Dne 10. 02. 19 v 22:58 John Stoffel napsal(a):
>>>>>> "Zdenek" == Zdenek Kabelac <zkabelac@redhat.com> writes:
> 
> Zdenek> Dne 05. 02. 19 v 1:47 Drew Hastings napsal(a):
>>> Hi,
>>>
>>> I'm assuming all user space code is expected to use the handle_errors feature,
>>> so this isn't that big of a deal. I'm also using 4.19.13, which I think is
>>> more recent than the latest update to dm-raid1.c
>>>
>>> That said, there may be a bug that causes the entire mirror to crash if there
>>> is an error on the first leg of the mirror, but not on the 2nd leg.
>>>
>>> Works fine if you do cause IO errors on the 2nd leg:
>>>
>>> root@kernel-dev:~# dmsetup create zerodev --table "0 1073741824 zero"
>>> root@kernel-dev:~# dmsetup create errordev --table "0 1073741824 error"
>>>
>>>
>>> root@kernel-dev:~# dmsetup create mirror-table --table "0 1073741824 mirror
>>> core 2 524288 nosync 2 /dev/mapper/zerodev 0 /dev/mapper/errordev 0 0"
> 
> Zdenek> There are 2 operational modes for old dm mirror target.
> Zdenek> One requires to handle errors externally. Please check i.e. the following
> Zdenek> guide for mirror target:
> 
> Zdenek> https://wiki.gentoo.org/wiki/Device-mapper
> 
> Zdenek> Lvm2 is using 'dmeventd' to service this error handling
> Zdenek> (i.e. dropping the mirror leg or allocating replacement one).
> 
> Is it time to remove the dm-mirror target then?  Or to deprecate it?
> Or to just silently replace it with something that does the right
> thing when errors happen?  I can't think of why *anyone* would want to
> use the dm-mirror target as it now seems to work.


Hi

The old dm mirror works differently then new dm raid - so if you have
an old one present/running (i.e. LVM mirrored volume) using this target - you 
need to have this target present to be able to access such volume.

You can (in lvm2) convert such mirror to use newer dm raid target - but
this requires some extra space (even though very small metadata volume) which 
is now required per each mirrored leg.

Both targets are still maintained and bugfixed.

There is also something about very high complexity of mdraid code and
relative simplicity of old mirror target.

As for comment about *anyone* - majority of users are consuming DM targets via 
lvm2 - there you get this transition automatically - when you ask to create 
mirrored  LV - lvm2 will use (if present in kernel)  newer md  raid variant. 
If anyone is using DM directly with 'dmsetup' command - it's assumed these are 
skilled users familiar with kernel doc (dmsetup is low-level admins tool). 
Lvm2 should be seen as 'friendly-face' of these DM targets - which otherwise 
do require pretty complex setup. If you do not want to use lvm2, the tool 
replacing lvm2 needs to reproduce this extra logic.

Regards

Zdenek

Re: Possible bug in mirror target

[John Stoffel]

>>>>> "Zdenek" == Zdenek Kabelac <zkabelac@redhat.com> writes:

Zdenek> Dne 10. 02. 19 v 22:58 John Stoffel napsal(a):
>>>>>>> "Zdenek" == Zdenek Kabelac <zkabelac@redhat.com> writes:
>> 
Zdenek> Dne 05. 02. 19 v 1:47 Drew Hastings napsal(a):
>>>> Hi,
>>>> 
>>>> I'm assuming all user space code is expected to use the handle_errors feature,
>>>> so this isn't that big of a deal. I'm also using 4.19.13, which I think is
>>>> more recent than the latest update to dm-raid1.c
>>>> 
>>>> That said, there may be a bug that causes the entire mirror to crash if there
>>>> is an error on the first leg of the mirror, but not on the 2nd leg.
>>>> 
>>>> Works fine if you do cause IO errors on the 2nd leg:
>>>> 
>>>> root@kernel-dev:~# dmsetup create zerodev --table "0 1073741824 zero"
>>>> root@kernel-dev:~# dmsetup create errordev --table "0 1073741824 error"
>>>> 
>>>> 
>>>> root@kernel-dev:~# dmsetup create mirror-table --table "0 1073741824 mirror
>>>> core 2 524288 nosync 2 /dev/mapper/zerodev 0 /dev/mapper/errordev 0 0"
>> 
Zdenek> There are 2 operational modes for old dm mirror target.
Zdenek> One requires to handle errors externally. Please check i.e. the following
Zdenek> guide for mirror target:
>> 
Zdenek> https://wiki.gentoo.org/wiki/Device-mapper
>> 
Zdenek> Lvm2 is using 'dmeventd' to service this error handling
Zdenek> (i.e. dropping the mirror leg or allocating replacement one).
>> 
>> Is it time to remove the dm-mirror target then?  Or to deprecate it?
>> Or to just silently replace it with something that does the right
>> thing when errors happen?  I can't think of why *anyone* would want to
>> use the dm-mirror target as it now seems to work.


Zdenek> The old dm mirror works differently then new dm raid - so if
Zdenek> you have an old one present/running (i.e. LVM mirrored volume)
Zdenek> using this target - you need to have this target present to be
Zdenek> able to access such volume.

Ok, I see that.  I just wonder if A) anyone still uses this target and
B) if it should be deprecated since it doesn't provide the protections
that the dm_raid target provides.

Getting rid of legacy stuff isn't a bad thing... :-)

Zdenek> You can (in lvm2) convert such mirror to use newer dm raid
Zdenek> target - but this requires some extra space (even though very
Zdenek> small metadata volume) which is now required per each mirrored
Zdenek> leg.

Not a bad thing.

Zdenek> Both targets are still maintained and bugfixed.

Zdenek> There is also something about very high complexity of mdraid
Zdenek> code and relative simplicity of old mirror target.

I'm thinking about the complexity of maintaing both in parallel, when
the older one doesn't provide the protections of the newer one. 

Zdenek> As for comment about *anyone* - majority of users are
Zdenek> consuming DM targets via lvm2 - there you get this transition
Zdenek> automatically - when you ask to create mirrored LV - lvm2 will
Zdenek> use (if present in kernel) newer md raid variant.  If anyone
Zdenek> is using DM directly with 'dmsetup' command - it's assumed
Zdenek> these are skilled users familiar with kernel doc (dmsetup is
Zdenek> low-level admins tool).  Lvm2 should be seen as
Zdenek> 'friendly-face' of these DM targets - which otherwise do
Zdenek> require pretty complex setup. If you do not want to use lvm2,
Zdenek> the tool replacing lvm2 needs to reproduce this extra logic.

Right, so how many use the old version?  

Re: Possible bug in mirror target

[Zdenek Kabelac]

Hi

Dne 12. 02. 19 v 23:36 John Stoffel napsal(a):
>>>>>> "Zdenek" == Zdenek Kabelac <zkabelac@redhat.com> writes:
> 
> Zdenek> Dne 10. 02. 19 v 22:58 John Stoffel napsal(a):
> 
> Zdenek> The old dm mirror works differently then new dm raid - so if
> Zdenek> you have an old one present/running (i.e. LVM mirrored volume)
> Zdenek> using this target - you need to have this target present to be
> Zdenek> able to access such volume.
> 
> Ok, I see that.  I just wonder if A) anyone still uses this target and
> B) if it should be deprecated since it doesn't provide the protections
> that the dm_raid target provides.

Well there is at least one existing user of old mirror target - and it's lvm2, 
  which is still actively  using it for pvmove operation - and there is not 
yet any new code to switch using mdraid - and it's not completely trivial
to make such change.

As said properties of each target are slightly different and number
of mdraid issue with regards of error detection needed by lvm2
have been fixed relatively recently (kernels >4.15)

> 
> Getting rid of legacy stuff isn't a bad thing... :-)
> 

Mirror target cannot be considered legacy stuff yet...

> Zdenek> You can (in lvm2) convert such mirror to use newer dm raid
> Zdenek> target - but this requires some extra space (even though very
> Zdenek> small metadata volume) which is now required per each mirrored
> Zdenek> leg.
> 
> Not a bad thing.

If you have this free space it's surely not a bad thing - but there
are many cases, you can get any free space - majority of filesystems
does not support shrinking - so the only 'free' space you can
get is by adding new storage to VG - again quite limiting factor.

> Zdenek> Both targets are still maintained and bugfixed.
> 
> Zdenek> There is also something about very high complexity of mdraid
> Zdenek> code and relative simplicity of old mirror target.
> 
> I'm thinking about the complexity of maintaing both in parallel, when
> the older one doesn't provide the protections of the newer one.

There is very little amount of work associated with maintenance of
old mirror target - as said it's relatively simple one.

I'm not sure about which missing protection you are talking about - write 
error is handled by user-space application - which has been a designed feature 
- so it's not missing - it just need to be correctly configured.

> 
> Zdenek> As for comment about *anyone* - majority of users are
> Zdenek> consuming DM targets via lvm2 - there you get this transition
> Zdenek> automatically - when you ask to create mirrored LV - lvm2 will
> Zdenek> use (if present in kernel) newer md raid variant.  If anyone
> Zdenek> is using DM directly with 'dmsetup' command - it's assumed
> Zdenek> these are skilled users familiar with kernel doc (dmsetup is
> Zdenek> low-level admins tool).  Lvm2 should be seen as
> Zdenek> 'friendly-face' of these DM targets - which otherwise do
> Zdenek> require pretty complex setup. If you do not want to use lvm2,
> Zdenek> the tool replacing lvm2 needs to reproduce this extra logic.
> 
> Right, so how many use the old version?

With lvm simplest way (with enabled monitoring - which is default):

lvcreate --type mirror -Lsize -m #MIRRORS -n LVNAME VGNAME

Regards

Zdenek

Re: Possible bug in mirror target

[Bryn M. Reeves]

On Wed, Feb 13, 2019 at 03:21:52PM +0100, Zdenek Kabelac wrote:
> If you have this free space it's surely not a bad thing - but there
> are many cases, you can get any free space - majority of filesystems
> does not support shrinking - so the only 'free' space you can
> get is by adding new storage to VG - again quite limiting factor.

Probably obvious to many, but if you find yourself in this situation
using the Red Hat / Fedora default layout created by Anaconda, the
easiest solution is normally to "steal" one extent from the auto-
configured swap LV, and donate it to the mirror conversion:

  # swapoff -a
  # lvresize -l -1 rhel/swap
  # mkswap /dev/rhel/swap
  # swapon -a
  # lvchange -m1 ...

This will always work with the automatic layouts provided by those
distros.

Regards,
Bryn.

Re: Possible bug in mirror target

[Zdenek Kabelac]

Dne 13. 02. 19 v 16:01 Bryn M. Reeves napsal(a):
> On Wed, Feb 13, 2019 at 03:21:52PM +0100, Zdenek Kabelac wrote:
>> If you have this free space it's surely not a bad thing - but there
>> are many cases, you can get any free space - majority of filesystems
>> does not support shrinking - so the only 'free' space you can
>> get is by adding new storage to VG - again quite limiting factor.
> 
> Probably obvious to many, but if you find yourself in this situation
> using the Red Hat / Fedora default layout created by Anaconda, the
> easiest solution is normally to "steal" one extent from the auto-
> configured swap LV, and donate it to the mirror conversion:
> 
>    # swapoff -a
>    # lvresize -l -1 rhel/swap
>    # mkswap /dev/rhel/swap
>    # swapon -a
>    # lvchange -m1 ...
> 
> This will always work with the automatic layouts provided by those
> distros.

Just to further clear this - yeah - stealing 1 extent this way 'mostly' 
temporarily works, however for conversion to 'raid' - the rule should be, that 
log/metadata device for each leg should be co-located on the same physical 
device as the leg itself - clearly storing multiple or all metadata devices on 
1 physical device will not have required purpose - although as long as all 
devices do run correctly such setup will appear as usable (and user will 
actually not get any warning that setup is actually not very safe for device 
failure)

Regards,
Zdenek

Re: Possible bug in mirror target

[Bryn M. Reeves]

On Wed, Feb 13, 2019 at 07:08:40PM +0100, Zdenek Kabelac wrote:
> Dne 13. 02. 19 v 16:01 Bryn M. Reeves napsal(a):
> > On Wed, Feb 13, 2019 at 03:21:52PM +0100, Zdenek Kabelac wrote:
> > > If you have this free space it's surely not a bad thing - but there
> > > are many cases, you can get any free space - majority of filesystems
> > > does not support shrinking - so the only 'free' space you can
> > > get is by adding new storage to VG - again quite limiting factor.
> > 
> > Probably obvious to many, but if you find yourself in this situation
> > using the Red Hat / Fedora default layout created by Anaconda, the
> > easiest solution is normally to "steal" one extent from the auto-
> > configured swap LV, and donate it to the mirror conversion:
> > 
> >    # swapoff -a
> >    # lvresize -l -1 rhel/swap
> >    # mkswap /dev/rhel/swap
> >    # swapon -a
> >    # lvchange -m1 ...
> > 
> > This will always work with the automatic layouts provided by those
> > distros.
> 
> Just to further clear this - yeah - stealing 1 extent this way 'mostly'
> temporarily works, however for conversion to 'raid' - the rule should be,
> that log/metadata device for each leg should be co-located on the same
> physical device as the leg itself - clearly storing multiple or all metadata
> devices on 1 physical device will not have required purpose - although as

That's exactly what those steps do (I used them two days ago to verify
an RHBZ). You steal the extent from the system PV that Anaconda originally
created the volume group on, providing a rmeta log on the same device as
the image volume.

Extra steps would be needed to coerce the logs onto the same devices:

  rhel-root (253:4)
   ├─rhel-root_rimage_1 (253:3)
   │  └─ (252:17)
   ├─rhel-root_rmeta_1 (253:2)
   │  └─ (252:17)
   ├─rhel-root_rimage_0 (253:1)
   │  └─ (252:2)
   └─rhel-root_rmeta_0 (253:0)
      └─ (252:2)

You obviously need to also provide sufficient space in any new PV to allow
for the extra log extent, but that shouldn't be either surprising or
difficult.

Regards,
Bryn.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel