During systemd/udev, device-mapper trying to work with non-LVM volumes

During systemd/udev, device-mapper trying to work with non-LVM volumes

[james harvey]

If I understand what's going on here, I think device-mapper is trying
to work with two volumes that don't involve LVM, causing the errors.
Is this a device-mapper bug?  A udev bug?  Something I have configured
wrong?

During boot, I see the following messages:

===============================
starting version 230
:: running early hook [lvm2]
:: running hook [udev]
:: Triggering uevents...
    [    3.964973] device-mapper: table: 253:21: thin: Unable to
activate thin device while pool is suspended
    [    4.023198] device-mapper: table: 253:22: thin: Unable to
activate thin device while pool is suspended
. . .
===============================

In dmesg, I see:

===============================
[    3.313426] Btrfs loaded
[    3.313987] BTRFS: device label terrasnapper_boot devid 1 transid 13 /dev/md2
[    3.314219] BTRFS: device label terramain_boot devid 1 transid 27 /dev/md1
[    3.391995] device-mapper: thin: Data device (dm-5) discard
unsupported: Disabling discard passdown.
[    3.392803] device-mapper: thin: Data device (dm-3) discard
unsupported: Disabling discard passdown.
[    3.520545] BTRFS: device label terrasnapper devid 3 transid 28 /dev/dm-13
[    3.520931] BTRFS: device label terramain devid 2 transid 35 /dev/dm-11
[    3.574622] BTRFS: device label persistent devid 3 transid 11 /dev/dm-14
[    3.590508] BTRFS: device label terrasnapper devid 2 transid 28 /dev/dm-12
[    3.604053] BTRFS: device label terramain devid 3 transid 35 /dev/dm-10
[    3.711390] BTRFS: device label persistent devid 2 transid 11 /dev/dm-15
[    3.788923] md: bind<dm-16>
[    3.790860] md/raid1:md0: active with 3 out of 3 mirrors
[    3.790909] md0: detected capacity change from 0 to 17163091968
[    3.961862] device-mapper: thin: Data device (dm-18) discard
unsupported: Disabling discard passdown.
[    3.964973] device-mapper: table: 253:21: thin: Unable to activate
thin device while pool is suspended
[    3.965075] device-mapper: ioctl: error adding target to table
[    4.023198] device-mapper: table: 253:22: thin: Unable to activate
thin device while pool is suspended
[    4.023295] device-mapper: ioctl: error adding target to table
[    4.111046] BTRFS: device label terrasnapper devid 1 transid 28 /dev/dm-21
[    4.188863] BTRFS: device label persistent devid 1 transid 11 /dev/dm-23
[    4.196150] BTRFS: device label terramain devid 1 transid 35 /dev/dm-22
===============================

There are 3 hard drives, all platter SATA.

dm-21 and dm-22 are NOT using LVM.  (If that is what is being referred
to by "table: 253:21" and "table: 253:22".) They are each a btrfs
filesystem, on a md RAID 1, of 3 hard drive partitions.  They're each
separate boot partitions, for separate installs.

All other volumes in the system are btrfs RAID's on LVM volumes, not
using md arrays.  (Except swap is an md array of raw LVM volumes.)

dmsetup table shows (253:21 and 253:22 are NOT on here)

===============================
disk3-disk3thin: 0 1048576000 linear 253:6 0
disk3-terramain3: 0 209715200 thin 253:6 1
disk3-disk3thin-tpool: 0 1048576000 thin-pool 253:4 253:5 512 0 0
disk3-disk3thin_tdata: 0 1048576000 linear 8:35 33818624
disk2-terraswap2: 0 33554432 linear 8:19 2048
disk1-disk1thin-tpool: 0 1048576000 thin-pool 253:17 253:18 512 0 0
disk1-disk1thin_tdata: 0 1048576000 linear 8:3 33818624
disk3-disk3thin_tmeta: 0 262144 linear 8:35 1082394624
disk1-disk1thin_tmeta: 0 262144 linear 8:3 1082394624
disk2-disk2thin: 0 1048576000 linear 253:7 0
disk3-terrasnapper3: 0 209715200 thin 253:6 2
disk2-persistent2: 0 209715200 thin 253:7 3
disk3-terraswap3: 0 33554432 linear 8:35 2048
disk1-disk1thin: 0 1048576000 linear 253:19 0
disk2-terrasnapper2: 0 209715200 thin 253:7 2
disk1-terramain1: 0 209715200 thin 253:19 1
disk2-disk2thin-tpool: 0 1048576000 thin-pool 253:2 253:3 512 0 0
disk2-disk2thin_tdata: 0 1048576000 linear 8:19 33818624
disk1-persistent1: 0 209715200 thin 253:19 3
disk2-disk2thin_tmeta: 0 262144 linear 8:19 1082394624
disk1-terrasnapper1: 0 209715200 thin 253:19 2
disk2-terramain2: 0 209715200 thin 253:7 1
disk3-persistent3: 0 209715200 thin 253:6 3
disk1-terraswap1: 0 33554432 linear 8:3 2048
===============================

Re: During systemd/udev, device-mapper trying to work with non-LVM volumes

[Marian Csontos]

On 07/23/2016 01:14 AM, james harvey wrote:
> If I understand what's going on here, I think device-mapper is trying
> to work with two volumes that don't involve LVM, causing the errors.

If I understand correctly, these volumes DO involve LVM.

It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on 
top of LVM.

Using BTRFS with thin-shapshots is not a good idea, especially, if you 
have multiple snapshots of btrfs' underlying device active.

Why are you usingn BTRFS on top of thin-pool?
BTRFS does have snapshots and IMHO you should pick either BTRFS or 
thin-pool.

> Is this a device-mapper bug?  A udev bug?  Something I have configured
> wrong?

Hard to say...

Which distribution?
Kernel, lvm version?

Ideally run `lvmdump -m` and post output, please.

>
> During boot, I see the following messages:
>
> ===============================
> starting version 230
> :: running early hook [lvm2]
> :: running hook [udev]
> :: Triggering uevents...
>     [    3.964973] device-mapper: table: 253:21: thin: Unable to
> activate thin device while pool is suspended
>     [    4.023198] device-mapper: table: 253:22: thin: Unable to
> activate thin device while pool is suspended
> . . .
> ===============================
>
> In dmesg, I see:
>
> ===============================
> [    3.313426] Btrfs loaded
> [    3.313987] BTRFS: device label terrasnapper_boot devid 1 transid 13 /dev/md2
> [    3.314219] BTRFS: device label terramain_boot devid 1 transid 27 /dev/md1
> [    3.391995] device-mapper: thin: Data device (dm-5) discard
> unsupported: Disabling discard passdown.
> [    3.392803] device-mapper: thin: Data device (dm-3) discard
> unsupported: Disabling discard passdown.
> [    3.520545] BTRFS: device label terrasnapper devid 3 transid 28 /dev/dm-13
> [    3.520931] BTRFS: device label terramain devid 2 transid 35 /dev/dm-11
> [    3.574622] BTRFS: device label persistent devid 3 transid 11 /dev/dm-14
> [    3.590508] BTRFS: device label terrasnapper devid 2 transid 28 /dev/dm-12
> [    3.604053] BTRFS: device label terramain devid 3 transid 35 /dev/dm-10
> [    3.711390] BTRFS: device label persistent devid 2 transid 11 /dev/dm-15
> [    3.788923] md: bind<dm-16>
> [    3.790860] md/raid1:md0: active with 3 out of 3 mirrors
> [    3.790909] md0: detected capacity change from 0 to 17163091968
> [    3.961862] device-mapper: thin: Data device (dm-18) discard
> unsupported: Disabling discard passdown.
> [    3.964973] device-mapper: table: 253:21: thin: Unable to activate
> thin device while pool is suspended
> [    3.965075] device-mapper: ioctl: error adding target to table
> [    4.023198] device-mapper: table: 253:22: thin: Unable to activate
> thin device while pool is suspended
> [    4.023295] device-mapper: ioctl: error adding target to table
> [    4.111046] BTRFS: device label terrasnapper devid 1 transid 28 /dev/dm-21
> [    4.188863] BTRFS: device label persistent devid 1 transid 11 /dev/dm-23
> [    4.196150] BTRFS: device label terramain devid 1 transid 35 /dev/dm-22
> ===============================
>
> There are 3 hard drives, all platter SATA.
>
> dm-21 and dm-22 are NOT using LVM.  (If that is what is being referred
> to by "table: 253:21" and "table: 253:22".) They are each a btrfs
> filesystem, on a md RAID 1, of 3 hard drive partitions.  They're each
> separate boot partitions, for separate installs.
>
> All other volumes in the system are btrfs RAID's on LVM volumes, not
> using md arrays.  (Except swap is an md array of raw LVM volumes.)
>
> dmsetup table shows (253:21 and 253:22 are NOT on here)
>
> ===============================
> disk3-disk3thin: 0 1048576000 linear 253:6 0
> disk3-terramain3: 0 209715200 thin 253:6 1
> disk3-disk3thin-tpool: 0 1048576000 thin-pool 253:4 253:5 512 0 0
> disk3-disk3thin_tdata: 0 1048576000 linear 8:35 33818624
> disk2-terraswap2: 0 33554432 linear 8:19 2048
> disk1-disk1thin-tpool: 0 1048576000 thin-pool 253:17 253:18 512 0 0
> disk1-disk1thin_tdata: 0 1048576000 linear 8:3 33818624
> disk3-disk3thin_tmeta: 0 262144 linear 8:35 1082394624
> disk1-disk1thin_tmeta: 0 262144 linear 8:3 1082394624
> disk2-disk2thin: 0 1048576000 linear 253:7 0
> disk3-terrasnapper3: 0 209715200 thin 253:6 2
> disk2-persistent2: 0 209715200 thin 253:7 3
> disk3-terraswap3: 0 33554432 linear 8:35 2048
> disk1-disk1thin: 0 1048576000 linear 253:19 0
> disk2-terrasnapper2: 0 209715200 thin 253:7 2
> disk1-terramain1: 0 209715200 thin 253:19 1
> disk2-disk2thin-tpool: 0 1048576000 thin-pool 253:2 253:3 512 0 0
> disk2-disk2thin_tdata: 0 1048576000 linear 8:19 33818624
> disk1-persistent1: 0 209715200 thin 253:19 3
> disk2-disk2thin_tmeta: 0 262144 linear 8:19 1082394624
> disk1-terrasnapper1: 0 209715200 thin 253:19 2
> disk2-terramain2: 0 209715200 thin 253:7 1
> disk3-persistent3: 0 209715200 thin 253:6 3
> disk1-terraswap1: 0 33554432 linear 8:3 2048
> ===============================
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

Re: During systemd/udev, device-mapper trying to work with non-LVM volumes

[james harvey]

[-- Attachment #1: Type: text/plain, Size: 2882 bytes --]

On Wed, Jul 27, 2016 at 2:49 PM, Marian Csontos <mcsontos@redhat.com> wrote:
> On 07/23/2016 01:14 AM, james harvey wrote:
>>
>> If I understand what's going on here, I think device-mapper is trying
>> to work with two volumes that don't involve LVM, causing the errors.
>
>
> If I understand correctly, these volumes DO involve LVM.
>
> It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on top
> of LVM.

I do have some BTRFS volumes on top of LVM, including my 2 root
volumes, but my 2 boot partitions don't involve LVM.  They're raw disk
partitions - MD RAID 1 - BTRFS.

The kernel error references "table: 253:21" and "table: 253:22".
These entries are not referred to by running dmsetup.  If these
correspond to dm-21 and dm-22, those are the boot volumes that don't
involve LVM at all.

> Using BTRFS with thin-shapshots is not a good idea, especially, if you have
> multiple snapshots of btrfs' underlying device active.
>
> Why are you usingn BTRFS on top of thin-pool?
> BTRFS does have snapshots and IMHO you should pick either BTRFS or
> thin-pool.

I'm not using thin-snapshots, just the thin-provisioning feature.  Is
running BTRFS in that scenario still a bad situation?  Why's that?
I'm going to be using a lot of virtual machines, which is my main
reason for wanting thin-provisioning.

I'm only using btrfs snapshots.

>> Is this a device-mapper bug?  A udev bug?  Something I have configured
>> wrong?
>
> Hard to say...
>
> Which distribution?
> Kernel, lvm version?

Sorry for not mentioning.  Arch, kernel 4.6.4, lvm 2.02.161, device
mapper 1.02.131, thin-pool 1.18.0

> Ideally run `lvmdump -m` and post output, please.

The number of kernel errors during boot that I'm getting seems to be
random.  (Probably some type of race condition?)  My original post
happened to be that it was using the ones not using LVM, but sometimes
it's doing it on LVM backed volumes too.  Occasionally it gives no
kernel errors.

On this boot, I have these errors:

==========
[    3.319387] device-mapper: table: 253:5: thin: Unable to activate
thin device while pool is suspended
[    3.394258] device-mapper: table: 253:6: thin: Unable to activate
thin device while pool is suspended
[    3.632259] device-mapper: table: 253:13: thin: Unable to activate
thin device while pool is suspended
[    3.698752] device-mapper: table: 253:14: thin: Unable to activate
thin device while pool is suspended
[    4.045282] device-mapper: table: 253:21: thin: Unable to activate
thin device while pool is suspended
[    4.117778] device-mapper: table: 253:22: thin: Unable to activate
thin device while pool is suspended
==========

I've attached lvmdump-terra-snapper-2016072803258.tgz created during
the boot that gave these 6 errors.

I've also attached lvmdump-terra-snapper-2016072812433.tgz created
during a different boot, that only errors on 253:13 and 253:21.

[-- Attachment #2: lvmdump-terra-snapper-2016072803258.tgz --]
[-- Type: application/x-gzip, Size: 61537 bytes --]

[-- Attachment #3: lvmdump-terra-snapper-2016072812433.tgz --]
[-- Type: application/x-gzip, Size: 61534 bytes --]

[-- Attachment #4: Type: text/plain, Size: 0 bytes --]

Re: During systemd/udev, device-mapper trying to work with non-LVM volumes

[Marian Csontos]

On 07/28/2016 03:33 AM, james harvey wrote:
> On Wed, Jul 27, 2016 at 2:49 PM, Marian Csontos <mcsontos@redhat.com> wrote:
>> On 07/23/2016 01:14 AM, james harvey wrote:
>>>
>>> If I understand what's going on here, I think device-mapper is trying
>>> to work with two volumes that don't involve LVM, causing the errors.
>>
>>
>> If I understand correctly, these volumes DO involve LVM.
>>
>> It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on top
>> of LVM.
>
> I do have some BTRFS volumes on top of LVM, including my 2 root
> volumes, but my 2 boot partitions don't involve LVM.  They're raw disk
> partitions - MD RAID 1 - BTRFS.
>
> The kernel error references "table: 253:21" and "table: 253:22".
> These entries are not referred to by running dmsetup.  If these
> correspond to dm-21 and dm-22, those are the boot volumes that don't
> involve LVM at all.
>
>> Using BTRFS with thin-shapshots is not a good idea, especially, if you have
>> multiple snapshots of btrfs' underlying device active.
>>
>> Why are you usingn BTRFS on top of thin-pool?
>> BTRFS does have snapshots and IMHO you should pick either BTRFS or
>> thin-pool.
>
> I'm not using thin-snapshots, just the thin-provisioning feature.  Is
> running BTRFS in that scenario still a bad situation?  Why's that?
> I'm going to be using a lot of virtual machines, which is my main
> reason for wanting thin-provisioning.
>
> I'm only using btrfs snapshots.
>
>>> Is this a device-mapper bug?  A udev bug?  Something I have configured
>>> wrong?
>>
>> Hard to say...
>>
>> Which distribution?
>> Kernel, lvm version?
>
> Sorry for not mentioning.  Arch, kernel 4.6.4, lvm 2.02.161, device
> mapper 1.02.131, thin-pool 1.18.0
>
>> Ideally run `lvmdump -m` and post output, please.
>
> The number of kernel errors during boot that I'm getting seems to be
> random.  (Probably some type of race condition?)  My original post
> happened to be that it was using the ones not using LVM, but sometimes
> it's doing it on LVM backed volumes too.  Occasionally it gives no
> kernel errors.
>
> On this boot, I have these errors:
>
> ==========
> [    3.319387] device-mapper: table: 253:5: thin: Unable to activate
> thin device while pool is suspended
> [    3.394258] device-mapper: table: 253:6: thin: Unable to activate
> thin device while pool is suspended
> [    3.632259] device-mapper: table: 253:13: thin: Unable to activate
> thin device while pool is suspended
> [    3.698752] device-mapper: table: 253:14: thin: Unable to activate
> thin device while pool is suspended
> [    4.045282] device-mapper: table: 253:21: thin: Unable to activate
> thin device while pool is suspended
> [    4.117778] device-mapper: table: 253:22: thin: Unable to activate
> thin device while pool is suspended
> ==========
>
> I've attached lvmdump-terra-snapper-2016072803258.tgz created during
> the boot that gave these 6 errors.

James, there is nothing in the logs suggesting any of the devices is not 
DM device.

*_boot devices are "md" devices, (9:{0,1,2})

These are all thin LVs (/disk({1,2,3})-(main|snapper)\1/):

For example 253:5 and 6:

Name                  Maj Min Stat Open Targ Event  UUID 

disk3-main3           253   6 L--w    0    1      0 
LVM-OHvenodGhFiZBRegTTkz6jt26MhoNxmCEdGScUBBGM5Sk6922y7HeBL0OJfOMO4v
disk3-snapper3        253   5 L--w    1    1      0 
LVM-OHvenodGhFiZBRegTTkz6jt26MhoNxmCZ3VcsAUU8fynWzcVnr81EJxpJh3Z8IE9


This looks like an issue with locking, where activation of thin-volume 
is attempted and gets past through locks while while the pool is 
suspended, or that there is something else in the system suspending the 
pool.

This should not normally happen, any chance "locking_type" in 
/etc/lvm/lvm.conf your initramdisk is set to 0 or 5?

May be full journal would help us to see more...

Also these seems to have no effect on running system, as I see in the 
dmsetup outputs all the affected LVs were activated afterwards, or have 
you activated them manually?

-- Martian

>
> I've also attached lvmdump-terra-snapper-2016072812433.tgz created
> during a different boot, that only errors on 253:13 and 253:21.



>
>
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

Re: During systemd/udev, device-mapper trying to work with non-LVM volumes

[Zdenek Kabelac]

Dne 28.7.2016 v 03:33 james harvey napsal(a):
> On Wed, Jul 27, 2016 at 2:49 PM, Marian Csontos <mcsontos@redhat.com> wrote:
>> On 07/23/2016 01:14 AM, james harvey wrote:
>>>
>>> If I understand what's going on here, I think device-mapper is trying
>>> to work with two volumes that don't involve LVM, causing the errors.
>>
>>
>> If I understand correctly, these volumes DO involve LVM.
>>
>> It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on top
>> of LVM.
>
> I do have some BTRFS volumes on top of LVM, including my 2 root
> volumes, but my 2 boot partitions don't involve LVM.  They're raw disk
> partitions - MD RAID 1 - BTRFS.
>
> The kernel error references "table: 253:21" and "table: 253:22".
> These entries are not referred to by running dmsetup.  If these
> correspond to dm-21 and dm-22, those are the boot volumes that don't
> involve LVM at all.

This doesn't make much sense.

253:XX are all DM devices -  few lines above you say   boot partitions are 
'raw disks'  now you say dm-21 & dm-22  are boot volumes ??

LVM is volume manager - LV is DM device (maintained by lvm2 command)
There is nothing like  lvm2 device - it's always 'dm' device.

lvm2  dm device has   LVM-   prefix in UUID

In your 'dmsetup into -c' output all DM device have this prefix - so
all your DM device are lvm2  maintained devices.

>
>> Using BTRFS with thin-shapshots is not a good idea, especially, if you have
>> multiple snapshots of btrfs' underlying device active.
>>
>> Why are you usingn BTRFS on top of thin-pool?
>> BTRFS does have snapshots and IMHO you should pick either BTRFS or
>> thin-pool.
>
> I'm not using thin-snapshots, just the thin-provisioning feature.  Is

Again doesn't make sense...


> running BTRFS in that scenario still a bad situation?  Why's that?
> I'm going to be using a lot of virtual machines, which is my main
> reason for wanting thin-provisioning.

HOWTO....

>
> I'm only using btrfs snapshots.
>
>>> Is this a device-mapper bug?  A udev bug?  Something I have configured
>>> wrong?

Seems like 99.99999% wrong configuration....

>>
>> Which distribution?
>> Kernel, lvm version?
>
> Sorry for not mentioning.  Arch, kernel 4.6.4, lvm 2.02.161, device
> mapper 1.02.131, thin-pool 1.18.0
>
>> Ideally run `lvmdump -m` and post output, please.
>
> The number of kernel errors during boot that I'm getting seems to be
> random.  (Probably some type of race condition?)  My original post
> happened to be that it was using the ones not using LVM, but sometimes
> it's doing it on LVM backed volumes too.  Occasionally it gives no
> kernel errors.
>
> On this boot, I have these errors:
>
> ==========
> [    3.319387] device-mapper: table: 253:5: thin: Unable to activate
> thin device while pool is suspended
> [    3.394258] device-mapper: table: 253:6: thin: Unable to activate
> thin device while pool is suspended
> [    3.632259] device-mapper: table: 253:13: thin: Unable to activate
> thin device while pool is suspended
> [    3.698752] device-mapper: table: 253:14: thin: Unable to activate
> thin device while pool is suspended
> [    4.045282] device-mapper: table: 253:21: thin: Unable to activate
> thin device while pool is suspended
> [    4.117778] device-mapper: table: 253:22: thin: Unable to activate
> thin device while pool is suspended
> ==========
>


Completely confused about this - are you trying to operate with thin devices 
yourself with some 'dmsetup' commands ?   Eventually using 'docker' ?
Or maybe you haven configured lockless lvm2, where volumes are activated with 
locking_type==0  ?

Lvm surely doesn't try to activate thinLV from a suspended thin-pool ?

So you really need to expose sequence of command you try to execute - we do 
not have have crystal ball to reverse engineer your wrongly issued commands 
out of kernel error messages -  i.e. is it some 'lvchange/vgchange' producing 
it - then take  '-vvvv' trace out of those commands.

Also -  why do you even mix  btrfs with mdadm & lvm2??

btrfs has it's own solution for raid as well as for volume management.

Combining  'btrfs' and lvm2 snapshot is basically a 'weapon of mass 
destruction'  since  btrfs  has no idea which disk to use when multiple same 
devices with same signature appears in the system.

I'd strongly recommend to read some doc first to get familiar with basic 
bricks of your device stack.

The usage presented in tgz doesn't look like a proper use-case for lvm2 at 
all, and rather a misuse based on misunderstanding how all these technologies 
do work.

Regards

Zdenek

Re: During systemd/udev, device-mapper trying to work with non-LVM volumes

[james harvey]

Before I respond to all the other questions, lets clear up the
underlying misunderstanding here.

Maybe I'm missing something, but I think most/all of the responders
are thinking I'm using LVM snapshots.  If I am missing something,
saying "doesn't make sense" or "howto" doesn't really help.  Although
I may not be using LVM in the same way most people use it, I don't see
what instructions I'm violating.  I am NEVER EVER planning on running
LVM snapshots.  I'm NEVER EVER going to run "lvcreate -s".  I know you
can't make block level copies of BTRFS volumes that it can see at
once.  I have NEVER EVER ran "{l,v}gchange".

I want to have thin provisioning with KVM virtual machines.  I'm never
going to use docker, and I'm never going to use snapper on an LVM
snapshot basis, only having it use BTRFS snapshots.

Can't someone use thinly-provisioned LOGICAL volumes, and NEVER EVER
use thinly-provisioned SNAPSHOT volumes?  I've repeatedly said I'm not
using thin-provisioned snapshots.

Isn't the problem running BTRFS on LVM2 when a LVM2-snapshot is made,
that BTRFS gets confused by duplicate signatures?

So, if a user is NEVER EVER going to use LVM2 snapshots, isn't that OK?

/dev/sd{a,b,c}1 3.5G Linux RAID - to be used as /dev/md1 labeled main_boot
/dev/sd{a,b,c}2 3.5G Linux RAID - to be used as /dev/md2 labeled snapper_boot
/dev/sd{a,b,c}3 4.6T Linux LVM
# mdadm --create --name=main_boot --level 1 --metadata=1.0
--raid-devices=3 /dev/md1 /dev/sda1 /dev/sdb1 /dev/sdc1
# mkfs.btrfs --label main_boot /dev/disk/by-id/md-name-main_boot
# mdadm --create --name=snapper_boot --level 1 --metadata=1.0
--raid-devices=3 /dev/md2 /dev/sda2 /dev/sdb2 /dev/sdc2
# mkfs.btrfs --label snapper_boot /dev/disk/by-id/md-name-snapper_boot
# pvcreate /dev/disk/sda3
# vgcreate disk1 /dev/disk/sda3
# pvcreate /dev/disk/sdb3
# vgcreate disk2 /dev/disk/sdb3
# pvcreate /dev/disk/sdc3
# vgcreate disk3 /dev/disk/sdc3
# lvcreate --size 500G --thinpool disk1thin disk1
# lvcreate --size 500G --thinpool disk2thin disk2
# lvcreate --size 500G --thinpool disk3thin disk3
# lvcreate --virtualsize 100G --name main1 disk1/disk1thin
# lvcreate --virtualsize 100G --name main2 disk2/disk2thin
# lvcreate --virtualsize 100G --name main3 disk3/disk3thin
# mkfs.btrfs --label main --metadata raid1 --data raid1
/dev/disk1/main1 /dev/disk2/main2 /dev/disk3/main3

Then install to /dev/disk/by-label/main and using
/dev/disk/by-label/main_boot as its /boot?

Then only using btrfs subvolumes (snapshots) and NEVER EVER running
again ANY lv command having to do with main1/main2/main3?

On Thu, Jul 28, 2016 at 10:12 AM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
> Dne 28.7.2016 v 03:33 james harvey napsal(a):
>>
>> On Wed, Jul 27, 2016 at 2:49 PM, Marian Csontos <mcsontos@redhat.com>
>> wrote:
>>>
>>> On 07/23/2016 01:14 AM, james harvey wrote:
>>>>
>>>>
>>>> If I understand what's going on here, I think device-mapper is trying
>>>> to work with two volumes that don't involve LVM, causing the errors.
>>>
>>>
>>>
>>> If I understand correctly, these volumes DO involve LVM.
>>>
>>> It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on
>>> top
>>> of LVM.
>>
>>
>> I do have some BTRFS volumes on top of LVM, including my 2 root
>> volumes, but my 2 boot partitions don't involve LVM.  They're raw disk
>> partitions - MD RAID 1 - BTRFS.
>>
>> The kernel error references "table: 253:21" and "table: 253:22".
>> These entries are not referred to by running dmsetup.  If these
>> correspond to dm-21 and dm-22, those are the boot volumes that don't
>> involve LVM at all.
>
>
> This doesn't make much sense.
>
> 253:XX are all DM devices -  few lines above you say   boot partitions are
> 'raw disks'  now you say dm-21 & dm-22  are boot volumes ??
>
> LVM is volume manager - LV is DM device (maintained by lvm2 command)
> There is nothing like  lvm2 device - it's always 'dm' device.
>
> lvm2  dm device has   LVM-   prefix in UUID
>
> In your 'dmsetup into -c' output all DM device have this prefix - so
> all your DM device are lvm2  maintained devices.
>
>>
>>> Using BTRFS with thin-shapshots is not a good idea, especially, if you
>>> have
>>> multiple snapshots of btrfs' underlying device active.
>>>
>>> Why are you usingn BTRFS on top of thin-pool?
>>> BTRFS does have snapshots and IMHO you should pick either BTRFS or
>>> thin-pool.
>>
>>
>> I'm not using thin-snapshots, just the thin-provisioning feature.  Is
>
>
> Again doesn't make sense...

Can you say a little more on this?  Just saying it doesn't make sense
gives me no direction.  I understand why many people might like to use
thin-snapshots, but can't a user just use thin-provisioning and NEVER
EVER use the LVM thin-snapshots?

>> running BTRFS in that scenario still a bad situation?  Why's that?
>> I'm going to be using a lot of virtual machines, which is my main
>> reason for wanting thin-provisioning.
>
>
> HOWTO....

Again, just saying I'm doing it wrong doesn't help.

>
>>
>> I'm only using btrfs snapshots.
>>
>>>> Is this a device-mapper bug?  A udev bug?  Something I have configured
>>>> wrong?
>
>
> Seems like 99.99999% wrong configuration....
>
>
>>>
>>> Which distribution?
>>> Kernel, lvm version?
>>
>>
>> Sorry for not mentioning.  Arch, kernel 4.6.4, lvm 2.02.161, device
>> mapper 1.02.131, thin-pool 1.18.0
>>
>>> Ideally run `lvmdump -m` and post output, please.
>>
>>
>> The number of kernel errors during boot that I'm getting seems to be
>> random.  (Probably some type of race condition?)  My original post
>> happened to be that it was using the ones not using LVM, but sometimes
>> it's doing it on LVM backed volumes too.  Occasionally it gives no
>> kernel errors.
>>
>> On this boot, I have these errors:
>>
>> ==========
>> [    3.319387] device-mapper: table: 253:5: thin: Unable to activate
>> thin device while pool is suspended
>> [    3.394258] device-mapper: table: 253:6: thin: Unable to activate
>> thin device while pool is suspended
>> [    3.632259] device-mapper: table: 253:13: thin: Unable to activate
>> thin device while pool is suspended
>> [    3.698752] device-mapper: table: 253:14: thin: Unable to activate
>> thin device while pool is suspended
>> [    4.045282] device-mapper: table: 253:21: thin: Unable to activate
>> thin device while pool is suspended
>> [    4.117778] device-mapper: table: 253:22: thin: Unable to activate
>> thin device while pool is suspended
>> ==========
>>
>
>
> Completely confused about this - are you trying to operate with thin devices
> yourself with some 'dmsetup' commands ?   Eventually using 'docker' ?
> Or maybe you haven configured lockless lvm2, where volumes are activated
> with locking_type==0  ?
>
> Lvm surely doesn't try to activate thinLV from a suspended thin-pool ?
>
> So you really need to expose sequence of command you try to execute - we do
> not have have crystal ball to reverse engineer your wrongly issued commands
> out of kernel error messages -  i.e. is it some 'lvchange/vgchange'
> producing it - then take  '-vvvv' trace out of those commands.
>
> Also -  why do you even mix  btrfs with mdadm & lvm2??
>
> btrfs has it's own solution for raid as well as for volume management.
>
> Combining  'btrfs' and lvm2 snapshot is basically a 'weapon of mass
> destruction'  since  btrfs  has no idea which disk to use when multiple same
> devices with same signature appears in the system.
>
> I'd strongly recommend to read some doc first to get familiar with basic
> bricks of your device stack.
>
> The usage presented in tgz doesn't look like a proper use-case for lvm2 at
> all, and rather a misuse based on misunderstanding how all these
> technologies do work.
>
> Regards
>
> Zdenek
>

Re: During systemd/udev, device-mapper trying to work with non-LVM volumes

[james harvey]

> # pvcreate /dev/disk/sda3
> # vgcreate disk1 /dev/disk/sda3
> # pvcreate /dev/disk/sdb3
> # vgcreate disk2 /dev/disk/sdb3
> # pvcreate /dev/disk/sdc3
> # vgcreate disk3 /dev/disk/sdc3

... I'm actually using
/dev/disk/by-id/ata-TOSHIBA_MD04ACA500_<serialNumber> but shortened
for simplicity and missed removing the /disk/ portion.

Re: During systemd/udev, device-mapper trying to work with non-LVM volumes

[Zdenek Kabelac]

Dne 3.8.2016 v 04:11 james harvey napsal(a):
> Before I respond to all the other questions, lets clear up the
> underlying misunderstanding here.
>
> Maybe I'm missing something, but I think most/all of the responders
> are thinking I'm using LVM snapshots.  If I am missing something,
> saying "doesn't make sense" or "howto" doesn't really help.  Although
> I may not be using LVM in the same way most people use it, I don't see
> what instructions I'm violating.  I am NEVER EVER planning on running
> LVM snapshots.  I'm NEVER EVER going to run "lvcreate -s".  I know you
> can't make block level copies of BTRFS volumes that it can see at
> once.  I have NEVER EVER ran "{l,v}gchange".
>
> I want to have thin provisioning with KVM virtual machines.  I'm never
> going to use docker, and I'm never going to use snapper on an LVM
> snapshot basis, only having it use BTRFS snapshots.
>
> Can't someone use thinly-provisioned LOGICAL volumes, and NEVER EVER
> use thinly-provisioned SNAPSHOT volumes?  I've repeatedly said I'm not
> using thin-provisioned snapshots.
>
> Isn't the problem running BTRFS on LVM2 when a LVM2-snapshot is made,
> that BTRFS gets confused by duplicate signatures?
>
> So, if a user is NEVER EVER going to use LVM2 snapshots, isn't that OK?
>
> /dev/sd{a,b,c}1 3.5G Linux RAID - to be used as /dev/md1 labeled main_boot
> /dev/sd{a,b,c}2 3.5G Linux RAID - to be used as /dev/md2 labeled snapper_boot
> /dev/sd{a,b,c}3 4.6T Linux LVM
> # mdadm --create --name=main_boot --level 1 --metadata=1.0
> --raid-devices=3 /dev/md1 /dev/sda1 /dev/sdb1 /dev/sdc1
> # mkfs.btrfs --label main_boot /dev/disk/by-id/md-name-main_boot
> # mdadm --create --name=snapper_boot --level 1 --metadata=1.0
> --raid-devices=3 /dev/md2 /dev/sda2 /dev/sdb2 /dev/sdc2
> # mkfs.btrfs --label snapper_boot /dev/disk/by-id/md-name-snapper_boot
> # pvcreate /dev/disk/sda3
> # vgcreate disk1 /dev/disk/sda3
> # pvcreate /dev/disk/sdb3
> # vgcreate disk2 /dev/disk/sdb3
> # pvcreate /dev/disk/sdc3
> # vgcreate disk3 /dev/disk/sdc3
> # lvcreate --size 500G --thinpool disk1thin disk1
> # lvcreate --size 500G --thinpool disk2thin disk2
> # lvcreate --size 500G --thinpool disk3thin disk3
> # lvcreate --virtualsize 100G --name main1 disk1/disk1thin
> # lvcreate --virtualsize 100G --name main2 disk2/disk2thin
> # lvcreate --virtualsize 100G --name main3 disk3/disk3thin
> # mkfs.btrfs --label main --metadata raid1 --data raid1
> /dev/disk1/main1 /dev/disk2/main2 /dev/disk3/main3
>
> Then install to /dev/disk/by-label/main and using
> /dev/disk/by-label/main_boot as its /boot?
>
> Then only using btrfs subvolumes (snapshots) and NEVER EVER running
> again ANY lv command having to do with main1/main2/main3?
>

Hi


These are the steps you want to use:

mdadm   whateverarrayyouwantouse   ->  mdXXX

vgcreate  VG  /dev/mdXXX

lvcreate -L1500G -T VG/pool
lvcreate -V300G  -n disk VG/pool

mkfs.btrfs --label main  /dev/VG/disk

---

It's unsupported/unadvised  to combine  your BTRFS volume from  LVs from 
different VG - you would need to be real expect to make all the activation 
running properly.

Handling autoexpansion of 3 separate pools used for single filesystem is 
something untested & unsupported...


---

Also note - there is no 'extra'  thin-snapshot.
Thin snapshot is  just ordinary  thin volume - no difference.
Thin volumes just have some mappings..

---

If you are not going to run  'lvcreate -s' - why you talk about snapshots?
Your emails are simply still too confusing...

Regards


Zdenek



> On Thu, Jul 28, 2016 at 10:12 AM, Zdenek Kabelac <zkabelac@redhat.com> wrote:
>> Dne 28.7.2016 v 03:33 james harvey napsal(a):
>>>
>>> On Wed, Jul 27, 2016 at 2:49 PM, Marian Csontos <mcsontos@redhat.com>
>>> wrote:
>>>>
>>>> On 07/23/2016 01:14 AM, james harvey wrote:
>>>>>
>>>>>
>>>>> If I understand what's going on here, I think device-mapper is trying
>>>>> to work with two volumes that don't involve LVM, causing the errors.
>>>>
>>>>
>>>>
>>>> If I understand correctly, these volumes DO involve LVM.
>>>>
>>>> It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on
>>>> top
>>>> of LVM.
>>>
>>>
>>> I do have some BTRFS volumes on top of LVM, including my 2 root
>>> volumes, but my 2 boot partitions don't involve LVM.  They're raw disk
>>> partitions - MD RAID 1 - BTRFS.
>>>
>>> The kernel error references "table: 253:21" and "table: 253:22".
>>> These entries are not referred to by running dmsetup.  If these
>>> correspond to dm-21 and dm-22, those are the boot volumes that don't
>>> involve LVM at all.
>>
>>
>> This doesn't make much sense.
>>
>> 253:XX are all DM devices -  few lines above you say   boot partitions are
>> 'raw disks'  now you say dm-21 & dm-22  are boot volumes ??
>>
>> LVM is volume manager - LV is DM device (maintained by lvm2 command)
>> There is nothing like  lvm2 device - it's always 'dm' device.
>>
>> lvm2  dm device has   LVM-   prefix in UUID
>>
>> In your 'dmsetup into -c' output all DM device have this prefix - so
>> all your DM device are lvm2  maintained devices.
>>
>>>
>>>> Using BTRFS with thin-shapshots is not a good idea, especially, if you
>>>> have
>>>> multiple snapshots of btrfs' underlying device active.
>>>>
>>>> Why are you usingn BTRFS on top of thin-pool?
>>>> BTRFS does have snapshots and IMHO you should pick either BTRFS or
>>>> thin-pool.
>>>
>>>
>>> I'm not using thin-snapshots, just the thin-provisioning feature.  Is
>>
>>
>> Again doesn't make sense...
>
> Can you say a little more on this?  Just saying it doesn't make sense
> gives me no direction.  I understand why many people might like to use
> thin-snapshots, but can't a user just use thin-provisioning and NEVER
> EVER use the LVM thin-snapshots?
>
>>> running BTRFS in that scenario still a bad situation?  Why's that?
>>> I'm going to be using a lot of virtual machines, which is my main
>>> reason for wanting thin-provisioning.
>>
>>
>> HOWTO....
>
> Again, just saying I'm doing it wrong doesn't help.
>
>>
>>>
>>> I'm only using btrfs snapshots.
>>>
>>>>> Is this a device-mapper bug?  A udev bug?  Something I have configured
>>>>> wrong?
>>
>>
>> Seems like 99.99999% wrong configuration....
>>
>>
>>>>
>>>> Which distribution?
>>>> Kernel, lvm version?
>>>
>>>
>>> Sorry for not mentioning.  Arch, kernel 4.6.4, lvm 2.02.161, device
>>> mapper 1.02.131, thin-pool 1.18.0
>>>
>>>> Ideally run `lvmdump -m` and post output, please.
>>>
>>>
>>> The number of kernel errors during boot that I'm getting seems to be
>>> random.  (Probably some type of race condition?)  My original post
>>> happened to be that it was using the ones not using LVM, but sometimes
>>> it's doing it on LVM backed volumes too.  Occasionally it gives no
>>> kernel errors.
>>>
>>> On this boot, I have these errors:
>>>
>>> ==========
>>> [    3.319387] device-mapper: table: 253:5: thin: Unable to activate
>>> thin device while pool is suspended
>>> [    3.394258] device-mapper: table: 253:6: thin: Unable to activate
>>> thin device while pool is suspended
>>> [    3.632259] device-mapper: table: 253:13: thin: Unable to activate
>>> thin device while pool is suspended
>>> [    3.698752] device-mapper: table: 253:14: thin: Unable to activate
>>> thin device while pool is suspended
>>> [    4.045282] device-mapper: table: 253:21: thin: Unable to activate
>>> thin device while pool is suspended
>>> [    4.117778] device-mapper: table: 253:22: thin: Unable to activate
>>> thin device while pool is suspended
>>> ==========
>>>
>>
>>
>> Completely confused about this - are you trying to operate with thin devices
>> yourself with some 'dmsetup' commands ?   Eventually using 'docker' ?
>> Or maybe you haven configured lockless lvm2, where volumes are activated
>> with locking_type==0  ?
>>
>> Lvm surely doesn't try to activate thinLV from a suspended thin-pool ?
>>
>> So you really need to expose sequence of command you try to execute - we do
>> not have have crystal ball to reverse engineer your wrongly issued commands
>> out of kernel error messages -  i.e. is it some 'lvchange/vgchange'
>> producing it - then take  '-vvvv' trace out of those commands.
>>
>> Also -  why do you even mix  btrfs with mdadm & lvm2??
>>
>> btrfs has it's own solution for raid as well as for volume management.
>>
>> Combining  'btrfs' and lvm2 snapshot is basically a 'weapon of mass
>> destruction'  since  btrfs  has no idea which disk to use when multiple same
>> devices with same signature appears in the system.
>>
>> I'd strongly recommend to read some doc first to get familiar with basic
>> bricks of your device stack.
>>
>> The usage presented in tgz doesn't look like a proper use-case for lvm2 at
>> all, and rather a misuse based on misunderstanding how all these
>> technologies do work.
>>
>> Regards
>>
>> Zdenek
>>
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

Re: During systemd/udev, device-mapper trying to work with non-LVM volumes

[james harvey]

[-- Attachment #1: Type: text/plain, Size: 6586 bytes --]

Ok, we can put the BTRFS issues aside.  I've reproduced using EXT4.

Below are close to the minimal steps I can use to create the problem.
If I remove some of the lvcreates that aren't being used, the problem
seems to happen less often.  So I'm not sure if that has an affect on
the race condition, or if I'm just seeing variance, so I've left those
in there even though they don't appear to do anything.

This appears to be a race condition that occurs while "udevadm trigger
--action=add --type=devices" is running.  (Editing the systemd udev
hook, inserting sleep's between udevadm calls shows this - without the
sleeps it shows during "udevadm settle" is running, since the trigger
call is running in the background.)  It happens very often, but not
always.

No idea if that means it's a udevadm (systemd) bug incorrectly giving
device-mapper something, or a kernel device-mapper bug doing something
wrong.

Up to date Arch, linux 4.6.4, lvm2 2.02.162, systemd 231.

In /etc/lvm/lvm.conf "locking_type = 1"

I haven't been activating anything manually.  Looks like everything
needing to be activated is somehow, but possibly things are trying to
be activated that aren't involved with thin volumes or even LVM.

I just noticed "dmsetup ls" and "dmsetup table" are showing different
253:X numbers.  I was only using "dmsetup table" in my original post.
I don't know which version of the numbers are showing during the
initramfs.

If the error corresponds to the "dmsetup ls" numbers, than the error
in this current boot refers to "disk2-disk2thin_tdata" and
"disk3-persistent3", which is ext4. Over an mdadm array, the error
occurs, and on a single lv it doesn't occur.

If the error corresponds to the "dmsetup table" numbers, then the
error refers to a device not in the table as well as possibly
"disk2-disk2thin-tpool".

Attached is another "lvmdump -m" file.  And a full dmesg (6MB, using
systemd log level debug, etc) is available at:
https://github.com/systemd/systemd/files/403118/udevError.shutdown-log.txt

If during the install I skip making the mdadm boot array and just use
a single partition, this doesn't happen.

The error on this run is:
:: Triggering uevents...
[ 3.106579] device-mapper: table: 253:7: thin: Unable to
activate thin device while pool is suspended
[ 3.290255] device-mapper: table: 253:17: thin: Unable to
activate thin device while pool is suspended
. . .

==============================

$ sudo dmsetup ls
disk3-disk3thin (253:15)
disk2-main2 (253:10)
disk3-disk3thin-tpool (253:14)
disk3-disk3thin_tdata (253:13)
disk1-disk1thin-tpool (253:2)
disk1-disk1thin_tdata (253:1)
disk3-disk3thin_tmeta (253:12)
disk3-main3 (253:16)
disk2-disk2thin (253:9)
disk1-disk1thin_tmeta (253:0)
disk2-persistent2 (253:11)
disk1-disk1thin (253:3)
disk2-disk2thin-tpool (253:8)
disk2-disk2thin_tdata (253:7)
disk1-persistent1 (253:5)
disk2-disk2thin_tmeta (253:6)
disk1-main1 (253:4)
disk3-persistent3 (253:17)

==============================

$ sudo dmsetup table

disk3-disk3thin: 0 1048576000 linear 253:14 0
disk2-main2: 0 209715200 thin 253:8 1
disk3-disk3thin-tpool: 0 1048576000 thin-pool 253:12 253:13 512 0 0
disk3-disk3thin_tdata: 0 1048576000 linear 8:35 264192
disk1-disk1thin-tpool: 0 1048576000 thin-pool 253:0 253:1 512 0 0
disk1-disk1thin_tdata: 0 1048576000 linear 8:3 264192
disk3-disk3thin_tmeta: 0 262144 linear 8:35 1048840192
disk3-main3: 0 209715200 thin 253:14 1
disk2-disk2thin: 0 1048576000 linear 253:8 0
disk1-disk1thin_tmeta: 0 262144 linear 8:3 1048840192
disk2-persistent2: 0 209715200 thin 253:8 2
disk1-disk1thin: 0 1048576000 linear 253:2 0
disk2-disk2thin-tpool: 0 1048576000 thin-pool 253:6 253:7 512 0 0
disk2-disk2thin_tdata: 0 1048576000 linear 8:19 264192
disk1-persistent1: 0 209715200 thin 253:2 2
disk2-disk2thin_tmeta: 0 262144 linear 8:19 1048840192
disk1-main1: 0 209715200 thin 253:2 1
disk3-persistent3: 0 209715200 thin 253:14 2

==============================

$ sudo lvmdump -m
Creating dump directory: /root/lvmdump-terra-2016080534140

Gathering LVM & device-mapper version info...
Gathering dmsetup info...
Gathering process info...
Gathering console messages...
Gathering /etc/lvm info...
Gathering /dev listing...
Gathering /sys/block listing...
Gathering LVM metadata from Physical Volumes...
/dev/sda3
/dev/sdb3
/dev/sdc3
Creating report tarball in /root/lvmdump-terra-2016080534140.tgz...

==============================

/dev/sd{a,b,c}1 3.5G Linux RAID
/dev/sd{a,b,c}2 3.5G Linux RAID
/dev/sd{a,b,c}3 4.5T Linux LVM

{ Setup LVM and filesystems }

({ This causes the issue })
# mdadm --create --verbose --name=main_boot --homehost="none" --level
1 --metadata 1.0 --raid-devices=3 /dev/md1 /dev/sda1 /dev/sdb1
/dev/sdc1
# mkfs.ext4 -F -L main_boot /dev/disk/by-id/md-name-main_boot

({ This appears to prevent the issue })
# mkfs.ext4 -F -L main_boot /dev/sda

({ Then, either way, continuing })
# pvcreate --yes --pvmetadatacopies 2 /dev/sda3
# vgcreate disk1 /dev/sda3
# pvcreate --yes --pvmetadatacopies 2 /dev/sdb3
# vgcreate disk2 /dev/sdb3
# pvcreate --yes --pvmetadatacopies 2 /dev/sdc3
# vgcreate disk3 /dev/sdc3
# lvcreate --size 500G --thinpool disk1thin disk1
# lvcreate --size 500G --thinpool disk2thin disk2
# lvcreate --size 500G --thinpool disk3thin disk3
# lvcreate --virtualsize 100G --name main1 disk1/disk1thin
# lvcreate --virtualsize 100G --name main2 disk2/disk2thin
# lvcreate --virtualsize 100G --name main3 disk3/disk3thin
# mkfs.ext4 -L main /dev/disk1/main1
# mount /dev/disk1/main1 /mnt
# mkdir /mnt/boot
# mount /dev/dev/sda1 /mnt/boot
# lvcreate --virtualsize 100G --name persistent1 disk1/disk1thin
# lvcreate --virtualsize 100G --name persistent2 disk2/disk2thin
# lvcreate --virtualsize 100G --name persistent3 disk3/disk3thin

{ Install Arch Linux}
# vi /etc/pacman.d/mirrorlist
# pacstrap -i base syslinux gptfdisk lvm2
# arch-chroot /mnt
# vi /etc/locale.gen
# locale-gen
# locale > /etc/locale.conf

({ If using the mdadm to cause the error })
# mdadm --detail --scan > /etc/mdadm.conf

({ Then, either way, continuing })
# vi /etc/nsswitch.conf
# systemd enable systemd-resolved systemd-networkd
# ln -s /usr/share/zoneinfo/America/Detroit /etc/localtime
# hwclock --utc --systohc
# passwd
{ Add lvm2 between block and filesystems}
# vi /etc/mkinitcpio.conf
# mkinitcpio -p linux
# echo hostname > /etc/hostname
# vi /etc/systemd/network/enp31s0.network
# syslinux-install_update -i -a -m
# vi /boot/syslinux/syslinux.cfg
{{ After exiting the arch-chroot }}
# ln -f -s /run/systemd/resolve/resolv.conf /mnt/etc/resolv.conf

{ Reboot }

==============================

[-- Attachment #2: lvmdump-terra-2016080534140.tgz --]
[-- Type: application/x-gzip, Size: 61102 bytes --]

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: During systemd/udev, device-mapper trying to work with non-LVM volumes

[james harvey]

Systemd says the LVM/DM developers install their own udev rules and
this is a LVM/DM issue, not a systemd issue.  See
https://github.com/systemd/systemd/issues/3899

On Thu, Aug 4, 2016 at 11:53 PM, james harvey <jamespharvey20@gmail.com> wrote:
> Ok, we can put the BTRFS issues aside.  I've reproduced using EXT4.
>
> Below are close to the minimal steps I can use to create the problem.
> If I remove some of the lvcreates that aren't being used, the problem
> seems to happen less often.  So I'm not sure if that has an affect on
> the race condition, or if I'm just seeing variance, so I've left those
> in there even though they don't appear to do anything.
>
> This appears to be a race condition that occurs while "udevadm trigger
> --action=add --type=devices" is running.  (Editing the systemd udev
> hook, inserting sleep's between udevadm calls shows this - without the
> sleeps it shows during "udevadm settle" is running, since the trigger
> call is running in the background.)  It happens very often, but not
> always.
>
> No idea if that means it's a udevadm (systemd) bug incorrectly giving
> device-mapper something, or a kernel device-mapper bug doing something
> wrong.
>
> Up to date Arch, linux 4.6.4, lvm2 2.02.162, systemd 231.
>
> In /etc/lvm/lvm.conf "locking_type = 1"
>
> I haven't been activating anything manually.  Looks like everything
> needing to be activated is somehow, but possibly things are trying to
> be activated that aren't involved with thin volumes or even LVM.
>
> I just noticed "dmsetup ls" and "dmsetup table" are showing different
> 253:X numbers.  I was only using "dmsetup table" in my original post.
> I don't know which version of the numbers are showing during the
> initramfs.
>
> If the error corresponds to the "dmsetup ls" numbers, than the error
> in this current boot refers to "disk2-disk2thin_tdata" and
> "disk3-persistent3", which is ext4. Over an mdadm array, the error
> occurs, and on a single lv it doesn't occur.
>
> If the error corresponds to the "dmsetup table" numbers, then the
> error refers to a device not in the table as well as possibly
> "disk2-disk2thin-tpool".
>
> Attached is another "lvmdump -m" file.  And a full dmesg (6MB, using
> systemd log level debug, etc) is available at:
> https://github.com/systemd/systemd/files/403118/udevError.shutdown-log.txt
>
> If during the install I skip making the mdadm boot array and just use
> a single partition, this doesn't happen.
>
> The error on this run is:
> :: Triggering uevents...
> [ 3.106579] device-mapper: table: 253:7: thin: Unable to
> activate thin device while pool is suspended
> [ 3.290255] device-mapper: table: 253:17: thin: Unable to
> activate thin device while pool is suspended
> . . .
>
> ==============================
>
> $ sudo dmsetup ls
> disk3-disk3thin (253:15)
> disk2-main2 (253:10)
> disk3-disk3thin-tpool (253:14)
> disk3-disk3thin_tdata (253:13)
> disk1-disk1thin-tpool (253:2)
> disk1-disk1thin_tdata (253:1)
> disk3-disk3thin_tmeta (253:12)
> disk3-main3 (253:16)
> disk2-disk2thin (253:9)
> disk1-disk1thin_tmeta (253:0)
> disk2-persistent2 (253:11)
> disk1-disk1thin (253:3)
> disk2-disk2thin-tpool (253:8)
> disk2-disk2thin_tdata (253:7)
> disk1-persistent1 (253:5)
> disk2-disk2thin_tmeta (253:6)
> disk1-main1 (253:4)
> disk3-persistent3 (253:17)
>
> ==============================
>
> $ sudo dmsetup table
>
> disk3-disk3thin: 0 1048576000 linear 253:14 0
> disk2-main2: 0 209715200 thin 253:8 1
> disk3-disk3thin-tpool: 0 1048576000 thin-pool 253:12 253:13 512 0 0
> disk3-disk3thin_tdata: 0 1048576000 linear 8:35 264192
> disk1-disk1thin-tpool: 0 1048576000 thin-pool 253:0 253:1 512 0 0
> disk1-disk1thin_tdata: 0 1048576000 linear 8:3 264192
> disk3-disk3thin_tmeta: 0 262144 linear 8:35 1048840192
> disk3-main3: 0 209715200 thin 253:14 1
> disk2-disk2thin: 0 1048576000 linear 253:8 0
> disk1-disk1thin_tmeta: 0 262144 linear 8:3 1048840192
> disk2-persistent2: 0 209715200 thin 253:8 2
> disk1-disk1thin: 0 1048576000 linear 253:2 0
> disk2-disk2thin-tpool: 0 1048576000 thin-pool 253:6 253:7 512 0 0
> disk2-disk2thin_tdata: 0 1048576000 linear 8:19 264192
> disk1-persistent1: 0 209715200 thin 253:2 2
> disk2-disk2thin_tmeta: 0 262144 linear 8:19 1048840192
> disk1-main1: 0 209715200 thin 253:2 1
> disk3-persistent3: 0 209715200 thin 253:14 2
>
> ==============================
>
> $ sudo lvmdump -m
> Creating dump directory: /root/lvmdump-terra-2016080534140
>
> Gathering LVM & device-mapper version info...
> Gathering dmsetup info...
> Gathering process info...
> Gathering console messages...
> Gathering /etc/lvm info...
> Gathering /dev listing...
> Gathering /sys/block listing...
> Gathering LVM metadata from Physical Volumes...
> /dev/sda3
> /dev/sdb3
> /dev/sdc3
> Creating report tarball in /root/lvmdump-terra-2016080534140.tgz...
>
> ==============================
>
> /dev/sd{a,b,c}1 3.5G Linux RAID
> /dev/sd{a,b,c}2 3.5G Linux RAID
> /dev/sd{a,b,c}3 4.5T Linux LVM
>
> { Setup LVM and filesystems }
>
> ({ This causes the issue })
> # mdadm --create --verbose --name=main_boot --homehost="none" --level
> 1 --metadata 1.0 --raid-devices=3 /dev/md1 /dev/sda1 /dev/sdb1
> /dev/sdc1
> # mkfs.ext4 -F -L main_boot /dev/disk/by-id/md-name-main_boot
>
> ({ This appears to prevent the issue })
> # mkfs.ext4 -F -L main_boot /dev/sda
>
> ({ Then, either way, continuing })
> # pvcreate --yes --pvmetadatacopies 2 /dev/sda3
> # vgcreate disk1 /dev/sda3
> # pvcreate --yes --pvmetadatacopies 2 /dev/sdb3
> # vgcreate disk2 /dev/sdb3
> # pvcreate --yes --pvmetadatacopies 2 /dev/sdc3
> # vgcreate disk3 /dev/sdc3
> # lvcreate --size 500G --thinpool disk1thin disk1
> # lvcreate --size 500G --thinpool disk2thin disk2
> # lvcreate --size 500G --thinpool disk3thin disk3
> # lvcreate --virtualsize 100G --name main1 disk1/disk1thin
> # lvcreate --virtualsize 100G --name main2 disk2/disk2thin
> # lvcreate --virtualsize 100G --name main3 disk3/disk3thin
> # mkfs.ext4 -L main /dev/disk1/main1
> # mount /dev/disk1/main1 /mnt
> # mkdir /mnt/boot
> # mount /dev/dev/sda1 /mnt/boot
> # lvcreate --virtualsize 100G --name persistent1 disk1/disk1thin
> # lvcreate --virtualsize 100G --name persistent2 disk2/disk2thin
> # lvcreate --virtualsize 100G --name persistent3 disk3/disk3thin
>
> { Install Arch Linux}
> # vi /etc/pacman.d/mirrorlist
> # pacstrap -i base syslinux gptfdisk lvm2
> # arch-chroot /mnt
> # vi /etc/locale.gen
> # locale-gen
> # locale > /etc/locale.conf
>
> ({ If using the mdadm to cause the error })
> # mdadm --detail --scan > /etc/mdadm.conf
>
> ({ Then, either way, continuing })
> # vi /etc/nsswitch.conf
> # systemd enable systemd-resolved systemd-networkd
> # ln -s /usr/share/zoneinfo/America/Detroit /etc/localtime
> # hwclock --utc --systohc
> # passwd
> { Add lvm2 between block and filesystems}
> # vi /etc/mkinitcpio.conf
> # mkinitcpio -p linux
> # echo hostname > /etc/hostname
> # vi /etc/systemd/network/enp31s0.network
> # syslinux-install_update -i -a -m
> # vi /boot/syslinux/syslinux.cfg
> {{ After exiting the arch-chroot }}
> # ln -f -s /run/systemd/resolve/resolv.conf /mnt/etc/resolv.conf
>
> { Reboot }
>
> ==============================