* During systemd/udev, device-mapper trying to work with non-LVM volumes
@ 2016-07-22 23:14 james harvey
2016-07-27 18:49 ` Marian Csontos
0 siblings, 1 reply; 10+ messages in thread
From: james harvey @ 2016-07-22 23:14 UTC (permalink / raw)
To: dm-devel
If I understand what's going on here, I think device-mapper is trying
to work with two volumes that don't involve LVM, causing the errors.
Is this a device-mapper bug? A udev bug? Something I have configured
wrong?
During boot, I see the following messages:
===============================
starting version 230
:: running early hook [lvm2]
:: running hook [udev]
:: Triggering uevents...
[ 3.964973] device-mapper: table: 253:21: thin: Unable to
activate thin device while pool is suspended
[ 4.023198] device-mapper: table: 253:22: thin: Unable to
activate thin device while pool is suspended
. . .
===============================
In dmesg, I see:
===============================
[ 3.313426] Btrfs loaded
[ 3.313987] BTRFS: device label terrasnapper_boot devid 1 transid 13 /dev/md2
[ 3.314219] BTRFS: device label terramain_boot devid 1 transid 27 /dev/md1
[ 3.391995] device-mapper: thin: Data device (dm-5) discard
unsupported: Disabling discard passdown.
[ 3.392803] device-mapper: thin: Data device (dm-3) discard
unsupported: Disabling discard passdown.
[ 3.520545] BTRFS: device label terrasnapper devid 3 transid 28 /dev/dm-13
[ 3.520931] BTRFS: device label terramain devid 2 transid 35 /dev/dm-11
[ 3.574622] BTRFS: device label persistent devid 3 transid 11 /dev/dm-14
[ 3.590508] BTRFS: device label terrasnapper devid 2 transid 28 /dev/dm-12
[ 3.604053] BTRFS: device label terramain devid 3 transid 35 /dev/dm-10
[ 3.711390] BTRFS: device label persistent devid 2 transid 11 /dev/dm-15
[ 3.788923] md: bind<dm-16>
[ 3.790860] md/raid1:md0: active with 3 out of 3 mirrors
[ 3.790909] md0: detected capacity change from 0 to 17163091968
[ 3.961862] device-mapper: thin: Data device (dm-18) discard
unsupported: Disabling discard passdown.
[ 3.964973] device-mapper: table: 253:21: thin: Unable to activate
thin device while pool is suspended
[ 3.965075] device-mapper: ioctl: error adding target to table
[ 4.023198] device-mapper: table: 253:22: thin: Unable to activate
thin device while pool is suspended
[ 4.023295] device-mapper: ioctl: error adding target to table
[ 4.111046] BTRFS: device label terrasnapper devid 1 transid 28 /dev/dm-21
[ 4.188863] BTRFS: device label persistent devid 1 transid 11 /dev/dm-23
[ 4.196150] BTRFS: device label terramain devid 1 transid 35 /dev/dm-22
===============================
There are 3 hard drives, all platter SATA.
dm-21 and dm-22 are NOT using LVM. (If that is what is being referred
to by "table: 253:21" and "table: 253:22".) They are each a btrfs
filesystem, on a md RAID 1, of 3 hard drive partitions. They're each
separate boot partitions, for separate installs.
All other volumes in the system are btrfs RAID's on LVM volumes, not
using md arrays. (Except swap is an md array of raw LVM volumes.)
dmsetup table shows (253:21 and 253:22 are NOT on here)
===============================
disk3-disk3thin: 0 1048576000 linear 253:6 0
disk3-terramain3: 0 209715200 thin 253:6 1
disk3-disk3thin-tpool: 0 1048576000 thin-pool 253:4 253:5 512 0 0
disk3-disk3thin_tdata: 0 1048576000 linear 8:35 33818624
disk2-terraswap2: 0 33554432 linear 8:19 2048
disk1-disk1thin-tpool: 0 1048576000 thin-pool 253:17 253:18 512 0 0
disk1-disk1thin_tdata: 0 1048576000 linear 8:3 33818624
disk3-disk3thin_tmeta: 0 262144 linear 8:35 1082394624
disk1-disk1thin_tmeta: 0 262144 linear 8:3 1082394624
disk2-disk2thin: 0 1048576000 linear 253:7 0
disk3-terrasnapper3: 0 209715200 thin 253:6 2
disk2-persistent2: 0 209715200 thin 253:7 3
disk3-terraswap3: 0 33554432 linear 8:35 2048
disk1-disk1thin: 0 1048576000 linear 253:19 0
disk2-terrasnapper2: 0 209715200 thin 253:7 2
disk1-terramain1: 0 209715200 thin 253:19 1
disk2-disk2thin-tpool: 0 1048576000 thin-pool 253:2 253:3 512 0 0
disk2-disk2thin_tdata: 0 1048576000 linear 8:19 33818624
disk1-persistent1: 0 209715200 thin 253:19 3
disk2-disk2thin_tmeta: 0 262144 linear 8:19 1082394624
disk1-terrasnapper1: 0 209715200 thin 253:19 2
disk2-terramain2: 0 209715200 thin 253:7 1
disk3-persistent3: 0 209715200 thin 253:6 3
disk1-terraswap1: 0 33554432 linear 8:3 2048
===============================
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: During systemd/udev, device-mapper trying to work with non-LVM volumes 2016-07-22 23:14 During systemd/udev, device-mapper trying to work with non-LVM volumes james harvey @ 2016-07-27 18:49 ` Marian Csontos 2016-07-28 1:33 ` james harvey 0 siblings, 1 reply; 10+ messages in thread From: Marian Csontos @ 2016-07-27 18:49 UTC (permalink / raw) To: james harvey, dm-devel On 07/23/2016 01:14 AM, james harvey wrote: > If I understand what's going on here, I think device-mapper is trying > to work with two volumes that don't involve LVM, causing the errors. If I understand correctly, these volumes DO involve LVM. It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on top of LVM. Using BTRFS with thin-shapshots is not a good idea, especially, if you have multiple snapshots of btrfs' underlying device active. Why are you usingn BTRFS on top of thin-pool? BTRFS does have snapshots and IMHO you should pick either BTRFS or thin-pool. > Is this a device-mapper bug? A udev bug? Something I have configured > wrong? Hard to say... Which distribution? Kernel, lvm version? Ideally run `lvmdump -m` and post output, please. > > During boot, I see the following messages: > > =============================== > starting version 230 > :: running early hook [lvm2] > :: running hook [udev] > :: Triggering uevents... > [ 3.964973] device-mapper: table: 253:21: thin: Unable to > activate thin device while pool is suspended > [ 4.023198] device-mapper: table: 253:22: thin: Unable to > activate thin device while pool is suspended > . . . > =============================== > > In dmesg, I see: > > =============================== > [ 3.313426] Btrfs loaded > [ 3.313987] BTRFS: device label terrasnapper_boot devid 1 transid 13 /dev/md2 > [ 3.314219] BTRFS: device label terramain_boot devid 1 transid 27 /dev/md1 > [ 3.391995] device-mapper: thin: Data device (dm-5) discard > unsupported: Disabling discard passdown. > [ 3.392803] device-mapper: thin: Data device (dm-3) discard > unsupported: Disabling discard passdown. > [ 3.520545] BTRFS: device label terrasnapper devid 3 transid 28 /dev/dm-13 > [ 3.520931] BTRFS: device label terramain devid 2 transid 35 /dev/dm-11 > [ 3.574622] BTRFS: device label persistent devid 3 transid 11 /dev/dm-14 > [ 3.590508] BTRFS: device label terrasnapper devid 2 transid 28 /dev/dm-12 > [ 3.604053] BTRFS: device label terramain devid 3 transid 35 /dev/dm-10 > [ 3.711390] BTRFS: device label persistent devid 2 transid 11 /dev/dm-15 > [ 3.788923] md: bind<dm-16> > [ 3.790860] md/raid1:md0: active with 3 out of 3 mirrors > [ 3.790909] md0: detected capacity change from 0 to 17163091968 > [ 3.961862] device-mapper: thin: Data device (dm-18) discard > unsupported: Disabling discard passdown. > [ 3.964973] device-mapper: table: 253:21: thin: Unable to activate > thin device while pool is suspended > [ 3.965075] device-mapper: ioctl: error adding target to table > [ 4.023198] device-mapper: table: 253:22: thin: Unable to activate > thin device while pool is suspended > [ 4.023295] device-mapper: ioctl: error adding target to table > [ 4.111046] BTRFS: device label terrasnapper devid 1 transid 28 /dev/dm-21 > [ 4.188863] BTRFS: device label persistent devid 1 transid 11 /dev/dm-23 > [ 4.196150] BTRFS: device label terramain devid 1 transid 35 /dev/dm-22 > =============================== > > There are 3 hard drives, all platter SATA. > > dm-21 and dm-22 are NOT using LVM. (If that is what is being referred > to by "table: 253:21" and "table: 253:22".) They are each a btrfs > filesystem, on a md RAID 1, of 3 hard drive partitions. They're each > separate boot partitions, for separate installs. > > All other volumes in the system are btrfs RAID's on LVM volumes, not > using md arrays. (Except swap is an md array of raw LVM volumes.) > > dmsetup table shows (253:21 and 253:22 are NOT on here) > > =============================== > disk3-disk3thin: 0 1048576000 linear 253:6 0 > disk3-terramain3: 0 209715200 thin 253:6 1 > disk3-disk3thin-tpool: 0 1048576000 thin-pool 253:4 253:5 512 0 0 > disk3-disk3thin_tdata: 0 1048576000 linear 8:35 33818624 > disk2-terraswap2: 0 33554432 linear 8:19 2048 > disk1-disk1thin-tpool: 0 1048576000 thin-pool 253:17 253:18 512 0 0 > disk1-disk1thin_tdata: 0 1048576000 linear 8:3 33818624 > disk3-disk3thin_tmeta: 0 262144 linear 8:35 1082394624 > disk1-disk1thin_tmeta: 0 262144 linear 8:3 1082394624 > disk2-disk2thin: 0 1048576000 linear 253:7 0 > disk3-terrasnapper3: 0 209715200 thin 253:6 2 > disk2-persistent2: 0 209715200 thin 253:7 3 > disk3-terraswap3: 0 33554432 linear 8:35 2048 > disk1-disk1thin: 0 1048576000 linear 253:19 0 > disk2-terrasnapper2: 0 209715200 thin 253:7 2 > disk1-terramain1: 0 209715200 thin 253:19 1 > disk2-disk2thin-tpool: 0 1048576000 thin-pool 253:2 253:3 512 0 0 > disk2-disk2thin_tdata: 0 1048576000 linear 8:19 33818624 > disk1-persistent1: 0 209715200 thin 253:19 3 > disk2-disk2thin_tmeta: 0 262144 linear 8:19 1082394624 > disk1-terrasnapper1: 0 209715200 thin 253:19 2 > disk2-terramain2: 0 209715200 thin 253:7 1 > disk3-persistent3: 0 209715200 thin 253:6 3 > disk1-terraswap1: 0 33554432 linear 8:3 2048 > =============================== > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: During systemd/udev, device-mapper trying to work with non-LVM volumes 2016-07-27 18:49 ` Marian Csontos @ 2016-07-28 1:33 ` james harvey 2016-07-28 13:13 ` Marian Csontos 2016-07-28 14:12 ` Zdenek Kabelac 0 siblings, 2 replies; 10+ messages in thread From: james harvey @ 2016-07-28 1:33 UTC (permalink / raw) To: Marian Csontos, dm-devel [-- Attachment #1: Type: text/plain, Size: 2882 bytes --] On Wed, Jul 27, 2016 at 2:49 PM, Marian Csontos <mcsontos@redhat.com> wrote: > On 07/23/2016 01:14 AM, james harvey wrote: >> >> If I understand what's going on here, I think device-mapper is trying >> to work with two volumes that don't involve LVM, causing the errors. > > > If I understand correctly, these volumes DO involve LVM. > > It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on top > of LVM. I do have some BTRFS volumes on top of LVM, including my 2 root volumes, but my 2 boot partitions don't involve LVM. They're raw disk partitions - MD RAID 1 - BTRFS. The kernel error references "table: 253:21" and "table: 253:22". These entries are not referred to by running dmsetup. If these correspond to dm-21 and dm-22, those are the boot volumes that don't involve LVM at all. > Using BTRFS with thin-shapshots is not a good idea, especially, if you have > multiple snapshots of btrfs' underlying device active. > > Why are you usingn BTRFS on top of thin-pool? > BTRFS does have snapshots and IMHO you should pick either BTRFS or > thin-pool. I'm not using thin-snapshots, just the thin-provisioning feature. Is running BTRFS in that scenario still a bad situation? Why's that? I'm going to be using a lot of virtual machines, which is my main reason for wanting thin-provisioning. I'm only using btrfs snapshots. >> Is this a device-mapper bug? A udev bug? Something I have configured >> wrong? > > Hard to say... > > Which distribution? > Kernel, lvm version? Sorry for not mentioning. Arch, kernel 4.6.4, lvm 2.02.161, device mapper 1.02.131, thin-pool 1.18.0 > Ideally run `lvmdump -m` and post output, please. The number of kernel errors during boot that I'm getting seems to be random. (Probably some type of race condition?) My original post happened to be that it was using the ones not using LVM, but sometimes it's doing it on LVM backed volumes too. Occasionally it gives no kernel errors. On this boot, I have these errors: ========== [ 3.319387] device-mapper: table: 253:5: thin: Unable to activate thin device while pool is suspended [ 3.394258] device-mapper: table: 253:6: thin: Unable to activate thin device while pool is suspended [ 3.632259] device-mapper: table: 253:13: thin: Unable to activate thin device while pool is suspended [ 3.698752] device-mapper: table: 253:14: thin: Unable to activate thin device while pool is suspended [ 4.045282] device-mapper: table: 253:21: thin: Unable to activate thin device while pool is suspended [ 4.117778] device-mapper: table: 253:22: thin: Unable to activate thin device while pool is suspended ========== I've attached lvmdump-terra-snapper-2016072803258.tgz created during the boot that gave these 6 errors. I've also attached lvmdump-terra-snapper-2016072812433.tgz created during a different boot, that only errors on 253:13 and 253:21. [-- Attachment #2: lvmdump-terra-snapper-2016072803258.tgz --] [-- Type: application/x-gzip, Size: 61537 bytes --] [-- Attachment #3: lvmdump-terra-snapper-2016072812433.tgz --] [-- Type: application/x-gzip, Size: 61534 bytes --] [-- Attachment #4: Type: text/plain, Size: 0 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: During systemd/udev, device-mapper trying to work with non-LVM volumes 2016-07-28 1:33 ` james harvey @ 2016-07-28 13:13 ` Marian Csontos 2016-07-28 14:12 ` Zdenek Kabelac 1 sibling, 0 replies; 10+ messages in thread From: Marian Csontos @ 2016-07-28 13:13 UTC (permalink / raw) To: james harvey, dm-devel On 07/28/2016 03:33 AM, james harvey wrote: > On Wed, Jul 27, 2016 at 2:49 PM, Marian Csontos <mcsontos@redhat.com> wrote: >> On 07/23/2016 01:14 AM, james harvey wrote: >>> >>> If I understand what's going on here, I think device-mapper is trying >>> to work with two volumes that don't involve LVM, causing the errors. >> >> >> If I understand correctly, these volumes DO involve LVM. >> >> It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on top >> of LVM. > > I do have some BTRFS volumes on top of LVM, including my 2 root > volumes, but my 2 boot partitions don't involve LVM. They're raw disk > partitions - MD RAID 1 - BTRFS. > > The kernel error references "table: 253:21" and "table: 253:22". > These entries are not referred to by running dmsetup. If these > correspond to dm-21 and dm-22, those are the boot volumes that don't > involve LVM at all. > >> Using BTRFS with thin-shapshots is not a good idea, especially, if you have >> multiple snapshots of btrfs' underlying device active. >> >> Why are you usingn BTRFS on top of thin-pool? >> BTRFS does have snapshots and IMHO you should pick either BTRFS or >> thin-pool. > > I'm not using thin-snapshots, just the thin-provisioning feature. Is > running BTRFS in that scenario still a bad situation? Why's that? > I'm going to be using a lot of virtual machines, which is my main > reason for wanting thin-provisioning. > > I'm only using btrfs snapshots. > >>> Is this a device-mapper bug? A udev bug? Something I have configured >>> wrong? >> >> Hard to say... >> >> Which distribution? >> Kernel, lvm version? > > Sorry for not mentioning. Arch, kernel 4.6.4, lvm 2.02.161, device > mapper 1.02.131, thin-pool 1.18.0 > >> Ideally run `lvmdump -m` and post output, please. > > The number of kernel errors during boot that I'm getting seems to be > random. (Probably some type of race condition?) My original post > happened to be that it was using the ones not using LVM, but sometimes > it's doing it on LVM backed volumes too. Occasionally it gives no > kernel errors. > > On this boot, I have these errors: > > ========== > [ 3.319387] device-mapper: table: 253:5: thin: Unable to activate > thin device while pool is suspended > [ 3.394258] device-mapper: table: 253:6: thin: Unable to activate > thin device while pool is suspended > [ 3.632259] device-mapper: table: 253:13: thin: Unable to activate > thin device while pool is suspended > [ 3.698752] device-mapper: table: 253:14: thin: Unable to activate > thin device while pool is suspended > [ 4.045282] device-mapper: table: 253:21: thin: Unable to activate > thin device while pool is suspended > [ 4.117778] device-mapper: table: 253:22: thin: Unable to activate > thin device while pool is suspended > ========== > > I've attached lvmdump-terra-snapper-2016072803258.tgz created during > the boot that gave these 6 errors. James, there is nothing in the logs suggesting any of the devices is not DM device. *_boot devices are "md" devices, (9:{0,1,2}) These are all thin LVs (/disk({1,2,3})-(main|snapper)\1/): For example 253:5 and 6: Name Maj Min Stat Open Targ Event UUID disk3-main3 253 6 L--w 0 1 0 LVM-OHvenodGhFiZBRegTTkz6jt26MhoNxmCEdGScUBBGM5Sk6922y7HeBL0OJfOMO4v disk3-snapper3 253 5 L--w 1 1 0 LVM-OHvenodGhFiZBRegTTkz6jt26MhoNxmCZ3VcsAUU8fynWzcVnr81EJxpJh3Z8IE9 This looks like an issue with locking, where activation of thin-volume is attempted and gets past through locks while while the pool is suspended, or that there is something else in the system suspending the pool. This should not normally happen, any chance "locking_type" in /etc/lvm/lvm.conf your initramdisk is set to 0 or 5? May be full journal would help us to see more... Also these seems to have no effect on running system, as I see in the dmsetup outputs all the affected LVs were activated afterwards, or have you activated them manually? -- Martian > > I've also attached lvmdump-terra-snapper-2016072812433.tgz created > during a different boot, that only errors on 253:13 and 253:21. > > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: During systemd/udev, device-mapper trying to work with non-LVM volumes 2016-07-28 1:33 ` james harvey 2016-07-28 13:13 ` Marian Csontos @ 2016-07-28 14:12 ` Zdenek Kabelac 2016-08-03 2:11 ` james harvey 1 sibling, 1 reply; 10+ messages in thread From: Zdenek Kabelac @ 2016-07-28 14:12 UTC (permalink / raw) To: james harvey, dm-devel Dne 28.7.2016 v 03:33 james harvey napsal(a): > On Wed, Jul 27, 2016 at 2:49 PM, Marian Csontos <mcsontos@redhat.com> wrote: >> On 07/23/2016 01:14 AM, james harvey wrote: >>> >>> If I understand what's going on here, I think device-mapper is trying >>> to work with two volumes that don't involve LVM, causing the errors. >> >> >> If I understand correctly, these volumes DO involve LVM. >> >> It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on top >> of LVM. > > I do have some BTRFS volumes on top of LVM, including my 2 root > volumes, but my 2 boot partitions don't involve LVM. They're raw disk > partitions - MD RAID 1 - BTRFS. > > The kernel error references "table: 253:21" and "table: 253:22". > These entries are not referred to by running dmsetup. If these > correspond to dm-21 and dm-22, those are the boot volumes that don't > involve LVM at all. This doesn't make much sense. 253:XX are all DM devices - few lines above you say boot partitions are 'raw disks' now you say dm-21 & dm-22 are boot volumes ?? LVM is volume manager - LV is DM device (maintained by lvm2 command) There is nothing like lvm2 device - it's always 'dm' device. lvm2 dm device has LVM- prefix in UUID In your 'dmsetup into -c' output all DM device have this prefix - so all your DM device are lvm2 maintained devices. > >> Using BTRFS with thin-shapshots is not a good idea, especially, if you have >> multiple snapshots of btrfs' underlying device active. >> >> Why are you usingn BTRFS on top of thin-pool? >> BTRFS does have snapshots and IMHO you should pick either BTRFS or >> thin-pool. > > I'm not using thin-snapshots, just the thin-provisioning feature. Is Again doesn't make sense... > running BTRFS in that scenario still a bad situation? Why's that? > I'm going to be using a lot of virtual machines, which is my main > reason for wanting thin-provisioning. HOWTO.... > > I'm only using btrfs snapshots. > >>> Is this a device-mapper bug? A udev bug? Something I have configured >>> wrong? Seems like 99.99999% wrong configuration.... >> >> Which distribution? >> Kernel, lvm version? > > Sorry for not mentioning. Arch, kernel 4.6.4, lvm 2.02.161, device > mapper 1.02.131, thin-pool 1.18.0 > >> Ideally run `lvmdump -m` and post output, please. > > The number of kernel errors during boot that I'm getting seems to be > random. (Probably some type of race condition?) My original post > happened to be that it was using the ones not using LVM, but sometimes > it's doing it on LVM backed volumes too. Occasionally it gives no > kernel errors. > > On this boot, I have these errors: > > ========== > [ 3.319387] device-mapper: table: 253:5: thin: Unable to activate > thin device while pool is suspended > [ 3.394258] device-mapper: table: 253:6: thin: Unable to activate > thin device while pool is suspended > [ 3.632259] device-mapper: table: 253:13: thin: Unable to activate > thin device while pool is suspended > [ 3.698752] device-mapper: table: 253:14: thin: Unable to activate > thin device while pool is suspended > [ 4.045282] device-mapper: table: 253:21: thin: Unable to activate > thin device while pool is suspended > [ 4.117778] device-mapper: table: 253:22: thin: Unable to activate > thin device while pool is suspended > ========== > Completely confused about this - are you trying to operate with thin devices yourself with some 'dmsetup' commands ? Eventually using 'docker' ? Or maybe you haven configured lockless lvm2, where volumes are activated with locking_type==0 ? Lvm surely doesn't try to activate thinLV from a suspended thin-pool ? So you really need to expose sequence of command you try to execute - we do not have have crystal ball to reverse engineer your wrongly issued commands out of kernel error messages - i.e. is it some 'lvchange/vgchange' producing it - then take '-vvvv' trace out of those commands. Also - why do you even mix btrfs with mdadm & lvm2?? btrfs has it's own solution for raid as well as for volume management. Combining 'btrfs' and lvm2 snapshot is basically a 'weapon of mass destruction' since btrfs has no idea which disk to use when multiple same devices with same signature appears in the system. I'd strongly recommend to read some doc first to get familiar with basic bricks of your device stack. The usage presented in tgz doesn't look like a proper use-case for lvm2 at all, and rather a misuse based on misunderstanding how all these technologies do work. Regards Zdenek ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: During systemd/udev, device-mapper trying to work with non-LVM volumes 2016-07-28 14:12 ` Zdenek Kabelac @ 2016-08-03 2:11 ` james harvey 2016-08-03 2:15 ` james harvey 2016-08-03 7:49 ` Zdenek Kabelac 0 siblings, 2 replies; 10+ messages in thread From: james harvey @ 2016-08-03 2:11 UTC (permalink / raw) To: Zdenek Kabelac, Marian Csontos; +Cc: dm-devel Before I respond to all the other questions, lets clear up the underlying misunderstanding here. Maybe I'm missing something, but I think most/all of the responders are thinking I'm using LVM snapshots. If I am missing something, saying "doesn't make sense" or "howto" doesn't really help. Although I may not be using LVM in the same way most people use it, I don't see what instructions I'm violating. I am NEVER EVER planning on running LVM snapshots. I'm NEVER EVER going to run "lvcreate -s". I know you can't make block level copies of BTRFS volumes that it can see at once. I have NEVER EVER ran "{l,v}gchange". I want to have thin provisioning with KVM virtual machines. I'm never going to use docker, and I'm never going to use snapper on an LVM snapshot basis, only having it use BTRFS snapshots. Can't someone use thinly-provisioned LOGICAL volumes, and NEVER EVER use thinly-provisioned SNAPSHOT volumes? I've repeatedly said I'm not using thin-provisioned snapshots. Isn't the problem running BTRFS on LVM2 when a LVM2-snapshot is made, that BTRFS gets confused by duplicate signatures? So, if a user is NEVER EVER going to use LVM2 snapshots, isn't that OK? /dev/sd{a,b,c}1 3.5G Linux RAID - to be used as /dev/md1 labeled main_boot /dev/sd{a,b,c}2 3.5G Linux RAID - to be used as /dev/md2 labeled snapper_boot /dev/sd{a,b,c}3 4.6T Linux LVM # mdadm --create --name=main_boot --level 1 --metadata=1.0 --raid-devices=3 /dev/md1 /dev/sda1 /dev/sdb1 /dev/sdc1 # mkfs.btrfs --label main_boot /dev/disk/by-id/md-name-main_boot # mdadm --create --name=snapper_boot --level 1 --metadata=1.0 --raid-devices=3 /dev/md2 /dev/sda2 /dev/sdb2 /dev/sdc2 # mkfs.btrfs --label snapper_boot /dev/disk/by-id/md-name-snapper_boot # pvcreate /dev/disk/sda3 # vgcreate disk1 /dev/disk/sda3 # pvcreate /dev/disk/sdb3 # vgcreate disk2 /dev/disk/sdb3 # pvcreate /dev/disk/sdc3 # vgcreate disk3 /dev/disk/sdc3 # lvcreate --size 500G --thinpool disk1thin disk1 # lvcreate --size 500G --thinpool disk2thin disk2 # lvcreate --size 500G --thinpool disk3thin disk3 # lvcreate --virtualsize 100G --name main1 disk1/disk1thin # lvcreate --virtualsize 100G --name main2 disk2/disk2thin # lvcreate --virtualsize 100G --name main3 disk3/disk3thin # mkfs.btrfs --label main --metadata raid1 --data raid1 /dev/disk1/main1 /dev/disk2/main2 /dev/disk3/main3 Then install to /dev/disk/by-label/main and using /dev/disk/by-label/main_boot as its /boot? Then only using btrfs subvolumes (snapshots) and NEVER EVER running again ANY lv command having to do with main1/main2/main3? On Thu, Jul 28, 2016 at 10:12 AM, Zdenek Kabelac <zkabelac@redhat.com> wrote: > Dne 28.7.2016 v 03:33 james harvey napsal(a): >> >> On Wed, Jul 27, 2016 at 2:49 PM, Marian Csontos <mcsontos@redhat.com> >> wrote: >>> >>> On 07/23/2016 01:14 AM, james harvey wrote: >>>> >>>> >>>> If I understand what's going on here, I think device-mapper is trying >>>> to work with two volumes that don't involve LVM, causing the errors. >>> >>> >>> >>> If I understand correctly, these volumes DO involve LVM. >>> >>> It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on >>> top >>> of LVM. >> >> >> I do have some BTRFS volumes on top of LVM, including my 2 root >> volumes, but my 2 boot partitions don't involve LVM. They're raw disk >> partitions - MD RAID 1 - BTRFS. >> >> The kernel error references "table: 253:21" and "table: 253:22". >> These entries are not referred to by running dmsetup. If these >> correspond to dm-21 and dm-22, those are the boot volumes that don't >> involve LVM at all. > > > This doesn't make much sense. > > 253:XX are all DM devices - few lines above you say boot partitions are > 'raw disks' now you say dm-21 & dm-22 are boot volumes ?? > > LVM is volume manager - LV is DM device (maintained by lvm2 command) > There is nothing like lvm2 device - it's always 'dm' device. > > lvm2 dm device has LVM- prefix in UUID > > In your 'dmsetup into -c' output all DM device have this prefix - so > all your DM device are lvm2 maintained devices. > >> >>> Using BTRFS with thin-shapshots is not a good idea, especially, if you >>> have >>> multiple snapshots of btrfs' underlying device active. >>> >>> Why are you usingn BTRFS on top of thin-pool? >>> BTRFS does have snapshots and IMHO you should pick either BTRFS or >>> thin-pool. >> >> >> I'm not using thin-snapshots, just the thin-provisioning feature. Is > > > Again doesn't make sense... Can you say a little more on this? Just saying it doesn't make sense gives me no direction. I understand why many people might like to use thin-snapshots, but can't a user just use thin-provisioning and NEVER EVER use the LVM thin-snapshots? >> running BTRFS in that scenario still a bad situation? Why's that? >> I'm going to be using a lot of virtual machines, which is my main >> reason for wanting thin-provisioning. > > > HOWTO.... Again, just saying I'm doing it wrong doesn't help. > >> >> I'm only using btrfs snapshots. >> >>>> Is this a device-mapper bug? A udev bug? Something I have configured >>>> wrong? > > > Seems like 99.99999% wrong configuration.... > > >>> >>> Which distribution? >>> Kernel, lvm version? >> >> >> Sorry for not mentioning. Arch, kernel 4.6.4, lvm 2.02.161, device >> mapper 1.02.131, thin-pool 1.18.0 >> >>> Ideally run `lvmdump -m` and post output, please. >> >> >> The number of kernel errors during boot that I'm getting seems to be >> random. (Probably some type of race condition?) My original post >> happened to be that it was using the ones not using LVM, but sometimes >> it's doing it on LVM backed volumes too. Occasionally it gives no >> kernel errors. >> >> On this boot, I have these errors: >> >> ========== >> [ 3.319387] device-mapper: table: 253:5: thin: Unable to activate >> thin device while pool is suspended >> [ 3.394258] device-mapper: table: 253:6: thin: Unable to activate >> thin device while pool is suspended >> [ 3.632259] device-mapper: table: 253:13: thin: Unable to activate >> thin device while pool is suspended >> [ 3.698752] device-mapper: table: 253:14: thin: Unable to activate >> thin device while pool is suspended >> [ 4.045282] device-mapper: table: 253:21: thin: Unable to activate >> thin device while pool is suspended >> [ 4.117778] device-mapper: table: 253:22: thin: Unable to activate >> thin device while pool is suspended >> ========== >> > > > Completely confused about this - are you trying to operate with thin devices > yourself with some 'dmsetup' commands ? Eventually using 'docker' ? > Or maybe you haven configured lockless lvm2, where volumes are activated > with locking_type==0 ? > > Lvm surely doesn't try to activate thinLV from a suspended thin-pool ? > > So you really need to expose sequence of command you try to execute - we do > not have have crystal ball to reverse engineer your wrongly issued commands > out of kernel error messages - i.e. is it some 'lvchange/vgchange' > producing it - then take '-vvvv' trace out of those commands. > > Also - why do you even mix btrfs with mdadm & lvm2?? > > btrfs has it's own solution for raid as well as for volume management. > > Combining 'btrfs' and lvm2 snapshot is basically a 'weapon of mass > destruction' since btrfs has no idea which disk to use when multiple same > devices with same signature appears in the system. > > I'd strongly recommend to read some doc first to get familiar with basic > bricks of your device stack. > > The usage presented in tgz doesn't look like a proper use-case for lvm2 at > all, and rather a misuse based on misunderstanding how all these > technologies do work. > > Regards > > Zdenek > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: During systemd/udev, device-mapper trying to work with non-LVM volumes 2016-08-03 2:11 ` james harvey @ 2016-08-03 2:15 ` james harvey 2016-08-03 7:49 ` Zdenek Kabelac 1 sibling, 0 replies; 10+ messages in thread From: james harvey @ 2016-08-03 2:15 UTC (permalink / raw) To: Zdenek Kabelac, Marian Csontos; +Cc: dm-devel > # pvcreate /dev/disk/sda3 > # vgcreate disk1 /dev/disk/sda3 > # pvcreate /dev/disk/sdb3 > # vgcreate disk2 /dev/disk/sdb3 > # pvcreate /dev/disk/sdc3 > # vgcreate disk3 /dev/disk/sdc3 ... I'm actually using /dev/disk/by-id/ata-TOSHIBA_MD04ACA500_<serialNumber> but shortened for simplicity and missed removing the /disk/ portion. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: During systemd/udev, device-mapper trying to work with non-LVM volumes 2016-08-03 2:11 ` james harvey 2016-08-03 2:15 ` james harvey @ 2016-08-03 7:49 ` Zdenek Kabelac 2016-08-05 3:53 ` james harvey 1 sibling, 1 reply; 10+ messages in thread From: Zdenek Kabelac @ 2016-08-03 7:49 UTC (permalink / raw) To: james harvey, Marian Csontos; +Cc: dm-devel Dne 3.8.2016 v 04:11 james harvey napsal(a): > Before I respond to all the other questions, lets clear up the > underlying misunderstanding here. > > Maybe I'm missing something, but I think most/all of the responders > are thinking I'm using LVM snapshots. If I am missing something, > saying "doesn't make sense" or "howto" doesn't really help. Although > I may not be using LVM in the same way most people use it, I don't see > what instructions I'm violating. I am NEVER EVER planning on running > LVM snapshots. I'm NEVER EVER going to run "lvcreate -s". I know you > can't make block level copies of BTRFS volumes that it can see at > once. I have NEVER EVER ran "{l,v}gchange". > > I want to have thin provisioning with KVM virtual machines. I'm never > going to use docker, and I'm never going to use snapper on an LVM > snapshot basis, only having it use BTRFS snapshots. > > Can't someone use thinly-provisioned LOGICAL volumes, and NEVER EVER > use thinly-provisioned SNAPSHOT volumes? I've repeatedly said I'm not > using thin-provisioned snapshots. > > Isn't the problem running BTRFS on LVM2 when a LVM2-snapshot is made, > that BTRFS gets confused by duplicate signatures? > > So, if a user is NEVER EVER going to use LVM2 snapshots, isn't that OK? > > /dev/sd{a,b,c}1 3.5G Linux RAID - to be used as /dev/md1 labeled main_boot > /dev/sd{a,b,c}2 3.5G Linux RAID - to be used as /dev/md2 labeled snapper_boot > /dev/sd{a,b,c}3 4.6T Linux LVM > # mdadm --create --name=main_boot --level 1 --metadata=1.0 > --raid-devices=3 /dev/md1 /dev/sda1 /dev/sdb1 /dev/sdc1 > # mkfs.btrfs --label main_boot /dev/disk/by-id/md-name-main_boot > # mdadm --create --name=snapper_boot --level 1 --metadata=1.0 > --raid-devices=3 /dev/md2 /dev/sda2 /dev/sdb2 /dev/sdc2 > # mkfs.btrfs --label snapper_boot /dev/disk/by-id/md-name-snapper_boot > # pvcreate /dev/disk/sda3 > # vgcreate disk1 /dev/disk/sda3 > # pvcreate /dev/disk/sdb3 > # vgcreate disk2 /dev/disk/sdb3 > # pvcreate /dev/disk/sdc3 > # vgcreate disk3 /dev/disk/sdc3 > # lvcreate --size 500G --thinpool disk1thin disk1 > # lvcreate --size 500G --thinpool disk2thin disk2 > # lvcreate --size 500G --thinpool disk3thin disk3 > # lvcreate --virtualsize 100G --name main1 disk1/disk1thin > # lvcreate --virtualsize 100G --name main2 disk2/disk2thin > # lvcreate --virtualsize 100G --name main3 disk3/disk3thin > # mkfs.btrfs --label main --metadata raid1 --data raid1 > /dev/disk1/main1 /dev/disk2/main2 /dev/disk3/main3 > > Then install to /dev/disk/by-label/main and using > /dev/disk/by-label/main_boot as its /boot? > > Then only using btrfs subvolumes (snapshots) and NEVER EVER running > again ANY lv command having to do with main1/main2/main3? > Hi These are the steps you want to use: mdadm whateverarrayyouwantouse -> mdXXX vgcreate VG /dev/mdXXX lvcreate -L1500G -T VG/pool lvcreate -V300G -n disk VG/pool mkfs.btrfs --label main /dev/VG/disk --- It's unsupported/unadvised to combine your BTRFS volume from LVs from different VG - you would need to be real expect to make all the activation running properly. Handling autoexpansion of 3 separate pools used for single filesystem is something untested & unsupported... --- Also note - there is no 'extra' thin-snapshot. Thin snapshot is just ordinary thin volume - no difference. Thin volumes just have some mappings.. --- If you are not going to run 'lvcreate -s' - why you talk about snapshots? Your emails are simply still too confusing... Regards Zdenek > On Thu, Jul 28, 2016 at 10:12 AM, Zdenek Kabelac <zkabelac@redhat.com> wrote: >> Dne 28.7.2016 v 03:33 james harvey napsal(a): >>> >>> On Wed, Jul 27, 2016 at 2:49 PM, Marian Csontos <mcsontos@redhat.com> >>> wrote: >>>> >>>> On 07/23/2016 01:14 AM, james harvey wrote: >>>>> >>>>> >>>>> If I understand what's going on here, I think device-mapper is trying >>>>> to work with two volumes that don't involve LVM, causing the errors. >>>> >>>> >>>> >>>> If I understand correctly, these volumes DO involve LVM. >>>> >>>> It is not LV on top of your BTRFS volumes, but your BTRFS volumes are on >>>> top >>>> of LVM. >>> >>> >>> I do have some BTRFS volumes on top of LVM, including my 2 root >>> volumes, but my 2 boot partitions don't involve LVM. They're raw disk >>> partitions - MD RAID 1 - BTRFS. >>> >>> The kernel error references "table: 253:21" and "table: 253:22". >>> These entries are not referred to by running dmsetup. If these >>> correspond to dm-21 and dm-22, those are the boot volumes that don't >>> involve LVM at all. >> >> >> This doesn't make much sense. >> >> 253:XX are all DM devices - few lines above you say boot partitions are >> 'raw disks' now you say dm-21 & dm-22 are boot volumes ?? >> >> LVM is volume manager - LV is DM device (maintained by lvm2 command) >> There is nothing like lvm2 device - it's always 'dm' device. >> >> lvm2 dm device has LVM- prefix in UUID >> >> In your 'dmsetup into -c' output all DM device have this prefix - so >> all your DM device are lvm2 maintained devices. >> >>> >>>> Using BTRFS with thin-shapshots is not a good idea, especially, if you >>>> have >>>> multiple snapshots of btrfs' underlying device active. >>>> >>>> Why are you usingn BTRFS on top of thin-pool? >>>> BTRFS does have snapshots and IMHO you should pick either BTRFS or >>>> thin-pool. >>> >>> >>> I'm not using thin-snapshots, just the thin-provisioning feature. Is >> >> >> Again doesn't make sense... > > Can you say a little more on this? Just saying it doesn't make sense > gives me no direction. I understand why many people might like to use > thin-snapshots, but can't a user just use thin-provisioning and NEVER > EVER use the LVM thin-snapshots? > >>> running BTRFS in that scenario still a bad situation? Why's that? >>> I'm going to be using a lot of virtual machines, which is my main >>> reason for wanting thin-provisioning. >> >> >> HOWTO.... > > Again, just saying I'm doing it wrong doesn't help. > >> >>> >>> I'm only using btrfs snapshots. >>> >>>>> Is this a device-mapper bug? A udev bug? Something I have configured >>>>> wrong? >> >> >> Seems like 99.99999% wrong configuration.... >> >> >>>> >>>> Which distribution? >>>> Kernel, lvm version? >>> >>> >>> Sorry for not mentioning. Arch, kernel 4.6.4, lvm 2.02.161, device >>> mapper 1.02.131, thin-pool 1.18.0 >>> >>>> Ideally run `lvmdump -m` and post output, please. >>> >>> >>> The number of kernel errors during boot that I'm getting seems to be >>> random. (Probably some type of race condition?) My original post >>> happened to be that it was using the ones not using LVM, but sometimes >>> it's doing it on LVM backed volumes too. Occasionally it gives no >>> kernel errors. >>> >>> On this boot, I have these errors: >>> >>> ========== >>> [ 3.319387] device-mapper: table: 253:5: thin: Unable to activate >>> thin device while pool is suspended >>> [ 3.394258] device-mapper: table: 253:6: thin: Unable to activate >>> thin device while pool is suspended >>> [ 3.632259] device-mapper: table: 253:13: thin: Unable to activate >>> thin device while pool is suspended >>> [ 3.698752] device-mapper: table: 253:14: thin: Unable to activate >>> thin device while pool is suspended >>> [ 4.045282] device-mapper: table: 253:21: thin: Unable to activate >>> thin device while pool is suspended >>> [ 4.117778] device-mapper: table: 253:22: thin: Unable to activate >>> thin device while pool is suspended >>> ========== >>> >> >> >> Completely confused about this - are you trying to operate with thin devices >> yourself with some 'dmsetup' commands ? Eventually using 'docker' ? >> Or maybe you haven configured lockless lvm2, where volumes are activated >> with locking_type==0 ? >> >> Lvm surely doesn't try to activate thinLV from a suspended thin-pool ? >> >> So you really need to expose sequence of command you try to execute - we do >> not have have crystal ball to reverse engineer your wrongly issued commands >> out of kernel error messages - i.e. is it some 'lvchange/vgchange' >> producing it - then take '-vvvv' trace out of those commands. >> >> Also - why do you even mix btrfs with mdadm & lvm2?? >> >> btrfs has it's own solution for raid as well as for volume management. >> >> Combining 'btrfs' and lvm2 snapshot is basically a 'weapon of mass >> destruction' since btrfs has no idea which disk to use when multiple same >> devices with same signature appears in the system. >> >> I'd strongly recommend to read some doc first to get familiar with basic >> bricks of your device stack. >> >> The usage presented in tgz doesn't look like a proper use-case for lvm2 at >> all, and rather a misuse based on misunderstanding how all these >> technologies do work. >> >> Regards >> >> Zdenek >> > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: During systemd/udev, device-mapper trying to work with non-LVM volumes 2016-08-03 7:49 ` Zdenek Kabelac @ 2016-08-05 3:53 ` james harvey 2016-08-05 9:27 ` james harvey 0 siblings, 1 reply; 10+ messages in thread From: james harvey @ 2016-08-05 3:53 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: dm-devel, Marian Csontos [-- Attachment #1: Type: text/plain, Size: 6586 bytes --] Ok, we can put the BTRFS issues aside. I've reproduced using EXT4. Below are close to the minimal steps I can use to create the problem. If I remove some of the lvcreates that aren't being used, the problem seems to happen less often. So I'm not sure if that has an affect on the race condition, or if I'm just seeing variance, so I've left those in there even though they don't appear to do anything. This appears to be a race condition that occurs while "udevadm trigger --action=add --type=devices" is running. (Editing the systemd udev hook, inserting sleep's between udevadm calls shows this - without the sleeps it shows during "udevadm settle" is running, since the trigger call is running in the background.) It happens very often, but not always. No idea if that means it's a udevadm (systemd) bug incorrectly giving device-mapper something, or a kernel device-mapper bug doing something wrong. Up to date Arch, linux 4.6.4, lvm2 2.02.162, systemd 231. In /etc/lvm/lvm.conf "locking_type = 1" I haven't been activating anything manually. Looks like everything needing to be activated is somehow, but possibly things are trying to be activated that aren't involved with thin volumes or even LVM. I just noticed "dmsetup ls" and "dmsetup table" are showing different 253:X numbers. I was only using "dmsetup table" in my original post. I don't know which version of the numbers are showing during the initramfs. If the error corresponds to the "dmsetup ls" numbers, than the error in this current boot refers to "disk2-disk2thin_tdata" and "disk3-persistent3", which is ext4. Over an mdadm array, the error occurs, and on a single lv it doesn't occur. If the error corresponds to the "dmsetup table" numbers, then the error refers to a device not in the table as well as possibly "disk2-disk2thin-tpool". Attached is another "lvmdump -m" file. And a full dmesg (6MB, using systemd log level debug, etc) is available at: https://github.com/systemd/systemd/files/403118/udevError.shutdown-log.txt If during the install I skip making the mdadm boot array and just use a single partition, this doesn't happen. The error on this run is: :: Triggering uevents... [ 3.106579] device-mapper: table: 253:7: thin: Unable to activate thin device while pool is suspended [ 3.290255] device-mapper: table: 253:17: thin: Unable to activate thin device while pool is suspended . . . ============================== $ sudo dmsetup ls disk3-disk3thin (253:15) disk2-main2 (253:10) disk3-disk3thin-tpool (253:14) disk3-disk3thin_tdata (253:13) disk1-disk1thin-tpool (253:2) disk1-disk1thin_tdata (253:1) disk3-disk3thin_tmeta (253:12) disk3-main3 (253:16) disk2-disk2thin (253:9) disk1-disk1thin_tmeta (253:0) disk2-persistent2 (253:11) disk1-disk1thin (253:3) disk2-disk2thin-tpool (253:8) disk2-disk2thin_tdata (253:7) disk1-persistent1 (253:5) disk2-disk2thin_tmeta (253:6) disk1-main1 (253:4) disk3-persistent3 (253:17) ============================== $ sudo dmsetup table disk3-disk3thin: 0 1048576000 linear 253:14 0 disk2-main2: 0 209715200 thin 253:8 1 disk3-disk3thin-tpool: 0 1048576000 thin-pool 253:12 253:13 512 0 0 disk3-disk3thin_tdata: 0 1048576000 linear 8:35 264192 disk1-disk1thin-tpool: 0 1048576000 thin-pool 253:0 253:1 512 0 0 disk1-disk1thin_tdata: 0 1048576000 linear 8:3 264192 disk3-disk3thin_tmeta: 0 262144 linear 8:35 1048840192 disk3-main3: 0 209715200 thin 253:14 1 disk2-disk2thin: 0 1048576000 linear 253:8 0 disk1-disk1thin_tmeta: 0 262144 linear 8:3 1048840192 disk2-persistent2: 0 209715200 thin 253:8 2 disk1-disk1thin: 0 1048576000 linear 253:2 0 disk2-disk2thin-tpool: 0 1048576000 thin-pool 253:6 253:7 512 0 0 disk2-disk2thin_tdata: 0 1048576000 linear 8:19 264192 disk1-persistent1: 0 209715200 thin 253:2 2 disk2-disk2thin_tmeta: 0 262144 linear 8:19 1048840192 disk1-main1: 0 209715200 thin 253:2 1 disk3-persistent3: 0 209715200 thin 253:14 2 ============================== $ sudo lvmdump -m Creating dump directory: /root/lvmdump-terra-2016080534140 Gathering LVM & device-mapper version info... Gathering dmsetup info... Gathering process info... Gathering console messages... Gathering /etc/lvm info... Gathering /dev listing... Gathering /sys/block listing... Gathering LVM metadata from Physical Volumes... /dev/sda3 /dev/sdb3 /dev/sdc3 Creating report tarball in /root/lvmdump-terra-2016080534140.tgz... ============================== /dev/sd{a,b,c}1 3.5G Linux RAID /dev/sd{a,b,c}2 3.5G Linux RAID /dev/sd{a,b,c}3 4.5T Linux LVM { Setup LVM and filesystems } ({ This causes the issue }) # mdadm --create --verbose --name=main_boot --homehost="none" --level 1 --metadata 1.0 --raid-devices=3 /dev/md1 /dev/sda1 /dev/sdb1 /dev/sdc1 # mkfs.ext4 -F -L main_boot /dev/disk/by-id/md-name-main_boot ({ This appears to prevent the issue }) # mkfs.ext4 -F -L main_boot /dev/sda ({ Then, either way, continuing }) # pvcreate --yes --pvmetadatacopies 2 /dev/sda3 # vgcreate disk1 /dev/sda3 # pvcreate --yes --pvmetadatacopies 2 /dev/sdb3 # vgcreate disk2 /dev/sdb3 # pvcreate --yes --pvmetadatacopies 2 /dev/sdc3 # vgcreate disk3 /dev/sdc3 # lvcreate --size 500G --thinpool disk1thin disk1 # lvcreate --size 500G --thinpool disk2thin disk2 # lvcreate --size 500G --thinpool disk3thin disk3 # lvcreate --virtualsize 100G --name main1 disk1/disk1thin # lvcreate --virtualsize 100G --name main2 disk2/disk2thin # lvcreate --virtualsize 100G --name main3 disk3/disk3thin # mkfs.ext4 -L main /dev/disk1/main1 # mount /dev/disk1/main1 /mnt # mkdir /mnt/boot # mount /dev/dev/sda1 /mnt/boot # lvcreate --virtualsize 100G --name persistent1 disk1/disk1thin # lvcreate --virtualsize 100G --name persistent2 disk2/disk2thin # lvcreate --virtualsize 100G --name persistent3 disk3/disk3thin { Install Arch Linux} # vi /etc/pacman.d/mirrorlist # pacstrap -i base syslinux gptfdisk lvm2 # arch-chroot /mnt # vi /etc/locale.gen # locale-gen # locale > /etc/locale.conf ({ If using the mdadm to cause the error }) # mdadm --detail --scan > /etc/mdadm.conf ({ Then, either way, continuing }) # vi /etc/nsswitch.conf # systemd enable systemd-resolved systemd-networkd # ln -s /usr/share/zoneinfo/America/Detroit /etc/localtime # hwclock --utc --systohc # passwd { Add lvm2 between block and filesystems} # vi /etc/mkinitcpio.conf # mkinitcpio -p linux # echo hostname > /etc/hostname # vi /etc/systemd/network/enp31s0.network # syslinux-install_update -i -a -m # vi /boot/syslinux/syslinux.cfg {{ After exiting the arch-chroot }} # ln -f -s /run/systemd/resolve/resolv.conf /mnt/etc/resolv.conf { Reboot } ============================== [-- Attachment #2: lvmdump-terra-2016080534140.tgz --] [-- Type: application/x-gzip, Size: 61102 bytes --] [-- Attachment #3: Type: text/plain, Size: 0 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: During systemd/udev, device-mapper trying to work with non-LVM volumes 2016-08-05 3:53 ` james harvey @ 2016-08-05 9:27 ` james harvey 0 siblings, 0 replies; 10+ messages in thread From: james harvey @ 2016-08-05 9:27 UTC (permalink / raw) To: Zdenek Kabelac; +Cc: dm-devel, Marian Csontos Systemd says the LVM/DM developers install their own udev rules and this is a LVM/DM issue, not a systemd issue. See https://github.com/systemd/systemd/issues/3899 On Thu, Aug 4, 2016 at 11:53 PM, james harvey <jamespharvey20@gmail.com> wrote: > Ok, we can put the BTRFS issues aside. I've reproduced using EXT4. > > Below are close to the minimal steps I can use to create the problem. > If I remove some of the lvcreates that aren't being used, the problem > seems to happen less often. So I'm not sure if that has an affect on > the race condition, or if I'm just seeing variance, so I've left those > in there even though they don't appear to do anything. > > This appears to be a race condition that occurs while "udevadm trigger > --action=add --type=devices" is running. (Editing the systemd udev > hook, inserting sleep's between udevadm calls shows this - without the > sleeps it shows during "udevadm settle" is running, since the trigger > call is running in the background.) It happens very often, but not > always. > > No idea if that means it's a udevadm (systemd) bug incorrectly giving > device-mapper something, or a kernel device-mapper bug doing something > wrong. > > Up to date Arch, linux 4.6.4, lvm2 2.02.162, systemd 231. > > In /etc/lvm/lvm.conf "locking_type = 1" > > I haven't been activating anything manually. Looks like everything > needing to be activated is somehow, but possibly things are trying to > be activated that aren't involved with thin volumes or even LVM. > > I just noticed "dmsetup ls" and "dmsetup table" are showing different > 253:X numbers. I was only using "dmsetup table" in my original post. > I don't know which version of the numbers are showing during the > initramfs. > > If the error corresponds to the "dmsetup ls" numbers, than the error > in this current boot refers to "disk2-disk2thin_tdata" and > "disk3-persistent3", which is ext4. Over an mdadm array, the error > occurs, and on a single lv it doesn't occur. > > If the error corresponds to the "dmsetup table" numbers, then the > error refers to a device not in the table as well as possibly > "disk2-disk2thin-tpool". > > Attached is another "lvmdump -m" file. And a full dmesg (6MB, using > systemd log level debug, etc) is available at: > https://github.com/systemd/systemd/files/403118/udevError.shutdown-log.txt > > If during the install I skip making the mdadm boot array and just use > a single partition, this doesn't happen. > > The error on this run is: > :: Triggering uevents... > [ 3.106579] device-mapper: table: 253:7: thin: Unable to > activate thin device while pool is suspended > [ 3.290255] device-mapper: table: 253:17: thin: Unable to > activate thin device while pool is suspended > . . . > > ============================== > > $ sudo dmsetup ls > disk3-disk3thin (253:15) > disk2-main2 (253:10) > disk3-disk3thin-tpool (253:14) > disk3-disk3thin_tdata (253:13) > disk1-disk1thin-tpool (253:2) > disk1-disk1thin_tdata (253:1) > disk3-disk3thin_tmeta (253:12) > disk3-main3 (253:16) > disk2-disk2thin (253:9) > disk1-disk1thin_tmeta (253:0) > disk2-persistent2 (253:11) > disk1-disk1thin (253:3) > disk2-disk2thin-tpool (253:8) > disk2-disk2thin_tdata (253:7) > disk1-persistent1 (253:5) > disk2-disk2thin_tmeta (253:6) > disk1-main1 (253:4) > disk3-persistent3 (253:17) > > ============================== > > $ sudo dmsetup table > > disk3-disk3thin: 0 1048576000 linear 253:14 0 > disk2-main2: 0 209715200 thin 253:8 1 > disk3-disk3thin-tpool: 0 1048576000 thin-pool 253:12 253:13 512 0 0 > disk3-disk3thin_tdata: 0 1048576000 linear 8:35 264192 > disk1-disk1thin-tpool: 0 1048576000 thin-pool 253:0 253:1 512 0 0 > disk1-disk1thin_tdata: 0 1048576000 linear 8:3 264192 > disk3-disk3thin_tmeta: 0 262144 linear 8:35 1048840192 > disk3-main3: 0 209715200 thin 253:14 1 > disk2-disk2thin: 0 1048576000 linear 253:8 0 > disk1-disk1thin_tmeta: 0 262144 linear 8:3 1048840192 > disk2-persistent2: 0 209715200 thin 253:8 2 > disk1-disk1thin: 0 1048576000 linear 253:2 0 > disk2-disk2thin-tpool: 0 1048576000 thin-pool 253:6 253:7 512 0 0 > disk2-disk2thin_tdata: 0 1048576000 linear 8:19 264192 > disk1-persistent1: 0 209715200 thin 253:2 2 > disk2-disk2thin_tmeta: 0 262144 linear 8:19 1048840192 > disk1-main1: 0 209715200 thin 253:2 1 > disk3-persistent3: 0 209715200 thin 253:14 2 > > ============================== > > $ sudo lvmdump -m > Creating dump directory: /root/lvmdump-terra-2016080534140 > > Gathering LVM & device-mapper version info... > Gathering dmsetup info... > Gathering process info... > Gathering console messages... > Gathering /etc/lvm info... > Gathering /dev listing... > Gathering /sys/block listing... > Gathering LVM metadata from Physical Volumes... > /dev/sda3 > /dev/sdb3 > /dev/sdc3 > Creating report tarball in /root/lvmdump-terra-2016080534140.tgz... > > ============================== > > /dev/sd{a,b,c}1 3.5G Linux RAID > /dev/sd{a,b,c}2 3.5G Linux RAID > /dev/sd{a,b,c}3 4.5T Linux LVM > > { Setup LVM and filesystems } > > ({ This causes the issue }) > # mdadm --create --verbose --name=main_boot --homehost="none" --level > 1 --metadata 1.0 --raid-devices=3 /dev/md1 /dev/sda1 /dev/sdb1 > /dev/sdc1 > # mkfs.ext4 -F -L main_boot /dev/disk/by-id/md-name-main_boot > > ({ This appears to prevent the issue }) > # mkfs.ext4 -F -L main_boot /dev/sda > > ({ Then, either way, continuing }) > # pvcreate --yes --pvmetadatacopies 2 /dev/sda3 > # vgcreate disk1 /dev/sda3 > # pvcreate --yes --pvmetadatacopies 2 /dev/sdb3 > # vgcreate disk2 /dev/sdb3 > # pvcreate --yes --pvmetadatacopies 2 /dev/sdc3 > # vgcreate disk3 /dev/sdc3 > # lvcreate --size 500G --thinpool disk1thin disk1 > # lvcreate --size 500G --thinpool disk2thin disk2 > # lvcreate --size 500G --thinpool disk3thin disk3 > # lvcreate --virtualsize 100G --name main1 disk1/disk1thin > # lvcreate --virtualsize 100G --name main2 disk2/disk2thin > # lvcreate --virtualsize 100G --name main3 disk3/disk3thin > # mkfs.ext4 -L main /dev/disk1/main1 > # mount /dev/disk1/main1 /mnt > # mkdir /mnt/boot > # mount /dev/dev/sda1 /mnt/boot > # lvcreate --virtualsize 100G --name persistent1 disk1/disk1thin > # lvcreate --virtualsize 100G --name persistent2 disk2/disk2thin > # lvcreate --virtualsize 100G --name persistent3 disk3/disk3thin > > { Install Arch Linux} > # vi /etc/pacman.d/mirrorlist > # pacstrap -i base syslinux gptfdisk lvm2 > # arch-chroot /mnt > # vi /etc/locale.gen > # locale-gen > # locale > /etc/locale.conf > > ({ If using the mdadm to cause the error }) > # mdadm --detail --scan > /etc/mdadm.conf > > ({ Then, either way, continuing }) > # vi /etc/nsswitch.conf > # systemd enable systemd-resolved systemd-networkd > # ln -s /usr/share/zoneinfo/America/Detroit /etc/localtime > # hwclock --utc --systohc > # passwd > { Add lvm2 between block and filesystems} > # vi /etc/mkinitcpio.conf > # mkinitcpio -p linux > # echo hostname > /etc/hostname > # vi /etc/systemd/network/enp31s0.network > # syslinux-install_update -i -a -m > # vi /boot/syslinux/syslinux.cfg > {{ After exiting the arch-chroot }} > # ln -f -s /run/systemd/resolve/resolv.conf /mnt/etc/resolv.conf > > { Reboot } > > ============================== ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2016-08-05 9:27 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-07-22 23:14 During systemd/udev, device-mapper trying to work with non-LVM volumes james harvey 2016-07-27 18:49 ` Marian Csontos 2016-07-28 1:33 ` james harvey 2016-07-28 13:13 ` Marian Csontos 2016-07-28 14:12 ` Zdenek Kabelac 2016-08-03 2:11 ` james harvey 2016-08-03 2:15 ` james harvey 2016-08-03 7:49 ` Zdenek Kabelac 2016-08-05 3:53 ` james harvey 2016-08-05 9:27 ` james harvey
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.