From mboxrd@z Thu Jan 1 00:00:00 1970 References: <47346a29-e6c7-6e22-4360-2d07e2ec7be3@redhat.com> <7839ff52-18e5-6a95-9a2a-12ea73457700@redhat.com> <62151b2e-c21a-177e-f66b-e2e08857be17@redhat.com> <6514b2be-67d7-3dfe-38b9-95e3bb39f55a@redhat.com> From: Zdenek Kabelac Message-ID: <7b17539c-749c-154f-d777-1d80d562a7f6@redhat.com> Date: Fri, 12 Apr 2019 12:05:33 +0200 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [linux-lvm] [lvm-devel] Aborting. LV mythinpool_tmeta is now incomplete Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: Content-Type: text/plain; charset="us-ascii"; format="flowed" To: LVM2 development , Eric Ren , LVM general discussion and development Dne 11. 04. 19 v 19:33 Eric Ren napsal(a): > Hi Zdenek, > >> Anyway - proper reproducer with full -vvvv log would be really the most >>> explanatory and needed to move on here. > > The activation error is reproduced once more. Please see lvm2.log attached. > > Please search the error message like this: > > $grep -rn Aborting lvm2.log > ./lvm2.log:1386:activate/dev_manager.c:2459 lvcreate Aborting. LV > mythinpool_tmeta is now incomplete and '--activationmode partial' was > not specified. > > $grep -rn "Failed to suspend" lvm2.log > ./lvm2.log:1399:metadata/lv_manip.c:7416 lvcreate Failed to suspend > thin snapshot origin vg0/6. > Hi Looking at provided log file - the system seems to be using some weird udev rule - which results in generating strange /dev/__ symlinks. My assumption can be the udev rule has missed to set some variable properly so instead of /dev/xxxx__yyyyy you end with /dev/__ There are already visible some other devices like i.e.: /dev/disk/by-id/virtio-instance-store0___O-part1 pointing to /dev/vda1 From log is visible lvm2 wants to actually store metadata on /dev/__ so for some reason thins symlink was preffered over real mknode device. So the fact there is missing device for _tmeta comes from the fact, there is probably mess in /dev directory and lvm2 becuase can't cope better with this particular case. IMHO fixing the udev rules should be enough to get things running properly. Since the log shows tons of 'nbd' device - it could be some race in udev rule processing as well. The selection of preferred name in lvm2 is somewhat out-of-data and now it would be probably always better to prefer real node name over any symlink at the price of less explanatory logging but with higher chance of figure out real state of system. But ATM it's important to figure out how this situation happens. BTW - you can always setup filter (and you really SHOULD in this case), to whitelist devices for scanning to avoid similar issues. I assume you want to set something like a|/dev/vd|, r|.*| Also you should collect udev rules and post them here so we can check which rule could be suspected. Regards Zdenek From mboxrd@z Thu Jan 1 00:00:00 1970 From: Zdenek Kabelac Date: Fri, 12 Apr 2019 12:05:33 +0200 Subject: Aborting. LV mythinpool_tmeta is now incomplete In-Reply-To: References: <47346a29-e6c7-6e22-4360-2d07e2ec7be3@redhat.com> <7839ff52-18e5-6a95-9a2a-12ea73457700@redhat.com> <62151b2e-c21a-177e-f66b-e2e08857be17@redhat.com> <6514b2be-67d7-3dfe-38b9-95e3bb39f55a@redhat.com> Message-ID: <7b17539c-749c-154f-d777-1d80d562a7f6@redhat.com> List-Id: To: lvm-devel@redhat.com MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Dne 11. 04. 19 v 19:33 Eric Ren napsal(a): > Hi Zdenek, > >> Anyway - proper reproducer with full -vvvv log would be really the most >>> explanatory and needed to move on here. > > The activation error is reproduced once more. Please see lvm2.log attached. > > Please search the error message like this: > > $grep -rn Aborting lvm2.log > ./lvm2.log:1386:activate/dev_manager.c:2459 lvcreate Aborting. LV > mythinpool_tmeta is now incomplete and '--activationmode partial' was > not specified. > > $grep -rn "Failed to suspend" lvm2.log > ./lvm2.log:1399:metadata/lv_manip.c:7416 lvcreate Failed to suspend > thin snapshot origin vg0/6. > Hi Looking at provided log file - the system seems to be using some weird udev rule - which results in generating strange /dev/__ symlinks. My assumption can be the udev rule has missed to set some variable properly so instead of /dev/xxxx__yyyyy you end with /dev/__ There are already visible some other devices like i.e.: /dev/disk/by-id/virtio-instance-store0___O-part1 pointing to /dev/vda1 From log is visible lvm2 wants to actually store metadata on /dev/__ so for some reason thins symlink was preffered over real mknode device. So the fact there is missing device for _tmeta comes from the fact, there is probably mess in /dev directory and lvm2 becuase can't cope better with this particular case. IMHO fixing the udev rules should be enough to get things running properly. Since the log shows tons of 'nbd' device - it could be some race in udev rule processing as well. The selection of preferred name in lvm2 is somewhat out-of-data and now it would be probably always better to prefer real node name over any symlink at the price of less explanatory logging but with higher chance of figure out real state of system. But ATM it's important to figure out how this situation happens. BTW - you can always setup filter (and you really SHOULD in this case), to whitelist devices for scanning to avoid similar issues. I assume you want to set something like a|/dev/vd|, r|.*| Also you should collect udev rules and post them here so we can check which rule could be suspected. Regards Zdenek