[linux-lvm] System completely unstable after migrating to thin pools

[linux-lvm] System completely unstable after migrating to thin pools

[Sreyan Chakravarty]

[-- Attachment #1.1: Type: text/plain, Size: 2259 bytes --]

Hi,

I have migrated to thin pools using partclone.

But now that I am using thin pools my system is completely unstable.

My swap is not activated :

swapon[1004]: swapon: /fedora.swap: read swap header failed
systemd[1]: fedora.swap.swap: Swap process exited, code=exited,
status=255/EXCEPTION

systemd[1]: fedora.swap.swap: Failed with result 'exit-code'.
 Failed to activate swap /fedora.swap.
 Dependency failed for Swap.
 swap.target: Job swap.target/start failed with result 'dependency'.

My System logging service is not started:

rsyslog.service: Failed with result 'core-dump'.
 Failed to start System Logging Service.
 SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295
subj=system_u:system_r:init_t:s0 msg='unit=rsyslog comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
 rsyslog.service: Scheduled restart job, restart counter is at 1.

And device mapper gives me weird messages during startup like:

 kernel: device-mapper: btree spine: node_check failed: csum 3318704195 !=
wanted 3318554075
 kernel: device-mapper: block manager: btree_node validator check failed
for block 220
 kernel: device-mapper: btree spine: node_check failed: csum 3318704195 !=
wanted 3318554075
 kernel: device-mapper: block manager: btree_node validator check failed
for block 220
 kernel: device-mapper: btree spine: node_check failed: csum 3318704195 !=
wanted 3318554075
 kernel: device-mapper: block manager: btree_node validator check failed
for block 220
 kernel: device-mapper: btree spine: node_check failed: csum 3318704195 !=
wanted 3318554075
 kernel: device-mapper: block manager: btree_node validator check failed
for block 220

Also some dracut module is missing:

dracut-initqueue[867]: /usr/sbin/thin_check: execvp failed: No such file or
directory
 dracut-initqueue[867]: WARNING: Check is skipped, please install
recommended missing binary /usr/sbin/thin_check!


In short my system has gone to hell.

This all started when I did a fstrim like this:

sudo fstrim -v /

And it returned an input/output error.

My logical volumes were marked with two 'XX', I don't know what that means.
After that I restarted.

What can I do to get my system back to normal ?

Please help.

-- 
Regards,
Sreyan Chakravarty

[-- Attachment #1.2: Type: text/html, Size: 2959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 197 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] System completely unstable after migrating to thin pools

[John Stoffel]

Sreyan> I have migrated to thin pools using partclone.

I suspect you're toast.  What are you trying to accomplish here
though?  Trying to stuff 10 pounds of data into a 5 pound bag?  *grin*

Or are you trying to have the ability to take snapshots of your system
to make rollbacks easier?  

Sreyan> But now that I am using thin pools my system is completely unstable.

You need to post the output of your configuration in more detail, with
dmesg output, logs, lvs, pvs, vgs output, etc. 

But really, you do have backups?

Sreyan> My swap is not activated :

Sreyan> swapon[1004]: swapon: /fedora.swap: read swap header failed
Sreyan> systemd[1]: fedora.swap.swap: Swap process exited, code=exited, status=255/EXCEPTION

Sreyan> systemd[1]: fedora.swap.swap: Failed with result 'exit-code'.
Sreyan>  Failed to activate swap /fedora.swap.
Sreyan>  Dependency failed for Swap.
Sreyan>  swap.target: Job swap.target/start failed with result 'dependency'.

Sreyan> My System logging service is not started:

Sreyan> rsyslog.service: Failed with result 'core-dump'.
Sreyan>  Failed to start System Logging Service.
Sreyan>  SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=
Sreyan> 'unit=rsyslog comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=
Sreyan> failed'
Sreyan>  rsyslog.service: Scheduled restart job, restart counter is at 1.

Sreyan> And device mapper gives me weird messages during startup like:

Sreyan>  kernel: device-mapper: btree spine: node_check failed: csum 3318704195 != wanted 3318554075
Sreyan>  kernel: device-mapper: block manager: btree_node validator check failed for block 220
Sreyan>  kernel: device-mapper: btree spine: node_check failed: csum 3318704195 != wanted 3318554075
Sreyan>  kernel: device-mapper: block manager: btree_node validator check failed for block 220
Sreyan>  kernel: device-mapper: btree spine: node_check failed: csum 3318704195 != wanted 3318554075
Sreyan>  kernel: device-mapper: block manager: btree_node validator check failed for block 220
Sreyan>  kernel: device-mapper: btree spine: node_check failed: csum 3318704195 != wanted 3318554075
Sreyan>  kernel: device-mapper: block manager: btree_node validator check failed for block 220

Sreyan> Also some dracut module is missing:

Sreyan> dracut-initqueue[867]: /usr/sbin/thin_check: execvp failed: No such file or directory
Sreyan>  dracut-initqueue[867]: WARNING: Check is skipped, please install recommended missing binary /usr/
Sreyan> sbin/thin_check!

Sreyan> In short my system has gone to hell.

Sreyan> This all started when I did a fstrim like this:

Sreyan> sudo fstrim -v /

Sreyan> And it returned an input/output error.

Sreyan> My logical volumes were marked with two 'XX', I don't know what that means. After that I
Sreyan> restarted.

Sreyan> What can I do to get my system back to normal ?

Sreyan> Please help.

Sreyan> --
Sreyan> Regards,
Sreyan> Sreyan Chakravarty

Sreyan> _______________________________________________
Sreyan> linux-lvm mailing list
Sreyan> linux-lvm@redhat.com
Sreyan> https://www.redhat.com/mailman/listinfo/linux-lvm
Sreyan> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] System completely unstable after migrating to thin pools

[Sreyan Chakravarty]

[-- Attachment #1.1: Type: text/plain, Size: 1189 bytes --]

On Tue, Dec 1, 2020 at 3:55 AM John Stoffel <john@stoffel.org> wrote:

>
> I suspect you're toast.  What are you trying to accomplish here
> though?  Trying to stuff 10 pounds of data into a 5 pound bag?  *grin*
>
>
I don't know what you mean by that. Do you mean that thin pools cannot be
migrated to from older LVMs ?


> Or are you trying to have the ability to take snapshots of your system
> to make rollbacks easier?
>
>
Yes.


> You need to post the output of your configuration in more detail, with
> dmesg output, logs, lvs, pvs, vgs output, etc.
>
>
$ lvs
  LV     VG       Attr       LSize    Pool   Origin Data%  Meta%  Move Log
Cpy%Sync Convert
  fedora vgfedora Vwi-aotz--  700.00g pool00        16.66

  pool00 vgfedora twi-aotz-- <929.76g               12.54  16.62


$ pvs
  PV                                                    VG       Fmt  Attr
PSize   PFree
  /dev/mapper/luks-2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4 vgfedora lvm2 a--
 929.99g 120.00m

$ vgs
  VG       #PV #LV #SN Attr   VSize   VFree
  vgfedora   1   2   0 wz--n- 929.99g 120.00m

dmesg:

https://pastebin.com/raw/svTX92SJ


> But really, you do have backups?
>
>
No.


-- 
Regards,
Sreyan Chakravarty

[-- Attachment #1.2: Type: text/html, Size: 2542 bytes --]

[-- Attachment #2: Type: text/plain, Size: 197 bytes --]

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] System completely unstable after migrating to thin pools

[John Stoffel]

>>>>> "Sreyan" == Sreyan Chakravarty <sreyan32@gmail.com> writes:

Sreyan> On Tue, Dec 1, 2020 at 3:55 AM John Stoffel <john@stoffel.org> wrote:
Sreyan>     I suspect you're toast.  What are you trying to accomplish here
Sreyan>     though?  Trying to stuff 10 pounds of data into a 5 pound bag?  *grin*

Sreyan> I don't know what you mean by that. Do you mean that thin
Sreyan> pools cannot be migrated to from older LVMs ?  

I don't think so.  The two versions of LVM thin pools aren't
compatible.  See the archives of this list for more details.  I also
don't quite trust them either with my data.  

Sreyan>     Or are you trying to have the ability to take snapshots of your system
Sreyan>     to make rollbacks easier? 

Sreyan> Yes.

You might want to think about using 'btrfs' instead, since it supports
filesystem level snapshots, and I think Ubuntu 20.x or newer uses it
for it's default filesystem now.  I'm pretty sure Linux MINT does as
well.

But this will require you to copy the data off the current setup (if
you can get it working again) and onto new disks.  A full re-install
of the OS will be needed.

Not that I truest brtfs either quite yet.  Nor do I like the mashing
of various storage layers into one unified whole (like ZFS does)
though I understand why they feel they need to do it.  

Sreyan>     You need to post the output of your configuration in more detail, with
Sreyan>     dmesg output, logs, lvs, pvs, vgs output, etc.

Sreyan> $ lvs
Sreyan>   LV     VG       Attr       LSize    Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
Sreyan>   fedora vgfedora Vwi-aotz--  700.00g pool00        16.66                                  
Sreyan>   pool00 vgfedora twi-aotz-- <929.76g               12.54  16.62                           

So it looks like you're using Fedora, which I honestly don't have alot
of experience with in situations like this.  I'm more of a Debian guy
at home and a CentOS/RHEL guy at work.  

Sreyan> $ pvs
Sreyan>   PV                                                    VG       Fmt  Attr PSize   PFree  
Sreyan>   /dev/mapper/luks-2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4 vgfedora lvm2 a--  929.99g 120.00m

Sreyan> $ vgs
Sreyan>   VG       #PV #LV #SN Attr   VSize   VFree  
Sreyan>   vgfedora   1   2   0 wz--n- 929.99g 120.00m

Sreyan> dmesg:

Sreyan> https://pastebin.com/raw/svTX92SJ
Sreyan>  

Sreyan>     But really, you do have backups?

Sreyan> No.

You're probably toast.

Can you boot off a rescue disk, or even a recent Fedora bootable ISO
and poke around at your current setup?  That might let your recover
your data.

Of course before you do that, you should copy the original disk onto a
new disk on save the original somewhere offline and unpowered, so you
can run tests on the copy of the data.

Good luck!


_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/