linux-lvm.redhat.com archive mirror
 help / color / mirror / Atom feed
From: Zdenek Kabelac <zdenek.kabelac@gmail.com>
To: LVM general discussion and development <linux-lvm@redhat.com>,
	Thomas Deutschmann <whissi@whissi.de>
Subject: Re: [linux-lvm] lvcreate hangs forever during snapshot creation when suspending volume
Date: Mon, 1 Aug 2022 19:29:41 +0200	[thread overview]
Message-ID: <d73e7506-a754-55d0-734f-084ffda1d6f2@gmail.com> (raw)
In-Reply-To: <000101d8a432$08688fe0$1939afa0$@whissi.de>

Dne 30. 07. 22 v 18:33 Thomas Deutschmann napsal(a):
> Hi,
> 
> while trying to backup a Dell R7525 system running
> Debian bookworm/testing using LVM snapshots I noticed that the system
> will 'freeze' sometimes (not all the times) when creating the snapshot.
> To recover from this, a power cycle is required.
> 
> Is this a problem caused by LVM or a kernel issue?
> 
> The command I run:
> 
>    /usr/sbin/lvcreate \
>    -vvvvv \
>    --size 100G \
>    --snapshot /dev/mapper/devDataStore1-volMachines \
>    --name volMachines_snap
> 
> The last 4 lines:
>> [Sat Jul 30 16:31:34 2022] debugfs: Directory 'dm-4' with parent 'block' already present!
>> [Sat Jul 30 16:31:34 2022] debugfs: Directory 'dm-7' with parent 'block' already present!
>> [Sat Jul 30 16:34:55 2022] INFO: task mariadbd:1607 blocked for more than 120 seconds.
>> [Sat Jul 30 16:34:55 2022]       Not tainted 5.18.0-2-amd64 #1 Debian 5.18.5-1
>> [Sat Jul 30 16:34:55 2022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> [Sat Jul 30 16:34:55 2022] task:mariadbd        state:D stack:    0 pid: 1607 ppid:  1289 flags:0x00000000
>> [Sat Jul 30 16:34:55 2022] Call Trace:
>> [Sat Jul 30 16:34:55 2022]  <TASK>
>> [Sat Jul 30 16:34:55 2022]  __schedule+0x30b/0x9e0
>> [Sat Jul 30 16:34:55 2022]  schedule+0x4e/0xb0
>> [Sat Jul 30 16:34:55 2022]  percpu_rwsem_wait+0x112/0x130
>> [Sat Jul 30 16:34:55 2022]  ? __percpu_rwsem_trylock.part.0+0x70/0x70
>> [Sat Jul 30 16:34:55 2022]  __percpu_down_read+0x5e/0x80
>> [Sat Jul 30 16:34:55 2022]  io_write+0x2e9/0x300
>> [Sat Jul 30 16:34:55 2022]  ? _raw_spin_lock+0x13/0x30
>> [Sat Jul 30 16:34:55 2022]  ? newidle_balance+0x26a/0x400
>> [Sat Jul 30 16:34:55 2022]  ? fget+0x7c/0xb0
>> [Sat Jul 30 16:34:55 2022]  io_issue_sqe+0x47c/0x2550
>> [Sat Jul 30 16:34:55 2022]  ? select_task_rq_fair+0x174/0x1240
>> [Sat Jul 30 16:34:55 2022]  ? hrtimer_try_to_cancel+0x78/0x110
>> [Sat Jul 30 16:34:55 2022]  io_submit_sqes+0x3ce/0x1aa0
>> [Sat Jul 30 16:34:55 2022]  ? _raw_spin_unlock_irqrestore+0x23/0x40
>> [Sat Jul 30 16:34:55 2022]  ? wake_up_q+0x4a/0x90
>> [Sat Jul 30 16:34:55 2022]  ? __do_sys_io_uring_enter+0x565/0xa60
>> [Sat Jul 30 16:34:55 2022]  __do_sys_io_uring_enter+0x565/0xa60
>> [Sat Jul 30 16:34:55 2022]  do_syscall_64+0x3b/0xc0
>> [Sat Jul 30 16:34:55 2022]  entry_SYSCALL_64_after_hwframe+0x44/0xae
>> [Sat Jul 30 16:34:55 2022] RIP: 0033:0x7f05b90229b9
>> [Sat Jul 30 16:34:55 2022] RSP: 002b:00007eff8e9efa38 EFLAGS: 00000216 ORIG_RAX: 00000000000001aa
>> [Sat Jul 30 16:34:55 2022] RAX: ffffffffffffffda RBX: 0000561c424f1d18 RCX: 00007f05b90229b9
>> [Sat Jul 30 16:34:55 2022] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000009
>> [Sat Jul 30 16:34:55 2022] RBP: 00007eff8e9efa90 R08: 0000000000000000 R09: 0000000000000008
>> [Sat Jul 30 16:34:55 2022] R10: 0000000000000000 R11: 0000000000000216 R12: 0000561c42500938
>> [Sat Jul 30 16:34:55 2022] R13: 00007f05b9821c00 R14: 0000561c425009e0 R15: 0000561c424f1d18
>> [Sat Jul 30 16:34:55 2022]  </TASK>
>> [Sat Jul 30 16:34:55 2022] INFO: task mariadbd:9955 blocked for more than 120 seconds.
>> [Sat Jul 30 16:34:55 2022]       Not tainted 5.18.0-2-amd64 #1 Debian 5.18.5-1
>> [Sat Jul 30 16:34:55 2022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> [Sat Jul 30 16:34:55 2022] task:mariadbd        state:D stack:    0 pid: 9955 ppid:  1289 flags:0x00000000
>> [Sat Jul 30 16:34:55 2022] Call Trace:
>> [Sat Jul 30 16:34:55 2022]  <TASK>
>> [...]
> 
> The message "mariadbd:1607 blocked for more than 120 seconds" will repeat.
> 
> MariaDB itself is running in a systemd-nspawn container. The container
> storage is located on the volume for which snapshot creation will hang.


Hi


Lvm2 is *NOT* supported to be used within containers!

This requires some very specific 'system' modification and its in overal very 
problematic - so rule #1 is - always run lvm2 command on your hosting machine.


Now - you suggests you are able to reproduce this issue also on your bare 
metal hw - in this case run these 3 commands  before  'lvcreate'


# dmsetup table
# dmsetup info -c
# dmsetup ls --tree

# lvcreate ....

If it blocks take again these:
# dmsetup table
# dmsetup info -c
# dmsetup ls --tree


You 'lvcreate -vvvv'  & 'dmesg' trace simply suggest that system is waiting to 
fulfill  'fsfreeze'  operation - it's unclear why it cannot be finished - 
maybe some problem with your 'raid' array ??

So far I do not see any bug on lvm2 side - all works from lvm2 side as 
expected - however it's unclear why your 'raid' is so slow ?

Note: you could always 'experiment' without lvm2 in the picture -
you can ran   'fsfreeze --freeze|--unfreeze'  yourself - to see whether even 
this command is able to finish  ?

Note2: if you system has lots of 'dirty' pages - it may potentially take a lot 
of time to 'fsfreeze' operation of a filesystem since all 'dirty' pages needs 
to be written to your disk..


Regards

Zdenek

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


  reply	other threads:[~2022-08-01 17:30 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-07-30 16:33 [linux-lvm] lvcreate hangs forever during snapshot creation when suspending volume Thomas Deutschmann
2022-08-01 17:29 ` Zdenek Kabelac [this message]
2022-08-01 17:34   ` Zdenek Kabelac
2022-08-01 20:34     ` Thomas Deutschmann
2022-08-02  3:01       ` Thomas Deutschmann
2022-08-02 14:52         ` Zdenek Kabelac
2022-08-03  1:37           ` Thomas Deutschmann
2022-08-28 22:38             ` Thomas Deutschmann
2022-08-29 10:32               ` Zdenek Kabelac

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d73e7506-a754-55d0-734f-084ffda1d6f2@gmail.com \
    --to=zdenek.kabelac@gmail.com \
    --cc=linux-lvm@redhat.com \
    --cc=whissi@whissi.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).