Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

[David Newall <btrfs@davidnewall.com>]

> echo w > /proc/sysrq-trigger

Interesting.

One material point which I failed to mention is that the btrfs volume is 
on an encrypted volume (cryptsetup luksOpen /dev/vdc backups).

The first step, "mount -r /dev/vg/ext2fs-snapshot 
/btrfs-backup-volume/local-snapshot", seemed to trigger the problem.  
When I did the echo to sysrq-trigger, it seemed to stop blocking, but 
that might have been a coincidence.  After the echo, kernel output 
exceeded 100KB, so I saved it to https://davidnewall.com/kern.1

During rsync (--archive --one-file-system --hard-links --inplace 
--numeric-ids --delete /btrfs-backup-volume/local-snapshot/ 
/btrfs-backup-volume/data/), initially there was no problem, but, then 
it (df) seemed to hang again.  The rsync took a long time to complete, 
and before it did finish, I did the echo to sysrq-trigger again; kernel 
output is saved to https://davidnewall.com/kern.2

The rsync finished not long after the echo to sysrq-trigger, but that's 
probably also a coincidence.  After rsync completed, df still hung.  I 
did another echo to sysrq-trigger, and saved kernel output to 
https://davidnewall.com/kern.3

I tried a minor change in procedure to see if it would bring the system 
back to normal response.  Normally I'd do "btrfs subvolume snapshot", 
but I tried unmounting the lvm2 snapshot first (umount 
/btrfs-backup-volume/local-snapshot).  It did not complete within the 
expected time, and another echo to sysrq-trigger resulted in 
https://davidnewall.com/kern.4

Eventually the umount completed and system came back to normal response.

I did the btrfs subvolume snapshot, and it completed faster than I could 
notice without causing any issues.

After unmounting the btrfs volume, I tried each step again, and 
everything completed within expected times without causing any hang.

Something which I did previously mention, but I'll repeat because it 
might well be important, is that the base ext2 filesystem is on a 
drbd-replicated volume.  I don't know if it's part of the problem, and I 
observe that the hang condition was not triggered at the point of 
creating the lvm2 snapshot.

I greatly appreciate your advice and help.

Thanks,

David