4+ hour hung task running btrfs receive - btrfs_delayed_meta_helper

4+ hour hung task running btrfs receive - btrfs_delayed_meta_helper

[Mike Javorski]

Greetings..

I was running btrfs receive to apply snapshots from another machine
and the receiving machine locked up with only a few KB received
(tracked with pv). I left it for ~ 4 hours with no change, and tried
to abort the receive and it would not exit. I then tried to "ls -al"
on the snapshot directory and that froze as well.

I then triggered a sysrq w, and have pastebined the latter section of
the output:  https://pastebin.com/fQFWq307
(I included everything after boot, there was no abnormal bits in the
boot and I didn't want to have to filter everything for PII)

Now it is ~16 hours later and both the "btrfs receive" and "ls"
processes are still locked in "D" state, but I don't know if I should
just reboot the machine (preferably not with a hard reset) or not. I
do have a keyboard attached so I can possibly do a SysRq forced
unmount if provided with "safe" instruction as I have never done that
before. I do find that I can list files in other snapshot subvolumes,
so the base FS isn't completely blocked, but is around this new
snapshot.

Other (possibly pertinent) info:
- Both machines are running the latest ArchLinux provided kernel of
4.19.8-arch1 and btrfs-progs 4.19-1
- The receiving machine had just replaced a failing disk with a new
one (via btrfs disk replace) and all indications were that it
completed successfully, correcting errors reading from the old disk as
they were encountered.
- I had also just deleted two snapshots which were of suspect
completeness and was re-sending one of those.

So that is the scenario. Happy to get more logs if needed, but would
welcome advice on how to proceed.

Thanks,

- mike