From: Chris Murphy <lists@colorremedies.com>
To: Btrfs BTRFS <linux-btrfs@vger.kernel.org>
Subject: 5.6, slow send/receive, < 1MB/s
Date: Mon, 4 May 2020 01:43:26 -0600 [thread overview]
Message-ID: <CAJCQCtTp+DJ3LQhfLhFh0eFBPvksrCWyDi9_KiWxM_wk+i=45w@mail.gmail.com> (raw)
btrfs-progs-5.6-1.fc32.x86_64
kernel-5.6.8-300.fc32.x86_64
I have one snapshot of a subvolume being sent from one Btrfs
filesystem to another. Both are single device. The HDD scrub
performance is ~90MB/s. The send/receive is averaging less than 2MB/s.
Upon attaching strace to the send PID, it dies.
# btrfs send /mnt/first/scratch/gits.20200503/ | btrfs receive /mnt/aviat
ERROR: command length 1718776930 too big for buffer 65536
#
$ sudo strace -p 5628
strace: Process 5628 attached
ioctl(4, BTRFS_IOC_SEND, {send_fd=6, clone_sources_count=0,
clone_sources=NULL, parent
_root=0, flags=0}) = ?
+++ killed by SIGPIPE +++
During the send/receive, top reports both of them using 90%+ CPU
rsync averages 65MBs
Another subvolume, also one snapshot, has long periods of time (1/2
hour) where the transfer rate is never above 500K/s, but both the
btrfs send and btrfs receive PIDS are pinned at 99%+ CPU.
perf top during high btrfs command CPU consumption and low IO throughput
34.01% [kernel] [k]
mutex_spin_on_owner
28.35% [kernel] [k] mutex_unlock
8.05% [kernel] [k]
mwait_idle_with_hints.constprop.
5.10% [kernel] [k] mutex_lock
perf top during moderate btrfs command CPU consumption and high IO throughput
15.16% [kernel] [k]
fuse_perform_write
10.27% [kernel] [k]
mwait_idle_with_hints.constprop.
7.97% [kernel] [k] mutex_unlock
7.35% [kernel] [k]
mutex_spin_on_owner
5.75% btrfs [.] __crc32c_le
I suspect many fragments for some files when performance is slow, and
somehow btrfs send is more adversely affected by this than cp or
rsync.
--
Chris Murphy
next reply other threads:[~2020-05-04 7:43 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-04 7:43 Chris Murphy [this message]
2020-05-04 7:47 ` 5.6, slow send/receive, < 1MB/s Nikolay Borisov
2020-05-04 8:03 ` Chris Murphy
2020-05-04 8:12 ` Chris Murphy
2020-05-04 8:13 ` Nikolay Borisov
2020-05-04 8:25 ` Chris Murphy
2020-05-04 8:43 ` Chris Murphy
2020-05-06 5:48 ` Chris Murphy
2020-05-06 21:04 ` Chris Murphy
2020-05-06 21:07 ` Chris Murphy
2020-05-23 16:11 ` Jorge Bastos
2020-05-23 16:42 ` A L
2020-06-06 16:16 ` Jorge Bastos
2020-06-06 16:24 ` Jorge Bastos
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAJCQCtTp+DJ3LQhfLhFh0eFBPvksrCWyDi9_KiWxM_wk+i=45w@mail.gmail.com' \
--to=lists@colorremedies.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).