linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Chris Murphy <lists@colorremedies.com>
To: Btrfs BTRFS <linux-btrfs@vger.kernel.org>
Subject: 5.6, slow send/receive, < 1MB/s
Date: Mon, 4 May 2020 01:43:26 -0600	[thread overview]
Message-ID: <CAJCQCtTp+DJ3LQhfLhFh0eFBPvksrCWyDi9_KiWxM_wk+i=45w@mail.gmail.com> (raw)

btrfs-progs-5.6-1.fc32.x86_64
kernel-5.6.8-300.fc32.x86_64

I have one snapshot of a subvolume being sent from one Btrfs
filesystem to another. Both are single device. The HDD scrub
performance is ~90MB/s. The send/receive is averaging less than 2MB/s.
Upon attaching strace to the send PID, it dies.

# btrfs send /mnt/first/scratch/gits.20200503/ | btrfs receive /mnt/aviat
ERROR: command length 1718776930 too big for buffer 65536
#

$ sudo strace -p 5628
strace: Process 5628 attached
ioctl(4, BTRFS_IOC_SEND, {send_fd=6, clone_sources_count=0,
clone_sources=NULL, parent
_root=0, flags=0}) = ?
+++ killed by SIGPIPE +++

During the send/receive, top reports both of them using 90%+ CPU

rsync averages 65MBs

Another subvolume, also one snapshot, has long periods of time (1/2
hour) where the transfer rate is never above 500K/s, but both the
btrfs send and btrfs receive PIDS are pinned at 99%+ CPU.

perf top during high btrfs command CPU consumption and low IO throughput

  34.01%  [kernel]                                [k]
mutex_spin_on_owner
  28.35%  [kernel]                                [k] mutex_unlock
    8.05%  [kernel]                                [k]
mwait_idle_with_hints.constprop.
    5.10%  [kernel]                                [k] mutex_lock

perf top during moderate btrfs command CPU consumption and high IO throughput

 15.16%  [kernel]                                [k]
fuse_perform_write
 10.27%  [kernel]                                [k]
mwait_idle_with_hints.constprop.
   7.97%  [kernel]                                [k] mutex_unlock
   7.35%  [kernel]                                [k]
mutex_spin_on_owner
    5.75%  btrfs                                   [.] __crc32c_le


I suspect many fragments for some files when performance is slow, and
somehow btrfs send is more adversely affected by this than cp or
rsync.

-- 
Chris Murphy

             reply	other threads:[~2020-05-04  7:43 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-04  7:43 Chris Murphy [this message]
2020-05-04  7:47 ` 5.6, slow send/receive, < 1MB/s Nikolay Borisov
2020-05-04  8:03 ` Chris Murphy
2020-05-04  8:12   ` Chris Murphy
2020-05-04  8:13   ` Nikolay Borisov
2020-05-04  8:25     ` Chris Murphy
2020-05-04  8:43       ` Chris Murphy
2020-05-06  5:48         ` Chris Murphy
2020-05-06 21:04           ` Chris Murphy
2020-05-06 21:07             ` Chris Murphy
2020-05-23 16:11               ` Jorge Bastos
2020-05-23 16:42                 ` A L
2020-06-06 16:16                   ` Jorge Bastos
2020-06-06 16:24                     ` Jorge Bastos

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAJCQCtTp+DJ3LQhfLhFh0eFBPvksrCWyDi9_KiWxM_wk+i=45w@mail.gmail.com' \
    --to=lists@colorremedies.com \
    --cc=linux-btrfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).