Re: 5.6, slow send/receive, < 1MB/s

[Chris Murphy <lists@colorremedies.com>]

btrfs-progs-5.6-1.fc32.x86_64
kernel-5.6.8-300.fc32.x86_64

I've created a new file system, filled it with ~600G, set the seed
flag, and I'm creating a sprout by adding a 2nd device and removing
the 1st. This used to go at device speeds. Today it's running at
~35M/s where I expect 95-120M/s. I don't know that it's related to
send/receive performance.

The remove process with 'perf top -g -U' is only using 7% CPU, unlike
the send/receive cases.

Samples
  Children      Self  Shared Objec  Symbol
-   82.39%     0.04%  [kernel]      [k] relocate_block_group
   - 17.88% relocate_block_group
      - 7.78% prepare_to_relocate
         - 7.21% btrfs_commit_transaction
            - 7.51% btrfs_run_delayed_refs
               - __btrfs_run_delayed_refs
                  - 3.19% __tracepoint_kvm_mmu_spte_requested
                     - 3.15% pipapo_get.constprop.0
                        - 3.57% __add_to_free_space_tree
                             3.61% btrfs_del_items
                             1.37% btrfs_insert_empty_items
                  - 1.41% __tracepoint_kvm_entry
                    0.92% __tracepoint_kvm_apic_accept_irq
      - 6.41% relocate_data_extent
         - 7.52% relocate_file_extent_cluster
            - 2.35% __do_page_cache_readahead
               - 1.70% read_pages.isra.0
                  - 2.03% extent_readpages
                       1.32% __do_readpage
                 0.93% __alloc_pages_nodemask
            - 1.67% clear_extent_bit
                 1.92% __clear_extent_bit
            - 1.40% lock_extent_bits
                 1.59% __set_extent_bit
              1.19% btrfs_delalloc_reserve_metadata
              1.15% __set_page_dirty_nobuffers
      - 2.23% prepare_to_merge
         - 1.84% btrfs_commit_transaction
            - 2.15% btrfs_run_delayed_refs
               - __btrfs_run_delayed_refs
                    1.47% __btrfs_inc_extent_ref.isra.0
+   38.75%     0.47%  [kernel]      [k] __btrfs_run_delayed_refs
+   32.98%     0.21%  [kernel]      [k] relocate_file_extent_cluster
+   29.88%     0.00%  [kernel]      [k] relocate_data_extent
+   21.63%     0.00%  libc-2.31.so  [.] __GI___ioctl
+   21.46%     0.00%  [kernel]      [k] entry_SYSCALL_64_after_hwframe
+   21.46%     0.00%  [kernel]      [k] do_syscall_64
+   21.46%     0.00%  [kernel]      [k] __x64_sys_ioctl
+   21.46%     0.00%  [kernel]      [k] ksys_ioctl
+   21.46%     0.00%  [kernel]      [k] rpc_info_open
+   21.46%     0.00%  [kernel]      [k] btrfs_rm_device
For a higher level overview, try: perf top --sort comm,dso


And also running perf doesn't improve performance, which it does
rather remarkably for send/receive.


--
Chris Murphy