Re: cpu bound I/O behaviour in linux 5.4 (possibly others)

[Nikolay Borisov <nborisov@suse.com>]

On 14.02.20 г. 13:30 ч., Marc Lehmann wrote:
> Hi!
> 
> I've upgraded a machine to linux 5.4.15 that runs a small netnews
> system. It normally pulls news with about 20MB/s. After upgrading (it
> seems) that this process is now CPU bound, and I get only about 10mb/s
> throughput. Otherwise, everything seems fine - no obvious bugs, and no
> obvious performance problems.
> 
> "CPU-bound" specifically means that the disk(s) seem pretty idle (it an
> 6x10TB raid5), I can do a lot of I/O without slowing down the transfer,
> but there is always a single kworker which is constantly at 100% cpu (i.e.
> one core) in top:

So this is a 50tb useful space, right?

> 
>  8963 root      20   0       0      0      0 R 2 100.0   0.0   2:04 [kworker/u8:15+flush-btrfs-3]
> 
> When I cat /proc/8963/task/8963/stack regularly, I get either no output or
> (most often) this single line:
> 
>    [<0>] tree_search_offset.isra.0+0x16a/0x1d0 [btrfs]

This points to freespace cache. One thing that I might suggest is try
using free-space-tree (aka free space cache v2 ) as opposed to v1. You
can achieve this with the space_cache=2 mount option. I suspect what
might be happening is the freespace cache is rather fragmented and the
rb tree which is really pointer chasing is having a hard time keeping
up. space cache v2 is a lot better in that regard, since it's a btree as
its backing data structure.