Weird behavior (no commit?) when writing to a really slow block device

Weird behavior (no commit?) when writing to a really slow block device

[Roman Mamedov]

Hello,

I am rsyncing some files from fast devices (100-200 MB/s) to a slow remote one
(15 MB/s) over nbd.

There is some strange behavior, at least on my current kernel 5.10.161:

  * free space shown in "df" does NOT decrease even after many hours and
    hundreds of GBs already written;

  * freshly written files (already fully copied by rsync log) are shown with
    their real size in "ls -la", but all have zero size in "du".

I keep an eye on: watch "grep -e Dirty -e Writeback: /proc/meminfo"
This currently shows:

  Dirty:            554936 kB
  Writeback:       3388616 kB

The latter figure hovers around 3.5-4 GB all the time. For these syncing
setups, I have these settings tweaked on this machine (so I guess it's not as
large as it could have been with 64 GB of RAM):

  vm.dirty_background_ratio=5
  vm.dirty_ratio=10

As I understand the described weird symptoms would have cleared up if there
was a complete flush of data and metadata. And I suspect issuing a "sync" on
command-line would cause that to occur. But the question is why this didn't
occur on its own, as said above, over many hours of copying? If I disconnect
the block device now, wouldn't everything copied so far be actually lost?

I know there are the "commit" and "flushoncommit" options, but in this case I
haven't overridden either. Mount options are:

  rw,noatime,nodiratime,compress=zstd:3,nossd,space_cache=v2,skip_balance,subvolid=5,subvol=/

So if I had to guess, given a slow block device and the constant ongoing write
load trying to stream at a much faster rate, the "commit" doesn't actually
happen, neither in its predefined default interval of 30 seconds, nor during
much much longer (hours).

-- 
With respect,
Roman

Re: Weird behavior (no commit?) when writing to a really slow block device

[Roman Mamedov]

On Sun, 8 Jan 2023 02:03:31 +0500
Roman Mamedov <rm@romanrm.net> wrote:

> if there was a complete flush of data and metadata. And I suspect issuing a
> "sync" on command-line would cause that to occur.

In fact, no. :D

Running a "sync" simply does not return while the rsync process is still
actively copying data to the slow device.

It's stuck for 30 minutes by now, despite it would take less than 4 minutes to
write out the 3.5 GB of the writeback buffer at 15 MB/sec of the target device.

And the free space in "df" still did not update from what it was when I started
the copy process 3 hours ago. rsync has managed to write 185 GB since then,
according to its /proc/xxx/io.


-- 
With respect,
Roman