Re: [6.2 regression][bisected]discard storm on idle since v6.1-rc8-59-g63a7cb130718 discard=async

[Anand Jain <anand.jain@oracle.com>]

On 3/2/23 03:30, Sergei Trofimovich wrote:
> Hi btrfs maintainers!
> 
> Tl;DR:
> 
>    After 63a7cb13071842 "btrfs: auto enable discard=async when possible" I
>    see constant DISCARD storm towards my NVME device be it idle or not.
> 
>    No storm: v6.1 and older
>    Has storm: v6.2 and newer
> 
> More words:
> 
> After upgrade from 6.1 to 6.2 I noticed that Disk led on my desktop
> started flashing incessantly regardless of present or absent workload.
> 
> I think I confirmed the storm with `perf`: led flashes align with output
> of:
> 
>      # perf ftrace -a -T 'nvme_setup*' | cat
> 
>      kworker/6:1H-298     [006]   2569.645201: nvme_setup_cmd <-nvme_queue_rq
>      kworker/6:1H-298     [006]   2569.645205: nvme_setup_discard <-nvme_setup_cmd
>      kworker/6:1H-298     [006]   2569.749198: nvme_setup_cmd <-nvme_queue_rq
>      kworker/6:1H-298     [006]   2569.749202: nvme_setup_discard <-nvme_setup_cmd
>      kworker/6:1H-298     [006]   2569.853204: nvme_setup_cmd <-nvme_queue_rq
>      kworker/6:1H-298     [006]   2569.853209: nvme_setup_discard <-nvme_setup_cmd
>      kworker/6:1H-298     [006]   2569.958198: nvme_setup_cmd <-nvme_queue_rq
>      kworker/6:1H-298     [006]   2569.958202: nvme_setup_discard <-nvme_setup_cmd
> 
> `iotop` shows no read/write IO at all (expected).
> 
> I was able to bisect it down to this commit:
> 
>    $ git bisect good
>    63a7cb13071842966c1ce931edacbc23573aada5 is the first bad commit
>    commit 63a7cb13071842966c1ce931edacbc23573aada5
>    Author: David Sterba <dsterba@suse.com>
>    Date:   Tue Jul 26 20:54:10 2022 +0200
> 
>      btrfs: auto enable discard=async when possible
> 
>      There's a request to automatically enable async discard for capable
>      devices. We can do that, the async mode is designed to wait for larger
>      freed extents and is not intrusive, with limits to iops, kbps or latency.
> 
>      The status and tunables will be exported in /sys/fs/btrfs/FSID/discard .
> 
>      The automatic selection is done if there's at least one discard capable
>      device in the filesystem (not capable devices are skipped). Mounting
>      with any other discard option will honor that option, notably mounting
>      with nodiscard will keep it disabled.
> 
>      Link: https://lore.kernel.org/linux-btrfs/CAEg-Je_b1YtdsCR0zS5XZ_SbvJgN70ezwvRwLiCZgDGLbeMB=w@mail.gmail.com/
>      Reviewed-by: Boris Burkov <boris@bur.io>
>      Signed-off-by: David Sterba <dsterba@suse.com>
> 
>     fs/btrfs/ctree.h   |  1 +
>     fs/btrfs/disk-io.c | 14 ++++++++++++++
>     fs/btrfs/super.c   |  2 ++
>     fs/btrfs/volumes.c |  3 +++
>     fs/btrfs/volumes.h |  2 ++
>     5 files changed, 22 insertions(+)
> 
> Is this storm a known issue? I did not dig too much into the patch. But
> glancing at it this bit looks slightly off:
> 
>      +       if (bdev_max_discard_sectors(bdev))
>      +               fs_devices->discardable = true;
> 
> Is it expected that there is no `= false` assignment?
> 
> This is the list of `btrfs` filesystems I have:
> 
>    $ cat /proc/mounts | fgrep btrfs
>    /dev/nvme0n1p3 / btrfs rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=848,subvol=/nixos 0 0
>    /dev/sda3 /mnt/archive btrfs rw,noatime,compress=zstd:3,space_cache,subvolid=5,subvol=/ 0 0
>    # skipped bind mounts
> 



> The device is:
> 
>    $ lspci | fgrep -i Solid
>    01:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)


  It is a SSD device with NVME interface, that needs regular discard.
  Why not try tune io intensity using

  /sys/fs/btrfs/<uuid>/discard

  options?

  Maybe not all discardable sectors are not issued at once. It is a good
  idea to try with a fresh mkfs (which runs discard at mkfs) to see if
  discard is being issued even if there are no fs activities.

Thanks, Anand