All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: [6.2 regression][bisected]discard storm on idle since v6.1-rc8-59-g63a7cb130718 discard=async
@ 2023-03-20 22:40 Christopher Price
  2023-03-21 21:26 ` Josef Bacik
  0 siblings, 1 reply; 30+ messages in thread
From: Christopher Price @ 2023-03-20 22:40 UTC (permalink / raw)
  To: slyich
  Cc: anand.jain, boris, clm, dsterba, josef, linux-btrfs, regressions,
	regressions

I can also confirm the issue occurred on and after kernel 6.2, and
confirm the workaround setting btrfs discard iops_limit works for my
device:

08:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd
NVMe SSD Controller PM9A1/PM9A3/9
80PRO

I had to use ~6000 for my device, instead of 1000.
I'm currently on kernel 6.2.1.

I am also interested to know what if this is expected.

^ permalink raw reply	[flat|nested] 30+ messages in thread
* [6.2 regression][bisected]discard storm on idle since v6.1-rc8-59-g63a7cb130718 discard=async
@ 2023-03-01 19:30 Sergei Trofimovich
  2023-03-02  8:04 ` Linux regression tracking #adding (Thorsten Leemhuis)
  2023-03-02  9:12 ` Anand Jain
  0 siblings, 2 replies; 30+ messages in thread
From: Sergei Trofimovich @ 2023-03-01 19:30 UTC (permalink / raw)
  To: linux-btrfs; +Cc: David Sterba, Boris Burkov, Chris Mason, Josef Bacik

Hi btrfs maintainers!

Tl;DR:

  After 63a7cb13071842 "btrfs: auto enable discard=async when possible" I
  see constant DISCARD storm towards my NVME device be it idle or not.

  No storm: v6.1 and older
  Has storm: v6.2 and newer

More words:

After upgrade from 6.1 to 6.2 I noticed that Disk led on my desktop
started flashing incessantly regardless of present or absent workload.

I think I confirmed the storm with `perf`: led flashes align with output
of:

    # perf ftrace -a -T 'nvme_setup*' | cat

    kworker/6:1H-298     [006]   2569.645201: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.645205: nvme_setup_discard <-nvme_setup_cmd
    kworker/6:1H-298     [006]   2569.749198: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.749202: nvme_setup_discard <-nvme_setup_cmd
    kworker/6:1H-298     [006]   2569.853204: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.853209: nvme_setup_discard <-nvme_setup_cmd
    kworker/6:1H-298     [006]   2569.958198: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.958202: nvme_setup_discard <-nvme_setup_cmd

`iotop` shows no read/write IO at all (expected).

I was able to bisect it down to this commit:

  $ git bisect good
  63a7cb13071842966c1ce931edacbc23573aada5 is the first bad commit
  commit 63a7cb13071842966c1ce931edacbc23573aada5
  Author: David Sterba <dsterba@suse.com>
  Date:   Tue Jul 26 20:54:10 2022 +0200

    btrfs: auto enable discard=async when possible

    There's a request to automatically enable async discard for capable
    devices. We can do that, the async mode is designed to wait for larger
    freed extents and is not intrusive, with limits to iops, kbps or latency.

    The status and tunables will be exported in /sys/fs/btrfs/FSID/discard .

    The automatic selection is done if there's at least one discard capable
    device in the filesystem (not capable devices are skipped). Mounting
    with any other discard option will honor that option, notably mounting
    with nodiscard will keep it disabled.

    Link: https://lore.kernel.org/linux-btrfs/CAEg-Je_b1YtdsCR0zS5XZ_SbvJgN70ezwvRwLiCZgDGLbeMB=w@mail.gmail.com/
    Reviewed-by: Boris Burkov <boris@bur.io>
    Signed-off-by: David Sterba <dsterba@suse.com>

   fs/btrfs/ctree.h   |  1 +
   fs/btrfs/disk-io.c | 14 ++++++++++++++
   fs/btrfs/super.c   |  2 ++
   fs/btrfs/volumes.c |  3 +++
   fs/btrfs/volumes.h |  2 ++
   5 files changed, 22 insertions(+)

Is this storm a known issue? I did not dig too much into the patch. But
glancing at it this bit looks slightly off:

    +       if (bdev_max_discard_sectors(bdev))
    +               fs_devices->discardable = true;

Is it expected that there is no `= false` assignment?

This is the list of `btrfs` filesystems I have:

  $ cat /proc/mounts | fgrep btrfs
  /dev/nvme0n1p3 / btrfs rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=848,subvol=/nixos 0 0
  /dev/sda3 /mnt/archive btrfs rw,noatime,compress=zstd:3,space_cache,subvolid=5,subvol=/ 0 0
  # skipped bind mounts

The device is:

  $ lspci | fgrep -i Solid
  01:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)

Can you help me debug the source of discards storm?

Is it an expected discard storm?

Is it problematic for SSD life span?

Thank you!

-- 

  Sergei

^ permalink raw reply	[flat|nested] 30+ messages in thread

end of thread, other threads:[~2023-04-21 13:56 UTC | newest]

Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-20 22:40 [6.2 regression][bisected]discard storm on idle since v6.1-rc8-59-g63a7cb130718 discard=async Christopher Price
2023-03-21 21:26 ` Josef Bacik
2023-03-22  8:38   ` Christoph Hellwig
2023-03-23 22:26     ` Sergei Trofimovich
2023-04-04 10:49       ` Linux regression tracking (Thorsten Leemhuis)
2023-04-04 16:04         ` Christoph Hellwig
2023-04-04 16:20           ` Roman Mamedov
2023-04-04 16:27             ` Christoph Hellwig
2023-04-04 23:37               ` Damien Le Moal
2023-04-04 18:15           ` Chris Mason
2023-04-04 18:51             ` Boris Burkov
2023-04-04 19:22               ` David Sterba
2023-04-04 19:39                 ` Boris Burkov
2023-04-05  8:17                   ` Linux regression tracking (Thorsten Leemhuis)
2023-04-10  2:03               ` Michael Bromilow
2023-04-11 17:52                 ` David Sterba
2023-04-11 18:15                   ` Linux regression tracking (Thorsten Leemhuis)
2023-04-04 19:08             ` Sergei Trofimovich
2023-04-05  6:18             ` Christoph Hellwig
2023-04-05 12:01               ` Chris Mason
2023-04-04 18:23         ` Boris Burkov
2023-04-04 19:12           ` Sergei Trofimovich
  -- strict thread matches above, loose matches on Subject: below --
2023-03-01 19:30 Sergei Trofimovich
2023-03-02  8:04 ` Linux regression tracking #adding (Thorsten Leemhuis)
2023-04-04 10:52   ` Linux regression tracking #update (Thorsten Leemhuis)
2023-04-21 13:56   ` Linux regression tracking #update (Thorsten Leemhuis)
2023-03-02  9:12 ` Anand Jain
2023-03-02 10:54   ` Sergei Trofimovich
2023-03-15 11:44     ` Linux regression tracking (Thorsten Leemhuis)
2023-03-15 16:34       ` Sergei Trofimovich

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.