[f2fs-dev] [Bug 213877] New: Mount multiple SMR block devices exceed certain number cause system non-response

[f2fs-dev] [Bug 213877] New: Mount multiple SMR block devices exceed certain number cause system non-response

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=213877

            Bug ID: 213877
           Summary: Mount multiple SMR block devices exceed certain number
                    cause system non-response
           Product: File System
           Version: 2.5
    Kernel Version: Linux DT1 5.13.4-200.fc34.x86_64 #1 SMP Tue Jul 20
                    20:27:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
          Hardware: Intel
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: f2fs
          Assignee: filesystem_f2fs@kernel-bugs.kernel.org
          Reporter: leftzheng@gmail.com
        Regression: No

[1.] One-line summary of the problem:
Mount multiple SMR block devices exceed certain number cause system
non-response

[2.] Full description of the problem/report:
Created some F2FS on SMR devices (mkfs.f2fs -m), then mounted in sequence. Each
device is the same Model: HGST HSH721414AL (Size 14TB).
Empirically, found that when the amount of SMR device * 1.5Gb > System RAM, the
system ran out of memory and hung. No dmesg output. For example, 24 SMR Disk
need 24*1.5GB = 36GB. A system with 32G RAM can only mount 21 devices, the 22nd
device will be a reproducible cause of system hang.
The number of SMR devices with other FS mounted on this system does not
interfere with the result above.

[3.] Keywords (i.e., modules, networking, kernel):
F2FS, SMR, Memory

[4.] Kernel information
[4.1.] Kernel version (uname -a):
Linux 5.13.4-200.fc34.x86_64 #1 SMP Tue Jul 20 20:27:29 UTC 2021 x86_64 x86_64
x86_64 GNU/Linux

[4.2.] Kernel .config file:
Default Fedora 34 with f2fs-tools-1.14.0-2.fc34.x86_64

[5.] Most recent kernel version which did not have the bug:
None

[6.] Output of Oops.. message (if applicable) with symbolic information
     resolved (see Documentation/admin-guide/oops-tracing.rst)
None

[7.] A small shell script or example program which triggers the
     problem (if possible)
mount /dev/sdX /mnt/0X

[8.] Memory consumption 

With 24 * 14T SMR Block device with F2FS
free -g
              total        used        free      shared  buff/cache   available
Mem:             46          36           0           0          10          10
Swap:             0           0           0


With 3 * 14T SMR Block device with F2FS
free -g
               total        used        free      shared  buff/cache  
available
Mem:               7           5           0           0           1          
1
Swap:              7           0           7

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 213877] Mount multiple SMR block devices exceed certain number cause system non-response

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=213877

James Z (leftzheng@gmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |trivial
     Kernel Version|Linux DT1                   |Linux
                   |5.13.4-200.fc34.x86_64 #1   |5.13.4-200.fc34.x86_64 #1
                   |SMP Tue Jul 20 20:27:29 UTC |SMP Tue Jul 20 20:27:29 UTC
                   |2021 x86_64 x86_64 x86_64   |2021 x86_64 x86_64 x86_64
                   |GNU/Linux                   |GNU/Linux

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 213877] Mount multiple SMR block devices exceed certain number cause system non-response

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=213877

Chao Yu (chao@kernel.org) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
                 CC|                            |chao@kernel.org

--- Comment #1 from Chao Yu (chao@kernel.org) ---
Could you please apply below patch, and try nosmall_discard option during
mount(), it expects that memory cost will decrease.

*Please note that*, I haven't do any test w/ this patch now, so please backup
your data before your test.

https://git.kernel.org/pub/scm/linux/kernel/git/chao/linux.git/commit/?h=misc&id=5dba79cc25c3c902942c2f68a6e2546586c65f96

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 213877] Mount multiple SMR block devices exceed certain number cause system non-response

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=213877

--- Comment #2 from James Z (leftzheng@gmail.com) ---
[1.] Brief
With this patch, memory usage dropped from 1.5G per SMR disk to about 1G.
The read/write function works fine on disks with plenty of free space. However,
on a near-full disk, read operations cause panic.

[2.] A small shell script or example program which triggers the problem
[James@DataT01 /mnt/03]$ touch 123

[3.] Kernel Output
[51671.050646] BUG: kernel NULL pointer dereference, address: 0000000000000000
[51671.050652] #PF: supervisor read access in kernel mode
[51671.050654] #PF: error_code(0x0000) - not-present page
[51671.050656] PGD 0 P4D 0 
[51671.050660] Oops: 0000 [#1] SMP NOPTI
[51671.050663] CPU: 4 PID: 52263 Comm: f2fs_ckpt-8:96 Tainted: G        W      
  5.13.4-200.fc34.x86_64 #1
[51671.050666] Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470
GAMING PRO MAX (MS-7B79), BIOS M.60 06/11/2020
[51671.050668] RIP: 0010:f2fs_issue_discard.isra.0+0x77/0x170 [f2fs]
[51671.050700] Code: 00 00 00 8b 40 48 8b bb 5c 04 00 00 41 29 c0 8d 4f ff 44
21 c1 89 c8 f7 d1 83 e1 07 c1 e8 03 48 03 42 18 ba 01 00 00 00 d3 e2 <0f> be 08
41 89 c8 41 09 d0 44 88 00 85 d1 75 07 83 ab 90 04 00 00
[51671.050702] RSP: 0018:ffffbe70c1893ca0 EFLAGS: 00010212
[51671.050705] RAX: 0000000000000000 RBX: ffff9f734424f000 RCX:
0000000000000007
[51671.050707] RDX: 0000000000000080 RSI: 0000000008270001 RDI:
0000000000000200
[51671.050709] RBP: 00000027ffffffd8 R08: 0000000008260000 R09:
0000000000000000
[51671.050711] R10: 000000000003de00 R11: 0000000000000004 R12:
0000000008270001
[51671.050713] R13: 0000000000000000 R14: 0000000008270000 R15:
ffff9f7345933840
[51671.050715] FS:  0000000000000000(0000) GS:ffff9f7456900000(0000)
knlGS:0000000000000000
[51671.050717] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[51671.050719] CR2: 0000000000000000 CR3: 0000000144932000 CR4:
0000000000350ee0
[51671.050721] Call Trace:
[51671.050726]  f2fs_clear_prefree_segments+0x439/0x6f0 [f2fs]
[51671.050750]  ? __raw_callee_save___native_queued_spin_unlock+0x11/0x1e
[51671.050756]  f2fs_write_checkpoint+0xccf/0x11d0 [f2fs]
[51671.050782]  __checkpoint_and_complete_reqs+0x78/0x160 [f2fs]
[51671.050803]  issue_checkpoint_thread+0x38/0xb0 [f2fs]
[51671.050823]  ? finish_wait+0x80/0x80
[51671.050827]  ? __checkpoint_and_complete_reqs+0x160/0x160 [f2fs]
[51671.050846]  kthread+0x127/0x150
[51671.050850]  ? set_kthread_struct+0x40/0x40
[51671.050852]  ret_from_fork+0x22/0x30
[51671.050858] Modules linked in: binfmt_misc f2fs nls_utf8 hfsplus hfs
crc32_generic lz4hc_compress lz4_compress isofs snd_seq_dummy snd_hrtimer
xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nf_conntrack_tftp
bridge stp llc bonding tls nft_objref nf_conntrack_netbios_ns
nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib
nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat
ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw
iptable_security rfkill ip_set nf_tables nfnetlink ip6table_filter ip6_tables
iptable_filter snd_hda_codec_realtek snd_hda_codec_generic sunrpc ledtrig_audio
snd_hda_codec_hdmi intel_rapl_msr intel_rapl_common snd_hda_intel
snd_intel_dspcfg vfat edac_mce_amd snd_intel_sdw_acpi fat snd_hda_codec kvm_amd
ppdev kvm snd_hda_core snd_hwdep irqbypass rapl snd_seq snd_seq_device pcspkr
wmi_bmof k10temp snd_pcm joydev
[51671.050910]  i2c_piix4 snd_timer snd soundcore parport_pc parport gpio_amdpt
gpio_generic acpi_cpufreq zram ip_tables radeon i2c_algo_bit drm_ttm_helper ttm
drm_kms_helper cec crct10dif_pclmul crc32_pclmul crc32c_intel
ghash_clmulni_intel drm sp5100_tco ixgbe ccp mdio dca r8169 wmi fuse
[51671.050932] CR2: 0000000000000000
[51671.050935] ---[ end trace 76f29393379353e4 ]---
[51671.050936] RIP: 0010:f2fs_issue_discard.isra.0+0x77/0x170 [f2fs]
[51671.050958] Code: 00 00 00 8b 40 48 8b bb 5c 04 00 00 41 29 c0 8d 4f ff 44
21 c1 89 c8 f7 d1 83 e1 07 c1 e8 03 48 03 42 18 ba 01 00 00 00 d3 e2 <0f> be 08
41 89 c8 41 09 d0 44 88 00 85 d1 75 07 83 ab 90 04 00 00
[51671.050960] RSP: 0018:ffffbe70c1893ca0 EFLAGS: 00010212
[51671.050962] RAX: 0000000000000000 RBX: ffff9f734424f000 RCX:
0000000000000007
[51671.050964] RDX: 0000000000000080 RSI: 0000000008270001 RDI:
0000000000000200
[51671.050966] RBP: 00000027ffffffd8 R08: 0000000008260000 R09:
0000000000000000
[51671.050967] R10: 000000000003de00 R11: 0000000000000004 R12:
0000000008270001
[51671.050969] R13: 0000000000000000 R14: 0000000008270000 R15:
ffff9f7345933840
[51671.050971] FS:  0000000000000000(0000) GS:ffff9f7456900000(0000)
knlGS:0000000000000000
[51671.050973] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[51671.050974] CR2: 0000000000000000 CR3: 0000000144932000 CR4:
0000000000350ee0

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 213877] Mount multiple SMR block devices exceed certain number cause system non-response

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=213877

--- Comment #3 from James Z (leftzheng@gmail.com) ---
However, on a near-full disk, *write* operations cause panic.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 213877] Mount multiple SMR block devices exceed certain number cause system non-response

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=213877

--- Comment #4 from Chao Yu (chao@kernel.org) ---
Sorry, I've fixed that case:

https://git.kernel.org/pub/scm/linux/kernel/git/chao/linux.git/commit/?h=misc&id=527a7d6a3e0307ad1b4bcc7811cc2920b2af45c5

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 213877] Mount multiple SMR block devices exceed certain number cause system non-response

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=213877

--- Comment #5 from James Z (leftzheng@gmail.com) ---

Could you please provide an available link to this patch? The above one is out
of date.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

[f2fs-dev] [Bug 213877] Mount multiple SMR block devices exceed certain number cause system non-response

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=213877

Jaegeuk Kim (jaegeuk@kernel.org) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jaegeuk@kernel.org

--- Comment #6 from Jaegeuk Kim (jaegeuk@kernel.org) ---
https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git/commit/?h=dev&id=4f993264fe2965c344f223d854ccbb549b16ed71

in pull request for 5.15-rc1.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel