* Re: [f2fs-dev] kernel BUG at fs/f2fs/segment.c:2391 during xfstest generic/204 [not found] <20190729174140.GA169027@gmail.com> @ 2019-08-16 8:37 ` Chao Yu 2019-08-18 16:03 ` Eric Biggers 0 siblings, 1 reply; 4+ messages in thread From: Chao Yu @ 2019-08-16 8:37 UTC (permalink / raw) To: Eric Biggers; +Cc: linux-f2fs-devel Hi Eric, Sorry for late reply... I only hit ENOSPC error: QA output created by 204 line 82: echo: write error: No space left on device What is your mkfs/mount option? On 2019/7/30 1:41, Eric Biggers wrote: > On xfstest generic/204 on f2fs, I'm getting a kernel BUG. > > It bisects to: > > commit 1cb50f87e10696e8cc61fb62d0d948e11b0e6dc1 > Author: Jaegeuk Kim <jaegeuk@kernel.org> > Date: Fri Jul 6 16:47:34 2018 -0700 > > f2fs: do checkpoint in kill_sb > > Here is the BUG, on v5.3-rc2: > > generic/204 1s ... [10:34:22][ 6.005271] run fstests generic/204 at 2019-07-29 10:34:22 > ------------[ cut here ]------------ > kernel BUG at fs/f2fs/segment.c:2391! > invalid opcode: 0000 [#1] SMP > CPU: 1 PID: 111 Comm: kworker/u4:2 Not tainted 5.3.0-rc2 #4 > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 > Workqueue: writeback wb_workfn (flush-254:32) > RIP: 0010:get_new_segment fs/f2fs/segment.c:2382 [inline] > RIP: 0010:new_curseg+0x350/0x360 fs/f2fs/segment.c:2512 > Code: 8b 7b 48 44 89 ea 41 0f af f6 e8 5b cb 06 00 41 8b 97 18 0b 00 00 41 89 c2 41 0f af d6 39 d0 0f 82 97 fe ff ff e9 a7 fd ff ff <0f> 0b 45 31 e4 45 31 ed e9 47 f2 > RSP: 0018:ffffc9000045f5c8 EFLAGS: 00010246 > RAX: 0000000000000031 RBX: ffff88807b533800 RCX: 0000000000000040 > RDX: 0000000000000031 RSI: 0000000000000000 RDI: ffff88807af35b60 > RBP: ffffc9000045f618 R08: 000000000000000a R09: 0000000000000001 > R10: ffff88807b533828 R11: 0000000000000001 R12: 0000000000000000 > R13: 0000000000000001 R14: 0000000000000000 R15: ffff888078d3c000 > FS: 0000000000000000(0000) GS:ffff88807fd00000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 000055db93c6b2d8 CR3: 0000000078c43005 CR4: 0000000000760ee0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > PKRU: 55555554 > Call Trace: > allocate_segment_by_default+0x9c/0x100 fs/f2fs/segment.c:2651 > f2fs_allocate_data_block+0x1f3/0x600 fs/f2fs/segment.c:3101 > do_write_page+0x9d/0x110 fs/f2fs/segment.c:3165 > f2fs_do_write_node_page+0x35/0xb0 fs/f2fs/segment.c:3216 > __write_node_page+0x290/0x660 fs/f2fs/node.c:1574 > f2fs_sync_node_pages+0x6e1/0x790 fs/f2fs/node.c:1843 > block_operations fs/f2fs/checkpoint.c:1225 [inline] > f2fs_write_checkpoint+0x282/0x920 fs/f2fs/checkpoint.c:1576 > f2fs_sync_fs+0xc6/0x1b0 fs/f2fs/super.c:1116 > f2fs_balance_fs_bg+0x107/0x300 fs/f2fs/segment.c:557 > f2fs_write_node_pages+0x55/0x310 fs/f2fs/node.c:1931 > do_writepages+0x43/0xf0 mm/page-writeback.c:2342 > __writeback_single_inode+0x56/0x5b0 fs/fs-writeback.c:1364 > writeback_sb_inodes+0x253/0x580 fs/fs-writeback.c:1628 > wb_writeback+0x10f/0x480 fs/fs-writeback.c:1804 > wb_do_writeback fs/fs-writeback.c:1949 [inline] > wb_workfn+0xa9/0x570 fs/fs-writeback.c:1990 > process_one_work+0x21c/0x5c0 kernel/workqueue.c:2269 > worker_thread+0x3a/0x3b0 kernel/workqueue.c:2415 > kthread+0x124/0x140 kernel/kthread.c:255 > ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 > ---[ end trace 63bbe1af16f6dd61 ]--- > RIP: 0010:get_new_segment fs/f2fs/segment.c:2382 [inline] > RIP: 0010:new_curseg+0x350/0x360 fs/f2fs/segment.c:2512 > Code: 8b 7b 48 44 89 ea 41 0f af f6 e8 5b cb 06 00 41 8b 97 18 0b 00 00 41 89 c2 41 0f af d6 39 d0 0f 82 97 fe ff ff e9 a7 fd ff ff <0f> 0b 45 31 e4 45 31 ed e9 47 f2 > RSP: 0018:ffffc9000045f5c8 EFLAGS: 00010246 > RAX: 0000000000000031 RBX: ffff88807b533800 RCX: 0000000000000040 > RDX: 0000000000000031 RSI: 0000000000000000 RDI: ffff88807af35b60 > RBP: ffffc9000045f618 R08: 000000000000000a R09: 0000000000000001 > R10: ffff88807b533828 R11: 0000000000000001 R12: 0000000000000000 > R13: 0000000000000001 R14: 0000000000000000 R15: ffff888078d3c000 > FS: 0000000000000000(0000) GS:ffff88807fd00000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 000055db93c6b2d8 CR3: 0000000078c43005 CR4: 0000000000760ee0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > PKRU: 55555554 > > > _______________________________________________ > Linux-f2fs-devel mailing list > Linux-f2fs-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel > . > _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [f2fs-dev] kernel BUG at fs/f2fs/segment.c:2391 during xfstest generic/204 2019-08-16 8:37 ` [f2fs-dev] kernel BUG at fs/f2fs/segment.c:2391 during xfstest generic/204 Chao Yu @ 2019-08-18 16:03 ` Eric Biggers 2019-08-22 8:29 ` Chao Yu 0 siblings, 1 reply; 4+ messages in thread From: Eric Biggers @ 2019-08-18 16:03 UTC (permalink / raw) To: Chao Yu; +Cc: linux-f2fs-devel On Fri, Aug 16, 2019 at 04:37:35PM +0800, Chao Yu wrote: > Hi Eric, > > Sorry for late reply... > > I only hit ENOSPC error: > > QA output created by 204 > line 82: echo: write error: No space left on device > > What is your mkfs/mount option? > It's: MKFS_OPTIONS -- /dev/vdc MOUNT_OPTIONS -- -o acl,user_xattr /dev/vdc /vdc Here's the output from 'kvm-xfstests -c f2fs/default generic/204': KERNEL: kernel 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 SMP Sun Aug 18 08:57:34 PDT 2019 x86_64 FSTESTVER: blktests 0ee7eba (Wed, 17 Apr 2019 14:41:39 -0700) FSTESTVER: fio fio-3.2 (Fri, 3 Nov 2017 15:23:49 -0600) FSTESTVER: fsverity 2151209 (Fri, 28 Jun 2019 14:34:41 -0700) FSTESTVER: quota 59b280e (Mon, 5 Feb 2018 16:48:22 +0100) FSTESTVER: xfsprogs v5.2.0-rc0-13-g189981e6 (Mon, 12 Aug 2019 10:34:57 -0700) FSTESTVER: xfstests v1.1.0-2641-g43d96649 (Mon, 12 Aug 2019 09:34:21 -0700) FSTESTVER: xfstests-bld b9e2900 (Mon, 29 Jul 2019 00:01:44 -0400) FSTESTCFG: "f2fs/default" FSTESTSET: "generic/204" FSTESTEXC: "" FSTESTOPT: "aex" MNTOPTS: "" CPUS: "2" MEM: "1970.49" total used free shared buff/cache available Mem: 1970 57 1828 8 84 1872 Swap: 0 0 0 [ 3.504915] blk_update_request: operation not supported error, dev vdd, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 [ 4.149867] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory [ 4.151515] NFSD: Using legacy client tracking operations. [ 4.248498] blk_update_request: operation not supported error, dev vdc, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 BEGIN TEST default (1 test): f2fs Sun Aug 18 09:00:05 PDT 2019 DEVICE: /dev/vdd MKFS_OPTIONS: F2FS_MOUNT_OPTIONS: FSTYP -- f2fs PLATFORM -- Linux/x86_64 kvm-xfstests 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 SMP Sun Aug 18 08:57:34 PDT 2019 MKFS_OPTIONS -- /dev/vdc MOUNT_OPTIONS -- -o acl,user_xattr /dev/vdc /vdc [ 5.044711] blk_update_request: operation not supported error, dev vdc, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 generic/204 [09:00:06][ 5.400706] run fstests generic/204 at 2019-08-18 09:00:06 [ 5.710696] blk_update_request: operation not supported error, dev vdc, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 [ 7.122703] ------------[ cut here ]------------ [ 7.123209] kernel BUG at fs/f2fs/segment.c:2391! [ 7.124232] invalid opcode: 0000 [#1] SMP NOPTI [ 7.124675] CPU: 0 PID: 155 Comm: kworker/u4:2 Not tainted 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 [ 7.125548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-20181126_142135-anatol 04/01/2014 [ 7.126565] Workqueue: writeback wb_workfn (flush-254:32) [ 7.127092] RIP: 0010:get_new_segment+0x275/0x280 [ 7.127545] Code: 0b 49 8b 7c 24 58 8d 6b 01 89 ca 0f af f5 e8 42 07 09 00 41 0f af af 68 0d 00 00 89 c7 39 c5 0f 86 20 fe ff ff e9 fc fe ff ff <0f> 0b 0f 0b 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 57 41 56 41 89 [ 7.129349] RSP: 0018:ffffae77401e35c0 EFLAGS: 00010246 [ 7.129850] RAX: 0000000000000031 RBX: 0000000000000000 RCX: 0000000000000031 [ 7.130534] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff942f740a76c0 [ 7.131216] RBP: 0000000000000001 R08: 00000000d4def660 R09: 0000000000000000 [ 7.131935] R10: 0000000000000001 R11: 0000000000000000 R12: ffff942f77a1ca80 [ 7.132614] R13: 0000000000000000 R14: 0000000000000000 R15: ffff942f7788a000 [ 7.133299] FS: 0000000000000000(0000) GS:ffff942f7d800000(0000) knlGS:0000000000000000 [ 7.134100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7.134651] CR2: 00005653dde81238 CR3: 00000000745f4000 CR4: 00000000003406f0 [ 7.135335] Call Trace: [ 7.135580] new_curseg+0xdc/0x160 [ 7.135917] allocate_segment_by_default+0x9a/0x100 [ 7.136357] f2fs_allocate_data_block+0x1ff/0x600 [ 7.136821] ? __write_node_page+0x4c1/0x5d0 [ 7.137239] do_write_page+0x6a/0x120 [ 7.137595] f2fs_do_write_node_page+0x3a/0xb0 [ 7.138026] __write_node_page+0x279/0x5d0 [ 7.138390] f2fs_sync_node_pages+0x614/0x6a0 [ 7.138773] ? __lock_acquire+0x447/0x7c0 [ 7.139144] block_operations+0x260/0x380 [ 7.139562] ? f2fs_write_checkpoint+0xa7/0x540 [ 7.140005] f2fs_write_checkpoint+0x101/0x540 [ 7.140434] f2fs_sync_fs+0xc8/0x1c0 [ 7.140782] f2fs_balance_fs_bg+0x18d/0x2d0 [ 7.141192] ? __lock_acquire+0x447/0x7c0 [ 7.141584] f2fs_write_node_pages+0x54/0x2f0 [ 7.142114] do_writepages+0x41/0xe0 [ 7.142580] __writeback_single_inode+0x54/0x5f0 [ 7.143159] writeback_sb_inodes+0x1fe/0x4c0 [ 7.143671] wb_writeback+0x112/0x470 [ 7.144053] wb_workfn+0xf3/0x5d0 [ 7.144425] ? process_one_work+0x1b0/0x560 [ 7.144827] process_one_work+0x233/0x560 [ 7.145220] worker_thread+0x50/0x3b0 [ 7.145573] kthread+0x108/0x140 [ 7.145890] ? process_one_work+0x560/0x560 [ 7.146290] ? __kthread_create_on_node+0x1a0/0x1a0 [ 7.146772] ret_from_fork+0x27/0x50 [ 7.147851] ---[ end trace c53970fd3fcc6658 ]--- [ 7.148321] RIP: 0010:get_new_segment+0x275/0x280 [ 7.148776] Code: 0b 49 8b 7c 24 58 8d 6b 01 89 ca 0f af f5 e8 42 07 09 00 41 0f af af 68 0d 00 00 89 c7 39 c5 0f 86 20 fe ff ff e9 fc fe ff ff <0f> 0b 0f 0b 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 57 41 56 41 89 [ 7.150587] RSP: 0018:ffffae77401e35c0 EFLAGS: 00010246 [ 7.151096] RAX: 0000000000000031 RBX: 0000000000000000 RCX: 0000000000000031 [ 7.151791] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff942f740a76c0 [ 7.152531] RBP: 0000000000000001 R08: 00000000d4def660 R09: 0000000000000000 [ 7.153218] R10: 0000000000000001 R11: 0000000000000000 R12: ffff942f77a1ca80 [ 7.153900] R13: 0000000000000000 R14: 0000000000000000 R15: ffff942f7788a000 [ 7.154578] FS: 0000000000000000(0000) GS:ffff942f7d800000(0000) knlGS:0000000000000000 [ 7.155400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7.155961] CR2: 00005653dde81238 CR3: 00000000745f4000 CR4: 00000000003406f0 [ 7.156623] ------------[ cut here ]------------ [ 7.157066] WARNING: CPU: 0 PID: 155 at kernel/exit.c:785 do_exit.cold+0x62/0xb9 [ 7.157802] CPU: 0 PID: 155 Comm: kworker/u4:2 Tainted: G D 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 [ 7.158802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-20181126_142135-anatol 04/01/2014 [ 7.159772] Workqueue: writeback wb_workfn (flush-254:32) [ 7.160330] RIP: 0010:do_exit.cold+0x62/0xb9 [ 7.160742] Code: 44 24 10 48 8b 44 24 10 e8 87 b0 a2 00 e9 81 f3 ff ff 48 c7 c7 10 41 27 a5 e8 1e 9d ff ff 48 c7 c7 50 21 26 a5 e8 81 7c 06 00 <0f> 0b e9 0b f3 ff ff 65 48 8b 04 25 c0 5d 01 00 8b 90 b0 03 00 00 [ 7.162517] RSP: 0018:ffffae77401e3ee8 EFLAGS: 00010246 [ 7.163054] RAX: 0000000000000024 RBX: ffff942f778600c0 RCX: 0000000000000006 [ 7.163734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff942f7d9d6600 [ 7.164416] RBP: 000000000000000b R08: 0000000000000000 R09: 0000000000000000 [ 7.165115] R10: 0000000000000000 R11: ffffae77401e3da5 R12: 000000000000000b [ 7.165875] R13: 0000000000000006 R14: ffff942f778600c0 R15: ffffffffa46d2ef5 [ 7.166560] FS: 0000000000000000(0000) GS:ffff942f7d800000(0000) knlGS:0000000000000000 [ 7.167333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7.167888] CR2: 00005653dde81238 CR3: 00000000745f4000 CR4: 00000000003406f0 [ 7.168592] Call Trace: [ 7.168836] ? kthread+0x108/0x140 [ 7.169172] ? get_new_segment+0x275/0x280 [ 7.169567] rewind_stack_do_exit+0x17/0x20 [ 7.169982] irq event stamp: 221765 [ 7.170322] hardirqs last enabled at (221765): [<ffffffffa400168a>] trace_hardirqs_on_thunk+0x1a/0x20 [ 7.171232] hardirqs last disabled at (221763): [<ffffffffa4e002a7>] __do_softirq+0x2a7/0x423 [ 7.172042] softirqs last enabled at (221764): [<ffffffffa4e00321>] __do_softirq+0x321/0x423 [ 7.172857] softirqs last disabled at (217645): [<ffffffffa40b92af>] irq_exit+0x8f/0xc0 [ 7.173673] ---[ end trace c53970fd3fcc6659 ]--- [ 7.174154] BUG: sleeping function called from invalid context at include/linux/percpu-rwsem.h:38 [ 7.175010] in_atomic(): 1, irqs_disabled(): 0, pid: 155, name: kworker/u4:2 [ 7.175679] INFO: lockdep is turned off. [ 7.176106] CPU: 0 PID: 155 Comm: kworker/u4:2 Tainted: G D W 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 [ 7.177106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-20181126_142135-anatol 04/01/2014 [ 7.178071] Workqueue: writeback wb_workfn (flush-254:32) [ 7.178590] Call Trace: [ 7.178841] dump_stack+0x67/0x90 [ 7.179169] ___might_sleep.cold+0x9f/0xaf [ 7.179563] exit_signals+0x1c/0x300 [ 7.179911] do_exit+0xb7/0x7b0 [ 7.180217] ? kthread+0x108/0x140 [ 7.180546] ? get_new_segment+0x275/0x280 [ 7.180945] rewind_stack_do_exit+0x17/0x20 [ 7.181365] note: kworker/u4:2[155] exited with preempt_count 1 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [f2fs-dev] kernel BUG at fs/f2fs/segment.c:2391 during xfstest generic/204 2019-08-18 16:03 ` Eric Biggers @ 2019-08-22 8:29 ` Chao Yu 2019-08-22 9:21 ` Chao Yu 0 siblings, 1 reply; 4+ messages in thread From: Chao Yu @ 2019-08-22 8:29 UTC (permalink / raw) To: Eric Biggers; +Cc: linux-f2fs-devel Hi Eric, I've figured out one patch for this issue, could you have a try? Thanks, On 2019/8/19 0:03, Eric Biggers wrote: > On Fri, Aug 16, 2019 at 04:37:35PM +0800, Chao Yu wrote: >> Hi Eric, >> >> Sorry for late reply... >> >> I only hit ENOSPC error: >> >> QA output created by 204 >> line 82: echo: write error: No space left on device >> >> What is your mkfs/mount option? >> > > It's: > > MKFS_OPTIONS -- /dev/vdc > MOUNT_OPTIONS -- -o acl,user_xattr /dev/vdc /vdc > > Here's the output from 'kvm-xfstests -c f2fs/default generic/204': > > KERNEL: kernel 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 SMP Sun Aug 18 08:57:34 PDT 2019 x86_64 > FSTESTVER: blktests 0ee7eba (Wed, 17 Apr 2019 14:41:39 -0700) > FSTESTVER: fio fio-3.2 (Fri, 3 Nov 2017 15:23:49 -0600) > FSTESTVER: fsverity 2151209 (Fri, 28 Jun 2019 14:34:41 -0700) > FSTESTVER: quota 59b280e (Mon, 5 Feb 2018 16:48:22 +0100) > FSTESTVER: xfsprogs v5.2.0-rc0-13-g189981e6 (Mon, 12 Aug 2019 10:34:57 -0700) > FSTESTVER: xfstests v1.1.0-2641-g43d96649 (Mon, 12 Aug 2019 09:34:21 -0700) > FSTESTVER: xfstests-bld b9e2900 (Mon, 29 Jul 2019 00:01:44 -0400) > FSTESTCFG: "f2fs/default" > FSTESTSET: "generic/204" > FSTESTEXC: "" > FSTESTOPT: "aex" > MNTOPTS: "" > CPUS: "2" > MEM: "1970.49" > total used free shared buff/cache available > Mem: 1970 57 1828 8 84 1872 > Swap: 0 0 0 > [ 3.504915] blk_update_request: operation not supported error, dev vdd, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 > [ 4.149867] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory > [ 4.151515] NFSD: Using legacy client tracking operations. > [ 4.248498] blk_update_request: operation not supported error, dev vdc, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 > BEGIN TEST default (1 test): f2fs Sun Aug 18 09:00:05 PDT 2019 > DEVICE: /dev/vdd > MKFS_OPTIONS: > F2FS_MOUNT_OPTIONS: > FSTYP -- f2fs > PLATFORM -- Linux/x86_64 kvm-xfstests 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 SMP Sun Aug 18 08:57:34 PDT 2019 > MKFS_OPTIONS -- /dev/vdc > MOUNT_OPTIONS -- -o acl,user_xattr /dev/vdc /vdc > > [ 5.044711] blk_update_request: operation not supported error, dev vdc, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 > generic/204 [09:00:06][ 5.400706] run fstests generic/204 at 2019-08-18 09:00:06 > [ 5.710696] blk_update_request: operation not supported error, dev vdc, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 > [ 7.122703] ------------[ cut here ]------------ > [ 7.123209] kernel BUG at fs/f2fs/segment.c:2391! > [ 7.124232] invalid opcode: 0000 [#1] SMP NOPTI > [ 7.124675] CPU: 0 PID: 155 Comm: kworker/u4:2 Not tainted 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 > [ 7.125548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-20181126_142135-anatol 04/01/2014 > [ 7.126565] Workqueue: writeback wb_workfn (flush-254:32) > [ 7.127092] RIP: 0010:get_new_segment+0x275/0x280 > [ 7.127545] Code: 0b 49 8b 7c 24 58 8d 6b 01 89 ca 0f af f5 e8 42 07 09 00 41 0f af af 68 0d 00 00 89 c7 39 c5 0f 86 20 fe ff ff e9 fc fe ff ff <0f> 0b 0f 0b 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 57 41 56 41 89 > [ 7.129349] RSP: 0018:ffffae77401e35c0 EFLAGS: 00010246 > [ 7.129850] RAX: 0000000000000031 RBX: 0000000000000000 RCX: 0000000000000031 > [ 7.130534] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff942f740a76c0 > [ 7.131216] RBP: 0000000000000001 R08: 00000000d4def660 R09: 0000000000000000 > [ 7.131935] R10: 0000000000000001 R11: 0000000000000000 R12: ffff942f77a1ca80 > [ 7.132614] R13: 0000000000000000 R14: 0000000000000000 R15: ffff942f7788a000 > [ 7.133299] FS: 0000000000000000(0000) GS:ffff942f7d800000(0000) knlGS:0000000000000000 > [ 7.134100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 7.134651] CR2: 00005653dde81238 CR3: 00000000745f4000 CR4: 00000000003406f0 > [ 7.135335] Call Trace: > [ 7.135580] new_curseg+0xdc/0x160 > [ 7.135917] allocate_segment_by_default+0x9a/0x100 > [ 7.136357] f2fs_allocate_data_block+0x1ff/0x600 > [ 7.136821] ? __write_node_page+0x4c1/0x5d0 > [ 7.137239] do_write_page+0x6a/0x120 > [ 7.137595] f2fs_do_write_node_page+0x3a/0xb0 > [ 7.138026] __write_node_page+0x279/0x5d0 > [ 7.138390] f2fs_sync_node_pages+0x614/0x6a0 > [ 7.138773] ? __lock_acquire+0x447/0x7c0 > [ 7.139144] block_operations+0x260/0x380 > [ 7.139562] ? f2fs_write_checkpoint+0xa7/0x540 > [ 7.140005] f2fs_write_checkpoint+0x101/0x540 > [ 7.140434] f2fs_sync_fs+0xc8/0x1c0 > [ 7.140782] f2fs_balance_fs_bg+0x18d/0x2d0 > [ 7.141192] ? __lock_acquire+0x447/0x7c0 > [ 7.141584] f2fs_write_node_pages+0x54/0x2f0 > [ 7.142114] do_writepages+0x41/0xe0 > [ 7.142580] __writeback_single_inode+0x54/0x5f0 > [ 7.143159] writeback_sb_inodes+0x1fe/0x4c0 > [ 7.143671] wb_writeback+0x112/0x470 > [ 7.144053] wb_workfn+0xf3/0x5d0 > [ 7.144425] ? process_one_work+0x1b0/0x560 > [ 7.144827] process_one_work+0x233/0x560 > [ 7.145220] worker_thread+0x50/0x3b0 > [ 7.145573] kthread+0x108/0x140 > [ 7.145890] ? process_one_work+0x560/0x560 > [ 7.146290] ? __kthread_create_on_node+0x1a0/0x1a0 > [ 7.146772] ret_from_fork+0x27/0x50 > [ 7.147851] ---[ end trace c53970fd3fcc6658 ]--- > [ 7.148321] RIP: 0010:get_new_segment+0x275/0x280 > [ 7.148776] Code: 0b 49 8b 7c 24 58 8d 6b 01 89 ca 0f af f5 e8 42 07 09 00 41 0f af af 68 0d 00 00 89 c7 39 c5 0f 86 20 fe ff ff e9 fc fe ff ff <0f> 0b 0f 0b 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 57 41 56 41 89 > [ 7.150587] RSP: 0018:ffffae77401e35c0 EFLAGS: 00010246 > [ 7.151096] RAX: 0000000000000031 RBX: 0000000000000000 RCX: 0000000000000031 > [ 7.151791] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff942f740a76c0 > [ 7.152531] RBP: 0000000000000001 R08: 00000000d4def660 R09: 0000000000000000 > [ 7.153218] R10: 0000000000000001 R11: 0000000000000000 R12: ffff942f77a1ca80 > [ 7.153900] R13: 0000000000000000 R14: 0000000000000000 R15: ffff942f7788a000 > [ 7.154578] FS: 0000000000000000(0000) GS:ffff942f7d800000(0000) knlGS:0000000000000000 > [ 7.155400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 7.155961] CR2: 00005653dde81238 CR3: 00000000745f4000 CR4: 00000000003406f0 > [ 7.156623] ------------[ cut here ]------------ > [ 7.157066] WARNING: CPU: 0 PID: 155 at kernel/exit.c:785 do_exit.cold+0x62/0xb9 > [ 7.157802] CPU: 0 PID: 155 Comm: kworker/u4:2 Tainted: G D 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 > [ 7.158802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-20181126_142135-anatol 04/01/2014 > [ 7.159772] Workqueue: writeback wb_workfn (flush-254:32) > [ 7.160330] RIP: 0010:do_exit.cold+0x62/0xb9 > [ 7.160742] Code: 44 24 10 48 8b 44 24 10 e8 87 b0 a2 00 e9 81 f3 ff ff 48 c7 c7 10 41 27 a5 e8 1e 9d ff ff 48 c7 c7 50 21 26 a5 e8 81 7c 06 00 <0f> 0b e9 0b f3 ff ff 65 48 8b 04 25 c0 5d 01 00 8b 90 b0 03 00 00 > [ 7.162517] RSP: 0018:ffffae77401e3ee8 EFLAGS: 00010246 > [ 7.163054] RAX: 0000000000000024 RBX: ffff942f778600c0 RCX: 0000000000000006 > [ 7.163734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff942f7d9d6600 > [ 7.164416] RBP: 000000000000000b R08: 0000000000000000 R09: 0000000000000000 > [ 7.165115] R10: 0000000000000000 R11: ffffae77401e3da5 R12: 000000000000000b > [ 7.165875] R13: 0000000000000006 R14: ffff942f778600c0 R15: ffffffffa46d2ef5 > [ 7.166560] FS: 0000000000000000(0000) GS:ffff942f7d800000(0000) knlGS:0000000000000000 > [ 7.167333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 7.167888] CR2: 00005653dde81238 CR3: 00000000745f4000 CR4: 00000000003406f0 > [ 7.168592] Call Trace: > [ 7.168836] ? kthread+0x108/0x140 > [ 7.169172] ? get_new_segment+0x275/0x280 > [ 7.169567] rewind_stack_do_exit+0x17/0x20 > [ 7.169982] irq event stamp: 221765 > [ 7.170322] hardirqs last enabled at (221765): [<ffffffffa400168a>] trace_hardirqs_on_thunk+0x1a/0x20 > [ 7.171232] hardirqs last disabled at (221763): [<ffffffffa4e002a7>] __do_softirq+0x2a7/0x423 > [ 7.172042] softirqs last enabled at (221764): [<ffffffffa4e00321>] __do_softirq+0x321/0x423 > [ 7.172857] softirqs last disabled at (217645): [<ffffffffa40b92af>] irq_exit+0x8f/0xc0 > [ 7.173673] ---[ end trace c53970fd3fcc6659 ]--- > [ 7.174154] BUG: sleeping function called from invalid context at include/linux/percpu-rwsem.h:38 > [ 7.175010] in_atomic(): 1, irqs_disabled(): 0, pid: 155, name: kworker/u4:2 > [ 7.175679] INFO: lockdep is turned off. > [ 7.176106] CPU: 0 PID: 155 Comm: kworker/u4:2 Tainted: G D W 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 > [ 7.177106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-20181126_142135-anatol 04/01/2014 > [ 7.178071] Workqueue: writeback wb_workfn (flush-254:32) > [ 7.178590] Call Trace: > [ 7.178841] dump_stack+0x67/0x90 > [ 7.179169] ___might_sleep.cold+0x9f/0xaf > [ 7.179563] exit_signals+0x1c/0x300 > [ 7.179911] do_exit+0xb7/0x7b0 > [ 7.180217] ? kthread+0x108/0x140 > [ 7.180546] ? get_new_segment+0x275/0x280 > [ 7.180945] rewind_stack_do_exit+0x17/0x20 > [ 7.181365] note: kworker/u4:2[155] exited with preempt_count 1 > . > _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [f2fs-dev] kernel BUG at fs/f2fs/segment.c:2391 during xfstest generic/204 2019-08-22 8:29 ` Chao Yu @ 2019-08-22 9:21 ` Chao Yu 0 siblings, 0 replies; 4+ messages in thread From: Chao Yu @ 2019-08-22 9:21 UTC (permalink / raw) To: Eric Biggers; +Cc: linux-f2fs-devel On 2019/8/22 16:29, Chao Yu wrote: > Hi Eric, > > I've figured out one patch for this issue, could you have a try? Could you only check 204? I just hit deadlock in 051, not sure it is caused by this patch. > > Thanks, > > On 2019/8/19 0:03, Eric Biggers wrote: >> On Fri, Aug 16, 2019 at 04:37:35PM +0800, Chao Yu wrote: >>> Hi Eric, >>> >>> Sorry for late reply... >>> >>> I only hit ENOSPC error: >>> >>> QA output created by 204 >>> line 82: echo: write error: No space left on device >>> >>> What is your mkfs/mount option? >>> >> >> It's: >> >> MKFS_OPTIONS -- /dev/vdc >> MOUNT_OPTIONS -- -o acl,user_xattr /dev/vdc /vdc >> >> Here's the output from 'kvm-xfstests -c f2fs/default generic/204': >> >> KERNEL: kernel 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 SMP Sun Aug 18 08:57:34 PDT 2019 x86_64 >> FSTESTVER: blktests 0ee7eba (Wed, 17 Apr 2019 14:41:39 -0700) >> FSTESTVER: fio fio-3.2 (Fri, 3 Nov 2017 15:23:49 -0600) >> FSTESTVER: fsverity 2151209 (Fri, 28 Jun 2019 14:34:41 -0700) >> FSTESTVER: quota 59b280e (Mon, 5 Feb 2018 16:48:22 +0100) >> FSTESTVER: xfsprogs v5.2.0-rc0-13-g189981e6 (Mon, 12 Aug 2019 10:34:57 -0700) >> FSTESTVER: xfstests v1.1.0-2641-g43d96649 (Mon, 12 Aug 2019 09:34:21 -0700) >> FSTESTVER: xfstests-bld b9e2900 (Mon, 29 Jul 2019 00:01:44 -0400) >> FSTESTCFG: "f2fs/default" >> FSTESTSET: "generic/204" >> FSTESTEXC: "" >> FSTESTOPT: "aex" >> MNTOPTS: "" >> CPUS: "2" >> MEM: "1970.49" >> total used free shared buff/cache available >> Mem: 1970 57 1828 8 84 1872 >> Swap: 0 0 0 >> [ 3.504915] blk_update_request: operation not supported error, dev vdd, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 >> [ 4.149867] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory >> [ 4.151515] NFSD: Using legacy client tracking operations. >> [ 4.248498] blk_update_request: operation not supported error, dev vdc, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 >> BEGIN TEST default (1 test): f2fs Sun Aug 18 09:00:05 PDT 2019 >> DEVICE: /dev/vdd >> MKFS_OPTIONS: >> F2FS_MOUNT_OPTIONS: >> FSTYP -- f2fs >> PLATFORM -- Linux/x86_64 kvm-xfstests 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 SMP Sun Aug 18 08:57:34 PDT 2019 >> MKFS_OPTIONS -- /dev/vdc >> MOUNT_OPTIONS -- -o acl,user_xattr /dev/vdc /vdc >> >> [ 5.044711] blk_update_request: operation not supported error, dev vdc, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 >> generic/204 [09:00:06][ 5.400706] run fstests generic/204 at 2019-08-18 09:00:06 >> [ 5.710696] blk_update_request: operation not supported error, dev vdc, sector 8388606 op 0x3:(DISCARD) flags 0x0 phys_seg 2 prio class 0 >> [ 7.122703] ------------[ cut here ]------------ >> [ 7.123209] kernel BUG at fs/f2fs/segment.c:2391! >> [ 7.124232] invalid opcode: 0000 [#1] SMP NOPTI >> [ 7.124675] CPU: 0 PID: 155 Comm: kworker/u4:2 Not tainted 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 >> [ 7.125548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-20181126_142135-anatol 04/01/2014 >> [ 7.126565] Workqueue: writeback wb_workfn (flush-254:32) >> [ 7.127092] RIP: 0010:get_new_segment+0x275/0x280 >> [ 7.127545] Code: 0b 49 8b 7c 24 58 8d 6b 01 89 ca 0f af f5 e8 42 07 09 00 41 0f af af 68 0d 00 00 89 c7 39 c5 0f 86 20 fe ff ff e9 fc fe ff ff <0f> 0b 0f 0b 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 57 41 56 41 89 >> [ 7.129349] RSP: 0018:ffffae77401e35c0 EFLAGS: 00010246 >> [ 7.129850] RAX: 0000000000000031 RBX: 0000000000000000 RCX: 0000000000000031 >> [ 7.130534] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff942f740a76c0 >> [ 7.131216] RBP: 0000000000000001 R08: 00000000d4def660 R09: 0000000000000000 >> [ 7.131935] R10: 0000000000000001 R11: 0000000000000000 R12: ffff942f77a1ca80 >> [ 7.132614] R13: 0000000000000000 R14: 0000000000000000 R15: ffff942f7788a000 >> [ 7.133299] FS: 0000000000000000(0000) GS:ffff942f7d800000(0000) knlGS:0000000000000000 >> [ 7.134100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 7.134651] CR2: 00005653dde81238 CR3: 00000000745f4000 CR4: 00000000003406f0 >> [ 7.135335] Call Trace: >> [ 7.135580] new_curseg+0xdc/0x160 >> [ 7.135917] allocate_segment_by_default+0x9a/0x100 >> [ 7.136357] f2fs_allocate_data_block+0x1ff/0x600 >> [ 7.136821] ? __write_node_page+0x4c1/0x5d0 >> [ 7.137239] do_write_page+0x6a/0x120 >> [ 7.137595] f2fs_do_write_node_page+0x3a/0xb0 >> [ 7.138026] __write_node_page+0x279/0x5d0 >> [ 7.138390] f2fs_sync_node_pages+0x614/0x6a0 >> [ 7.138773] ? __lock_acquire+0x447/0x7c0 >> [ 7.139144] block_operations+0x260/0x380 >> [ 7.139562] ? f2fs_write_checkpoint+0xa7/0x540 >> [ 7.140005] f2fs_write_checkpoint+0x101/0x540 >> [ 7.140434] f2fs_sync_fs+0xc8/0x1c0 >> [ 7.140782] f2fs_balance_fs_bg+0x18d/0x2d0 >> [ 7.141192] ? __lock_acquire+0x447/0x7c0 >> [ 7.141584] f2fs_write_node_pages+0x54/0x2f0 >> [ 7.142114] do_writepages+0x41/0xe0 >> [ 7.142580] __writeback_single_inode+0x54/0x5f0 >> [ 7.143159] writeback_sb_inodes+0x1fe/0x4c0 >> [ 7.143671] wb_writeback+0x112/0x470 >> [ 7.144053] wb_workfn+0xf3/0x5d0 >> [ 7.144425] ? process_one_work+0x1b0/0x560 >> [ 7.144827] process_one_work+0x233/0x560 >> [ 7.145220] worker_thread+0x50/0x3b0 >> [ 7.145573] kthread+0x108/0x140 >> [ 7.145890] ? process_one_work+0x560/0x560 >> [ 7.146290] ? __kthread_create_on_node+0x1a0/0x1a0 >> [ 7.146772] ret_from_fork+0x27/0x50 >> [ 7.147851] ---[ end trace c53970fd3fcc6658 ]--- >> [ 7.148321] RIP: 0010:get_new_segment+0x275/0x280 >> [ 7.148776] Code: 0b 49 8b 7c 24 58 8d 6b 01 89 ca 0f af f5 e8 42 07 09 00 41 0f af af 68 0d 00 00 89 c7 39 c5 0f 86 20 fe ff ff e9 fc fe ff ff <0f> 0b 0f 0b 0f 1f 80 00 00 00 00 0f 1f 44 00 00 41 57 41 56 41 89 >> [ 7.150587] RSP: 0018:ffffae77401e35c0 EFLAGS: 00010246 >> [ 7.151096] RAX: 0000000000000031 RBX: 0000000000000000 RCX: 0000000000000031 >> [ 7.151791] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff942f740a76c0 >> [ 7.152531] RBP: 0000000000000001 R08: 00000000d4def660 R09: 0000000000000000 >> [ 7.153218] R10: 0000000000000001 R11: 0000000000000000 R12: ffff942f77a1ca80 >> [ 7.153900] R13: 0000000000000000 R14: 0000000000000000 R15: ffff942f7788a000 >> [ 7.154578] FS: 0000000000000000(0000) GS:ffff942f7d800000(0000) knlGS:0000000000000000 >> [ 7.155400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 7.155961] CR2: 00005653dde81238 CR3: 00000000745f4000 CR4: 00000000003406f0 >> [ 7.156623] ------------[ cut here ]------------ >> [ 7.157066] WARNING: CPU: 0 PID: 155 at kernel/exit.c:785 do_exit.cold+0x62/0xb9 >> [ 7.157802] CPU: 0 PID: 155 Comm: kworker/u4:2 Tainted: G D 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 >> [ 7.158802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-20181126_142135-anatol 04/01/2014 >> [ 7.159772] Workqueue: writeback wb_workfn (flush-254:32) >> [ 7.160330] RIP: 0010:do_exit.cold+0x62/0xb9 >> [ 7.160742] Code: 44 24 10 48 8b 44 24 10 e8 87 b0 a2 00 e9 81 f3 ff ff 48 c7 c7 10 41 27 a5 e8 1e 9d ff ff 48 c7 c7 50 21 26 a5 e8 81 7c 06 00 <0f> 0b e9 0b f3 ff ff 65 48 8b 04 25 c0 5d 01 00 8b 90 b0 03 00 00 >> [ 7.162517] RSP: 0018:ffffae77401e3ee8 EFLAGS: 00010246 >> [ 7.163054] RAX: 0000000000000024 RBX: ffff942f778600c0 RCX: 0000000000000006 >> [ 7.163734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff942f7d9d6600 >> [ 7.164416] RBP: 000000000000000b R08: 0000000000000000 R09: 0000000000000000 >> [ 7.165115] R10: 0000000000000000 R11: ffffae77401e3da5 R12: 000000000000000b >> [ 7.165875] R13: 0000000000000006 R14: ffff942f778600c0 R15: ffffffffa46d2ef5 >> [ 7.166560] FS: 0000000000000000(0000) GS:ffff942f7d800000(0000) knlGS:0000000000000000 >> [ 7.167333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 7.167888] CR2: 00005653dde81238 CR3: 00000000745f4000 CR4: 00000000003406f0 >> [ 7.168592] Call Trace: >> [ 7.168836] ? kthread+0x108/0x140 >> [ 7.169172] ? get_new_segment+0x275/0x280 >> [ 7.169567] rewind_stack_do_exit+0x17/0x20 >> [ 7.169982] irq event stamp: 221765 >> [ 7.170322] hardirqs last enabled at (221765): [<ffffffffa400168a>] trace_hardirqs_on_thunk+0x1a/0x20 >> [ 7.171232] hardirqs last disabled at (221763): [<ffffffffa4e002a7>] __do_softirq+0x2a7/0x423 >> [ 7.172042] softirqs last enabled at (221764): [<ffffffffa4e00321>] __do_softirq+0x321/0x423 >> [ 7.172857] softirqs last disabled at (217645): [<ffffffffa40b92af>] irq_exit+0x8f/0xc0 >> [ 7.173673] ---[ end trace c53970fd3fcc6659 ]--- >> [ 7.174154] BUG: sleeping function called from invalid context at include/linux/percpu-rwsem.h:38 >> [ 7.175010] in_atomic(): 1, irqs_disabled(): 0, pid: 155, name: kworker/u4:2 >> [ 7.175679] INFO: lockdep is turned off. >> [ 7.176106] CPU: 0 PID: 155 Comm: kworker/u4:2 Tainted: G D W 5.3.0-rc4-xfstests-00142-g8fde2832bd0b #1 >> [ 7.177106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-20181126_142135-anatol 04/01/2014 >> [ 7.178071] Workqueue: writeback wb_workfn (flush-254:32) >> [ 7.178590] Call Trace: >> [ 7.178841] dump_stack+0x67/0x90 >> [ 7.179169] ___might_sleep.cold+0x9f/0xaf >> [ 7.179563] exit_signals+0x1c/0x300 >> [ 7.179911] do_exit+0xb7/0x7b0 >> [ 7.180217] ? kthread+0x108/0x140 >> [ 7.180546] ? get_new_segment+0x275/0x280 >> [ 7.180945] rewind_stack_do_exit+0x17/0x20 >> [ 7.181365] note: kworker/u4:2[155] exited with preempt_count 1 >> . >> > > > _______________________________________________ > Linux-f2fs-devel mailing list > Linux-f2fs-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel > . > _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-08-22 9:21 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20190729174140.GA169027@gmail.com>
2019-08-16 8:37 ` [f2fs-dev] kernel BUG at fs/f2fs/segment.c:2391 during xfstest generic/204 Chao Yu
2019-08-18 16:03 ` Eric Biggers
2019-08-22 8:29 ` Chao Yu
2019-08-22 9:21 ` Chao Yu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).