* PROBLEM: bcache related kernel BUG() since Linux 5.12
@ 2021-05-15 19:06 Thorsten Knabe
2021-05-16 8:42 ` Matthias Ferdinand
2021-05-31 9:37 ` Rolf Fokkens
0 siblings, 2 replies; 15+ messages in thread
From: Thorsten Knabe @ 2021-05-15 19:06 UTC (permalink / raw)
To: linux-bcache
Hello.
Starting with Linux 5.12 bcache triggers a BUG() after a few minutes of
usage.
Linux up to 5.11.x is not affected by this bug.
Environment:
- Debian 10 AMD 64
- Kernel 5.12 - 5.12.4
- Filesystem ext4
- Backing device: degraded software RAID-6 (MD) with 3 of 4 disks active
(unsure if the degraded RAID-6 has an effect or not)
- Cache device: Single SSD
Kernel log:
May 12 20:22:24 tek04 kernel: nr_vecs=472
May 12 20:22:24 tek04 kernel: ------------[ cut here ]------------
May 12 20:22:24 tek04 kernel: kernel BUG at block/bio.c:53!
May 12 20:22:24 tek04 kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI
May 12 20:22:24 tek04 kernel: CPU: 1 PID: 1670 Comm: grep Tainted: G
I 5.12.3 #2
May 12 20:22:24 tek04 kernel: Hardware name: To Be Filled By O.E.M. To
Be Filled By O.E.M./X58 Deluxe, BIOS P2.20 10/30/2009
May 12 20:22:24 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
May 12 20:22:24 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
e8 d4 80 fe ff
May 12 20:22:24 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
May 12 20:22:24 tek04 kernel: RAX: 000000000000000b RBX:
ffffc9000274b764 RCX: 0000000000000000
May 12 20:22:24 tek04 kernel: bch_count_backing_io_errors: 1 callbacks
suppressed
May 12 20:22:24 tek04 kernel: bcache: bch_count_backing_io_errors() md1:
Read-ahead I/O failed on backing device, ignore
May 12 20:22:24 tek04 kernel: RDX: ffff888333c5e400 RSI:
ffff888333c57480 RDI: ffff888333c57480
May 12 20:22:24 tek04 kernel: RBP: 0000000000000800 R08:
0000000000000000 R09: c0000000ffffdfff
May 12 20:22:24 tek04 kernel: R10: ffffc9000274b580 R11:
ffffc9000274b578 R12: ffff8881170f0118
May 12 20:22:24 tek04 kernel: R13: ffff888109911b00 R14:
ffff8881170f00d0 R15: 0000000000000800
May 12 20:22:24 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
GS:ffff888333c40000(0000) knlGS:0000000000000000
May 12 20:22:24 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
May 12 20:22:24 tek04 kernel: CR2: 00005628963f4fd0 CR3:
000000016c43c000 CR4: 00000000000006e0
May 12 20:22:24 tek04 kernel: Call Trace:
May 12 20:22:24 tek04 kernel: bvec_alloc+0x22/0x90
May 12 20:22:24 tek04 kernel: bio_alloc_bioset+0x176/0x230
May 12 20:22:24 tek04 kernel: cached_dev_cache_miss+0x1a8/0x300
May 12 20:22:24 tek04 kernel: cache_lookup_fn+0x110/0x2e0
May 12 20:22:24 tek04 kernel: ? bch_ptr_invalid+0x10/0x10
May 12 20:22:24 tek04 kernel: ? bch_btree_iter_next_filter+0x1af/0x2d0
May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0x69/0x160
May 12 20:22:24 tek04 kernel: ? __bch_bset_search+0x315/0x440
May 12 20:22:24 tek04 kernel: ? downgrade_write+0xb0/0xb0
May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0xcf/0x160
May 12 20:22:24 tek04 kernel: ? raid5_make_request+0x5c4/0xaa0
May 12 20:22:24 tek04 kernel: ? recalibrate_cpu_khz+0x10/0x10
May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x30/0x400
May 12 20:22:24 tek04 kernel: ? rwsem_wake.isra.11+0x80/0x80
May 12 20:22:24 tek04 kernel: bch_btree_map_keys+0xf2/0x140
May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
May 12 20:22:24 tek04 kernel: cache_lookup+0xb1/0x190
May 12 20:22:24 tek04 kernel: cached_dev_submit_bio+0x9ab/0xc90
May 12 20:22:24 tek04 kernel: ? submit_bio_checks+0x197/0x4a0
May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x3b7/0x400
May 12 20:22:24 tek04 kernel: submit_bio_noacct+0x10e/0x4c0
May 12 20:22:24 tek04 kernel: submit_bio+0x2e/0x160
May 12 20:22:24 tek04 kernel: ? xa_load+0x66/0x70
May 12 20:22:24 tek04 kernel: ? bio_add_page+0x2f/0x70
May 12 20:22:24 tek04 kernel: ext4_mpage_readpages+0x1ae/0xa00
May 12 20:22:24 tek04 kernel: ? __mod_lruvec_state+0x29/0x60
May 12 20:22:24 tek04 kernel: read_pages+0x78/0x1d0
May 12 20:22:24 tek04 kernel: page_cache_ra_unbounded+0x127/0x1b0
May 12 20:22:24 tek04 kernel: filemap_get_pages+0x1d0/0x4a0
May 12 20:22:24 tek04 kernel: filemap_read+0x91/0x2d0
May 12 20:22:24 tek04 kernel: new_sync_read+0x103/0x180
May 12 20:22:24 tek04 kernel: vfs_read+0x11b/0x1b0
May 12 20:22:24 tek04 kernel: ksys_read+0x55/0xd0
May 12 20:22:24 tek04 kernel: do_syscall_64+0x33/0x80
May 12 20:22:24 tek04 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
May 12 20:22:24 tek04 kernel: RIP: 0033:0x7f3a1cb89461
May 12 20:22:24 tek04 kernel: Code: fe ff ff 50 48 8d 3d fe d0 09 00 e8
e9 03 02 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 99 62 0d 00 8b 00 85 c0
75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54
49 89 d4 55 48
May 12 20:22:24 tek04 kernel: RSP: 002b:00007fff4052fff8 EFLAGS:
00000246 ORIG_RAX: 0000000000000000
May 12 20:22:24 tek04 kernel: RAX: ffffffffffffffda RBX:
0000000000018000 RCX: 00007f3a1cb89461
May 12 20:22:24 tek04 kernel: RDX: 0000000000018000 RSI:
000055ff0d01a000 RDI: 0000000000000004
May 12 20:22:24 tek04 kernel: RBP: 0000000000018000 R08:
0000000000000002 R09: 000055ff0d0194b0
May 12 20:22:24 tek04 kernel: R10: 0000000000000000 R11:
0000000000000246 R12: 000055ff0d01a000
May 12 20:22:24 tek04 kernel: R13: 0000000000000004 R14:
000055ff0d0194b0 R15: 0000000000000004
May 12 20:22:24 tek04 kernel: Modules linked in: cmac bnep
intel_powerclamp snd_hda_codec_realtek snd_hda_codec_generic btusb
ledtrig_audio snd_hda_codec_hdmi btrtl kvm_intel snd_hda_intel btbcm
snd_intel_dspcfg btintel kvm snd_hda_codec irqbypass bluetooth serio_raw
pcspkr hfcpci snd_hda_core iTCO_wdt evdev input_leds joydev sg snd_hwdep
intel_pmc_bxt ecdh_generic rfkill iTCO_vendor_support mISDN_core ecc
snd_pcm snd_timer tiny_power_button snd soundcore button i7core_edac
acpi_cpufreq wmi nft_counter nf_log_ipv6 nf_log_ipv
May 12 20:22:24 tek04 kernel: ---[ end trace 9a03f30c7b4aa246 ]---
May 12 20:22:25 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
May 12 20:22:25 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
e8 d4 80 fe ff
May 12 20:22:25 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
May 12 20:22:25 tek04 kernel: RAX: 000000000000000b RBX:
ffffc9000274b764 RCX: 0000000000000000
May 12 20:22:25 tek04 kernel: RDX: ffff888333c5e400 RSI:
ffff888333c57480 RDI: ffff888333c57480
May 12 20:22:25 tek04 kernel: RBP: 0000000000000800 R08:
0000000000000000 R09: c0000000ffffdfff
May 12 20:22:25 tek04 kernel: R10: ffffc9000274b580 R11:
ffffc9000274b578 R12: ffff8881170f0118
May 12 20:22:25 tek04 kernel: R13: ffff888109911b00 R14:
ffff8881170f00d0 R15: 0000000000000800
May 12 20:22:25 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
GS:ffff888333c40000(0000) knlGS:0000000000000000
May 12 20:22:25 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
May 12 20:22:25 tek04 kernel: CR2: 00005628963f4fd0 CR3:
000000016c43c000 CR4: 00000000000006e0
A printk has been added to line 52 of block/bio.c to dump the nr_vecs
variable to the kernel log before the BUG(). Obviously nr_vecs (472
logged) is bigger than expected by bvec_alloc/bio_alloc_bioset (max
256), which finally triggers the BUG().
Removing the BUG() from line 52 of block/bio.c, thus basically restoring
the Linux 5.11.x behavior of bvec_alloc/bio_alloc_bioset to just return
NULL, when nr_vecs is too big seems to resolve the issue.
Regards
Thorsten
--
___
| | / E-Mail: linux@thorsten-knabe.de
|horsten |/\nabe WWW: http://linux.thorsten-knabe.de
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-05-15 19:06 PROBLEM: bcache related kernel BUG() since Linux 5.12 Thorsten Knabe
@ 2021-05-16 8:42 ` Matthias Ferdinand
2021-05-16 14:43 ` Coly Li
2021-05-31 9:37 ` Rolf Fokkens
1 sibling, 1 reply; 15+ messages in thread
From: Matthias Ferdinand @ 2021-05-16 8:42 UTC (permalink / raw)
To: Thorsten Knabe; +Cc: linux-bcache
On Sat, May 15, 2021 at 09:06:07PM +0200, Thorsten Knabe wrote:
> Hello.
>
> Starting with Linux 5.12 bcache triggers a BUG() after a few minutes of
> usage.
>
> Linux up to 5.11.x is not affected by this bug.
>
> Environment:
> - Debian 10 AMD 64
> - Kernel 5.12 - 5.12.4
> - Filesystem ext4
> - Backing device: degraded software RAID-6 (MD) with 3 of 4 disks active
> (unsure if the degraded RAID-6 has an effect or not)
> - Cache device: Single SSD
Sorry I can't immediately help with bcache, but for DRBD, there was a
similar problem with DRBD on degraded md-raid fixed just recently:
https://lists.linbit.com/pipermail/drbd-user/2021-May/025904.html
Although they had silent data corruption AFAICT, not a loud BUG(), and
they stated problems started with kernel 4.3.
For DRBD it had to do with split BIOs and readahead, which degraded
md-raid may or may not fail, and missing a fail on parts of a split-up
readahead BIO.
Matthias
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-05-16 8:42 ` Matthias Ferdinand
@ 2021-05-16 14:43 ` Coly Li
0 siblings, 0 replies; 15+ messages in thread
From: Coly Li @ 2021-05-16 14:43 UTC (permalink / raw)
To: Matthias Ferdinand, Thorsten Knabe; +Cc: linux-bcache
[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]
On 5/16/21 4:42 PM, Matthias Ferdinand wrote:
> On Sat, May 15, 2021 at 09:06:07PM +0200, Thorsten Knabe wrote:
>> Hello.
>>
>> Starting with Linux 5.12 bcache triggers a BUG() after a few minutes of
>> usage.
>>
>> Linux up to 5.11.x is not affected by this bug.
>>
>> Environment:
>> - Debian 10 AMD 64
>> - Kernel 5.12 - 5.12.4
>> - Filesystem ext4
>> - Backing device: degraded software RAID-6 (MD) with 3 of 4 disks active
>> (unsure if the degraded RAID-6 has an effect or not)
>> - Cache device: Single SSD
>
> Sorry I can't immediately help with bcache, but for DRBD, there was a
> similar problem with DRBD on degraded md-raid fixed just recently:
>
> https://lists.linbit.com/pipermail/drbd-user/2021-May/025904.html
>
> Although they had silent data corruption AFAICT, not a loud BUG(), and
> they stated problems started with kernel 4.3.
>
> For DRBD it had to do with split BIOs and readahead, which degraded
> md-raid may or may not fail, and missing a fail on parts of a split-up
> readahead BIO.
>
> Matthias
>
This is caused by a hidden issue which is triggered by the bio code
change in v5.12.
The attached patch can help to avoid the panic, and the finally fixes
are under testing and will be posted very soon.
Coly Li
[-- Attachment #2: 0001-bcache-avoid-oversized-bio_alloc_bioset-call-in-cach.patch --]
[-- Type: text/plain, Size: 2265 bytes --]
From 6f2edee7100efabf2ccccb84e4a92ccbfbddd8c5 Mon Sep 17 00:00:00 2001
From: Coly Li <colyli@suse.de>
Date: Thu, 6 May 2021 10:38:41 +0800
Subject: [PATCH] bcache: avoid oversized bio_alloc_bioset() call in
cached_dev_cache_miss()
Since Linux v5.12, calling bio_alloc_bioset() with oversized bio vectors
number will cause a BUG() panic in biovec_slab(). There are 2 locations
in bcache code calling bio_alloc_bioset(), and only the location in
cached_dev_cache_miss() has such potential oversized risk.
In cached_dev_cache_miss() the bio vectors number is calculated by
DIV_ROUND_UP(s->insert_bio_sectors, PAGE_SECTORS), this patch checks the
calculated result, if it is larger than BIO_MAX_VECS, then give up the
allocation of cache_bio and sending request to backing device directly.
By this restriction, the potential BUG() panic can be avoided from the
cache missing code path.
Signed-off-by: Coly Li <colyli@suse.de>
---
drivers/md/bcache/request.c | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c
index 29c231758293..a657d3a2b624 100644
--- a/drivers/md/bcache/request.c
+++ b/drivers/md/bcache/request.c
@@ -879,7 +879,7 @@ static void cached_dev_read_done_bh(struct closure *cl)
static int cached_dev_cache_miss(struct btree *b, struct search *s,
struct bio *bio, unsigned int sectors)
{
- int ret = MAP_CONTINUE;
+ int ret = MAP_CONTINUE, nr_iovecs = 0;
unsigned int reada = 0;
struct cached_dev *dc = container_of(s->d, struct cached_dev, disk);
struct bio *miss, *cache_bio;
@@ -916,9 +916,14 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s,
/* btree_search_recurse()'s btree iterator is no good anymore */
ret = miss == bio ? MAP_DONE : -EINTR;
- cache_bio = bio_alloc_bioset(GFP_NOWAIT,
- DIV_ROUND_UP(s->insert_bio_sectors, PAGE_SECTORS),
- &dc->disk.bio_split);
+ nr_iovecs = DIV_ROUND_UP(s->insert_bio_sectors, PAGE_SECTORS);
+ if (nr_iovecs > BIO_MAX_VECS) {
+ pr_warn("inserting bio is too large: %d iovecs, not intsert.\n",
+ nr_iovecs);
+ goto out_submit;
+ }
+ cache_bio = bio_alloc_bioset(GFP_NOWAIT, nr_iovecs,
+ &dc->disk.bio_split);
if (!cache_bio)
goto out_submit;
--
2.26.2
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-05-15 19:06 PROBLEM: bcache related kernel BUG() since Linux 5.12 Thorsten Knabe
2021-05-16 8:42 ` Matthias Ferdinand
@ 2021-05-31 9:37 ` Rolf Fokkens
2021-06-01 15:34 ` Coly Li
1 sibling, 1 reply; 15+ messages in thread
From: Rolf Fokkens @ 2021-05-31 9:37 UTC (permalink / raw)
To: Thorsten Knabe, linux-bcache
Same here, more details: https://bugzilla.redhat.com/show_bug.cgi?id=1965809
On 5/15/21 9:06 PM, Thorsten Knabe wrote:
> Hello.
>
> Starting with Linux 5.12 bcache triggers a BUG() after a few minutes of
> usage.
>
> Linux up to 5.11.x is not affected by this bug.
>
> Environment:
> - Debian 10 AMD 64
> - Kernel 5.12 - 5.12.4
> - Filesystem ext4
> - Backing device: degraded software RAID-6 (MD) with 3 of 4 disks active
> (unsure if the degraded RAID-6 has an effect or not)
> - Cache device: Single SSD
>
> Kernel log:
>
> May 12 20:22:24 tek04 kernel: nr_vecs=472
> May 12 20:22:24 tek04 kernel: ------------[ cut here ]------------
> May 12 20:22:24 tek04 kernel: kernel BUG at block/bio.c:53!
> May 12 20:22:24 tek04 kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI
> May 12 20:22:24 tek04 kernel: CPU: 1 PID: 1670 Comm: grep Tainted: G
> I 5.12.3 #2
> May 12 20:22:24 tek04 kernel: Hardware name: To Be Filled By O.E.M. To
> Be Filled By O.E.M./X58 Deluxe, BIOS P2.20 10/30/2009
> May 12 20:22:24 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
> May 12 20:22:24 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
> e8 d4 80 fe ff
> May 12 20:22:24 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
> May 12 20:22:24 tek04 kernel: RAX: 000000000000000b RBX:
> ffffc9000274b764 RCX: 0000000000000000
> May 12 20:22:24 tek04 kernel: bch_count_backing_io_errors: 1 callbacks
> suppressed
> May 12 20:22:24 tek04 kernel: bcache: bch_count_backing_io_errors() md1:
> Read-ahead I/O failed on backing device, ignore
> May 12 20:22:24 tek04 kernel: RDX: ffff888333c5e400 RSI:
> ffff888333c57480 RDI: ffff888333c57480
> May 12 20:22:24 tek04 kernel: RBP: 0000000000000800 R08:
> 0000000000000000 R09: c0000000ffffdfff
> May 12 20:22:24 tek04 kernel: R10: ffffc9000274b580 R11:
> ffffc9000274b578 R12: ffff8881170f0118
> May 12 20:22:24 tek04 kernel: R13: ffff888109911b00 R14:
> ffff8881170f00d0 R15: 0000000000000800
> May 12 20:22:24 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
> GS:ffff888333c40000(0000) knlGS:0000000000000000
> May 12 20:22:24 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033
> May 12 20:22:24 tek04 kernel: CR2: 00005628963f4fd0 CR3:
> 000000016c43c000 CR4: 00000000000006e0
> May 12 20:22:24 tek04 kernel: Call Trace:
> May 12 20:22:24 tek04 kernel: bvec_alloc+0x22/0x90
> May 12 20:22:24 tek04 kernel: bio_alloc_bioset+0x176/0x230
> May 12 20:22:24 tek04 kernel: cached_dev_cache_miss+0x1a8/0x300
> May 12 20:22:24 tek04 kernel: cache_lookup_fn+0x110/0x2e0
> May 12 20:22:24 tek04 kernel: ? bch_ptr_invalid+0x10/0x10
> May 12 20:22:24 tek04 kernel: ? bch_btree_iter_next_filter+0x1af/0x2d0
> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0x69/0x160
> May 12 20:22:24 tek04 kernel: ? __bch_bset_search+0x315/0x440
> May 12 20:22:24 tek04 kernel: ? downgrade_write+0xb0/0xb0
> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0xcf/0x160
> May 12 20:22:24 tek04 kernel: ? raid5_make_request+0x5c4/0xaa0
> May 12 20:22:24 tek04 kernel: ? recalibrate_cpu_khz+0x10/0x10
> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x30/0x400
> May 12 20:22:24 tek04 kernel: ? rwsem_wake.isra.11+0x80/0x80
> May 12 20:22:24 tek04 kernel: bch_btree_map_keys+0xf2/0x140
> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
> May 12 20:22:24 tek04 kernel: cache_lookup+0xb1/0x190
> May 12 20:22:24 tek04 kernel: cached_dev_submit_bio+0x9ab/0xc90
> May 12 20:22:24 tek04 kernel: ? submit_bio_checks+0x197/0x4a0
> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x3b7/0x400
> May 12 20:22:24 tek04 kernel: submit_bio_noacct+0x10e/0x4c0
> May 12 20:22:24 tek04 kernel: submit_bio+0x2e/0x160
> May 12 20:22:24 tek04 kernel: ? xa_load+0x66/0x70
> May 12 20:22:24 tek04 kernel: ? bio_add_page+0x2f/0x70
> May 12 20:22:24 tek04 kernel: ext4_mpage_readpages+0x1ae/0xa00
> May 12 20:22:24 tek04 kernel: ? __mod_lruvec_state+0x29/0x60
> May 12 20:22:24 tek04 kernel: read_pages+0x78/0x1d0
> May 12 20:22:24 tek04 kernel: page_cache_ra_unbounded+0x127/0x1b0
> May 12 20:22:24 tek04 kernel: filemap_get_pages+0x1d0/0x4a0
> May 12 20:22:24 tek04 kernel: filemap_read+0x91/0x2d0
> May 12 20:22:24 tek04 kernel: new_sync_read+0x103/0x180
> May 12 20:22:24 tek04 kernel: vfs_read+0x11b/0x1b0
> May 12 20:22:24 tek04 kernel: ksys_read+0x55/0xd0
> May 12 20:22:24 tek04 kernel: do_syscall_64+0x33/0x80
> May 12 20:22:24 tek04 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
> May 12 20:22:24 tek04 kernel: RIP: 0033:0x7f3a1cb89461
> May 12 20:22:24 tek04 kernel: Code: fe ff ff 50 48 8d 3d fe d0 09 00 e8
> e9 03 02 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 99 62 0d 00 8b 00 85 c0
> 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54
> 49 89 d4 55 48
> May 12 20:22:24 tek04 kernel: RSP: 002b:00007fff4052fff8 EFLAGS:
> 00000246 ORIG_RAX: 0000000000000000
> May 12 20:22:24 tek04 kernel: RAX: ffffffffffffffda RBX:
> 0000000000018000 RCX: 00007f3a1cb89461
> May 12 20:22:24 tek04 kernel: RDX: 0000000000018000 RSI:
> 000055ff0d01a000 RDI: 0000000000000004
> May 12 20:22:24 tek04 kernel: RBP: 0000000000018000 R08:
> 0000000000000002 R09: 000055ff0d0194b0
> May 12 20:22:24 tek04 kernel: R10: 0000000000000000 R11:
> 0000000000000246 R12: 000055ff0d01a000
> May 12 20:22:24 tek04 kernel: R13: 0000000000000004 R14:
> 000055ff0d0194b0 R15: 0000000000000004
> May 12 20:22:24 tek04 kernel: Modules linked in: cmac bnep
> intel_powerclamp snd_hda_codec_realtek snd_hda_codec_generic btusb
> ledtrig_audio snd_hda_codec_hdmi btrtl kvm_intel snd_hda_intel btbcm
> snd_intel_dspcfg btintel kvm snd_hda_codec irqbypass bluetooth serio_raw
> pcspkr hfcpci snd_hda_core iTCO_wdt evdev input_leds joydev sg snd_hwdep
> intel_pmc_bxt ecdh_generic rfkill iTCO_vendor_support mISDN_core ecc
> snd_pcm snd_timer tiny_power_button snd soundcore button i7core_edac
> acpi_cpufreq wmi nft_counter nf_log_ipv6 nf_log_ipv
> May 12 20:22:24 tek04 kernel: ---[ end trace 9a03f30c7b4aa246 ]---
> May 12 20:22:25 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
> May 12 20:22:25 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
> e8 d4 80 fe ff
> May 12 20:22:25 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
> May 12 20:22:25 tek04 kernel: RAX: 000000000000000b RBX:
> ffffc9000274b764 RCX: 0000000000000000
> May 12 20:22:25 tek04 kernel: RDX: ffff888333c5e400 RSI:
> ffff888333c57480 RDI: ffff888333c57480
> May 12 20:22:25 tek04 kernel: RBP: 0000000000000800 R08:
> 0000000000000000 R09: c0000000ffffdfff
> May 12 20:22:25 tek04 kernel: R10: ffffc9000274b580 R11:
> ffffc9000274b578 R12: ffff8881170f0118
> May 12 20:22:25 tek04 kernel: R13: ffff888109911b00 R14:
> ffff8881170f00d0 R15: 0000000000000800
> May 12 20:22:25 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
> GS:ffff888333c40000(0000) knlGS:0000000000000000
> May 12 20:22:25 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033
> May 12 20:22:25 tek04 kernel: CR2: 00005628963f4fd0 CR3:
> 000000016c43c000 CR4: 00000000000006e0
>
> A printk has been added to line 52 of block/bio.c to dump the nr_vecs
> variable to the kernel log before the BUG(). Obviously nr_vecs (472
> logged) is bigger than expected by bvec_alloc/bio_alloc_bioset (max
> 256), which finally triggers the BUG().
>
> Removing the BUG() from line 52 of block/bio.c, thus basically restoring
> the Linux 5.11.x behavior of bvec_alloc/bio_alloc_bioset to just return
> NULL, when nr_vecs is too big seems to resolve the issue.
>
> Regards
> Thorsten
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-05-31 9:37 ` Rolf Fokkens
@ 2021-06-01 15:34 ` Coly Li
2021-06-01 21:04 ` Rolf Fokkens
2021-06-02 8:33 ` Thorsten Knabe
0 siblings, 2 replies; 15+ messages in thread
From: Coly Li @ 2021-06-01 15:34 UTC (permalink / raw)
To: Rolf Fokkens; +Cc: Thorsten Knabe, linux-bcache
[-- Attachment #1: Type: text/plain, Size: 8253 bytes --]
On 5/31/21 5:37 PM, Rolf Fokkens wrote:
> Same here, more details: https://bugzilla.redhat.com/show_bug.cgi?id=1965809
Could you please try these attached patches ? I'd like to have more
confirmation before submitting to Jens.
Thanks.
Coly Li
>
> On 5/15/21 9:06 PM, Thorsten Knabe wrote:
>> Hello.
>>
>> Starting with Linux 5.12 bcache triggers a BUG() after a few minutes of
>> usage.
>>
>> Linux up to 5.11.x is not affected by this bug.
>>
>> Environment:
>> - Debian 10 AMD 64
>> - Kernel 5.12 - 5.12.4
>> - Filesystem ext4
>> - Backing device: degraded software RAID-6 (MD) with 3 of 4 disks active
>> (unsure if the degraded RAID-6 has an effect or not)
>> - Cache device: Single SSD
>>
>> Kernel log:
>>
>> May 12 20:22:24 tek04 kernel: nr_vecs=472
>> May 12 20:22:24 tek04 kernel: ------------[ cut here ]------------
>> May 12 20:22:24 tek04 kernel: kernel BUG at block/bio.c:53!
>> May 12 20:22:24 tek04 kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI
>> May 12 20:22:24 tek04 kernel: CPU: 1 PID: 1670 Comm: grep Tainted: G
>> I 5.12.3 #2
>> May 12 20:22:24 tek04 kernel: Hardware name: To Be Filled By O.E.M. To
>> Be Filled By O.E.M./X58 Deluxe, BIOS P2.20 10/30/2009
>> May 12 20:22:24 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
>> May 12 20:22:24 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
>> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
>> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
>> e8 d4 80 fe ff
>> May 12 20:22:24 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
>> May 12 20:22:24 tek04 kernel: RAX: 000000000000000b RBX:
>> ffffc9000274b764 RCX: 0000000000000000
>> May 12 20:22:24 tek04 kernel: bch_count_backing_io_errors: 1 callbacks
>> suppressed
>> May 12 20:22:24 tek04 kernel: bcache: bch_count_backing_io_errors() md1:
>> Read-ahead I/O failed on backing device, ignore
>> May 12 20:22:24 tek04 kernel: RDX: ffff888333c5e400 RSI:
>> ffff888333c57480 RDI: ffff888333c57480
>> May 12 20:22:24 tek04 kernel: RBP: 0000000000000800 R08:
>> 0000000000000000 R09: c0000000ffffdfff
>> May 12 20:22:24 tek04 kernel: R10: ffffc9000274b580 R11:
>> ffffc9000274b578 R12: ffff8881170f0118
>> May 12 20:22:24 tek04 kernel: R13: ffff888109911b00 R14:
>> ffff8881170f00d0 R15: 0000000000000800
>> May 12 20:22:24 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
>> GS:ffff888333c40000(0000) knlGS:0000000000000000
>> May 12 20:22:24 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>> 0000000080050033
>> May 12 20:22:24 tek04 kernel: CR2: 00005628963f4fd0 CR3:
>> 000000016c43c000 CR4: 00000000000006e0
>> May 12 20:22:24 tek04 kernel: Call Trace:
>> May 12 20:22:24 tek04 kernel: bvec_alloc+0x22/0x90
>> May 12 20:22:24 tek04 kernel: bio_alloc_bioset+0x176/0x230
>> May 12 20:22:24 tek04 kernel: cached_dev_cache_miss+0x1a8/0x300
>> May 12 20:22:24 tek04 kernel: cache_lookup_fn+0x110/0x2e0
>> May 12 20:22:24 tek04 kernel: ? bch_ptr_invalid+0x10/0x10
>> May 12 20:22:24 tek04 kernel: ? bch_btree_iter_next_filter+0x1af/0x2d0
>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0x69/0x160
>> May 12 20:22:24 tek04 kernel: ? __bch_bset_search+0x315/0x440
>> May 12 20:22:24 tek04 kernel: ? downgrade_write+0xb0/0xb0
>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0xcf/0x160
>> May 12 20:22:24 tek04 kernel: ? raid5_make_request+0x5c4/0xaa0
>> May 12 20:22:24 tek04 kernel: ? recalibrate_cpu_khz+0x10/0x10
>> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x30/0x400
>> May 12 20:22:24 tek04 kernel: ? rwsem_wake.isra.11+0x80/0x80
>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys+0xf2/0x140
>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>> May 12 20:22:24 tek04 kernel: cache_lookup+0xb1/0x190
>> May 12 20:22:24 tek04 kernel: cached_dev_submit_bio+0x9ab/0xc90
>> May 12 20:22:24 tek04 kernel: ? submit_bio_checks+0x197/0x4a0
>> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x3b7/0x400
>> May 12 20:22:24 tek04 kernel: submit_bio_noacct+0x10e/0x4c0
>> May 12 20:22:24 tek04 kernel: submit_bio+0x2e/0x160
>> May 12 20:22:24 tek04 kernel: ? xa_load+0x66/0x70
>> May 12 20:22:24 tek04 kernel: ? bio_add_page+0x2f/0x70
>> May 12 20:22:24 tek04 kernel: ext4_mpage_readpages+0x1ae/0xa00
>> May 12 20:22:24 tek04 kernel: ? __mod_lruvec_state+0x29/0x60
>> May 12 20:22:24 tek04 kernel: read_pages+0x78/0x1d0
>> May 12 20:22:24 tek04 kernel: page_cache_ra_unbounded+0x127/0x1b0
>> May 12 20:22:24 tek04 kernel: filemap_get_pages+0x1d0/0x4a0
>> May 12 20:22:24 tek04 kernel: filemap_read+0x91/0x2d0
>> May 12 20:22:24 tek04 kernel: new_sync_read+0x103/0x180
>> May 12 20:22:24 tek04 kernel: vfs_read+0x11b/0x1b0
>> May 12 20:22:24 tek04 kernel: ksys_read+0x55/0xd0
>> May 12 20:22:24 tek04 kernel: do_syscall_64+0x33/0x80
>> May 12 20:22:24 tek04 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>> May 12 20:22:24 tek04 kernel: RIP: 0033:0x7f3a1cb89461
>> May 12 20:22:24 tek04 kernel: Code: fe ff ff 50 48 8d 3d fe d0 09 00 e8
>> e9 03 02 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 99 62 0d 00 8b 00 85 c0
>> 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54
>> 49 89 d4 55 48
>> May 12 20:22:24 tek04 kernel: RSP: 002b:00007fff4052fff8 EFLAGS:
>> 00000246 ORIG_RAX: 0000000000000000
>> May 12 20:22:24 tek04 kernel: RAX: ffffffffffffffda RBX:
>> 0000000000018000 RCX: 00007f3a1cb89461
>> May 12 20:22:24 tek04 kernel: RDX: 0000000000018000 RSI:
>> 000055ff0d01a000 RDI: 0000000000000004
>> May 12 20:22:24 tek04 kernel: RBP: 0000000000018000 R08:
>> 0000000000000002 R09: 000055ff0d0194b0
>> May 12 20:22:24 tek04 kernel: R10: 0000000000000000 R11:
>> 0000000000000246 R12: 000055ff0d01a000
>> May 12 20:22:24 tek04 kernel: R13: 0000000000000004 R14:
>> 000055ff0d0194b0 R15: 0000000000000004
>> May 12 20:22:24 tek04 kernel: Modules linked in: cmac bnep
>> intel_powerclamp snd_hda_codec_realtek snd_hda_codec_generic btusb
>> ledtrig_audio snd_hda_codec_hdmi btrtl kvm_intel snd_hda_intel btbcm
>> snd_intel_dspcfg btintel kvm snd_hda_codec irqbypass bluetooth serio_raw
>> pcspkr hfcpci snd_hda_core iTCO_wdt evdev input_leds joydev sg snd_hwdep
>> intel_pmc_bxt ecdh_generic rfkill iTCO_vendor_support mISDN_core ecc
>> snd_pcm snd_timer tiny_power_button snd soundcore button i7core_edac
>> acpi_cpufreq wmi nft_counter nf_log_ipv6 nf_log_ipv
>> May 12 20:22:24 tek04 kernel: ---[ end trace 9a03f30c7b4aa246 ]---
>> May 12 20:22:25 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
>> May 12 20:22:25 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
>> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
>> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
>> e8 d4 80 fe ff
>> May 12 20:22:25 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
>> May 12 20:22:25 tek04 kernel: RAX: 000000000000000b RBX:
>> ffffc9000274b764 RCX: 0000000000000000
>> May 12 20:22:25 tek04 kernel: RDX: ffff888333c5e400 RSI:
>> ffff888333c57480 RDI: ffff888333c57480
>> May 12 20:22:25 tek04 kernel: RBP: 0000000000000800 R08:
>> 0000000000000000 R09: c0000000ffffdfff
>> May 12 20:22:25 tek04 kernel: R10: ffffc9000274b580 R11:
>> ffffc9000274b578 R12: ffff8881170f0118
>> May 12 20:22:25 tek04 kernel: R13: ffff888109911b00 R14:
>> ffff8881170f00d0 R15: 0000000000000800
>> May 12 20:22:25 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
>> GS:ffff888333c40000(0000) knlGS:0000000000000000
>> May 12 20:22:25 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>> 0000000080050033
>> May 12 20:22:25 tek04 kernel: CR2: 00005628963f4fd0 CR3:
>> 000000016c43c000 CR4: 00000000000006e0
>>
>> A printk has been added to line 52 of block/bio.c to dump the nr_vecs
>> variable to the kernel log before the BUG(). Obviously nr_vecs (472
>> logged) is bigger than expected by bvec_alloc/bio_alloc_bioset (max
>> 256), which finally triggers the BUG().
>>
>> Removing the BUG() from line 52 of block/bio.c, thus basically restoring
>> the Linux 5.11.x behavior of bvec_alloc/bio_alloc_bioset to just return
>> NULL, when nr_vecs is too big seems to resolve the issue.
>>
>> Regards
>> Thorsten
>>
[-- Attachment #2: v5-0001-bcache-remove-bcache-device-self-defined-readahea.patch --]
[-- Type: text/plain, Size: 7638 bytes --]
From 79c8b6fe6b1ae95e6a75e49aa907b80f4cf1f813 Mon Sep 17 00:00:00 2001
From: Coly Li <colyli@suse.de>
Date: Mon, 31 May 2021 22:30:37 +0800
Subject: [PATCH v5 1/2] bcache: remove bcache device self-defined readahead
For read cache missing, bcache defines a readahead size for the read I/O
request to the backing device for the missing data. This readahead size
is initialized to 0, and almost no one uses it to avoid unnecessary read
amplifying onto backing device and write amplifying onto cache device.
Considering upper layer file system code has readahead logic allready
and works fine with readahead_cache_policy sysfile interface, we don't
have to keep bcache self-defined readahead anymore.
This patch removes the bcache self-defined readahead for cache missing
request for backing device, and the readahead sysfs file interfaces are
removed as well.
This is the preparation for next patch to fix potential kernel panic due
to oversized request in a simpler method.
Reported-by: Alexander Ullrich <ealex1979@gmail.com>
Reported-by: Diego Ercolani <diego.ercolani@gmail.com>
Reported-by: Jan Szubiak <jan.szubiak@linuxpolska.pl>
Reported-by: Marco Rebhan <me@dblsaiko.net>
Reported-by: Matthias Ferdinand <bcache@mfedv.net>
Reported-by: Thorsten Knabe <linux@thorsten-knabe.de>
Reported-by: Victor Westerhuis <victor@westerhu.is>
Reported-by: Vojtech Pavlik <vojtech@suse.cz>
Signed-off-by: Coly Li <colyli@suse.de>
Cc: stable@vger.kernel.org
Cc: Christoph Hellwig <hch@lst.de>
Cc: Kent Overstreet <kent.overstreet@gmail.com>
Cc: Nix <nix@esperi.org.uk>
Cc: Takashi Iwai <tiwai@suse.com>
---
Changlog,
v1, the initial version by hint from Christoph Hellwig.
drivers/md/bcache/bcache.h | 1 -
drivers/md/bcache/request.c | 13 +------------
drivers/md/bcache/stats.c | 14 --------------
drivers/md/bcache/stats.h | 1 -
drivers/md/bcache/sysfs.c | 4 ----
5 files changed, 1 insertion(+), 32 deletions(-)
diff --git a/drivers/md/bcache/bcache.h b/drivers/md/bcache/bcache.h
index 0a4551e165ab..5fc989a6d452 100644
--- a/drivers/md/bcache/bcache.h
+++ b/drivers/md/bcache/bcache.h
@@ -364,7 +364,6 @@ struct cached_dev {
/* The rest of this all shows up in sysfs */
unsigned int sequential_cutoff;
- unsigned int readahead;
unsigned int io_disable:1;
unsigned int verify:1;
diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c
index 29c231758293..ab8ff18df32a 100644
--- a/drivers/md/bcache/request.c
+++ b/drivers/md/bcache/request.c
@@ -880,7 +880,6 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s,
struct bio *bio, unsigned int sectors)
{
int ret = MAP_CONTINUE;
- unsigned int reada = 0;
struct cached_dev *dc = container_of(s->d, struct cached_dev, disk);
struct bio *miss, *cache_bio;
@@ -892,14 +891,7 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s,
goto out_submit;
}
- if (!(bio->bi_opf & REQ_RAHEAD) &&
- !(bio->bi_opf & (REQ_META|REQ_PRIO)) &&
- s->iop.c->gc_stats.in_use < CUTOFF_CACHE_READA)
- reada = min_t(sector_t, dc->readahead >> 9,
- get_capacity(bio->bi_bdev->bd_disk) -
- bio_end_sector(bio));
-
- s->insert_bio_sectors = min(sectors, bio_sectors(bio) + reada);
+ s->insert_bio_sectors = min(sectors, bio_sectors(bio));
s->iop.replace_key = KEY(s->iop.inode,
bio->bi_iter.bi_sector + s->insert_bio_sectors,
@@ -933,9 +925,6 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s,
if (bch_bio_alloc_pages(cache_bio, __GFP_NOWARN|GFP_NOIO))
goto out_put;
- if (reada)
- bch_mark_cache_readahead(s->iop.c, s->d);
-
s->cache_miss = miss;
s->iop.bio = cache_bio;
bio_get(cache_bio);
diff --git a/drivers/md/bcache/stats.c b/drivers/md/bcache/stats.c
index 503aafe188dc..4c7ee5fedb9d 100644
--- a/drivers/md/bcache/stats.c
+++ b/drivers/md/bcache/stats.c
@@ -46,7 +46,6 @@ read_attribute(cache_misses);
read_attribute(cache_bypass_hits);
read_attribute(cache_bypass_misses);
read_attribute(cache_hit_ratio);
-read_attribute(cache_readaheads);
read_attribute(cache_miss_collisions);
read_attribute(bypassed);
@@ -64,7 +63,6 @@ SHOW(bch_stats)
DIV_SAFE(var(cache_hits) * 100,
var(cache_hits) + var(cache_misses)));
- var_print(cache_readaheads);
var_print(cache_miss_collisions);
sysfs_hprint(bypassed, var(sectors_bypassed) << 9);
#undef var
@@ -86,7 +84,6 @@ static struct attribute *bch_stats_files[] = {
&sysfs_cache_bypass_hits,
&sysfs_cache_bypass_misses,
&sysfs_cache_hit_ratio,
- &sysfs_cache_readaheads,
&sysfs_cache_miss_collisions,
&sysfs_bypassed,
NULL
@@ -113,7 +110,6 @@ void bch_cache_accounting_clear(struct cache_accounting *acc)
acc->total.cache_misses = 0;
acc->total.cache_bypass_hits = 0;
acc->total.cache_bypass_misses = 0;
- acc->total.cache_readaheads = 0;
acc->total.cache_miss_collisions = 0;
acc->total.sectors_bypassed = 0;
}
@@ -145,7 +141,6 @@ static void scale_stats(struct cache_stats *stats, unsigned long rescale_at)
scale_stat(&stats->cache_misses);
scale_stat(&stats->cache_bypass_hits);
scale_stat(&stats->cache_bypass_misses);
- scale_stat(&stats->cache_readaheads);
scale_stat(&stats->cache_miss_collisions);
scale_stat(&stats->sectors_bypassed);
}
@@ -168,7 +163,6 @@ static void scale_accounting(struct timer_list *t)
move_stat(cache_misses);
move_stat(cache_bypass_hits);
move_stat(cache_bypass_misses);
- move_stat(cache_readaheads);
move_stat(cache_miss_collisions);
move_stat(sectors_bypassed);
@@ -209,14 +203,6 @@ void bch_mark_cache_accounting(struct cache_set *c, struct bcache_device *d,
mark_cache_stats(&c->accounting.collector, hit, bypass);
}
-void bch_mark_cache_readahead(struct cache_set *c, struct bcache_device *d)
-{
- struct cached_dev *dc = container_of(d, struct cached_dev, disk);
-
- atomic_inc(&dc->accounting.collector.cache_readaheads);
- atomic_inc(&c->accounting.collector.cache_readaheads);
-}
-
void bch_mark_cache_miss_collision(struct cache_set *c, struct bcache_device *d)
{
struct cached_dev *dc = container_of(d, struct cached_dev, disk);
diff --git a/drivers/md/bcache/stats.h b/drivers/md/bcache/stats.h
index abfaabf7e7fc..ca4f435f7216 100644
--- a/drivers/md/bcache/stats.h
+++ b/drivers/md/bcache/stats.h
@@ -7,7 +7,6 @@ struct cache_stat_collector {
atomic_t cache_misses;
atomic_t cache_bypass_hits;
atomic_t cache_bypass_misses;
- atomic_t cache_readaheads;
atomic_t cache_miss_collisions;
atomic_t sectors_bypassed;
};
diff --git a/drivers/md/bcache/sysfs.c b/drivers/md/bcache/sysfs.c
index cc89f3156d1a..05ac1d6fbbf3 100644
--- a/drivers/md/bcache/sysfs.c
+++ b/drivers/md/bcache/sysfs.c
@@ -137,7 +137,6 @@ rw_attribute(io_disable);
rw_attribute(discard);
rw_attribute(running);
rw_attribute(label);
-rw_attribute(readahead);
rw_attribute(errors);
rw_attribute(io_error_limit);
rw_attribute(io_error_halflife);
@@ -260,7 +259,6 @@ SHOW(__bch_cached_dev)
var_printf(partial_stripes_expensive, "%u");
var_hprint(sequential_cutoff);
- var_hprint(readahead);
sysfs_print(running, atomic_read(&dc->running));
sysfs_print(state, states[BDEV_STATE(&dc->sb)]);
@@ -365,7 +363,6 @@ STORE(__cached_dev)
sysfs_strtoul_clamp(sequential_cutoff,
dc->sequential_cutoff,
0, UINT_MAX);
- d_strtoi_h(readahead);
if (attr == &sysfs_clear_stats)
bch_cache_accounting_clear(&dc->accounting);
@@ -538,7 +535,6 @@ static struct attribute *bch_cached_dev_files[] = {
&sysfs_running,
&sysfs_state,
&sysfs_label,
- &sysfs_readahead,
#ifdef CONFIG_BCACHE_DEBUG
&sysfs_verify,
&sysfs_bypass_torture_test,
--
2.26.2
[-- Attachment #3: v5-0002-bcache-avoid-oversized-read-request-in-cache-miss.patch --]
[-- Type: text/plain, Size: 8619 bytes --]
From ad9ed825299930298c2e12699d855e0b89c2953a Mon Sep 17 00:00:00 2001
From: Coly Li <colyli@suse.de>
Date: Mon, 31 May 2021 22:47:12 +0800
Subject: [PATCH v5 2/2] bcache: avoid oversized read request in cache missing
code path
In the cache missing code path of cached device, if a proper location
from the internal B+ tree is matched for a cache miss range, function
cached_dev_cache_miss() will be called in cache_lookup_fn() in the
following code block,
[code block 1]
526 unsigned int sectors = KEY_INODE(k) == s->iop.inode
527 ? min_t(uint64_t, INT_MAX,
528 KEY_START(k) - bio->bi_iter.bi_sector)
529 : INT_MAX;
530 int ret = s->d->cache_miss(b, s, bio, sectors);
Here s->d->cache_miss() is the call backfunction pointer initialized as
cached_dev_cache_miss(), the last parameter 'sectors' is an important
hint to calculate the size of read request to backing device of the
missing cache data.
Current calculation in above code block may generate oversized value of
'sectors', which consequently may trigger 2 different potential kernel
panics by BUG() or BUG_ON() as listed below,
1) BUG_ON() inside bch_btree_insert_key(),
[code block 2]
886 BUG_ON(b->ops->is_extents && !KEY_SIZE(k));
2) BUG() inside biovec_slab(),
[code block 3]
51 default:
52 BUG();
53 return NULL;
All the above panics are original from cached_dev_cache_miss() by the
oversized parameter 'sectors'.
Inside cached_dev_cache_miss(), parameter 'sectors' is used to calculate
the size of data read from backing device for the cache missing. This
size is stored in s->insert_bio_sectors by the following lines of code,
[code block 4]
909 s->insert_bio_sectors = min(sectors, bio_sectors(bio) + reada);
Then the actual key inserting to the internal B+ tree is generated and
stored in s->iop.replace_key by the following lines of code,
[code block 5]
911 s->iop.replace_key = KEY(s->iop.inode,
912 bio->bi_iter.bi_sector + s->insert_bio_sectors,
913 s->insert_bio_sectors);
The oversized parameter 'sectors' may trigger panic 1) by BUG_ON() from
the above code block.
And the bio sending to backing device for the missing data is allocated
with hint from s->insert_bio_sectors by the following lines of code,
[code block 6]
926 cache_bio = bio_alloc_bioset(GFP_NOWAIT,
927 DIV_ROUND_UP(s->insert_bio_sectors, PAGE_SECTORS),
928 &dc->disk.bio_split);
The oversized parameter 'sectors' may trigger panic 2) by BUG() from the
agove code block.
Now let me explain how the panics happen with the oversized 'sectors'.
In code block 5, replace_key is generated by macro KEY(). From the
definition of macro KEY(),
[code block 7]
71 #define KEY(inode, offset, size) \
72 ((struct bkey) { \
73 .high = (1ULL << 63) | ((__u64) (size) << 20) | (inode), \
74 .low = (offset) \
75 })
Here 'size' is 16bits width embedded in 64bits member 'high' of struct
bkey. But in code block 1, if "KEY_START(k) - bio->bi_iter.bi_sector" is
very probably to be larger than (1<<16) - 1, which makes the bkey size
calculation in code block 5 is overflowed. In one bug report the value
of parameter 'sectors' is 131072 (= 1 << 17), the overflowed 'sectors'
results the overflowed s->insert_bio_sectors in code block 4, then makes
size field of s->iop.replace_key to be 0 in code block 5. Then the 0-
sized s->iop.replace_key is inserted into the internal B+ tree as cache
missing check key (a special key to detect and avoid a racing between
normal write request and cache missing read request) as,
[code block 8]
915 ret = bch_btree_insert_check_key(b, &s->op, &s->iop.replace_key);
Then the 0-sized s->iop.replace_key as 3rd parameter triggers the bkey
size check BUG_ON() in code block 2, and causes the kernel panic 1).
Another kernel panic is from code block 6, is by the bvecs number
oversized value s->insert_bio_sectors from code block 4,
min(sectors, bio_sectors(bio) + reada)
There are two possibility for oversized reresult,
- bio_sectors(bio) is valid, but bio_sectors(bio) + reada is oversized.
- sectors < bio_sectors(bio) + reada, but sectors is oversized.
From a bug report the result of "DIV_ROUND_UP(s->insert_bio_sectors,
PAGE_SECTORS)" from code block 6 can be 344, 282, 946, 342 and many
other values which larther than BIO_MAX_VECS (a.k.a 256). When calling
bio_alloc_bioset() with such larger-than-256 value as the 2nd parameter,
this value will eventually be sent to biovec_slab() as parameter
'nr_vecs' in following code path,
bio_alloc_bioset() ==> bvec_alloc() ==> biovec_slab()
Because parameter 'nr_vecs' is larger-than-256 value, the panic by BUG()
in code block 3 is triggered inside biovec_slab().
From the above analysis, we know that the 4th parameter 'sector' sent
into cached_dev_cache_miss() may cause overflow in code block 5 and 6,
and finally cause kernel panic in code block 2 and 3. And if result of
bio_sectors(bio) + reada exceeds valid bvecs number, it may also trigger
kernel panic in code block 3 from code block 6.
Now the almost-useless readahead size for cache missing request back to
backing device is removed, this patch can fix the oversized issue with
more simpler method.
- add a local variable size_limit, set it by the minimum value from
the max bkey size and max bio bvecs number.
- set s->insert_bio_sectors by the minimum value from size_limit,
sectors, and the sectors size of bio.
- replace sectors by s->insert_bio_sectors to do bio_next_split.
By the above method with size_limit, s->insert_bio_sectors will never
result oversized replace_key size or bio bvecs number. And split bio
'miss' from bio_next_split() will always match the size of 'cache_bio',
that is the current maximum bio size we can sent to backing device for
fetching the cache missing data.
Current problmatic code can be partially found since Linux v3.13-rc1,
therefore all maintained stable kernels should try to apply this fix.
Reported-by: Alexander Ullrich <ealex1979@gmail.com>
Reported-by: Diego Ercolani <diego.ercolani@gmail.com>
Reported-by: Jan Szubiak <jan.szubiak@linuxpolska.pl>
Reported-by: Marco Rebhan <me@dblsaiko.net>
Reported-by: Matthias Ferdinand <bcache@mfedv.net>
Reported-by: Thorsten Knabe <linux@thorsten-knabe.de>
Reported-by: Victor Westerhuis <victor@westerhu.is>
Reported-by: Vojtech Pavlik <vojtech@suse.cz>
Signed-off-by: Coly Li <colyli@suse.de>
Cc: stable@vger.kernel.org
Cc: Christoph Hellwig <hch@lst.de>
Cc: Kent Overstreet <kent.overstreet@gmail.com>
Cc: Nix <nix@esperi.org.uk>
Cc: Takashi Iwai <tiwai@suse.com>
---
Changelog,
v5, improvement and fix based on v4 comments from Christoph Hellwig
and Nix.
v4, not directly access BIO_MAX_VECS and reduce reada value to avoid
oversized bvecs number, by hint from Christoph Hellwig.
v3, fix typo in v2.
v2, fix the bypass bio size calculation in v1.
v1, the initial version.
drivers/md/bcache/request.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c
index ab8ff18df32a..d855a8882cbc 100644
--- a/drivers/md/bcache/request.c
+++ b/drivers/md/bcache/request.c
@@ -882,6 +882,7 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s,
int ret = MAP_CONTINUE;
struct cached_dev *dc = container_of(s->d, struct cached_dev, disk);
struct bio *miss, *cache_bio;
+ unsigned int size_limit;
s->cache_missed = 1;
@@ -891,7 +892,10 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s,
goto out_submit;
}
- s->insert_bio_sectors = min(sectors, bio_sectors(bio));
+ /* Limitation for valid replace key size and cache_bio bvecs number */
+ size_limit = min_t(unsigned int, bio_max_segs(UINT_MAX) * PAGE_SECTORS,
+ (1 << KEY_SIZE_BITS) - 1);
+ s->insert_bio_sectors = min3(size_limit, sectors, bio_sectors(bio));
s->iop.replace_key = KEY(s->iop.inode,
bio->bi_iter.bi_sector + s->insert_bio_sectors,
@@ -903,7 +907,7 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s,
s->iop.replace = true;
- miss = bio_next_split(bio, sectors, GFP_NOIO, &s->d->bio_split);
+ miss = bio_next_split(bio, s->insert_bio_sectors, GFP_NOIO, &s->d->bio_split);
/* btree_search_recurse()'s btree iterator is no good anymore */
ret = miss == bio ? MAP_DONE : -EINTR;
--
2.26.2
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-06-01 15:34 ` Coly Li
@ 2021-06-01 21:04 ` Rolf Fokkens
2021-06-01 21:46 ` Rolf Fokkens
2021-06-02 8:33 ` Thorsten Knabe
1 sibling, 1 reply; 15+ messages in thread
From: Rolf Fokkens @ 2021-06-01 21:04 UTC (permalink / raw)
To: Coly Li; +Cc: Thorsten Knabe, linux-bcache
Building a patched Fedora 5.12.8 kernel.....
Will keep you posted,
Rolf
On 6/1/21 5:34 PM, Coly Li wrote:
> On 5/31/21 5:37 PM, Rolf Fokkens wrote:
>> Same here, more details: https://bugzilla.redhat.com/show_bug.cgi?id=1965809
> Could you please try these attached patches ? I'd like to have more
> confirmation before submitting to Jens.
>
> Thanks.
>
> Coly Li
>
>
>
>> On 5/15/21 9:06 PM, Thorsten Knabe wrote:
>>> Hello.
>>>
>>> Starting with Linux 5.12 bcache triggers a BUG() after a few minutes of
>>> usage.
>>>
>>> Linux up to 5.11.x is not affected by this bug.
>>>
>>> Environment:
>>> - Debian 10 AMD 64
>>> - Kernel 5.12 - 5.12.4
>>> - Filesystem ext4
>>> - Backing device: degraded software RAID-6 (MD) with 3 of 4 disks active
>>> (unsure if the degraded RAID-6 has an effect or not)
>>> - Cache device: Single SSD
>>>
>>> Kernel log:
>>>
>>> May 12 20:22:24 tek04 kernel: nr_vecs=472
>>> May 12 20:22:24 tek04 kernel: ------------[ cut here ]------------
>>> May 12 20:22:24 tek04 kernel: kernel BUG at block/bio.c:53!
>>> May 12 20:22:24 tek04 kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI
>>> May 12 20:22:24 tek04 kernel: CPU: 1 PID: 1670 Comm: grep Tainted: G
>>> I 5.12.3 #2
>>> May 12 20:22:24 tek04 kernel: Hardware name: To Be Filled By O.E.M. To
>>> Be Filled By O.E.M./X58 Deluxe, BIOS P2.20 10/30/2009
>>> May 12 20:22:24 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
>>> May 12 20:22:24 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
>>> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
>>> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
>>> e8 d4 80 fe ff
>>> May 12 20:22:24 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
>>> May 12 20:22:24 tek04 kernel: RAX: 000000000000000b RBX:
>>> ffffc9000274b764 RCX: 0000000000000000
>>> May 12 20:22:24 tek04 kernel: bch_count_backing_io_errors: 1 callbacks
>>> suppressed
>>> May 12 20:22:24 tek04 kernel: bcache: bch_count_backing_io_errors() md1:
>>> Read-ahead I/O failed on backing device, ignore
>>> May 12 20:22:24 tek04 kernel: RDX: ffff888333c5e400 RSI:
>>> ffff888333c57480 RDI: ffff888333c57480
>>> May 12 20:22:24 tek04 kernel: RBP: 0000000000000800 R08:
>>> 0000000000000000 R09: c0000000ffffdfff
>>> May 12 20:22:24 tek04 kernel: R10: ffffc9000274b580 R11:
>>> ffffc9000274b578 R12: ffff8881170f0118
>>> May 12 20:22:24 tek04 kernel: R13: ffff888109911b00 R14:
>>> ffff8881170f00d0 R15: 0000000000000800
>>> May 12 20:22:24 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
>>> GS:ffff888333c40000(0000) knlGS:0000000000000000
>>> May 12 20:22:24 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>> 0000000080050033
>>> May 12 20:22:24 tek04 kernel: CR2: 00005628963f4fd0 CR3:
>>> 000000016c43c000 CR4: 00000000000006e0
>>> May 12 20:22:24 tek04 kernel: Call Trace:
>>> May 12 20:22:24 tek04 kernel: bvec_alloc+0x22/0x90
>>> May 12 20:22:24 tek04 kernel: bio_alloc_bioset+0x176/0x230
>>> May 12 20:22:24 tek04 kernel: cached_dev_cache_miss+0x1a8/0x300
>>> May 12 20:22:24 tek04 kernel: cache_lookup_fn+0x110/0x2e0
>>> May 12 20:22:24 tek04 kernel: ? bch_ptr_invalid+0x10/0x10
>>> May 12 20:22:24 tek04 kernel: ? bch_btree_iter_next_filter+0x1af/0x2d0
>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0x69/0x160
>>> May 12 20:22:24 tek04 kernel: ? __bch_bset_search+0x315/0x440
>>> May 12 20:22:24 tek04 kernel: ? downgrade_write+0xb0/0xb0
>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0xcf/0x160
>>> May 12 20:22:24 tek04 kernel: ? raid5_make_request+0x5c4/0xaa0
>>> May 12 20:22:24 tek04 kernel: ? recalibrate_cpu_khz+0x10/0x10
>>> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x30/0x400
>>> May 12 20:22:24 tek04 kernel: ? rwsem_wake.isra.11+0x80/0x80
>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys+0xf2/0x140
>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>> May 12 20:22:24 tek04 kernel: cache_lookup+0xb1/0x190
>>> May 12 20:22:24 tek04 kernel: cached_dev_submit_bio+0x9ab/0xc90
>>> May 12 20:22:24 tek04 kernel: ? submit_bio_checks+0x197/0x4a0
>>> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x3b7/0x400
>>> May 12 20:22:24 tek04 kernel: submit_bio_noacct+0x10e/0x4c0
>>> May 12 20:22:24 tek04 kernel: submit_bio+0x2e/0x160
>>> May 12 20:22:24 tek04 kernel: ? xa_load+0x66/0x70
>>> May 12 20:22:24 tek04 kernel: ? bio_add_page+0x2f/0x70
>>> May 12 20:22:24 tek04 kernel: ext4_mpage_readpages+0x1ae/0xa00
>>> May 12 20:22:24 tek04 kernel: ? __mod_lruvec_state+0x29/0x60
>>> May 12 20:22:24 tek04 kernel: read_pages+0x78/0x1d0
>>> May 12 20:22:24 tek04 kernel: page_cache_ra_unbounded+0x127/0x1b0
>>> May 12 20:22:24 tek04 kernel: filemap_get_pages+0x1d0/0x4a0
>>> May 12 20:22:24 tek04 kernel: filemap_read+0x91/0x2d0
>>> May 12 20:22:24 tek04 kernel: new_sync_read+0x103/0x180
>>> May 12 20:22:24 tek04 kernel: vfs_read+0x11b/0x1b0
>>> May 12 20:22:24 tek04 kernel: ksys_read+0x55/0xd0
>>> May 12 20:22:24 tek04 kernel: do_syscall_64+0x33/0x80
>>> May 12 20:22:24 tek04 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>>> May 12 20:22:24 tek04 kernel: RIP: 0033:0x7f3a1cb89461
>>> May 12 20:22:24 tek04 kernel: Code: fe ff ff 50 48 8d 3d fe d0 09 00 e8
>>> e9 03 02 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 99 62 0d 00 8b 00 85 c0
>>> 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54
>>> 49 89 d4 55 48
>>> May 12 20:22:24 tek04 kernel: RSP: 002b:00007fff4052fff8 EFLAGS:
>>> 00000246 ORIG_RAX: 0000000000000000
>>> May 12 20:22:24 tek04 kernel: RAX: ffffffffffffffda RBX:
>>> 0000000000018000 RCX: 00007f3a1cb89461
>>> May 12 20:22:24 tek04 kernel: RDX: 0000000000018000 RSI:
>>> 000055ff0d01a000 RDI: 0000000000000004
>>> May 12 20:22:24 tek04 kernel: RBP: 0000000000018000 R08:
>>> 0000000000000002 R09: 000055ff0d0194b0
>>> May 12 20:22:24 tek04 kernel: R10: 0000000000000000 R11:
>>> 0000000000000246 R12: 000055ff0d01a000
>>> May 12 20:22:24 tek04 kernel: R13: 0000000000000004 R14:
>>> 000055ff0d0194b0 R15: 0000000000000004
>>> May 12 20:22:24 tek04 kernel: Modules linked in: cmac bnep
>>> intel_powerclamp snd_hda_codec_realtek snd_hda_codec_generic btusb
>>> ledtrig_audio snd_hda_codec_hdmi btrtl kvm_intel snd_hda_intel btbcm
>>> snd_intel_dspcfg btintel kvm snd_hda_codec irqbypass bluetooth serio_raw
>>> pcspkr hfcpci snd_hda_core iTCO_wdt evdev input_leds joydev sg snd_hwdep
>>> intel_pmc_bxt ecdh_generic rfkill iTCO_vendor_support mISDN_core ecc
>>> snd_pcm snd_timer tiny_power_button snd soundcore button i7core_edac
>>> acpi_cpufreq wmi nft_counter nf_log_ipv6 nf_log_ipv
>>> May 12 20:22:24 tek04 kernel: ---[ end trace 9a03f30c7b4aa246 ]---
>>> May 12 20:22:25 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
>>> May 12 20:22:25 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
>>> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
>>> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
>>> e8 d4 80 fe ff
>>> May 12 20:22:25 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
>>> May 12 20:22:25 tek04 kernel: RAX: 000000000000000b RBX:
>>> ffffc9000274b764 RCX: 0000000000000000
>>> May 12 20:22:25 tek04 kernel: RDX: ffff888333c5e400 RSI:
>>> ffff888333c57480 RDI: ffff888333c57480
>>> May 12 20:22:25 tek04 kernel: RBP: 0000000000000800 R08:
>>> 0000000000000000 R09: c0000000ffffdfff
>>> May 12 20:22:25 tek04 kernel: R10: ffffc9000274b580 R11:
>>> ffffc9000274b578 R12: ffff8881170f0118
>>> May 12 20:22:25 tek04 kernel: R13: ffff888109911b00 R14:
>>> ffff8881170f00d0 R15: 0000000000000800
>>> May 12 20:22:25 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
>>> GS:ffff888333c40000(0000) knlGS:0000000000000000
>>> May 12 20:22:25 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>> 0000000080050033
>>> May 12 20:22:25 tek04 kernel: CR2: 00005628963f4fd0 CR3:
>>> 000000016c43c000 CR4: 00000000000006e0
>>>
>>> A printk has been added to line 52 of block/bio.c to dump the nr_vecs
>>> variable to the kernel log before the BUG(). Obviously nr_vecs (472
>>> logged) is bigger than expected by bvec_alloc/bio_alloc_bioset (max
>>> 256), which finally triggers the BUG().
>>>
>>> Removing the BUG() from line 52 of block/bio.c, thus basically restoring
>>> the Linux 5.11.x behavior of bvec_alloc/bio_alloc_bioset to just return
>>> NULL, when nr_vecs is too big seems to resolve the issue.
>>>
>>> Regards
>>> Thorsten
>>>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-06-01 21:04 ` Rolf Fokkens
@ 2021-06-01 21:46 ` Rolf Fokkens
0 siblings, 0 replies; 15+ messages in thread
From: Rolf Fokkens @ 2021-06-01 21:46 UTC (permalink / raw)
To: Coly Li; +Cc: Thorsten Knabe, linux-bcache
System is running:
bash-5.0$ cat /proc/version
Linux version 5.12.8-200.rf.fc33.x86_64
(mockbuild@tb-sandbox-mjolnir.local.tbai.nl) (gcc (GCC) 10.3.1 20210422
(Red Hat 10.3.1-1), GNU ld version 2.35-18.fc33) #1 SMP Tue Jun 1
23:10:39 CEST 2021
bash-5.0$
And it's running longer now than any other 5.12 kernel so far,
promising. I'll leave it running and will use it as my daily driver
tomorrow.
Will keep you posted,
Rolf
On 6/1/21 11:04 PM, Rolf Fokkens wrote:
> Building a patched Fedora 5.12.8 kernel.....
>
> Will keep you posted,
>
> Rolf
>
> On 6/1/21 5:34 PM, Coly Li wrote:
>> On 5/31/21 5:37 PM, Rolf Fokkens wrote:
>>> Same here, more details: https://bugzilla.redhat.com/show_bug.cgi?id=1965809
>> Could you please try these attached patches ? I'd like to have more
>> confirmation before submitting to Jens.
>>
>> Thanks.
>>
>> Coly Li
>>
>>
>>
>>> On 5/15/21 9:06 PM, Thorsten Knabe wrote:
>>>> Hello.
>>>>
>>>> Starting with Linux 5.12 bcache triggers a BUG() after a few minutes of
>>>> usage.
>>>>
>>>> Linux up to 5.11.x is not affected by this bug.
>>>>
>>>> Environment:
>>>> - Debian 10 AMD 64
>>>> - Kernel 5.12 - 5.12.4
>>>> - Filesystem ext4
>>>> - Backing device: degraded software RAID-6 (MD) with 3 of 4 disks active
>>>> (unsure if the degraded RAID-6 has an effect or not)
>>>> - Cache device: Single SSD
>>>>
>>>> Kernel log:
>>>>
>>>> May 12 20:22:24 tek04 kernel: nr_vecs=472
>>>> May 12 20:22:24 tek04 kernel: ------------[ cut here ]------------
>>>> May 12 20:22:24 tek04 kernel: kernel BUG at block/bio.c:53!
>>>> May 12 20:22:24 tek04 kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI
>>>> May 12 20:22:24 tek04 kernel: CPU: 1 PID: 1670 Comm: grep Tainted: G
>>>> I 5.12.3 #2
>>>> May 12 20:22:24 tek04 kernel: Hardware name: To Be Filled By O.E.M. To
>>>> Be Filled By O.E.M./X58 Deluxe, BIOS P2.20 10/30/2009
>>>> May 12 20:22:24 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
>>>> May 12 20:22:24 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
>>>> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
>>>> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
>>>> e8 d4 80 fe ff
>>>> May 12 20:22:24 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
>>>> May 12 20:22:24 tek04 kernel: RAX: 000000000000000b RBX:
>>>> ffffc9000274b764 RCX: 0000000000000000
>>>> May 12 20:22:24 tek04 kernel: bch_count_backing_io_errors: 1 callbacks
>>>> suppressed
>>>> May 12 20:22:24 tek04 kernel: bcache: bch_count_backing_io_errors() md1:
>>>> Read-ahead I/O failed on backing device, ignore
>>>> May 12 20:22:24 tek04 kernel: RDX: ffff888333c5e400 RSI:
>>>> ffff888333c57480 RDI: ffff888333c57480
>>>> May 12 20:22:24 tek04 kernel: RBP: 0000000000000800 R08:
>>>> 0000000000000000 R09: c0000000ffffdfff
>>>> May 12 20:22:24 tek04 kernel: R10: ffffc9000274b580 R11:
>>>> ffffc9000274b578 R12: ffff8881170f0118
>>>> May 12 20:22:24 tek04 kernel: R13: ffff888109911b00 R14:
>>>> ffff8881170f00d0 R15: 0000000000000800
>>>> May 12 20:22:24 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
>>>> GS:ffff888333c40000(0000) knlGS:0000000000000000
>>>> May 12 20:22:24 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>>> 0000000080050033
>>>> May 12 20:22:24 tek04 kernel: CR2: 00005628963f4fd0 CR3:
>>>> 000000016c43c000 CR4: 00000000000006e0
>>>> May 12 20:22:24 tek04 kernel: Call Trace:
>>>> May 12 20:22:24 tek04 kernel: bvec_alloc+0x22/0x90
>>>> May 12 20:22:24 tek04 kernel: bio_alloc_bioset+0x176/0x230
>>>> May 12 20:22:24 tek04 kernel: cached_dev_cache_miss+0x1a8/0x300
>>>> May 12 20:22:24 tek04 kernel: cache_lookup_fn+0x110/0x2e0
>>>> May 12 20:22:24 tek04 kernel: ? bch_ptr_invalid+0x10/0x10
>>>> May 12 20:22:24 tek04 kernel: ? bch_btree_iter_next_filter+0x1af/0x2d0
>>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0x69/0x160
>>>> May 12 20:22:24 tek04 kernel: ? __bch_bset_search+0x315/0x440
>>>> May 12 20:22:24 tek04 kernel: ? downgrade_write+0xb0/0xb0
>>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0xcf/0x160
>>>> May 12 20:22:24 tek04 kernel: ? raid5_make_request+0x5c4/0xaa0
>>>> May 12 20:22:24 tek04 kernel: ? recalibrate_cpu_khz+0x10/0x10
>>>> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x30/0x400
>>>> May 12 20:22:24 tek04 kernel: ? rwsem_wake.isra.11+0x80/0x80
>>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys+0xf2/0x140
>>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>>> May 12 20:22:24 tek04 kernel: cache_lookup+0xb1/0x190
>>>> May 12 20:22:24 tek04 kernel: cached_dev_submit_bio+0x9ab/0xc90
>>>> May 12 20:22:24 tek04 kernel: ? submit_bio_checks+0x197/0x4a0
>>>> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x3b7/0x400
>>>> May 12 20:22:24 tek04 kernel: submit_bio_noacct+0x10e/0x4c0
>>>> May 12 20:22:24 tek04 kernel: submit_bio+0x2e/0x160
>>>> May 12 20:22:24 tek04 kernel: ? xa_load+0x66/0x70
>>>> May 12 20:22:24 tek04 kernel: ? bio_add_page+0x2f/0x70
>>>> May 12 20:22:24 tek04 kernel: ext4_mpage_readpages+0x1ae/0xa00
>>>> May 12 20:22:24 tek04 kernel: ? __mod_lruvec_state+0x29/0x60
>>>> May 12 20:22:24 tek04 kernel: read_pages+0x78/0x1d0
>>>> May 12 20:22:24 tek04 kernel: page_cache_ra_unbounded+0x127/0x1b0
>>>> May 12 20:22:24 tek04 kernel: filemap_get_pages+0x1d0/0x4a0
>>>> May 12 20:22:24 tek04 kernel: filemap_read+0x91/0x2d0
>>>> May 12 20:22:24 tek04 kernel: new_sync_read+0x103/0x180
>>>> May 12 20:22:24 tek04 kernel: vfs_read+0x11b/0x1b0
>>>> May 12 20:22:24 tek04 kernel: ksys_read+0x55/0xd0
>>>> May 12 20:22:24 tek04 kernel: do_syscall_64+0x33/0x80
>>>> May 12 20:22:24 tek04 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>>>> May 12 20:22:24 tek04 kernel: RIP: 0033:0x7f3a1cb89461
>>>> May 12 20:22:24 tek04 kernel: Code: fe ff ff 50 48 8d 3d fe d0 09 00 e8
>>>> e9 03 02 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 99 62 0d 00 8b 00 85 c0
>>>> 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54
>>>> 49 89 d4 55 48
>>>> May 12 20:22:24 tek04 kernel: RSP: 002b:00007fff4052fff8 EFLAGS:
>>>> 00000246 ORIG_RAX: 0000000000000000
>>>> May 12 20:22:24 tek04 kernel: RAX: ffffffffffffffda RBX:
>>>> 0000000000018000 RCX: 00007f3a1cb89461
>>>> May 12 20:22:24 tek04 kernel: RDX: 0000000000018000 RSI:
>>>> 000055ff0d01a000 RDI: 0000000000000004
>>>> May 12 20:22:24 tek04 kernel: RBP: 0000000000018000 R08:
>>>> 0000000000000002 R09: 000055ff0d0194b0
>>>> May 12 20:22:24 tek04 kernel: R10: 0000000000000000 R11:
>>>> 0000000000000246 R12: 000055ff0d01a000
>>>> May 12 20:22:24 tek04 kernel: R13: 0000000000000004 R14:
>>>> 000055ff0d0194b0 R15: 0000000000000004
>>>> May 12 20:22:24 tek04 kernel: Modules linked in: cmac bnep
>>>> intel_powerclamp snd_hda_codec_realtek snd_hda_codec_generic btusb
>>>> ledtrig_audio snd_hda_codec_hdmi btrtl kvm_intel snd_hda_intel btbcm
>>>> snd_intel_dspcfg btintel kvm snd_hda_codec irqbypass bluetooth serio_raw
>>>> pcspkr hfcpci snd_hda_core iTCO_wdt evdev input_leds joydev sg snd_hwdep
>>>> intel_pmc_bxt ecdh_generic rfkill iTCO_vendor_support mISDN_core ecc
>>>> snd_pcm snd_timer tiny_power_button snd soundcore button i7core_edac
>>>> acpi_cpufreq wmi nft_counter nf_log_ipv6 nf_log_ipv
>>>> May 12 20:22:24 tek04 kernel: ---[ end trace 9a03f30c7b4aa246 ]---
>>>> May 12 20:22:25 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
>>>> May 12 20:22:25 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
>>>> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
>>>> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
>>>> e8 d4 80 fe ff
>>>> May 12 20:22:25 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
>>>> May 12 20:22:25 tek04 kernel: RAX: 000000000000000b RBX:
>>>> ffffc9000274b764 RCX: 0000000000000000
>>>> May 12 20:22:25 tek04 kernel: RDX: ffff888333c5e400 RSI:
>>>> ffff888333c57480 RDI: ffff888333c57480
>>>> May 12 20:22:25 tek04 kernel: RBP: 0000000000000800 R08:
>>>> 0000000000000000 R09: c0000000ffffdfff
>>>> May 12 20:22:25 tek04 kernel: R10: ffffc9000274b580 R11:
>>>> ffffc9000274b578 R12: ffff8881170f0118
>>>> May 12 20:22:25 tek04 kernel: R13: ffff888109911b00 R14:
>>>> ffff8881170f00d0 R15: 0000000000000800
>>>> May 12 20:22:25 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
>>>> GS:ffff888333c40000(0000) knlGS:0000000000000000
>>>> May 12 20:22:25 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>>> 0000000080050033
>>>> May 12 20:22:25 tek04 kernel: CR2: 00005628963f4fd0 CR3:
>>>> 000000016c43c000 CR4: 00000000000006e0
>>>>
>>>> A printk has been added to line 52 of block/bio.c to dump the nr_vecs
>>>> variable to the kernel log before the BUG(). Obviously nr_vecs (472
>>>> logged) is bigger than expected by bvec_alloc/bio_alloc_bioset (max
>>>> 256), which finally triggers the BUG().
>>>>
>>>> Removing the BUG() from line 52 of block/bio.c, thus basically restoring
>>>> the Linux 5.11.x behavior of bvec_alloc/bio_alloc_bioset to just return
>>>> NULL, when nr_vecs is too big seems to resolve the issue.
>>>>
>>>> Regards
>>>> Thorsten
>>>>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-06-01 15:34 ` Coly Li
2021-06-01 21:04 ` Rolf Fokkens
@ 2021-06-02 8:33 ` Thorsten Knabe
2021-06-02 9:45 ` Rolf Fokkens
1 sibling, 1 reply; 15+ messages in thread
From: Thorsten Knabe @ 2021-06-02 8:33 UTC (permalink / raw)
To: Coly Li; +Cc: linux-bcache, Rolf Fokkens
Hello Coly.
On 6/1/21 5:34 PM, Coly Li wrote:
> On 5/31/21 5:37 PM, Rolf Fokkens wrote:
>> Same here, more details: https://bugzilla.redhat.com/show_bug.cgi?id=1965809
>
> Could you please try these attached patches ? I'd like to have more
> confirmation before submitting to Jens.
Running 5.12.8 + your v5 patches for a few hours now on that degraded
RAID-6 system without any issues. If anything changes, I'll let you know.
Thorsten
>
> Thanks.
>
> Coly Li
>
>
>
>>
>> On 5/15/21 9:06 PM, Thorsten Knabe wrote:
>>> Hello.
>>>
>>> Starting with Linux 5.12 bcache triggers a BUG() after a few minutes of
>>> usage.
>>>
>>> Linux up to 5.11.x is not affected by this bug.
>>>
>>> Environment:
>>> - Debian 10 AMD 64
>>> - Kernel 5.12 - 5.12.4
>>> - Filesystem ext4
>>> - Backing device: degraded software RAID-6 (MD) with 3 of 4 disks active
>>> (unsure if the degraded RAID-6 has an effect or not)
>>> - Cache device: Single SSD
>>>
>>> Kernel log:
>>>
>>> May 12 20:22:24 tek04 kernel: nr_vecs=472
>>> May 12 20:22:24 tek04 kernel: ------------[ cut here ]------------
>>> May 12 20:22:24 tek04 kernel: kernel BUG at block/bio.c:53!
>>> May 12 20:22:24 tek04 kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI
>>> May 12 20:22:24 tek04 kernel: CPU: 1 PID: 1670 Comm: grep Tainted: G
>>> I 5.12.3 #2
>>> May 12 20:22:24 tek04 kernel: Hardware name: To Be Filled By O.E.M. To
>>> Be Filled By O.E.M./X58 Deluxe, BIOS P2.20 10/30/2009
>>> May 12 20:22:24 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
>>> May 12 20:22:24 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
>>> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
>>> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
>>> e8 d4 80 fe ff
>>> May 12 20:22:24 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
>>> May 12 20:22:24 tek04 kernel: RAX: 000000000000000b RBX:
>>> ffffc9000274b764 RCX: 0000000000000000
>>> May 12 20:22:24 tek04 kernel: bch_count_backing_io_errors: 1 callbacks
>>> suppressed
>>> May 12 20:22:24 tek04 kernel: bcache: bch_count_backing_io_errors() md1:
>>> Read-ahead I/O failed on backing device, ignore
>>> May 12 20:22:24 tek04 kernel: RDX: ffff888333c5e400 RSI:
>>> ffff888333c57480 RDI: ffff888333c57480
>>> May 12 20:22:24 tek04 kernel: RBP: 0000000000000800 R08:
>>> 0000000000000000 R09: c0000000ffffdfff
>>> May 12 20:22:24 tek04 kernel: R10: ffffc9000274b580 R11:
>>> ffffc9000274b578 R12: ffff8881170f0118
>>> May 12 20:22:24 tek04 kernel: R13: ffff888109911b00 R14:
>>> ffff8881170f00d0 R15: 0000000000000800
>>> May 12 20:22:24 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
>>> GS:ffff888333c40000(0000) knlGS:0000000000000000
>>> May 12 20:22:24 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>> 0000000080050033
>>> May 12 20:22:24 tek04 kernel: CR2: 00005628963f4fd0 CR3:
>>> 000000016c43c000 CR4: 00000000000006e0
>>> May 12 20:22:24 tek04 kernel: Call Trace:
>>> May 12 20:22:24 tek04 kernel: bvec_alloc+0x22/0x90
>>> May 12 20:22:24 tek04 kernel: bio_alloc_bioset+0x176/0x230
>>> May 12 20:22:24 tek04 kernel: cached_dev_cache_miss+0x1a8/0x300
>>> May 12 20:22:24 tek04 kernel: cache_lookup_fn+0x110/0x2e0
>>> May 12 20:22:24 tek04 kernel: ? bch_ptr_invalid+0x10/0x10
>>> May 12 20:22:24 tek04 kernel: ? bch_btree_iter_next_filter+0x1af/0x2d0
>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0x69/0x160
>>> May 12 20:22:24 tek04 kernel: ? __bch_bset_search+0x315/0x440
>>> May 12 20:22:24 tek04 kernel: ? downgrade_write+0xb0/0xb0
>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0xcf/0x160
>>> May 12 20:22:24 tek04 kernel: ? raid5_make_request+0x5c4/0xaa0
>>> May 12 20:22:24 tek04 kernel: ? recalibrate_cpu_khz+0x10/0x10
>>> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x30/0x400
>>> May 12 20:22:24 tek04 kernel: ? rwsem_wake.isra.11+0x80/0x80
>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys+0xf2/0x140
>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>> May 12 20:22:24 tek04 kernel: cache_lookup+0xb1/0x190
>>> May 12 20:22:24 tek04 kernel: cached_dev_submit_bio+0x9ab/0xc90
>>> May 12 20:22:24 tek04 kernel: ? submit_bio_checks+0x197/0x4a0
>>> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x3b7/0x400
>>> May 12 20:22:24 tek04 kernel: submit_bio_noacct+0x10e/0x4c0
>>> May 12 20:22:24 tek04 kernel: submit_bio+0x2e/0x160
>>> May 12 20:22:24 tek04 kernel: ? xa_load+0x66/0x70
>>> May 12 20:22:24 tek04 kernel: ? bio_add_page+0x2f/0x70
>>> May 12 20:22:24 tek04 kernel: ext4_mpage_readpages+0x1ae/0xa00
>>> May 12 20:22:24 tek04 kernel: ? __mod_lruvec_state+0x29/0x60
>>> May 12 20:22:24 tek04 kernel: read_pages+0x78/0x1d0
>>> May 12 20:22:24 tek04 kernel: page_cache_ra_unbounded+0x127/0x1b0
>>> May 12 20:22:24 tek04 kernel: filemap_get_pages+0x1d0/0x4a0
>>> May 12 20:22:24 tek04 kernel: filemap_read+0x91/0x2d0
>>> May 12 20:22:24 tek04 kernel: new_sync_read+0x103/0x180
>>> May 12 20:22:24 tek04 kernel: vfs_read+0x11b/0x1b0
>>> May 12 20:22:24 tek04 kernel: ksys_read+0x55/0xd0
>>> May 12 20:22:24 tek04 kernel: do_syscall_64+0x33/0x80
>>> May 12 20:22:24 tek04 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>>> May 12 20:22:24 tek04 kernel: RIP: 0033:0x7f3a1cb89461
>>> May 12 20:22:24 tek04 kernel: Code: fe ff ff 50 48 8d 3d fe d0 09 00 e8
>>> e9 03 02 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 99 62 0d 00 8b 00 85 c0
>>> 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54
>>> 49 89 d4 55 48
>>> May 12 20:22:24 tek04 kernel: RSP: 002b:00007fff4052fff8 EFLAGS:
>>> 00000246 ORIG_RAX: 0000000000000000
>>> May 12 20:22:24 tek04 kernel: RAX: ffffffffffffffda RBX:
>>> 0000000000018000 RCX: 00007f3a1cb89461
>>> May 12 20:22:24 tek04 kernel: RDX: 0000000000018000 RSI:
>>> 000055ff0d01a000 RDI: 0000000000000004
>>> May 12 20:22:24 tek04 kernel: RBP: 0000000000018000 R08:
>>> 0000000000000002 R09: 000055ff0d0194b0
>>> May 12 20:22:24 tek04 kernel: R10: 0000000000000000 R11:
>>> 0000000000000246 R12: 000055ff0d01a000
>>> May 12 20:22:24 tek04 kernel: R13: 0000000000000004 R14:
>>> 000055ff0d0194b0 R15: 0000000000000004
>>> May 12 20:22:24 tek04 kernel: Modules linked in: cmac bnep
>>> intel_powerclamp snd_hda_codec_realtek snd_hda_codec_generic btusb
>>> ledtrig_audio snd_hda_codec_hdmi btrtl kvm_intel snd_hda_intel btbcm
>>> snd_intel_dspcfg btintel kvm snd_hda_codec irqbypass bluetooth serio_raw
>>> pcspkr hfcpci snd_hda_core iTCO_wdt evdev input_leds joydev sg snd_hwdep
>>> intel_pmc_bxt ecdh_generic rfkill iTCO_vendor_support mISDN_core ecc
>>> snd_pcm snd_timer tiny_power_button snd soundcore button i7core_edac
>>> acpi_cpufreq wmi nft_counter nf_log_ipv6 nf_log_ipv
>>> May 12 20:22:24 tek04 kernel: ---[ end trace 9a03f30c7b4aa246 ]---
>>> May 12 20:22:25 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
>>> May 12 20:22:25 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
>>> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
>>> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
>>> e8 d4 80 fe ff
>>> May 12 20:22:25 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
>>> May 12 20:22:25 tek04 kernel: RAX: 000000000000000b RBX:
>>> ffffc9000274b764 RCX: 0000000000000000
>>> May 12 20:22:25 tek04 kernel: RDX: ffff888333c5e400 RSI:
>>> ffff888333c57480 RDI: ffff888333c57480
>>> May 12 20:22:25 tek04 kernel: RBP: 0000000000000800 R08:
>>> 0000000000000000 R09: c0000000ffffdfff
>>> May 12 20:22:25 tek04 kernel: R10: ffffc9000274b580 R11:
>>> ffffc9000274b578 R12: ffff8881170f0118
>>> May 12 20:22:25 tek04 kernel: R13: ffff888109911b00 R14:
>>> ffff8881170f00d0 R15: 0000000000000800
>>> May 12 20:22:25 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
>>> GS:ffff888333c40000(0000) knlGS:0000000000000000
>>> May 12 20:22:25 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>> 0000000080050033
>>> May 12 20:22:25 tek04 kernel: CR2: 00005628963f4fd0 CR3:
>>> 000000016c43c000 CR4: 00000000000006e0
>>>
>>> A printk has been added to line 52 of block/bio.c to dump the nr_vecs
>>> variable to the kernel log before the BUG(). Obviously nr_vecs (472
>>> logged) is bigger than expected by bvec_alloc/bio_alloc_bioset (max
>>> 256), which finally triggers the BUG().
>>>
>>> Removing the BUG() from line 52 of block/bio.c, thus basically restoring
>>> the Linux 5.11.x behavior of bvec_alloc/bio_alloc_bioset to just return
>>> NULL, when nr_vecs is too big seems to resolve the issue.
>>>
>>> Regards
>>> Thorsten
>>>
>
--
___
| | / E-Mail: linux@thorsten-knabe.de
|horsten |/\nabe WWW: http://linux.thorsten-knabe.de
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-06-02 8:33 ` Thorsten Knabe
@ 2021-06-02 9:45 ` Rolf Fokkens
2021-06-02 11:08 ` Coly Li
0 siblings, 1 reply; 15+ messages in thread
From: Rolf Fokkens @ 2021-06-02 9:45 UTC (permalink / raw)
To: Thorsten Knabe, Coly Li; +Cc: linux-bcache
Hi Coli,
Things are stable so far, looks really promising:
bash-5.0$ cat /proc/version
Linux version 5.12.8-200.rf.fc33.x86_64
(mockbuild@tb-sandbox-mjolnir.local.tbai.nl) (gcc (GCC) 10.3.1 20210422
(Red Hat 10.3.1-1), GNU ld version 2.35-18.fc33) #1 SMP Tue Jun 1
23:10:39 CEST 2021
bash-5.0$ uptime
11:42:05 up 12:01, 1 user, load average: 0.84, 2.43, 3.20
bash-5.0$
I left the system running during the night, and have been using it
actively for 3 hours now.
I'll keep you posted if anything changes, but this is a major step
forward for sure (before your patches the system would freeze in minutes
after booting).
Rolf
On 6/2/21 10:33 AM, Thorsten Knabe wrote:
> Hello Coly.
>
> On 6/1/21 5:34 PM, Coly Li wrote:
>> On 5/31/21 5:37 PM, Rolf Fokkens wrote:
>>> Same here, more details: https://bugzilla.redhat.com/show_bug.cgi?id=1965809
>> Could you please try these attached patches ? I'd like to have more
>> confirmation before submitting to Jens.
> Running 5.12.8 + your v5 patches for a few hours now on that degraded
> RAID-6 system without any issues. If anything changes, I'll let you know.
>
> Thorsten
>> Thanks.
>>
>> Coly Li
>>
>>
>>
>>> On 5/15/21 9:06 PM, Thorsten Knabe wrote:
>>>> Hello.
>>>>
>>>> Starting with Linux 5.12 bcache triggers a BUG() after a few minutes of
>>>> usage.
>>>>
>>>> Linux up to 5.11.x is not affected by this bug.
>>>>
>>>> Environment:
>>>> - Debian 10 AMD 64
>>>> - Kernel 5.12 - 5.12.4
>>>> - Filesystem ext4
>>>> - Backing device: degraded software RAID-6 (MD) with 3 of 4 disks active
>>>> (unsure if the degraded RAID-6 has an effect or not)
>>>> - Cache device: Single SSD
>>>>
>>>> Kernel log:
>>>>
>>>> May 12 20:22:24 tek04 kernel: nr_vecs=472
>>>> May 12 20:22:24 tek04 kernel: ------------[ cut here ]------------
>>>> May 12 20:22:24 tek04 kernel: kernel BUG at block/bio.c:53!
>>>> May 12 20:22:24 tek04 kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI
>>>> May 12 20:22:24 tek04 kernel: CPU: 1 PID: 1670 Comm: grep Tainted: G
>>>> I 5.12.3 #2
>>>> May 12 20:22:24 tek04 kernel: Hardware name: To Be Filled By O.E.M. To
>>>> Be Filled By O.E.M./X58 Deluxe, BIOS P2.20 10/30/2009
>>>> May 12 20:22:24 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
>>>> May 12 20:22:24 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
>>>> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
>>>> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
>>>> e8 d4 80 fe ff
>>>> May 12 20:22:24 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
>>>> May 12 20:22:24 tek04 kernel: RAX: 000000000000000b RBX:
>>>> ffffc9000274b764 RCX: 0000000000000000
>>>> May 12 20:22:24 tek04 kernel: bch_count_backing_io_errors: 1 callbacks
>>>> suppressed
>>>> May 12 20:22:24 tek04 kernel: bcache: bch_count_backing_io_errors() md1:
>>>> Read-ahead I/O failed on backing device, ignore
>>>> May 12 20:22:24 tek04 kernel: RDX: ffff888333c5e400 RSI:
>>>> ffff888333c57480 RDI: ffff888333c57480
>>>> May 12 20:22:24 tek04 kernel: RBP: 0000000000000800 R08:
>>>> 0000000000000000 R09: c0000000ffffdfff
>>>> May 12 20:22:24 tek04 kernel: R10: ffffc9000274b580 R11:
>>>> ffffc9000274b578 R12: ffff8881170f0118
>>>> May 12 20:22:24 tek04 kernel: R13: ffff888109911b00 R14:
>>>> ffff8881170f00d0 R15: 0000000000000800
>>>> May 12 20:22:24 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
>>>> GS:ffff888333c40000(0000) knlGS:0000000000000000
>>>> May 12 20:22:24 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>>> 0000000080050033
>>>> May 12 20:22:24 tek04 kernel: CR2: 00005628963f4fd0 CR3:
>>>> 000000016c43c000 CR4: 00000000000006e0
>>>> May 12 20:22:24 tek04 kernel: Call Trace:
>>>> May 12 20:22:24 tek04 kernel: bvec_alloc+0x22/0x90
>>>> May 12 20:22:24 tek04 kernel: bio_alloc_bioset+0x176/0x230
>>>> May 12 20:22:24 tek04 kernel: cached_dev_cache_miss+0x1a8/0x300
>>>> May 12 20:22:24 tek04 kernel: cache_lookup_fn+0x110/0x2e0
>>>> May 12 20:22:24 tek04 kernel: ? bch_ptr_invalid+0x10/0x10
>>>> May 12 20:22:24 tek04 kernel: ? bch_btree_iter_next_filter+0x1af/0x2d0
>>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0x69/0x160
>>>> May 12 20:22:24 tek04 kernel: ? __bch_bset_search+0x315/0x440
>>>> May 12 20:22:24 tek04 kernel: ? downgrade_write+0xb0/0xb0
>>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys_recurse+0xcf/0x160
>>>> May 12 20:22:24 tek04 kernel: ? raid5_make_request+0x5c4/0xaa0
>>>> May 12 20:22:24 tek04 kernel: ? recalibrate_cpu_khz+0x10/0x10
>>>> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x30/0x400
>>>> May 12 20:22:24 tek04 kernel: ? rwsem_wake.isra.11+0x80/0x80
>>>> May 12 20:22:24 tek04 kernel: bch_btree_map_keys+0xf2/0x140
>>>> May 12 20:22:24 tek04 kernel: ? cache_lookup+0x190/0x190
>>>> May 12 20:22:24 tek04 kernel: cache_lookup+0xb1/0x190
>>>> May 12 20:22:24 tek04 kernel: cached_dev_submit_bio+0x9ab/0xc90
>>>> May 12 20:22:24 tek04 kernel: ? submit_bio_checks+0x197/0x4a0
>>>> May 12 20:22:24 tek04 kernel: ? kmem_cache_alloc+0x3b7/0x400
>>>> May 12 20:22:24 tek04 kernel: submit_bio_noacct+0x10e/0x4c0
>>>> May 12 20:22:24 tek04 kernel: submit_bio+0x2e/0x160
>>>> May 12 20:22:24 tek04 kernel: ? xa_load+0x66/0x70
>>>> May 12 20:22:24 tek04 kernel: ? bio_add_page+0x2f/0x70
>>>> May 12 20:22:24 tek04 kernel: ext4_mpage_readpages+0x1ae/0xa00
>>>> May 12 20:22:24 tek04 kernel: ? __mod_lruvec_state+0x29/0x60
>>>> May 12 20:22:24 tek04 kernel: read_pages+0x78/0x1d0
>>>> May 12 20:22:24 tek04 kernel: page_cache_ra_unbounded+0x127/0x1b0
>>>> May 12 20:22:24 tek04 kernel: filemap_get_pages+0x1d0/0x4a0
>>>> May 12 20:22:24 tek04 kernel: filemap_read+0x91/0x2d0
>>>> May 12 20:22:24 tek04 kernel: new_sync_read+0x103/0x180
>>>> May 12 20:22:24 tek04 kernel: vfs_read+0x11b/0x1b0
>>>> May 12 20:22:24 tek04 kernel: ksys_read+0x55/0xd0
>>>> May 12 20:22:24 tek04 kernel: do_syscall_64+0x33/0x80
>>>> May 12 20:22:24 tek04 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
>>>> May 12 20:22:24 tek04 kernel: RIP: 0033:0x7f3a1cb89461
>>>> May 12 20:22:24 tek04 kernel: Code: fe ff ff 50 48 8d 3d fe d0 09 00 e8
>>>> e9 03 02 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 99 62 0d 00 8b 00 85 c0
>>>> 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54
>>>> 49 89 d4 55 48
>>>> May 12 20:22:24 tek04 kernel: RSP: 002b:00007fff4052fff8 EFLAGS:
>>>> 00000246 ORIG_RAX: 0000000000000000
>>>> May 12 20:22:24 tek04 kernel: RAX: ffffffffffffffda RBX:
>>>> 0000000000018000 RCX: 00007f3a1cb89461
>>>> May 12 20:22:24 tek04 kernel: RDX: 0000000000018000 RSI:
>>>> 000055ff0d01a000 RDI: 0000000000000004
>>>> May 12 20:22:24 tek04 kernel: RBP: 0000000000018000 R08:
>>>> 0000000000000002 R09: 000055ff0d0194b0
>>>> May 12 20:22:24 tek04 kernel: R10: 0000000000000000 R11:
>>>> 0000000000000246 R12: 000055ff0d01a000
>>>> May 12 20:22:24 tek04 kernel: R13: 0000000000000004 R14:
>>>> 000055ff0d0194b0 R15: 0000000000000004
>>>> May 12 20:22:24 tek04 kernel: Modules linked in: cmac bnep
>>>> intel_powerclamp snd_hda_codec_realtek snd_hda_codec_generic btusb
>>>> ledtrig_audio snd_hda_codec_hdmi btrtl kvm_intel snd_hda_intel btbcm
>>>> snd_intel_dspcfg btintel kvm snd_hda_codec irqbypass bluetooth serio_raw
>>>> pcspkr hfcpci snd_hda_core iTCO_wdt evdev input_leds joydev sg snd_hwdep
>>>> intel_pmc_bxt ecdh_generic rfkill iTCO_vendor_support mISDN_core ecc
>>>> snd_pcm snd_timer tiny_power_button snd soundcore button i7core_edac
>>>> acpi_cpufreq wmi nft_counter nf_log_ipv6 nf_log_ipv
>>>> May 12 20:22:24 tek04 kernel: ---[ end trace 9a03f30c7b4aa246 ]---
>>>> May 12 20:22:25 tek04 kernel: RIP: 0010:biovec_slab.cold.45+0xf/0x11
>>>> May 12 20:22:25 tek04 kernel: Code: b3 ae ff 89 c6 48 c7 c7 30 82 21 82
>>>> e8 03 81 fe ff b8 b6 ff ff ff e9 3d bc ae ff 0f b7 f7 48 c7 c7 c4 82 21
>>>> 82 e8 ea 80 fe ff <0f> 0b 49 8b b4 24 d0 00 00 00 48 c7 c7 40 84 21 82
>>>> e8 d4 80 fe ff
>>>> May 12 20:22:25 tek04 kernel: RSP: 0018:ffffc9000274b730 EFLAGS: 00010292
>>>> May 12 20:22:25 tek04 kernel: RAX: 000000000000000b RBX:
>>>> ffffc9000274b764 RCX: 0000000000000000
>>>> May 12 20:22:25 tek04 kernel: RDX: ffff888333c5e400 RSI:
>>>> ffff888333c57480 RDI: ffff888333c57480
>>>> May 12 20:22:25 tek04 kernel: RBP: 0000000000000800 R08:
>>>> 0000000000000000 R09: c0000000ffffdfff
>>>> May 12 20:22:25 tek04 kernel: R10: ffffc9000274b580 R11:
>>>> ffffc9000274b578 R12: ffff8881170f0118
>>>> May 12 20:22:25 tek04 kernel: R13: ffff888109911b00 R14:
>>>> ffff8881170f00d0 R15: 0000000000000800
>>>> May 12 20:22:25 tek04 kernel: FS: 00007f3a1ca7cb80(0000)
>>>> GS:ffff888333c40000(0000) knlGS:0000000000000000
>>>> May 12 20:22:25 tek04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>>> 0000000080050033
>>>> May 12 20:22:25 tek04 kernel: CR2: 00005628963f4fd0 CR3:
>>>> 000000016c43c000 CR4: 00000000000006e0
>>>>
>>>> A printk has been added to line 52 of block/bio.c to dump the nr_vecs
>>>> variable to the kernel log before the BUG(). Obviously nr_vecs (472
>>>> logged) is bigger than expected by bvec_alloc/bio_alloc_bioset (max
>>>> 256), which finally triggers the BUG().
>>>>
>>>> Removing the BUG() from line 52 of block/bio.c, thus basically restoring
>>>> the Linux 5.11.x behavior of bvec_alloc/bio_alloc_bioset to just return
>>>> NULL, when nr_vecs is too big seems to resolve the issue.
>>>>
>>>> Regards
>>>> Thorsten
>>>>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-06-02 9:45 ` Rolf Fokkens
@ 2021-06-02 11:08 ` Coly Li
2021-06-04 9:07 ` Rolf Fokkens
0 siblings, 1 reply; 15+ messages in thread
From: Coly Li @ 2021-06-02 11:08 UTC (permalink / raw)
To: Rolf Fokkens, Thorsten Knabe; +Cc: linux-bcache
On 6/2/21 5:45 PM, Rolf Fokkens wrote:
> Hi Coli,
>
> Things are stable so far, looks really promising:
>
> bash-5.0$ cat /proc/version
> Linux version 5.12.8-200.rf.fc33.x86_64
> (mockbuild@tb-sandbox-mjolnir.local.tbai.nl) (gcc (GCC) 10.3.1 20210422
> (Red Hat 10.3.1-1), GNU ld version 2.35-18.fc33) #1 SMP Tue Jun 1
> 23:10:39 CEST 2021
> bash-5.0$ uptime
> 11:42:05 up 12:01, 1 user, load average: 0.84, 2.43, 3.20
> bash-5.0$
>
> I left the system running during the night, and have been using it
> actively for 3 hours now.
Hi Rolf and Thorsten,
Thank you all for the status update!
>
> I'll keep you posted if anything changes, but this is a major step
> forward for sure (before your patches the system would freeze in minutes
> after booting).
Hope we have the luck in next 48 hours.
Coly Li
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-06-02 11:08 ` Coly Li
@ 2021-06-04 9:07 ` Rolf Fokkens
2021-06-04 10:35 ` Coly Li
0 siblings, 1 reply; 15+ messages in thread
From: Rolf Fokkens @ 2021-06-04 9:07 UTC (permalink / raw)
To: Coly Li, Thorsten Knabe; +Cc: linux-bcache
Hi Coly, Thorsten,
I survived 48 hours perfectly:
bash-5.0$ uptime
10:45:53 up 2 days, 11:05, 1 user, load average: 2.82, 3.45, 2.67
bash-5.0$ cat /proc/version
Linux version 5.12.8-200.rf.fc33.x86_64
(mockbuild@tb-sandbox-mjolnir.local.tbai.nl) (gcc (GCC) 10.3.1 20210422
(Red Hat 10.3.1-1), GNU ld version 2.35-18.fc33) #1 SMP Tue Jun 1
23:10:39 CEST 2021
bash-5.0$
Furthermore there are no concerning messages in the syslog.
Best,
Rolf
On 6/2/21 1:08 PM, Coly Li wrote:
> On 6/2/21 5:45 PM, Rolf Fokkens wrote:
>> Hi Coli,
>>
>> Things are stable so far, looks really promising:
>>
>> bash-5.0$ cat /proc/version
>> Linux version 5.12.8-200.rf.fc33.x86_64
>> (mockbuild@tb-sandbox-mjolnir.local.tbai.nl) (gcc (GCC) 10.3.1 20210422
>> (Red Hat 10.3.1-1), GNU ld version 2.35-18.fc33) #1 SMP Tue Jun 1
>> 23:10:39 CEST 2021
>> bash-5.0$ uptime
>> 11:42:05 up 12:01, 1 user, load average: 0.84, 2.43, 3.20
>> bash-5.0$
>>
>> I left the system running during the night, and have been using it
>> actively for 3 hours now.
>
> Hi Rolf and Thorsten,
>
> Thank you all for the status update!
>
>> I'll keep you posted if anything changes, but this is a major step
>> forward for sure (before your patches the system would freeze in minutes
>> after booting).
>
> Hope we have the luck in next 48 hours.
>
> Coly Li
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-06-04 9:07 ` Rolf Fokkens
@ 2021-06-04 10:35 ` Coly Li
2021-06-04 12:06 ` Thorsten Knabe
[not found] ` <ec9f73fa-a16c-b0c1-d1f8-2bf4e585be5f@rolffokkens.nl>
0 siblings, 2 replies; 15+ messages in thread
From: Coly Li @ 2021-06-04 10:35 UTC (permalink / raw)
To: Rolf Fokkens; +Cc: linux-bcache, Thorsten Knabe
On 6/4/21 5:07 PM, Rolf Fokkens wrote:
> Hi Coly, Thorsten,
>
> I survived 48 hours perfectly:
>
> bash-5.0$ uptime
> 10:45:53 up 2 days, 11:05, 1 user, load average: 2.82, 3.45, 2.67
> bash-5.0$ cat /proc/version
> Linux version 5.12.8-200.rf.fc33.x86_64
> (mockbuild@tb-sandbox-mjolnir.local.tbai.nl) (gcc (GCC) 10.3.1 20210422
> (Red Hat 10.3.1-1), GNU ld version 2.35-18.fc33) #1 SMP Tue Jun 1
> 23:10:39 CEST 2021
> bash-5.0$
>
> Furthermore there are no concerning messages in the syslog.
Hi Rolf,
Thanks for your update. Which kind of applications/workload are running
in the past 2 days ?
Coly Li
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-06-04 10:35 ` Coly Li
@ 2021-06-04 12:06 ` Thorsten Knabe
2021-06-05 15:20 ` Coly Li
[not found] ` <ec9f73fa-a16c-b0c1-d1f8-2bf4e585be5f@rolffokkens.nl>
1 sibling, 1 reply; 15+ messages in thread
From: Thorsten Knabe @ 2021-06-04 12:06 UTC (permalink / raw)
To: Coly Li; +Cc: linux-bcache, Rolf Fokkens
Hi Coly.
On 6/4/21 12:35 PM, Coly Li wrote:
> On 6/4/21 5:07 PM, Rolf Fokkens wrote:
>> Hi Coly, Thorsten,
>>
>> I survived 48 hours perfectly:
>>
>> bash-5.0$ uptime
>> 10:45:53 up 2 days, 11:05, 1 user, load average: 2.82, 3.45, 2.67
>> bash-5.0$ cat /proc/version
>> Linux version 5.12.8-200.rf.fc33.x86_64
>> (mockbuild@tb-sandbox-mjolnir.local.tbai.nl) (gcc (GCC) 10.3.1 20210422
>> (Red Hat 10.3.1-1), GNU ld version 2.35-18.fc33) #1 SMP Tue Jun 1
>> 23:10:39 CEST 2021
>> bash-5.0$
>>
>> Furthermore there are no concerning messages in the syslog.
>
> Hi Rolf,
>
> Thanks for your update. Which kind of applications/workload are running
> in the past 2 days ?
>
> Coly Li
>
No problems observed here too. Made a few (intended) reboots, ran a some
kernel builds and made a full backup during the last two days. Kernel is
now a stock 5.12.9.
Just booted the system from an USB stick (kernel here: debian
5.10.0-0.bpo.5-amd64) and performed a filesystem (ext4) check, which
reported no errors.
Thorsten
--
___
| | / E-Mail: linux@thorsten-knabe.de
|horsten |/\nabe WWW: http://linux.thorsten-knabe.de
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
2021-06-04 12:06 ` Thorsten Knabe
@ 2021-06-05 15:20 ` Coly Li
0 siblings, 0 replies; 15+ messages in thread
From: Coly Li @ 2021-06-05 15:20 UTC (permalink / raw)
To: Thorsten Knabe, Rolf Fokkens; +Cc: linux-bcache
On 6/4/21 8:06 PM, Thorsten Knabe wrote:
> Hi Coly.
>
> On 6/4/21 12:35 PM, Coly Li wrote:
>> On 6/4/21 5:07 PM, Rolf Fokkens wrote:
>>> Hi Coly, Thorsten,
>>>
>>> I survived 48 hours perfectly:
>>>
>>> bash-5.0$ uptime
>>> 10:45:53 up 2 days, 11:05, 1 user, load average: 2.82, 3.45, 2.67
>>> bash-5.0$ cat /proc/version
>>> Linux version 5.12.8-200.rf.fc33.x86_64
>>> (mockbuild@tb-sandbox-mjolnir.local.tbai.nl) (gcc (GCC) 10.3.1 20210422
>>> (Red Hat 10.3.1-1), GNU ld version 2.35-18.fc33) #1 SMP Tue Jun 1
>>> 23:10:39 CEST 2021
>>> bash-5.0$
>>>
>>> Furthermore there are no concerning messages in the syslog.
>> Hi Rolf,
>>
>> Thanks for your update. Which kind of applications/workload are running
>> in the past 2 days ?
>>
>> Coly Li
>>
> No problems observed here too. Made a few (intended) reboots, ran a some
> kernel builds and made a full backup during the last two days. Kernel is
> now a stock 5.12.9.
> Just booted the system from an USB stick (kernel here: debian
> 5.10.0-0.bpo.5-amd64) and performed a filesystem (ext4) check, which
> reported no errors.
Hi Thorsten and Rolf,
Thank you all for the testing and detail information.
Now I am installing a dedicate machine to use bcache for my git mirrors,
daily downloading and compiling testing.
Let's me see how it works for such workloads.
Coly Li
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: PROBLEM: bcache related kernel BUG() since Linux 5.12
[not found] ` <ec9f73fa-a16c-b0c1-d1f8-2bf4e585be5f@rolffokkens.nl>
@ 2021-06-07 10:11 ` Coly Li
0 siblings, 0 replies; 15+ messages in thread
From: Coly Li @ 2021-06-07 10:11 UTC (permalink / raw)
To: Rolf Fokkens, Thorsten Knabe; +Cc: linux-bcache
On 6/4/21 9:38 PM, Rolf Fokkens wrote:
> Hoi Coly,
>
> The common applications I used, which for sure benefit from bcache:
>
> * Gnome 3.38
> * Libreoffice-7.0.6.2
> * teams-1.4.00.7556
> * Terminator
> * Evolution-3.38.4
> * Thunderbird evolution-3.38.4
> * thunderbird-78.10.1
> * firefox-88.0.1
> * google-chrome-stable-91.0.4472.77
>
> So it's a typical desktop wordload with most I/O during startup of
> applications.
>
> For overall stress/stability testing:
>
> * Doom (steam/proton)
> * Rise of the Tombraider (steam)
> * The Witcher 3 (steam)
>
> Overall stats after (should have done before) starting the steam games:
>
> bash-5.0$ bcache-status
> --- bcache ---
> UUID b191549d-4455-43ca-b9b8-8e32dd68751c
> Block Size 512 B
> Bucket Size 512.00 KiB
> Congested? False
> Read Congestion 0.0ms
> Write Congestion 20.0ms
> Total Cache Size 128 GiB
> Total Cache Used 128 GiB (100%)
> Total Cache Unused 0 B (0%)
> Evictable Cache 106 GiB (83%)
> Replacement Policy [lru] fifo random
> Cache Mode writethrough [writeback] writearound none
> Total Hits 3006361 (95%)
> Total Misses 126463
> Total Bypass Hits 4552 (68%)
> Total Bypass Misses 2061
> Total Bypassed 730.1 MiB
> bash-5.0$
>
> Hope this gives you a good impression of the workload.
>
> Let me know if you like me to do specific stress tests.
[snipped]
Hi Rolf and Thorsten,
I run the following workloads for 48+ hours, no panic or data corruption
so far,
- tar, untar, gzip, gunzip
- git clone/fsck/gc/archive
- copy iso files, checksum calculation and check for all the iso files
- kernel source code compiling
- ext4 file system check
The fix might not be perfect yet, but IMHO we should provide a fix now.
If there is any other known issue triggered, let's fix and verify later.
Thank you all for the testing and verification these days.
Coly Li
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2021-06-07 10:11 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-15 19:06 PROBLEM: bcache related kernel BUG() since Linux 5.12 Thorsten Knabe
2021-05-16 8:42 ` Matthias Ferdinand
2021-05-16 14:43 ` Coly Li
2021-05-31 9:37 ` Rolf Fokkens
2021-06-01 15:34 ` Coly Li
2021-06-01 21:04 ` Rolf Fokkens
2021-06-01 21:46 ` Rolf Fokkens
2021-06-02 8:33 ` Thorsten Knabe
2021-06-02 9:45 ` Rolf Fokkens
2021-06-02 11:08 ` Coly Li
2021-06-04 9:07 ` Rolf Fokkens
2021-06-04 10:35 ` Coly Li
2021-06-04 12:06 ` Thorsten Knabe
2021-06-05 15:20 ` Coly Li
[not found] ` <ec9f73fa-a16c-b0c1-d1f8-2bf4e585be5f@rolffokkens.nl>
2021-06-07 10:11 ` Coly Li
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).