From: "Paweł Wiejacha" <pawel.wiejacha@rtbhouse.com>
To: Song Liu <song@kernel.org>
Cc: linux-raid <linux-raid@vger.kernel.org>,
Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Subject: Re: PROBLEM: double fault in md_end_io
Date: Thu, 15 Apr 2021 17:35:59 +0200 [thread overview]
Message-ID: <CADLTsw3pHkqnykSHRGhdksy8yEpSkoyo4n50cPiV6hFk25Yn7w@mail.gmail.com> (raw)
In-Reply-To: <CAPhsuW6V4-ujDZJopCyAfTpLqDuW1bXX+SGgxqwnbFmR3uEWGQ@mail.gmail.com>
> Pawel, have you tried to repro with md-next branch?
Yes, and it also crashes:
$ git log --decorate --format=oneline
ca882ba4c0478c68f927fa7b622ec17bde9361ce (HEAD -> md-next-pw2) using
slab_pool for md_io_pool
2715e61834586cef8292fcaa457cbf2da955a3b8 (song-md/md-next) md/bitmap:
wait for external bitmap writes to complete during tear down
c84917aa5dfc4c809634120fb429f0ff590a1f75 md: do not return existing
mddevs from mddev_find_or_alloc
<0>[ 2086.596361] traps: PANIC: double fault, error_code: 0x0
<4>[ 2086.596364] double fault: 0000 [#1] SMP NOPTI
<4>[ 2086.596365] CPU: 40 PID: 0 Comm: swapper/40 Tainted: G
OE 5.12.0-rc3-md-next-pw-fix2 #1
<4>[ 2086.596365] Hardware name: empty S8030GM2NE/S8030GM2NE, BIOS
V4.00 03/11/2021
<4>[ 2086.596366] RIP: 0010:__slab_free+0x26/0x380
<4>[ 2086.596367] Code: 1f 44 00 00 0f 1f 44 00 00 55 49 89 ca 45 89
c3 48 89 e5 41 57 41 56 49 89 fe 41 55 41 54 49 89 f4 53 48 83 e4 f0
48 83 ec 70 <48> 89 54 24 28 0f 1f 44 00 00 41 8b 46 28 4d 8b 6c 24 20
49 8b 5c
<4>[ 2086.596368] RSP: 0018:ffff96fa4d1c8f90 EFLAGS: 00010086
<4>[ 2086.596369] RAX: ffff892f4fc35300 RBX: ffff890fdca3ff78 RCX:
ffff890fdca3ff78
<4>[ 2086.596370] RDX: ffff890fdca3ff78 RSI: fffff393c1728fc0 RDI:
ffff892fd3a9b300
<4>[ 2086.596370] RBP: ffff96fa4d1c9030 R08: 0000000000000001 R09:
ffffffffb66500a7
<4>[ 2086.596371] R10: ffff890fdca3ff78 R11: 0000000000000001 R12:
fffff393c1728fc0
<4>[ 2086.596371] R13: fffff393c1728fc0 R14: ffff892fd3a9b300 R15:
0000000000000000
<4>[ 2086.596372] FS: 0000000000000000(0000)
GS:ffff892f4fc00000(0000) knlGS:0000000000000000
<4>[ 2086.596373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[ 2086.596373] CR2: ffff96fa4d1c8f88 CR3: 00000010c8210000 CR4:
0000000000350ee0
<4>[ 2086.596373] Call Trace:
<4>[ 2086.596374] <IRQ>
<4>[ 2086.596374] ? kmem_cache_free+0x3d2/0x420
<4>[ 2086.596374] ? mempool_free_slab+0x17/0x20
<4>[ 2086.596375] ? mempool_free_slab+0x17/0x20
<4>[ 2086.596375] ? mempool_free+0x2f/0x80
<4>[ 2086.596376] ? md_end_io+0x47/0x60
<4>[ 2086.596376] ? bio_endio+0xee/0x140
<4>[ 2086.596376] ? bio_chain_endio+0x2d/0x40
<4>[ 2086.596377] ? md_end_io+0x59/0x60
<4>[ 2086.596377] ? bio_endio+0xee/0x140
<4>[ 2086.596378] ? bio_chain_endio+0x2d/0x40
...
<4>[ 2086.596485] Lost 340 message(s)!
Dumping last: `watch -n0.1 ...`:
stat 451 0 13738 76 67823521 0 17162428974 801228700 351 1886784
801228776 0 0 0 0 0 0
inflight 0 354
md_io/aliases 4
md_io/align 8
md_io/cache_dma 0
md_io/cpu_partial 30
md_io/cpu_slabs 1235 N0=288 N1=239 N2=463 N3=245
md_io/destroy_by_rcu 0
md_io/hwcache_align 0
md_io/min_partial 5
md_io/objects 126582 N0=26928 N1=28254 N2=44064 N3=27336
md_io/object_size 40
md_io/objects_partial 0
md_io/objs_per_slab 102
md_io/order 0
md_io/partial 8 N2=4 N3=4
md_io/poison 0
md_io/reclaim_account 0
md_io/red_zone 0
md_io/remote_node_defrag_ratio 100
md_io/sanity_checks 0
md_io/slabs 1249 N0=264 N1=277 N2=436 N3=272
md_io/slabs_cpu_partial 1172(1172) C0=14(14) C1=17(17) C2=17(17)
C3=17(17) C4=18(18) C5=17(17) C6=18(18) C7=17(17) C8=21(21) C9=4(4)
C10=21(21) C11=22(22) C12=21(21) C13=25(25) C14=15(15) C15
=19(19) C16=24(24) C17=19(19) C18=22(22) C19=15(15) C20=28(28)
C21=18(18) C22=18(18) C23=24(24) C24=17(17) C25=27(27) C26=8(8)
C27=18(18) C28=17(17) C29=18(18) C30=21(21) C31=21(21) C32=9(9)
C33=9(9) C34=24(24) C35=11(11) C36=19(19) C37=14(14) C38=18(18)
C39=4(4) C40=23(23) C41=20(20) C42=18(18) C43=22(22) C44=21(21)
C45=24(24) C46=20(20) C47=15(15) C48=17(17) C49=15(15) C50=16(1
6) C51=26(26) C52=21(21) C53=19(19) C54=16(16) C55=18(18) C56=26(26)
C57=14(14) C58=18(18) C59=23(23) C60=18(18) C61=20(20) C62=19(19)
C63=17(17)
md_io/slab_size 40
md_io/store_user 0
md_io/total_objects 127398 N0=26928 N1=28254 N2=44472 N3=27744
md_io/trace 0
md_io/usersize 0
> I am not able to reproduce the issue [...]
It crashes on two different machines. Next week I'm going to upgrade a
distro on an older machine (with Intel NVMe disks, different
motherboard and Xeon instead of EPYC 2 CPU) running currently
linux-5.4 without this issue. So I will let you know if switching to a
newer kernel with "improved io stats accounting" makes it unstable or
not.
Best regards,
Pawel Wiejacha.
On Thu, 15 Apr 2021 at 08:35, Song Liu <song@kernel.org> wrote:
>
> On Wed, Apr 14, 2021 at 5:36 PM Song Liu <song@kernel.org> wrote:
> >
> > On Tue, Apr 13, 2021 at 5:05 AM Paweł Wiejacha
> > <pawel.wiejacha@rtbhouse.com> wrote:
> > >
> > > Hello Song,
> > >
> > > That code does not compile, but I guess that what you meant was
> > > something like this:
> >
> > Yeah.. I am really sorry for the noise.
> >
> > >
> > > diff --git drivers/md/md.c drivers/md/md.c
> > > index 04384452a..cbc97a96b 100644
> > > --- drivers/md/md.c
> > > +++ drivers/md/md.c
> > > @@ -78,6 +78,7 @@ static DEFINE_SPINLOCK(pers_lock);
> > >
> > > static struct kobj_type md_ktype;
> > >
> > > +struct kmem_cache *md_io_cache;
> > > struct md_cluster_operations *md_cluster_ops;
> > > EXPORT_SYMBOL(md_cluster_ops);
> > > static struct module *md_cluster_mod;
> > > @@ -5701,8 +5702,8 @@ static int md_alloc(dev_t dev, char *name)
> > > */
> > > mddev->hold_active = UNTIL_STOP;
> > >
> > > - error = mempool_init_kmalloc_pool(&mddev->md_io_pool, BIO_POOL_SIZE,
> > > - sizeof(struct md_io));
> > > + error = mempool_init_slab_pool(&mddev->md_io_pool, BIO_POOL_SIZE,
> > > + md_io_cache);
> > > if (error)
> > > goto abort;
> > >
> > > @@ -9542,6 +9543,10 @@ static int __init md_init(void)
> > > {
> > > int ret = -ENOMEM;
> > >
> > > + md_io_cache = KMEM_CACHE(md_io, 0);
> > > + if (!md_io_cache)
> > > + goto err_md_io_cache;
> > > +
> > > md_wq = alloc_workqueue("md", WQ_MEM_RECLAIM, 0);
> > > if (!md_wq)
> > > goto err_wq;
> > > @@ -9578,6 +9583,8 @@ static int __init md_init(void)
> > > err_misc_wq:
> > > destroy_workqueue(md_wq);
> > > err_wq:
> > > + kmem_cache_destroy(md_io_cache);
> > > +err_md_io_cache:
> > > return ret;
> > > }
> > >
> > > @@ -9863,6 +9870,7 @@ static __exit void md_exit(void)
> > > destroy_workqueue(md_rdev_misc_wq);
> > > destroy_workqueue(md_misc_wq);
> > > destroy_workqueue(md_wq);
> > > + kmem_cache_destroy(md_io_cache);
> > > }
> > >
> > > subsys_initcall(md_init);
> >
> > [...]
> >
> > >
> > > $ watch -n0.2 'cat /proc/meminfo | paste - - | tee -a ~/meminfo'
> > > MemTotal: 528235648 kB MemFree: 20002732 kB
> > > MemAvailable: 483890268 kB Buffers: 7356 kB
> > > Cached: 495416180 kB SwapCached: 0 kB
> > > Active: 96396800 kB Inactive: 399891308 kB
> > > Active(anon): 10976 kB Inactive(anon): 890908 kB
> > > Active(file): 96385824 kB Inactive(file): 399000400 kB
> > > Unevictable: 78768 kB Mlocked: 78768 kB
> > > SwapTotal: 0 kB SwapFree: 0 kB
> > > Dirty: 88422072 kB Writeback: 948756 kB
> > > AnonPages: 945772 kB Mapped: 57300 kB
> > > Shmem: 26300 kB KReclaimable: 7248160 kB
> > > Slab: 7962748 kB SReclaimable: 7248160 kB
> > > SUnreclaim: 714588 kB KernelStack: 18288 kB
> > > PageTables: 10796 kB NFS_Unstable: 0 kB
> > > Bounce: 0 kB WritebackTmp: 0 kB
> > > CommitLimit: 264117824 kB Committed_AS: 21816824 kB
> > > VmallocTotal: 34359738367 kB VmallocUsed: 561588 kB
> > > VmallocChunk: 0 kB Percpu: 65792 kB
> > > HardwareCorrupted: 0 kB AnonHugePages: 0 kB
> > > ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB
> > > FileHugePages: 0 kB FilePmdMapped: 0 kB
> > > HugePages_Total: 0 HugePages_Free: 0
> > > HugePages_Rsvd: 0 HugePages_Surp: 0
> > > Hugepagesize: 2048 kB Hugetlb: 0 kB
> > > DirectMap4k: 541000 kB DirectMap2M: 11907072 kB
> > > DirectMap1G: 525336576 kB
> > >
> >
> > And thanks for these information.
> >
> > I have set up a system to run the test, the code I am using is the top of the
> > md-next branch. I will update later tonight on the status.
>
> I am not able to reproduce the issue after 6 hours. Maybe it is because I run
> tests on 3 partitions of the same nvme SSD. I will try on a different host with
> multiple SSDs.
>
> Pawel, have you tried to repro with md-next branch?
>
> Thanks,
> Song
next prev parent reply other threads:[~2021-04-15 15:36 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-09 21:40 PROBLEM: double fault in md_end_io Paweł Wiejacha
2021-04-12 6:48 ` Song Liu
2021-04-13 12:05 ` Paweł Wiejacha
2021-04-15 0:36 ` Song Liu
2021-04-15 6:35 ` Song Liu
2021-04-15 15:35 ` Paweł Wiejacha [this message]
2021-04-22 15:40 ` Paweł Wiejacha
2021-04-23 2:36 ` Guoqing Jiang
2021-04-23 6:44 ` Song Liu
2021-05-04 21:17 ` Paweł Wiejacha
2021-05-06 5:48 ` Song Liu
2021-05-06 23:46 ` Guoqing Jiang
2021-05-08 1:17 ` Guoqing Jiang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CADLTsw3pHkqnykSHRGhdksy8yEpSkoyo4n50cPiV6hFk25Yn7w@mail.gmail.com \
--to=pawel.wiejacha@rtbhouse.com \
--cc=artur.paszkiewicz@intel.com \
--cc=linux-raid@vger.kernel.org \
--cc=song@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.