* "mm: reparent slab memory on cgroup removal" series triggers SLUB_DEBUG errors
@ 2019-06-18 21:43 Qian Cai
2019-06-19 3:09 ` Roman Gushchin
0 siblings, 1 reply; 3+ messages in thread
From: Qian Cai @ 2019-06-18 21:43 UTC (permalink / raw)
To: Roman Gushchin
Cc: Vladimir Davydov, Andrew Morton, Linux MM,
Linux Kernel Mailing List, Johannes Weiner
Booting linux-next on both arm64 and powerpc triggers SLUB_DEBUG errors below. Reverted the whole series “mm: reparent slab memory on cgroup removal” [1] fixed the issue.
[1] https://lore.kernel.org/lkml/20190611231813.3148843-1-guro@fb.com/
[ 151.773224][ T1650] BUG kmem_cache (Tainted: G B W ): Poison overwritten
[ 151.780969][ T1650] -----------------------------------------------------------------------------
[ 151.780969][ T1650]
[ 151.792016][ T1650] INFO: 0x000000001fd6fdef-0x0000000007f6bb36. First byte 0x0 instead of 0x6b
[ 151.800726][ T1650] INFO: Allocated in create_cache+0x6c/0x1bc age=24301 cpu=97 pid=1444
[ 151.808821][ T1650] kmem_cache_alloc+0x514/0x568
[ 151.813527][ T1650] create_cache+0x6c/0x1bc
[ 151.817800][ T1650] memcg_create_kmem_cache+0xfc/0x11c
[ 151.823028][ T1650] memcg_kmem_cache_create_func+0x40/0x170
[ 151.828691][ T1650] process_one_work+0x4e0/0xa54
[ 151.833398][ T1650] worker_thread+0x498/0x650
[ 151.837843][ T1650] kthread+0x1b8/0x1d4
[ 151.841770][ T1650] ret_from_fork+0x10/0x18
[ 151.846046][ T1650] INFO: Freed in slab_kmem_cache_release+0x3c/0x48 age=23341 cpu=28 pid=1480
[ 151.854659][ T1650] slab_kmem_cache_release+0x3c/0x48
[ 151.859799][ T1650] kmem_cache_release+0x1c/0x28
[ 151.864507][ T1650] kobject_cleanup+0x134/0x288
[ 151.869127][ T1650] kobject_put+0x5c/0x68
[ 151.873226][ T1650] sysfs_slab_release+0x2c/0x38
[ 151.877931][ T1650] shutdown_cache+0x198/0x23c
[ 151.882464][ T1650] kmemcg_cache_shutdown_fn+0x1c/0x34
[ 151.887691][ T1650] kmemcg_workfn+0x44/0x68
[ 151.891963][ T1650] process_one_work+0x4e0/0xa54
[ 151.896668][ T1650] worker_thread+0x498/0x650
[ 151.901113][ T1650] kthread+0x1b8/0x1d4
[ 151.905037][ T1650] ret_from_fork+0x10/0x18
[ 151.909324][ T1650] INFO: Slab 0x00000000406d65a6 objects=64 used=64 fp=0x000000004d988e71 flags=0x7ffffffc000200
[ 151.919596][ T1650] INFO: Object 0x0000000040f4b79e @offset=15420325124116637824 fp=0x00000000e038adbf
[ 151.919596][ T1650]
[ 151.931079][ T1650] Redzone 00000000fc4c04f0: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
[ 151.941168][ T1650] Redzone 000000009a25c019: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
[ 151.951256][ T1650] Redzone 000000000b05c7cc: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
[ 151.961345][ T1650] Redzone 00000000a08ae38b: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
[ 151.971433][ T1650] Redzone 00000000e0eccd41: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
[ 151.981520][ T1650] Redzone 0000000016ee2661: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
[ 151.991608][ T1650] Redzone 000000009364e729: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
[ 152.001695][ T1650] Redzone 00000000f2202456: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
[ 152.011784][ T1650] Object 0000000040f4b79e: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.021783][ T1650] Object 000000002df21fec: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.031779][ T1650] Object 0000000041cf0887: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.041775][ T1650] Object 00000000bfb91e8f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.051770][ T1650] Object 00000000da315b1c: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.061765][ T1650] Object 00000000b362de78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.071761][ T1650] Object 00000000ad4f72bf: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.081756][ T1650] Object 00000000aa32d346: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.091751][ T1650] Object 00000000ad1cf22c: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.101746][ T1650] Object 000000001cee47e4: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.111741][ T1650] Object 00000000418720ed: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.121736][ T1650] Object 00000000dee1c3f2: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.131731][ T1650] Object 00000000a23397c1: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.141727][ T1650] Object 000000002ed01641: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.151721][ T1650] Object 00000000915ec720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.161716][ T1650] Object 00000000915988c1: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.171711][ T1650] Object 000000004a0cc60f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.181707][ T1650] Object 0000000054a294c9: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.191701][ T1650] Object 0000000054f61682: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.201697][ T1650] Object 0000000018d04328: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.211692][ T1650] Object 00000000703cf2c7: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.221687][ T1650] Object 000000004d3ac5d5: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 kkkkkkkk........
[ 152.231682][ T1650] Object 00000000726ce587: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.241676][ T1650] Object 00000000c709b64e: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.251672][ T1650] Object 0000000044d6a5c6: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.261667][ T1650] Object 000000009c76a6a2: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.271662][ T1650] Object 0000000033d01d12: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.281657][ T1650] Object 00000000c50ff26f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.291652][ T1650] Object 00000000ebc3aaae: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.301647][ T1650] Object 00000000a2072fe3: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 152.311641][ T1650] Object 000000003d5911a3: 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkk.
[ 152.320942][ T1650] Redzone 000000009a2feac1: bb bb bb bb bb bb bb bb ........
[ 152.330330][ T1650] Padding 00000000c1b3cb8b: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 152.340412][ T1650] Padding 000000003715421a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 152.350493][ T1650] Padding 0000000066b51ba7: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 152.360575][ T1650] Padding 00000000ca240306: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 152.370657][ T1650] Padding 0000000014a2af5d: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
[ 152.380048][ T1650] CPU: 82 PID: 1650 Comm: kworker/82:1 Tainted: G B W 5.2.0-rc5-next-20190617 #18
[ 152.390216][ T1650] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.0.9 03/01/2019
[ 152.400741][ T1650] Workqueue: memcg_kmem_cache memcg_kmem_cache_create_func
[ 152.407786][ T1650] Call trace:
[ 152.410926][ T1650] dump_backtrace+0x0/0x268
[ 152.415280][ T1650] show_stack+0x20/0x2c
[ 152.419287][ T1650] dump_stack+0xb4/0x108
[ 152.423384][ T1650] print_trailer+0x274/0x298
[ 152.427825][ T1650] check_bytes_and_report+0xc4/0x118
[ 152.432959][ T1650] check_object+0x2fc/0x36c
[ 152.437312][ T1650] alloc_debug_processing+0x154/0x240
[ 152.442532][ T1650] ___slab_alloc+0x710/0xa68
[ 152.446972][ T1650] kmem_cache_alloc+0x514/0x568
[ 152.451672][ T1650] create_cache+0x6c/0x1bc
[ 152.455938][ T1650] memcg_create_kmem_cache+0xfc/0x11c
[ 152.461158][ T1650] memcg_kmem_cache_create_func+0x40/0x170
[ 152.466814][ T1650] process_one_work+0x4e0/0xa54
[ 152.471515][ T1650] worker_thread+0x498/0x650
[ 152.475953][ T1650] kthread+0x1b8/0x1d4
[ 152.479872][ T1650] ret_from_fork+0x10/0x18
[ 152.484139][ T1650] FIX kmem_cache: Restoring 0x000000001fd6fdef-0x0000000007f6bb36=0x6b
[ 152.484139][ T1650]
[ 152.494395][ T1650] FIX kmem_cache: Marking all objects used
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: "mm: reparent slab memory on cgroup removal" series triggers SLUB_DEBUG errors
2019-06-18 21:43 "mm: reparent slab memory on cgroup removal" series triggers SLUB_DEBUG errors Qian Cai
@ 2019-06-19 3:09 ` Roman Gushchin
2019-06-19 13:36 ` Qian Cai
0 siblings, 1 reply; 3+ messages in thread
From: Roman Gushchin @ 2019-06-19 3:09 UTC (permalink / raw)
To: Qian Cai
Cc: Vladimir Davydov, Andrew Morton, Linux MM,
Linux Kernel Mailing List, Johannes Weiner
On Tue, Jun 18, 2019 at 05:43:04PM -0400, Qian Cai wrote:
> Booting linux-next on both arm64 and powerpc triggers SLUB_DEBUG errors below. Reverted the whole series “mm: reparent slab memory on cgroup removal” [1] fixed the issue.
Hi Qian!
Thank you for the report!
Didn't you try to reproduce it on x86? All the code changed in this series
isn't arch-specific, so if it can be seen only on ppc and arm64, that's
interesting.
I'm currently on PTO and have a very limited internet connection,
so I won't be able to reproduce the issue up to Sunday, when I'll be back.
If you can try reverting only the last patch from the series,
I will appreciate it.
Thanks!
>
> [1] https://lore.kernel.org/lkml/20190611231813.3148843-1-guro@fb.com/
>
> [ 151.773224][ T1650] BUG kmem_cache (Tainted: G B W ): Poison overwritten
> [ 151.780969][ T1650] -----------------------------------------------------------------------------
> [ 151.780969][ T1650]
> [ 151.792016][ T1650] INFO: 0x000000001fd6fdef-0x0000000007f6bb36. First byte 0x0 instead of 0x6b
> [ 151.800726][ T1650] INFO: Allocated in create_cache+0x6c/0x1bc age=24301 cpu=97 pid=1444
> [ 151.808821][ T1650] kmem_cache_alloc+0x514/0x568
> [ 151.813527][ T1650] create_cache+0x6c/0x1bc
> [ 151.817800][ T1650] memcg_create_kmem_cache+0xfc/0x11c
> [ 151.823028][ T1650] memcg_kmem_cache_create_func+0x40/0x170
> [ 151.828691][ T1650] process_one_work+0x4e0/0xa54
> [ 151.833398][ T1650] worker_thread+0x498/0x650
> [ 151.837843][ T1650] kthread+0x1b8/0x1d4
> [ 151.841770][ T1650] ret_from_fork+0x10/0x18
> [ 151.846046][ T1650] INFO: Freed in slab_kmem_cache_release+0x3c/0x48 age=23341 cpu=28 pid=1480
> [ 151.854659][ T1650] slab_kmem_cache_release+0x3c/0x48
> [ 151.859799][ T1650] kmem_cache_release+0x1c/0x28
> [ 151.864507][ T1650] kobject_cleanup+0x134/0x288
> [ 151.869127][ T1650] kobject_put+0x5c/0x68
> [ 151.873226][ T1650] sysfs_slab_release+0x2c/0x38
> [ 151.877931][ T1650] shutdown_cache+0x198/0x23c
> [ 151.882464][ T1650] kmemcg_cache_shutdown_fn+0x1c/0x34
> [ 151.887691][ T1650] kmemcg_workfn+0x44/0x68
> [ 151.891963][ T1650] process_one_work+0x4e0/0xa54
> [ 151.896668][ T1650] worker_thread+0x498/0x650
> [ 151.901113][ T1650] kthread+0x1b8/0x1d4
> [ 151.905037][ T1650] ret_from_fork+0x10/0x18
> [ 151.909324][ T1650] INFO: Slab 0x00000000406d65a6 objects=64 used=64 fp=0x000000004d988e71 flags=0x7ffffffc000200
> [ 151.919596][ T1650] INFO: Object 0x0000000040f4b79e @offset=15420325124116637824 fp=0x00000000e038adbf
> [ 151.919596][ T1650]
> [ 151.931079][ T1650] Redzone 00000000fc4c04f0: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
> [ 151.941168][ T1650] Redzone 000000009a25c019: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
> [ 151.951256][ T1650] Redzone 000000000b05c7cc: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
> [ 151.961345][ T1650] Redzone 00000000a08ae38b: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
> [ 151.971433][ T1650] Redzone 00000000e0eccd41: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
> [ 151.981520][ T1650] Redzone 0000000016ee2661: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
> [ 151.991608][ T1650] Redzone 000000009364e729: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
> [ 152.001695][ T1650] Redzone 00000000f2202456: bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb bb ................
> [ 152.011784][ T1650] Object 0000000040f4b79e: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.021783][ T1650] Object 000000002df21fec: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.031779][ T1650] Object 0000000041cf0887: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.041775][ T1650] Object 00000000bfb91e8f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.051770][ T1650] Object 00000000da315b1c: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.061765][ T1650] Object 00000000b362de78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.071761][ T1650] Object 00000000ad4f72bf: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.081756][ T1650] Object 00000000aa32d346: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.091751][ T1650] Object 00000000ad1cf22c: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.101746][ T1650] Object 000000001cee47e4: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.111741][ T1650] Object 00000000418720ed: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.121736][ T1650] Object 00000000dee1c3f2: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.131731][ T1650] Object 00000000a23397c1: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.141727][ T1650] Object 000000002ed01641: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.151721][ T1650] Object 00000000915ec720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.161716][ T1650] Object 00000000915988c1: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.171711][ T1650] Object 000000004a0cc60f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.181707][ T1650] Object 0000000054a294c9: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.191701][ T1650] Object 0000000054f61682: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.201697][ T1650] Object 0000000018d04328: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.211692][ T1650] Object 00000000703cf2c7: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.221687][ T1650] Object 000000004d3ac5d5: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 kkkkkkkk........
> [ 152.231682][ T1650] Object 00000000726ce587: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.241676][ T1650] Object 00000000c709b64e: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.251672][ T1650] Object 0000000044d6a5c6: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.261667][ T1650] Object 000000009c76a6a2: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.271662][ T1650] Object 0000000033d01d12: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.281657][ T1650] Object 00000000c50ff26f: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.291652][ T1650] Object 00000000ebc3aaae: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.301647][ T1650] Object 00000000a2072fe3: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> [ 152.311641][ T1650] Object 000000003d5911a3: 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkk.
> [ 152.320942][ T1650] Redzone 000000009a2feac1: bb bb bb bb bb bb bb bb ........
> [ 152.330330][ T1650] Padding 00000000c1b3cb8b: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 152.340412][ T1650] Padding 000000003715421a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 152.350493][ T1650] Padding 0000000066b51ba7: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 152.360575][ T1650] Padding 00000000ca240306: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 152.370657][ T1650] Padding 0000000014a2af5d: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
> [ 152.380048][ T1650] CPU: 82 PID: 1650 Comm: kworker/82:1 Tainted: G B W 5.2.0-rc5-next-20190617 #18
> [ 152.390216][ T1650] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.0.9 03/01/2019
> [ 152.400741][ T1650] Workqueue: memcg_kmem_cache memcg_kmem_cache_create_func
> [ 152.407786][ T1650] Call trace:
> [ 152.410926][ T1650] dump_backtrace+0x0/0x268
> [ 152.415280][ T1650] show_stack+0x20/0x2c
> [ 152.419287][ T1650] dump_stack+0xb4/0x108
> [ 152.423384][ T1650] print_trailer+0x274/0x298
> [ 152.427825][ T1650] check_bytes_and_report+0xc4/0x118
> [ 152.432959][ T1650] check_object+0x2fc/0x36c
> [ 152.437312][ T1650] alloc_debug_processing+0x154/0x240
> [ 152.442532][ T1650] ___slab_alloc+0x710/0xa68
> [ 152.446972][ T1650] kmem_cache_alloc+0x514/0x568
> [ 152.451672][ T1650] create_cache+0x6c/0x1bc
> [ 152.455938][ T1650] memcg_create_kmem_cache+0xfc/0x11c
> [ 152.461158][ T1650] memcg_kmem_cache_create_func+0x40/0x170
> [ 152.466814][ T1650] process_one_work+0x4e0/0xa54
> [ 152.471515][ T1650] worker_thread+0x498/0x650
> [ 152.475953][ T1650] kthread+0x1b8/0x1d4
> [ 152.479872][ T1650] ret_from_fork+0x10/0x18
> [ 152.484139][ T1650] FIX kmem_cache: Restoring 0x000000001fd6fdef-0x0000000007f6bb36=0x6b
> [ 152.484139][ T1650]
> [ 152.494395][ T1650] FIX kmem_cache: Marking all objects used
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: "mm: reparent slab memory on cgroup removal" series triggers SLUB_DEBUG errors
2019-06-19 3:09 ` Roman Gushchin
@ 2019-06-19 13:36 ` Qian Cai
0 siblings, 0 replies; 3+ messages in thread
From: Qian Cai @ 2019-06-19 13:36 UTC (permalink / raw)
To: Roman Gushchin
Cc: Vladimir Davydov, Andrew Morton, Linux MM,
Linux Kernel Mailing List, Johannes Weiner
On Wed, 2019-06-19 at 03:09 +0000, Roman Gushchin wrote:
> On Tue, Jun 18, 2019 at 05:43:04PM -0400, Qian Cai wrote:
> > Booting linux-next on both arm64 and powerpc triggers SLUB_DEBUG errors
> > below. Reverted the whole series “mm: reparent slab memory on cgroup
> > removal” [1] fixed the issue.
>
> Hi Qian!
>
> Thank you for the report!
>
> Didn't you try to reproduce it on x86? All the code changed in this series
> isn't arch-specific, so if it can be seen only on ppc and arm64, that's
> interesting.
Yes, it is not reproducible on x86 yet.
>
> I'm currently on PTO and have a very limited internet connection,
> so I won't be able to reproduce the issue up to Sunday, when I'll be back.
>
> If you can try reverting only the last patch from the series,
> I will appreciate it.
No, that does not help.
>
> Thanks!
>
> >
> > [1] https://lore.kernel.org/lkml/20190611231813.3148843-1-guro@fb.com/
> >
> > [ 151.773224][ T1650] BUG kmem_cache (Tainted: G B W ): Poison
> > overwritten
> > [ 151.780969][ T1650] ---------------------------------------------------
> > --------------------------
> > [ 151.780969][ T1650]
> > [ 151.792016][ T1650] INFO: 0x000000001fd6fdef-0x0000000007f6bb36. First
> > byte 0x0 instead of 0x6b
> > [ 151.800726][ T1650] INFO: Allocated in create_cache+0x6c/0x1bc age=24301
> > cpu=97 pid=1444
> > [ 151.808821][ T1650] kmem_cache_alloc+0x514/0x568
> > [ 151.813527][ T1650] create_cache+0x6c/0x1bc
> > [ 151.817800][ T1650] memcg_create_kmem_cache+0xfc/0x11c
> > [ 151.823028][ T1650] memcg_kmem_cache_create_func+0x40/0x170
> > [ 151.828691][ T1650] process_one_work+0x4e0/0xa54
> > [ 151.833398][ T1650] worker_thread+0x498/0x650
> > [ 151.837843][ T1650] kthread+0x1b8/0x1d4
> > [ 151.841770][ T1650] ret_from_fork+0x10/0x18
> > [ 151.846046][ T1650] INFO: Freed in slab_kmem_cache_release+0x3c/0x48
> > age=23341 cpu=28 pid=1480
> > [ 151.854659][ T1650] slab_kmem_cache_release+0x3c/0x48
> > [ 151.859799][ T1650] kmem_cache_release+0x1c/0x28
> > [ 151.864507][ T1650] kobject_cleanup+0x134/0x288
> > [ 151.869127][ T1650] kobject_put+0x5c/0x68
> > [ 151.873226][ T1650] sysfs_slab_release+0x2c/0x38
> > [ 151.877931][ T1650] shutdown_cache+0x198/0x23c
> > [ 151.882464][ T1650] kmemcg_cache_shutdown_fn+0x1c/0x34
> > [ 151.887691][ T1650] kmemcg_workfn+0x44/0x68
> > [ 151.891963][ T1650] process_one_work+0x4e0/0xa54
> > [ 151.896668][ T1650] worker_thread+0x498/0x650
> > [ 151.901113][ T1650] kthread+0x1b8/0x1d4
> > [ 151.905037][ T1650] ret_from_fork+0x10/0x18
> > [ 151.909324][ T1650] INFO: Slab 0x00000000406d65a6 objects=64 used=64
> > fp=0x000000004d988e71 flags=0x7ffffffc000200
> > [ 151.919596][ T1650] INFO: Object 0x0000000040f4b79e
> > @offset=15420325124116637824 fp=0x00000000e038adbf
> > [ 151.919596][ T1650]
> > [ 151.931079][ T1650] Redzone 00000000fc4c04f0: bb bb bb bb bb bb bb bb bb
> > bb bb bb bb bb bb bb ................
> > [ 151.941168][ T1650] Redzone 000000009a25c019: bb bb bb bb bb bb bb bb bb
> > bb bb bb bb bb bb bb ................
> > [ 151.951256][ T1650] Redzone 000000000b05c7cc: bb bb bb bb bb bb bb bb bb
> > bb bb bb bb bb bb bb ................
> > [ 151.961345][ T1650] Redzone 00000000a08ae38b: bb bb bb bb bb bb bb bb bb
> > bb bb bb bb bb bb bb ................
> > [ 151.971433][ T1650] Redzone 00000000e0eccd41: bb bb bb bb bb bb bb bb bb
> > bb bb bb bb bb bb bb ................
> > [ 151.981520][ T1650] Redzone 0000000016ee2661: bb bb bb bb bb bb bb bb bb
> > bb bb bb bb bb bb bb ................
> > [ 151.991608][ T1650] Redzone 000000009364e729: bb bb bb bb bb bb bb bb bb
> > bb bb bb bb bb bb bb ................
> > [ 152.001695][ T1650] Redzone 00000000f2202456: bb bb bb bb bb bb bb bb bb
> > bb bb bb bb bb bb bb ................
> > [ 152.011784][ T1650] Object 0000000040f4b79e: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.021783][ T1650] Object 000000002df21fec: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.031779][ T1650] Object 0000000041cf0887: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.041775][ T1650] Object 00000000bfb91e8f: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.051770][ T1650] Object 00000000da315b1c: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.061765][ T1650] Object 00000000b362de78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.071761][ T1650] Object 00000000ad4f72bf: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.081756][ T1650] Object 00000000aa32d346: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.091751][ T1650] Object 00000000ad1cf22c: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.101746][ T1650] Object 000000001cee47e4: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.111741][ T1650] Object 00000000418720ed: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.121736][ T1650] Object 00000000dee1c3f2: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.131731][ T1650] Object 00000000a23397c1: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.141727][ T1650] Object 000000002ed01641: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.151721][ T1650] Object 00000000915ec720: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.161716][ T1650] Object 00000000915988c1: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.171711][ T1650] Object 000000004a0cc60f: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.181707][ T1650] Object 0000000054a294c9: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.191701][ T1650] Object 0000000054f61682: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.201697][ T1650] Object 0000000018d04328: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.211692][ T1650] Object 00000000703cf2c7: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.221687][ T1650] Object 000000004d3ac5d5: 6b 6b 6b 6b 6b 6b 6b 6b 00
> > 00 00 00 00 00 00 00 kkkkkkkk........
> > [ 152.231682][ T1650] Object 00000000726ce587: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.241676][ T1650] Object 00000000c709b64e: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.251672][ T1650] Object 0000000044d6a5c6: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.261667][ T1650] Object 000000009c76a6a2: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.271662][ T1650] Object 0000000033d01d12: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.281657][ T1650] Object 00000000c50ff26f: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.291652][ T1650] Object 00000000ebc3aaae: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.301647][ T1650] Object 00000000a2072fe3: 6b 6b 6b 6b 6b 6b 6b 6b 6b
> > 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> > [ 152.311641][ T1650] Object 000000003d5911a3: 6b 6b 6b 6b 6b 6b 6b
> > a5 kkkkkkk.
> > [ 152.320942][ T1650] Redzone 000000009a2feac1: bb bb bb bb bb bb bb
> > bb ........
> > [ 152.330330][ T1650] Padding 00000000c1b3cb8b: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> > 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> > [ 152.340412][ T1650] Padding 000000003715421a: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> > 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> > [ 152.350493][ T1650] Padding 0000000066b51ba7: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> > 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> > [ 152.360575][ T1650] Padding 00000000ca240306: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> > 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> > [ 152.370657][ T1650] Padding 0000000014a2af5d: 5a 5a 5a 5a 5a 5a 5a
> > 5a ZZZZZZZZ
> > [ 152.380048][ T1650] CPU: 82 PID: 1650 Comm: kworker/82:1 Tainted:
> > G B W 5.2.0-rc5-next-20190617 #18
> > [ 152.390216][ T1650] Hardware name: HPE Apollo
> > 70 /C01_APACHE_MB , BIOS L50_5.13_1.0.9 03/01/2019
> > [ 152.400741][ T1650] Workqueue: memcg_kmem_cache
> > memcg_kmem_cache_create_func
> > [ 152.407786][ T1650] Call trace:
> > [ 152.410926][ T1650] dump_backtrace+0x0/0x268
> > [ 152.415280][ T1650] show_stack+0x20/0x2c
> > [ 152.419287][ T1650] dump_stack+0xb4/0x108
> > [ 152.423384][ T1650] print_trailer+0x274/0x298
> > [ 152.427825][ T1650] check_bytes_and_report+0xc4/0x118
> > [ 152.432959][ T1650] check_object+0x2fc/0x36c
> > [ 152.437312][ T1650] alloc_debug_processing+0x154/0x240
> > [ 152.442532][ T1650] ___slab_alloc+0x710/0xa68
> > [ 152.446972][ T1650] kmem_cache_alloc+0x514/0x568
> > [ 152.451672][ T1650] create_cache+0x6c/0x1bc
> > [ 152.455938][ T1650] memcg_create_kmem_cache+0xfc/0x11c
> > [ 152.461158][ T1650] memcg_kmem_cache_create_func+0x40/0x170
> > [ 152.466814][ T1650] process_one_work+0x4e0/0xa54
> > [ 152.471515][ T1650] worker_thread+0x498/0x650
> > [ 152.475953][ T1650] kthread+0x1b8/0x1d4
> > [ 152.479872][ T1650] ret_from_fork+0x10/0x18
> > [ 152.484139][ T1650] FIX kmem_cache: Restoring 0x000000001fd6fdef-
> > 0x0000000007f6bb36=0x6b
> > [ 152.484139][ T1650]
> > [ 152.494395][ T1650] FIX kmem_cache: Marking all objects used
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-06-19 13:36 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-18 21:43 "mm: reparent slab memory on cgroup removal" series triggers SLUB_DEBUG errors Qian Cai
2019-06-19 3:09 ` Roman Gushchin
2019-06-19 13:36 ` Qian Cai
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).