* Deprecating and removing SLOB
@ 2022-11-08 15:55 Vlastimil Babka
2022-11-08 18:18 ` Christophe Leroy
` (2 more replies)
0 siblings, 3 replies; 112+ messages in thread
From: Vlastimil Babka @ 2022-11-08 15:55 UTC (permalink / raw)
To: Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg
Cc: Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds,
linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev,
Andrew Morton
Hi,
as we all know, we currently have three slab allocators. As we discussed
at LPC [1], it is my hope that one of these allocators has a future, and
two of them do not.
The unsurprising reasons include code maintenance burden, other features
compatible with only a subset of allocators (or more effort spent on the
features), blocking API improvements (more on that below), and my
inability to pronounce SLAB and SLUB in a properly distinguishable way,
without resorting to spelling out the letters.
I think (but may be proven wrong) that SLOB is the easier target of the
two to be removed, so I'd like to focus on it first.
I believe SLOB can be removed because:
- AFAIK nobody really uses it? It strives for minimal memory footprint
by putting all objects together, which has its CPU performance costs
(locking, lack of percpu caching, searching for free space...). I'm not
aware of any "tiny linux" deployment that opts for this. For example,
OpenWRT seems to use SLUB and the devices these days have e.g. 128MB
RAM, not up to 16 MB anymore. I've heard anecdotes that the performance
SLOB impact is too much for those who tried. Googling for
"CONFIG_SLOB=y" yielded nothing useful.
- Last time we discussed it [2], it seemed SLUB memory requirements can
be brought very close to SLOB's if needed. Of course it can never have
as small footprint as SLOB due to separate kmem_caches, but the
difference is not that significant, unless somebody still tries to use
Linux on very tiny systems (goes back to the previous point).
Besides the smaller maintenance burden, removing SLOB would allow us to
do a useful API improvement - the ability to use kfree() for both
objects allocated by kmalloc() and kmem_cache_alloc(). Currently the
latter has to be freed by kmem_cache_free(), passing a kmem_cache
pointer in addition to the object pointer. With SLUB and SLAB, it is
however possible to use kfree() instead, as the kmalloc caches and the
rest of kmem_caches are the same and kfree() can lookup the kmem_cache
from object pointer easily for any of those. XFS has apparently did that
for years without anyone noticing it's broken on SLOB [3], and
legitimizing and expanding this would help some use cases beside XFS
(IIRC Matthew mentioned rcu-based freeing for example).
However for SLOB to support kfree() on all allocations, it would need to
store object size of allocated objects (which it currently does only for
kmalloc() objects, prepending a size header to the object), but for
kmem_cache_alloc() allocations as well. This has been attempted in the
thread [3] but it bloats the memory usage, especially on architectures
with large ARCH_KMALLOC_MINALIGN, where the prepended header basically
has to occupy the whole ARCH_KMALLOC_MINALIGN block to be DMA safe.
There are ongoing efforts to reduce this minalign, but the memory
footprint would still increase, going against the purpose of SLOB, so
again it would be easier if we could just remove it.
So with this thread I'm interested in hearing arguments/use cases for
keeping SLOB. There might be obviously users of SLOB whom this
conversation will not reach, so I assume the eventual next step would be
to deprecate it in a way that those users are notified when building a
new kernel and can raise their voice then. Is there a good proven way
how to do that for a config option like this one?
Thanks,
Vlastimil
[1] https://lpc.events/event/16/contributions/1272/ - slides in the
slabs.pdf linked there
[2]
https://lore.kernel.org/all/20211017135708.GA8442@kvm.asia-northeast3-a.c.our-ratio-313919.internal/#t
[3]
https://lore.kernel.org/all/20210930044202.GP2361455@dread.disaster.area/
^ permalink raw reply [flat|nested] 112+ messages in thread* Re: Deprecating and removing SLOB 2022-11-08 15:55 Deprecating and removing SLOB Vlastimil Babka @ 2022-11-08 18:18 ` Christophe Leroy 2022-11-08 19:17 ` Andrew Morton 2022-11-08 18:46 ` Roman Gushchin 2022-11-08 21:44 ` Pasha Tatashin 2 siblings, 1 reply; 112+ messages in thread From: Christophe Leroy @ 2022-11-08 18:18 UTC (permalink / raw) To: Vlastimil Babka, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg Cc: Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton Le 08/11/2022 à 16:55, Vlastimil Babka a écrit : > Hi, > > as we all know, we currently have three slab allocators. As we discussed > at LPC [1], it is my hope that one of these allocators has a future, and > two of them do not. Well, the only one that supports PREEMPT_RT is SLUB as far as I can see in mm/Kconfig, so I guess both SLOB and SLAB will go away ? > > The unsurprising reasons include code maintenance burden, other features > compatible with only a subset of allocators (or more effort spent on the > features), blocking API improvements (more on that below), and my > inability to pronounce SLAB and SLUB in a properly distinguishable way, > without resorting to spelling out the letters. > > I think (but may be proven wrong) that SLOB is the easier target of the > two to be removed, so I'd like to focus on it first. > > I believe SLOB can be removed because: > > - AFAIK nobody really uses it? It strives for minimal memory footprint > by putting all objects together, which has its CPU performance costs > (locking, lack of percpu caching, searching for free space...). I'm not > aware of any "tiny linux" deployment that opts for this. For example, > OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > SLOB impact is too much for those who tried. Googling for > "CONFIG_SLOB=y" yielded nothing useful. I still have devices (powerpc) with only 32MB today and for the next ten years at least. But they have been using SLUB. > > - Last time we discussed it [2], it seemed SLUB memory requirements can > be brought very close to SLOB's if needed. Of course it can never have > as small footprint as SLOB due to separate kmem_caches, but the > difference is not that significant, unless somebody still tries to use > Linux on very tiny systems (goes back to the previous point). > > Besides the smaller maintenance burden, removing SLOB would allow us to > do a useful API improvement - the ability to use kfree() for both > objects allocated by kmalloc() and kmem_cache_alloc(). Currently the > latter has to be freed by kmem_cache_free(), passing a kmem_cache > pointer in addition to the object pointer. With SLUB and SLAB, it is > however possible to use kfree() instead, as the kmalloc caches and the > rest of kmem_caches are the same and kfree() can lookup the kmem_cache > from object pointer easily for any of those. XFS has apparently did that > for years without anyone noticing it's broken on SLOB [3], and > legitimizing and expanding this would help some use cases beside XFS > (IIRC Matthew mentioned rcu-based freeing for example). > > However for SLOB to support kfree() on all allocations, it would need to > store object size of allocated objects (which it currently does only for > kmalloc() objects, prepending a size header to the object), but for > kmem_cache_alloc() allocations as well. This has been attempted in the > thread [3] but it bloats the memory usage, especially on architectures > with large ARCH_KMALLOC_MINALIGN, where the prepended header basically > has to occupy the whole ARCH_KMALLOC_MINALIGN block to be DMA safe. > There are ongoing efforts to reduce this minalign, but the memory > footprint would still increase, going against the purpose of SLOB, so > again it would be easier if we could just remove it. > > So with this thread I'm interested in hearing arguments/use cases for > keeping SLOB. There might be obviously users of SLOB whom this > conversation will not reach, so I assume the eventual next step would be > to deprecate it in a way that those users are notified when building a > new kernel and can raise their voice then. Is there a good proven way > how to do that for a config option like this one? Mark them as dependent on CONFIG_BROKEN ? Christophe > > Thanks, > Vlastimil > > [1] https://lpc.events/event/16/contributions/1272/ - slides in the > slabs.pdf linked there > [2] > https://lore.kernel.org/all/20211017135708.GA8442@kvm.asia-northeast3-a.c.our-ratio-313919.internal/#t > [3] > https://lore.kernel.org/all/20210930044202.GP2361455@dread.disaster.area/ > > ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-08 18:18 ` Christophe Leroy @ 2022-11-08 19:17 ` Andrew Morton 0 siblings, 0 replies; 112+ messages in thread From: Andrew Morton @ 2022-11-08 19:17 UTC (permalink / raw) To: Christophe Leroy Cc: Vlastimil Babka, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev On Tue, 8 Nov 2022 18:18:46 +0000 Christophe Leroy <christophe.leroy@csgroup.eu> wrote: > Mark them as dependent on CONFIG_BROKEN ? For SLOB at least, yes. Leave it a couple of cycles and unless someone comes out with a really really really good reason for retaining, away it goes. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-08 15:55 Deprecating and removing SLOB Vlastimil Babka 2022-11-08 18:18 ` Christophe Leroy @ 2022-11-08 18:46 ` Roman Gushchin 2022-11-08 20:13 ` Yosry Ahmed 2022-11-08 21:44 ` Pasha Tatashin 2 siblings, 1 reply; 112+ messages in thread From: Roman Gushchin @ 2022-11-08 18:46 UTC (permalink / raw) To: Vlastimil Babka Cc: Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton On Tue, Nov 08, 2022 at 04:55:29PM +0100, Vlastimil Babka wrote: > Hi, > > as we all know, we currently have three slab allocators. As we discussed at > LPC [1], it is my hope that one of these allocators has a future, and two of > them do not. > > The unsurprising reasons include code maintenance burden, other features > compatible with only a subset of allocators (or more effort spent on the > features), blocking API improvements (more on that below), and my inability > to pronounce SLAB and SLUB in a properly distinguishable way, without > resorting to spelling out the letters. > > I think (but may be proven wrong) that SLOB is the easier target of the two > to be removed, so I'd like to focus on it first. Great! SLOB is not supported by the kernel memory accounting code, so if we'll deprecate SLOB, we can remove all those annoying ifndefs. But I wonder if we can deprecate SLAB too? Or at least use the moment to ask every non-SLUB user on why they can't/don't want to use SLUB. Are there any known advantages of SLAB over SLUB? Also, for memory-constrained users we might want to add some guide on how to configure SLUB to minimize the memory footprint. Thank you! Roman ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-08 18:46 ` Roman Gushchin @ 2022-11-08 20:13 ` Yosry Ahmed 2022-11-09 9:09 ` Vlastimil Babka 0 siblings, 1 reply; 112+ messages in thread From: Yosry Ahmed @ 2022-11-08 20:13 UTC (permalink / raw) To: Roman Gushchin, David Rientjes, Greg Thelen Cc: Vlastimil Babka, Christoph Lameter, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton On Tue, Nov 8, 2022 at 10:47 AM Roman Gushchin <roman.gushchin@linux.dev> wrote: > > On Tue, Nov 08, 2022 at 04:55:29PM +0100, Vlastimil Babka wrote: > > Hi, > > > > as we all know, we currently have three slab allocators. As we discussed at > > LPC [1], it is my hope that one of these allocators has a future, and two of > > them do not. > > > > The unsurprising reasons include code maintenance burden, other features > > compatible with only a subset of allocators (or more effort spent on the > > features), blocking API improvements (more on that below), and my inability > > to pronounce SLAB and SLUB in a properly distinguishable way, without > > resorting to spelling out the letters. > > > > I think (but may be proven wrong) that SLOB is the easier target of the two > > to be removed, so I'd like to focus on it first. > > Great! > > SLOB is not supported by the kernel memory accounting code, so if we'll > deprecate SLOB, we can remove all those annoying ifndefs. > > But I wonder if we can deprecate SLAB too? Or at least use the moment to > ask every non-SLUB user on why they can't/don't want to use SLUB. > Are there any known advantages of SLAB over SLUB? We use SLAB at Google, but I am not the right person to answer the question of why we can't/don't use SLUB. Adding Greg here who recently looked into this and might have answers. I see David is already tagged, he might have a good answer as well. > > Also, for memory-constrained users we might want to add some guide on how > to configure SLUB to minimize the memory footprint. > > Thank you! > > Roman > ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-08 20:13 ` Yosry Ahmed @ 2022-11-09 9:09 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-09 9:09 UTC (permalink / raw) To: Yosry Ahmed, Roman Gushchin, David Rientjes, Greg Thelen Cc: Christoph Lameter, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton On 11/8/22 21:13, Yosry Ahmed wrote: > On Tue, Nov 8, 2022 at 10:47 AM Roman Gushchin <roman.gushchin@linux.dev> wrote: >> >> On Tue, Nov 08, 2022 at 04:55:29PM +0100, Vlastimil Babka wrote: >> > Hi, >> > >> > as we all know, we currently have three slab allocators. As we discussed at >> > LPC [1], it is my hope that one of these allocators has a future, and two of >> > them do not. >> > >> > The unsurprising reasons include code maintenance burden, other features >> > compatible with only a subset of allocators (or more effort spent on the >> > features), blocking API improvements (more on that below), and my inability >> > to pronounce SLAB and SLUB in a properly distinguishable way, without >> > resorting to spelling out the letters. >> > >> > I think (but may be proven wrong) that SLOB is the easier target of the two >> > to be removed, so I'd like to focus on it first. >> >> Great! >> >> SLOB is not supported by the kernel memory accounting code, so if we'll >> deprecate SLOB, we can remove all those annoying ifndefs. >> >> But I wonder if we can deprecate SLAB too? Or at least use the moment to >> ask every non-SLUB user on why they can't/don't want to use SLUB. >> Are there any known advantages of SLAB over SLUB? Yeah it was my plan to inquire about SLAB next, once SLOB's fate is settled, as I did expect greater resistance there. My hope is that if there are still workloads that benefit from SLAB's percpu arrays, we could e.g. add (per-cache opt-in) percpu arrays to SLUB, as that would be still better than having two different complete implementations. > We use SLAB at Google, but I am not the right person to answer the > question of why we can't/don't use SLUB. Adding Greg here who recently > looked into this and might have answers. I see David is already > tagged, he might have a good answer as well. Yeah, Google folks were indeed against removing SLAB due to such workloads in the past discussions. >> >> Also, for memory-constrained users we might want to add some guide on how >> to configure SLUB to minimize the memory footprint. >> >> Thank you! >> >> Roman >> ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-08 15:55 Deprecating and removing SLOB Vlastimil Babka 2022-11-08 18:18 ` Christophe Leroy 2022-11-08 18:46 ` Roman Gushchin @ 2022-11-08 21:44 ` Pasha Tatashin 2022-11-09 9:00 ` Vlastimil Babka ` (2 more replies) 2 siblings, 3 replies; 112+ messages in thread From: Pasha Tatashin @ 2022-11-08 21:44 UTC (permalink / raw) To: Vlastimil Babka Cc: Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > Hi, > > as we all know, we currently have three slab allocators. As we discussed > at LPC [1], it is my hope that one of these allocators has a future, and > two of them do not. > > The unsurprising reasons include code maintenance burden, other features > compatible with only a subset of allocators (or more effort spent on the > features), blocking API improvements (more on that below), and my > inability to pronounce SLAB and SLUB in a properly distinguishable way, > without resorting to spelling out the letters. > > I think (but may be proven wrong) that SLOB is the easier target of the > two to be removed, so I'd like to focus on it first. > > I believe SLOB can be removed because: > > - AFAIK nobody really uses it? It strives for minimal memory footprint > by putting all objects together, which has its CPU performance costs > (locking, lack of percpu caching, searching for free space...). I'm not > aware of any "tiny linux" deployment that opts for this. For example, > OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > SLOB impact is too much for those who tried. Googling for > "CONFIG_SLOB=y" yielded nothing useful. I am all for removing SLOB. There are some devices with configs where SLOB is enabled by default. Perhaps, the owners/maintainers of those devices/configs should be included into this thread: tatashin@soleen:~/x/linux$ git grep SLOB=y arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y arch/arm/configs/collie_defconfig:CONFIG_SLOB=y arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y kernel/configs/tiny.config:CONFIG_SLOB=y > > - Last time we discussed it [2], it seemed SLUB memory requirements can > be brought very close to SLOB's if needed. Of course it can never have > as small footprint as SLOB due to separate kmem_caches, but the > difference is not that significant, unless somebody still tries to use > Linux on very tiny systems (goes back to the previous point). > > Besides the smaller maintenance burden, removing SLOB would allow us to > do a useful API improvement - the ability to use kfree() for both > objects allocated by kmalloc() and kmem_cache_alloc(). Currently the > latter has to be freed by kmem_cache_free(), passing a kmem_cache > pointer in addition to the object pointer. With SLUB and SLAB, it is > however possible to use kfree() instead, as the kmalloc caches and the > rest of kmem_caches are the same and kfree() can lookup the kmem_cache > from object pointer easily for any of those. XFS has apparently did that > for years without anyone noticing it's broken on SLOB [3], and > legitimizing and expanding this would help some use cases beside XFS > (IIRC Matthew mentioned rcu-based freeing for example). > > However for SLOB to support kfree() on all allocations, it would need to > store object size of allocated objects (which it currently does only for > kmalloc() objects, prepending a size header to the object), but for > kmem_cache_alloc() allocations as well. This has been attempted in the > thread [3] but it bloats the memory usage, especially on architectures > with large ARCH_KMALLOC_MINALIGN, where the prepended header basically > has to occupy the whole ARCH_KMALLOC_MINALIGN block to be DMA safe. > There are ongoing efforts to reduce this minalign, but the memory > footprint would still increase, going against the purpose of SLOB, so > again it would be easier if we could just remove it. > > So with this thread I'm interested in hearing arguments/use cases for > keeping SLOB. There might be obviously users of SLOB whom this > conversation will not reach, so I assume the eventual next step would be > to deprecate it in a way that those users are notified when building a > new kernel and can raise their voice then. Is there a good proven way > how to do that for a config option like this one? > > Thanks, > Vlastimil > > [1] https://lpc.events/event/16/contributions/1272/ - slides in the > slabs.pdf linked there > [2] > https://lore.kernel.org/all/20211017135708.GA8442@kvm.asia-northeast3-a.c.our-ratio-313919.internal/#t > [3] > https://lore.kernel.org/all/20210930044202.GP2361455@dread.disaster.area/ > > > ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-08 21:44 ` Pasha Tatashin 2022-11-09 9:00 ` Vlastimil Babka 2022-11-09 17:57 ` Conor.Dooley @ 2022-11-09 9:00 ` Vlastimil Babka 2 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-09 9:00 UTC (permalink / raw) To: Pasha Tatashin Cc: Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, linux-olinux-omap@vger.kernel.orgmap, openrisc, linux-riscv, linux-sh On 11/8/22 22:44, Pasha Tatashin wrote: > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >> >> Hi, >> >> as we all know, we currently have three slab allocators. As we discussed >> at LPC [1], it is my hope that one of these allocators has a future, and >> two of them do not. >> >> The unsurprising reasons include code maintenance burden, other features >> compatible with only a subset of allocators (or more effort spent on the >> features), blocking API improvements (more on that below), and my >> inability to pronounce SLAB and SLUB in a properly distinguishable way, >> without resorting to spelling out the letters. >> >> I think (but may be proven wrong) that SLOB is the easier target of the >> two to be removed, so I'd like to focus on it first. >> >> I believe SLOB can be removed because: >> >> - AFAIK nobody really uses it? It strives for minimal memory footprint >> by putting all objects together, which has its CPU performance costs >> (locking, lack of percpu caching, searching for free space...). I'm not >> aware of any "tiny linux" deployment that opts for this. For example, >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >> SLOB impact is too much for those who tried. Googling for >> "CONFIG_SLOB=y" yielded nothing useful. > > I am all for removing SLOB. > > There are some devices with configs where SLOB is enabled by default. > Perhaps, the owners/maintainers of those devices/configs should be > included into this thread: > > tatashin@soleen:~/x/linux$ git grep SLOB=y > arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y > arch/arm/configs/collie_defconfig:CONFIG_SLOB=y > arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y > arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y > arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y > arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y > arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y > arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y > kernel/configs/tiny.config:CONFIG_SLOB=y Great point, thanks. Ccing. First mail here: https://lore.kernel.org/all/CA%2BCK2bD-uVGJ0%3D9uc7Lt5zwY%2B2PM2RTcfOhxEd65S7TvTrJULA@mail.gmail.com/ >> >> - Last time we discussed it [2], it seemed SLUB memory requirements can >> be brought very close to SLOB's if needed. Of course it can never have >> as small footprint as SLOB due to separate kmem_caches, but the >> difference is not that significant, unless somebody still tries to use >> Linux on very tiny systems (goes back to the previous point). >> >> Besides the smaller maintenance burden, removing SLOB would allow us to >> do a useful API improvement - the ability to use kfree() for both >> objects allocated by kmalloc() and kmem_cache_alloc(). Currently the >> latter has to be freed by kmem_cache_free(), passing a kmem_cache >> pointer in addition to the object pointer. With SLUB and SLAB, it is >> however possible to use kfree() instead, as the kmalloc caches and the >> rest of kmem_caches are the same and kfree() can lookup the kmem_cache >> from object pointer easily for any of those. XFS has apparently did that >> for years without anyone noticing it's broken on SLOB [3], and >> legitimizing and expanding this would help some use cases beside XFS >> (IIRC Matthew mentioned rcu-based freeing for example). >> >> However for SLOB to support kfree() on all allocations, it would need to >> store object size of allocated objects (which it currently does only for >> kmalloc() objects, prepending a size header to the object), but for >> kmem_cache_alloc() allocations as well. This has been attempted in the >> thread [3] but it bloats the memory usage, especially on architectures >> with large ARCH_KMALLOC_MINALIGN, where the prepended header basically >> has to occupy the whole ARCH_KMALLOC_MINALIGN block to be DMA safe. >> There are ongoing efforts to reduce this minalign, but the memory >> footprint would still increase, going against the purpose of SLOB, so >> again it would be easier if we could just remove it. >> >> So with this thread I'm interested in hearing arguments/use cases for >> keeping SLOB. There might be obviously users of SLOB whom this >> conversation will not reach, so I assume the eventual next step would be >> to deprecate it in a way that those users are notified when building a >> new kernel and can raise their voice then. Is there a good proven way >> how to do that for a config option like this one? >> >> Thanks, >> Vlastimil >> >> [1] https://lpc.events/event/16/contributions/1272/ - slides in the >> slabs.pdf linked there >> [2] >> https://lore.kernel.org/all/20211017135708.GA8442@kvm.asia-northeast3-a.c.our-ratio-313919.internal/#t >> [3] >> https://lore.kernel.org/all/20210930044202.GP2361455@dread.disaster.area/ >> >> >> _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 9:00 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-09 9:00 UTC (permalink / raw) To: Pasha Tatashin Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, linux-mm, Hyeonggon Yoo, Christoph Lameter, linux-olinux-omap@vger.kernel.orgmap, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, linux-riscv, Andrew Morton, Linus Torvalds On 11/8/22 22:44, Pasha Tatashin wrote: > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >> >> Hi, >> >> as we all know, we currently have three slab allocators. As we discussed >> at LPC [1], it is my hope that one of these allocators has a future, and >> two of them do not. >> >> The unsurprising reasons include code maintenance burden, other features >> compatible with only a subset of allocators (or more effort spent on the >> features), blocking API improvements (more on that below), and my >> inability to pronounce SLAB and SLUB in a properly distinguishable way, >> without resorting to spelling out the letters. >> >> I think (but may be proven wrong) that SLOB is the easier target of the >> two to be removed, so I'd like to focus on it first. >> >> I believe SLOB can be removed because: >> >> - AFAIK nobody really uses it? It strives for minimal memory footprint >> by putting all objects together, which has its CPU performance costs >> (locking, lack of percpu caching, searching for free space...). I'm not >> aware of any "tiny linux" deployment that opts for this. For example, >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >> SLOB impact is too much for those who tried. Googling for >> "CONFIG_SLOB=y" yielded nothing useful. > > I am all for removing SLOB. > > There are some devices with configs where SLOB is enabled by default. > Perhaps, the owners/maintainers of those devices/configs should be > included into this thread: > > tatashin@soleen:~/x/linux$ git grep SLOB=y > arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y > arch/arm/configs/collie_defconfig:CONFIG_SLOB=y > arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y > arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y > arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y > arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y > arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y > arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y > kernel/configs/tiny.config:CONFIG_SLOB=y Great point, thanks. Ccing. First mail here: https://lore.kernel.org/all/CA%2BCK2bD-uVGJ0%3D9uc7Lt5zwY%2B2PM2RTcfOhxEd65S7TvTrJULA@mail.gmail.com/ >> >> - Last time we discussed it [2], it seemed SLUB memory requirements can >> be brought very close to SLOB's if needed. Of course it can never have >> as small footprint as SLOB due to separate kmem_caches, but the >> difference is not that significant, unless somebody still tries to use >> Linux on very tiny systems (goes back to the previous point). >> >> Besides the smaller maintenance burden, removing SLOB would allow us to >> do a useful API improvement - the ability to use kfree() for both >> objects allocated by kmalloc() and kmem_cache_alloc(). Currently the >> latter has to be freed by kmem_cache_free(), passing a kmem_cache >> pointer in addition to the object pointer. With SLUB and SLAB, it is >> however possible to use kfree() instead, as the kmalloc caches and the >> rest of kmem_caches are the same and kfree() can lookup the kmem_cache >> from object pointer easily for any of those. XFS has apparently did that >> for years without anyone noticing it's broken on SLOB [3], and >> legitimizing and expanding this would help some use cases beside XFS >> (IIRC Matthew mentioned rcu-based freeing for example). >> >> However for SLOB to support kfree() on all allocations, it would need to >> store object size of allocated objects (which it currently does only for >> kmalloc() objects, prepending a size header to the object), but for >> kmem_cache_alloc() allocations as well. This has been attempted in the >> thread [3] but it bloats the memory usage, especially on architectures >> with large ARCH_KMALLOC_MINALIGN, where the prepended header basically >> has to occupy the whole ARCH_KMALLOC_MINALIGN block to be DMA safe. >> There are ongoing efforts to reduce this minalign, but the memory >> footprint would still increase, going against the purpose of SLOB, so >> again it would be easier if we could just remove it. >> >> So with this thread I'm interested in hearing arguments/use cases for >> keeping SLOB. There might be obviously users of SLOB whom this >> conversation will not reach, so I assume the eventual next step would be >> to deprecate it in a way that those users are notified when building a >> new kernel and can raise their voice then. Is there a good proven way >> how to do that for a config option like this one? >> >> Thanks, >> Vlastimil >> >> [1] https://lpc.events/event/16/contributions/1272/ - slides in the >> slabs.pdf linked there >> [2] >> https://lore.kernel.org/all/20211017135708.GA8442@kvm.asia-northeast3-a.c.our-ratio-313919.internal/#t >> [3] >> https://lore.kernel.org/all/20210930044202.GP2361455@dread.disaster.area/ >> >> >> ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 9:00 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-09 9:00 UTC (permalink / raw) To: Pasha Tatashin Cc: Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, linux-olinux-omap@vger.kernel.orgmap, openrisc, linux-riscv, linux-sh On 11/8/22 22:44, Pasha Tatashin wrote: > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >> >> Hi, >> >> as we all know, we currently have three slab allocators. As we discussed >> at LPC [1], it is my hope that one of these allocators has a future, and >> two of them do not. >> >> The unsurprising reasons include code maintenance burden, other features >> compatible with only a subset of allocators (or more effort spent on the >> features), blocking API improvements (more on that below), and my >> inability to pronounce SLAB and SLUB in a properly distinguishable way, >> without resorting to spelling out the letters. >> >> I think (but may be proven wrong) that SLOB is the easier target of the >> two to be removed, so I'd like to focus on it first. >> >> I believe SLOB can be removed because: >> >> - AFAIK nobody really uses it? It strives for minimal memory footprint >> by putting all objects together, which has its CPU performance costs >> (locking, lack of percpu caching, searching for free space...). I'm not >> aware of any "tiny linux" deployment that opts for this. For example, >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >> SLOB impact is too much for those who tried. Googling for >> "CONFIG_SLOB=y" yielded nothing useful. > > I am all for removing SLOB. > > There are some devices with configs where SLOB is enabled by default. > Perhaps, the owners/maintainers of those devices/configs should be > included into this thread: > > tatashin@soleen:~/x/linux$ git grep SLOB=y > arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y > arch/arm/configs/collie_defconfig:CONFIG_SLOB=y > arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y > arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y > arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y > arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y > arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y > arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y > kernel/configs/tiny.config:CONFIG_SLOB=y Great point, thanks. Ccing. First mail here: https://lore.kernel.org/all/CA%2BCK2bD-uVGJ0%3D9uc7Lt5zwY%2B2PM2RTcfOhxEd65S7TvTrJULA@mail.gmail.com/ >> >> - Last time we discussed it [2], it seemed SLUB memory requirements can >> be brought very close to SLOB's if needed. Of course it can never have >> as small footprint as SLOB due to separate kmem_caches, but the >> difference is not that significant, unless somebody still tries to use >> Linux on very tiny systems (goes back to the previous point). >> >> Besides the smaller maintenance burden, removing SLOB would allow us to >> do a useful API improvement - the ability to use kfree() for both >> objects allocated by kmalloc() and kmem_cache_alloc(). Currently the >> latter has to be freed by kmem_cache_free(), passing a kmem_cache >> pointer in addition to the object pointer. With SLUB and SLAB, it is >> however possible to use kfree() instead, as the kmalloc caches and the >> rest of kmem_caches are the same and kfree() can lookup the kmem_cache >> from object pointer easily for any of those. XFS has apparently did that >> for years without anyone noticing it's broken on SLOB [3], and >> legitimizing and expanding this would help some use cases beside XFS >> (IIRC Matthew mentioned rcu-based freeing for example). >> >> However for SLOB to support kfree() on all allocations, it would need to >> store object size of allocated objects (which it currently does only for >> kmalloc() objects, prepending a size header to the object), but for >> kmem_cache_alloc() allocations as well. This has been attempted in the >> thread [3] but it bloats the memory usage, especially on architectures >> with large ARCH_KMALLOC_MINALIGN, where the prepended header basically >> has to occupy the whole ARCH_KMALLOC_MINALIGN block to be DMA safe. >> There are ongoing efforts to reduce this minalign, but the memory >> footprint would still increase, going against the purpose of SLOB, so >> again it would be easier if we could just remove it. >> >> So with this thread I'm interested in hearing arguments/use cases for >> keeping SLOB. There might be obviously users of SLOB whom this >> conversation will not reach, so I assume the eventual next step would be >> to deprecate it in a way that those users are notified when building a >> new kernel and can raise their voice then. Is there a good proven way >> how to do that for a config option like this one? >> >> Thanks, >> Vlastimil >> >> [1] https://lpc.events/event/16/contributions/1272/ - slides in the >> slabs.pdf linked there >> [2] >> https://lore.kernel.org/all/20211017135708.GA8442@kvm.asia-northeast3-a.c.our-ratio-313919.internal/#t >> [3] >> https://lore.kernel.org/all/20210930044202.GP2361455@dread.disaster.area/ >> >> >> _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 9:00 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-09 9:00 UTC (permalink / raw) To: Pasha Tatashin Cc: Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, linux-olinux-omap@vger.kernel.orgmap, openrisc, linux-riscv, linux-sh On 11/8/22 22:44, Pasha Tatashin wrote: > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >> >> Hi, >> >> as we all know, we currently have three slab allocators. As we discussed >> at LPC [1], it is my hope that one of these allocators has a future, and >> two of them do not. >> >> The unsurprising reasons include code maintenance burden, other features >> compatible with only a subset of allocators (or more effort spent on the >> features), blocking API improvements (more on that below), and my >> inability to pronounce SLAB and SLUB in a properly distinguishable way, >> without resorting to spelling out the letters. >> >> I think (but may be proven wrong) that SLOB is the easier target of the >> two to be removed, so I'd like to focus on it first. >> >> I believe SLOB can be removed because: >> >> - AFAIK nobody really uses it? It strives for minimal memory footprint >> by putting all objects together, which has its CPU performance costs >> (locking, lack of percpu caching, searching for free space...). I'm not >> aware of any "tiny linux" deployment that opts for this. For example, >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >> SLOB impact is too much for those who tried. Googling for >> "CONFIG_SLOB=y" yielded nothing useful. > > I am all for removing SLOB. > > There are some devices with configs where SLOB is enabled by default. > Perhaps, the owners/maintainers of those devices/configs should be > included into this thread: > > tatashin@soleen:~/x/linux$ git grep SLOB=y > arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y > arch/arm/configs/collie_defconfig:CONFIG_SLOB=y > arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y > arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y > arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y > arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y > arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y > arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y > kernel/configs/tiny.config:CONFIG_SLOB=y Great point, thanks. Ccing. First mail here: https://lore.kernel.org/all/CA%2BCK2bD-uVGJ0%3D9uc7Lt5zwY%2B2PM2RTcfOhxEd65S7TvTrJULA@mail.gmail.com/ >> >> - Last time we discussed it [2], it seemed SLUB memory requirements can >> be brought very close to SLOB's if needed. Of course it can never have >> as small footprint as SLOB due to separate kmem_caches, but the >> difference is not that significant, unless somebody still tries to use >> Linux on very tiny systems (goes back to the previous point). >> >> Besides the smaller maintenance burden, removing SLOB would allow us to >> do a useful API improvement - the ability to use kfree() for both >> objects allocated by kmalloc() and kmem_cache_alloc(). Currently the >> latter has to be freed by kmem_cache_free(), passing a kmem_cache >> pointer in addition to the object pointer. With SLUB and SLAB, it is >> however possible to use kfree() instead, as the kmalloc caches and the >> rest of kmem_caches are the same and kfree() can lookup the kmem_cache >> from object pointer easily for any of those. XFS has apparently did that >> for years without anyone noticing it's broken on SLOB [3], and >> legitimizing and expanding this would help some use cases beside XFS >> (IIRC Matthew mentioned rcu-based freeing for example). >> >> However for SLOB to support kfree() on all allocations, it would need to >> store object size of allocated objects (which it currently does only for >> kmalloc() objects, prepending a size header to the object), but for >> kmem_cache_alloc() allocations as well. This has been attempted in the >> thread [3] but it bloats the memory usage, especially on architectures >> with large ARCH_KMALLOC_MINALIGN, where the prepended header basically >> has to occupy the whole ARCH_KMALLOC_MINALIGN block to be DMA safe. >> There are ongoing efforts to reduce this minalign, but the memory >> footprint would still increase, going against the purpose of SLOB, so >> again it would be easier if we could just remove it. >> >> So with this thread I'm interested in hearing arguments/use cases for >> keeping SLOB. There might be obviously users of SLOB whom this >> conversation will not reach, so I assume the eventual next step would be >> to deprecate it in a way that those users are notified when building a >> new kernel and can raise their voice then. Is there a good proven way >> how to do that for a config option like this one? >> >> Thanks, >> Vlastimil >> >> [1] https://lpc.events/event/16/contributions/1272/ - slides in the >> slabs.pdf linked there >> [2] >> https://lore.kernel.org/all/20211017135708.GA8442@kvm.asia-northeast3-a.c.our-ratio-313919.internal/#t >> [3] >> https://lore.kernel.org/all/20210930044202.GP2361455@dread.disaster.area/ >> >> >> ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-09 9:00 ` Vlastimil Babka (?) (?) @ 2022-11-09 15:50 ` Aaro Koskinen -1 siblings, 0 replies; 112+ messages in thread From: Aaro Koskinen @ 2022-11-09 15:50 UTC (permalink / raw) To: Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh Hi, On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >> I believe SLOB can be removed because: > >> > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > >> by putting all objects together, which has its CPU performance costs > >> (locking, lack of percpu caching, searching for free space...). I'm not > >> aware of any "tiny linux" deployment that opts for this. For example, > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >> SLOB impact is too much for those who tried. Googling for > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > I am all for removing SLOB. > > > > There are some devices with configs where SLOB is enabled by default. > > Perhaps, the owners/maintainers of those devices/configs should be > > included into this thread: [...] > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of better debugging features and the memory footprint difference doesn't really matter for my use cases. Looking at history why SLOB was added there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") when separate boards configs were merged, and SX1 board happened to have SLOB in there. This board is nowadays only used in QEMU anyway. There are OMAP1 boards with only 16 MB, but support for those boards will be removed. So from OMAP1 side, I don't think there is any real need for SLOB anymore. A. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 15:50 ` Aaro Koskinen 0 siblings, 0 replies; 112+ messages in thread From: Aaro Koskinen @ 2022-11-09 15:50 UTC (permalink / raw) To: Vlastimil Babka Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, linux-mm, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds Hi, On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >> I believe SLOB can be removed because: > >> > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > >> by putting all objects together, which has its CPU performance costs > >> (locking, lack of percpu caching, searching for free space...). I'm not > >> aware of any "tiny linux" deployment that opts for this. For example, > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >> SLOB impact is too much for those who tried. Googling for > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > I am all for removing SLOB. > > > > There are some devices with configs where SLOB is enabled by default. > > Perhaps, the owners/maintainers of those devices/configs should be > > included into this thread: [...] > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of better debugging features and the memory footprint difference doesn't really matter for my use cases. Looking at history why SLOB was added there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") when separate boards configs were merged, and SX1 board happened to have SLOB in there. This board is nowadays only used in QEMU anyway. There are OMAP1 boards with only 16 MB, but support for those boards will be removed. So from OMAP1 side, I don't think there is any real need for SLOB anymore. A. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 15:50 ` Aaro Koskinen 0 siblings, 0 replies; 112+ messages in thread From: Aaro Koskinen @ 2022-11-09 15:50 UTC (permalink / raw) To: Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh Hi, On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >> I believe SLOB can be removed because: > >> > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > >> by putting all objects together, which has its CPU performance costs > >> (locking, lack of percpu caching, searching for free space...). I'm not > >> aware of any "tiny linux" deployment that opts for this. For example, > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >> SLOB impact is too much for those who tried. Googling for > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > I am all for removing SLOB. > > > > There are some devices with configs where SLOB is enabled by default. > > Perhaps, the owners/maintainers of those devices/configs should be > > included into this thread: [...] > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of better debugging features and the memory footprint difference doesn't really matter for my use cases. Looking at history why SLOB was added there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") when separate boards configs were merged, and SX1 board happened to have SLOB in there. This board is nowadays only used in QEMU anyway. There are OMAP1 boards with only 16 MB, but support for those boards will be removed. So from OMAP1 side, I don't think there is any real need for SLOB anymore. A. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 15:50 ` Aaro Koskinen 0 siblings, 0 replies; 112+ messages in thread From: Aaro Koskinen @ 2022-11-09 15:50 UTC (permalink / raw) To: Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh Hi, On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >> I believe SLOB can be removed because: > >> > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > >> by putting all objects together, which has its CPU performance costs > >> (locking, lack of percpu caching, searching for free space...). I'm not > >> aware of any "tiny linux" deployment that opts for this. For example, > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >> SLOB impact is too much for those who tried. Googling for > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > I am all for removing SLOB. > > > > There are some devices with configs where SLOB is enabled by default. > > Perhaps, the owners/maintainers of those devices/configs should be > > included into this thread: [...] > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of better debugging features and the memory footprint difference doesn't really matter for my use cases. Looking at history why SLOB was added there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") when separate boards configs were merged, and SX1 board happened to have SLOB in there. This board is nowadays only used in QEMU anyway. There are OMAP1 boards with only 16 MB, but support for those boards will be removed. So from OMAP1 side, I don't think there is any real need for SLOB anymore. A. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-09 15:50 ` Aaro Koskinen (?) (?) @ 2022-11-09 16:45 ` Geert Uytterhoeven -1 siblings, 0 replies; 112+ messages in thread From: Geert Uytterhoeven @ 2022-11-09 16:45 UTC (permalink / raw) To: Aaro Koskinen Cc: Vlastimil Babka, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh On Wed, Nov 9, 2022 at 4:53 PM Aaro Koskinen <aaro.koskinen@iki.fi> wrote: > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. > > There are OMAP1 boards with only 16 MB, but support for those boards > will be removed. So from OMAP1 side, I don't think there is any real > need for SLOB anymore. Interestingly, the m68k defconfigs use either SLAB, or the default (SLUB). So the poor old m68k machines (many of which have less than 32 MiB) seem to do fine without SLOB... Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 16:45 ` Geert Uytterhoeven 0 siblings, 0 replies; 112+ messages in thread From: Geert Uytterhoeven @ 2022-11-09 16:45 UTC (permalink / raw) To: Aaro Koskinen Cc: Vlastimil Babka, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh On Wed, Nov 9, 2022 at 4:53 PM Aaro Koskinen <aaro.koskinen@iki.fi> wrote: > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. > > There are OMAP1 boards with only 16 MB, but support for those boards > will be removed. So from OMAP1 side, I don't think there is any real > need for SLOB anymore. Interestingly, the m68k defconfigs use either SLAB, or the default (SLUB). So the poor old m68k machines (many of which have less than 32 MiB) seem to do fine without SLOB... Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 16:45 ` Geert Uytterhoeven 0 siblings, 0 replies; 112+ messages in thread From: Geert Uytterhoeven @ 2022-11-09 16:45 UTC (permalink / raw) To: Aaro Koskinen Cc: Vlastimil Babka, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh On Wed, Nov 9, 2022 at 4:53 PM Aaro Koskinen <aaro.koskinen@iki.fi> wrote: > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. > > There are OMAP1 boards with only 16 MB, but support for those boards > will be removed. So from OMAP1 side, I don't think there is any real > need for SLOB anymore. Interestingly, the m68k defconfigs use either SLAB, or the default (SLUB). So the poor old m68k machines (many of which have less than 32 MiB) seem to do fine without SLOB... Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 16:45 ` Geert Uytterhoeven 0 siblings, 0 replies; 112+ messages in thread From: Geert Uytterhoeven @ 2022-11-09 16:45 UTC (permalink / raw) To: Aaro Koskinen Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, linux-mm, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Janusz Krzysztofik, Russell King, Matthew Wilcox, linux-arm-kernel, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, Vlastimil Babka, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On Wed, Nov 9, 2022 at 4:53 PM Aaro Koskinen <aaro.koskinen@iki.fi> wrote: > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. > > There are OMAP1 boards with only 16 MB, but support for those boards > will be removed. So from OMAP1 side, I don't think there is any real > need for SLOB anymore. Interestingly, the m68k defconfigs use either SLAB, or the default (SLUB). So the poor old m68k machines (many of which have less than 32 MiB) seem to do fine without SLOB... Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-09 15:50 ` Aaro Koskinen (?) (?) @ 2022-11-09 17:45 ` Mike Rapoport -1 siblings, 0 replies; 112+ messages in thread From: Mike Rapoport @ 2022-11-09 17:45 UTC (permalink / raw) To: Aaro Koskinen Cc: Vlastimil Babka, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh On Wed, Nov 09, 2022 at 05:50:08PM +0200, Aaro Koskinen wrote: > Hi, > > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. Looks like the same happened with arch/arm/configs/pxa_defconfig. XCEP board had SLOB in its defconfig and when common pxa_defconfig was created it apparently used it. Looks like the board has 64M of RAM, so dropping CONFIG_SLOB=y from arch/arm/configs/pxa_defconfig and arch/arm/configs/xcep_defconfig seems very reasonable. -- Sincerely yours, Mike. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 17:45 ` Mike Rapoport 0 siblings, 0 replies; 112+ messages in thread From: Mike Rapoport @ 2022-11-09 17:45 UTC (permalink / raw) To: Aaro Koskinen Cc: Vlastimil Babka, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh On Wed, Nov 09, 2022 at 05:50:08PM +0200, Aaro Koskinen wrote: > Hi, > > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. Looks like the same happened with arch/arm/configs/pxa_defconfig. XCEP board had SLOB in its defconfig and when common pxa_defconfig was created it apparently used it. Looks like the board has 64M of RAM, so dropping CONFIG_SLOB=y from arch/arm/configs/pxa_defconfig and arch/arm/configs/xcep_defconfig seems very reasonable. -- Sincerely yours, Mike. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 17:45 ` Mike Rapoport 0 siblings, 0 replies; 112+ messages in thread From: Mike Rapoport @ 2022-11-09 17:45 UTC (permalink / raw) To: Aaro Koskinen Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, linux-mm, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Janusz Krzysztofik, Russell King, Matthew Wilcox, linux-arm-kernel, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, Vlastimil Babka, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On Wed, Nov 09, 2022 at 05:50:08PM +0200, Aaro Koskinen wrote: > Hi, > > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. Looks like the same happened with arch/arm/configs/pxa_defconfig. XCEP board had SLOB in its defconfig and when common pxa_defconfig was created it apparently used it. Looks like the board has 64M of RAM, so dropping CONFIG_SLOB=y from arch/arm/configs/pxa_defconfig and arch/arm/configs/xcep_defconfig seems very reasonable. -- Sincerely yours, Mike. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 17:45 ` Mike Rapoport 0 siblings, 0 replies; 112+ messages in thread From: Mike Rapoport @ 2022-11-09 17:45 UTC (permalink / raw) To: Aaro Koskinen Cc: Vlastimil Babka, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh On Wed, Nov 09, 2022 at 05:50:08PM +0200, Aaro Koskinen wrote: > Hi, > > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. Looks like the same happened with arch/arm/configs/pxa_defconfig. XCEP board had SLOB in its defconfig and when common pxa_defconfig was created it apparently used it. Looks like the board has 64M of RAM, so dropping CONFIG_SLOB=y from arch/arm/configs/pxa_defconfig and arch/arm/configs/xcep_defconfig seems very reasonable. -- Sincerely yours, Mike. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-09 15:50 ` Aaro Koskinen (?) (?) @ 2022-11-09 21:16 ` Janusz Krzysztofik -1 siblings, 0 replies; 112+ messages in thread From: Janusz Krzysztofik @ 2022-11-09 21:16 UTC (permalink / raw) To: Vlastimil Babka, Aaro Koskinen Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh On Wednesday, 9 November 2022 16:50:08 CET Aaro Koskinen wrote: > Hi, > > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. > > There are OMAP1 boards with only 16 MB, but support for those boards > will be removed. So from OMAP1 side, I don't think there is any real > need for SLOB anymore. Moreover, I always had issues with availability of socket buffers during USB device setup when trying to use SLOB on Amstrad Delta based on OMAP1510, the least powerful OMAP1. Then, +1 for SLOB removal. Thanks, Janusz ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 21:16 ` Janusz Krzysztofik 0 siblings, 0 replies; 112+ messages in thread From: Janusz Krzysztofik @ 2022-11-09 21:16 UTC (permalink / raw) To: Vlastimil Babka, Aaro Koskinen Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, linux-mm, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On Wednesday, 9 November 2022 16:50:08 CET Aaro Koskinen wrote: > Hi, > > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. > > There are OMAP1 boards with only 16 MB, but support for those boards > will be removed. So from OMAP1 side, I don't think there is any real > need for SLOB anymore. Moreover, I always had issues with availability of socket buffers during USB device setup when trying to use SLOB on Amstrad Delta based on OMAP1510, the least powerful OMAP1. Then, +1 for SLOB removal. Thanks, Janusz ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 21:16 ` Janusz Krzysztofik 0 siblings, 0 replies; 112+ messages in thread From: Janusz Krzysztofik @ 2022-11-09 21:16 UTC (permalink / raw) To: Vlastimil Babka, Aaro Koskinen Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh On Wednesday, 9 November 2022 16:50:08 CET Aaro Koskinen wrote: > Hi, > > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. > > There are OMAP1 boards with only 16 MB, but support for those boards > will be removed. So from OMAP1 side, I don't think there is any real > need for SLOB anymore. Moreover, I always had issues with availability of socket buffers during USB device setup when trying to use SLOB on Amstrad Delta based on OMAP1510, the least powerful OMAP1. Then, +1 for SLOB removal. Thanks, Janusz _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 21:16 ` Janusz Krzysztofik 0 siblings, 0 replies; 112+ messages in thread From: Janusz Krzysztofik @ 2022-11-09 21:16 UTC (permalink / raw) To: Vlastimil Babka, Aaro Koskinen Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh On Wednesday, 9 November 2022 16:50:08 CET Aaro Koskinen wrote: > Hi, > > On Wed, Nov 09, 2022 at 10:00:25AM +0100, Vlastimil Babka wrote: > > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > > >> I believe SLOB can be removed because: > > >> > > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > > >> by putting all objects together, which has its CPU performance costs > > >> (locking, lack of percpu caching, searching for free space...). I'm not > > >> aware of any "tiny linux" deployment that opts for this. For example, > > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > > >> SLOB impact is too much for those who tried. Googling for > > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > > > I am all for removing SLOB. > > > > > > There are some devices with configs where SLOB is enabled by default. > > > Perhaps, the owners/maintainers of those devices/configs should be > > > included into this thread: > > [...] > > > > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > > I have been using SLUB on my OMAP1 boards with 32 MB RAM, because of > better debugging features and the memory footprint difference doesn't > really matter for my use cases. Looking at history why SLOB was added > there, it seems it came from 6cfce27c14aa ("omap1: Add omap1_defconfig") > when separate boards configs were merged, and SX1 board happened to have > SLOB in there. This board is nowadays only used in QEMU anyway. > > There are OMAP1 boards with only 16 MB, but support for those boards > will be removed. So from OMAP1 side, I don't think there is any real > need for SLOB anymore. Moreover, I always had issues with availability of socket buffers during USB device setup when trying to use SLOB on Amstrad Delta based on OMAP1510, the least powerful OMAP1. Then, +1 for SLOB removal. Thanks, Janusz _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-08 21:44 ` Pasha Tatashin 2022-11-09 9:00 ` Vlastimil Babka @ 2022-11-09 17:57 ` Conor.Dooley 2022-11-09 23:00 ` Damien Le Moal 2022-11-11 10:33 ` Vlastimil Babka 2 siblings, 1 reply; 112+ messages in thread From: Conor.Dooley @ 2022-11-09 17:57 UTC (permalink / raw) To: damien.lemoal Cc: cl, rientjes, iamjoonsoo.kim, penberg, 42.hyeyoo, willy, roman.gushchin, pasha.tatashin, torvalds, linux-mm, vbabka, linux-kernel, catalin.marinas, rkovhaev, akpm +CC Damien > There are some devices with configs where SLOB is enabled by default. > Perhaps, the owners/maintainers of those devices/configs should be > included into this thread: > > tatashin@soleen:~/x/linux$ git grep SLOB=y > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y Saw you were not added to the CC Damien & I know you don't want your baby broken! On 08/11/2022 21:44, Pasha Tatashin wrote: > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >> >> Hi, >> >> as we all know, we currently have three slab allocators. As we discussed >> at LPC [1], it is my hope that one of these allocators has a future, and >> two of them do not. >> >> The unsurprising reasons include code maintenance burden, other features >> compatible with only a subset of allocators (or more effort spent on the >> features), blocking API improvements (more on that below), and my >> inability to pronounce SLAB and SLUB in a properly distinguishable way, >> without resorting to spelling out the letters. >> >> I think (but may be proven wrong) that SLOB is the easier target of the >> two to be removed, so I'd like to focus on it first. >> >> I believe SLOB can be removed because: >> >> - AFAIK nobody really uses it? It strives for minimal memory footprint >> by putting all objects together, which has its CPU performance costs >> (locking, lack of percpu caching, searching for free space...). I'm not >> aware of any "tiny linux" deployment that opts for this. For example, >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >> SLOB impact is too much for those who tried. Googling for >> "CONFIG_SLOB=y" yielded nothing useful. > > I am all for removing SLOB. > > There are some devices with configs where SLOB is enabled by default. > Perhaps, the owners/maintainers of those devices/configs should be > included into this thread: > > tatashin@soleen:~/x/linux$ git grep SLOB=y > arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y > arch/arm/configs/collie_defconfig:CONFIG_SLOB=y > arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y > arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y > arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y > arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y > arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y > arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y > kernel/configs/tiny.config:CONFIG_SLOB=y > >> >> - Last time we discussed it [2], it seemed SLUB memory requirements can >> be brought very close to SLOB's if needed. Of course it can never have >> as small footprint as SLOB due to separate kmem_caches, but the >> difference is not that significant, unless somebody still tries to use >> Linux on very tiny systems (goes back to the previous point). >> >> Besides the smaller maintenance burden, removing SLOB would allow us to >> do a useful API improvement - the ability to use kfree() for both >> objects allocated by kmalloc() and kmem_cache_alloc(). Currently the >> latter has to be freed by kmem_cache_free(), passing a kmem_cache >> pointer in addition to the object pointer. With SLUB and SLAB, it is >> however possible to use kfree() instead, as the kmalloc caches and the >> rest of kmem_caches are the same and kfree() can lookup the kmem_cache >> from object pointer easily for any of those. XFS has apparently did that >> for years without anyone noticing it's broken on SLOB [3], and >> legitimizing and expanding this would help some use cases beside XFS >> (IIRC Matthew mentioned rcu-based freeing for example). >> >> However for SLOB to support kfree() on all allocations, it would need to >> store object size of allocated objects (which it currently does only for >> kmalloc() objects, prepending a size header to the object), but for >> kmem_cache_alloc() allocations as well. This has been attempted in the >> thread [3] but it bloats the memory usage, especially on architectures >> with large ARCH_KMALLOC_MINALIGN, where the prepended header basically >> has to occupy the whole ARCH_KMALLOC_MINALIGN block to be DMA safe. >> There are ongoing efforts to reduce this minalign, but the memory >> footprint would still increase, going against the purpose of SLOB, so >> again it would be easier if we could just remove it. >> >> So with this thread I'm interested in hearing arguments/use cases for >> keeping SLOB. There might be obviously users of SLOB whom this >> conversation will not reach, so I assume the eventual next step would be >> to deprecate it in a way that those users are notified when building a >> new kernel and can raise their voice then. Is there a good proven way >> how to do that for a config option like this one? >> >> Thanks, >> Vlastimil >> >> [1] https://lpc.events/event/16/contributions/1272/ - slides in the >> slabs.pdf linked there >> [2] >> https://lore.kernel.org/all/20211017135708.GA8442@kvm.asia-northeast3-a.c.our-ratio-313919.internal/#t >> [3] >> https://lore.kernel.org/all/20210930044202.GP2361455@dread.disaster.area/ >> >> >> ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-09 17:57 ` Conor.Dooley @ 2022-11-09 23:00 ` Damien Le Moal 2022-11-11 10:25 ` Vlastimil Babka 0 siblings, 1 reply; 112+ messages in thread From: Damien Le Moal @ 2022-11-09 23:00 UTC (permalink / raw) To: Conor.Dooley Cc: cl, rientjes, iamjoonsoo.kim, penberg, 42.hyeyoo, willy, roman.gushchin, pasha.tatashin, torvalds, linux-mm, vbabka, linux-kernel, catalin.marinas, rkovhaev, akpm On 11/10/22 02:57, Conor.Dooley@microchip.com wrote: > +CC Damien > >> There are some devices with configs where SLOB is enabled by default. >> Perhaps, the owners/maintainers of those devices/configs should be >> included into this thread: >> >> tatashin@soleen:~/x/linux$ git grep SLOB=y > >> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > > Saw you were not added to the CC Damien & I know you don't want your > baby broken! :) I set SLOB=y for the K210 as the config help mentions it is a bit more efficient in low memory cases. I did run a few times with SLAB and it was OK, so removing slob should not be a problem. Can check again. Cheers. > > > On 08/11/2022 21:44, Pasha Tatashin wrote: >> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>> >>> Hi, >>> >>> as we all know, we currently have three slab allocators. As we discussed >>> at LPC [1], it is my hope that one of these allocators has a future, and >>> two of them do not. >>> >>> The unsurprising reasons include code maintenance burden, other features >>> compatible with only a subset of allocators (or more effort spent on the >>> features), blocking API improvements (more on that below), and my >>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>> without resorting to spelling out the letters. >>> >>> I think (but may be proven wrong) that SLOB is the easier target of the >>> two to be removed, so I'd like to focus on it first. >>> >>> I believe SLOB can be removed because: >>> >>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>> by putting all objects together, which has its CPU performance costs >>> (locking, lack of percpu caching, searching for free space...). I'm not >>> aware of any "tiny linux" deployment that opts for this. For example, >>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>> SLOB impact is too much for those who tried. Googling for >>> "CONFIG_SLOB=y" yielded nothing useful. >> >> I am all for removing SLOB. >> >> There are some devices with configs where SLOB is enabled by default. >> Perhaps, the owners/maintainers of those devices/configs should be >> included into this thread: >> >> tatashin@soleen:~/x/linux$ git grep SLOB=y >> arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y >> arch/arm/configs/collie_defconfig:CONFIG_SLOB=y >> arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y >> arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y >> arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y >> arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y >> arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y >> arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y >> arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y >> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >> arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y >> arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y >> arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y >> arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y >> arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y >> kernel/configs/tiny.config:CONFIG_SLOB=y >> >>> >>> - Last time we discussed it [2], it seemed SLUB memory requirements can >>> be brought very close to SLOB's if needed. Of course it can never have >>> as small footprint as SLOB due to separate kmem_caches, but the >>> difference is not that significant, unless somebody still tries to use >>> Linux on very tiny systems (goes back to the previous point). >>> >>> Besides the smaller maintenance burden, removing SLOB would allow us to >>> do a useful API improvement - the ability to use kfree() for both >>> objects allocated by kmalloc() and kmem_cache_alloc(). Currently the >>> latter has to be freed by kmem_cache_free(), passing a kmem_cache >>> pointer in addition to the object pointer. With SLUB and SLAB, it is >>> however possible to use kfree() instead, as the kmalloc caches and the >>> rest of kmem_caches are the same and kfree() can lookup the kmem_cache >>> from object pointer easily for any of those. XFS has apparently did that >>> for years without anyone noticing it's broken on SLOB [3], and >>> legitimizing and expanding this would help some use cases beside XFS >>> (IIRC Matthew mentioned rcu-based freeing for example). >>> >>> However for SLOB to support kfree() on all allocations, it would need to >>> store object size of allocated objects (which it currently does only for >>> kmalloc() objects, prepending a size header to the object), but for >>> kmem_cache_alloc() allocations as well. This has been attempted in the >>> thread [3] but it bloats the memory usage, especially on architectures >>> with large ARCH_KMALLOC_MINALIGN, where the prepended header basically >>> has to occupy the whole ARCH_KMALLOC_MINALIGN block to be DMA safe. >>> There are ongoing efforts to reduce this minalign, but the memory >>> footprint would still increase, going against the purpose of SLOB, so >>> again it would be easier if we could just remove it. >>> >>> So with this thread I'm interested in hearing arguments/use cases for >>> keeping SLOB. There might be obviously users of SLOB whom this >>> conversation will not reach, so I assume the eventual next step would be >>> to deprecate it in a way that those users are notified when building a >>> new kernel and can raise their voice then. Is there a good proven way >>> how to do that for a config option like this one? >>> >>> Thanks, >>> Vlastimil >>> >>> [1] https://lpc.events/event/16/contributions/1272/ - slides in the >>> slabs.pdf linked there >>> [2] >>> https://lore.kernel.org/all/20211017135708.GA8442@kvm.asia-northeast3-a.c.our-ratio-313919.internal/#t >>> [3] >>> https://lore.kernel.org/all/20210930044202.GP2361455@dread.disaster.area/ >>> >>> >>> -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-09 23:00 ` Damien Le Moal @ 2022-11-11 10:25 ` Vlastimil Babka 2022-11-12 1:40 ` Damien Le Moal 0 siblings, 1 reply; 112+ messages in thread From: Vlastimil Babka @ 2022-11-11 10:25 UTC (permalink / raw) To: Damien Le Moal, Conor.Dooley Cc: cl, rientjes, iamjoonsoo.kim, penberg, 42.hyeyoo, willy, roman.gushchin, pasha.tatashin, torvalds, linux-mm, linux-kernel, catalin.marinas, rkovhaev, akpm On 11/10/22 00:00, Damien Le Moal wrote: > On 11/10/22 02:57, Conor.Dooley@microchip.com wrote: >> +CC Damien >> >>> There are some devices with configs where SLOB is enabled by default. >>> Perhaps, the owners/maintainers of those devices/configs should be >>> included into this thread: >>> >>> tatashin@soleen:~/x/linux$ git grep SLOB=y >> >>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >> >> Saw you were not added to the CC Damien & I know you don't want your >> baby broken! > > :) > > I set SLOB=y for the K210 as the config help mentions it is a bit more > efficient in low memory cases. I did run a few times with SLAB and it > was OK, so removing slob should not be a problem. Can check again. Thanks, but please check with SLUB, not SLAB, if possible. Disable SLUB_CPU_PARTIAL (default y on SMP) if you want to minimize the memory usage. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-11 10:25 ` Vlastimil Babka @ 2022-11-12 1:40 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-12 1:40 UTC (permalink / raw) To: Vlastimil Babka, Conor.Dooley Cc: cl, rientjes, iamjoonsoo.kim, penberg, 42.hyeyoo, willy, roman.gushchin, pasha.tatashin, torvalds, linux-mm, linux-kernel, catalin.marinas, rkovhaev, akpm On 11/11/22 19:25, Vlastimil Babka wrote: > On 11/10/22 00:00, Damien Le Moal wrote: >> On 11/10/22 02:57, Conor.Dooley@microchip.com wrote: >>> +CC Damien >>> >>>> There are some devices with configs where SLOB is enabled by default. >>>> Perhaps, the owners/maintainers of those devices/configs should be >>>> included into this thread: >>>> >>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>> >>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>> >>> Saw you were not added to the CC Damien & I know you don't want your >>> baby broken! >> >> :) >> >> I set SLOB=y for the K210 as the config help mentions it is a bit more >> efficient in low memory cases. I did run a few times with SLAB and it >> was OK, so removing slob should not be a problem. Can check again. > > Thanks, but please check with SLUB, not SLAB, if possible. > Disable SLUB_CPU_PARTIAL (default y on SMP) if you want to minimize the > memory usage. Thanks for the hint. Will try that. -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-08 21:44 ` Pasha Tatashin 2022-11-09 9:00 ` Vlastimil Babka 2022-11-09 17:57 ` Conor.Dooley @ 2022-11-11 10:33 ` Vlastimil Babka 2 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-11 10:33 UTC (permalink / raw) To: Pasha Tatashin Cc: Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, linux-olinux-omap@vger.kernel.orgmap, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Damien Le Moal, Paul Cercueil On 11/8/22 22:44, Pasha Tatashin wrote: > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >> >> Hi, >> >> as we all know, we currently have three slab allocators. As we discussed >> at LPC [1], it is my hope that one of these allocators has a future, and >> two of them do not. >> >> The unsurprising reasons include code maintenance burden, other features >> compatible with only a subset of allocators (or more effort spent on the >> features), blocking API improvements (more on that below), and my >> inability to pronounce SLAB and SLUB in a properly distinguishable way, >> without resorting to spelling out the letters. >> >> I think (but may be proven wrong) that SLOB is the easier target of the >> two to be removed, so I'd like to focus on it first. >> >> I believe SLOB can be removed because: >> >> - AFAIK nobody really uses it? It strives for minimal memory footprint >> by putting all objects together, which has its CPU performance costs >> (locking, lack of percpu caching, searching for free space...). I'm not >> aware of any "tiny linux" deployment that opts for this. For example, >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >> SLOB impact is too much for those who tried. Googling for >> "CONFIG_SLOB=y" yielded nothing useful. > > I am all for removing SLOB. > > There are some devices with configs where SLOB is enabled by default. > Perhaps, the owners/maintainers of those devices/configs should be > included into this thread: > > tatashin@soleen:~/x/linux$ git grep SLOB=y > arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y > arch/arm/configs/collie_defconfig:CONFIG_SLOB=y > arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y > arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y > arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y > arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y > arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y > arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y > kernel/configs/tiny.config:CONFIG_SLOB=y Turns out that since SLOB depends on EXPERT, many of those lack it so running make defconfig ends up with SLUB anyway, unless I miss something. Only a subset has both SLOB and EXPERT: > git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` arch/arm/configs/collie_defconfig:CONFIG_EXPERT=y arch/arm/configs/omap1_defconfig:CONFIG_EXPERT=y arch/arm/configs/tct_hammer_defconfig:CONFIG_EXPERT=y arch/arm/configs/xcep_defconfig:CONFIG_EXPERT=y arch/openrisc/configs/or1ksim_defconfig:CONFIG_EXPERT=y arch/openrisc/configs/simple_smp_defconfig:CONFIG_EXPERT=y arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y arch/x86/configs/x86_64_defconfig:CONFIG_EXPERT=y So how about the following for -next and 6.2? I did try depending on BROKEN as suggested, but didn't find a way to override it by enabling CONFIG_BROKEN=y without modifying a Kconfig file, so this seems more graceful to me? ----8<---- From 58028e06eee8a7fb8a11908b6941fae000660e4d Mon Sep 17 00:00:00 2001 From: Vlastimil Babka <vbabka@suse.cz> Date: Fri, 11 Nov 2022 11:04:55 +0100 Subject: [PATCH] mm, slob: rename CONFIG_SLOB to CONFIG_SLOB_DEPRECATED As explained in [1], we would like to remove SLOB if possible. - There are no known users that need its somewhat lower memory footprint so much that they cannot handle SLUB instead. - It is an extra maintenance burden, and a number of features are incompatible with it. - It blocks the API improvement of allowing kfree() on objects allocated via kmem_cache_alloc(). As the first step, rename the CONFIG_SLOB option in the slab allocator configuration choice to CONFIG_SLOB_DEPRECATED. Add CONFIG_SLOB depending on CONFIG_SLOB_DEPRECATED as an internal option to avoid code churn. This will cause existing .config files and defconfigs with CONFIG_SLOB=y to silently switch to the default (and recommended replacement) SLUB, while still allowing SLOB to be configured by anyone that notices and needs it. But those should contact the slab maintainers and linux-mm@kvack.org as explained in the updated help. With no valid objections, the plan is to update the existing defconfigs to SLUB and remove SLOB in a few cycles. [1] https://lore.kernel.org/all/b35c3f82-f67b-2103-7d82-7a7ba7521439@suse.cz/ Signed-off-by: Vlastimil Babka <vbabka@suse.cz> --- mm/Kconfig | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 57e1d8c5b505..27c0d13314be 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -219,17 +219,28 @@ config SLUB and has enhanced diagnostics. SLUB is the default choice for a slab allocator. -config SLOB +config SLOB_DEPRECATED depends on EXPERT - bool "SLOB (Simple Allocator)" + bool "SLOB (Simple Allocator - DEPRECATED)" depends on !PREEMPT_RT help + Deprecated and scheduled for removal in a few cycles. SLUB + recommended as replacement. If you need SLOB to stay, please + contact linux-mm@kvack.org and people listed in the SLAB + ALLOCATOR section of MAINTAINERS file, and state your use + case. + SLOB replaces the stock allocator with a drastically simpler allocator. SLOB is generally more space efficient but does not perform as well on large systems. endchoice +config SLOB + bool + default y + depends on SLOB_DEPRECATED + config SLAB_MERGE_DEFAULT bool "Allow slab caches to be merged" default y -- 2.38.0 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply related [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-11 10:33 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-11 10:33 UTC (permalink / raw) To: Pasha Tatashin Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Hyeonggon Yoo, Christoph Lameter, linux-olinux-omap@vger.kernel.orgmap, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Damien Le Moal, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, linux-riscv, Andrew Morton, Linus Torvalds On 11/8/22 22:44, Pasha Tatashin wrote: > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >> >> Hi, >> >> as we all know, we currently have three slab allocators. As we discussed >> at LPC [1], it is my hope that one of these allocators has a future, and >> two of them do not. >> >> The unsurprising reasons include code maintenance burden, other features >> compatible with only a subset of allocators (or more effort spent on the >> features), blocking API improvements (more on that below), and my >> inability to pronounce SLAB and SLUB in a properly distinguishable way, >> without resorting to spelling out the letters. >> >> I think (but may be proven wrong) that SLOB is the easier target of the >> two to be removed, so I'd like to focus on it first. >> >> I believe SLOB can be removed because: >> >> - AFAIK nobody really uses it? It strives for minimal memory footprint >> by putting all objects together, which has its CPU performance costs >> (locking, lack of percpu caching, searching for free space...). I'm not >> aware of any "tiny linux" deployment that opts for this. For example, >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >> SLOB impact is too much for those who tried. Googling for >> "CONFIG_SLOB=y" yielded nothing useful. > > I am all for removing SLOB. > > There are some devices with configs where SLOB is enabled by default. > Perhaps, the owners/maintainers of those devices/configs should be > included into this thread: > > tatashin@soleen:~/x/linux$ git grep SLOB=y > arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y > arch/arm/configs/collie_defconfig:CONFIG_SLOB=y > arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y > arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y > arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y > arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y > arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y > arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y > kernel/configs/tiny.config:CONFIG_SLOB=y Turns out that since SLOB depends on EXPERT, many of those lack it so running make defconfig ends up with SLUB anyway, unless I miss something. Only a subset has both SLOB and EXPERT: > git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` arch/arm/configs/collie_defconfig:CONFIG_EXPERT=y arch/arm/configs/omap1_defconfig:CONFIG_EXPERT=y arch/arm/configs/tct_hammer_defconfig:CONFIG_EXPERT=y arch/arm/configs/xcep_defconfig:CONFIG_EXPERT=y arch/openrisc/configs/or1ksim_defconfig:CONFIG_EXPERT=y arch/openrisc/configs/simple_smp_defconfig:CONFIG_EXPERT=y arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y arch/x86/configs/x86_64_defconfig:CONFIG_EXPERT=y So how about the following for -next and 6.2? I did try depending on BROKEN as suggested, but didn't find a way to override it by enabling CONFIG_BROKEN=y without modifying a Kconfig file, so this seems more graceful to me? ----8<---- From 58028e06eee8a7fb8a11908b6941fae000660e4d Mon Sep 17 00:00:00 2001 From: Vlastimil Babka <vbabka@suse.cz> Date: Fri, 11 Nov 2022 11:04:55 +0100 Subject: [PATCH] mm, slob: rename CONFIG_SLOB to CONFIG_SLOB_DEPRECATED As explained in [1], we would like to remove SLOB if possible. - There are no known users that need its somewhat lower memory footprint so much that they cannot handle SLUB instead. - It is an extra maintenance burden, and a number of features are incompatible with it. - It blocks the API improvement of allowing kfree() on objects allocated via kmem_cache_alloc(). As the first step, rename the CONFIG_SLOB option in the slab allocator configuration choice to CONFIG_SLOB_DEPRECATED. Add CONFIG_SLOB depending on CONFIG_SLOB_DEPRECATED as an internal option to avoid code churn. This will cause existing .config files and defconfigs with CONFIG_SLOB=y to silently switch to the default (and recommended replacement) SLUB, while still allowing SLOB to be configured by anyone that notices and needs it. But those should contact the slab maintainers and linux-mm@kvack.org as explained in the updated help. With no valid objections, the plan is to update the existing defconfigs to SLUB and remove SLOB in a few cycles. [1] https://lore.kernel.org/all/b35c3f82-f67b-2103-7d82-7a7ba7521439@suse.cz/ Signed-off-by: Vlastimil Babka <vbabka@suse.cz> --- mm/Kconfig | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 57e1d8c5b505..27c0d13314be 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -219,17 +219,28 @@ config SLUB and has enhanced diagnostics. SLUB is the default choice for a slab allocator. -config SLOB +config SLOB_DEPRECATED depends on EXPERT - bool "SLOB (Simple Allocator)" + bool "SLOB (Simple Allocator - DEPRECATED)" depends on !PREEMPT_RT help + Deprecated and scheduled for removal in a few cycles. SLUB + recommended as replacement. If you need SLOB to stay, please + contact linux-mm@kvack.org and people listed in the SLAB + ALLOCATOR section of MAINTAINERS file, and state your use + case. + SLOB replaces the stock allocator with a drastically simpler allocator. SLOB is generally more space efficient but does not perform as well on large systems. endchoice +config SLOB + bool + default y + depends on SLOB_DEPRECATED + config SLAB_MERGE_DEFAULT bool "Allow slab caches to be merged" default y -- 2.38.0 ^ permalink raw reply related [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-11 10:33 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-11 10:33 UTC (permalink / raw) To: Pasha Tatashin Cc: Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, linux-olinux-omap@vger.kernel.orgmap, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Damien Le Moal, Paul Cercueil On 11/8/22 22:44, Pasha Tatashin wrote: > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >> >> Hi, >> >> as we all know, we currently have three slab allocators. As we discussed >> at LPC [1], it is my hope that one of these allocators has a future, and >> two of them do not. >> >> The unsurprising reasons include code maintenance burden, other features >> compatible with only a subset of allocators (or more effort spent on the >> features), blocking API improvements (more on that below), and my >> inability to pronounce SLAB and SLUB in a properly distinguishable way, >> without resorting to spelling out the letters. >> >> I think (but may be proven wrong) that SLOB is the easier target of the >> two to be removed, so I'd like to focus on it first. >> >> I believe SLOB can be removed because: >> >> - AFAIK nobody really uses it? It strives for minimal memory footprint >> by putting all objects together, which has its CPU performance costs >> (locking, lack of percpu caching, searching for free space...). I'm not >> aware of any "tiny linux" deployment that opts for this. For example, >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >> SLOB impact is too much for those who tried. Googling for >> "CONFIG_SLOB=y" yielded nothing useful. > > I am all for removing SLOB. > > There are some devices with configs where SLOB is enabled by default. > Perhaps, the owners/maintainers of those devices/configs should be > included into this thread: > > tatashin@soleen:~/x/linux$ git grep SLOB=y > arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y > arch/arm/configs/collie_defconfig:CONFIG_SLOB=y > arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y > arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y > arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y > arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y > arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y > arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y > kernel/configs/tiny.config:CONFIG_SLOB=y Turns out that since SLOB depends on EXPERT, many of those lack it so running make defconfig ends up with SLUB anyway, unless I miss something. Only a subset has both SLOB and EXPERT: > git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` arch/arm/configs/collie_defconfig:CONFIG_EXPERT=y arch/arm/configs/omap1_defconfig:CONFIG_EXPERT=y arch/arm/configs/tct_hammer_defconfig:CONFIG_EXPERT=y arch/arm/configs/xcep_defconfig:CONFIG_EXPERT=y arch/openrisc/configs/or1ksim_defconfig:CONFIG_EXPERT=y arch/openrisc/configs/simple_smp_defconfig:CONFIG_EXPERT=y arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y arch/x86/configs/x86_64_defconfig:CONFIG_EXPERT=y So how about the following for -next and 6.2? I did try depending on BROKEN as suggested, but didn't find a way to override it by enabling CONFIG_BROKEN=y without modifying a Kconfig file, so this seems more graceful to me? ----8<---- From 58028e06eee8a7fb8a11908b6941fae000660e4d Mon Sep 17 00:00:00 2001 From: Vlastimil Babka <vbabka@suse.cz> Date: Fri, 11 Nov 2022 11:04:55 +0100 Subject: [PATCH] mm, slob: rename CONFIG_SLOB to CONFIG_SLOB_DEPRECATED As explained in [1], we would like to remove SLOB if possible. - There are no known users that need its somewhat lower memory footprint so much that they cannot handle SLUB instead. - It is an extra maintenance burden, and a number of features are incompatible with it. - It blocks the API improvement of allowing kfree() on objects allocated via kmem_cache_alloc(). As the first step, rename the CONFIG_SLOB option in the slab allocator configuration choice to CONFIG_SLOB_DEPRECATED. Add CONFIG_SLOB depending on CONFIG_SLOB_DEPRECATED as an internal option to avoid code churn. This will cause existing .config files and defconfigs with CONFIG_SLOB=y to silently switch to the default (and recommended replacement) SLUB, while still allowing SLOB to be configured by anyone that notices and needs it. But those should contact the slab maintainers and linux-mm@kvack.org as explained in the updated help. With no valid objections, the plan is to update the existing defconfigs to SLUB and remove SLOB in a few cycles. [1] https://lore.kernel.org/all/b35c3f82-f67b-2103-7d82-7a7ba7521439@suse.cz/ Signed-off-by: Vlastimil Babka <vbabka@suse.cz> --- mm/Kconfig | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 57e1d8c5b505..27c0d13314be 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -219,17 +219,28 @@ config SLUB and has enhanced diagnostics. SLUB is the default choice for a slab allocator. -config SLOB +config SLOB_DEPRECATED depends on EXPERT - bool "SLOB (Simple Allocator)" + bool "SLOB (Simple Allocator - DEPRECATED)" depends on !PREEMPT_RT help + Deprecated and scheduled for removal in a few cycles. SLUB + recommended as replacement. If you need SLOB to stay, please + contact linux-mm@kvack.org and people listed in the SLAB + ALLOCATOR section of MAINTAINERS file, and state your use + case. + SLOB replaces the stock allocator with a drastically simpler allocator. SLOB is generally more space efficient but does not perform as well on large systems. endchoice +config SLOB + bool + default y + depends on SLOB_DEPRECATED + config SLAB_MERGE_DEFAULT bool "Allow slab caches to be merged" default y -- 2.38.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-11 10:33 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-11 10:33 UTC (permalink / raw) To: Pasha Tatashin Cc: Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, linux-olinux-omap@vger.kernel.orgmap, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Damien Le Moal, Paul Cercueil On 11/8/22 22:44, Pasha Tatashin wrote: > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >> >> Hi, >> >> as we all know, we currently have three slab allocators. As we discussed >> at LPC [1], it is my hope that one of these allocators has a future, and >> two of them do not. >> >> The unsurprising reasons include code maintenance burden, other features >> compatible with only a subset of allocators (or more effort spent on the >> features), blocking API improvements (more on that below), and my >> inability to pronounce SLAB and SLUB in a properly distinguishable way, >> without resorting to spelling out the letters. >> >> I think (but may be proven wrong) that SLOB is the easier target of the >> two to be removed, so I'd like to focus on it first. >> >> I believe SLOB can be removed because: >> >> - AFAIK nobody really uses it? It strives for minimal memory footprint >> by putting all objects together, which has its CPU performance costs >> (locking, lack of percpu caching, searching for free space...). I'm not >> aware of any "tiny linux" deployment that opts for this. For example, >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >> SLOB impact is too much for those who tried. Googling for >> "CONFIG_SLOB=y" yielded nothing useful. > > I am all for removing SLOB. > > There are some devices with configs where SLOB is enabled by default. > Perhaps, the owners/maintainers of those devices/configs should be > included into this thread: > > tatashin@soleen:~/x/linux$ git grep SLOB=y > arch/arm/configs/clps711x_defconfig:CONFIG_SLOB=y > arch/arm/configs/collie_defconfig:CONFIG_SLOB=y > arch/arm/configs/multi_v4t_defconfig:CONFIG_SLOB=y > arch/arm/configs/omap1_defconfig:CONFIG_SLOB=y > arch/arm/configs/pxa_defconfig:CONFIG_SLOB=y > arch/arm/configs/tct_hammer_defconfig:CONFIG_SLOB=y > arch/arm/configs/xcep_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/or1ksim_defconfig:CONFIG_SLOB=y > arch/openrisc/configs/simple_smp_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7201_defconfig:CONFIG_SLOB=y > arch/sh/configs/rsk7203_defconfig:CONFIG_SLOB=y > arch/sh/configs/se7206_defconfig:CONFIG_SLOB=y > arch/sh/configs/shmin_defconfig:CONFIG_SLOB=y > arch/sh/configs/shx3_defconfig:CONFIG_SLOB=y > kernel/configs/tiny.config:CONFIG_SLOB=y Turns out that since SLOB depends on EXPERT, many of those lack it so running make defconfig ends up with SLUB anyway, unless I miss something. Only a subset has both SLOB and EXPERT: > git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` arch/arm/configs/collie_defconfig:CONFIG_EXPERT=y arch/arm/configs/omap1_defconfig:CONFIG_EXPERT=y arch/arm/configs/tct_hammer_defconfig:CONFIG_EXPERT=y arch/arm/configs/xcep_defconfig:CONFIG_EXPERT=y arch/openrisc/configs/or1ksim_defconfig:CONFIG_EXPERT=y arch/openrisc/configs/simple_smp_defconfig:CONFIG_EXPERT=y arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y arch/x86/configs/x86_64_defconfig:CONFIG_EXPERT=y So how about the following for -next and 6.2? I did try depending on BROKEN as suggested, but didn't find a way to override it by enabling CONFIG_BROKEN=y without modifying a Kconfig file, so this seems more graceful to me? ----8<---- From 58028e06eee8a7fb8a11908b6941fae000660e4d Mon Sep 17 00:00:00 2001 From: Vlastimil Babka <vbabka@suse.cz> Date: Fri, 11 Nov 2022 11:04:55 +0100 Subject: [PATCH] mm, slob: rename CONFIG_SLOB to CONFIG_SLOB_DEPRECATED As explained in [1], we would like to remove SLOB if possible. - There are no known users that need its somewhat lower memory footprint so much that they cannot handle SLUB instead. - It is an extra maintenance burden, and a number of features are incompatible with it. - It blocks the API improvement of allowing kfree() on objects allocated via kmem_cache_alloc(). As the first step, rename the CONFIG_SLOB option in the slab allocator configuration choice to CONFIG_SLOB_DEPRECATED. Add CONFIG_SLOB depending on CONFIG_SLOB_DEPRECATED as an internal option to avoid code churn. This will cause existing .config files and defconfigs with CONFIG_SLOB=y to silently switch to the default (and recommended replacement) SLUB, while still allowing SLOB to be configured by anyone that notices and needs it. But those should contact the slab maintainers and linux-mm@kvack.org as explained in the updated help. With no valid objections, the plan is to update the existing defconfigs to SLUB and remove SLOB in a few cycles. [1] https://lore.kernel.org/all/b35c3f82-f67b-2103-7d82-7a7ba7521439@suse.cz/ Signed-off-by: Vlastimil Babka <vbabka@suse.cz> --- mm/Kconfig | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 57e1d8c5b505..27c0d13314be 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -219,17 +219,28 @@ config SLUB and has enhanced diagnostics. SLUB is the default choice for a slab allocator. -config SLOB +config SLOB_DEPRECATED depends on EXPERT - bool "SLOB (Simple Allocator)" + bool "SLOB (Simple Allocator - DEPRECATED)" depends on !PREEMPT_RT help + Deprecated and scheduled for removal in a few cycles. SLUB + recommended as replacement. If you need SLOB to stay, please + contact linux-mm@kvack.org and people listed in the SLAB + ALLOCATOR section of MAINTAINERS file, and state your use + case. + SLOB replaces the stock allocator with a drastically simpler allocator. SLOB is generally more space efficient but does not perform as well on large systems. endchoice +config SLOB + bool + default y + depends on SLOB_DEPRECATED + config SLAB_MERGE_DEFAULT bool "Allow slab caches to be merged" default y -- 2.38.0 ^ permalink raw reply related [flat|nested] 112+ messages in thread
* [lkp] [+5395 bytes kernel size regression] [i386-tinyconfig] [b7c8731082] Deprecating and removing SLOB 2022-11-11 10:33 ` Vlastimil Babka ` (2 preceding siblings ...) (?) @ 2022-11-11 12:49 ` kernel test robot 2022-11-11 17:24 ` Josh Triplett -1 siblings, 1 reply; 112+ messages in thread From: kernel test robot @ 2022-11-11 12:49 UTC (permalink / raw) To: Vlastimil Babka; +Cc: oe-kbuild-all, lkp, Josh Triplett FYI, we noticed a +5395 bytes kernel size regression due to commit: commit: b7c873108294f0fd19c7e4d6b038a05375b3cd33 (Deprecating and removing SLOB) url: https://github.com/intel-lab-lkp/linux/commits/Vlastimil-Babka/mm-slob-rename-CONFIG_SLOB-to-CONFIG_SLOB_DEPRECATED/20221111-183529 base: https://git.kernel.org/cgit/linux/kernel/git/akpm/mm.git mm-everything patch subject: [PATCH] mm, slob: rename CONFIG_SLOB to CONFIG_SLOB_DEPRECATED Details as below (size data is obtained by `nm --size-sort vmlinux`): e1e3de44: Merge branch 'mm-nonmm-unstable' into mm-everything b7c87310: Deprecating and removing SLOB +-----------------------------------------+----------+----------+-------+ | symbol | e1e3de44 | b7c87310 | delta | +-----------------------------------------+----------+----------+-------+ | bzImage | 491680 | 494688 | 3008 | | nm.t.___slab_alloc | 0 | 666 | 666 | | nm.t.calculate_sizes | 0 | 381 | 381 | | nm.t.kmem_cache_free_bulk | 0 | 375 | 375 | | nm.t.__kmem_cache_do_shrink | 0 | 314 | 314 | | nm.t.deactivate_slab | 0 | 306 | 306 | | nm.t.allocate_slab | 0 | 276 | 276 | | nm.T.__kmem_cache_create | 26 | 291 | 265 | | nm.D.kmalloc_info | 0 | 264 | 264 | | nm.t.property_entry_copy_data | 0 | 247 | 247 | | nm.t.__slab_free | 0 | 241 | 241 | | nm.T.__kmem_cache_alloc_node | 0 | 178 | 178 | | nm.T.kmem_cache_alloc_node | 5 | 182 | 177 | | nm.T.kmem_cache_alloc | 8 | 177 | 169 | | nm.T.kmem_cache_alloc_lru | 10 | 179 | 169 | | nm.t.slab_free_freelist_hook | 0 | 167 | 167 | | nm.t.do_initcalls | 0 | 166 | 166 | | nm.T.kmem_cache_alloc_bulk | 67 | 224 | 157 | | nm.T.__kmem_cache_shutdown | 3 | 154 | 151 | | nm.T.kmem_cache_init | 21 | 167 | 146 | | nm.t.flush_all_cpus_locked | 0 | 131 | 131 | | nm.T.__kmem_cache_free | 0 | 125 | 125 | | nm.t.calc_slab_order | 0 | 124 | 124 | | nm.t.__free_slab | 0 | 123 | 123 | | nm.T.create_kmalloc_caches | 0 | 123 | 123 | | nm.D.kmalloc_caches | 0 | 112 | 112 | | nm.T.create_kmalloc_cache | 0 | 111 | 111 | | nm.T.create_boot_cache | 0 | 106 | 106 | | nm.t.bootstrap | 0 | 104 | 104 | | nm.T.kmem_cache_free | 35 | 130 | 95 | | nm.t.alloc_callchain_buffers | 0 | 92 | 92 | | nm.d.boot_kmem_cache | 0 | 88 | 88 | | nm.d.boot_kmem_cache_node | 0 | 88 | 88 | | nm.t.cmpxchg_double_slab.constprop | 0 | 80 | 80 | | nm.T.kmalloc_slab | 0 | 74 | 74 | | nm.T.__ksize | 60 | 127 | 67 | | nm.t.__cmpxchg_double_slab.constprop | 0 | 66 | 66 | | nm.t.slab_post_alloc_hook | 0 | 62 | 62 | | nm.t.new_kmalloc_cache | 0 | 61 | 61 | | nm.T.__kmem_cache_alias | 0 | 60 | 60 | | nm.t.flush_cpu_slab | 0 | 60 | 60 | | nm.t.slub_cpu_dead | 0 | 56 | 56 | | nm.T.free_large_kmalloc | 0 | 49 | 49 | | nm.t.alloc_slab_page | 0 | 48 | 48 | | nm.T.kmem_cache_create_usercopy | 359 | 404 | 45 | | nm.T.kmalloc_size_roundup | 16 | 60 | 44 | | nm.t.setup_slub_max_order | 0 | 43 | 43 | | nm.t.new_slab | 0 | 40 | 40 | | nm.t.slab_want_init_on_alloc | 0 | 40 | 40 | | nm.t.__flush_cpu_slab | 0 | 38 | 38 | | nm.t.slab_want_init_on_free | 0 | 37 | 37 | | nm.t.remove_partial | 0 | 30 | 30 | | nm.t.slab_pre_alloc_hook | 0 | 28 | 28 | | nm.t.free_kmem_cache_nodes | 0 | 25 | 25 | | nm.t.free_slab | 0 | 24 | 24 | | nm.d.size_index | 0 | 24 | 24 | | nm.d.slub_flush | 0 | 24 | 24 | | nm.t.setup_slub_min_objects | 0 | 23 | 23 | | nm.t.setup_slub_min_order | 0 | 23 | 23 | | nm.t.pfmemalloc_match | 0 | 19 | 19 | | nm.t.virt_to_slab | 0 | 19 | 19 | | nm.T.mempool_create_node | 86 | 105 | 19 | | nm.d.__setup_str_setup_slub_min_objects | 0 | 18 | 18 | | nm.T.find_mergeable | 135 | 153 | 18 | | nm.t.kzalloc | 10 | 27 | 17 | | nm.T.__kmem_cache_empty | 0 | 17 | 17 | | nm.T.__kmem_cache_release | 1 | 18 | 17 | | nm.d.__setup_str_setup_slub_max_order | 0 | 16 | 16 | | nm.d.__setup_str_setup_slub_min_order | 0 | 16 | 16 | | nm.T.percpu_ref_init | 164 | 180 | 16 | | nm.T.__kmem_cache_shrink | 3 | 18 | 15 | | nm.d.__setup_str_slub_nomerge | 0 | 13 | 13 | | nm.d.__setup_setup_slub_max_order | 0 | 12 | 12 | | nm.d.__setup_setup_slub_min_objects | 0 | 12 | 12 | | nm.d.__setup_setup_slub_min_order | 0 | 12 | 12 | | nm.d.__setup_slub_merge | 0 | 12 | 12 | | nm.d.__setup_slub_nomerge | 0 | 12 | 12 | | nm.d.flush_lock | 0 | 12 | 12 | | nm.T.kmem_cache_init_late | 11 | 23 | 12 | | nm.T.call_usermodehelper_setup | 86 | 98 | 12 | | nm.T.devres_open_group | 113 | 125 | 12 | | nm.T.dma_alloc_noncontiguous | 173 | 185 | 12 | | nm.d.__setup_str_slub_merge | 0 | 11 | 11 | | nm.t.rcu_free_slab | 0 | 11 | 11 | | nm.t.virt_to_folio | 19 | 29 | 10 | | nm.T.alloc_pipe_info | 255 | 260 | 5 | | nm.b.flushwq | 0 | 4 | 4 | | nm.b.kmem_cache_node | 0 | 4 | 4 | | nm.b.slab_nodes | 0 | 4 | 4 | | nm.d.slub_max_order | 0 | 4 | 4 | | nm.b.slub_min_objects | 0 | 4 | 4 | | nm.b.slub_min_order | 0 | 4 | 4 | | nm.T.fixup_red_left | 0 | 3 | 3 | | nm.T.kmem_cache_flags | 0 | 3 | 3 | | nm.T.setup_kmalloc_cache_index_table | 0 | 1 | 1 | | nm.T.kmem_cache_size | 3 | 4 | 1 | | nm.T.pipe_resize_ring | 242 | 243 | 1 | | nm.T.dma_pool_alloc | 265 | 264 | -1 | | nm.T.kobject_get_path | 115 | 112 | -3 | | nm.T.ida_alloc_range | 645 | 642 | -3 | | nm.d.free_slob_large | 8 | 0 | -8 | | nm.d.free_slob_medium | 8 | 0 | -8 | | nm.d.free_slob_small | 8 | 0 | -8 | | nm.t.kmem_rcu_free | 10 | 0 | -10 | | nm.t.slob_next | 32 | 0 | -32 | | nm.T.kfree | 81 | 47 | -34 | | nm.t.clear_slob_page_free | 35 | 0 | -35 | | nm.t.__kmem_cache_free | 36 | 0 | -36 | | nm.T.kmem_cache_free_bulk | 48 | 10 | -38 | | nm.t.set_slob | 40 | 0 | -40 | | nm.d.kmem_cache_boot | 44 | 0 | -44 | | nm.T.__kmalloc | 108 | 50 | -58 | | nm.T.__kmalloc_node | 112 | 53 | -59 | | nm.T.__kmalloc_node_track_caller | 112 | 50 | -62 | | nm.t.slob_new_pages | 62 | 0 | -62 | | nm.T.get_callchain_buffers | 163 | 90 | -73 | | nm.t.slob_free_pages | 80 | 0 | -80 | | nm.t.slob_alloc_node | 86 | 0 | -86 | | nm.t.kernel_init_freeable | 324 | 162 | -162 | | nm.t.property_entries_dup | 405 | 119 | -286 | | nm.t.slob_alloc | 358 | 0 | -358 | | nm.t.slob_page_alloc | 366 | 0 | -366 | | nm.t.slob_free | 471 | 0 | -471 | +-----------------------------------------+----------+----------+-------+ Thanks ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: [lkp] [+5395 bytes kernel size regression] [i386-tinyconfig] [b7c8731082] Deprecating and removing SLOB 2022-11-11 12:49 ` [lkp] [+5395 bytes kernel size regression] [i386-tinyconfig] [b7c8731082] " kernel test robot @ 2022-11-11 17:24 ` Josh Triplett 2022-11-11 21:11 ` Vlastimil Babka 2022-11-17 8:57 ` Vlastimil Babka 0 siblings, 2 replies; 112+ messages in thread From: Josh Triplett @ 2022-11-11 17:24 UTC (permalink / raw) To: Vlastimil Babka, oe-kbuild-all; +Cc: kernel test robot On Fri, Nov 11, 2022 at 08:49:57PM +0800, kernel test robot wrote: > FYI, we noticed a +5395 bytes kernel size regression due to commit: > > commit: b7c873108294f0fd19c7e4d6b038a05375b3cd33 (Deprecating and removing SLOB) > url: https://github.com/intel-lab-lkp/linux/commits/Vlastimil-Babka/mm-slob-rename-CONFIG_SLOB-to-CONFIG_SLOB_DEPRECATED/20221111-183529 > base: https://git.kernel.org/cgit/linux/kernel/git/akpm/mm.git mm-everything > patch subject: [PATCH] mm, slob: rename CONFIG_SLOB to CONFIG_SLOB_DEPRECATED > > > Details as below (size data is obtained by `nm --size-sort vmlinux`): > > e1e3de44: Merge branch 'mm-nonmm-unstable' into mm-everything > b7c87310: Deprecating and removing SLOB This patch has the net effect of switching tiny over from SLOB to SLUB, which causes a 5k+ size regression. I *absolutely* appreciate the value of removing this much code, and I'm not going to propose keeping SLOB for the sole reason of being 5k smaller; it's not worth keeping a whole allocator for that. I also wouldn't be surprised if we get *some* of that back by simplifications this change enables in the rest of the kernel. But I'm wondering if there might be any low-hanging fruit that could slim down SLUB to the tune of 5k or so, that has been previously ignored because SLOB served that use case? For instance, largely self-contained bits of SLUB functionality that could be omitted on the tiniest of systems? - Josh Triplett ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: [lkp] [+5395 bytes kernel size regression] [i386-tinyconfig] [b7c8731082] Deprecating and removing SLOB 2022-11-11 17:24 ` Josh Triplett @ 2022-11-11 21:11 ` Vlastimil Babka 2022-11-17 8:57 ` Vlastimil Babka 1 sibling, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-11 21:11 UTC (permalink / raw) To: Josh Triplett, oe-kbuild-all, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Roman Gushchin Cc: kernel test robot, linux-mm On 11/11/22 18:24, Josh Triplett wrote: > On Fri, Nov 11, 2022 at 08:49:57PM +0800, kernel test robot wrote: >> FYI, we noticed a +5395 bytes kernel size regression due to commit: >> >> commit: b7c873108294f0fd19c7e4d6b038a05375b3cd33 (Deprecating and removing SLOB) >> url: https://github.com/intel-lab-lkp/linux/commits/Vlastimil-Babka/mm-slob-rename-CONFIG_SLOB-to-CONFIG_SLOB_DEPRECATED/20221111-183529 >> base: https://git.kernel.org/cgit/linux/kernel/git/akpm/mm.git mm-everything >> patch subject: [PATCH] mm, slob: rename CONFIG_SLOB to CONFIG_SLOB_DEPRECATED >> >> >> Details as below (size data is obtained by `nm --size-sort vmlinux`): >> >> e1e3de44: Merge branch 'mm-nonmm-unstable' into mm-everything >> b7c87310: Deprecating and removing SLOB > > This patch has the net effect of switching tiny over from SLOB to SLUB, > which causes a 5k+ size regression. > > I *absolutely* appreciate the value of removing this much code, and I'm > not going to propose keeping SLOB for the sole reason of being 5k > smaller; it's not worth keeping a whole allocator for that. I also > wouldn't be surprised if we get *some* of that back by simplifications > this change enables in the rest of the kernel. +Cc more slab people and linux-mm, the lkp report is here: https://lore.kernel.org/all/Y25E9cJbhDAKi1vd@99bb1221be19/ Yeah, probably not immediately but if we succeed in removing SLAB as well, the mm/slab_common.c wrapper can go away which should reduce the code. > But I'm wondering if there might be any low-hanging fruit that could > slim down SLUB to the tune of 5k or so, that has been previously ignored > because SLOB served that use case? For instance, largely self-contained > bits of SLUB functionality that could be omitted on the tiniest of > systems? Maybe not the whole of 5k, but looking at the symbol sizes we could get something back with less aggressive inlining. Maybe add something like a SLUB_TINY config that sets the other configs correctly for minimal footprint and does tweaks like that to inlining or more code removal. Besides the code size, the less obvious but probably more important is how many slab pages it needs to allocate at runtime. E.g. for the configs, "make tinyconfig" seems to not set those like SLUB_DEBUG and SLUB_CPU_PARTIAL, which is good for making both code and page allocation smaller, but it also doesn't set SLAB_MERGE_DEFAULT which should result in fewer slab pages allocated if enabled. With SLUB_TINY we could also ditch completely percpu caches and associated code and just rely on the spin_lock protected partial list... > - Josh Triplett ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: [lkp] [+5395 bytes kernel size regression] [i386-tinyconfig] [b7c8731082] Deprecating and removing SLOB 2022-11-11 17:24 ` Josh Triplett 2022-11-11 21:11 ` Vlastimil Babka @ 2022-11-17 8:57 ` Vlastimil Babka 1 sibling, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-17 8:57 UTC (permalink / raw) To: Josh Triplett, oe-kbuild-all; +Cc: kernel test robot On 11/11/22 18:24, Josh Triplett wrote: > On Fri, Nov 11, 2022 at 08:49:57PM +0800, kernel test robot wrote: >> FYI, we noticed a +5395 bytes kernel size regression due to commit: >> >> commit: b7c873108294f0fd19c7e4d6b038a05375b3cd33 (Deprecating and removing SLOB) >> url: https://github.com/intel-lab-lkp/linux/commits/Vlastimil-Babka/mm-slob-rename-CONFIG_SLOB-to-CONFIG_SLOB_DEPRECATED/20221111-183529 >> base: https://git.kernel.org/cgit/linux/kernel/git/akpm/mm.git mm-everything >> patch subject: [PATCH] mm, slob: rename CONFIG_SLOB to CONFIG_SLOB_DEPRECATED >> >> >> Details as below (size data is obtained by `nm --size-sort vmlinux`): >> >> e1e3de44: Merge branch 'mm-nonmm-unstable' into mm-everything >> b7c87310: Deprecating and removing SLOB > > This patch has the net effect of switching tiny over from SLOB to SLUB, > which causes a 5k+ size regression. > > I *absolutely* appreciate the value of removing this much code, and I'm > not going to propose keeping SLOB for the sole reason of being 5k > smaller; it's not worth keeping a whole allocator for that. I also > wouldn't be surprised if we get *some* of that back by simplifications > this change enables in the rest of the kernel. > > But I'm wondering if there might be any low-hanging fruit that could > slim down SLUB to the tune of 5k or so, that has been previously ignored > because SLOB served that use case? For instance, largely self-contained > bits of SLUB functionality that could be omitted on the tiniest of > systems? Hi, could you test this one please https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r1 I've updated kernel/configs/tiny.config there to include CONFIG_SLUB_TINY so should be picked up. > - Josh Triplett ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-11 10:33 ` Vlastimil Babka (?) (?) @ 2022-11-11 20:46 ` Conor Dooley -1 siblings, 0 replies; 112+ messages in thread From: Conor Dooley @ 2022-11-11 20:46 UTC (permalink / raw) To: Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Damien Le Moal, Paul Cercueil On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > On 11/8/22 22:44, Pasha Tatashin wrote: > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >> > >> Hi, > >> > >> as we all know, we currently have three slab allocators. As we discussed > >> at LPC [1], it is my hope that one of these allocators has a future, and > >> two of them do not. > >> > >> The unsurprising reasons include code maintenance burden, other features > >> compatible with only a subset of allocators (or more effort spent on the > >> features), blocking API improvements (more on that below), and my > >> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >> without resorting to spelling out the letters. > >> > >> I think (but may be proven wrong) that SLOB is the easier target of the > >> two to be removed, so I'd like to focus on it first. > >> > >> I believe SLOB can be removed because: > >> > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > >> by putting all objects together, which has its CPU performance costs > >> (locking, lack of percpu caching, searching for free space...). I'm not > >> aware of any "tiny linux" deployment that opts for this. For example, > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >> SLOB impact is too much for those who tried. Googling for > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > I am all for removing SLOB. > > > > There are some devices with configs where SLOB is enabled by default. > > Perhaps, the owners/maintainers of those devices/configs should be > > included into this thread: > > > > tatashin@soleen:~/x/linux$ git grep SLOB=y > > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > > Turns out that since SLOB depends on EXPERT, many of those lack it so > running make defconfig ends up with SLUB anyway, unless I miss something. > Only a subset has both SLOB and EXPERT: > > > git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y I suppose there's not really a concern with the virt defconfig, but I did check the output of `make nommu_k210_defconfig" and despite not having expert it seems to end up CONFIG_SLOB=y in the generated .config. I do have a board with a k210 so I checked with s/SLOB/SLUB and it still boots etc, but I have no workloads or w/e to run on it. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-11 20:46 ` Conor Dooley 0 siblings, 0 replies; 112+ messages in thread From: Conor Dooley @ 2022-11-11 20:46 UTC (permalink / raw) To: Vlastimil Babka Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Damien Le Moal, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > On 11/8/22 22:44, Pasha Tatashin wrote: > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >> > >> Hi, > >> > >> as we all know, we currently have three slab allocators. As we discussed > >> at LPC [1], it is my hope that one of these allocators has a future, and > >> two of them do not. > >> > >> The unsurprising reasons include code maintenance burden, other features > >> compatible with only a subset of allocators (or more effort spent on the > >> features), blocking API improvements (more on that below), and my > >> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >> without resorting to spelling out the letters. > >> > >> I think (but may be proven wrong) that SLOB is the easier target of the > >> two to be removed, so I'd like to focus on it first. > >> > >> I believe SLOB can be removed because: > >> > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > >> by putting all objects together, which has its CPU performance costs > >> (locking, lack of percpu caching, searching for free space...). I'm not > >> aware of any "tiny linux" deployment that opts for this. For example, > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >> SLOB impact is too much for those who tried. Googling for > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > I am all for removing SLOB. > > > > There are some devices with configs where SLOB is enabled by default. > > Perhaps, the owners/maintainers of those devices/configs should be > > included into this thread: > > > > tatashin@soleen:~/x/linux$ git grep SLOB=y > > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > > Turns out that since SLOB depends on EXPERT, many of those lack it so > running make defconfig ends up with SLUB anyway, unless I miss something. > Only a subset has both SLOB and EXPERT: > > > git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y I suppose there's not really a concern with the virt defconfig, but I did check the output of `make nommu_k210_defconfig" and despite not having expert it seems to end up CONFIG_SLOB=y in the generated .config. I do have a board with a k210 so I checked with s/SLOB/SLUB and it still boots etc, but I have no workloads or w/e to run on it. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-11 20:46 ` Conor Dooley 0 siblings, 0 replies; 112+ messages in thread From: Conor Dooley @ 2022-11-11 20:46 UTC (permalink / raw) To: Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Damien Le Moal, Paul Cercueil On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > On 11/8/22 22:44, Pasha Tatashin wrote: > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >> > >> Hi, > >> > >> as we all know, we currently have three slab allocators. As we discussed > >> at LPC [1], it is my hope that one of these allocators has a future, and > >> two of them do not. > >> > >> The unsurprising reasons include code maintenance burden, other features > >> compatible with only a subset of allocators (or more effort spent on the > >> features), blocking API improvements (more on that below), and my > >> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >> without resorting to spelling out the letters. > >> > >> I think (but may be proven wrong) that SLOB is the easier target of the > >> two to be removed, so I'd like to focus on it first. > >> > >> I believe SLOB can be removed because: > >> > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > >> by putting all objects together, which has its CPU performance costs > >> (locking, lack of percpu caching, searching for free space...). I'm not > >> aware of any "tiny linux" deployment that opts for this. For example, > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >> SLOB impact is too much for those who tried. Googling for > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > I am all for removing SLOB. > > > > There are some devices with configs where SLOB is enabled by default. > > Perhaps, the owners/maintainers of those devices/configs should be > > included into this thread: > > > > tatashin@soleen:~/x/linux$ git grep SLOB=y > > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > > Turns out that since SLOB depends on EXPERT, many of those lack it so > running make defconfig ends up with SLUB anyway, unless I miss something. > Only a subset has both SLOB and EXPERT: > > > git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y I suppose there's not really a concern with the virt defconfig, but I did check the output of `make nommu_k210_defconfig" and despite not having expert it seems to end up CONFIG_SLOB=y in the generated .config. I do have a board with a k210 so I checked with s/SLOB/SLUB and it still boots etc, but I have no workloads or w/e to run on it. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-11 20:46 ` Conor Dooley 0 siblings, 0 replies; 112+ messages in thread From: Conor Dooley @ 2022-11-11 20:46 UTC (permalink / raw) To: Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Damien Le Moal, Paul Cercueil On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > On 11/8/22 22:44, Pasha Tatashin wrote: > > On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >> > >> Hi, > >> > >> as we all know, we currently have three slab allocators. As we discussed > >> at LPC [1], it is my hope that one of these allocators has a future, and > >> two of them do not. > >> > >> The unsurprising reasons include code maintenance burden, other features > >> compatible with only a subset of allocators (or more effort spent on the > >> features), blocking API improvements (more on that below), and my > >> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >> without resorting to spelling out the letters. > >> > >> I think (but may be proven wrong) that SLOB is the easier target of the > >> two to be removed, so I'd like to focus on it first. > >> > >> I believe SLOB can be removed because: > >> > >> - AFAIK nobody really uses it? It strives for minimal memory footprint > >> by putting all objects together, which has its CPU performance costs > >> (locking, lack of percpu caching, searching for free space...). I'm not > >> aware of any "tiny linux" deployment that opts for this. For example, > >> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >> SLOB impact is too much for those who tried. Googling for > >> "CONFIG_SLOB=y" yielded nothing useful. > > > > I am all for removing SLOB. > > > > There are some devices with configs where SLOB is enabled by default. > > Perhaps, the owners/maintainers of those devices/configs should be > > included into this thread: > > > > tatashin@soleen:~/x/linux$ git grep SLOB=y > > arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > > arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > > arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > > Turns out that since SLOB depends on EXPERT, many of those lack it so > running make defconfig ends up with SLUB anyway, unless I miss something. > Only a subset has both SLOB and EXPERT: > > > git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y I suppose there's not really a concern with the virt defconfig, but I did check the output of `make nommu_k210_defconfig" and despite not having expert it seems to end up CONFIG_SLOB=y in the generated .config. I do have a board with a k210 so I checked with s/SLOB/SLUB and it still boots etc, but I have no workloads or w/e to run on it. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-11 20:46 ` Conor Dooley (?) (?) @ 2022-11-12 1:40 ` Damien Le Moal -1 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-12 1:40 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/12/22 05:46, Conor Dooley wrote: > On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >> On 11/8/22 22:44, Pasha Tatashin wrote: >>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>> >>>> Hi, >>>> >>>> as we all know, we currently have three slab allocators. As we discussed >>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>> two of them do not. >>>> >>>> The unsurprising reasons include code maintenance burden, other features >>>> compatible with only a subset of allocators (or more effort spent on the >>>> features), blocking API improvements (more on that below), and my >>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>> without resorting to spelling out the letters. >>>> >>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>> two to be removed, so I'd like to focus on it first. >>>> >>>> I believe SLOB can be removed because: >>>> >>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>> by putting all objects together, which has its CPU performance costs >>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>> aware of any "tiny linux" deployment that opts for this. For example, >>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>> SLOB impact is too much for those who tried. Googling for >>>> "CONFIG_SLOB=y" yielded nothing useful. >>> >>> I am all for removing SLOB. >>> >>> There are some devices with configs where SLOB is enabled by default. >>> Perhaps, the owners/maintainers of those devices/configs should be >>> included into this thread: >>> >>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >> >> Turns out that since SLOB depends on EXPERT, many of those lack it so >> running make defconfig ends up with SLUB anyway, unless I miss something. >> Only a subset has both SLOB and EXPERT: >> >>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > > I suppose there's not really a concern with the virt defconfig, but I > did check the output of `make nommu_k210_defconfig" and despite not > having expert it seems to end up CONFIG_SLOB=y in the generated .config. > > I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > boots etc, but I have no workloads or w/e to run on it. I will try with SLUB over the weekend. -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-12 1:40 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-12 1:40 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 11/12/22 05:46, Conor Dooley wrote: > On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >> On 11/8/22 22:44, Pasha Tatashin wrote: >>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>> >>>> Hi, >>>> >>>> as we all know, we currently have three slab allocators. As we discussed >>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>> two of them do not. >>>> >>>> The unsurprising reasons include code maintenance burden, other features >>>> compatible with only a subset of allocators (or more effort spent on the >>>> features), blocking API improvements (more on that below), and my >>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>> without resorting to spelling out the letters. >>>> >>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>> two to be removed, so I'd like to focus on it first. >>>> >>>> I believe SLOB can be removed because: >>>> >>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>> by putting all objects together, which has its CPU performance costs >>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>> aware of any "tiny linux" deployment that opts for this. For example, >>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>> SLOB impact is too much for those who tried. Googling for >>>> "CONFIG_SLOB=y" yielded nothing useful. >>> >>> I am all for removing SLOB. >>> >>> There are some devices with configs where SLOB is enabled by default. >>> Perhaps, the owners/maintainers of those devices/configs should be >>> included into this thread: >>> >>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >> >> Turns out that since SLOB depends on EXPERT, many of those lack it so >> running make defconfig ends up with SLUB anyway, unless I miss something. >> Only a subset has both SLOB and EXPERT: >> >>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > > I suppose there's not really a concern with the virt defconfig, but I > did check the output of `make nommu_k210_defconfig" and despite not > having expert it seems to end up CONFIG_SLOB=y in the generated .config. > > I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > boots etc, but I have no workloads or w/e to run on it. I will try with SLUB over the weekend. -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-12 1:40 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-12 1:40 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/12/22 05:46, Conor Dooley wrote: > On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >> On 11/8/22 22:44, Pasha Tatashin wrote: >>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>> >>>> Hi, >>>> >>>> as we all know, we currently have three slab allocators. As we discussed >>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>> two of them do not. >>>> >>>> The unsurprising reasons include code maintenance burden, other features >>>> compatible with only a subset of allocators (or more effort spent on the >>>> features), blocking API improvements (more on that below), and my >>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>> without resorting to spelling out the letters. >>>> >>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>> two to be removed, so I'd like to focus on it first. >>>> >>>> I believe SLOB can be removed because: >>>> >>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>> by putting all objects together, which has its CPU performance costs >>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>> aware of any "tiny linux" deployment that opts for this. For example, >>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>> SLOB impact is too much for those who tried. Googling for >>>> "CONFIG_SLOB=y" yielded nothing useful. >>> >>> I am all for removing SLOB. >>> >>> There are some devices with configs where SLOB is enabled by default. >>> Perhaps, the owners/maintainers of those devices/configs should be >>> included into this thread: >>> >>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >> >> Turns out that since SLOB depends on EXPERT, many of those lack it so >> running make defconfig ends up with SLUB anyway, unless I miss something. >> Only a subset has both SLOB and EXPERT: >> >>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > > I suppose there's not really a concern with the virt defconfig, but I > did check the output of `make nommu_k210_defconfig" and despite not > having expert it seems to end up CONFIG_SLOB=y in the generated .config. > > I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > boots etc, but I have no workloads or w/e to run on it. I will try with SLUB over the weekend. -- Damien Le Moal Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-12 1:40 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-12 1:40 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/12/22 05:46, Conor Dooley wrote: > On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >> On 11/8/22 22:44, Pasha Tatashin wrote: >>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>> >>>> Hi, >>>> >>>> as we all know, we currently have three slab allocators. As we discussed >>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>> two of them do not. >>>> >>>> The unsurprising reasons include code maintenance burden, other features >>>> compatible with only a subset of allocators (or more effort spent on the >>>> features), blocking API improvements (more on that below), and my >>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>> without resorting to spelling out the letters. >>>> >>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>> two to be removed, so I'd like to focus on it first. >>>> >>>> I believe SLOB can be removed because: >>>> >>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>> by putting all objects together, which has its CPU performance costs >>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>> aware of any "tiny linux" deployment that opts for this. For example, >>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>> SLOB impact is too much for those who tried. Googling for >>>> "CONFIG_SLOB=y" yielded nothing useful. >>> >>> I am all for removing SLOB. >>> >>> There are some devices with configs where SLOB is enabled by default. >>> Perhaps, the owners/maintainers of those devices/configs should be >>> included into this thread: >>> >>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >> >> Turns out that since SLOB depends on EXPERT, many of those lack it so >> running make defconfig ends up with SLUB anyway, unless I miss something. >> Only a subset has both SLOB and EXPERT: >> >>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > > I suppose there's not really a concern with the virt defconfig, but I > did check the output of `make nommu_k210_defconfig" and despite not > having expert it seems to end up CONFIG_SLOB=y in the generated .config. > > I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > boots etc, but I have no workloads or w/e to run on it. I will try with SLUB over the weekend. -- Damien Le Moal Western Digital Research _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-11 20:46 ` Conor Dooley (?) (?) @ 2022-11-14 1:55 ` Damien Le Moal -1 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 1:55 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/12/22 05:46, Conor Dooley wrote: > On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >> On 11/8/22 22:44, Pasha Tatashin wrote: >>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>> >>>> Hi, >>>> >>>> as we all know, we currently have three slab allocators. As we discussed >>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>> two of them do not. >>>> >>>> The unsurprising reasons include code maintenance burden, other features >>>> compatible with only a subset of allocators (or more effort spent on the >>>> features), blocking API improvements (more on that below), and my >>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>> without resorting to spelling out the letters. >>>> >>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>> two to be removed, so I'd like to focus on it first. >>>> >>>> I believe SLOB can be removed because: >>>> >>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>> by putting all objects together, which has its CPU performance costs >>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>> aware of any "tiny linux" deployment that opts for this. For example, >>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>> SLOB impact is too much for those who tried. Googling for >>>> "CONFIG_SLOB=y" yielded nothing useful. >>> >>> I am all for removing SLOB. >>> >>> There are some devices with configs where SLOB is enabled by default. >>> Perhaps, the owners/maintainers of those devices/configs should be >>> included into this thread: >>> >>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >> >> Turns out that since SLOB depends on EXPERT, many of those lack it so >> running make defconfig ends up with SLUB anyway, unless I miss something. >> Only a subset has both SLOB and EXPERT: >> >>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > > I suppose there's not really a concern with the virt defconfig, but I > did check the output of `make nommu_k210_defconfig" and despite not > having expert it seems to end up CONFIG_SLOB=y in the generated .config. > > I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > boots etc, but I have no workloads or w/e to run on it. I sent a patch to change the k210 defconfig to using SLUB. However... The current default config using SLOB gives about 630 free memory pages after boot (cat /proc/vmstat). Switching to SLUB, this is down to about 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). This is with a buildroot kernel 5.19 build including a shell and sd-card boot. With SLUB, I get clean boots and a shell prompt as expected. But I definitely see more errors with shell commands failing due to allocation failures for the shell process fork. So as far as the K210 is concerned, switching to SLUB is not ideal. I would not want to hold on kernel mm improvements because of this toy k210 though, so I am not going to prevent SLOB deprecation. I just wish SLUB itself used less memory :) -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 1:55 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 1:55 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/12/22 05:46, Conor Dooley wrote: > On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >> On 11/8/22 22:44, Pasha Tatashin wrote: >>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>> >>>> Hi, >>>> >>>> as we all know, we currently have three slab allocators. As we discussed >>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>> two of them do not. >>>> >>>> The unsurprising reasons include code maintenance burden, other features >>>> compatible with only a subset of allocators (or more effort spent on the >>>> features), blocking API improvements (more on that below), and my >>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>> without resorting to spelling out the letters. >>>> >>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>> two to be removed, so I'd like to focus on it first. >>>> >>>> I believe SLOB can be removed because: >>>> >>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>> by putting all objects together, which has its CPU performance costs >>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>> aware of any "tiny linux" deployment that opts for this. For example, >>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>> SLOB impact is too much for those who tried. Googling for >>>> "CONFIG_SLOB=y" yielded nothing useful. >>> >>> I am all for removing SLOB. >>> >>> There are some devices with configs where SLOB is enabled by default. >>> Perhaps, the owners/maintainers of those devices/configs should be >>> included into this thread: >>> >>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >> >> Turns out that since SLOB depends on EXPERT, many of those lack it so >> running make defconfig ends up with SLUB anyway, unless I miss something. >> Only a subset has both SLOB and EXPERT: >> >>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > > I suppose there's not really a concern with the virt defconfig, but I > did check the output of `make nommu_k210_defconfig" and despite not > having expert it seems to end up CONFIG_SLOB=y in the generated .config. > > I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > boots etc, but I have no workloads or w/e to run on it. I sent a patch to change the k210 defconfig to using SLUB. However... The current default config using SLOB gives about 630 free memory pages after boot (cat /proc/vmstat). Switching to SLUB, this is down to about 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). This is with a buildroot kernel 5.19 build including a shell and sd-card boot. With SLUB, I get clean boots and a shell prompt as expected. But I definitely see more errors with shell commands failing due to allocation failures for the shell process fork. So as far as the K210 is concerned, switching to SLUB is not ideal. I would not want to hold on kernel mm improvements because of this toy k210 though, so I am not going to prevent SLOB deprecation. I just wish SLUB itself used less memory :) -- Damien Le Moal Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 1:55 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 1:55 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 11/12/22 05:46, Conor Dooley wrote: > On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >> On 11/8/22 22:44, Pasha Tatashin wrote: >>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>> >>>> Hi, >>>> >>>> as we all know, we currently have three slab allocators. As we discussed >>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>> two of them do not. >>>> >>>> The unsurprising reasons include code maintenance burden, other features >>>> compatible with only a subset of allocators (or more effort spent on the >>>> features), blocking API improvements (more on that below), and my >>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>> without resorting to spelling out the letters. >>>> >>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>> two to be removed, so I'd like to focus on it first. >>>> >>>> I believe SLOB can be removed because: >>>> >>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>> by putting all objects together, which has its CPU performance costs >>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>> aware of any "tiny linux" deployment that opts for this. For example, >>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>> SLOB impact is too much for those who tried. Googling for >>>> "CONFIG_SLOB=y" yielded nothing useful. >>> >>> I am all for removing SLOB. >>> >>> There are some devices with configs where SLOB is enabled by default. >>> Perhaps, the owners/maintainers of those devices/configs should be >>> included into this thread: >>> >>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >> >> Turns out that since SLOB depends on EXPERT, many of those lack it so >> running make defconfig ends up with SLUB anyway, unless I miss something. >> Only a subset has both SLOB and EXPERT: >> >>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > > I suppose there's not really a concern with the virt defconfig, but I > did check the output of `make nommu_k210_defconfig" and despite not > having expert it seems to end up CONFIG_SLOB=y in the generated .config. > > I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > boots etc, but I have no workloads or w/e to run on it. I sent a patch to change the k210 defconfig to using SLUB. However... The current default config using SLOB gives about 630 free memory pages after boot (cat /proc/vmstat). Switching to SLUB, this is down to about 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). This is with a buildroot kernel 5.19 build including a shell and sd-card boot. With SLUB, I get clean boots and a shell prompt as expected. But I definitely see more errors with shell commands failing due to allocation failures for the shell process fork. So as far as the K210 is concerned, switching to SLUB is not ideal. I would not want to hold on kernel mm improvements because of this toy k210 though, so I am not going to prevent SLOB deprecation. I just wish SLUB itself used less memory :) -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 1:55 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 1:55 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/12/22 05:46, Conor Dooley wrote: > On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >> On 11/8/22 22:44, Pasha Tatashin wrote: >>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>> >>>> Hi, >>>> >>>> as we all know, we currently have three slab allocators. As we discussed >>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>> two of them do not. >>>> >>>> The unsurprising reasons include code maintenance burden, other features >>>> compatible with only a subset of allocators (or more effort spent on the >>>> features), blocking API improvements (more on that below), and my >>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>> without resorting to spelling out the letters. >>>> >>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>> two to be removed, so I'd like to focus on it first. >>>> >>>> I believe SLOB can be removed because: >>>> >>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>> by putting all objects together, which has its CPU performance costs >>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>> aware of any "tiny linux" deployment that opts for this. For example, >>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>> SLOB impact is too much for those who tried. Googling for >>>> "CONFIG_SLOB=y" yielded nothing useful. >>> >>> I am all for removing SLOB. >>> >>> There are some devices with configs where SLOB is enabled by default. >>> Perhaps, the owners/maintainers of those devices/configs should be >>> included into this thread: >>> >>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >> >> Turns out that since SLOB depends on EXPERT, many of those lack it so >> running make defconfig ends up with SLUB anyway, unless I miss something. >> Only a subset has both SLOB and EXPERT: >> >>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > > I suppose there's not really a concern with the virt defconfig, but I > did check the output of `make nommu_k210_defconfig" and despite not > having expert it seems to end up CONFIG_SLOB=y in the generated .config. > > I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > boots etc, but I have no workloads or w/e to run on it. I sent a patch to change the k210 defconfig to using SLUB. However... The current default config using SLOB gives about 630 free memory pages after boot (cat /proc/vmstat). Switching to SLUB, this is down to about 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). This is with a buildroot kernel 5.19 build including a shell and sd-card boot. With SLUB, I get clean boots and a shell prompt as expected. But I definitely see more errors with shell commands failing due to allocation failures for the shell process fork. So as far as the K210 is concerned, switching to SLUB is not ideal. I would not want to hold on kernel mm improvements because of this toy k210 though, so I am not going to prevent SLOB deprecation. I just wish SLUB itself used less memory :) -- Damien Le Moal Western Digital Research _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-14 1:55 ` Damien Le Moal (?) (?) @ 2022-11-14 5:48 ` Damien Le Moal -1 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 5:48 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 10:55, Damien Le Moal wrote: > On 11/12/22 05:46, Conor Dooley wrote: >> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>> >>>>> Hi, >>>>> >>>>> as we all know, we currently have three slab allocators. As we discussed >>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>> two of them do not. >>>>> >>>>> The unsurprising reasons include code maintenance burden, other features >>>>> compatible with only a subset of allocators (or more effort spent on the >>>>> features), blocking API improvements (more on that below), and my >>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>> without resorting to spelling out the letters. >>>>> >>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>> two to be removed, so I'd like to focus on it first. >>>>> >>>>> I believe SLOB can be removed because: >>>>> >>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>> by putting all objects together, which has its CPU performance costs >>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>> SLOB impact is too much for those who tried. Googling for >>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>> >>>> I am all for removing SLOB. >>>> >>>> There are some devices with configs where SLOB is enabled by default. >>>> Perhaps, the owners/maintainers of those devices/configs should be >>>> included into this thread: >>>> >>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >> >>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >> >>> >>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>> running make defconfig ends up with SLUB anyway, unless I miss something. >>> Only a subset has both SLOB and EXPERT: >>> >>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >> >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >> >> I suppose there's not really a concern with the virt defconfig, but I >> did check the output of `make nommu_k210_defconfig" and despite not >> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >> >> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >> boots etc, but I have no workloads or w/e to run on it. > > I sent a patch to change the k210 defconfig to using SLUB. However... > > The current default config using SLOB gives about 630 free memory pages > after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > This is with a buildroot kernel 5.19 build including a shell and sd-card > boot. With SLUB, I get clean boots and a shell prompt as expected. But I > definitely see more errors with shell commands failing due to allocation > failures for the shell process fork. So as far as the K210 is concerned, > switching to SLUB is not ideal. > > I would not want to hold on kernel mm improvements because of this toy > k210 though, so I am not going to prevent SLOB deprecation. I just wish > SLUB itself used less memory :) Did further tests with kernel 6.0.1: * SLOB: 630 free pages after boot, shell working (occasional shell fork failure happen though) * SLAB: getting memory allocation for order 7 failures on boot already (init process). Shell barely working (high frequency of shell command fork failures) * SLUB: getting memory allocation for order 7 failures on boot. I do get a shell prompt but cannot run any shell command that involves forking a new process. So if we want to keep the k210 support functional with a shell, we need slob. If we reduce that board support to only one application started as the init process, then I guess anything is OK. -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 5:48 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 5:48 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 10:55, Damien Le Moal wrote: > On 11/12/22 05:46, Conor Dooley wrote: >> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>> >>>>> Hi, >>>>> >>>>> as we all know, we currently have three slab allocators. As we discussed >>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>> two of them do not. >>>>> >>>>> The unsurprising reasons include code maintenance burden, other features >>>>> compatible with only a subset of allocators (or more effort spent on the >>>>> features), blocking API improvements (more on that below), and my >>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>> without resorting to spelling out the letters. >>>>> >>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>> two to be removed, so I'd like to focus on it first. >>>>> >>>>> I believe SLOB can be removed because: >>>>> >>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>> by putting all objects together, which has its CPU performance costs >>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>> SLOB impact is too much for those who tried. Googling for >>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>> >>>> I am all for removing SLOB. >>>> >>>> There are some devices with configs where SLOB is enabled by default. >>>> Perhaps, the owners/maintainers of those devices/configs should be >>>> included into this thread: >>>> >>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >> >>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >> >>> >>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>> running make defconfig ends up with SLUB anyway, unless I miss something. >>> Only a subset has both SLOB and EXPERT: >>> >>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >> >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >> >> I suppose there's not really a concern with the virt defconfig, but I >> did check the output of `make nommu_k210_defconfig" and despite not >> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >> >> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >> boots etc, but I have no workloads or w/e to run on it. > > I sent a patch to change the k210 defconfig to using SLUB. However... > > The current default config using SLOB gives about 630 free memory pages > after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > This is with a buildroot kernel 5.19 build including a shell and sd-card > boot. With SLUB, I get clean boots and a shell prompt as expected. But I > definitely see more errors with shell commands failing due to allocation > failures for the shell process fork. So as far as the K210 is concerned, > switching to SLUB is not ideal. > > I would not want to hold on kernel mm improvements because of this toy > k210 though, so I am not going to prevent SLOB deprecation. I just wish > SLUB itself used less memory :) Did further tests with kernel 6.0.1: * SLOB: 630 free pages after boot, shell working (occasional shell fork failure happen though) * SLAB: getting memory allocation for order 7 failures on boot already (init process). Shell barely working (high frequency of shell command fork failures) * SLUB: getting memory allocation for order 7 failures on boot. I do get a shell prompt but cannot run any shell command that involves forking a new process. So if we want to keep the k210 support functional with a shell, we need slob. If we reduce that board support to only one application started as the init process, then I guess anything is OK. -- Damien Le Moal Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 5:48 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 5:48 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 11/14/22 10:55, Damien Le Moal wrote: > On 11/12/22 05:46, Conor Dooley wrote: >> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>> >>>>> Hi, >>>>> >>>>> as we all know, we currently have three slab allocators. As we discussed >>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>> two of them do not. >>>>> >>>>> The unsurprising reasons include code maintenance burden, other features >>>>> compatible with only a subset of allocators (or more effort spent on the >>>>> features), blocking API improvements (more on that below), and my >>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>> without resorting to spelling out the letters. >>>>> >>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>> two to be removed, so I'd like to focus on it first. >>>>> >>>>> I believe SLOB can be removed because: >>>>> >>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>> by putting all objects together, which has its CPU performance costs >>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>> SLOB impact is too much for those who tried. Googling for >>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>> >>>> I am all for removing SLOB. >>>> >>>> There are some devices with configs where SLOB is enabled by default. >>>> Perhaps, the owners/maintainers of those devices/configs should be >>>> included into this thread: >>>> >>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >> >>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >> >>> >>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>> running make defconfig ends up with SLUB anyway, unless I miss something. >>> Only a subset has both SLOB and EXPERT: >>> >>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >> >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >> >> I suppose there's not really a concern with the virt defconfig, but I >> did check the output of `make nommu_k210_defconfig" and despite not >> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >> >> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >> boots etc, but I have no workloads or w/e to run on it. > > I sent a patch to change the k210 defconfig to using SLUB. However... > > The current default config using SLOB gives about 630 free memory pages > after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > This is with a buildroot kernel 5.19 build including a shell and sd-card > boot. With SLUB, I get clean boots and a shell prompt as expected. But I > definitely see more errors with shell commands failing due to allocation > failures for the shell process fork. So as far as the K210 is concerned, > switching to SLUB is not ideal. > > I would not want to hold on kernel mm improvements because of this toy > k210 though, so I am not going to prevent SLOB deprecation. I just wish > SLUB itself used less memory :) Did further tests with kernel 6.0.1: * SLOB: 630 free pages after boot, shell working (occasional shell fork failure happen though) * SLAB: getting memory allocation for order 7 failures on boot already (init process). Shell barely working (high frequency of shell command fork failures) * SLUB: getting memory allocation for order 7 failures on boot. I do get a shell prompt but cannot run any shell command that involves forking a new process. So if we want to keep the k210 support functional with a shell, we need slob. If we reduce that board support to only one application started as the init process, then I guess anything is OK. -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 5:48 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 5:48 UTC (permalink / raw) To: Conor Dooley, Vlastimil Babka Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 10:55, Damien Le Moal wrote: > On 11/12/22 05:46, Conor Dooley wrote: >> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>> >>>>> Hi, >>>>> >>>>> as we all know, we currently have three slab allocators. As we discussed >>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>> two of them do not. >>>>> >>>>> The unsurprising reasons include code maintenance burden, other features >>>>> compatible with only a subset of allocators (or more effort spent on the >>>>> features), blocking API improvements (more on that below), and my >>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>> without resorting to spelling out the letters. >>>>> >>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>> two to be removed, so I'd like to focus on it first. >>>>> >>>>> I believe SLOB can be removed because: >>>>> >>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>> by putting all objects together, which has its CPU performance costs >>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>> SLOB impact is too much for those who tried. Googling for >>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>> >>>> I am all for removing SLOB. >>>> >>>> There are some devices with configs where SLOB is enabled by default. >>>> Perhaps, the owners/maintainers of those devices/configs should be >>>> included into this thread: >>>> >>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >> >>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >> >>> >>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>> running make defconfig ends up with SLUB anyway, unless I miss something. >>> Only a subset has both SLOB and EXPERT: >>> >>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >> >>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >> >> I suppose there's not really a concern with the virt defconfig, but I >> did check the output of `make nommu_k210_defconfig" and despite not >> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >> >> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >> boots etc, but I have no workloads or w/e to run on it. > > I sent a patch to change the k210 defconfig to using SLUB. However... > > The current default config using SLOB gives about 630 free memory pages > after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > This is with a buildroot kernel 5.19 build including a shell and sd-card > boot. With SLUB, I get clean boots and a shell prompt as expected. But I > definitely see more errors with shell commands failing due to allocation > failures for the shell process fork. So as far as the K210 is concerned, > switching to SLUB is not ideal. > > I would not want to hold on kernel mm improvements because of this toy > k210 though, so I am not going to prevent SLOB deprecation. I just wish > SLUB itself used less memory :) Did further tests with kernel 6.0.1: * SLOB: 630 free pages after boot, shell working (occasional shell fork failure happen though) * SLAB: getting memory allocation for order 7 failures on boot already (init process). Shell barely working (high frequency of shell command fork failures) * SLUB: getting memory allocation for order 7 failures on boot. I do get a shell prompt but cannot run any shell command that involves forking a new process. So if we want to keep the k210 support functional with a shell, we need slob. If we reduce that board support to only one application started as the init process, then I guess anything is OK. -- Damien Le Moal Western Digital Research _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-14 5:48 ` Damien Le Moal (?) (?) @ 2022-11-14 9:36 ` Vlastimil Babka -1 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-14 9:36 UTC (permalink / raw) To: Damien Le Moal, Conor Dooley Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 06:48, Damien Le Moal wrote: > On 11/14/22 10:55, Damien Le Moal wrote: >> On 11/12/22 05:46, Conor Dooley wrote: >>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>> two of them do not. >>>>>> >>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>> features), blocking API improvements (more on that below), and my >>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>> without resorting to spelling out the letters. >>>>>> >>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>> two to be removed, so I'd like to focus on it first. >>>>>> >>>>>> I believe SLOB can be removed because: >>>>>> >>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>> by putting all objects together, which has its CPU performance costs >>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>> SLOB impact is too much for those who tried. Googling for >>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>> >>>>> I am all for removing SLOB. >>>>> >>>>> There are some devices with configs where SLOB is enabled by default. >>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>> included into this thread: >>>>> >>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>> >>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>> >>>> >>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>> Only a subset has both SLOB and EXPERT: >>>> >>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>> >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>> >>> I suppose there's not really a concern with the virt defconfig, but I >>> did check the output of `make nommu_k210_defconfig" and despite not >>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>> >>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>> boots etc, but I have no workloads or w/e to run on it. >> >> I sent a patch to change the k210 defconfig to using SLUB. However... Thanks! >> The current default config using SLOB gives about 630 free memory pages >> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). Thanks for the testing! How much RAM does the system have btw? I found 8MB somewhere, is that correct? So 230 pages that's a ~920 kB difference. Last time we saw less dramatic difference [1]. But that was looking at Slab pages, not free pages. The extra overhead could be also in percpu allocations, code etc. >> This is with a buildroot kernel 5.19 build including a shell and sd-card >> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >> definitely see more errors with shell commands failing due to allocation >> failures for the shell process fork. So as far as the K210 is concerned, >> switching to SLUB is not ideal. >> >> I would not want to hold on kernel mm improvements because of this toy >> k210 though, so I am not going to prevent SLOB deprecation. I just wish >> SLUB itself used less memory :) > > Did further tests with kernel 6.0.1: > * SLOB: 630 free pages after boot, shell working (occasional shell fork > failure happen though) > * SLAB: getting memory allocation for order 7 failures on boot already > (init process). Shell barely working (high frequency of shell command fork > failures) > * SLUB: getting memory allocation for order 7 failures on boot. I do get a > shell prompt but cannot run any shell command that involves forking a new > process. > > So if we want to keep the k210 support functional with a shell, we need > slob. If we reduce that board support to only one application started as > the init process, then I guess anything is OK. In [1] it was possible to save some more memory with more tuning. Some of that required boot parameters and other code changes. In another reply [2] I considered adding something like SLUB_TINY to take care of all that, so looks like it would make sense to proceed with that. [1] https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 9:36 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-14 9:36 UTC (permalink / raw) To: Damien Le Moal, Conor Dooley Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 06:48, Damien Le Moal wrote: > On 11/14/22 10:55, Damien Le Moal wrote: >> On 11/12/22 05:46, Conor Dooley wrote: >>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>> two of them do not. >>>>>> >>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>> features), blocking API improvements (more on that below), and my >>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>> without resorting to spelling out the letters. >>>>>> >>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>> two to be removed, so I'd like to focus on it first. >>>>>> >>>>>> I believe SLOB can be removed because: >>>>>> >>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>> by putting all objects together, which has its CPU performance costs >>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>> SLOB impact is too much for those who tried. Googling for >>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>> >>>>> I am all for removing SLOB. >>>>> >>>>> There are some devices with configs where SLOB is enabled by default. >>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>> included into this thread: >>>>> >>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>> >>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>> >>>> >>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>> Only a subset has both SLOB and EXPERT: >>>> >>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>> >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>> >>> I suppose there's not really a concern with the virt defconfig, but I >>> did check the output of `make nommu_k210_defconfig" and despite not >>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>> >>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>> boots etc, but I have no workloads or w/e to run on it. >> >> I sent a patch to change the k210 defconfig to using SLUB. However... Thanks! >> The current default config using SLOB gives about 630 free memory pages >> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). Thanks for the testing! How much RAM does the system have btw? I found 8MB somewhere, is that correct? So 230 pages that's a ~920 kB difference. Last time we saw less dramatic difference [1]. But that was looking at Slab pages, not free pages. The extra overhead could be also in percpu allocations, code etc. >> This is with a buildroot kernel 5.19 build including a shell and sd-card >> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >> definitely see more errors with shell commands failing due to allocation >> failures for the shell process fork. So as far as the K210 is concerned, >> switching to SLUB is not ideal. >> >> I would not want to hold on kernel mm improvements because of this toy >> k210 though, so I am not going to prevent SLOB deprecation. I just wish >> SLUB itself used less memory :) > > Did further tests with kernel 6.0.1: > * SLOB: 630 free pages after boot, shell working (occasional shell fork > failure happen though) > * SLAB: getting memory allocation for order 7 failures on boot already > (init process). Shell barely working (high frequency of shell command fork > failures) > * SLUB: getting memory allocation for order 7 failures on boot. I do get a > shell prompt but cannot run any shell command that involves forking a new > process. > > So if we want to keep the k210 support functional with a shell, we need > slob. If we reduce that board support to only one application started as > the init process, then I guess anything is OK. In [1] it was possible to save some more memory with more tuning. Some of that required boot parameters and other code changes. In another reply [2] I considered adding something like SLUB_TINY to take care of all that, so looks like it would make sense to proceed with that. [1] https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 9:36 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-14 9:36 UTC (permalink / raw) To: Damien Le Moal, Conor Dooley Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 11/14/22 06:48, Damien Le Moal wrote: > On 11/14/22 10:55, Damien Le Moal wrote: >> On 11/12/22 05:46, Conor Dooley wrote: >>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>> two of them do not. >>>>>> >>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>> features), blocking API improvements (more on that below), and my >>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>> without resorting to spelling out the letters. >>>>>> >>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>> two to be removed, so I'd like to focus on it first. >>>>>> >>>>>> I believe SLOB can be removed because: >>>>>> >>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>> by putting all objects together, which has its CPU performance costs >>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>> SLOB impact is too much for those who tried. Googling for >>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>> >>>>> I am all for removing SLOB. >>>>> >>>>> There are some devices with configs where SLOB is enabled by default. >>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>> included into this thread: >>>>> >>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>> >>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>> >>>> >>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>> Only a subset has both SLOB and EXPERT: >>>> >>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>> >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>> >>> I suppose there's not really a concern with the virt defconfig, but I >>> did check the output of `make nommu_k210_defconfig" and despite not >>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>> >>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>> boots etc, but I have no workloads or w/e to run on it. >> >> I sent a patch to change the k210 defconfig to using SLUB. However... Thanks! >> The current default config using SLOB gives about 630 free memory pages >> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). Thanks for the testing! How much RAM does the system have btw? I found 8MB somewhere, is that correct? So 230 pages that's a ~920 kB difference. Last time we saw less dramatic difference [1]. But that was looking at Slab pages, not free pages. The extra overhead could be also in percpu allocations, code etc. >> This is with a buildroot kernel 5.19 build including a shell and sd-card >> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >> definitely see more errors with shell commands failing due to allocation >> failures for the shell process fork. So as far as the K210 is concerned, >> switching to SLUB is not ideal. >> >> I would not want to hold on kernel mm improvements because of this toy >> k210 though, so I am not going to prevent SLOB deprecation. I just wish >> SLUB itself used less memory :) > > Did further tests with kernel 6.0.1: > * SLOB: 630 free pages after boot, shell working (occasional shell fork > failure happen though) > * SLAB: getting memory allocation for order 7 failures on boot already > (init process). Shell barely working (high frequency of shell command fork > failures) > * SLUB: getting memory allocation for order 7 failures on boot. I do get a > shell prompt but cannot run any shell command that involves forking a new > process. > > So if we want to keep the k210 support functional with a shell, we need > slob. If we reduce that board support to only one application started as > the init process, then I guess anything is OK. In [1] it was possible to save some more memory with more tuning. Some of that required boot parameters and other code changes. In another reply [2] I considered adding something like SLUB_TINY to take care of all that, so looks like it would make sense to proceed with that. [1] https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 9:36 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-14 9:36 UTC (permalink / raw) To: Damien Le Moal, Conor Dooley Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 06:48, Damien Le Moal wrote: > On 11/14/22 10:55, Damien Le Moal wrote: >> On 11/12/22 05:46, Conor Dooley wrote: >>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>> two of them do not. >>>>>> >>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>> features), blocking API improvements (more on that below), and my >>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>> without resorting to spelling out the letters. >>>>>> >>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>> two to be removed, so I'd like to focus on it first. >>>>>> >>>>>> I believe SLOB can be removed because: >>>>>> >>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>> by putting all objects together, which has its CPU performance costs >>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>> SLOB impact is too much for those who tried. Googling for >>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>> >>>>> I am all for removing SLOB. >>>>> >>>>> There are some devices with configs where SLOB is enabled by default. >>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>> included into this thread: >>>>> >>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>> >>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>> >>>> >>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>> Only a subset has both SLOB and EXPERT: >>>> >>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>> >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>> >>> I suppose there's not really a concern with the virt defconfig, but I >>> did check the output of `make nommu_k210_defconfig" and despite not >>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>> >>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>> boots etc, but I have no workloads or w/e to run on it. >> >> I sent a patch to change the k210 defconfig to using SLUB. However... Thanks! >> The current default config using SLOB gives about 630 free memory pages >> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). Thanks for the testing! How much RAM does the system have btw? I found 8MB somewhere, is that correct? So 230 pages that's a ~920 kB difference. Last time we saw less dramatic difference [1]. But that was looking at Slab pages, not free pages. The extra overhead could be also in percpu allocations, code etc. >> This is with a buildroot kernel 5.19 build including a shell and sd-card >> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >> definitely see more errors with shell commands failing due to allocation >> failures for the shell process fork. So as far as the K210 is concerned, >> switching to SLUB is not ideal. >> >> I would not want to hold on kernel mm improvements because of this toy >> k210 though, so I am not going to prevent SLOB deprecation. I just wish >> SLUB itself used less memory :) > > Did further tests with kernel 6.0.1: > * SLOB: 630 free pages after boot, shell working (occasional shell fork > failure happen though) > * SLAB: getting memory allocation for order 7 failures on boot already > (init process). Shell barely working (high frequency of shell command fork > failures) > * SLUB: getting memory allocation for order 7 failures on boot. I do get a > shell prompt but cannot run any shell command that involves forking a new > process. > > So if we want to keep the k210 support functional with a shell, we need > slob. If we reduce that board support to only one application started as > the init process, then I guess anything is OK. In [1] it was possible to save some more memory with more tuning. Some of that required boot parameters and other code changes. In another reply [2] I considered adding something like SLUB_TINY to take care of all that, so looks like it would make sense to proceed with that. [1] https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-14 9:36 ` Vlastimil Babka (?) (?) @ 2022-11-14 11:35 ` Damien Le Moal -1 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 11:35 UTC (permalink / raw) To: Vlastimil Babka, Conor Dooley Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 18:36, Vlastimil Babka wrote: > On 11/14/22 06:48, Damien Le Moal wrote: >> On 11/14/22 10:55, Damien Le Moal wrote: >>> On 11/12/22 05:46, Conor Dooley wrote: >>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>> two of them do not. >>>>>>> >>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>> features), blocking API improvements (more on that below), and my >>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>> without resorting to spelling out the letters. >>>>>>> >>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>> >>>>>>> I believe SLOB can be removed because: >>>>>>> >>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>> >>>>>> I am all for removing SLOB. >>>>>> >>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>> included into this thread: >>>>>> >>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>> >>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>> >>>>> >>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>> Only a subset has both SLOB and EXPERT: >>>>> >>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>> >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>> >>>> I suppose there's not really a concern with the virt defconfig, but I >>>> did check the output of `make nommu_k210_defconfig" and despite not >>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>> >>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>> boots etc, but I have no workloads or w/e to run on it. >>> >>> I sent a patch to change the k210 defconfig to using SLUB. However... > > Thanks! > >>> The current default config using SLOB gives about 630 free memory pages >>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > somewhere, is that correct? Yep, 8MB, that's it. > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > difference [1]. But that was looking at Slab pages, not free pages. The > extra overhead could be also in percpu allocations, code etc. > >>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>> definitely see more errors with shell commands failing due to allocation >>> failures for the shell process fork. So as far as the K210 is concerned, >>> switching to SLUB is not ideal. >>> >>> I would not want to hold on kernel mm improvements because of this toy >>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>> SLUB itself used less memory :) >> >> Did further tests with kernel 6.0.1: >> * SLOB: 630 free pages after boot, shell working (occasional shell fork >> failure happen though) >> * SLAB: getting memory allocation for order 7 failures on boot already >> (init process). Shell barely working (high frequency of shell command fork >> failures) I forgot to add here that the system was down to about 500 free pages after boot (again from the shell with "cat /proc/vmstat"). >> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >> shell prompt but cannot run any shell command that involves forking a new >> process. For both slab and slub, I had cpu partial off, debug off and slab merge on, as I suspected that would lead to less memory overhead. I suspected memory fragmentation may be an issue but doing echo 3 > /proc/sys/vm/drop_caches before trying a shell command did not help much at all (it usually does on that board with SLOB). Note that this is all with buildroot, so this echo & redirect always works as it does not cause a shell fork. >> >> So if we want to keep the k210 support functional with a shell, we need >> slob. If we reduce that board support to only one application started as >> the init process, then I guess anything is OK. > > In [1] it was possible to save some more memory with more tuning. Some of > that required boot parameters and other code changes. In another reply [2] I > considered adding something like SLUB_TINY to take care of all that, so > looks like it would make sense to proceed with that. If you want me to test something, let me know. > > [1] > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 11:35 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 11:35 UTC (permalink / raw) To: Vlastimil Babka, Conor Dooley Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 18:36, Vlastimil Babka wrote: > On 11/14/22 06:48, Damien Le Moal wrote: >> On 11/14/22 10:55, Damien Le Moal wrote: >>> On 11/12/22 05:46, Conor Dooley wrote: >>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>> two of them do not. >>>>>>> >>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>> features), blocking API improvements (more on that below), and my >>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>> without resorting to spelling out the letters. >>>>>>> >>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>> >>>>>>> I believe SLOB can be removed because: >>>>>>> >>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>> >>>>>> I am all for removing SLOB. >>>>>> >>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>> included into this thread: >>>>>> >>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>> >>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>> >>>>> >>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>> Only a subset has both SLOB and EXPERT: >>>>> >>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>> >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>> >>>> I suppose there's not really a concern with the virt defconfig, but I >>>> did check the output of `make nommu_k210_defconfig" and despite not >>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>> >>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>> boots etc, but I have no workloads or w/e to run on it. >>> >>> I sent a patch to change the k210 defconfig to using SLUB. However... > > Thanks! > >>> The current default config using SLOB gives about 630 free memory pages >>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > somewhere, is that correct? Yep, 8MB, that's it. > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > difference [1]. But that was looking at Slab pages, not free pages. The > extra overhead could be also in percpu allocations, code etc. > >>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>> definitely see more errors with shell commands failing due to allocation >>> failures for the shell process fork. So as far as the K210 is concerned, >>> switching to SLUB is not ideal. >>> >>> I would not want to hold on kernel mm improvements because of this toy >>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>> SLUB itself used less memory :) >> >> Did further tests with kernel 6.0.1: >> * SLOB: 630 free pages after boot, shell working (occasional shell fork >> failure happen though) >> * SLAB: getting memory allocation for order 7 failures on boot already >> (init process). Shell barely working (high frequency of shell command fork >> failures) I forgot to add here that the system was down to about 500 free pages after boot (again from the shell with "cat /proc/vmstat"). >> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >> shell prompt but cannot run any shell command that involves forking a new >> process. For both slab and slub, I had cpu partial off, debug off and slab merge on, as I suspected that would lead to less memory overhead. I suspected memory fragmentation may be an issue but doing echo 3 > /proc/sys/vm/drop_caches before trying a shell command did not help much at all (it usually does on that board with SLOB). Note that this is all with buildroot, so this echo & redirect always works as it does not cause a shell fork. >> >> So if we want to keep the k210 support functional with a shell, we need >> slob. If we reduce that board support to only one application started as >> the init process, then I guess anything is OK. > > In [1] it was possible to save some more memory with more tuning. Some of > that required boot parameters and other code changes. In another reply [2] I > considered adding something like SLUB_TINY to take care of all that, so > looks like it would make sense to proceed with that. If you want me to test something, let me know. > > [1] > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ -- Damien Le Moal Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 11:35 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 11:35 UTC (permalink / raw) To: Vlastimil Babka, Conor Dooley Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 11/14/22 18:36, Vlastimil Babka wrote: > On 11/14/22 06:48, Damien Le Moal wrote: >> On 11/14/22 10:55, Damien Le Moal wrote: >>> On 11/12/22 05:46, Conor Dooley wrote: >>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>> two of them do not. >>>>>>> >>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>> features), blocking API improvements (more on that below), and my >>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>> without resorting to spelling out the letters. >>>>>>> >>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>> >>>>>>> I believe SLOB can be removed because: >>>>>>> >>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>> >>>>>> I am all for removing SLOB. >>>>>> >>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>> included into this thread: >>>>>> >>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>> >>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>> >>>>> >>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>> Only a subset has both SLOB and EXPERT: >>>>> >>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>> >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>> >>>> I suppose there's not really a concern with the virt defconfig, but I >>>> did check the output of `make nommu_k210_defconfig" and despite not >>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>> >>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>> boots etc, but I have no workloads or w/e to run on it. >>> >>> I sent a patch to change the k210 defconfig to using SLUB. However... > > Thanks! > >>> The current default config using SLOB gives about 630 free memory pages >>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > somewhere, is that correct? Yep, 8MB, that's it. > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > difference [1]. But that was looking at Slab pages, not free pages. The > extra overhead could be also in percpu allocations, code etc. > >>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>> definitely see more errors with shell commands failing due to allocation >>> failures for the shell process fork. So as far as the K210 is concerned, >>> switching to SLUB is not ideal. >>> >>> I would not want to hold on kernel mm improvements because of this toy >>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>> SLUB itself used less memory :) >> >> Did further tests with kernel 6.0.1: >> * SLOB: 630 free pages after boot, shell working (occasional shell fork >> failure happen though) >> * SLAB: getting memory allocation for order 7 failures on boot already >> (init process). Shell barely working (high frequency of shell command fork >> failures) I forgot to add here that the system was down to about 500 free pages after boot (again from the shell with "cat /proc/vmstat"). >> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >> shell prompt but cannot run any shell command that involves forking a new >> process. For both slab and slub, I had cpu partial off, debug off and slab merge on, as I suspected that would lead to less memory overhead. I suspected memory fragmentation may be an issue but doing echo 3 > /proc/sys/vm/drop_caches before trying a shell command did not help much at all (it usually does on that board with SLOB). Note that this is all with buildroot, so this echo & redirect always works as it does not cause a shell fork. >> >> So if we want to keep the k210 support functional with a shell, we need >> slob. If we reduce that board support to only one application started as >> the init process, then I guess anything is OK. > > In [1] it was possible to save some more memory with more tuning. Some of > that required boot parameters and other code changes. In another reply [2] I > considered adding something like SLUB_TINY to take care of all that, so > looks like it would make sense to proceed with that. If you want me to test something, let me know. > > [1] > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 11:35 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-14 11:35 UTC (permalink / raw) To: Vlastimil Babka, Conor Dooley Cc: Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Hyeonggon Yoo, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 18:36, Vlastimil Babka wrote: > On 11/14/22 06:48, Damien Le Moal wrote: >> On 11/14/22 10:55, Damien Le Moal wrote: >>> On 11/12/22 05:46, Conor Dooley wrote: >>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>> two of them do not. >>>>>>> >>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>> features), blocking API improvements (more on that below), and my >>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>> without resorting to spelling out the letters. >>>>>>> >>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>> >>>>>>> I believe SLOB can be removed because: >>>>>>> >>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>> >>>>>> I am all for removing SLOB. >>>>>> >>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>> included into this thread: >>>>>> >>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>> >>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>> >>>>> >>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>> Only a subset has both SLOB and EXPERT: >>>>> >>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>> >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>> >>>> I suppose there's not really a concern with the virt defconfig, but I >>>> did check the output of `make nommu_k210_defconfig" and despite not >>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>> >>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>> boots etc, but I have no workloads or w/e to run on it. >>> >>> I sent a patch to change the k210 defconfig to using SLUB. However... > > Thanks! > >>> The current default config using SLOB gives about 630 free memory pages >>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > somewhere, is that correct? Yep, 8MB, that's it. > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > difference [1]. But that was looking at Slab pages, not free pages. The > extra overhead could be also in percpu allocations, code etc. > >>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>> definitely see more errors with shell commands failing due to allocation >>> failures for the shell process fork. So as far as the K210 is concerned, >>> switching to SLUB is not ideal. >>> >>> I would not want to hold on kernel mm improvements because of this toy >>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>> SLUB itself used less memory :) >> >> Did further tests with kernel 6.0.1: >> * SLOB: 630 free pages after boot, shell working (occasional shell fork >> failure happen though) >> * SLAB: getting memory allocation for order 7 failures on boot already >> (init process). Shell barely working (high frequency of shell command fork >> failures) I forgot to add here that the system was down to about 500 free pages after boot (again from the shell with "cat /proc/vmstat"). >> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >> shell prompt but cannot run any shell command that involves forking a new >> process. For both slab and slub, I had cpu partial off, debug off and slab merge on, as I suspected that would lead to less memory overhead. I suspected memory fragmentation may be an issue but doing echo 3 > /proc/sys/vm/drop_caches before trying a shell command did not help much at all (it usually does on that board with SLOB). Note that this is all with buildroot, so this echo & redirect always works as it does not cause a shell fork. >> >> So if we want to keep the k210 support functional with a shell, we need >> slob. If we reduce that board support to only one application started as >> the init process, then I guess anything is OK. > > In [1] it was possible to save some more memory with more tuning. Some of > that required boot parameters and other code changes. In another reply [2] I > considered adding something like SLUB_TINY to take care of all that, so > looks like it would make sense to proceed with that. If you want me to test something, let me know. > > [1] > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ -- Damien Le Moal Western Digital Research _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-14 11:35 ` Damien Le Moal (?) (?) @ 2022-11-14 14:47 ` Hyeonggon Yoo -1 siblings, 0 replies; 112+ messages in thread From: Hyeonggon Yoo @ 2022-11-14 14:47 UTC (permalink / raw) To: Damien Le Moal Cc: Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: > On 11/14/22 18:36, Vlastimil Babka wrote: > > On 11/14/22 06:48, Damien Le Moal wrote: > >> On 11/14/22 10:55, Damien Le Moal wrote: > >>> On 11/12/22 05:46, Conor Dooley wrote: > >>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > >>>>> On 11/8/22 22:44, Pasha Tatashin wrote: > >>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> as we all know, we currently have three slab allocators. As we discussed > >>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and > >>>>>>> two of them do not. > >>>>>>> > >>>>>>> The unsurprising reasons include code maintenance burden, other features > >>>>>>> compatible with only a subset of allocators (or more effort spent on the > >>>>>>> features), blocking API improvements (more on that below), and my > >>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >>>>>>> without resorting to spelling out the letters. > >>>>>>> > >>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the > >>>>>>> two to be removed, so I'd like to focus on it first. > >>>>>>> > >>>>>>> I believe SLOB can be removed because: > >>>>>>> > >>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint > >>>>>>> by putting all objects together, which has its CPU performance costs > >>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not > >>>>>>> aware of any "tiny linux" deployment that opts for this. For example, > >>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >>>>>>> SLOB impact is too much for those who tried. Googling for > >>>>>>> "CONFIG_SLOB=y" yielded nothing useful. > >>>>>> > >>>>>> I am all for removing SLOB. > >>>>>> > >>>>>> There are some devices with configs where SLOB is enabled by default. > >>>>>> Perhaps, the owners/maintainers of those devices/configs should be > >>>>>> included into this thread: > >>>>>> > >>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>>> > >>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > >>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > >>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >>>> > >>>>> > >>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so > >>>>> running make defconfig ends up with SLUB anyway, unless I miss something. > >>>>> Only a subset has both SLOB and EXPERT: > >>>>> > >>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >>>> > >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > >>>> > >>>> I suppose there's not really a concern with the virt defconfig, but I > >>>> did check the output of `make nommu_k210_defconfig" and despite not > >>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. > >>>> > >>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > >>>> boots etc, but I have no workloads or w/e to run on it. > >>> > >>> I sent a patch to change the k210 defconfig to using SLUB. However... > > > > Thanks! > > > >>> The current default config using SLOB gives about 630 free memory pages > >>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > >>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > > somewhere, is that correct? > > Yep, 8MB, that's it. > > > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > > difference [1]. But that was looking at Slab pages, not free pages. The > > extra overhead could be also in percpu allocations, code etc. > > > >>> This is with a buildroot kernel 5.19 build including a shell and sd-card > >>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I > >>> definitely see more errors with shell commands failing due to allocation > >>> failures for the shell process fork. So as far as the K210 is concerned, > >>> switching to SLUB is not ideal. > >>> > >>> I would not want to hold on kernel mm improvements because of this toy > >>> k210 though, so I am not going to prevent SLOB deprecation. I just wish > >>> SLUB itself used less memory :) > >> > >> Did further tests with kernel 6.0.1: > >> * SLOB: 630 free pages after boot, shell working (occasional shell fork > >> failure happen though) > >> * SLAB: getting memory allocation for order 7 failures on boot already > >> (init process). Shell barely working (high frequency of shell command fork > >> failures) > > I forgot to add here that the system was down to about 500 free pages > after boot (again from the shell with "cat /proc/vmstat"). > > >> * SLUB: getting memory allocation for order 7 failures on boot. I do get a > >> shell prompt but cannot run any shell command that involves forking a new > >> process. > > For both slab and slub, I had cpu partial off, debug off and slab merge > on, as I suspected that would lead to less memory overhead. > I suspected memory fragmentation may be an issue but doing > > echo 3 > /proc/sys/vm/drop_caches > > before trying a shell command did not help much at all (it usually does on > that board with SLOB). Note that this is all with buildroot, so this echo > & redirect always works as it does not cause a shell fork. > > >> > >> So if we want to keep the k210 support functional with a shell, we need > >> slob. If we reduce that board support to only one application started as > >> the init process, then I guess anything is OK. > > > > In [1] it was possible to save some more memory with more tuning. Some of > > that required boot parameters and other code changes. In another reply [2] I > > considered adding something like SLUB_TINY to take care of all that, so > > looks like it would make sense to proceed with that. > > If you want me to test something, let me know. Would you try this please? diff --git a/mm/slub.c b/mm/slub.c index a24b71041b26..1c36c4b9aaa0 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) * The larger the object size is, the more slabs we want on the partial * list to avoid pounding the page allocator excessively. */ - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); - + s->min_partial = 0; set_cpu_partial(s); #ifdef CONFIG_NUMA and booting with and without boot parameter slub_max_order=0? Thanks, Hyeonggon > > > > > [1] > > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ > > -- > Damien Le Moal > Western Digital Research ^ permalink raw reply related [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 14:47 ` Hyeonggon Yoo 0 siblings, 0 replies; 112+ messages in thread From: Hyeonggon Yoo @ 2022-11-14 14:47 UTC (permalink / raw) To: Damien Le Moal Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Christoph Lameter, linux-riscv, Jonas Bonn, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, linux-arm-kernel, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, Vlastimil Babka, Yoshinori Sato, linux-kernel, Conor Dooley, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: > On 11/14/22 18:36, Vlastimil Babka wrote: > > On 11/14/22 06:48, Damien Le Moal wrote: > >> On 11/14/22 10:55, Damien Le Moal wrote: > >>> On 11/12/22 05:46, Conor Dooley wrote: > >>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > >>>>> On 11/8/22 22:44, Pasha Tatashin wrote: > >>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> as we all know, we currently have three slab allocators. As we discussed > >>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and > >>>>>>> two of them do not. > >>>>>>> > >>>>>>> The unsurprising reasons include code maintenance burden, other features > >>>>>>> compatible with only a subset of allocators (or more effort spent on the > >>>>>>> features), blocking API improvements (more on that below), and my > >>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >>>>>>> without resorting to spelling out the letters. > >>>>>>> > >>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the > >>>>>>> two to be removed, so I'd like to focus on it first. > >>>>>>> > >>>>>>> I believe SLOB can be removed because: > >>>>>>> > >>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint > >>>>>>> by putting all objects together, which has its CPU performance costs > >>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not > >>>>>>> aware of any "tiny linux" deployment that opts for this. For example, > >>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >>>>>>> SLOB impact is too much for those who tried. Googling for > >>>>>>> "CONFIG_SLOB=y" yielded nothing useful. > >>>>>> > >>>>>> I am all for removing SLOB. > >>>>>> > >>>>>> There are some devices with configs where SLOB is enabled by default. > >>>>>> Perhaps, the owners/maintainers of those devices/configs should be > >>>>>> included into this thread: > >>>>>> > >>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>>> > >>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > >>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > >>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >>>> > >>>>> > >>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so > >>>>> running make defconfig ends up with SLUB anyway, unless I miss something. > >>>>> Only a subset has both SLOB and EXPERT: > >>>>> > >>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >>>> > >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > >>>> > >>>> I suppose there's not really a concern with the virt defconfig, but I > >>>> did check the output of `make nommu_k210_defconfig" and despite not > >>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. > >>>> > >>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > >>>> boots etc, but I have no workloads or w/e to run on it. > >>> > >>> I sent a patch to change the k210 defconfig to using SLUB. However... > > > > Thanks! > > > >>> The current default config using SLOB gives about 630 free memory pages > >>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > >>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > > somewhere, is that correct? > > Yep, 8MB, that's it. > > > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > > difference [1]. But that was looking at Slab pages, not free pages. The > > extra overhead could be also in percpu allocations, code etc. > > > >>> This is with a buildroot kernel 5.19 build including a shell and sd-card > >>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I > >>> definitely see more errors with shell commands failing due to allocation > >>> failures for the shell process fork. So as far as the K210 is concerned, > >>> switching to SLUB is not ideal. > >>> > >>> I would not want to hold on kernel mm improvements because of this toy > >>> k210 though, so I am not going to prevent SLOB deprecation. I just wish > >>> SLUB itself used less memory :) > >> > >> Did further tests with kernel 6.0.1: > >> * SLOB: 630 free pages after boot, shell working (occasional shell fork > >> failure happen though) > >> * SLAB: getting memory allocation for order 7 failures on boot already > >> (init process). Shell barely working (high frequency of shell command fork > >> failures) > > I forgot to add here that the system was down to about 500 free pages > after boot (again from the shell with "cat /proc/vmstat"). > > >> * SLUB: getting memory allocation for order 7 failures on boot. I do get a > >> shell prompt but cannot run any shell command that involves forking a new > >> process. > > For both slab and slub, I had cpu partial off, debug off and slab merge > on, as I suspected that would lead to less memory overhead. > I suspected memory fragmentation may be an issue but doing > > echo 3 > /proc/sys/vm/drop_caches > > before trying a shell command did not help much at all (it usually does on > that board with SLOB). Note that this is all with buildroot, so this echo > & redirect always works as it does not cause a shell fork. > > >> > >> So if we want to keep the k210 support functional with a shell, we need > >> slob. If we reduce that board support to only one application started as > >> the init process, then I guess anything is OK. > > > > In [1] it was possible to save some more memory with more tuning. Some of > > that required boot parameters and other code changes. In another reply [2] I > > considered adding something like SLUB_TINY to take care of all that, so > > looks like it would make sense to proceed with that. > > If you want me to test something, let me know. Would you try this please? diff --git a/mm/slub.c b/mm/slub.c index a24b71041b26..1c36c4b9aaa0 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) * The larger the object size is, the more slabs we want on the partial * list to avoid pounding the page allocator excessively. */ - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); - + s->min_partial = 0; set_cpu_partial(s); #ifdef CONFIG_NUMA and booting with and without boot parameter slub_max_order=0? Thanks, Hyeonggon > > > > > [1] > > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ > > -- > Damien Le Moal > Western Digital Research ^ permalink raw reply related [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 14:47 ` Hyeonggon Yoo 0 siblings, 0 replies; 112+ messages in thread From: Hyeonggon Yoo @ 2022-11-14 14:47 UTC (permalink / raw) To: Damien Le Moal Cc: Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: > On 11/14/22 18:36, Vlastimil Babka wrote: > > On 11/14/22 06:48, Damien Le Moal wrote: > >> On 11/14/22 10:55, Damien Le Moal wrote: > >>> On 11/12/22 05:46, Conor Dooley wrote: > >>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > >>>>> On 11/8/22 22:44, Pasha Tatashin wrote: > >>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> as we all know, we currently have three slab allocators. As we discussed > >>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and > >>>>>>> two of them do not. > >>>>>>> > >>>>>>> The unsurprising reasons include code maintenance burden, other features > >>>>>>> compatible with only a subset of allocators (or more effort spent on the > >>>>>>> features), blocking API improvements (more on that below), and my > >>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >>>>>>> without resorting to spelling out the letters. > >>>>>>> > >>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the > >>>>>>> two to be removed, so I'd like to focus on it first. > >>>>>>> > >>>>>>> I believe SLOB can be removed because: > >>>>>>> > >>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint > >>>>>>> by putting all objects together, which has its CPU performance costs > >>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not > >>>>>>> aware of any "tiny linux" deployment that opts for this. For example, > >>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >>>>>>> SLOB impact is too much for those who tried. Googling for > >>>>>>> "CONFIG_SLOB=y" yielded nothing useful. > >>>>>> > >>>>>> I am all for removing SLOB. > >>>>>> > >>>>>> There are some devices with configs where SLOB is enabled by default. > >>>>>> Perhaps, the owners/maintainers of those devices/configs should be > >>>>>> included into this thread: > >>>>>> > >>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>>> > >>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > >>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > >>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >>>> > >>>>> > >>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so > >>>>> running make defconfig ends up with SLUB anyway, unless I miss something. > >>>>> Only a subset has both SLOB and EXPERT: > >>>>> > >>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >>>> > >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > >>>> > >>>> I suppose there's not really a concern with the virt defconfig, but I > >>>> did check the output of `make nommu_k210_defconfig" and despite not > >>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. > >>>> > >>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > >>>> boots etc, but I have no workloads or w/e to run on it. > >>> > >>> I sent a patch to change the k210 defconfig to using SLUB. However... > > > > Thanks! > > > >>> The current default config using SLOB gives about 630 free memory pages > >>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > >>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > > somewhere, is that correct? > > Yep, 8MB, that's it. > > > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > > difference [1]. But that was looking at Slab pages, not free pages. The > > extra overhead could be also in percpu allocations, code etc. > > > >>> This is with a buildroot kernel 5.19 build including a shell and sd-card > >>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I > >>> definitely see more errors with shell commands failing due to allocation > >>> failures for the shell process fork. So as far as the K210 is concerned, > >>> switching to SLUB is not ideal. > >>> > >>> I would not want to hold on kernel mm improvements because of this toy > >>> k210 though, so I am not going to prevent SLOB deprecation. I just wish > >>> SLUB itself used less memory :) > >> > >> Did further tests with kernel 6.0.1: > >> * SLOB: 630 free pages after boot, shell working (occasional shell fork > >> failure happen though) > >> * SLAB: getting memory allocation for order 7 failures on boot already > >> (init process). Shell barely working (high frequency of shell command fork > >> failures) > > I forgot to add here that the system was down to about 500 free pages > after boot (again from the shell with "cat /proc/vmstat"). > > >> * SLUB: getting memory allocation for order 7 failures on boot. I do get a > >> shell prompt but cannot run any shell command that involves forking a new > >> process. > > For both slab and slub, I had cpu partial off, debug off and slab merge > on, as I suspected that would lead to less memory overhead. > I suspected memory fragmentation may be an issue but doing > > echo 3 > /proc/sys/vm/drop_caches > > before trying a shell command did not help much at all (it usually does on > that board with SLOB). Note that this is all with buildroot, so this echo > & redirect always works as it does not cause a shell fork. > > >> > >> So if we want to keep the k210 support functional with a shell, we need > >> slob. If we reduce that board support to only one application started as > >> the init process, then I guess anything is OK. > > > > In [1] it was possible to save some more memory with more tuning. Some of > > that required boot parameters and other code changes. In another reply [2] I > > considered adding something like SLUB_TINY to take care of all that, so > > looks like it would make sense to proceed with that. > > If you want me to test something, let me know. Would you try this please? diff --git a/mm/slub.c b/mm/slub.c index a24b71041b26..1c36c4b9aaa0 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) * The larger the object size is, the more slabs we want on the partial * list to avoid pounding the page allocator excessively. */ - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); - + s->min_partial = 0; set_cpu_partial(s); #ifdef CONFIG_NUMA and booting with and without boot parameter slub_max_order=0? Thanks, Hyeonggon > > > > > [1] > > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ > > -- > Damien Le Moal > Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 14:47 ` Hyeonggon Yoo 0 siblings, 0 replies; 112+ messages in thread From: Hyeonggon Yoo @ 2022-11-14 14:47 UTC (permalink / raw) To: Damien Le Moal Cc: Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: > On 11/14/22 18:36, Vlastimil Babka wrote: > > On 11/14/22 06:48, Damien Le Moal wrote: > >> On 11/14/22 10:55, Damien Le Moal wrote: > >>> On 11/12/22 05:46, Conor Dooley wrote: > >>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > >>>>> On 11/8/22 22:44, Pasha Tatashin wrote: > >>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> as we all know, we currently have three slab allocators. As we discussed > >>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and > >>>>>>> two of them do not. > >>>>>>> > >>>>>>> The unsurprising reasons include code maintenance burden, other features > >>>>>>> compatible with only a subset of allocators (or more effort spent on the > >>>>>>> features), blocking API improvements (more on that below), and my > >>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >>>>>>> without resorting to spelling out the letters. > >>>>>>> > >>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the > >>>>>>> two to be removed, so I'd like to focus on it first. > >>>>>>> > >>>>>>> I believe SLOB can be removed because: > >>>>>>> > >>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint > >>>>>>> by putting all objects together, which has its CPU performance costs > >>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not > >>>>>>> aware of any "tiny linux" deployment that opts for this. For example, > >>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >>>>>>> SLOB impact is too much for those who tried. Googling for > >>>>>>> "CONFIG_SLOB=y" yielded nothing useful. > >>>>>> > >>>>>> I am all for removing SLOB. > >>>>>> > >>>>>> There are some devices with configs where SLOB is enabled by default. > >>>>>> Perhaps, the owners/maintainers of those devices/configs should be > >>>>>> included into this thread: > >>>>>> > >>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>>> > >>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > >>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > >>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >>>> > >>>>> > >>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so > >>>>> running make defconfig ends up with SLUB anyway, unless I miss something. > >>>>> Only a subset has both SLOB and EXPERT: > >>>>> > >>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >>>> > >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > >>>> > >>>> I suppose there's not really a concern with the virt defconfig, but I > >>>> did check the output of `make nommu_k210_defconfig" and despite not > >>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. > >>>> > >>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > >>>> boots etc, but I have no workloads or w/e to run on it. > >>> > >>> I sent a patch to change the k210 defconfig to using SLUB. However... > > > > Thanks! > > > >>> The current default config using SLOB gives about 630 free memory pages > >>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > >>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > > somewhere, is that correct? > > Yep, 8MB, that's it. > > > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > > difference [1]. But that was looking at Slab pages, not free pages. The > > extra overhead could be also in percpu allocations, code etc. > > > >>> This is with a buildroot kernel 5.19 build including a shell and sd-card > >>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I > >>> definitely see more errors with shell commands failing due to allocation > >>> failures for the shell process fork. So as far as the K210 is concerned, > >>> switching to SLUB is not ideal. > >>> > >>> I would not want to hold on kernel mm improvements because of this toy > >>> k210 though, so I am not going to prevent SLOB deprecation. I just wish > >>> SLUB itself used less memory :) > >> > >> Did further tests with kernel 6.0.1: > >> * SLOB: 630 free pages after boot, shell working (occasional shell fork > >> failure happen though) > >> * SLAB: getting memory allocation for order 7 failures on boot already > >> (init process). Shell barely working (high frequency of shell command fork > >> failures) > > I forgot to add here that the system was down to about 500 free pages > after boot (again from the shell with "cat /proc/vmstat"). > > >> * SLUB: getting memory allocation for order 7 failures on boot. I do get a > >> shell prompt but cannot run any shell command that involves forking a new > >> process. > > For both slab and slub, I had cpu partial off, debug off and slab merge > on, as I suspected that would lead to less memory overhead. > I suspected memory fragmentation may be an issue but doing > > echo 3 > /proc/sys/vm/drop_caches > > before trying a shell command did not help much at all (it usually does on > that board with SLOB). Note that this is all with buildroot, so this echo > & redirect always works as it does not cause a shell fork. > > >> > >> So if we want to keep the k210 support functional with a shell, we need > >> slob. If we reduce that board support to only one application started as > >> the init process, then I guess anything is OK. > > > > In [1] it was possible to save some more memory with more tuning. Some of > > that required boot parameters and other code changes. In another reply [2] I > > considered adding something like SLUB_TINY to take care of all that, so > > looks like it would make sense to proceed with that. > > If you want me to test something, let me know. Would you try this please? diff --git a/mm/slub.c b/mm/slub.c index a24b71041b26..1c36c4b9aaa0 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) * The larger the object size is, the more slabs we want on the partial * list to avoid pounding the page allocator excessively. */ - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); - + s->min_partial = 0; set_cpu_partial(s); #ifdef CONFIG_NUMA and booting with and without boot parameter slub_max_order=0? Thanks, Hyeonggon > > > > > [1] > > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ > > -- > Damien Le Moal > Western Digital Research _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply related [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-14 14:47 ` Hyeonggon Yoo (?) (?) @ 2022-11-15 4:24 ` Damien Le Moal -1 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-15 4:24 UTC (permalink / raw) To: Hyeonggon Yoo Cc: Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 23:47, Hyeonggon Yoo wrote: > On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >> On 11/14/22 18:36, Vlastimil Babka wrote: >>> On 11/14/22 06:48, Damien Le Moal wrote: >>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>> two of them do not. >>>>>>>>> >>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>> without resorting to spelling out the letters. >>>>>>>>> >>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>> >>>>>>>>> I believe SLOB can be removed because: >>>>>>>>> >>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>> >>>>>>>> I am all for removing SLOB. >>>>>>>> >>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>> included into this thread: >>>>>>>> >>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>> >>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>> >>>>>>> >>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>> >>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>> >>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>> >>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>> >>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>> >>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>> >>> Thanks! >>> >>>>> The current default config using SLOB gives about 630 free memory pages >>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>> >>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>> somewhere, is that correct? >> >> Yep, 8MB, that's it. >> >>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>> difference [1]. But that was looking at Slab pages, not free pages. The >>> extra overhead could be also in percpu allocations, code etc. >>> >>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>> definitely see more errors with shell commands failing due to allocation >>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>> switching to SLUB is not ideal. >>>>> >>>>> I would not want to hold on kernel mm improvements because of this toy >>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>> SLUB itself used less memory :) >>>> >>>> Did further tests with kernel 6.0.1: >>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>> failure happen though) >>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>> (init process). Shell barely working (high frequency of shell command fork >>>> failures) >> >> I forgot to add here that the system was down to about 500 free pages >> after boot (again from the shell with "cat /proc/vmstat"). >> >>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>> shell prompt but cannot run any shell command that involves forking a new >>>> process. >> >> For both slab and slub, I had cpu partial off, debug off and slab merge >> on, as I suspected that would lead to less memory overhead. >> I suspected memory fragmentation may be an issue but doing >> >> echo 3 > /proc/sys/vm/drop_caches >> >> before trying a shell command did not help much at all (it usually does on >> that board with SLOB). Note that this is all with buildroot, so this echo >> & redirect always works as it does not cause a shell fork. >> >>>> >>>> So if we want to keep the k210 support functional with a shell, we need >>>> slob. If we reduce that board support to only one application started as >>>> the init process, then I guess anything is OK. >>> >>> In [1] it was possible to save some more memory with more tuning. Some of >>> that required boot parameters and other code changes. In another reply [2] I >>> considered adding something like SLUB_TINY to take care of all that, so >>> looks like it would make sense to proceed with that. >> >> If you want me to test something, let me know. > > Would you try this please? > > diff --git a/mm/slub.c b/mm/slub.c > index a24b71041b26..1c36c4b9aaa0 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) > * The larger the object size is, the more slabs we want on the partial > * list to avoid pounding the page allocator excessively. > */ > - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); > - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); > - > + s->min_partial = 0; > set_cpu_partial(s); > > #ifdef CONFIG_NUMA > > > and booting with and without boot parameter slub_max_order=0? Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I changed in buildroot default config for the sipeed maix bit card, booting with SD card. The test is: booting and run "cat /proc/vmstat" and register the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. Here are the results: 6.1-rc5, SLOB: - 623 free pages - 629 free pages - 629 free pages 6.1-rc5, SLUB: - 448 free pages - 448 free pages - 429 free pages 6.1-rc5, SLUB + slub_max_order=0: - Init error, shell prompt but no shell command working - Init error, no shell prompt - 508 free pages - Init error, shell prompt but no shell command working 6.1-rc5, SLUB + patch: - Init error, shell prompt but no shell command working - 433 free pages - 448 free pages - 423 free pages 6.1-rc5, SLUB + slub_max_order=0 + patch: - Init error, no shell prompt - Init error, shell prompt, 499 free pages - Init error, shell prompt but no shell command working - Init error, no shell prompt No changes for SLOB results, expected. For default SLUB, I did get all clean boots this time and could run the cat command. But I do see shell fork failures if I keep running commands. For SLUB + slub_max_order=0, I only got one clean boot with 508 free pages. Remaining runs failed to give a shell prompt or allow running cat command. For the clean boot, I do see higher number of free pages. SLUB with the patch was nearly identical to SLUB without the patch. And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I could run the cat command only once, giving 499 free pages, so better than regular SLUB. But it seems that the memory is more fragmented as allocations fail more often. Hope this helps. Let me know if you want to test something else. Cheers. -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-15 4:24 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-15 4:24 UTC (permalink / raw) To: Hyeonggon Yoo Cc: Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 23:47, Hyeonggon Yoo wrote: > On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >> On 11/14/22 18:36, Vlastimil Babka wrote: >>> On 11/14/22 06:48, Damien Le Moal wrote: >>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>> two of them do not. >>>>>>>>> >>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>> without resorting to spelling out the letters. >>>>>>>>> >>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>> >>>>>>>>> I believe SLOB can be removed because: >>>>>>>>> >>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>> >>>>>>>> I am all for removing SLOB. >>>>>>>> >>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>> included into this thread: >>>>>>>> >>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>> >>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>> >>>>>>> >>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>> >>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>> >>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>> >>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>> >>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>> >>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>> >>> Thanks! >>> >>>>> The current default config using SLOB gives about 630 free memory pages >>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>> >>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>> somewhere, is that correct? >> >> Yep, 8MB, that's it. >> >>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>> difference [1]. But that was looking at Slab pages, not free pages. The >>> extra overhead could be also in percpu allocations, code etc. >>> >>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>> definitely see more errors with shell commands failing due to allocation >>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>> switching to SLUB is not ideal. >>>>> >>>>> I would not want to hold on kernel mm improvements because of this toy >>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>> SLUB itself used less memory :) >>>> >>>> Did further tests with kernel 6.0.1: >>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>> failure happen though) >>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>> (init process). Shell barely working (high frequency of shell command fork >>>> failures) >> >> I forgot to add here that the system was down to about 500 free pages >> after boot (again from the shell with "cat /proc/vmstat"). >> >>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>> shell prompt but cannot run any shell command that involves forking a new >>>> process. >> >> For both slab and slub, I had cpu partial off, debug off and slab merge >> on, as I suspected that would lead to less memory overhead. >> I suspected memory fragmentation may be an issue but doing >> >> echo 3 > /proc/sys/vm/drop_caches >> >> before trying a shell command did not help much at all (it usually does on >> that board with SLOB). Note that this is all with buildroot, so this echo >> & redirect always works as it does not cause a shell fork. >> >>>> >>>> So if we want to keep the k210 support functional with a shell, we need >>>> slob. If we reduce that board support to only one application started as >>>> the init process, then I guess anything is OK. >>> >>> In [1] it was possible to save some more memory with more tuning. Some of >>> that required boot parameters and other code changes. In another reply [2] I >>> considered adding something like SLUB_TINY to take care of all that, so >>> looks like it would make sense to proceed with that. >> >> If you want me to test something, let me know. > > Would you try this please? > > diff --git a/mm/slub.c b/mm/slub.c > index a24b71041b26..1c36c4b9aaa0 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) > * The larger the object size is, the more slabs we want on the partial > * list to avoid pounding the page allocator excessively. > */ > - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); > - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); > - > + s->min_partial = 0; > set_cpu_partial(s); > > #ifdef CONFIG_NUMA > > > and booting with and without boot parameter slub_max_order=0? Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I changed in buildroot default config for the sipeed maix bit card, booting with SD card. The test is: booting and run "cat /proc/vmstat" and register the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. Here are the results: 6.1-rc5, SLOB: - 623 free pages - 629 free pages - 629 free pages 6.1-rc5, SLUB: - 448 free pages - 448 free pages - 429 free pages 6.1-rc5, SLUB + slub_max_order=0: - Init error, shell prompt but no shell command working - Init error, no shell prompt - 508 free pages - Init error, shell prompt but no shell command working 6.1-rc5, SLUB + patch: - Init error, shell prompt but no shell command working - 433 free pages - 448 free pages - 423 free pages 6.1-rc5, SLUB + slub_max_order=0 + patch: - Init error, no shell prompt - Init error, shell prompt, 499 free pages - Init error, shell prompt but no shell command working - Init error, no shell prompt No changes for SLOB results, expected. For default SLUB, I did get all clean boots this time and could run the cat command. But I do see shell fork failures if I keep running commands. For SLUB + slub_max_order=0, I only got one clean boot with 508 free pages. Remaining runs failed to give a shell prompt or allow running cat command. For the clean boot, I do see higher number of free pages. SLUB with the patch was nearly identical to SLUB without the patch. And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I could run the cat command only once, giving 499 free pages, so better than regular SLUB. But it seems that the memory is more fragmented as allocations fail more often. Hope this helps. Let me know if you want to test something else. Cheers. -- Damien Le Moal Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-15 4:24 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-15 4:24 UTC (permalink / raw) To: Hyeonggon Yoo Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Christoph Lameter, linux-riscv, Jonas Bonn, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, linux-arm-kernel, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, Vlastimil Babka, Yoshinori Sato, linux-kernel, Conor Dooley, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 11/14/22 23:47, Hyeonggon Yoo wrote: > On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >> On 11/14/22 18:36, Vlastimil Babka wrote: >>> On 11/14/22 06:48, Damien Le Moal wrote: >>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>> two of them do not. >>>>>>>>> >>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>> without resorting to spelling out the letters. >>>>>>>>> >>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>> >>>>>>>>> I believe SLOB can be removed because: >>>>>>>>> >>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>> >>>>>>>> I am all for removing SLOB. >>>>>>>> >>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>> included into this thread: >>>>>>>> >>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>> >>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>> >>>>>>> >>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>> >>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>> >>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>> >>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>> >>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>> >>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>> >>> Thanks! >>> >>>>> The current default config using SLOB gives about 630 free memory pages >>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>> >>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>> somewhere, is that correct? >> >> Yep, 8MB, that's it. >> >>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>> difference [1]. But that was looking at Slab pages, not free pages. The >>> extra overhead could be also in percpu allocations, code etc. >>> >>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>> definitely see more errors with shell commands failing due to allocation >>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>> switching to SLUB is not ideal. >>>>> >>>>> I would not want to hold on kernel mm improvements because of this toy >>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>> SLUB itself used less memory :) >>>> >>>> Did further tests with kernel 6.0.1: >>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>> failure happen though) >>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>> (init process). Shell barely working (high frequency of shell command fork >>>> failures) >> >> I forgot to add here that the system was down to about 500 free pages >> after boot (again from the shell with "cat /proc/vmstat"). >> >>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>> shell prompt but cannot run any shell command that involves forking a new >>>> process. >> >> For both slab and slub, I had cpu partial off, debug off and slab merge >> on, as I suspected that would lead to less memory overhead. >> I suspected memory fragmentation may be an issue but doing >> >> echo 3 > /proc/sys/vm/drop_caches >> >> before trying a shell command did not help much at all (it usually does on >> that board with SLOB). Note that this is all with buildroot, so this echo >> & redirect always works as it does not cause a shell fork. >> >>>> >>>> So if we want to keep the k210 support functional with a shell, we need >>>> slob. If we reduce that board support to only one application started as >>>> the init process, then I guess anything is OK. >>> >>> In [1] it was possible to save some more memory with more tuning. Some of >>> that required boot parameters and other code changes. In another reply [2] I >>> considered adding something like SLUB_TINY to take care of all that, so >>> looks like it would make sense to proceed with that. >> >> If you want me to test something, let me know. > > Would you try this please? > > diff --git a/mm/slub.c b/mm/slub.c > index a24b71041b26..1c36c4b9aaa0 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) > * The larger the object size is, the more slabs we want on the partial > * list to avoid pounding the page allocator excessively. > */ > - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); > - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); > - > + s->min_partial = 0; > set_cpu_partial(s); > > #ifdef CONFIG_NUMA > > > and booting with and without boot parameter slub_max_order=0? Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I changed in buildroot default config for the sipeed maix bit card, booting with SD card. The test is: booting and run "cat /proc/vmstat" and register the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. Here are the results: 6.1-rc5, SLOB: - 623 free pages - 629 free pages - 629 free pages 6.1-rc5, SLUB: - 448 free pages - 448 free pages - 429 free pages 6.1-rc5, SLUB + slub_max_order=0: - Init error, shell prompt but no shell command working - Init error, no shell prompt - 508 free pages - Init error, shell prompt but no shell command working 6.1-rc5, SLUB + patch: - Init error, shell prompt but no shell command working - 433 free pages - 448 free pages - 423 free pages 6.1-rc5, SLUB + slub_max_order=0 + patch: - Init error, no shell prompt - Init error, shell prompt, 499 free pages - Init error, shell prompt but no shell command working - Init error, no shell prompt No changes for SLOB results, expected. For default SLUB, I did get all clean boots this time and could run the cat command. But I do see shell fork failures if I keep running commands. For SLUB + slub_max_order=0, I only got one clean boot with 508 free pages. Remaining runs failed to give a shell prompt or allow running cat command. For the clean boot, I do see higher number of free pages. SLUB with the patch was nearly identical to SLUB without the patch. And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I could run the cat command only once, giving 499 free pages, so better than regular SLUB. But it seems that the memory is more fragmented as allocations fail more often. Hope this helps. Let me know if you want to test something else. Cheers. -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-15 4:24 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-15 4:24 UTC (permalink / raw) To: Hyeonggon Yoo Cc: Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/14/22 23:47, Hyeonggon Yoo wrote: > On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >> On 11/14/22 18:36, Vlastimil Babka wrote: >>> On 11/14/22 06:48, Damien Le Moal wrote: >>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>> two of them do not. >>>>>>>>> >>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>> without resorting to spelling out the letters. >>>>>>>>> >>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>> >>>>>>>>> I believe SLOB can be removed because: >>>>>>>>> >>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>> >>>>>>>> I am all for removing SLOB. >>>>>>>> >>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>> included into this thread: >>>>>>>> >>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>> >>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>> >>>>>>> >>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>> >>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>> >>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>> >>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>> >>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>> >>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>> >>> Thanks! >>> >>>>> The current default config using SLOB gives about 630 free memory pages >>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>> >>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>> somewhere, is that correct? >> >> Yep, 8MB, that's it. >> >>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>> difference [1]. But that was looking at Slab pages, not free pages. The >>> extra overhead could be also in percpu allocations, code etc. >>> >>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>> definitely see more errors with shell commands failing due to allocation >>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>> switching to SLUB is not ideal. >>>>> >>>>> I would not want to hold on kernel mm improvements because of this toy >>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>> SLUB itself used less memory :) >>>> >>>> Did further tests with kernel 6.0.1: >>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>> failure happen though) >>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>> (init process). Shell barely working (high frequency of shell command fork >>>> failures) >> >> I forgot to add here that the system was down to about 500 free pages >> after boot (again from the shell with "cat /proc/vmstat"). >> >>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>> shell prompt but cannot run any shell command that involves forking a new >>>> process. >> >> For both slab and slub, I had cpu partial off, debug off and slab merge >> on, as I suspected that would lead to less memory overhead. >> I suspected memory fragmentation may be an issue but doing >> >> echo 3 > /proc/sys/vm/drop_caches >> >> before trying a shell command did not help much at all (it usually does on >> that board with SLOB). Note that this is all with buildroot, so this echo >> & redirect always works as it does not cause a shell fork. >> >>>> >>>> So if we want to keep the k210 support functional with a shell, we need >>>> slob. If we reduce that board support to only one application started as >>>> the init process, then I guess anything is OK. >>> >>> In [1] it was possible to save some more memory with more tuning. Some of >>> that required boot parameters and other code changes. In another reply [2] I >>> considered adding something like SLUB_TINY to take care of all that, so >>> looks like it would make sense to proceed with that. >> >> If you want me to test something, let me know. > > Would you try this please? > > diff --git a/mm/slub.c b/mm/slub.c > index a24b71041b26..1c36c4b9aaa0 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) > * The larger the object size is, the more slabs we want on the partial > * list to avoid pounding the page allocator excessively. > */ > - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); > - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); > - > + s->min_partial = 0; > set_cpu_partial(s); > > #ifdef CONFIG_NUMA > > > and booting with and without boot parameter slub_max_order=0? Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I changed in buildroot default config for the sipeed maix bit card, booting with SD card. The test is: booting and run "cat /proc/vmstat" and register the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. Here are the results: 6.1-rc5, SLOB: - 623 free pages - 629 free pages - 629 free pages 6.1-rc5, SLUB: - 448 free pages - 448 free pages - 429 free pages 6.1-rc5, SLUB + slub_max_order=0: - Init error, shell prompt but no shell command working - Init error, no shell prompt - 508 free pages - Init error, shell prompt but no shell command working 6.1-rc5, SLUB + patch: - Init error, shell prompt but no shell command working - 433 free pages - 448 free pages - 423 free pages 6.1-rc5, SLUB + slub_max_order=0 + patch: - Init error, no shell prompt - Init error, shell prompt, 499 free pages - Init error, shell prompt but no shell command working - Init error, no shell prompt No changes for SLOB results, expected. For default SLUB, I did get all clean boots this time and could run the cat command. But I do see shell fork failures if I keep running commands. For SLUB + slub_max_order=0, I only got one clean boot with 508 free pages. Remaining runs failed to give a shell prompt or allow running cat command. For the clean boot, I do see higher number of free pages. SLUB with the patch was nearly identical to SLUB without the patch. And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I could run the cat command only once, giving 499 free pages, so better than regular SLUB. But it seems that the memory is more fragmented as allocations fail more often. Hope this helps. Let me know if you want to test something else. Cheers. -- Damien Le Moal Western Digital Research _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-15 4:24 ` Damien Le Moal (?) (?) @ 2022-11-15 4:28 ` Damien Le Moal -1 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-15 4:28 UTC (permalink / raw) To: Hyeonggon Yoo Cc: Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/15/22 13:24, Damien Le Moal wrote: > On 11/14/22 23:47, Hyeonggon Yoo wrote: >> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>> two of them do not. >>>>>>>>>> >>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>> >>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>> >>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>> >>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>> >>>>>>>>> I am all for removing SLOB. >>>>>>>>> >>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>> included into this thread: >>>>>>>>> >>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>> >>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>> >>>>>>>> >>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>> >>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>> >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>> >>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>> >>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>> >>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>> >>>> Thanks! >>>> >>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>> >>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>> somewhere, is that correct? >>> >>> Yep, 8MB, that's it. >>> >>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>> extra overhead could be also in percpu allocations, code etc. >>>> >>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>> definitely see more errors with shell commands failing due to allocation >>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>> switching to SLUB is not ideal. >>>>>> >>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>> SLUB itself used less memory :) >>>>> >>>>> Did further tests with kernel 6.0.1: >>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>> failure happen though) >>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>> (init process). Shell barely working (high frequency of shell command fork >>>>> failures) >>> >>> I forgot to add here that the system was down to about 500 free pages >>> after boot (again from the shell with "cat /proc/vmstat"). >>> >>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>> shell prompt but cannot run any shell command that involves forking a new >>>>> process. >>> >>> For both slab and slub, I had cpu partial off, debug off and slab merge >>> on, as I suspected that would lead to less memory overhead. >>> I suspected memory fragmentation may be an issue but doing >>> >>> echo 3 > /proc/sys/vm/drop_caches >>> >>> before trying a shell command did not help much at all (it usually does on >>> that board with SLOB). Note that this is all with buildroot, so this echo >>> & redirect always works as it does not cause a shell fork. >>> >>>>> >>>>> So if we want to keep the k210 support functional with a shell, we need >>>>> slob. If we reduce that board support to only one application started as >>>>> the init process, then I guess anything is OK. >>>> >>>> In [1] it was possible to save some more memory with more tuning. Some of >>>> that required boot parameters and other code changes. In another reply [2] I >>>> considered adding something like SLUB_TINY to take care of all that, so >>>> looks like it would make sense to proceed with that. >>> >>> If you want me to test something, let me know. >> >> Would you try this please? >> >> diff --git a/mm/slub.c b/mm/slub.c >> index a24b71041b26..1c36c4b9aaa0 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >> * The larger the object size is, the more slabs we want on the partial >> * list to avoid pounding the page allocator excessively. >> */ >> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >> - >> + s->min_partial = 0; >> set_cpu_partial(s); >> >> #ifdef CONFIG_NUMA >> >> >> and booting with and without boot parameter slub_max_order=0? > > Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I > changed in buildroot default config for the sipeed maix bit card, booting > with SD card. The test is: booting and run "cat /proc/vmstat" and register > the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. > > Here are the results: > > 6.1-rc5, SLOB: > - 623 free pages > - 629 free pages > - 629 free pages > 6.1-rc5, SLUB: > - 448 free pages > - 448 free pages > - 429 free pages > 6.1-rc5, SLUB + slub_max_order=0: > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > - 508 free pages > - Init error, shell prompt but no shell command working > 6.1-rc5, SLUB + patch: > - Init error, shell prompt but no shell command working > - 433 free pages > - 448 free pages > - 423 free pages > 6.1-rc5, SLUB + slub_max_order=0 + patch: > - Init error, no shell prompt > - Init error, shell prompt, 499 free pages > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > > No changes for SLOB results, expected. > > For default SLUB, I did get all clean boots this time and could run the > cat command. But I do see shell fork failures if I keep running commands. > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > pages. Remaining runs failed to give a shell prompt or allow running cat > command. For the clean boot, I do see higher number of free pages. > > SLUB with the patch was nearly identical to SLUB without the patch. > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > could run the cat command only once, giving 499 free pages, so better than > regular SLUB. But it seems that the memory is more fragmented as > allocations fail more often. Note about the last case (SLUB+patch+slub_max_order=0). Here are the messages I got when the init shell process fork failed: [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 [ 1.224098] unevictable:0 dirty:0 writeback:0 [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 [ 1.224098] mapped:0 shmem:0 pagetables:0 [ 1.224098] sec_pagetables:0 bounce:0 [ 1.224098] kernel_misc_reclaimable:0 [ 1.224098] free:859 free_pcp:0 free_cma:0 [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1.310654] lowmem_reserve[]: 0 0 0 [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB [ 1.326883] 33 total pagecache pages [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, errno -12 [ 1.337858] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-15 4:28 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-15 4:28 UTC (permalink / raw) To: Hyeonggon Yoo Cc: Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/15/22 13:24, Damien Le Moal wrote: > On 11/14/22 23:47, Hyeonggon Yoo wrote: >> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>> two of them do not. >>>>>>>>>> >>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>> >>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>> >>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>> >>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>> >>>>>>>>> I am all for removing SLOB. >>>>>>>>> >>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>> included into this thread: >>>>>>>>> >>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>> >>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>> >>>>>>>> >>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>> >>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>> >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>> >>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>> >>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>> >>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>> >>>> Thanks! >>>> >>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>> >>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>> somewhere, is that correct? >>> >>> Yep, 8MB, that's it. >>> >>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>> extra overhead could be also in percpu allocations, code etc. >>>> >>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>> definitely see more errors with shell commands failing due to allocation >>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>> switching to SLUB is not ideal. >>>>>> >>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>> SLUB itself used less memory :) >>>>> >>>>> Did further tests with kernel 6.0.1: >>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>> failure happen though) >>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>> (init process). Shell barely working (high frequency of shell command fork >>>>> failures) >>> >>> I forgot to add here that the system was down to about 500 free pages >>> after boot (again from the shell with "cat /proc/vmstat"). >>> >>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>> shell prompt but cannot run any shell command that involves forking a new >>>>> process. >>> >>> For both slab and slub, I had cpu partial off, debug off and slab merge >>> on, as I suspected that would lead to less memory overhead. >>> I suspected memory fragmentation may be an issue but doing >>> >>> echo 3 > /proc/sys/vm/drop_caches >>> >>> before trying a shell command did not help much at all (it usually does on >>> that board with SLOB). Note that this is all with buildroot, so this echo >>> & redirect always works as it does not cause a shell fork. >>> >>>>> >>>>> So if we want to keep the k210 support functional with a shell, we need >>>>> slob. If we reduce that board support to only one application started as >>>>> the init process, then I guess anything is OK. >>>> >>>> In [1] it was possible to save some more memory with more tuning. Some of >>>> that required boot parameters and other code changes. In another reply [2] I >>>> considered adding something like SLUB_TINY to take care of all that, so >>>> looks like it would make sense to proceed with that. >>> >>> If you want me to test something, let me know. >> >> Would you try this please? >> >> diff --git a/mm/slub.c b/mm/slub.c >> index a24b71041b26..1c36c4b9aaa0 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >> * The larger the object size is, the more slabs we want on the partial >> * list to avoid pounding the page allocator excessively. >> */ >> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >> - >> + s->min_partial = 0; >> set_cpu_partial(s); >> >> #ifdef CONFIG_NUMA >> >> >> and booting with and without boot parameter slub_max_order=0? > > Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I > changed in buildroot default config for the sipeed maix bit card, booting > with SD card. The test is: booting and run "cat /proc/vmstat" and register > the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. > > Here are the results: > > 6.1-rc5, SLOB: > - 623 free pages > - 629 free pages > - 629 free pages > 6.1-rc5, SLUB: > - 448 free pages > - 448 free pages > - 429 free pages > 6.1-rc5, SLUB + slub_max_order=0: > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > - 508 free pages > - Init error, shell prompt but no shell command working > 6.1-rc5, SLUB + patch: > - Init error, shell prompt but no shell command working > - 433 free pages > - 448 free pages > - 423 free pages > 6.1-rc5, SLUB + slub_max_order=0 + patch: > - Init error, no shell prompt > - Init error, shell prompt, 499 free pages > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > > No changes for SLOB results, expected. > > For default SLUB, I did get all clean boots this time and could run the > cat command. But I do see shell fork failures if I keep running commands. > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > pages. Remaining runs failed to give a shell prompt or allow running cat > command. For the clean boot, I do see higher number of free pages. > > SLUB with the patch was nearly identical to SLUB without the patch. > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > could run the cat command only once, giving 499 free pages, so better than > regular SLUB. But it seems that the memory is more fragmented as > allocations fail more often. Note about the last case (SLUB+patch+slub_max_order=0). Here are the messages I got when the init shell process fork failed: [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 [ 1.224098] unevictable:0 dirty:0 writeback:0 [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 [ 1.224098] mapped:0 shmem:0 pagetables:0 [ 1.224098] sec_pagetables:0 bounce:0 [ 1.224098] kernel_misc_reclaimable:0 [ 1.224098] free:859 free_pcp:0 free_cma:0 [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1.310654] lowmem_reserve[]: 0 0 0 [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB [ 1.326883] 33 total pagecache pages [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, errno -12 [ 1.337858] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b -- Damien Le Moal Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-15 4:28 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-15 4:28 UTC (permalink / raw) To: Hyeonggon Yoo Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Christoph Lameter, linux-riscv, Jonas Bonn, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, linux-arm-kernel, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, Vlastimil Babka, Yoshinori Sato, linux-kernel, Conor Dooley, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 11/15/22 13:24, Damien Le Moal wrote: > On 11/14/22 23:47, Hyeonggon Yoo wrote: >> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>> two of them do not. >>>>>>>>>> >>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>> >>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>> >>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>> >>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>> >>>>>>>>> I am all for removing SLOB. >>>>>>>>> >>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>> included into this thread: >>>>>>>>> >>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>> >>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>> >>>>>>>> >>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>> >>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>> >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>> >>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>> >>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>> >>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>> >>>> Thanks! >>>> >>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>> >>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>> somewhere, is that correct? >>> >>> Yep, 8MB, that's it. >>> >>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>> extra overhead could be also in percpu allocations, code etc. >>>> >>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>> definitely see more errors with shell commands failing due to allocation >>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>> switching to SLUB is not ideal. >>>>>> >>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>> SLUB itself used less memory :) >>>>> >>>>> Did further tests with kernel 6.0.1: >>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>> failure happen though) >>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>> (init process). Shell barely working (high frequency of shell command fork >>>>> failures) >>> >>> I forgot to add here that the system was down to about 500 free pages >>> after boot (again from the shell with "cat /proc/vmstat"). >>> >>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>> shell prompt but cannot run any shell command that involves forking a new >>>>> process. >>> >>> For both slab and slub, I had cpu partial off, debug off and slab merge >>> on, as I suspected that would lead to less memory overhead. >>> I suspected memory fragmentation may be an issue but doing >>> >>> echo 3 > /proc/sys/vm/drop_caches >>> >>> before trying a shell command did not help much at all (it usually does on >>> that board with SLOB). Note that this is all with buildroot, so this echo >>> & redirect always works as it does not cause a shell fork. >>> >>>>> >>>>> So if we want to keep the k210 support functional with a shell, we need >>>>> slob. If we reduce that board support to only one application started as >>>>> the init process, then I guess anything is OK. >>>> >>>> In [1] it was possible to save some more memory with more tuning. Some of >>>> that required boot parameters and other code changes. In another reply [2] I >>>> considered adding something like SLUB_TINY to take care of all that, so >>>> looks like it would make sense to proceed with that. >>> >>> If you want me to test something, let me know. >> >> Would you try this please? >> >> diff --git a/mm/slub.c b/mm/slub.c >> index a24b71041b26..1c36c4b9aaa0 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >> * The larger the object size is, the more slabs we want on the partial >> * list to avoid pounding the page allocator excessively. >> */ >> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >> - >> + s->min_partial = 0; >> set_cpu_partial(s); >> >> #ifdef CONFIG_NUMA >> >> >> and booting with and without boot parameter slub_max_order=0? > > Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I > changed in buildroot default config for the sipeed maix bit card, booting > with SD card. The test is: booting and run "cat /proc/vmstat" and register > the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. > > Here are the results: > > 6.1-rc5, SLOB: > - 623 free pages > - 629 free pages > - 629 free pages > 6.1-rc5, SLUB: > - 448 free pages > - 448 free pages > - 429 free pages > 6.1-rc5, SLUB + slub_max_order=0: > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > - 508 free pages > - Init error, shell prompt but no shell command working > 6.1-rc5, SLUB + patch: > - Init error, shell prompt but no shell command working > - 433 free pages > - 448 free pages > - 423 free pages > 6.1-rc5, SLUB + slub_max_order=0 + patch: > - Init error, no shell prompt > - Init error, shell prompt, 499 free pages > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > > No changes for SLOB results, expected. > > For default SLUB, I did get all clean boots this time and could run the > cat command. But I do see shell fork failures if I keep running commands. > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > pages. Remaining runs failed to give a shell prompt or allow running cat > command. For the clean boot, I do see higher number of free pages. > > SLUB with the patch was nearly identical to SLUB without the patch. > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > could run the cat command only once, giving 499 free pages, so better than > regular SLUB. But it seems that the memory is more fragmented as > allocations fail more often. Note about the last case (SLUB+patch+slub_max_order=0). Here are the messages I got when the init shell process fork failed: [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 [ 1.224098] unevictable:0 dirty:0 writeback:0 [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 [ 1.224098] mapped:0 shmem:0 pagetables:0 [ 1.224098] sec_pagetables:0 bounce:0 [ 1.224098] kernel_misc_reclaimable:0 [ 1.224098] free:859 free_pcp:0 free_cma:0 [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1.310654] lowmem_reserve[]: 0 0 0 [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB [ 1.326883] 33 total pagecache pages [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, errno -12 [ 1.337858] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-15 4:28 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-15 4:28 UTC (permalink / raw) To: Hyeonggon Yoo Cc: Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/15/22 13:24, Damien Le Moal wrote: > On 11/14/22 23:47, Hyeonggon Yoo wrote: >> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>> two of them do not. >>>>>>>>>> >>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>> >>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>> >>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>> >>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>> >>>>>>>>> I am all for removing SLOB. >>>>>>>>> >>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>> included into this thread: >>>>>>>>> >>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>> >>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>> >>>>>>>> >>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>> >>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>> >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>> >>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>> >>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>> >>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>> >>>> Thanks! >>>> >>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>> >>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>> somewhere, is that correct? >>> >>> Yep, 8MB, that's it. >>> >>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>> extra overhead could be also in percpu allocations, code etc. >>>> >>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>> definitely see more errors with shell commands failing due to allocation >>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>> switching to SLUB is not ideal. >>>>>> >>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>> SLUB itself used less memory :) >>>>> >>>>> Did further tests with kernel 6.0.1: >>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>> failure happen though) >>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>> (init process). Shell barely working (high frequency of shell command fork >>>>> failures) >>> >>> I forgot to add here that the system was down to about 500 free pages >>> after boot (again from the shell with "cat /proc/vmstat"). >>> >>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>> shell prompt but cannot run any shell command that involves forking a new >>>>> process. >>> >>> For both slab and slub, I had cpu partial off, debug off and slab merge >>> on, as I suspected that would lead to less memory overhead. >>> I suspected memory fragmentation may be an issue but doing >>> >>> echo 3 > /proc/sys/vm/drop_caches >>> >>> before trying a shell command did not help much at all (it usually does on >>> that board with SLOB). Note that this is all with buildroot, so this echo >>> & redirect always works as it does not cause a shell fork. >>> >>>>> >>>>> So if we want to keep the k210 support functional with a shell, we need >>>>> slob. If we reduce that board support to only one application started as >>>>> the init process, then I guess anything is OK. >>>> >>>> In [1] it was possible to save some more memory with more tuning. Some of >>>> that required boot parameters and other code changes. In another reply [2] I >>>> considered adding something like SLUB_TINY to take care of all that, so >>>> looks like it would make sense to proceed with that. >>> >>> If you want me to test something, let me know. >> >> Would you try this please? >> >> diff --git a/mm/slub.c b/mm/slub.c >> index a24b71041b26..1c36c4b9aaa0 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >> * The larger the object size is, the more slabs we want on the partial >> * list to avoid pounding the page allocator excessively. >> */ >> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >> - >> + s->min_partial = 0; >> set_cpu_partial(s); >> >> #ifdef CONFIG_NUMA >> >> >> and booting with and without boot parameter slub_max_order=0? > > Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I > changed in buildroot default config for the sipeed maix bit card, booting > with SD card. The test is: booting and run "cat /proc/vmstat" and register > the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. > > Here are the results: > > 6.1-rc5, SLOB: > - 623 free pages > - 629 free pages > - 629 free pages > 6.1-rc5, SLUB: > - 448 free pages > - 448 free pages > - 429 free pages > 6.1-rc5, SLUB + slub_max_order=0: > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > - 508 free pages > - Init error, shell prompt but no shell command working > 6.1-rc5, SLUB + patch: > - Init error, shell prompt but no shell command working > - 433 free pages > - 448 free pages > - 423 free pages > 6.1-rc5, SLUB + slub_max_order=0 + patch: > - Init error, no shell prompt > - Init error, shell prompt, 499 free pages > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > > No changes for SLOB results, expected. > > For default SLUB, I did get all clean boots this time and could run the > cat command. But I do see shell fork failures if I keep running commands. > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > pages. Remaining runs failed to give a shell prompt or allow running cat > command. For the clean boot, I do see higher number of free pages. > > SLUB with the patch was nearly identical to SLUB without the patch. > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > could run the cat command only once, giving 499 free pages, so better than > regular SLUB. But it seems that the memory is more fragmented as > allocations fail more often. Note about the last case (SLUB+patch+slub_max_order=0). Here are the messages I got when the init shell process fork failed: [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 [ 1.224098] unevictable:0 dirty:0 writeback:0 [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 [ 1.224098] mapped:0 shmem:0 pagetables:0 [ 1.224098] sec_pagetables:0 bounce:0 [ 1.224098] kernel_misc_reclaimable:0 [ 1.224098] free:859 free_pcp:0 free_cma:0 [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1.310654] lowmem_reserve[]: 0 0 0 [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB [ 1.326883] 33 total pagecache pages [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, errno -12 [ 1.337858] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b -- Damien Le Moal Western Digital Research _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-15 4:28 ` Damien Le Moal (?) (?) @ 2022-11-16 7:57 ` Matthew Wilcox -1 siblings, 0 replies; 112+ messages in thread From: Matthew Wilcox @ 2022-11-16 7:57 UTC (permalink / raw) To: Damien Le Moal Cc: Hyeonggon Yoo, Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On Tue, Nov 15, 2022 at 01:28:14PM +0900, Damien Le Moal wrote: > On 11/15/22 13:24, Damien Le Moal wrote: > > 6.1-rc5, SLOB: > > - 623 free pages > > - 629 free pages > > - 629 free pages > > 6.1-rc5, SLUB: > > - 448 free pages > > - 448 free pages > > - 429 free pages > > 6.1-rc5, SLUB + slub_max_order=0: > > - Init error, shell prompt but no shell command working > > - Init error, no shell prompt > > - 508 free pages > > - Init error, shell prompt but no shell command working > > 6.1-rc5, SLUB + patch: > > - Init error, shell prompt but no shell command working > > - 433 free pages > > - 448 free pages > > - 423 free pages > > 6.1-rc5, SLUB + slub_max_order=0 + patch: > > - Init error, no shell prompt > > - Init error, shell prompt, 499 free pages > > - Init error, shell prompt but no shell command working > > - Init error, no shell prompt > > > > No changes for SLOB results, expected. > > > > For default SLUB, I did get all clean boots this time and could run the > > cat command. But I do see shell fork failures if I keep running commands. > > > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > > pages. Remaining runs failed to give a shell prompt or allow running cat > > command. For the clean boot, I do see higher number of free pages. > > > > SLUB with the patch was nearly identical to SLUB without the patch. > > > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > > could run the cat command only once, giving 499 free pages, so better than > > regular SLUB. But it seems that the memory is more fragmented as > > allocations fail more often. > > Note about the last case (SLUB+patch+slub_max_order=0). Here are the > messages I got when the init shell process fork failed: > > [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed > [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 > [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 > [ 1.224098] unevictable:0 dirty:0 writeback:0 > [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 > [ 1.224098] mapped:0 shmem:0 pagetables:0 > [ 1.224098] sec_pagetables:0 bounce:0 > [ 1.224098] kernel_misc_reclaimable:0 > [ 1.224098] free:859 free_pcp:0 free_cma:0 > [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB > inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB > kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no > [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB > inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB > managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [ 1.310654] lowmem_reserve[]: 0 0 0 > [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB > (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB > [ 1.326883] 33 total pagecache pages > [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, > errno -12 What you're seeing here is memory fragmentation. There's more than 512kB of memory available, but nommu requires it to be contiguous, and it's not. This is pretty bad, really. We didn't even finish starting up and already we've managed to allocate at least one page from each of the 16 512kB chunks which existed. Commit df48a5f7a3bb was supposed to improve matters by making exact allocations reassemble once they were freed. Maybe the problem is entirely different. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-16 7:57 ` Matthew Wilcox 0 siblings, 0 replies; 112+ messages in thread From: Matthew Wilcox @ 2022-11-16 7:57 UTC (permalink / raw) To: Damien Le Moal Cc: Hyeonggon Yoo, Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On Tue, Nov 15, 2022 at 01:28:14PM +0900, Damien Le Moal wrote: > On 11/15/22 13:24, Damien Le Moal wrote: > > 6.1-rc5, SLOB: > > - 623 free pages > > - 629 free pages > > - 629 free pages > > 6.1-rc5, SLUB: > > - 448 free pages > > - 448 free pages > > - 429 free pages > > 6.1-rc5, SLUB + slub_max_order=0: > > - Init error, shell prompt but no shell command working > > - Init error, no shell prompt > > - 508 free pages > > - Init error, shell prompt but no shell command working > > 6.1-rc5, SLUB + patch: > > - Init error, shell prompt but no shell command working > > - 433 free pages > > - 448 free pages > > - 423 free pages > > 6.1-rc5, SLUB + slub_max_order=0 + patch: > > - Init error, no shell prompt > > - Init error, shell prompt, 499 free pages > > - Init error, shell prompt but no shell command working > > - Init error, no shell prompt > > > > No changes for SLOB results, expected. > > > > For default SLUB, I did get all clean boots this time and could run the > > cat command. But I do see shell fork failures if I keep running commands. > > > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > > pages. Remaining runs failed to give a shell prompt or allow running cat > > command. For the clean boot, I do see higher number of free pages. > > > > SLUB with the patch was nearly identical to SLUB without the patch. > > > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > > could run the cat command only once, giving 499 free pages, so better than > > regular SLUB. But it seems that the memory is more fragmented as > > allocations fail more often. > > Note about the last case (SLUB+patch+slub_max_order=0). Here are the > messages I got when the init shell process fork failed: > > [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed > [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 > [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 > [ 1.224098] unevictable:0 dirty:0 writeback:0 > [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 > [ 1.224098] mapped:0 shmem:0 pagetables:0 > [ 1.224098] sec_pagetables:0 bounce:0 > [ 1.224098] kernel_misc_reclaimable:0 > [ 1.224098] free:859 free_pcp:0 free_cma:0 > [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB > inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB > kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no > [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB > inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB > managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [ 1.310654] lowmem_reserve[]: 0 0 0 > [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB > (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB > [ 1.326883] 33 total pagecache pages > [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, > errno -12 What you're seeing here is memory fragmentation. There's more than 512kB of memory available, but nommu requires it to be contiguous, and it's not. This is pretty bad, really. We didn't even finish starting up and already we've managed to allocate at least one page from each of the 16 512kB chunks which existed. Commit df48a5f7a3bb was supposed to improve matters by making exact allocations reassemble once they were freed. Maybe the problem is entirely different. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-16 7:57 ` Matthew Wilcox 0 siblings, 0 replies; 112+ messages in thread From: Matthew Wilcox @ 2022-11-16 7:57 UTC (permalink / raw) To: Damien Le Moal Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Russell King, linux-arm-kernel, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, Vlastimil Babka, Yoshinori Sato, linux-kernel, Conor Dooley, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On Tue, Nov 15, 2022 at 01:28:14PM +0900, Damien Le Moal wrote: > On 11/15/22 13:24, Damien Le Moal wrote: > > 6.1-rc5, SLOB: > > - 623 free pages > > - 629 free pages > > - 629 free pages > > 6.1-rc5, SLUB: > > - 448 free pages > > - 448 free pages > > - 429 free pages > > 6.1-rc5, SLUB + slub_max_order=0: > > - Init error, shell prompt but no shell command working > > - Init error, no shell prompt > > - 508 free pages > > - Init error, shell prompt but no shell command working > > 6.1-rc5, SLUB + patch: > > - Init error, shell prompt but no shell command working > > - 433 free pages > > - 448 free pages > > - 423 free pages > > 6.1-rc5, SLUB + slub_max_order=0 + patch: > > - Init error, no shell prompt > > - Init error, shell prompt, 499 free pages > > - Init error, shell prompt but no shell command working > > - Init error, no shell prompt > > > > No changes for SLOB results, expected. > > > > For default SLUB, I did get all clean boots this time and could run the > > cat command. But I do see shell fork failures if I keep running commands. > > > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > > pages. Remaining runs failed to give a shell prompt or allow running cat > > command. For the clean boot, I do see higher number of free pages. > > > > SLUB with the patch was nearly identical to SLUB without the patch. > > > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > > could run the cat command only once, giving 499 free pages, so better than > > regular SLUB. But it seems that the memory is more fragmented as > > allocations fail more often. > > Note about the last case (SLUB+patch+slub_max_order=0). Here are the > messages I got when the init shell process fork failed: > > [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed > [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 > [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 > [ 1.224098] unevictable:0 dirty:0 writeback:0 > [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 > [ 1.224098] mapped:0 shmem:0 pagetables:0 > [ 1.224098] sec_pagetables:0 bounce:0 > [ 1.224098] kernel_misc_reclaimable:0 > [ 1.224098] free:859 free_pcp:0 free_cma:0 > [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB > inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB > kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no > [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB > inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB > managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [ 1.310654] lowmem_reserve[]: 0 0 0 > [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB > (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB > [ 1.326883] 33 total pagecache pages > [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, > errno -12 What you're seeing here is memory fragmentation. There's more than 512kB of memory available, but nommu requires it to be contiguous, and it's not. This is pretty bad, really. We didn't even finish starting up and already we've managed to allocate at least one page from each of the 16 512kB chunks which existed. Commit df48a5f7a3bb was supposed to improve matters by making exact allocations reassemble once they were freed. Maybe the problem is entirely different. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-16 7:57 ` Matthew Wilcox 0 siblings, 0 replies; 112+ messages in thread From: Matthew Wilcox @ 2022-11-16 7:57 UTC (permalink / raw) To: Damien Le Moal Cc: Hyeonggon Yoo, Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On Tue, Nov 15, 2022 at 01:28:14PM +0900, Damien Le Moal wrote: > On 11/15/22 13:24, Damien Le Moal wrote: > > 6.1-rc5, SLOB: > > - 623 free pages > > - 629 free pages > > - 629 free pages > > 6.1-rc5, SLUB: > > - 448 free pages > > - 448 free pages > > - 429 free pages > > 6.1-rc5, SLUB + slub_max_order=0: > > - Init error, shell prompt but no shell command working > > - Init error, no shell prompt > > - 508 free pages > > - Init error, shell prompt but no shell command working > > 6.1-rc5, SLUB + patch: > > - Init error, shell prompt but no shell command working > > - 433 free pages > > - 448 free pages > > - 423 free pages > > 6.1-rc5, SLUB + slub_max_order=0 + patch: > > - Init error, no shell prompt > > - Init error, shell prompt, 499 free pages > > - Init error, shell prompt but no shell command working > > - Init error, no shell prompt > > > > No changes for SLOB results, expected. > > > > For default SLUB, I did get all clean boots this time and could run the > > cat command. But I do see shell fork failures if I keep running commands. > > > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > > pages. Remaining runs failed to give a shell prompt or allow running cat > > command. For the clean boot, I do see higher number of free pages. > > > > SLUB with the patch was nearly identical to SLUB without the patch. > > > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > > could run the cat command only once, giving 499 free pages, so better than > > regular SLUB. But it seems that the memory is more fragmented as > > allocations fail more often. > > Note about the last case (SLUB+patch+slub_max_order=0). Here are the > messages I got when the init shell process fork failed: > > [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed > [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 > [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 > [ 1.224098] unevictable:0 dirty:0 writeback:0 > [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 > [ 1.224098] mapped:0 shmem:0 pagetables:0 > [ 1.224098] sec_pagetables:0 bounce:0 > [ 1.224098] kernel_misc_reclaimable:0 > [ 1.224098] free:859 free_pcp:0 free_cma:0 > [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB > inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB > kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no > [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB > inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB > managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB > [ 1.310654] lowmem_reserve[]: 0 0 0 > [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB > (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB > [ 1.326883] 33 total pagecache pages > [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, > errno -12 What you're seeing here is memory fragmentation. There's more than 512kB of memory available, but nommu requires it to be contiguous, and it's not. This is pretty bad, really. We didn't even finish starting up and already we've managed to allocate at least one page from each of the 16 512kB chunks which existed. Commit df48a5f7a3bb was supposed to improve matters by making exact allocations reassemble once they were freed. Maybe the problem is entirely different. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-16 7:57 ` Matthew Wilcox (?) (?) @ 2022-11-16 8:02 ` Damien Le Moal -1 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-16 8:02 UTC (permalink / raw) To: Matthew Wilcox Cc: Hyeonggon Yoo, Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 2022/11/16 16:57, Matthew Wilcox wrote: > On Tue, Nov 15, 2022 at 01:28:14PM +0900, Damien Le Moal wrote: >> On 11/15/22 13:24, Damien Le Moal wrote: >>> 6.1-rc5, SLOB: >>> - 623 free pages >>> - 629 free pages >>> - 629 free pages >>> 6.1-rc5, SLUB: >>> - 448 free pages >>> - 448 free pages >>> - 429 free pages >>> 6.1-rc5, SLUB + slub_max_order=0: >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> - 508 free pages >>> - Init error, shell prompt but no shell command working >>> 6.1-rc5, SLUB + patch: >>> - Init error, shell prompt but no shell command working >>> - 433 free pages >>> - 448 free pages >>> - 423 free pages >>> 6.1-rc5, SLUB + slub_max_order=0 + patch: >>> - Init error, no shell prompt >>> - Init error, shell prompt, 499 free pages >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> >>> No changes for SLOB results, expected. >>> >>> For default SLUB, I did get all clean boots this time and could run the >>> cat command. But I do see shell fork failures if I keep running commands. >>> >>> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >>> pages. Remaining runs failed to give a shell prompt or allow running cat >>> command. For the clean boot, I do see higher number of free pages. >>> >>> SLUB with the patch was nearly identical to SLUB without the patch. >>> >>> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >>> could run the cat command only once, giving 499 free pages, so better than >>> regular SLUB. But it seems that the memory is more fragmented as >>> allocations fail more often. >> >> Note about the last case (SLUB+patch+slub_max_order=0). Here are the >> messages I got when the init shell process fork failed: >> >> [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed >> [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 >> [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 >> [ 1.224098] unevictable:0 dirty:0 writeback:0 >> [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 >> [ 1.224098] mapped:0 shmem:0 pagetables:0 >> [ 1.224098] sec_pagetables:0 bounce:0 >> [ 1.224098] kernel_misc_reclaimable:0 >> [ 1.224098] free:859 free_pcp:0 free_cma:0 >> [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB >> inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB >> mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB >> kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no >> [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB >> reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB >> inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB >> managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB >> [ 1.310654] lowmem_reserve[]: 0 0 0 >> [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB >> (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB >> [ 1.326883] 33 total pagecache pages >> [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, >> errno -12 > > What you're seeing here is memory fragmentation. There's more than 512kB > of memory available, but nommu requires it to be contiguous, and it's > not. This is pretty bad, really. We didn't even finish starting up > and already we've managed to allocate at least one page from each of > the 16 512kB chunks which existed. Commit df48a5f7a3bb was supposed > to improve matters by making exact allocations reassemble once they > were freed. Maybe the problem is entirely different. I suspected something like this when seeing the reported "free:859" :) What I can try next is booting without SD card and the bare minimum set of drivers to see if the fragmentation is still there or not. Would that help ? These one page allocations may be for device drivers so never freed, no ? -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-16 8:02 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-16 8:02 UTC (permalink / raw) To: Matthew Wilcox Cc: Hyeonggon Yoo, Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 2022/11/16 16:57, Matthew Wilcox wrote: > On Tue, Nov 15, 2022 at 01:28:14PM +0900, Damien Le Moal wrote: >> On 11/15/22 13:24, Damien Le Moal wrote: >>> 6.1-rc5, SLOB: >>> - 623 free pages >>> - 629 free pages >>> - 629 free pages >>> 6.1-rc5, SLUB: >>> - 448 free pages >>> - 448 free pages >>> - 429 free pages >>> 6.1-rc5, SLUB + slub_max_order=0: >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> - 508 free pages >>> - Init error, shell prompt but no shell command working >>> 6.1-rc5, SLUB + patch: >>> - Init error, shell prompt but no shell command working >>> - 433 free pages >>> - 448 free pages >>> - 423 free pages >>> 6.1-rc5, SLUB + slub_max_order=0 + patch: >>> - Init error, no shell prompt >>> - Init error, shell prompt, 499 free pages >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> >>> No changes for SLOB results, expected. >>> >>> For default SLUB, I did get all clean boots this time and could run the >>> cat command. But I do see shell fork failures if I keep running commands. >>> >>> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >>> pages. Remaining runs failed to give a shell prompt or allow running cat >>> command. For the clean boot, I do see higher number of free pages. >>> >>> SLUB with the patch was nearly identical to SLUB without the patch. >>> >>> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >>> could run the cat command only once, giving 499 free pages, so better than >>> regular SLUB. But it seems that the memory is more fragmented as >>> allocations fail more often. >> >> Note about the last case (SLUB+patch+slub_max_order=0). Here are the >> messages I got when the init shell process fork failed: >> >> [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed >> [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 >> [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 >> [ 1.224098] unevictable:0 dirty:0 writeback:0 >> [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 >> [ 1.224098] mapped:0 shmem:0 pagetables:0 >> [ 1.224098] sec_pagetables:0 bounce:0 >> [ 1.224098] kernel_misc_reclaimable:0 >> [ 1.224098] free:859 free_pcp:0 free_cma:0 >> [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB >> inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB >> mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB >> kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no >> [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB >> reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB >> inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB >> managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB >> [ 1.310654] lowmem_reserve[]: 0 0 0 >> [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB >> (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB >> [ 1.326883] 33 total pagecache pages >> [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, >> errno -12 > > What you're seeing here is memory fragmentation. There's more than 512kB > of memory available, but nommu requires it to be contiguous, and it's > not. This is pretty bad, really. We didn't even finish starting up > and already we've managed to allocate at least one page from each of > the 16 512kB chunks which existed. Commit df48a5f7a3bb was supposed > to improve matters by making exact allocations reassemble once they > were freed. Maybe the problem is entirely different. I suspected something like this when seeing the reported "free:859" :) What I can try next is booting without SD card and the bare minimum set of drivers to see if the fragmentation is still there or not. Would that help ? These one page allocations may be for device drivers so never freed, no ? -- Damien Le Moal Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-16 8:02 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-16 8:02 UTC (permalink / raw) To: Matthew Wilcox Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Hyeonggon Yoo, Christoph Lameter, linux-riscv, Jonas Bonn, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Russell King, linux-arm-kernel, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, Vlastimil Babka, Yoshinori Sato, linux-kernel, Conor Dooley, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 2022/11/16 16:57, Matthew Wilcox wrote: > On Tue, Nov 15, 2022 at 01:28:14PM +0900, Damien Le Moal wrote: >> On 11/15/22 13:24, Damien Le Moal wrote: >>> 6.1-rc5, SLOB: >>> - 623 free pages >>> - 629 free pages >>> - 629 free pages >>> 6.1-rc5, SLUB: >>> - 448 free pages >>> - 448 free pages >>> - 429 free pages >>> 6.1-rc5, SLUB + slub_max_order=0: >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> - 508 free pages >>> - Init error, shell prompt but no shell command working >>> 6.1-rc5, SLUB + patch: >>> - Init error, shell prompt but no shell command working >>> - 433 free pages >>> - 448 free pages >>> - 423 free pages >>> 6.1-rc5, SLUB + slub_max_order=0 + patch: >>> - Init error, no shell prompt >>> - Init error, shell prompt, 499 free pages >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> >>> No changes for SLOB results, expected. >>> >>> For default SLUB, I did get all clean boots this time and could run the >>> cat command. But I do see shell fork failures if I keep running commands. >>> >>> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >>> pages. Remaining runs failed to give a shell prompt or allow running cat >>> command. For the clean boot, I do see higher number of free pages. >>> >>> SLUB with the patch was nearly identical to SLUB without the patch. >>> >>> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >>> could run the cat command only once, giving 499 free pages, so better than >>> regular SLUB. But it seems that the memory is more fragmented as >>> allocations fail more often. >> >> Note about the last case (SLUB+patch+slub_max_order=0). Here are the >> messages I got when the init shell process fork failed: >> >> [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed >> [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 >> [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 >> [ 1.224098] unevictable:0 dirty:0 writeback:0 >> [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 >> [ 1.224098] mapped:0 shmem:0 pagetables:0 >> [ 1.224098] sec_pagetables:0 bounce:0 >> [ 1.224098] kernel_misc_reclaimable:0 >> [ 1.224098] free:859 free_pcp:0 free_cma:0 >> [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB >> inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB >> mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB >> kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no >> [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB >> reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB >> inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB >> managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB >> [ 1.310654] lowmem_reserve[]: 0 0 0 >> [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB >> (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB >> [ 1.326883] 33 total pagecache pages >> [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, >> errno -12 > > What you're seeing here is memory fragmentation. There's more than 512kB > of memory available, but nommu requires it to be contiguous, and it's > not. This is pretty bad, really. We didn't even finish starting up > and already we've managed to allocate at least one page from each of > the 16 512kB chunks which existed. Commit df48a5f7a3bb was supposed > to improve matters by making exact allocations reassemble once they > were freed. Maybe the problem is entirely different. I suspected something like this when seeing the reported "free:859" :) What I can try next is booting without SD card and the bare minimum set of drivers to see if the fragmentation is still there or not. Would that help ? These one page allocations may be for device drivers so never freed, no ? -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-16 8:02 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-16 8:02 UTC (permalink / raw) To: Matthew Wilcox Cc: Hyeonggon Yoo, Vlastimil Babka, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 2022/11/16 16:57, Matthew Wilcox wrote: > On Tue, Nov 15, 2022 at 01:28:14PM +0900, Damien Le Moal wrote: >> On 11/15/22 13:24, Damien Le Moal wrote: >>> 6.1-rc5, SLOB: >>> - 623 free pages >>> - 629 free pages >>> - 629 free pages >>> 6.1-rc5, SLUB: >>> - 448 free pages >>> - 448 free pages >>> - 429 free pages >>> 6.1-rc5, SLUB + slub_max_order=0: >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> - 508 free pages >>> - Init error, shell prompt but no shell command working >>> 6.1-rc5, SLUB + patch: >>> - Init error, shell prompt but no shell command working >>> - 433 free pages >>> - 448 free pages >>> - 423 free pages >>> 6.1-rc5, SLUB + slub_max_order=0 + patch: >>> - Init error, no shell prompt >>> - Init error, shell prompt, 499 free pages >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> >>> No changes for SLOB results, expected. >>> >>> For default SLUB, I did get all clean boots this time and could run the >>> cat command. But I do see shell fork failures if I keep running commands. >>> >>> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >>> pages. Remaining runs failed to give a shell prompt or allow running cat >>> command. For the clean boot, I do see higher number of free pages. >>> >>> SLUB with the patch was nearly identical to SLUB without the patch. >>> >>> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >>> could run the cat command only once, giving 499 free pages, so better than >>> regular SLUB. But it seems that the memory is more fragmented as >>> allocations fail more often. >> >> Note about the last case (SLUB+patch+slub_max_order=0). Here are the >> messages I got when the init shell process fork failed: >> >> [ 1.217998] nommu: Allocation of length 491520 from process 1 (sh) failed >> [ 1.224098] active_anon:0 inactive_anon:0 isolated_anon:0 >> [ 1.224098] active_file:5 inactive_file:12 isolated_file:0 >> [ 1.224098] unevictable:0 dirty:0 writeback:0 >> [ 1.224098] slab_reclaimable:38 slab_unreclaimable:459 >> [ 1.224098] mapped:0 shmem:0 pagetables:0 >> [ 1.224098] sec_pagetables:0 bounce:0 >> [ 1.224098] kernel_misc_reclaimable:0 >> [ 1.224098] free:859 free_pcp:0 free_cma:0 >> [ 1.260419] Node 0 active_anon:0kB inactive_anon:0kB active_file:20kB >> inactive_file:48kB unevictable:0kB isolated(anon):0kB isolated(file):0kB >> mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB >> kernel_stack:576kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no >> [ 1.285147] DMA32 free:3436kB boost:0kB min:312kB low:388kB high:464kB >> reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB >> inactive_file:28kB unevictable:0kB writepending:0kB present:8192kB >> managed:6240kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB >> [ 1.310654] lowmem_reserve[]: 0 0 0 >> [ 1.314089] DMA32: 17*4kB (U) 10*8kB (U) 7*16kB (U) 6*32kB (U) 11*64kB >> (U) 6*128kB (U) 6*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3460kB >> [ 1.326883] 33 total pagecache pages >> [ 1.330420] binfmt_flat: Unable to allocate RAM for process text/data, >> errno -12 > > What you're seeing here is memory fragmentation. There's more than 512kB > of memory available, but nommu requires it to be contiguous, and it's > not. This is pretty bad, really. We didn't even finish starting up > and already we've managed to allocate at least one page from each of > the 16 512kB chunks which existed. Commit df48a5f7a3bb was supposed > to improve matters by making exact allocations reassemble once they > were freed. Maybe the problem is entirely different. I suspected something like this when seeing the reported "free:859" :) What I can try next is booting without SD card and the bare minimum set of drivers to see if the fragmentation is still there or not. Would that help ? These one page allocations may be for device drivers so never freed, no ? -- Damien Le Moal Western Digital Research _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-15 4:24 ` Damien Le Moal (?) (?) @ 2022-11-16 17:51 ` Vlastimil Babka -1 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-16 17:51 UTC (permalink / raw) To: Damien Le Moal, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/15/22 05:24, Damien Le Moal wrote: > On 11/14/22 23:47, Hyeonggon Yoo wrote: >> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>> two of them do not. >>>>>>>>>> >>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>> >>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>> >>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>> >>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>> >>>>>>>>> I am all for removing SLOB. >>>>>>>>> >>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>> included into this thread: >>>>>>>>> >>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>> >>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>> >>>>>>>> >>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>> >>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>> >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>> >>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>> >>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>> >>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>> >>>> Thanks! >>>> >>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>> >>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>> somewhere, is that correct? >>> >>> Yep, 8MB, that's it. >>> >>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>> extra overhead could be also in percpu allocations, code etc. >>>> >>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>> definitely see more errors with shell commands failing due to allocation >>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>> switching to SLUB is not ideal. >>>>>> >>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>> SLUB itself used less memory :) >>>>> >>>>> Did further tests with kernel 6.0.1: >>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>> failure happen though) >>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>> (init process). Shell barely working (high frequency of shell command fork >>>>> failures) >>> >>> I forgot to add here that the system was down to about 500 free pages >>> after boot (again from the shell with "cat /proc/vmstat"). >>> >>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>> shell prompt but cannot run any shell command that involves forking a new >>>>> process. >>> >>> For both slab and slub, I had cpu partial off, debug off and slab merge >>> on, as I suspected that would lead to less memory overhead. >>> I suspected memory fragmentation may be an issue but doing >>> >>> echo 3 > /proc/sys/vm/drop_caches >>> >>> before trying a shell command did not help much at all (it usually does on >>> that board with SLOB). Note that this is all with buildroot, so this echo >>> & redirect always works as it does not cause a shell fork. >>> >>>>> >>>>> So if we want to keep the k210 support functional with a shell, we need >>>>> slob. If we reduce that board support to only one application started as >>>>> the init process, then I guess anything is OK. >>>> >>>> In [1] it was possible to save some more memory with more tuning. Some of >>>> that required boot parameters and other code changes. In another reply [2] I >>>> considered adding something like SLUB_TINY to take care of all that, so >>>> looks like it would make sense to proceed with that. >>> >>> If you want me to test something, let me know. >> >> Would you try this please? >> >> diff --git a/mm/slub.c b/mm/slub.c >> index a24b71041b26..1c36c4b9aaa0 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >> * The larger the object size is, the more slabs we want on the partial >> * list to avoid pounding the page allocator excessively. >> */ >> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >> - >> + s->min_partial = 0; >> set_cpu_partial(s); >> >> #ifdef CONFIG_NUMA >> >> >> and booting with and without boot parameter slub_max_order=0? > > Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I > changed in buildroot default config for the sipeed maix bit card, booting > with SD card. The test is: booting and run "cat /proc/vmstat" and register > the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. > > Here are the results: > > 6.1-rc5, SLOB: > - 623 free pages > - 629 free pages > - 629 free pages > 6.1-rc5, SLUB: > - 448 free pages > - 448 free pages > - 429 free pages > 6.1-rc5, SLUB + slub_max_order=0: > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > - 508 free pages > - Init error, shell prompt but no shell command working > 6.1-rc5, SLUB + patch: > - Init error, shell prompt but no shell command working > - 433 free pages > - 448 free pages > - 423 free pages > 6.1-rc5, SLUB + slub_max_order=0 + patch: > - Init error, no shell prompt > - Init error, shell prompt, 499 free pages > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > > No changes for SLOB results, expected. > > For default SLUB, I did get all clean boots this time and could run the > cat command. But I do see shell fork failures if I keep running commands. > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > pages. Remaining runs failed to give a shell prompt or allow running cat > command. For the clean boot, I do see higher number of free pages. > > SLUB with the patch was nearly identical to SLUB without the patch. > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > could run the cat command only once, giving 499 free pages, so better than > regular SLUB. But it seems that the memory is more fragmented as > allocations fail more often. > > Hope this helps. Let me know if you want to test something else. Could you please try this branch with CONFIG_SLUB_TINY=y? https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 Seeing your results I didn't modify default slub_max_order by this new CONFIG (yet?) so maybe after trying the default, trying then also with manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise it should be all changes to lower SLUB memory footprint. Hopefully it will be visible in the number of free pages. But if fragmentation is an issue, it might not be enough. BTW, during boot there should be a line "Built X zonelists, mobility grouping ..." can you grep for it and provide please, I wonder if mobility grouping ends up being off or on on that system. Thanks! > Cheers. > ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-16 17:51 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-16 17:51 UTC (permalink / raw) To: Damien Le Moal, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/15/22 05:24, Damien Le Moal wrote: > On 11/14/22 23:47, Hyeonggon Yoo wrote: >> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>> two of them do not. >>>>>>>>>> >>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>> >>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>> >>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>> >>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>> >>>>>>>>> I am all for removing SLOB. >>>>>>>>> >>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>> included into this thread: >>>>>>>>> >>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>> >>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>> >>>>>>>> >>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>> >>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>> >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>> >>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>> >>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>> >>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>> >>>> Thanks! >>>> >>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>> >>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>> somewhere, is that correct? >>> >>> Yep, 8MB, that's it. >>> >>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>> extra overhead could be also in percpu allocations, code etc. >>>> >>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>> definitely see more errors with shell commands failing due to allocation >>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>> switching to SLUB is not ideal. >>>>>> >>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>> SLUB itself used less memory :) >>>>> >>>>> Did further tests with kernel 6.0.1: >>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>> failure happen though) >>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>> (init process). Shell barely working (high frequency of shell command fork >>>>> failures) >>> >>> I forgot to add here that the system was down to about 500 free pages >>> after boot (again from the shell with "cat /proc/vmstat"). >>> >>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>> shell prompt but cannot run any shell command that involves forking a new >>>>> process. >>> >>> For both slab and slub, I had cpu partial off, debug off and slab merge >>> on, as I suspected that would lead to less memory overhead. >>> I suspected memory fragmentation may be an issue but doing >>> >>> echo 3 > /proc/sys/vm/drop_caches >>> >>> before trying a shell command did not help much at all (it usually does on >>> that board with SLOB). Note that this is all with buildroot, so this echo >>> & redirect always works as it does not cause a shell fork. >>> >>>>> >>>>> So if we want to keep the k210 support functional with a shell, we need >>>>> slob. If we reduce that board support to only one application started as >>>>> the init process, then I guess anything is OK. >>>> >>>> In [1] it was possible to save some more memory with more tuning. Some of >>>> that required boot parameters and other code changes. In another reply [2] I >>>> considered adding something like SLUB_TINY to take care of all that, so >>>> looks like it would make sense to proceed with that. >>> >>> If you want me to test something, let me know. >> >> Would you try this please? >> >> diff --git a/mm/slub.c b/mm/slub.c >> index a24b71041b26..1c36c4b9aaa0 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >> * The larger the object size is, the more slabs we want on the partial >> * list to avoid pounding the page allocator excessively. >> */ >> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >> - >> + s->min_partial = 0; >> set_cpu_partial(s); >> >> #ifdef CONFIG_NUMA >> >> >> and booting with and without boot parameter slub_max_order=0? > > Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I > changed in buildroot default config for the sipeed maix bit card, booting > with SD card. The test is: booting and run "cat /proc/vmstat" and register > the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. > > Here are the results: > > 6.1-rc5, SLOB: > - 623 free pages > - 629 free pages > - 629 free pages > 6.1-rc5, SLUB: > - 448 free pages > - 448 free pages > - 429 free pages > 6.1-rc5, SLUB + slub_max_order=0: > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > - 508 free pages > - Init error, shell prompt but no shell command working > 6.1-rc5, SLUB + patch: > - Init error, shell prompt but no shell command working > - 433 free pages > - 448 free pages > - 423 free pages > 6.1-rc5, SLUB + slub_max_order=0 + patch: > - Init error, no shell prompt > - Init error, shell prompt, 499 free pages > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > > No changes for SLOB results, expected. > > For default SLUB, I did get all clean boots this time and could run the > cat command. But I do see shell fork failures if I keep running commands. > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > pages. Remaining runs failed to give a shell prompt or allow running cat > command. For the clean boot, I do see higher number of free pages. > > SLUB with the patch was nearly identical to SLUB without the patch. > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > could run the cat command only once, giving 499 free pages, so better than > regular SLUB. But it seems that the memory is more fragmented as > allocations fail more often. > > Hope this helps. Let me know if you want to test something else. Could you please try this branch with CONFIG_SLUB_TINY=y? https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 Seeing your results I didn't modify default slub_max_order by this new CONFIG (yet?) so maybe after trying the default, trying then also with manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise it should be all changes to lower SLUB memory footprint. Hopefully it will be visible in the number of free pages. But if fragmentation is an issue, it might not be enough. BTW, during boot there should be a line "Built X zonelists, mobility grouping ..." can you grep for it and provide please, I wonder if mobility grouping ends up being off or on on that system. Thanks! > Cheers. > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-16 17:51 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-16 17:51 UTC (permalink / raw) To: Damien Le Moal, Hyeonggon Yoo Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Conor Dooley, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 11/15/22 05:24, Damien Le Moal wrote: > On 11/14/22 23:47, Hyeonggon Yoo wrote: >> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>> two of them do not. >>>>>>>>>> >>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>> >>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>> >>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>> >>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>> >>>>>>>>> I am all for removing SLOB. >>>>>>>>> >>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>> included into this thread: >>>>>>>>> >>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>> >>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>> >>>>>>>> >>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>> >>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>> >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>> >>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>> >>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>> >>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>> >>>> Thanks! >>>> >>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>> >>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>> somewhere, is that correct? >>> >>> Yep, 8MB, that's it. >>> >>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>> extra overhead could be also in percpu allocations, code etc. >>>> >>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>> definitely see more errors with shell commands failing due to allocation >>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>> switching to SLUB is not ideal. >>>>>> >>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>> SLUB itself used less memory :) >>>>> >>>>> Did further tests with kernel 6.0.1: >>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>> failure happen though) >>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>> (init process). Shell barely working (high frequency of shell command fork >>>>> failures) >>> >>> I forgot to add here that the system was down to about 500 free pages >>> after boot (again from the shell with "cat /proc/vmstat"). >>> >>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>> shell prompt but cannot run any shell command that involves forking a new >>>>> process. >>> >>> For both slab and slub, I had cpu partial off, debug off and slab merge >>> on, as I suspected that would lead to less memory overhead. >>> I suspected memory fragmentation may be an issue but doing >>> >>> echo 3 > /proc/sys/vm/drop_caches >>> >>> before trying a shell command did not help much at all (it usually does on >>> that board with SLOB). Note that this is all with buildroot, so this echo >>> & redirect always works as it does not cause a shell fork. >>> >>>>> >>>>> So if we want to keep the k210 support functional with a shell, we need >>>>> slob. If we reduce that board support to only one application started as >>>>> the init process, then I guess anything is OK. >>>> >>>> In [1] it was possible to save some more memory with more tuning. Some of >>>> that required boot parameters and other code changes. In another reply [2] I >>>> considered adding something like SLUB_TINY to take care of all that, so >>>> looks like it would make sense to proceed with that. >>> >>> If you want me to test something, let me know. >> >> Would you try this please? >> >> diff --git a/mm/slub.c b/mm/slub.c >> index a24b71041b26..1c36c4b9aaa0 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >> * The larger the object size is, the more slabs we want on the partial >> * list to avoid pounding the page allocator excessively. >> */ >> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >> - >> + s->min_partial = 0; >> set_cpu_partial(s); >> >> #ifdef CONFIG_NUMA >> >> >> and booting with and without boot parameter slub_max_order=0? > > Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I > changed in buildroot default config for the sipeed maix bit card, booting > with SD card. The test is: booting and run "cat /proc/vmstat" and register > the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. > > Here are the results: > > 6.1-rc5, SLOB: > - 623 free pages > - 629 free pages > - 629 free pages > 6.1-rc5, SLUB: > - 448 free pages > - 448 free pages > - 429 free pages > 6.1-rc5, SLUB + slub_max_order=0: > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > - 508 free pages > - Init error, shell prompt but no shell command working > 6.1-rc5, SLUB + patch: > - Init error, shell prompt but no shell command working > - 433 free pages > - 448 free pages > - 423 free pages > 6.1-rc5, SLUB + slub_max_order=0 + patch: > - Init error, no shell prompt > - Init error, shell prompt, 499 free pages > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > > No changes for SLOB results, expected. > > For default SLUB, I did get all clean boots this time and could run the > cat command. But I do see shell fork failures if I keep running commands. > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > pages. Remaining runs failed to give a shell prompt or allow running cat > command. For the clean boot, I do see higher number of free pages. > > SLUB with the patch was nearly identical to SLUB without the patch. > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > could run the cat command only once, giving 499 free pages, so better than > regular SLUB. But it seems that the memory is more fragmented as > allocations fail more often. > > Hope this helps. Let me know if you want to test something else. Could you please try this branch with CONFIG_SLUB_TINY=y? https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 Seeing your results I didn't modify default slub_max_order by this new CONFIG (yet?) so maybe after trying the default, trying then also with manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise it should be all changes to lower SLUB memory footprint. Hopefully it will be visible in the number of free pages. But if fragmentation is an issue, it might not be enough. BTW, during boot there should be a line "Built X zonelists, mobility grouping ..." can you grep for it and provide please, I wonder if mobility grouping ends up being off or on on that system. Thanks! > Cheers. > ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-16 17:51 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-16 17:51 UTC (permalink / raw) To: Damien Le Moal, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/15/22 05:24, Damien Le Moal wrote: > On 11/14/22 23:47, Hyeonggon Yoo wrote: >> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>> two of them do not. >>>>>>>>>> >>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>> >>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>> >>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>> >>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>> >>>>>>>>> I am all for removing SLOB. >>>>>>>>> >>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>> included into this thread: >>>>>>>>> >>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>> >>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>> >>>>>>>> >>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>> >>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>> >>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>> >>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>> >>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>> >>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>> >>>> Thanks! >>>> >>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>> >>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>> somewhere, is that correct? >>> >>> Yep, 8MB, that's it. >>> >>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>> extra overhead could be also in percpu allocations, code etc. >>>> >>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>> definitely see more errors with shell commands failing due to allocation >>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>> switching to SLUB is not ideal. >>>>>> >>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>> SLUB itself used less memory :) >>>>> >>>>> Did further tests with kernel 6.0.1: >>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>> failure happen though) >>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>> (init process). Shell barely working (high frequency of shell command fork >>>>> failures) >>> >>> I forgot to add here that the system was down to about 500 free pages >>> after boot (again from the shell with "cat /proc/vmstat"). >>> >>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>> shell prompt but cannot run any shell command that involves forking a new >>>>> process. >>> >>> For both slab and slub, I had cpu partial off, debug off and slab merge >>> on, as I suspected that would lead to less memory overhead. >>> I suspected memory fragmentation may be an issue but doing >>> >>> echo 3 > /proc/sys/vm/drop_caches >>> >>> before trying a shell command did not help much at all (it usually does on >>> that board with SLOB). Note that this is all with buildroot, so this echo >>> & redirect always works as it does not cause a shell fork. >>> >>>>> >>>>> So if we want to keep the k210 support functional with a shell, we need >>>>> slob. If we reduce that board support to only one application started as >>>>> the init process, then I guess anything is OK. >>>> >>>> In [1] it was possible to save some more memory with more tuning. Some of >>>> that required boot parameters and other code changes. In another reply [2] I >>>> considered adding something like SLUB_TINY to take care of all that, so >>>> looks like it would make sense to proceed with that. >>> >>> If you want me to test something, let me know. >> >> Would you try this please? >> >> diff --git a/mm/slub.c b/mm/slub.c >> index a24b71041b26..1c36c4b9aaa0 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >> * The larger the object size is, the more slabs we want on the partial >> * list to avoid pounding the page allocator excessively. >> */ >> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >> - >> + s->min_partial = 0; >> set_cpu_partial(s); >> >> #ifdef CONFIG_NUMA >> >> >> and booting with and without boot parameter slub_max_order=0? > > Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I > changed in buildroot default config for the sipeed maix bit card, booting > with SD card. The test is: booting and run "cat /proc/vmstat" and register > the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. > > Here are the results: > > 6.1-rc5, SLOB: > - 623 free pages > - 629 free pages > - 629 free pages > 6.1-rc5, SLUB: > - 448 free pages > - 448 free pages > - 429 free pages > 6.1-rc5, SLUB + slub_max_order=0: > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > - 508 free pages > - Init error, shell prompt but no shell command working > 6.1-rc5, SLUB + patch: > - Init error, shell prompt but no shell command working > - 433 free pages > - 448 free pages > - 423 free pages > 6.1-rc5, SLUB + slub_max_order=0 + patch: > - Init error, no shell prompt > - Init error, shell prompt, 499 free pages > - Init error, shell prompt but no shell command working > - Init error, no shell prompt > > No changes for SLOB results, expected. > > For default SLUB, I did get all clean boots this time and could run the > cat command. But I do see shell fork failures if I keep running commands. > > For SLUB + slub_max_order=0, I only got one clean boot with 508 free > pages. Remaining runs failed to give a shell prompt or allow running cat > command. For the clean boot, I do see higher number of free pages. > > SLUB with the patch was nearly identical to SLUB without the patch. > > And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I > could run the cat command only once, giving 499 free pages, so better than > regular SLUB. But it seems that the memory is more fragmented as > allocations fail more often. > > Hope this helps. Let me know if you want to test something else. Could you please try this branch with CONFIG_SLUB_TINY=y? https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 Seeing your results I didn't modify default slub_max_order by this new CONFIG (yet?) so maybe after trying the default, trying then also with manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise it should be all changes to lower SLUB memory footprint. Hopefully it will be visible in the number of free pages. But if fragmentation is an issue, it might not be enough. BTW, during boot there should be a line "Built X zonelists, mobility grouping ..." can you grep for it and provide please, I wonder if mobility grouping ends up being off or on on that system. Thanks! > Cheers. > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-16 17:51 ` Vlastimil Babka (?) (?) @ 2022-11-17 0:22 ` Damien Le Moal -1 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-17 0:22 UTC (permalink / raw) To: Vlastimil Babka, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 2022/11/17 2:51, Vlastimil Babka wrote: > On 11/15/22 05:24, Damien Le Moal wrote: >> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>>> two of them do not. >>>>>>>>>>> >>>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>>> >>>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>>> >>>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>>> >>>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>>> >>>>>>>>>> I am all for removing SLOB. >>>>>>>>>> >>>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>>> included into this thread: >>>>>>>>>> >>>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>>> >>>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>>> >>>>>>>>> >>>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>>> >>>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>>> >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>>> >>>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>>> >>>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>>> >>>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>>> >>>>> Thanks! >>>>> >>>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>>> >>>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>>> somewhere, is that correct? >>>> >>>> Yep, 8MB, that's it. >>>> >>>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>>> extra overhead could be also in percpu allocations, code etc. >>>>> >>>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>>> definitely see more errors with shell commands failing due to allocation >>>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>>> switching to SLUB is not ideal. >>>>>>> >>>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>>> SLUB itself used less memory :) >>>>>> >>>>>> Did further tests with kernel 6.0.1: >>>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>>> failure happen though) >>>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>>> (init process). Shell barely working (high frequency of shell command fork >>>>>> failures) >>>> >>>> I forgot to add here that the system was down to about 500 free pages >>>> after boot (again from the shell with "cat /proc/vmstat"). >>>> >>>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>>> shell prompt but cannot run any shell command that involves forking a new >>>>>> process. >>>> >>>> For both slab and slub, I had cpu partial off, debug off and slab merge >>>> on, as I suspected that would lead to less memory overhead. >>>> I suspected memory fragmentation may be an issue but doing >>>> >>>> echo 3 > /proc/sys/vm/drop_caches >>>> >>>> before trying a shell command did not help much at all (it usually does on >>>> that board with SLOB). Note that this is all with buildroot, so this echo >>>> & redirect always works as it does not cause a shell fork. >>>> >>>>>> >>>>>> So if we want to keep the k210 support functional with a shell, we need >>>>>> slob. If we reduce that board support to only one application started as >>>>>> the init process, then I guess anything is OK. >>>>> >>>>> In [1] it was possible to save some more memory with more tuning. Some of >>>>> that required boot parameters and other code changes. In another reply [2] I >>>>> considered adding something like SLUB_TINY to take care of all that, so >>>>> looks like it would make sense to proceed with that. >>>> >>>> If you want me to test something, let me know. >>> >>> Would you try this please? >>> >>> diff --git a/mm/slub.c b/mm/slub.c >>> index a24b71041b26..1c36c4b9aaa0 100644 >>> --- a/mm/slub.c >>> +++ b/mm/slub.c >>> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >>> * The larger the object size is, the more slabs we want on the partial >>> * list to avoid pounding the page allocator excessively. >>> */ >>> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >>> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >>> - >>> + s->min_partial = 0; >>> set_cpu_partial(s); >>> >>> #ifdef CONFIG_NUMA >>> >>> >>> and booting with and without boot parameter slub_max_order=0? >> >> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >> changed in buildroot default config for the sipeed maix bit card, booting >> with SD card. The test is: booting and run "cat /proc/vmstat" and register >> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >> >> Here are the results: >> >> 6.1-rc5, SLOB: >> - 623 free pages >> - 629 free pages >> - 629 free pages >> 6.1-rc5, SLUB: >> - 448 free pages >> - 448 free pages >> - 429 free pages >> 6.1-rc5, SLUB + slub_max_order=0: >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> - 508 free pages >> - Init error, shell prompt but no shell command working >> 6.1-rc5, SLUB + patch: >> - Init error, shell prompt but no shell command working >> - 433 free pages >> - 448 free pages >> - 423 free pages >> 6.1-rc5, SLUB + slub_max_order=0 + patch: >> - Init error, no shell prompt >> - Init error, shell prompt, 499 free pages >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> >> No changes for SLOB results, expected. >> >> For default SLUB, I did get all clean boots this time and could run the >> cat command. But I do see shell fork failures if I keep running commands. >> >> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >> pages. Remaining runs failed to give a shell prompt or allow running cat >> command. For the clean boot, I do see higher number of free pages. >> >> SLUB with the patch was nearly identical to SLUB without the patch. >> >> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >> could run the cat command only once, giving 499 free pages, so better than >> regular SLUB. But it seems that the memory is more fragmented as >> allocations fail more often. >> >> Hope this helps. Let me know if you want to test something else. > > Could you please try this branch with CONFIG_SLUB_TINY=y? > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 > > Seeing your results I didn't modify default slub_max_order by this new > CONFIG (yet?) so maybe after trying the default, trying then also with > manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise > it should be all changes to lower SLUB memory footprint. Hopefully it will > be visible in the number of free pages. But if fragmentation is an issue, it > might not be enough. BTW, during boot there should be a line "Built X > zonelists, mobility grouping ..." can you grep for it and provide please, I > wonder if mobility grouping ends up being off or on on that system. OK. Will try as soon as I can (before end of week hopefully). My pipeline is a little crowded right now :) > > Thanks! > >> Cheers. >> > -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-17 0:22 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-17 0:22 UTC (permalink / raw) To: Vlastimil Babka, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 2022/11/17 2:51, Vlastimil Babka wrote: > On 11/15/22 05:24, Damien Le Moal wrote: >> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>>> two of them do not. >>>>>>>>>>> >>>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>>> >>>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>>> >>>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>>> >>>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>>> >>>>>>>>>> I am all for removing SLOB. >>>>>>>>>> >>>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>>> included into this thread: >>>>>>>>>> >>>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>>> >>>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>>> >>>>>>>>> >>>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>>> >>>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>>> >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>>> >>>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>>> >>>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>>> >>>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>>> >>>>> Thanks! >>>>> >>>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>>> >>>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>>> somewhere, is that correct? >>>> >>>> Yep, 8MB, that's it. >>>> >>>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>>> extra overhead could be also in percpu allocations, code etc. >>>>> >>>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>>> definitely see more errors with shell commands failing due to allocation >>>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>>> switching to SLUB is not ideal. >>>>>>> >>>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>>> SLUB itself used less memory :) >>>>>> >>>>>> Did further tests with kernel 6.0.1: >>>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>>> failure happen though) >>>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>>> (init process). Shell barely working (high frequency of shell command fork >>>>>> failures) >>>> >>>> I forgot to add here that the system was down to about 500 free pages >>>> after boot (again from the shell with "cat /proc/vmstat"). >>>> >>>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>>> shell prompt but cannot run any shell command that involves forking a new >>>>>> process. >>>> >>>> For both slab and slub, I had cpu partial off, debug off and slab merge >>>> on, as I suspected that would lead to less memory overhead. >>>> I suspected memory fragmentation may be an issue but doing >>>> >>>> echo 3 > /proc/sys/vm/drop_caches >>>> >>>> before trying a shell command did not help much at all (it usually does on >>>> that board with SLOB). Note that this is all with buildroot, so this echo >>>> & redirect always works as it does not cause a shell fork. >>>> >>>>>> >>>>>> So if we want to keep the k210 support functional with a shell, we need >>>>>> slob. If we reduce that board support to only one application started as >>>>>> the init process, then I guess anything is OK. >>>>> >>>>> In [1] it was possible to save some more memory with more tuning. Some of >>>>> that required boot parameters and other code changes. In another reply [2] I >>>>> considered adding something like SLUB_TINY to take care of all that, so >>>>> looks like it would make sense to proceed with that. >>>> >>>> If you want me to test something, let me know. >>> >>> Would you try this please? >>> >>> diff --git a/mm/slub.c b/mm/slub.c >>> index a24b71041b26..1c36c4b9aaa0 100644 >>> --- a/mm/slub.c >>> +++ b/mm/slub.c >>> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >>> * The larger the object size is, the more slabs we want on the partial >>> * list to avoid pounding the page allocator excessively. >>> */ >>> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >>> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >>> - >>> + s->min_partial = 0; >>> set_cpu_partial(s); >>> >>> #ifdef CONFIG_NUMA >>> >>> >>> and booting with and without boot parameter slub_max_order=0? >> >> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >> changed in buildroot default config for the sipeed maix bit card, booting >> with SD card. The test is: booting and run "cat /proc/vmstat" and register >> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >> >> Here are the results: >> >> 6.1-rc5, SLOB: >> - 623 free pages >> - 629 free pages >> - 629 free pages >> 6.1-rc5, SLUB: >> - 448 free pages >> - 448 free pages >> - 429 free pages >> 6.1-rc5, SLUB + slub_max_order=0: >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> - 508 free pages >> - Init error, shell prompt but no shell command working >> 6.1-rc5, SLUB + patch: >> - Init error, shell prompt but no shell command working >> - 433 free pages >> - 448 free pages >> - 423 free pages >> 6.1-rc5, SLUB + slub_max_order=0 + patch: >> - Init error, no shell prompt >> - Init error, shell prompt, 499 free pages >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> >> No changes for SLOB results, expected. >> >> For default SLUB, I did get all clean boots this time and could run the >> cat command. But I do see shell fork failures if I keep running commands. >> >> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >> pages. Remaining runs failed to give a shell prompt or allow running cat >> command. For the clean boot, I do see higher number of free pages. >> >> SLUB with the patch was nearly identical to SLUB without the patch. >> >> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >> could run the cat command only once, giving 499 free pages, so better than >> regular SLUB. But it seems that the memory is more fragmented as >> allocations fail more often. >> >> Hope this helps. Let me know if you want to test something else. > > Could you please try this branch with CONFIG_SLUB_TINY=y? > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 > > Seeing your results I didn't modify default slub_max_order by this new > CONFIG (yet?) so maybe after trying the default, trying then also with > manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise > it should be all changes to lower SLUB memory footprint. Hopefully it will > be visible in the number of free pages. But if fragmentation is an issue, it > might not be enough. BTW, during boot there should be a line "Built X > zonelists, mobility grouping ..." can you grep for it and provide please, I > wonder if mobility grouping ends up being off or on on that system. OK. Will try as soon as I can (before end of week hopefully). My pipeline is a little crowded right now :) > > Thanks! > >> Cheers. >> > -- Damien Le Moal Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-17 0:22 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-17 0:22 UTC (permalink / raw) To: Vlastimil Babka, Hyeonggon Yoo Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Conor Dooley, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 2022/11/17 2:51, Vlastimil Babka wrote: > On 11/15/22 05:24, Damien Le Moal wrote: >> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>>> two of them do not. >>>>>>>>>>> >>>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>>> >>>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>>> >>>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>>> >>>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>>> >>>>>>>>>> I am all for removing SLOB. >>>>>>>>>> >>>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>>> included into this thread: >>>>>>>>>> >>>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>>> >>>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>>> >>>>>>>>> >>>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>>> >>>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>>> >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>>> >>>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>>> >>>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>>> >>>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>>> >>>>> Thanks! >>>>> >>>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>>> >>>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>>> somewhere, is that correct? >>>> >>>> Yep, 8MB, that's it. >>>> >>>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>>> extra overhead could be also in percpu allocations, code etc. >>>>> >>>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>>> definitely see more errors with shell commands failing due to allocation >>>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>>> switching to SLUB is not ideal. >>>>>>> >>>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>>> SLUB itself used less memory :) >>>>>> >>>>>> Did further tests with kernel 6.0.1: >>>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>>> failure happen though) >>>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>>> (init process). Shell barely working (high frequency of shell command fork >>>>>> failures) >>>> >>>> I forgot to add here that the system was down to about 500 free pages >>>> after boot (again from the shell with "cat /proc/vmstat"). >>>> >>>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>>> shell prompt but cannot run any shell command that involves forking a new >>>>>> process. >>>> >>>> For both slab and slub, I had cpu partial off, debug off and slab merge >>>> on, as I suspected that would lead to less memory overhead. >>>> I suspected memory fragmentation may be an issue but doing >>>> >>>> echo 3 > /proc/sys/vm/drop_caches >>>> >>>> before trying a shell command did not help much at all (it usually does on >>>> that board with SLOB). Note that this is all with buildroot, so this echo >>>> & redirect always works as it does not cause a shell fork. >>>> >>>>>> >>>>>> So if we want to keep the k210 support functional with a shell, we need >>>>>> slob. If we reduce that board support to only one application started as >>>>>> the init process, then I guess anything is OK. >>>>> >>>>> In [1] it was possible to save some more memory with more tuning. Some of >>>>> that required boot parameters and other code changes. In another reply [2] I >>>>> considered adding something like SLUB_TINY to take care of all that, so >>>>> looks like it would make sense to proceed with that. >>>> >>>> If you want me to test something, let me know. >>> >>> Would you try this please? >>> >>> diff --git a/mm/slub.c b/mm/slub.c >>> index a24b71041b26..1c36c4b9aaa0 100644 >>> --- a/mm/slub.c >>> +++ b/mm/slub.c >>> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >>> * The larger the object size is, the more slabs we want on the partial >>> * list to avoid pounding the page allocator excessively. >>> */ >>> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >>> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >>> - >>> + s->min_partial = 0; >>> set_cpu_partial(s); >>> >>> #ifdef CONFIG_NUMA >>> >>> >>> and booting with and without boot parameter slub_max_order=0? >> >> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >> changed in buildroot default config for the sipeed maix bit card, booting >> with SD card. The test is: booting and run "cat /proc/vmstat" and register >> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >> >> Here are the results: >> >> 6.1-rc5, SLOB: >> - 623 free pages >> - 629 free pages >> - 629 free pages >> 6.1-rc5, SLUB: >> - 448 free pages >> - 448 free pages >> - 429 free pages >> 6.1-rc5, SLUB + slub_max_order=0: >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> - 508 free pages >> - Init error, shell prompt but no shell command working >> 6.1-rc5, SLUB + patch: >> - Init error, shell prompt but no shell command working >> - 433 free pages >> - 448 free pages >> - 423 free pages >> 6.1-rc5, SLUB + slub_max_order=0 + patch: >> - Init error, no shell prompt >> - Init error, shell prompt, 499 free pages >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> >> No changes for SLOB results, expected. >> >> For default SLUB, I did get all clean boots this time and could run the >> cat command. But I do see shell fork failures if I keep running commands. >> >> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >> pages. Remaining runs failed to give a shell prompt or allow running cat >> command. For the clean boot, I do see higher number of free pages. >> >> SLUB with the patch was nearly identical to SLUB without the patch. >> >> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >> could run the cat command only once, giving 499 free pages, so better than >> regular SLUB. But it seems that the memory is more fragmented as >> allocations fail more often. >> >> Hope this helps. Let me know if you want to test something else. > > Could you please try this branch with CONFIG_SLUB_TINY=y? > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 > > Seeing your results I didn't modify default slub_max_order by this new > CONFIG (yet?) so maybe after trying the default, trying then also with > manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise > it should be all changes to lower SLUB memory footprint. Hopefully it will > be visible in the number of free pages. But if fragmentation is an issue, it > might not be enough. BTW, during boot there should be a line "Built X > zonelists, mobility grouping ..." can you grep for it and provide please, I > wonder if mobility grouping ends up being off or on on that system. OK. Will try as soon as I can (before end of week hopefully). My pipeline is a little crowded right now :) > > Thanks! > >> Cheers. >> > -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-17 0:22 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-17 0:22 UTC (permalink / raw) To: Vlastimil Babka, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 2022/11/17 2:51, Vlastimil Babka wrote: > On 11/15/22 05:24, Damien Le Moal wrote: >> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>>> two of them do not. >>>>>>>>>>> >>>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>>> >>>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>>> >>>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>>> >>>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>>> >>>>>>>>>> I am all for removing SLOB. >>>>>>>>>> >>>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>>> included into this thread: >>>>>>>>>> >>>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>>> >>>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>>> >>>>>>>>> >>>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>>> >>>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>>> >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>>> >>>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>>> >>>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>>> >>>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>>> >>>>> Thanks! >>>>> >>>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>>> >>>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>>> somewhere, is that correct? >>>> >>>> Yep, 8MB, that's it. >>>> >>>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>>> extra overhead could be also in percpu allocations, code etc. >>>>> >>>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>>> definitely see more errors with shell commands failing due to allocation >>>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>>> switching to SLUB is not ideal. >>>>>>> >>>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>>> SLUB itself used less memory :) >>>>>> >>>>>> Did further tests with kernel 6.0.1: >>>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>>> failure happen though) >>>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>>> (init process). Shell barely working (high frequency of shell command fork >>>>>> failures) >>>> >>>> I forgot to add here that the system was down to about 500 free pages >>>> after boot (again from the shell with "cat /proc/vmstat"). >>>> >>>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>>> shell prompt but cannot run any shell command that involves forking a new >>>>>> process. >>>> >>>> For both slab and slub, I had cpu partial off, debug off and slab merge >>>> on, as I suspected that would lead to less memory overhead. >>>> I suspected memory fragmentation may be an issue but doing >>>> >>>> echo 3 > /proc/sys/vm/drop_caches >>>> >>>> before trying a shell command did not help much at all (it usually does on >>>> that board with SLOB). Note that this is all with buildroot, so this echo >>>> & redirect always works as it does not cause a shell fork. >>>> >>>>>> >>>>>> So if we want to keep the k210 support functional with a shell, we need >>>>>> slob. If we reduce that board support to only one application started as >>>>>> the init process, then I guess anything is OK. >>>>> >>>>> In [1] it was possible to save some more memory with more tuning. Some of >>>>> that required boot parameters and other code changes. In another reply [2] I >>>>> considered adding something like SLUB_TINY to take care of all that, so >>>>> looks like it would make sense to proceed with that. >>>> >>>> If you want me to test something, let me know. >>> >>> Would you try this please? >>> >>> diff --git a/mm/slub.c b/mm/slub.c >>> index a24b71041b26..1c36c4b9aaa0 100644 >>> --- a/mm/slub.c >>> +++ b/mm/slub.c >>> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >>> * The larger the object size is, the more slabs we want on the partial >>> * list to avoid pounding the page allocator excessively. >>> */ >>> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >>> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >>> - >>> + s->min_partial = 0; >>> set_cpu_partial(s); >>> >>> #ifdef CONFIG_NUMA >>> >>> >>> and booting with and without boot parameter slub_max_order=0? >> >> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >> changed in buildroot default config for the sipeed maix bit card, booting >> with SD card. The test is: booting and run "cat /proc/vmstat" and register >> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >> >> Here are the results: >> >> 6.1-rc5, SLOB: >> - 623 free pages >> - 629 free pages >> - 629 free pages >> 6.1-rc5, SLUB: >> - 448 free pages >> - 448 free pages >> - 429 free pages >> 6.1-rc5, SLUB + slub_max_order=0: >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> - 508 free pages >> - Init error, shell prompt but no shell command working >> 6.1-rc5, SLUB + patch: >> - Init error, shell prompt but no shell command working >> - 433 free pages >> - 448 free pages >> - 423 free pages >> 6.1-rc5, SLUB + slub_max_order=0 + patch: >> - Init error, no shell prompt >> - Init error, shell prompt, 499 free pages >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> >> No changes for SLOB results, expected. >> >> For default SLUB, I did get all clean boots this time and could run the >> cat command. But I do see shell fork failures if I keep running commands. >> >> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >> pages. Remaining runs failed to give a shell prompt or allow running cat >> command. For the clean boot, I do see higher number of free pages. >> >> SLUB with the patch was nearly identical to SLUB without the patch. >> >> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >> could run the cat command only once, giving 499 free pages, so better than >> regular SLUB. But it seems that the memory is more fragmented as >> allocations fail more often. >> >> Hope this helps. Let me know if you want to test something else. > > Could you please try this branch with CONFIG_SLUB_TINY=y? > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 > > Seeing your results I didn't modify default slub_max_order by this new > CONFIG (yet?) so maybe after trying the default, trying then also with > manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise > it should be all changes to lower SLUB memory footprint. Hopefully it will > be visible in the number of free pages. But if fragmentation is an issue, it > might not be enough. BTW, during boot there should be a line "Built X > zonelists, mobility grouping ..." can you grep for it and provide please, I > wonder if mobility grouping ends up being off or on on that system. OK. Will try as soon as I can (before end of week hopefully). My pipeline is a little crowded right now :) > > Thanks! > >> Cheers. >> > -- Damien Le Moal Western Digital Research _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-16 17:51 ` Vlastimil Babka (?) (?) @ 2022-11-21 4:30 ` Damien Le Moal -1 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-21 4:30 UTC (permalink / raw) To: Vlastimil Babka, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/17/22 02:51, Vlastimil Babka wrote: > On 11/15/22 05:24, Damien Le Moal wrote: >> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>>> two of them do not. >>>>>>>>>>> >>>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>>> >>>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>>> >>>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>>> >>>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>>> >>>>>>>>>> I am all for removing SLOB. >>>>>>>>>> >>>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>>> included into this thread: >>>>>>>>>> >>>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>>> >>>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>>> >>>>>>>>> >>>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>>> >>>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>>> >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>>> >>>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>>> >>>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>>> >>>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>>> >>>>> Thanks! >>>>> >>>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>>> >>>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>>> somewhere, is that correct? >>>> >>>> Yep, 8MB, that's it. >>>> >>>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>>> extra overhead could be also in percpu allocations, code etc. >>>>> >>>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>>> definitely see more errors with shell commands failing due to allocation >>>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>>> switching to SLUB is not ideal. >>>>>>> >>>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>>> SLUB itself used less memory :) >>>>>> >>>>>> Did further tests with kernel 6.0.1: >>>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>>> failure happen though) >>>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>>> (init process). Shell barely working (high frequency of shell command fork >>>>>> failures) >>>> >>>> I forgot to add here that the system was down to about 500 free pages >>>> after boot (again from the shell with "cat /proc/vmstat"). >>>> >>>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>>> shell prompt but cannot run any shell command that involves forking a new >>>>>> process. >>>> >>>> For both slab and slub, I had cpu partial off, debug off and slab merge >>>> on, as I suspected that would lead to less memory overhead. >>>> I suspected memory fragmentation may be an issue but doing >>>> >>>> echo 3 > /proc/sys/vm/drop_caches >>>> >>>> before trying a shell command did not help much at all (it usually does on >>>> that board with SLOB). Note that this is all with buildroot, so this echo >>>> & redirect always works as it does not cause a shell fork. >>>> >>>>>> >>>>>> So if we want to keep the k210 support functional with a shell, we need >>>>>> slob. If we reduce that board support to only one application started as >>>>>> the init process, then I guess anything is OK. >>>>> >>>>> In [1] it was possible to save some more memory with more tuning. Some of >>>>> that required boot parameters and other code changes. In another reply [2] I >>>>> considered adding something like SLUB_TINY to take care of all that, so >>>>> looks like it would make sense to proceed with that. >>>> >>>> If you want me to test something, let me know. >>> >>> Would you try this please? >>> >>> diff --git a/mm/slub.c b/mm/slub.c >>> index a24b71041b26..1c36c4b9aaa0 100644 >>> --- a/mm/slub.c >>> +++ b/mm/slub.c >>> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >>> * The larger the object size is, the more slabs we want on the partial >>> * list to avoid pounding the page allocator excessively. >>> */ >>> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >>> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >>> - >>> + s->min_partial = 0; >>> set_cpu_partial(s); >>> >>> #ifdef CONFIG_NUMA >>> >>> >>> and booting with and without boot parameter slub_max_order=0? >> >> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >> changed in buildroot default config for the sipeed maix bit card, booting >> with SD card. The test is: booting and run "cat /proc/vmstat" and register >> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >> >> Here are the results: >> >> 6.1-rc5, SLOB: >> - 623 free pages >> - 629 free pages >> - 629 free pages >> 6.1-rc5, SLUB: >> - 448 free pages >> - 448 free pages >> - 429 free pages >> 6.1-rc5, SLUB + slub_max_order=0: >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> - 508 free pages >> - Init error, shell prompt but no shell command working >> 6.1-rc5, SLUB + patch: >> - Init error, shell prompt but no shell command working >> - 433 free pages >> - 448 free pages >> - 423 free pages >> 6.1-rc5, SLUB + slub_max_order=0 + patch: >> - Init error, no shell prompt >> - Init error, shell prompt, 499 free pages >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> >> No changes for SLOB results, expected. >> >> For default SLUB, I did get all clean boots this time and could run the >> cat command. But I do see shell fork failures if I keep running commands. >> >> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >> pages. Remaining runs failed to give a shell prompt or allow running cat >> command. For the clean boot, I do see higher number of free pages. >> >> SLUB with the patch was nearly identical to SLUB without the patch. >> >> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >> could run the cat command only once, giving 499 free pages, so better than >> regular SLUB. But it seems that the memory is more fragmented as >> allocations fail more often. >> >> Hope this helps. Let me know if you want to test something else. > > Could you please try this branch with CONFIG_SLUB_TINY=y? > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 > > Seeing your results I didn't modify default slub_max_order by this new > CONFIG (yet?) so maybe after trying the default, trying then also with > manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise > it should be all changes to lower SLUB memory footprint. Hopefully it will > be visible in the number of free pages. But if fragmentation is an issue, it > might not be enough. BTW, during boot there should be a line "Built X > zonelists, mobility grouping ..." can you grep for it and provide please, I > wonder if mobility grouping ends up being off or on on that system. I ran your branch with CONFIG_SLUB_TINY=y. Here are the results with 3-4 runs per config: * tiny slub with default slub_max_order: - Clean boot, 579 free pages - Clean boot, 575 free pages - Clean boot, 579 free pages * tiny slub with slub_max_order=0 as boot argument: - Init error, shell prompt but no shell command working - Init error, shell prompt, 592 free pages - Init error, shell prompt, 591 free pages - Init error, shell prompt, 591 free pages * tiny slub with slub_max_order=1 as boot argument: - Clean boot, 601 free pages - Clean boot, 601 free pages - Clean boot, 591 free pages - Clean boot, 601 free pages For all cases, mobility grouping was reported as off: [ 0.000000] Built 1 zonelists, mobility grouping off. Total pages: 2020 So it looks like your tiny slub branch with slub_max_order=1 puts us almost on par with slob and that slub_max_order=0 seems to be generating more fragmentation leading to unreliable boot. I also tried slub_max_order=2, which gives clean boot and around 582 free pages, almost the same as the default. With this branch applied, I have no issues with having slob deprecated :) Thanks ! > > Thanks! > >> Cheers. >> > -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-21 4:30 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-21 4:30 UTC (permalink / raw) To: Vlastimil Babka, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/17/22 02:51, Vlastimil Babka wrote: > On 11/15/22 05:24, Damien Le Moal wrote: >> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>>> two of them do not. >>>>>>>>>>> >>>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>>> >>>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>>> >>>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>>> >>>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>>> >>>>>>>>>> I am all for removing SLOB. >>>>>>>>>> >>>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>>> included into this thread: >>>>>>>>>> >>>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>>> >>>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>>> >>>>>>>>> >>>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>>> >>>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>>> >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>>> >>>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>>> >>>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>>> >>>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>>> >>>>> Thanks! >>>>> >>>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>>> >>>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>>> somewhere, is that correct? >>>> >>>> Yep, 8MB, that's it. >>>> >>>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>>> extra overhead could be also in percpu allocations, code etc. >>>>> >>>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>>> definitely see more errors with shell commands failing due to allocation >>>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>>> switching to SLUB is not ideal. >>>>>>> >>>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>>> SLUB itself used less memory :) >>>>>> >>>>>> Did further tests with kernel 6.0.1: >>>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>>> failure happen though) >>>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>>> (init process). Shell barely working (high frequency of shell command fork >>>>>> failures) >>>> >>>> I forgot to add here that the system was down to about 500 free pages >>>> after boot (again from the shell with "cat /proc/vmstat"). >>>> >>>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>>> shell prompt but cannot run any shell command that involves forking a new >>>>>> process. >>>> >>>> For both slab and slub, I had cpu partial off, debug off and slab merge >>>> on, as I suspected that would lead to less memory overhead. >>>> I suspected memory fragmentation may be an issue but doing >>>> >>>> echo 3 > /proc/sys/vm/drop_caches >>>> >>>> before trying a shell command did not help much at all (it usually does on >>>> that board with SLOB). Note that this is all with buildroot, so this echo >>>> & redirect always works as it does not cause a shell fork. >>>> >>>>>> >>>>>> So if we want to keep the k210 support functional with a shell, we need >>>>>> slob. If we reduce that board support to only one application started as >>>>>> the init process, then I guess anything is OK. >>>>> >>>>> In [1] it was possible to save some more memory with more tuning. Some of >>>>> that required boot parameters and other code changes. In another reply [2] I >>>>> considered adding something like SLUB_TINY to take care of all that, so >>>>> looks like it would make sense to proceed with that. >>>> >>>> If you want me to test something, let me know. >>> >>> Would you try this please? >>> >>> diff --git a/mm/slub.c b/mm/slub.c >>> index a24b71041b26..1c36c4b9aaa0 100644 >>> --- a/mm/slub.c >>> +++ b/mm/slub.c >>> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >>> * The larger the object size is, the more slabs we want on the partial >>> * list to avoid pounding the page allocator excessively. >>> */ >>> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >>> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >>> - >>> + s->min_partial = 0; >>> set_cpu_partial(s); >>> >>> #ifdef CONFIG_NUMA >>> >>> >>> and booting with and without boot parameter slub_max_order=0? >> >> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >> changed in buildroot default config for the sipeed maix bit card, booting >> with SD card. The test is: booting and run "cat /proc/vmstat" and register >> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >> >> Here are the results: >> >> 6.1-rc5, SLOB: >> - 623 free pages >> - 629 free pages >> - 629 free pages >> 6.1-rc5, SLUB: >> - 448 free pages >> - 448 free pages >> - 429 free pages >> 6.1-rc5, SLUB + slub_max_order=0: >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> - 508 free pages >> - Init error, shell prompt but no shell command working >> 6.1-rc5, SLUB + patch: >> - Init error, shell prompt but no shell command working >> - 433 free pages >> - 448 free pages >> - 423 free pages >> 6.1-rc5, SLUB + slub_max_order=0 + patch: >> - Init error, no shell prompt >> - Init error, shell prompt, 499 free pages >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> >> No changes for SLOB results, expected. >> >> For default SLUB, I did get all clean boots this time and could run the >> cat command. But I do see shell fork failures if I keep running commands. >> >> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >> pages. Remaining runs failed to give a shell prompt or allow running cat >> command. For the clean boot, I do see higher number of free pages. >> >> SLUB with the patch was nearly identical to SLUB without the patch. >> >> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >> could run the cat command only once, giving 499 free pages, so better than >> regular SLUB. But it seems that the memory is more fragmented as >> allocations fail more often. >> >> Hope this helps. Let me know if you want to test something else. > > Could you please try this branch with CONFIG_SLUB_TINY=y? > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 > > Seeing your results I didn't modify default slub_max_order by this new > CONFIG (yet?) so maybe after trying the default, trying then also with > manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise > it should be all changes to lower SLUB memory footprint. Hopefully it will > be visible in the number of free pages. But if fragmentation is an issue, it > might not be enough. BTW, during boot there should be a line "Built X > zonelists, mobility grouping ..." can you grep for it and provide please, I > wonder if mobility grouping ends up being off or on on that system. I ran your branch with CONFIG_SLUB_TINY=y. Here are the results with 3-4 runs per config: * tiny slub with default slub_max_order: - Clean boot, 579 free pages - Clean boot, 575 free pages - Clean boot, 579 free pages * tiny slub with slub_max_order=0 as boot argument: - Init error, shell prompt but no shell command working - Init error, shell prompt, 592 free pages - Init error, shell prompt, 591 free pages - Init error, shell prompt, 591 free pages * tiny slub with slub_max_order=1 as boot argument: - Clean boot, 601 free pages - Clean boot, 601 free pages - Clean boot, 591 free pages - Clean boot, 601 free pages For all cases, mobility grouping was reported as off: [ 0.000000] Built 1 zonelists, mobility grouping off. Total pages: 2020 So it looks like your tiny slub branch with slub_max_order=1 puts us almost on par with slob and that slub_max_order=0 seems to be generating more fragmentation leading to unreliable boot. I also tried slub_max_order=2, which gives clean boot and around 582 free pages, almost the same as the default. With this branch applied, I have no issues with having slob deprecated :) Thanks ! > > Thanks! > >> Cheers. >> > -- Damien Le Moal Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-21 4:30 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-21 4:30 UTC (permalink / raw) To: Vlastimil Babka, Hyeonggon Yoo Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Conor Dooley, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 11/17/22 02:51, Vlastimil Babka wrote: > On 11/15/22 05:24, Damien Le Moal wrote: >> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>>> two of them do not. >>>>>>>>>>> >>>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>>> >>>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>>> >>>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>>> >>>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>>> >>>>>>>>>> I am all for removing SLOB. >>>>>>>>>> >>>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>>> included into this thread: >>>>>>>>>> >>>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>>> >>>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>>> >>>>>>>>> >>>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>>> >>>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>>> >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>>> >>>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>>> >>>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>>> >>>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>>> >>>>> Thanks! >>>>> >>>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>>> >>>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>>> somewhere, is that correct? >>>> >>>> Yep, 8MB, that's it. >>>> >>>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>>> extra overhead could be also in percpu allocations, code etc. >>>>> >>>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>>> definitely see more errors with shell commands failing due to allocation >>>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>>> switching to SLUB is not ideal. >>>>>>> >>>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>>> SLUB itself used less memory :) >>>>>> >>>>>> Did further tests with kernel 6.0.1: >>>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>>> failure happen though) >>>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>>> (init process). Shell barely working (high frequency of shell command fork >>>>>> failures) >>>> >>>> I forgot to add here that the system was down to about 500 free pages >>>> after boot (again from the shell with "cat /proc/vmstat"). >>>> >>>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>>> shell prompt but cannot run any shell command that involves forking a new >>>>>> process. >>>> >>>> For both slab and slub, I had cpu partial off, debug off and slab merge >>>> on, as I suspected that would lead to less memory overhead. >>>> I suspected memory fragmentation may be an issue but doing >>>> >>>> echo 3 > /proc/sys/vm/drop_caches >>>> >>>> before trying a shell command did not help much at all (it usually does on >>>> that board with SLOB). Note that this is all with buildroot, so this echo >>>> & redirect always works as it does not cause a shell fork. >>>> >>>>>> >>>>>> So if we want to keep the k210 support functional with a shell, we need >>>>>> slob. If we reduce that board support to only one application started as >>>>>> the init process, then I guess anything is OK. >>>>> >>>>> In [1] it was possible to save some more memory with more tuning. Some of >>>>> that required boot parameters and other code changes. In another reply [2] I >>>>> considered adding something like SLUB_TINY to take care of all that, so >>>>> looks like it would make sense to proceed with that. >>>> >>>> If you want me to test something, let me know. >>> >>> Would you try this please? >>> >>> diff --git a/mm/slub.c b/mm/slub.c >>> index a24b71041b26..1c36c4b9aaa0 100644 >>> --- a/mm/slub.c >>> +++ b/mm/slub.c >>> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >>> * The larger the object size is, the more slabs we want on the partial >>> * list to avoid pounding the page allocator excessively. >>> */ >>> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >>> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >>> - >>> + s->min_partial = 0; >>> set_cpu_partial(s); >>> >>> #ifdef CONFIG_NUMA >>> >>> >>> and booting with and without boot parameter slub_max_order=0? >> >> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >> changed in buildroot default config for the sipeed maix bit card, booting >> with SD card. The test is: booting and run "cat /proc/vmstat" and register >> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >> >> Here are the results: >> >> 6.1-rc5, SLOB: >> - 623 free pages >> - 629 free pages >> - 629 free pages >> 6.1-rc5, SLUB: >> - 448 free pages >> - 448 free pages >> - 429 free pages >> 6.1-rc5, SLUB + slub_max_order=0: >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> - 508 free pages >> - Init error, shell prompt but no shell command working >> 6.1-rc5, SLUB + patch: >> - Init error, shell prompt but no shell command working >> - 433 free pages >> - 448 free pages >> - 423 free pages >> 6.1-rc5, SLUB + slub_max_order=0 + patch: >> - Init error, no shell prompt >> - Init error, shell prompt, 499 free pages >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> >> No changes for SLOB results, expected. >> >> For default SLUB, I did get all clean boots this time and could run the >> cat command. But I do see shell fork failures if I keep running commands. >> >> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >> pages. Remaining runs failed to give a shell prompt or allow running cat >> command. For the clean boot, I do see higher number of free pages. >> >> SLUB with the patch was nearly identical to SLUB without the patch. >> >> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >> could run the cat command only once, giving 499 free pages, so better than >> regular SLUB. But it seems that the memory is more fragmented as >> allocations fail more often. >> >> Hope this helps. Let me know if you want to test something else. > > Could you please try this branch with CONFIG_SLUB_TINY=y? > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 > > Seeing your results I didn't modify default slub_max_order by this new > CONFIG (yet?) so maybe after trying the default, trying then also with > manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise > it should be all changes to lower SLUB memory footprint. Hopefully it will > be visible in the number of free pages. But if fragmentation is an issue, it > might not be enough. BTW, during boot there should be a line "Built X > zonelists, mobility grouping ..." can you grep for it and provide please, I > wonder if mobility grouping ends up being off or on on that system. I ran your branch with CONFIG_SLUB_TINY=y. Here are the results with 3-4 runs per config: * tiny slub with default slub_max_order: - Clean boot, 579 free pages - Clean boot, 575 free pages - Clean boot, 579 free pages * tiny slub with slub_max_order=0 as boot argument: - Init error, shell prompt but no shell command working - Init error, shell prompt, 592 free pages - Init error, shell prompt, 591 free pages - Init error, shell prompt, 591 free pages * tiny slub with slub_max_order=1 as boot argument: - Clean boot, 601 free pages - Clean boot, 601 free pages - Clean boot, 591 free pages - Clean boot, 601 free pages For all cases, mobility grouping was reported as off: [ 0.000000] Built 1 zonelists, mobility grouping off. Total pages: 2020 So it looks like your tiny slub branch with slub_max_order=1 puts us almost on par with slob and that slub_max_order=0 seems to be generating more fragmentation leading to unreliable boot. I also tried slub_max_order=2, which gives clean boot and around 582 free pages, almost the same as the default. With this branch applied, I have no issues with having slob deprecated :) Thanks ! > > Thanks! > >> Cheers. >> > -- Damien Le Moal Western Digital Research ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-21 4:30 ` Damien Le Moal 0 siblings, 0 replies; 112+ messages in thread From: Damien Le Moal @ 2022-11-21 4:30 UTC (permalink / raw) To: Vlastimil Babka, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/17/22 02:51, Vlastimil Babka wrote: > On 11/15/22 05:24, Damien Le Moal wrote: >> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>>> On 11/14/22 18:36, Vlastimil Babka wrote: >>>>> On 11/14/22 06:48, Damien Le Moal wrote: >>>>>> On 11/14/22 10:55, Damien Le Moal wrote: >>>>>>> On 11/12/22 05:46, Conor Dooley wrote: >>>>>>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>>>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>>>>>> two of them do not. >>>>>>>>>>> >>>>>>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>>>>>> features), blocking API improvements (more on that below), and my >>>>>>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>>>>>> without resorting to spelling out the letters. >>>>>>>>>>> >>>>>>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>>>>>> >>>>>>>>>>> I believe SLOB can be removed because: >>>>>>>>>>> >>>>>>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>>>>>> >>>>>>>>>> I am all for removing SLOB. >>>>>>>>>> >>>>>>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>>>>>> included into this thread: >>>>>>>>>> >>>>>>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>>>>>> >>>>>>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>>>>>> >>>>>>>>> >>>>>>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>>>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>>>>>> Only a subset has both SLOB and EXPERT: >>>>>>>>> >>>>>>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>>>>>> >>>>>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>>>>>> >>>>>>>> I suppose there's not really a concern with the virt defconfig, but I >>>>>>>> did check the output of `make nommu_k210_defconfig" and despite not >>>>>>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>>>>>> >>>>>>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>>>>>> boots etc, but I have no workloads or w/e to run on it. >>>>>>> >>>>>>> I sent a patch to change the k210 defconfig to using SLUB. However... >>>>> >>>>> Thanks! >>>>> >>>>>>> The current default config using SLOB gives about 630 free memory pages >>>>>>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>>>>>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). >>>>> >>>>> Thanks for the testing! How much RAM does the system have btw? I found 8MB >>>>> somewhere, is that correct? >>>> >>>> Yep, 8MB, that's it. >>>> >>>>> So 230 pages that's a ~920 kB difference. Last time we saw less dramatic >>>>> difference [1]. But that was looking at Slab pages, not free pages. The >>>>> extra overhead could be also in percpu allocations, code etc. >>>>> >>>>>>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>>>>>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>>>>>> definitely see more errors with shell commands failing due to allocation >>>>>>> failures for the shell process fork. So as far as the K210 is concerned, >>>>>>> switching to SLUB is not ideal. >>>>>>> >>>>>>> I would not want to hold on kernel mm improvements because of this toy >>>>>>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>>>>>> SLUB itself used less memory :) >>>>>> >>>>>> Did further tests with kernel 6.0.1: >>>>>> * SLOB: 630 free pages after boot, shell working (occasional shell fork >>>>>> failure happen though) >>>>>> * SLAB: getting memory allocation for order 7 failures on boot already >>>>>> (init process). Shell barely working (high frequency of shell command fork >>>>>> failures) >>>> >>>> I forgot to add here that the system was down to about 500 free pages >>>> after boot (again from the shell with "cat /proc/vmstat"). >>>> >>>>>> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >>>>>> shell prompt but cannot run any shell command that involves forking a new >>>>>> process. >>>> >>>> For both slab and slub, I had cpu partial off, debug off and slab merge >>>> on, as I suspected that would lead to less memory overhead. >>>> I suspected memory fragmentation may be an issue but doing >>>> >>>> echo 3 > /proc/sys/vm/drop_caches >>>> >>>> before trying a shell command did not help much at all (it usually does on >>>> that board with SLOB). Note that this is all with buildroot, so this echo >>>> & redirect always works as it does not cause a shell fork. >>>> >>>>>> >>>>>> So if we want to keep the k210 support functional with a shell, we need >>>>>> slob. If we reduce that board support to only one application started as >>>>>> the init process, then I guess anything is OK. >>>>> >>>>> In [1] it was possible to save some more memory with more tuning. Some of >>>>> that required boot parameters and other code changes. In another reply [2] I >>>>> considered adding something like SLUB_TINY to take care of all that, so >>>>> looks like it would make sense to proceed with that. >>>> >>>> If you want me to test something, let me know. >>> >>> Would you try this please? >>> >>> diff --git a/mm/slub.c b/mm/slub.c >>> index a24b71041b26..1c36c4b9aaa0 100644 >>> --- a/mm/slub.c >>> +++ b/mm/slub.c >>> @@ -4367,9 +4367,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >>> * The larger the object size is, the more slabs we want on the partial >>> * list to avoid pounding the page allocator excessively. >>> */ >>> - s->min_partial = min_t(unsigned long, MAX_PARTIAL, ilog2(s->size) / 2); >>> - s->min_partial = max_t(unsigned long, MIN_PARTIAL, s->min_partial); >>> - >>> + s->min_partial = 0; >>> set_cpu_partial(s); >>> >>> #ifdef CONFIG_NUMA >>> >>> >>> and booting with and without boot parameter slub_max_order=0? >> >> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >> changed in buildroot default config for the sipeed maix bit card, booting >> with SD card. The test is: booting and run "cat /proc/vmstat" and register >> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >> >> Here are the results: >> >> 6.1-rc5, SLOB: >> - 623 free pages >> - 629 free pages >> - 629 free pages >> 6.1-rc5, SLUB: >> - 448 free pages >> - 448 free pages >> - 429 free pages >> 6.1-rc5, SLUB + slub_max_order=0: >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> - 508 free pages >> - Init error, shell prompt but no shell command working >> 6.1-rc5, SLUB + patch: >> - Init error, shell prompt but no shell command working >> - 433 free pages >> - 448 free pages >> - 423 free pages >> 6.1-rc5, SLUB + slub_max_order=0 + patch: >> - Init error, no shell prompt >> - Init error, shell prompt, 499 free pages >> - Init error, shell prompt but no shell command working >> - Init error, no shell prompt >> >> No changes for SLOB results, expected. >> >> For default SLUB, I did get all clean boots this time and could run the >> cat command. But I do see shell fork failures if I keep running commands. >> >> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >> pages. Remaining runs failed to give a shell prompt or allow running cat >> command. For the clean boot, I do see higher number of free pages. >> >> SLUB with the patch was nearly identical to SLUB without the patch. >> >> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >> could run the cat command only once, giving 499 free pages, so better than >> regular SLUB. But it seems that the memory is more fragmented as >> allocations fail more often. >> >> Hope this helps. Let me know if you want to test something else. > > Could you please try this branch with CONFIG_SLUB_TINY=y? > https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 > > Seeing your results I didn't modify default slub_max_order by this new > CONFIG (yet?) so maybe after trying the default, trying then also with > manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise > it should be all changes to lower SLUB memory footprint. Hopefully it will > be visible in the number of free pages. But if fragmentation is an issue, it > might not be enough. BTW, during boot there should be a line "Built X > zonelists, mobility grouping ..." can you grep for it and provide please, I > wonder if mobility grouping ends up being off or on on that system. I ran your branch with CONFIG_SLUB_TINY=y. Here are the results with 3-4 runs per config: * tiny slub with default slub_max_order: - Clean boot, 579 free pages - Clean boot, 575 free pages - Clean boot, 579 free pages * tiny slub with slub_max_order=0 as boot argument: - Init error, shell prompt but no shell command working - Init error, shell prompt, 592 free pages - Init error, shell prompt, 591 free pages - Init error, shell prompt, 591 free pages * tiny slub with slub_max_order=1 as boot argument: - Clean boot, 601 free pages - Clean boot, 601 free pages - Clean boot, 591 free pages - Clean boot, 601 free pages For all cases, mobility grouping was reported as off: [ 0.000000] Built 1 zonelists, mobility grouping off. Total pages: 2020 So it looks like your tiny slub branch with slub_max_order=1 puts us almost on par with slob and that slub_max_order=0 seems to be generating more fragmentation leading to unreliable boot. I also tried slub_max_order=2, which gives clean boot and around 582 free pages, almost the same as the default. With this branch applied, I have no issues with having slob deprecated :) Thanks ! > > Thanks! > >> Cheers. >> > -- Damien Le Moal Western Digital Research _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-21 4:30 ` Damien Le Moal (?) (?) @ 2022-11-21 17:02 ` Vlastimil Babka -1 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-21 17:02 UTC (permalink / raw) To: Damien Le Moal, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/21/22 05:30, Damien Le Moal wrote: > On 11/17/22 02:51, Vlastimil Babka wrote: >> On 11/15/22 05:24, Damien Le Moal wrote: >>> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> >>> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >>> changed in buildroot default config for the sipeed maix bit card, booting >>> with SD card. The test is: booting and run "cat /proc/vmstat" and register >>> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >>> >>> Here are the results: >>> >>> 6.1-rc5, SLOB: >>> - 623 free pages >>> - 629 free pages >>> - 629 free pages >>> 6.1-rc5, SLUB: >>> - 448 free pages >>> - 448 free pages >>> - 429 free pages >>> 6.1-rc5, SLUB + slub_max_order=0: >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> - 508 free pages >>> - Init error, shell prompt but no shell command working >>> 6.1-rc5, SLUB + patch: >>> - Init error, shell prompt but no shell command working >>> - 433 free pages >>> - 448 free pages >>> - 423 free pages >>> 6.1-rc5, SLUB + slub_max_order=0 + patch: >>> - Init error, no shell prompt >>> - Init error, shell prompt, 499 free pages >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> >>> No changes for SLOB results, expected. >>> >>> For default SLUB, I did get all clean boots this time and could run the >>> cat command. But I do see shell fork failures if I keep running commands. >>> >>> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >>> pages. Remaining runs failed to give a shell prompt or allow running cat >>> command. For the clean boot, I do see higher number of free pages. >>> >>> SLUB with the patch was nearly identical to SLUB without the patch. >>> >>> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >>> could run the cat command only once, giving 499 free pages, so better than >>> regular SLUB. But it seems that the memory is more fragmented as >>> allocations fail more often. >>> >>> Hope this helps. Let me know if you want to test something else. >> >> Could you please try this branch with CONFIG_SLUB_TINY=y? >> https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 >> >> Seeing your results I didn't modify default slub_max_order by this new >> CONFIG (yet?) so maybe after trying the default, trying then also with >> manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise >> it should be all changes to lower SLUB memory footprint. Hopefully it will >> be visible in the number of free pages. But if fragmentation is an issue, it >> might not be enough. BTW, during boot there should be a line "Built X >> zonelists, mobility grouping ..." can you grep for it and provide please, I >> wonder if mobility grouping ends up being off or on on that system. > > I ran your branch with CONFIG_SLUB_TINY=y. Here are the results with 3-4 > runs per config: > > * tiny slub with default slub_max_order: > - Clean boot, 579 free pages > - Clean boot, 575 free pages > - Clean boot, 579 free pages > > * tiny slub with slub_max_order=0 as boot argument: > - Init error, shell prompt but no shell command working > - Init error, shell prompt, 592 free pages > - Init error, shell prompt, 591 free pages > - Init error, shell prompt, 591 free pages > > * tiny slub with slub_max_order=1 as boot argument: > - Clean boot, 601 free pages > - Clean boot, 601 free pages > - Clean boot, 591 free pages > - Clean boot, 601 free pages Oh that's great result, better than I'd hope! I'll change the default slub_max_order=1 with CONFIG_SLUB_TINY then. > For all cases, mobility grouping was reported as off: > > [ 0.000000] Built 1 zonelists, mobility grouping off. Total pages: 2020 Yeah, expected that would be the case, thanks for confirming. > So it looks like your tiny slub branch with slub_max_order=1 puts us > almost on par with slob and that slub_max_order=0 seems to be generating > more fragmentation leading to unreliable boot. I also tried > slub_max_order=2, which gives clean boot and around 582 free pages, almost > the same as the default. > > With this branch applied, I have no issues with having slob deprecated :) > Thanks ! Great, thanks for the testing! >> >> Thanks! >> >>> Cheers. >>> >> > ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-21 17:02 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-21 17:02 UTC (permalink / raw) To: Damien Le Moal, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/21/22 05:30, Damien Le Moal wrote: > On 11/17/22 02:51, Vlastimil Babka wrote: >> On 11/15/22 05:24, Damien Le Moal wrote: >>> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> >>> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >>> changed in buildroot default config for the sipeed maix bit card, booting >>> with SD card. The test is: booting and run "cat /proc/vmstat" and register >>> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >>> >>> Here are the results: >>> >>> 6.1-rc5, SLOB: >>> - 623 free pages >>> - 629 free pages >>> - 629 free pages >>> 6.1-rc5, SLUB: >>> - 448 free pages >>> - 448 free pages >>> - 429 free pages >>> 6.1-rc5, SLUB + slub_max_order=0: >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> - 508 free pages >>> - Init error, shell prompt but no shell command working >>> 6.1-rc5, SLUB + patch: >>> - Init error, shell prompt but no shell command working >>> - 433 free pages >>> - 448 free pages >>> - 423 free pages >>> 6.1-rc5, SLUB + slub_max_order=0 + patch: >>> - Init error, no shell prompt >>> - Init error, shell prompt, 499 free pages >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> >>> No changes for SLOB results, expected. >>> >>> For default SLUB, I did get all clean boots this time and could run the >>> cat command. But I do see shell fork failures if I keep running commands. >>> >>> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >>> pages. Remaining runs failed to give a shell prompt or allow running cat >>> command. For the clean boot, I do see higher number of free pages. >>> >>> SLUB with the patch was nearly identical to SLUB without the patch. >>> >>> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >>> could run the cat command only once, giving 499 free pages, so better than >>> regular SLUB. But it seems that the memory is more fragmented as >>> allocations fail more often. >>> >>> Hope this helps. Let me know if you want to test something else. >> >> Could you please try this branch with CONFIG_SLUB_TINY=y? >> https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 >> >> Seeing your results I didn't modify default slub_max_order by this new >> CONFIG (yet?) so maybe after trying the default, trying then also with >> manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise >> it should be all changes to lower SLUB memory footprint. Hopefully it will >> be visible in the number of free pages. But if fragmentation is an issue, it >> might not be enough. BTW, during boot there should be a line "Built X >> zonelists, mobility grouping ..." can you grep for it and provide please, I >> wonder if mobility grouping ends up being off or on on that system. > > I ran your branch with CONFIG_SLUB_TINY=y. Here are the results with 3-4 > runs per config: > > * tiny slub with default slub_max_order: > - Clean boot, 579 free pages > - Clean boot, 575 free pages > - Clean boot, 579 free pages > > * tiny slub with slub_max_order=0 as boot argument: > - Init error, shell prompt but no shell command working > - Init error, shell prompt, 592 free pages > - Init error, shell prompt, 591 free pages > - Init error, shell prompt, 591 free pages > > * tiny slub with slub_max_order=1 as boot argument: > - Clean boot, 601 free pages > - Clean boot, 601 free pages > - Clean boot, 591 free pages > - Clean boot, 601 free pages Oh that's great result, better than I'd hope! I'll change the default slub_max_order=1 with CONFIG_SLUB_TINY then. > For all cases, mobility grouping was reported as off: > > [ 0.000000] Built 1 zonelists, mobility grouping off. Total pages: 2020 Yeah, expected that would be the case, thanks for confirming. > So it looks like your tiny slub branch with slub_max_order=1 puts us > almost on par with slob and that slub_max_order=0 seems to be generating > more fragmentation leading to unreliable boot. I also tried > slub_max_order=2, which gives clean boot and around 582 free pages, almost > the same as the default. > > With this branch applied, I have no issues with having slob deprecated :) > Thanks ! Great, thanks for the testing! >> >> Thanks! >> >>> Cheers. >>> >> > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-21 17:02 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-21 17:02 UTC (permalink / raw) To: Damien Le Moal, Hyeonggon Yoo Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Christoph Lameter, linux-riscv, Jonas Bonn, Yoshinori Sato, Aaro Koskinen, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Alexander Shiyan, linux-kernel, Conor Dooley, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On 11/21/22 05:30, Damien Le Moal wrote: > On 11/17/22 02:51, Vlastimil Babka wrote: >> On 11/15/22 05:24, Damien Le Moal wrote: >>> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> >>> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >>> changed in buildroot default config for the sipeed maix bit card, booting >>> with SD card. The test is: booting and run "cat /proc/vmstat" and register >>> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >>> >>> Here are the results: >>> >>> 6.1-rc5, SLOB: >>> - 623 free pages >>> - 629 free pages >>> - 629 free pages >>> 6.1-rc5, SLUB: >>> - 448 free pages >>> - 448 free pages >>> - 429 free pages >>> 6.1-rc5, SLUB + slub_max_order=0: >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> - 508 free pages >>> - Init error, shell prompt but no shell command working >>> 6.1-rc5, SLUB + patch: >>> - Init error, shell prompt but no shell command working >>> - 433 free pages >>> - 448 free pages >>> - 423 free pages >>> 6.1-rc5, SLUB + slub_max_order=0 + patch: >>> - Init error, no shell prompt >>> - Init error, shell prompt, 499 free pages >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> >>> No changes for SLOB results, expected. >>> >>> For default SLUB, I did get all clean boots this time and could run the >>> cat command. But I do see shell fork failures if I keep running commands. >>> >>> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >>> pages. Remaining runs failed to give a shell prompt or allow running cat >>> command. For the clean boot, I do see higher number of free pages. >>> >>> SLUB with the patch was nearly identical to SLUB without the patch. >>> >>> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >>> could run the cat command only once, giving 499 free pages, so better than >>> regular SLUB. But it seems that the memory is more fragmented as >>> allocations fail more often. >>> >>> Hope this helps. Let me know if you want to test something else. >> >> Could you please try this branch with CONFIG_SLUB_TINY=y? >> https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 >> >> Seeing your results I didn't modify default slub_max_order by this new >> CONFIG (yet?) so maybe after trying the default, trying then also with >> manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise >> it should be all changes to lower SLUB memory footprint. Hopefully it will >> be visible in the number of free pages. But if fragmentation is an issue, it >> might not be enough. BTW, during boot there should be a line "Built X >> zonelists, mobility grouping ..." can you grep for it and provide please, I >> wonder if mobility grouping ends up being off or on on that system. > > I ran your branch with CONFIG_SLUB_TINY=y. Here are the results with 3-4 > runs per config: > > * tiny slub with default slub_max_order: > - Clean boot, 579 free pages > - Clean boot, 575 free pages > - Clean boot, 579 free pages > > * tiny slub with slub_max_order=0 as boot argument: > - Init error, shell prompt but no shell command working > - Init error, shell prompt, 592 free pages > - Init error, shell prompt, 591 free pages > - Init error, shell prompt, 591 free pages > > * tiny slub with slub_max_order=1 as boot argument: > - Clean boot, 601 free pages > - Clean boot, 601 free pages > - Clean boot, 591 free pages > - Clean boot, 601 free pages Oh that's great result, better than I'd hope! I'll change the default slub_max_order=1 with CONFIG_SLUB_TINY then. > For all cases, mobility grouping was reported as off: > > [ 0.000000] Built 1 zonelists, mobility grouping off. Total pages: 2020 Yeah, expected that would be the case, thanks for confirming. > So it looks like your tiny slub branch with slub_max_order=1 puts us > almost on par with slob and that slub_max_order=0 seems to be generating > more fragmentation leading to unreliable boot. I also tried > slub_max_order=2, which gives clean boot and around 582 free pages, almost > the same as the default. > > With this branch applied, I have no issues with having slob deprecated :) > Thanks ! Great, thanks for the testing! >> >> Thanks! >> >>> Cheers. >>> >> > ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-21 17:02 ` Vlastimil Babka 0 siblings, 0 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-21 17:02 UTC (permalink / raw) To: Damien Le Moal, Hyeonggon Yoo Cc: Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On 11/21/22 05:30, Damien Le Moal wrote: > On 11/17/22 02:51, Vlastimil Babka wrote: >> On 11/15/22 05:24, Damien Le Moal wrote: >>> On 11/14/22 23:47, Hyeonggon Yoo wrote: >>>> On Mon, Nov 14, 2022 at 08:35:31PM +0900, Damien Le Moal wrote: >>> >>> Test notes: I used Linus 6.1-rc5 as the base. That is the only thing I >>> changed in buildroot default config for the sipeed maix bit card, booting >>> with SD card. The test is: booting and run "cat /proc/vmstat" and register >>> the nr_free_pages value. I repeated the boot + cat 3 to 4 times for each case. >>> >>> Here are the results: >>> >>> 6.1-rc5, SLOB: >>> - 623 free pages >>> - 629 free pages >>> - 629 free pages >>> 6.1-rc5, SLUB: >>> - 448 free pages >>> - 448 free pages >>> - 429 free pages >>> 6.1-rc5, SLUB + slub_max_order=0: >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> - 508 free pages >>> - Init error, shell prompt but no shell command working >>> 6.1-rc5, SLUB + patch: >>> - Init error, shell prompt but no shell command working >>> - 433 free pages >>> - 448 free pages >>> - 423 free pages >>> 6.1-rc5, SLUB + slub_max_order=0 + patch: >>> - Init error, no shell prompt >>> - Init error, shell prompt, 499 free pages >>> - Init error, shell prompt but no shell command working >>> - Init error, no shell prompt >>> >>> No changes for SLOB results, expected. >>> >>> For default SLUB, I did get all clean boots this time and could run the >>> cat command. But I do see shell fork failures if I keep running commands. >>> >>> For SLUB + slub_max_order=0, I only got one clean boot with 508 free >>> pages. Remaining runs failed to give a shell prompt or allow running cat >>> command. For the clean boot, I do see higher number of free pages. >>> >>> SLUB with the patch was nearly identical to SLUB without the patch. >>> >>> And SLUB+patch+slub_max_order=0 gave again a lot of errors/bad boot. I >>> could run the cat command only once, giving 499 free pages, so better than >>> regular SLUB. But it seems that the memory is more fragmented as >>> allocations fail more often. >>> >>> Hope this helps. Let me know if you want to test something else. >> >> Could you please try this branch with CONFIG_SLUB_TINY=y? >> https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-tiny-v1r0 >> >> Seeing your results I didn't modify default slub_max_order by this new >> CONFIG (yet?) so maybe after trying the default, trying then also with >> manual slub_max_order=0 and slub_max_order=1 would be useful too. Otherwise >> it should be all changes to lower SLUB memory footprint. Hopefully it will >> be visible in the number of free pages. But if fragmentation is an issue, it >> might not be enough. BTW, during boot there should be a line "Built X >> zonelists, mobility grouping ..." can you grep for it and provide please, I >> wonder if mobility grouping ends up being off or on on that system. > > I ran your branch with CONFIG_SLUB_TINY=y. Here are the results with 3-4 > runs per config: > > * tiny slub with default slub_max_order: > - Clean boot, 579 free pages > - Clean boot, 575 free pages > - Clean boot, 579 free pages > > * tiny slub with slub_max_order=0 as boot argument: > - Init error, shell prompt but no shell command working > - Init error, shell prompt, 592 free pages > - Init error, shell prompt, 591 free pages > - Init error, shell prompt, 591 free pages > > * tiny slub with slub_max_order=1 as boot argument: > - Clean boot, 601 free pages > - Clean boot, 601 free pages > - Clean boot, 591 free pages > - Clean boot, 601 free pages Oh that's great result, better than I'd hope! I'll change the default slub_max_order=1 with CONFIG_SLUB_TINY then. > For all cases, mobility grouping was reported as off: > > [ 0.000000] Built 1 zonelists, mobility grouping off. Total pages: 2020 Yeah, expected that would be the case, thanks for confirming. > So it looks like your tiny slub branch with slub_max_order=1 puts us > almost on par with slob and that slub_max_order=0 seems to be generating > more fragmentation leading to unreliable boot. I also tried > slub_max_order=2, which gives clean boot and around 582 free pages, almost > the same as the default. > > With this branch applied, I have no issues with having slob deprecated :) > Thanks ! Great, thanks for the testing! >> >> Thanks! >> >>> Cheers. >>> >> > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-14 9:36 ` Vlastimil Babka (?) (?) @ 2022-11-14 11:50 ` Hyeonggon Yoo -1 siblings, 0 replies; 112+ messages in thread From: Hyeonggon Yoo @ 2022-11-14 11:50 UTC (permalink / raw) To: Vlastimil Babka Cc: Damien Le Moal, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On Mon, Nov 14, 2022 at 10:36:31AM +0100, Vlastimil Babka wrote: > On 11/14/22 06:48, Damien Le Moal wrote: > > On 11/14/22 10:55, Damien Le Moal wrote: > >> On 11/12/22 05:46, Conor Dooley wrote: > >>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > >>>> On 11/8/22 22:44, Pasha Tatashin wrote: > >>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> as we all know, we currently have three slab allocators. As we discussed > >>>>>> at LPC [1], it is my hope that one of these allocators has a future, and > >>>>>> two of them do not. > >>>>>> > >>>>>> The unsurprising reasons include code maintenance burden, other features > >>>>>> compatible with only a subset of allocators (or more effort spent on the > >>>>>> features), blocking API improvements (more on that below), and my > >>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >>>>>> without resorting to spelling out the letters. > >>>>>> > >>>>>> I think (but may be proven wrong) that SLOB is the easier target of the > >>>>>> two to be removed, so I'd like to focus on it first. > >>>>>> > >>>>>> I believe SLOB can be removed because: > >>>>>> > >>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint > >>>>>> by putting all objects together, which has its CPU performance costs > >>>>>> (locking, lack of percpu caching, searching for free space...). I'm not > >>>>>> aware of any "tiny linux" deployment that opts for this. For example, > >>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >>>>>> SLOB impact is too much for those who tried. Googling for > >>>>>> "CONFIG_SLOB=y" yielded nothing useful. > >>>>> > >>>>> I am all for removing SLOB. > >>>>> > >>>>> There are some devices with configs where SLOB is enabled by default. > >>>>> Perhaps, the owners/maintainers of those devices/configs should be > >>>>> included into this thread: > >>>>> > >>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> > >>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > >>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >>> > >>>> > >>>> Turns out that since SLOB depends on EXPERT, many of those lack it so > >>>> running make defconfig ends up with SLUB anyway, unless I miss something. > >>>> Only a subset has both SLOB and EXPERT: > >>>> > >>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >>> > >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > >>> > >>> I suppose there's not really a concern with the virt defconfig, but I > >>> did check the output of `make nommu_k210_defconfig" and despite not > >>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. > >>> > >>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > >>> boots etc, but I have no workloads or w/e to run on it. > >> > >> I sent a patch to change the k210 defconfig to using SLUB. However... > > Thanks! > > >> The current default config using SLOB gives about 630 free memory pages > >> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > >> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > somewhere, is that correct? > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > difference [1]. But that was looking at Slab pages, not free pages. IIRC overhead of s->min_partial (between 5 and 10) was pretty big because SLUB caches at most (s->min_partial) * (nr of caches) * (size of slab) bytes of unused memory. Passing slub_max_order=0 also may help a little bit. > The extra overhead could be also in percpu allocations, code etc. SLUB do not use large amount of percpu allocator I think, less than 30kB on such a small machine. Maybe also it would help reducing code size to disable CONFIG_MEMCG and CONFIG_TRACING, CONFIG_SLUB_DEBUG and CONFIG_SYSFS. I started from tinyconfig and enabled only necessary configs when testing in [1] (it's a bit laborious cuz pure tinyconfig does not even boot...). > >> This is with a buildroot kernel 5.19 build including a shell and sd-card > >> boot. With SLUB, I get clean boots and a shell prompt as expected. But I > >> definitely see more errors with shell commands failing due to allocation > >> failures for the shell process fork. So as far as the K210 is concerned, > >> switching to SLUB is not ideal. > >> > >> I would not want to hold on kernel mm improvements because of this toy > >> k210 though, so I am not going to prevent SLOB deprecation. I just wish > >> SLUB itself used less memory :) > > > > Did further tests with kernel 6.0.1: > > * SLOB: 630 free pages after boot, shell working (occasional shell fork > > failure happen though) > > * SLAB: getting memory allocation for order 7 failures on boot already > > (init process). Shell barely working (high frequency of shell command fork > > failures) > > * SLUB: getting memory allocation for order 7 failures on boot. I do get a > > shell prompt but cannot run any shell command that involves forking a new > > process. > > > > So if we want to keep the k210 support functional with a shell, we need > > slob. If we reduce that board support to only one application started as > > the init process, then I guess anything is OK. > > In [1] it was possible to save some more memory with more tuning. Some of > that required boot parameters and other code changes. In another reply [2] I > considered adding something like SLUB_TINY to take care of all that, so > looks like it would make sense to proceed with that. > > [1] > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ -- Thanks, Hyeonggon _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 11:50 ` Hyeonggon Yoo 0 siblings, 0 replies; 112+ messages in thread From: Hyeonggon Yoo @ 2022-11-14 11:50 UTC (permalink / raw) To: Vlastimil Babka Cc: Rich Felker, linux-sh, Tony Lindgren, Catalin Marinas, Roman Gushchin, Paul Cercueil, linux-mm, Conor.Dooley, Christoph Lameter, linux-riscv, Jonas Bonn, Alexander Shiyan, Aaro Koskinen, Damien Le Moal, Janusz Krzysztofik, Russell King, Matthew Wilcox, David Rientjes, Pasha Tatashin, Arnd Bergmann, Josh Triplett, openrisc, Joonsoo Kim, linux-arm-kernel, Yoshinori Sato, linux-kernel, Conor Dooley, Rustam Kovhaev, Pekka Enberg, Andrew Morton, Linus Torvalds On Mon, Nov 14, 2022 at 10:36:31AM +0100, Vlastimil Babka wrote: > On 11/14/22 06:48, Damien Le Moal wrote: > > On 11/14/22 10:55, Damien Le Moal wrote: > >> On 11/12/22 05:46, Conor Dooley wrote: > >>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > >>>> On 11/8/22 22:44, Pasha Tatashin wrote: > >>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> as we all know, we currently have three slab allocators. As we discussed > >>>>>> at LPC [1], it is my hope that one of these allocators has a future, and > >>>>>> two of them do not. > >>>>>> > >>>>>> The unsurprising reasons include code maintenance burden, other features > >>>>>> compatible with only a subset of allocators (or more effort spent on the > >>>>>> features), blocking API improvements (more on that below), and my > >>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >>>>>> without resorting to spelling out the letters. > >>>>>> > >>>>>> I think (but may be proven wrong) that SLOB is the easier target of the > >>>>>> two to be removed, so I'd like to focus on it first. > >>>>>> > >>>>>> I believe SLOB can be removed because: > >>>>>> > >>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint > >>>>>> by putting all objects together, which has its CPU performance costs > >>>>>> (locking, lack of percpu caching, searching for free space...). I'm not > >>>>>> aware of any "tiny linux" deployment that opts for this. For example, > >>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >>>>>> SLOB impact is too much for those who tried. Googling for > >>>>>> "CONFIG_SLOB=y" yielded nothing useful. > >>>>> > >>>>> I am all for removing SLOB. > >>>>> > >>>>> There are some devices with configs where SLOB is enabled by default. > >>>>> Perhaps, the owners/maintainers of those devices/configs should be > >>>>> included into this thread: > >>>>> > >>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> > >>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > >>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >>> > >>>> > >>>> Turns out that since SLOB depends on EXPERT, many of those lack it so > >>>> running make defconfig ends up with SLUB anyway, unless I miss something. > >>>> Only a subset has both SLOB and EXPERT: > >>>> > >>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >>> > >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > >>> > >>> I suppose there's not really a concern with the virt defconfig, but I > >>> did check the output of `make nommu_k210_defconfig" and despite not > >>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. > >>> > >>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > >>> boots etc, but I have no workloads or w/e to run on it. > >> > >> I sent a patch to change the k210 defconfig to using SLUB. However... > > Thanks! > > >> The current default config using SLOB gives about 630 free memory pages > >> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > >> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > somewhere, is that correct? > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > difference [1]. But that was looking at Slab pages, not free pages. IIRC overhead of s->min_partial (between 5 and 10) was pretty big because SLUB caches at most (s->min_partial) * (nr of caches) * (size of slab) bytes of unused memory. Passing slub_max_order=0 also may help a little bit. > The extra overhead could be also in percpu allocations, code etc. SLUB do not use large amount of percpu allocator I think, less than 30kB on such a small machine. Maybe also it would help reducing code size to disable CONFIG_MEMCG and CONFIG_TRACING, CONFIG_SLUB_DEBUG and CONFIG_SYSFS. I started from tinyconfig and enabled only necessary configs when testing in [1] (it's a bit laborious cuz pure tinyconfig does not even boot...). > >> This is with a buildroot kernel 5.19 build including a shell and sd-card > >> boot. With SLUB, I get clean boots and a shell prompt as expected. But I > >> definitely see more errors with shell commands failing due to allocation > >> failures for the shell process fork. So as far as the K210 is concerned, > >> switching to SLUB is not ideal. > >> > >> I would not want to hold on kernel mm improvements because of this toy > >> k210 though, so I am not going to prevent SLOB deprecation. I just wish > >> SLUB itself used less memory :) > > > > Did further tests with kernel 6.0.1: > > * SLOB: 630 free pages after boot, shell working (occasional shell fork > > failure happen though) > > * SLAB: getting memory allocation for order 7 failures on boot already > > (init process). Shell barely working (high frequency of shell command fork > > failures) > > * SLUB: getting memory allocation for order 7 failures on boot. I do get a > > shell prompt but cannot run any shell command that involves forking a new > > process. > > > > So if we want to keep the k210 support functional with a shell, we need > > slob. If we reduce that board support to only one application started as > > the init process, then I guess anything is OK. > > In [1] it was possible to save some more memory with more tuning. Some of > that required boot parameters and other code changes. In another reply [2] I > considered adding something like SLUB_TINY to take care of all that, so > looks like it would make sense to proceed with that. > > [1] > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ -- Thanks, Hyeonggon ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 11:50 ` Hyeonggon Yoo 0 siblings, 0 replies; 112+ messages in thread From: Hyeonggon Yoo @ 2022-11-14 11:50 UTC (permalink / raw) To: Vlastimil Babka Cc: Damien Le Moal, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On Mon, Nov 14, 2022 at 10:36:31AM +0100, Vlastimil Babka wrote: > On 11/14/22 06:48, Damien Le Moal wrote: > > On 11/14/22 10:55, Damien Le Moal wrote: > >> On 11/12/22 05:46, Conor Dooley wrote: > >>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > >>>> On 11/8/22 22:44, Pasha Tatashin wrote: > >>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> as we all know, we currently have three slab allocators. As we discussed > >>>>>> at LPC [1], it is my hope that one of these allocators has a future, and > >>>>>> two of them do not. > >>>>>> > >>>>>> The unsurprising reasons include code maintenance burden, other features > >>>>>> compatible with only a subset of allocators (or more effort spent on the > >>>>>> features), blocking API improvements (more on that below), and my > >>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >>>>>> without resorting to spelling out the letters. > >>>>>> > >>>>>> I think (but may be proven wrong) that SLOB is the easier target of the > >>>>>> two to be removed, so I'd like to focus on it first. > >>>>>> > >>>>>> I believe SLOB can be removed because: > >>>>>> > >>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint > >>>>>> by putting all objects together, which has its CPU performance costs > >>>>>> (locking, lack of percpu caching, searching for free space...). I'm not > >>>>>> aware of any "tiny linux" deployment that opts for this. For example, > >>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >>>>>> SLOB impact is too much for those who tried. Googling for > >>>>>> "CONFIG_SLOB=y" yielded nothing useful. > >>>>> > >>>>> I am all for removing SLOB. > >>>>> > >>>>> There are some devices with configs where SLOB is enabled by default. > >>>>> Perhaps, the owners/maintainers of those devices/configs should be > >>>>> included into this thread: > >>>>> > >>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> > >>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > >>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >>> > >>>> > >>>> Turns out that since SLOB depends on EXPERT, many of those lack it so > >>>> running make defconfig ends up with SLUB anyway, unless I miss something. > >>>> Only a subset has both SLOB and EXPERT: > >>>> > >>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >>> > >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > >>> > >>> I suppose there's not really a concern with the virt defconfig, but I > >>> did check the output of `make nommu_k210_defconfig" and despite not > >>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. > >>> > >>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > >>> boots etc, but I have no workloads or w/e to run on it. > >> > >> I sent a patch to change the k210 defconfig to using SLUB. However... > > Thanks! > > >> The current default config using SLOB gives about 630 free memory pages > >> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > >> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > somewhere, is that correct? > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > difference [1]. But that was looking at Slab pages, not free pages. IIRC overhead of s->min_partial (between 5 and 10) was pretty big because SLUB caches at most (s->min_partial) * (nr of caches) * (size of slab) bytes of unused memory. Passing slub_max_order=0 also may help a little bit. > The extra overhead could be also in percpu allocations, code etc. SLUB do not use large amount of percpu allocator I think, less than 30kB on such a small machine. Maybe also it would help reducing code size to disable CONFIG_MEMCG and CONFIG_TRACING, CONFIG_SLUB_DEBUG and CONFIG_SYSFS. I started from tinyconfig and enabled only necessary configs when testing in [1] (it's a bit laborious cuz pure tinyconfig does not even boot...). > >> This is with a buildroot kernel 5.19 build including a shell and sd-card > >> boot. With SLUB, I get clean boots and a shell prompt as expected. But I > >> definitely see more errors with shell commands failing due to allocation > >> failures for the shell process fork. So as far as the K210 is concerned, > >> switching to SLUB is not ideal. > >> > >> I would not want to hold on kernel mm improvements because of this toy > >> k210 though, so I am not going to prevent SLOB deprecation. I just wish > >> SLUB itself used less memory :) > > > > Did further tests with kernel 6.0.1: > > * SLOB: 630 free pages after boot, shell working (occasional shell fork > > failure happen though) > > * SLAB: getting memory allocation for order 7 failures on boot already > > (init process). Shell barely working (high frequency of shell command fork > > failures) > > * SLUB: getting memory allocation for order 7 failures on boot. I do get a > > shell prompt but cannot run any shell command that involves forking a new > > process. > > > > So if we want to keep the k210 support functional with a shell, we need > > slob. If we reduce that board support to only one application started as > > the init process, then I guess anything is OK. > > In [1] it was possible to save some more memory with more tuning. Some of > that required boot parameters and other code changes. In another reply [2] I > considered adding something like SLUB_TINY to take care of all that, so > looks like it would make sense to proceed with that. > > [1] > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ -- Thanks, Hyeonggon _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-14 11:50 ` Hyeonggon Yoo 0 siblings, 0 replies; 112+ messages in thread From: Hyeonggon Yoo @ 2022-11-14 11:50 UTC (permalink / raw) To: Vlastimil Babka Cc: Damien Le Moal, Conor Dooley, Pasha Tatashin, Christoph Lameter, David Rientjes, Joonsoo Kim, Pekka Enberg, Matthew Wilcox, Roman Gushchin, Linus Torvalds, linux-mm, linux-kernel, Catalin Marinas, Rustam Kovhaev, Andrew Morton, Josh Triplett, Arnd Bergmann, Russell King, Alexander Shiyan, Aaro Koskinen, Janusz Krzysztofik, Tony Lindgren, Yoshinori Sato, Rich Felker, Jonas Bonn, Stefan Kristiansson, Stafford Horne, linux-arm-kernel, openrisc, linux-riscv, linux-sh, Geert Uytterhoeven, Conor.Dooley, Paul Cercueil On Mon, Nov 14, 2022 at 10:36:31AM +0100, Vlastimil Babka wrote: > On 11/14/22 06:48, Damien Le Moal wrote: > > On 11/14/22 10:55, Damien Le Moal wrote: > >> On 11/12/22 05:46, Conor Dooley wrote: > >>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: > >>>> On 11/8/22 22:44, Pasha Tatashin wrote: > >>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka <vbabka@suse.cz> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> as we all know, we currently have three slab allocators. As we discussed > >>>>>> at LPC [1], it is my hope that one of these allocators has a future, and > >>>>>> two of them do not. > >>>>>> > >>>>>> The unsurprising reasons include code maintenance burden, other features > >>>>>> compatible with only a subset of allocators (or more effort spent on the > >>>>>> features), blocking API improvements (more on that below), and my > >>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, > >>>>>> without resorting to spelling out the letters. > >>>>>> > >>>>>> I think (but may be proven wrong) that SLOB is the easier target of the > >>>>>> two to be removed, so I'd like to focus on it first. > >>>>>> > >>>>>> I believe SLOB can be removed because: > >>>>>> > >>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint > >>>>>> by putting all objects together, which has its CPU performance costs > >>>>>> (locking, lack of percpu caching, searching for free space...). I'm not > >>>>>> aware of any "tiny linux" deployment that opts for this. For example, > >>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB > >>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance > >>>>>> SLOB impact is too much for those who tried. Googling for > >>>>>> "CONFIG_SLOB=y" yielded nothing useful. > >>>>> > >>>>> I am all for removing SLOB. > >>>>> > >>>>> There are some devices with configs where SLOB is enabled by default. > >>>>> Perhaps, the owners/maintainers of those devices/configs should be > >>>>> included into this thread: > >>>>> > >>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y > >>> > >>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y > >>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y > >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y > >>> > >>>> > >>>> Turns out that since SLOB depends on EXPERT, many of those lack it so > >>>> running make defconfig ends up with SLUB anyway, unless I miss something. > >>>> Only a subset has both SLOB and EXPERT: > >>>> > >>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` > >>> > >>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y > >>> > >>> I suppose there's not really a concern with the virt defconfig, but I > >>> did check the output of `make nommu_k210_defconfig" and despite not > >>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. > >>> > >>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still > >>> boots etc, but I have no workloads or w/e to run on it. > >> > >> I sent a patch to change the k210 defconfig to using SLUB. However... > > Thanks! > > >> The current default config using SLOB gives about 630 free memory pages > >> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about > >> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > somewhere, is that correct? > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > difference [1]. But that was looking at Slab pages, not free pages. IIRC overhead of s->min_partial (between 5 and 10) was pretty big because SLUB caches at most (s->min_partial) * (nr of caches) * (size of slab) bytes of unused memory. Passing slub_max_order=0 also may help a little bit. > The extra overhead could be also in percpu allocations, code etc. SLUB do not use large amount of percpu allocator I think, less than 30kB on such a small machine. Maybe also it would help reducing code size to disable CONFIG_MEMCG and CONFIG_TRACING, CONFIG_SLUB_DEBUG and CONFIG_SYSFS. I started from tinyconfig and enabled only necessary configs when testing in [1] (it's a bit laborious cuz pure tinyconfig does not even boot...). > >> This is with a buildroot kernel 5.19 build including a shell and sd-card > >> boot. With SLUB, I get clean boots and a shell prompt as expected. But I > >> definitely see more errors with shell commands failing due to allocation > >> failures for the shell process fork. So as far as the K210 is concerned, > >> switching to SLUB is not ideal. > >> > >> I would not want to hold on kernel mm improvements because of this toy > >> k210 though, so I am not going to prevent SLOB deprecation. I just wish > >> SLUB itself used less memory :) > > > > Did further tests with kernel 6.0.1: > > * SLOB: 630 free pages after boot, shell working (occasional shell fork > > failure happen though) > > * SLAB: getting memory allocation for order 7 failures on boot already > > (init process). Shell barely working (high frequency of shell command fork > > failures) > > * SLUB: getting memory allocation for order 7 failures on boot. I do get a > > shell prompt but cannot run any shell command that involves forking a new > > process. > > > > So if we want to keep the k210 support functional with a shell, we need > > slob. If we reduce that board support to only one application started as > > the init process, then I guess anything is OK. > > In [1] it was possible to save some more memory with more tuning. Some of > that required boot parameters and other code changes. In another reply [2] I > considered adding something like SLUB_TINY to take care of all that, so > looks like it would make sense to proceed with that. > > [1] > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ -- Thanks, Hyeonggon ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB @ 2022-11-09 20:56 Paul Cercueil 2022-11-09 21:39 ` Linus Torvalds 0 siblings, 1 reply; 112+ messages in thread From: Paul Cercueil @ 2022-11-09 20:56 UTC (permalink / raw) To: vbabka Cc: 42.hyeyoo, akpm, catalin.marinas, cl, iamjoonsoo.kim, linux-kernel, linux-mm, penberg, rientjes, rkovhaev, roman.gushchin, torvalds, willy Hi Vlastimil, I was actually using SLOB until recently for a device flasher program (kernel + initramfs + dtb, booted over USB) for Ingenic SoCs. I picked SLOB just because it said "embedded systems" in menuconfig and some of my boards have as little as 32 MiB RAM. It worked fine on some boards, but on others it had about a 25% chance of booting, and 75% chance of hanging at boot. I tried printk-debugging it, and was coming to the conclusion that it's memory corruption of some sort. Then I switched to SLUB and all the problems are gone. Same with SLAB. So while I can't say for sure that SLOB is broken (it might be triggering a bug somewhere else), I am highly suspicious that it is. So yeah... axe it. Cheers, -Paul ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-09 20:56 Paul Cercueil @ 2022-11-09 21:39 ` Linus Torvalds 2022-11-09 23:48 ` Aaro Koskinen 0 siblings, 1 reply; 112+ messages in thread From: Linus Torvalds @ 2022-11-09 21:39 UTC (permalink / raw) To: Paul Cercueil Cc: vbabka, 42.hyeyoo, akpm, catalin.marinas, cl, iamjoonsoo.kim, linux-kernel, linux-mm, penberg, rientjes, rkovhaev, roman.gushchin, willy On Wed, Nov 9, 2022 at 12:56 PM Paul Cercueil <paul@crapouillou.net> wrote: > > It worked fine on some boards, but on others it had about a 25% chance > of booting, and 75% chance of hanging at boot. I tried printk-debugging > it, and was coming to the conclusion that it's memory corruption of > some sort. > > Then I switched to SLUB and all the problems are gone. Same with SLAB. > > So while I can't say for sure that SLOB is broken (it might be > triggering a bug somewhere else), I am highly suspicious that it is. I have this distinct memory of having seen other reports like this, but my google-fu is not strong enough to back that up. There definitely has been recurring noise about SLOB issues. There's a reason people have wanted to remove it for years and years. Linus ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-09 21:39 ` Linus Torvalds @ 2022-11-09 23:48 ` Aaro Koskinen 2022-11-09 23:51 ` Aaro Koskinen 2022-11-10 4:40 ` Theodore Ts'o 0 siblings, 2 replies; 112+ messages in thread From: Aaro Koskinen @ 2022-11-09 23:48 UTC (permalink / raw) To: Linus Torvalds Cc: Paul Cercueil, vbabka, 42.hyeyoo, akpm, catalin.marinas, cl, iamjoonsoo.kim, linux-kernel, linux-mm, penberg, rientjes, rkovhaev, roman.gushchin, willy Hi, On Wed, Nov 09, 2022 at 01:39:12PM -0800, Linus Torvalds wrote: > On Wed, Nov 9, 2022 at 12:56 PM Paul Cercueil <paul@crapouillou.net> wrote: > > > > It worked fine on some boards, but on others it had about a 25% chance > > of booting, and 75% chance of hanging at boot. I tried printk-debugging > > it, and was coming to the conclusion that it's memory corruption of > > some sort. > > > > Then I switched to SLUB and all the problems are gone. Same with SLAB. > > > > So while I can't say for sure that SLOB is broken (it might be > > triggering a bug somewhere else), I am highly suspicious that it is. > > I have this distinct memory of having seen other reports like this, > but my google-fu is not strong enough to back that up. > > There definitely has been recurring noise about SLOB issues. There's a > reason people have wanted to remove it for years and years. Some of the reported SLOB issues have been actually real driver bugs, that go unnoticed when SLUB/SLAB are used (unless perhaps debug stuff is enabled). I'm not saying kernel should keep SLUB, but it's good at failing early when there is a bug. See e.g. commit 120ee599b5bf ("staging: octeon-usb: prevent memory corruption") A. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-09 23:48 ` Aaro Koskinen @ 2022-11-09 23:51 ` Aaro Koskinen 2022-11-10 4:40 ` Theodore Ts'o 1 sibling, 0 replies; 112+ messages in thread From: Aaro Koskinen @ 2022-11-09 23:51 UTC (permalink / raw) To: Linus Torvalds Cc: Paul Cercueil, vbabka, 42.hyeyoo, akpm, catalin.marinas, cl, iamjoonsoo.kim, linux-kernel, linux-mm, penberg, rientjes, rkovhaev, roman.gushchin, willy On Thu, Nov 10, 2022 at 01:48:35AM +0200, Aaro Koskinen wrote: > On Wed, Nov 09, 2022 at 01:39:12PM -0800, Linus Torvalds wrote: > > On Wed, Nov 9, 2022 at 12:56 PM Paul Cercueil <paul@crapouillou.net> wrote: > > > > > > It worked fine on some boards, but on others it had about a 25% chance > > > of booting, and 75% chance of hanging at boot. I tried printk-debugging > > > it, and was coming to the conclusion that it's memory corruption of > > > some sort. > > > > > > Then I switched to SLUB and all the problems are gone. Same with SLAB. > > > > > > So while I can't say for sure that SLOB is broken (it might be > > > triggering a bug somewhere else), I am highly suspicious that it is. > > > > I have this distinct memory of having seen other reports like this, > > but my google-fu is not strong enough to back that up. > > > > There definitely has been recurring noise about SLOB issues. There's a > > reason people have wanted to remove it for years and years. > > Some of the reported SLOB issues have been actually real driver bugs, > that go unnoticed when SLUB/SLAB are used (unless perhaps debug stuff > is enabled). I'm not saying kernel should keep SLUB, but it's good at ^^^^ SLOB, sorry for a typo. > failing early when there is a bug. See e.g. commit 120ee599b5bf ("staging: > octeon-usb: prevent memory corruption") > > A. ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-09 23:48 ` Aaro Koskinen 2022-11-09 23:51 ` Aaro Koskinen @ 2022-11-10 4:40 ` Theodore Ts'o 2022-11-10 7:31 ` Vlastimil Babka 1 sibling, 1 reply; 112+ messages in thread From: Theodore Ts'o @ 2022-11-10 4:40 UTC (permalink / raw) To: Aaro Koskinen Cc: Linus Torvalds, Paul Cercueil, vbabka, 42.hyeyoo, akpm, catalin.marinas, cl, iamjoonsoo.kim, linux-kernel, linux-mm, penberg, rientjes, rkovhaev, roman.gushchin, willy On Thu, Nov 10, 2022 at 01:48:32AM +0200, Aaro Koskinen wrote: > > Some of the reported SLOB issues have been actually real driver bugs, > that go unnoticed when SLUB/SLAB are used (unless perhaps debug stuff > is enabled). I'm not saying kernel should keep SLOB, but it's good at > failing early when there is a bug. See e.g. commit 120ee599b5bf ("staging: > octeon-usb: prevent memory corruption") Out of curiosity, are these bugs that would have been found using KASAN or some of the other kernel sanitizers and/or other debugging tools we have at our disposal? - Ted ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-10 4:40 ` Theodore Ts'o @ 2022-11-10 7:31 ` Vlastimil Babka 2022-11-10 7:54 ` Feng Tang 2022-11-10 16:20 ` Matthew Wilcox 0 siblings, 2 replies; 112+ messages in thread From: Vlastimil Babka @ 2022-11-10 7:31 UTC (permalink / raw) To: Theodore Ts'o, Aaro Koskinen Cc: Linus Torvalds, Paul Cercueil, 42.hyeyoo, akpm, catalin.marinas, cl, iamjoonsoo.kim, linux-kernel, linux-mm, penberg, rientjes, rkovhaev, roman.gushchin, willy, Feng Tang On 11/10/22 05:40, Theodore Ts'o wrote: > On Thu, Nov 10, 2022 at 01:48:32AM +0200, Aaro Koskinen wrote: >> >> Some of the reported SLOB issues have been actually real driver bugs, >> that go unnoticed when SLUB/SLAB are used (unless perhaps debug stuff >> is enabled). I'm not saying kernel should keep SLOB, but it's good at >> failing early when there is a bug. See e.g. commit 120ee599b5bf ("staging: >> octeon-usb: prevent memory corruption") > > Out of curiosity, are these bugs that would have been found using > KASAN or some of the other kernel sanitizers and/or other debugging > tools we have at our disposal? Hopefully slub_debug redzoning would be able to trigger the bug described in commit 120ee599b5bf above, which is: > octeon-hcd will crash the kernel when SLOB is used. This usually happens > after the 18-byte control transfer when a device descriptor is read. > The DMA engine is always transfering full 32-bit words and if the > transfer is shorter, some random garbage appears after the buffer. > The problem is not visible with SLUB since it rounds up the allocations > to word boundary, and the extra bytes will go undetected. Ah, actually it wouldn't *now* as SLUB would make the allocation fall into kmalloc-32 cache and only add redzone beyond 32 bytes. But with upcoming changes by Feng Tang, this should work. slub_debug would also have a chance of catching buffer overflows by kernel code itself, not DMA, and tell you about it more sooner and gracefully than crashing. KASAN also, even with a higher chance and precision, if it's available for your arch and your device constraints can tolerate its larger overhead. > - Ted > > ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-10 7:31 ` Vlastimil Babka @ 2022-11-10 7:54 ` Feng Tang 2022-11-10 16:20 ` Matthew Wilcox 1 sibling, 0 replies; 112+ messages in thread From: Feng Tang @ 2022-11-10 7:54 UTC (permalink / raw) To: Vlastimil Babka Cc: Theodore Ts'o, Aaro Koskinen, Torvalds, Linus, Paul Cercueil, 42.hyeyoo, akpm, catalin.marinas, cl, iamjoonsoo.kim, linux-kernel, linux-mm, penberg, rientjes, rkovhaev, roman.gushchin, willy On Thu, Nov 10, 2022 at 03:31:31PM +0800, Vlastimil Babka wrote: > On 11/10/22 05:40, Theodore Ts'o wrote: > > On Thu, Nov 10, 2022 at 01:48:32AM +0200, Aaro Koskinen wrote: > >> > >> Some of the reported SLOB issues have been actually real driver bugs, > >> that go unnoticed when SLUB/SLAB are used (unless perhaps debug stuff > >> is enabled). I'm not saying kernel should keep SLOB, but it's good at > >> failing early when there is a bug. See e.g. commit 120ee599b5bf ("staging: > >> octeon-usb: prevent memory corruption") > > > > Out of curiosity, are these bugs that would have been found using > > KASAN or some of the other kernel sanitizers and/or other debugging > > tools we have at our disposal? > > Hopefully slub_debug redzoning would be able to trigger the bug described in > commit 120ee599b5bf above, which is: > > > octeon-hcd will crash the kernel when SLOB is used. This usually happens > > after the 18-byte control transfer when a device descriptor is read. > > The DMA engine is always transfering full 32-bit words and if the > > transfer is shorter, some random garbage appears after the buffer. > > The problem is not visible with SLUB since it rounds up the allocations > > to word boundary, and the extra bytes will go undetected. > > Ah, actually it wouldn't *now* as SLUB would make the allocation fall into > kmalloc-32 cache and only add redzone beyond 32 bytes. But with upcoming > changes by Feng Tang, this should work. I wrote a simple case trying simulating this: static noinline void dma_align_test(void) { char *buf; buf = kmalloc(18, GFP_KERNEL); buf[18] = 0; buf[19] = 0; kfree(buf); } And with slub_debug on and the slub_redzone patchset[1], it did catch the out-of-bound access, as in the dmesg: " ============================================================================= BUG kmalloc-32 (Not tainted): kmalloc Redzone overwritten ----------------------------------------------------------------------------- 0xffff888005ebb032-0xffff888005ebb033 @offset=50. First byte 0x0 instead of 0xcc Allocated in dma_align_test+0x1b/0x29 age=6554 cpu=1 pid=1 __kmem_cache_alloc_node+0x2a7/0x320 kmalloc_trace+0x27/0xa0 dma_align_test+0x1b/0x29 late_slub_debug+0xa/0x11 do_one_initcall+0x87/0x2a0 ... Slab 0xffffea000017aec0 objects=21 used=19 fp=0xffff888005ebbf20 flags=0xfffffc0000200(slab|node=0|zone=1|lastcpupid=0x1fffff) Object 0xffff888005ebb020 @offset=32 fp=0x0000000000000000 Redzone ffff888005ebb000: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................ Redzone ffff888005ebb010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................ Object ffff888005ebb020: 50 92 28 00 81 88 ff ff 01 00 00 00 11 00 b6 07 P.(............. Object ffff888005ebb030: 6b a5 00 00 cc cc cc cc cc cc cc cc cc cc cc cc k............... Redzone ffff888005ebb040: cc cc cc cc cc cc cc cc ........ Padding ffff888005ebb0a4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ Padding ffff888005ebb0b4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZ " [1]. https://lore.kernel.org/lkml/20221021032405.1825078-1-feng.tang@intel.com/ Thanks, Feng > slub_debug would also have a chance of catching buffer overflows by kernel > code itself, not DMA, and tell you about it more sooner and gracefully than > crashing. KASAN also, even with a higher chance and precision, if it's > available for your arch and your device constraints can tolerate its larger > overhead. > > > - Ted > > > > > > ^ permalink raw reply [flat|nested] 112+ messages in thread
* Re: Deprecating and removing SLOB 2022-11-10 7:31 ` Vlastimil Babka 2022-11-10 7:54 ` Feng Tang @ 2022-11-10 16:20 ` Matthew Wilcox 2022-11-11 9:37 ` David Laight 1 sibling, 1 reply; 112+ messages in thread From: Matthew Wilcox @ 2022-11-10 16:20 UTC (permalink / raw) To: Vlastimil Babka Cc: Theodore Ts'o, Aaro Koskinen, Linus Torvalds, Paul Cercueil, 42.hyeyoo, akpm, catalin.marinas, cl, iamjoonsoo.kim, linux-kernel, linux-mm, penberg, rientjes, rkovhaev, roman.gushchin, Feng Tang On Thu, Nov 10, 2022 at 08:31:31AM +0100, Vlastimil Babka wrote: > > octeon-hcd will crash the kernel when SLOB is used. This usually happens > > after the 18-byte control transfer when a device descriptor is read. > > The DMA engine is always transfering full 32-bit words and if the > > transfer is shorter, some random garbage appears after the buffer. > > The problem is not visible with SLUB since it rounds up the allocations > > to word boundary, and the extra bytes will go undetected. > > Ah, actually it wouldn't *now* as SLUB would make the allocation fall into > kmalloc-32 cache and only add redzone beyond 32 bytes. But with upcoming > changes by Feng Tang, this should work. This is kind of "if a bug stings a tree in a forest, does it hurt" problem. If all allocations of 18 bytes are rounded up to 20 or more bytes, then it doesn't matter that the device has this bug. Sure, it may end up hurting in the future if we decide to create 18-byte slab caches, but it's not actually going to affect anything today (and we seem to be moving towards less precision in order to get more performance) ^ permalink raw reply [flat|nested] 112+ messages in thread
* RE: Deprecating and removing SLOB 2022-11-10 16:20 ` Matthew Wilcox @ 2022-11-11 9:37 ` David Laight 0 siblings, 0 replies; 112+ messages in thread From: David Laight @ 2022-11-11 9:37 UTC (permalink / raw) To: 'Matthew Wilcox', Vlastimil Babka Cc: Theodore Ts'o, Aaro Koskinen, Linus Torvalds, Paul Cercueil, 42.hyeyoo, akpm, catalin.marinas, cl, iamjoonsoo.kim, linux-kernel, linux-mm, penberg, rientjes, rkovhaev, roman.gushchin, Feng Tang From: Matthew Wilcox > Sent: 10 November 2022 16:20 > > On Thu, Nov 10, 2022 at 08:31:31AM +0100, Vlastimil Babka wrote: > > > octeon-hcd will crash the kernel when SLOB is used. This usually happens > > > after the 18-byte control transfer when a device descriptor is read. > > > The DMA engine is always transfering full 32-bit words and if the > > > transfer is shorter, some random garbage appears after the buffer. > > > The problem is not visible with SLUB since it rounds up the allocations > > > to word boundary, and the extra bytes will go undetected. > > > > Ah, actually it wouldn't *now* as SLUB would make the allocation fall into > > kmalloc-32 cache and only add redzone beyond 32 bytes. But with upcoming > > changes by Feng Tang, this should work. > > This is kind of "if a bug stings a tree in a forest, does it hurt" > problem. If all allocations of 18 bytes are rounded up to 20 or more > bytes, then it doesn't matter that the device has this bug. Sure, it > may end up hurting in the future if we decide to create 18-byte slab > caches, but it's not actually going to affect anything today (and we > seem to be moving towards less precision in order to get more > performance) Yes, even on dma-coherent systems allocated blocks have to be moderately aligned - so the space after an 18 byte block can't be used. I also doubt there is any benefit (and many bugs) from allowing 2 bytes alignment on m68k. So the 'overwrite to a whole number of words' maybe reasonably expected to not cause any real bugs. x86 (even 32bit) probably requires 16 byte alignment (for some corner cases) - ok for a power-of-2 allocator that doesn't add a header. (Although 1, 2, 4 and 8 byte allocates are valid.) To reduce memory wastage what you really don't want is an allocator that adds a header/trailer and then rounds up to a power of 2. Coders write in binary and do kmalloc(256) not kmalloc(200) and rounding 256 up to 512 is rather wasteful. (Search for the kmalloc(PAGE_SIZE+1) :-) I also think that one of the allocators only cuts pages into power-of-2 sizes. It is probably sensible to return cache-aligned (probably 64 byte) buffers for requests larger than a cache line. But a 4k page can be split into 21 192-byte buffers. As well as using less memory for allocates between 129 and 192 bytes it may reduce pressure on the d-cache by evening out cache line usage. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales) ^ permalink raw reply [flat|nested] 112+ messages in thread
end of thread, other threads:[~2022-11-21 17:04 UTC | newest] Thread overview: 112+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-11-08 15:55 Deprecating and removing SLOB Vlastimil Babka 2022-11-08 18:18 ` Christophe Leroy 2022-11-08 19:17 ` Andrew Morton 2022-11-08 18:46 ` Roman Gushchin 2022-11-08 20:13 ` Yosry Ahmed 2022-11-09 9:09 ` Vlastimil Babka 2022-11-08 21:44 ` Pasha Tatashin 2022-11-09 9:00 ` Vlastimil Babka 2022-11-09 9:00 ` Vlastimil Babka 2022-11-09 9:00 ` Vlastimil Babka 2022-11-09 9:00 ` Vlastimil Babka 2022-11-09 15:50 ` Aaro Koskinen 2022-11-09 15:50 ` Aaro Koskinen 2022-11-09 15:50 ` Aaro Koskinen 2022-11-09 15:50 ` Aaro Koskinen 2022-11-09 16:45 ` Geert Uytterhoeven 2022-11-09 16:45 ` Geert Uytterhoeven 2022-11-09 16:45 ` Geert Uytterhoeven 2022-11-09 16:45 ` Geert Uytterhoeven 2022-11-09 17:45 ` Mike Rapoport 2022-11-09 17:45 ` Mike Rapoport 2022-11-09 17:45 ` Mike Rapoport 2022-11-09 17:45 ` Mike Rapoport 2022-11-09 21:16 ` Janusz Krzysztofik 2022-11-09 21:16 ` Janusz Krzysztofik 2022-11-09 21:16 ` Janusz Krzysztofik 2022-11-09 21:16 ` Janusz Krzysztofik 2022-11-09 17:57 ` Conor.Dooley 2022-11-09 23:00 ` Damien Le Moal 2022-11-11 10:25 ` Vlastimil Babka 2022-11-12 1:40 ` Damien Le Moal 2022-11-11 10:33 ` Vlastimil Babka 2022-11-11 10:33 ` Vlastimil Babka 2022-11-11 10:33 ` Vlastimil Babka 2022-11-11 10:33 ` Vlastimil Babka 2022-11-11 12:49 ` [lkp] [+5395 bytes kernel size regression] [i386-tinyconfig] [b7c8731082] " kernel test robot 2022-11-11 17:24 ` Josh Triplett 2022-11-11 21:11 ` Vlastimil Babka 2022-11-17 8:57 ` Vlastimil Babka 2022-11-11 20:46 ` Conor Dooley 2022-11-11 20:46 ` Conor Dooley 2022-11-11 20:46 ` Conor Dooley 2022-11-11 20:46 ` Conor Dooley 2022-11-12 1:40 ` Damien Le Moal 2022-11-12 1:40 ` Damien Le Moal 2022-11-12 1:40 ` Damien Le Moal 2022-11-12 1:40 ` Damien Le Moal 2022-11-14 1:55 ` Damien Le Moal 2022-11-14 1:55 ` Damien Le Moal 2022-11-14 1:55 ` Damien Le Moal 2022-11-14 1:55 ` Damien Le Moal 2022-11-14 5:48 ` Damien Le Moal 2022-11-14 5:48 ` Damien Le Moal 2022-11-14 5:48 ` Damien Le Moal 2022-11-14 5:48 ` Damien Le Moal 2022-11-14 9:36 ` Vlastimil Babka 2022-11-14 9:36 ` Vlastimil Babka 2022-11-14 9:36 ` Vlastimil Babka 2022-11-14 9:36 ` Vlastimil Babka 2022-11-14 11:35 ` Damien Le Moal 2022-11-14 11:35 ` Damien Le Moal 2022-11-14 11:35 ` Damien Le Moal 2022-11-14 11:35 ` Damien Le Moal 2022-11-14 14:47 ` Hyeonggon Yoo 2022-11-14 14:47 ` Hyeonggon Yoo 2022-11-14 14:47 ` Hyeonggon Yoo 2022-11-14 14:47 ` Hyeonggon Yoo 2022-11-15 4:24 ` Damien Le Moal 2022-11-15 4:24 ` Damien Le Moal 2022-11-15 4:24 ` Damien Le Moal 2022-11-15 4:24 ` Damien Le Moal 2022-11-15 4:28 ` Damien Le Moal 2022-11-15 4:28 ` Damien Le Moal 2022-11-15 4:28 ` Damien Le Moal 2022-11-15 4:28 ` Damien Le Moal 2022-11-16 7:57 ` Matthew Wilcox 2022-11-16 7:57 ` Matthew Wilcox 2022-11-16 7:57 ` Matthew Wilcox 2022-11-16 7:57 ` Matthew Wilcox 2022-11-16 8:02 ` Damien Le Moal 2022-11-16 8:02 ` Damien Le Moal 2022-11-16 8:02 ` Damien Le Moal 2022-11-16 8:02 ` Damien Le Moal 2022-11-16 17:51 ` Vlastimil Babka 2022-11-16 17:51 ` Vlastimil Babka 2022-11-16 17:51 ` Vlastimil Babka 2022-11-16 17:51 ` Vlastimil Babka 2022-11-17 0:22 ` Damien Le Moal 2022-11-17 0:22 ` Damien Le Moal 2022-11-17 0:22 ` Damien Le Moal 2022-11-17 0:22 ` Damien Le Moal 2022-11-21 4:30 ` Damien Le Moal 2022-11-21 4:30 ` Damien Le Moal 2022-11-21 4:30 ` Damien Le Moal 2022-11-21 4:30 ` Damien Le Moal 2022-11-21 17:02 ` Vlastimil Babka 2022-11-21 17:02 ` Vlastimil Babka 2022-11-21 17:02 ` Vlastimil Babka 2022-11-21 17:02 ` Vlastimil Babka 2022-11-14 11:50 ` Hyeonggon Yoo 2022-11-14 11:50 ` Hyeonggon Yoo 2022-11-14 11:50 ` Hyeonggon Yoo 2022-11-14 11:50 ` Hyeonggon Yoo 2022-11-09 20:56 Paul Cercueil 2022-11-09 21:39 ` Linus Torvalds 2022-11-09 23:48 ` Aaro Koskinen 2022-11-09 23:51 ` Aaro Koskinen 2022-11-10 4:40 ` Theodore Ts'o 2022-11-10 7:31 ` Vlastimil Babka 2022-11-10 7:54 ` Feng Tang 2022-11-10 16:20 ` Matthew Wilcox 2022-11-11 9:37 ` David Laight
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.