[PATCH] dma-debug: increase HASH_SIZE

[PATCH] dma-debug: increase HASH_SIZE

[Eric Dumazet]

With modern NIC, it is not unusual having about ~256,000 active dma
mappings. Hash size of 1024 buckets is too small.

Forcing full cache line per bucket does not seem useful,
especially now that we have a contention on free_entries_lock
for allocations and freeing of entries. Better using space
to fit more buckets.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Marek Szyprowski <m.szyprowski@samsung.com>
---
 kernel/dma/debug.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/dma/debug.c b/kernel/dma/debug.c
index 4ad74f5987ea9e95f9bb5e2d1592254e367d24fb..35e2a853bff9c482d789ab331d79aaee07753a97 100644
--- a/kernel/dma/debug.c
+++ b/kernel/dma/debug.c
@@ -27,7 +27,7 @@
 
 #include <asm/sections.h>
 
-#define HASH_SIZE       1024ULL
+#define HASH_SIZE       16384ULL
 #define HASH_FN_SHIFT   13
 #define HASH_FN_MASK    (HASH_SIZE - 1)
 
@@ -87,7 +87,7 @@ typedef bool (*match_fn)(struct dma_debug_entry *, struct dma_debug_entry *);
 struct hash_bucket {
 	struct list_head list;
 	spinlock_t lock;
-} ____cacheline_aligned_in_smp;
+};
 
 /* Hash list to save the allocated dma addresses */
 static struct hash_bucket dma_entry_hash[HASH_SIZE];
-- 
2.24.0.rc0.303.g954a862665-goog

Re: [PATCH] dma-debug: increase HASH_SIZE

[Christoph Hellwig]

Thanks,

applied to the dma-mapping tree for 5.5.

Re: [PATCH] dma-debug: increase HASH_SIZE

[Geert Uytterhoeven]

On Wed, Oct 30, 2019 at 8:13 PM Eric Dumazet <edumazet@google.com> wrote:
> With modern NIC, it is not unusual having about ~256,000 active dma
> mappings. Hash size of 1024 buckets is too small.
>
> Forcing full cache line per bucket does not seem useful,
> especially now that we have a contention on free_entries_lock
> for allocations and freeing of entries. Better using space
> to fit more buckets.
>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Marek Szyprowski <m.szyprowski@samsung.com>
> ---
>  kernel/dma/debug.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/dma/debug.c b/kernel/dma/debug.c
> index 4ad74f5987ea9e95f9bb5e2d1592254e367d24fb..35e2a853bff9c482d789ab331d79aaee07753a97 100644
> --- a/kernel/dma/debug.c
> +++ b/kernel/dma/debug.c
> @@ -27,7 +27,7 @@
>
>  #include <asm/sections.h>
>
> -#define HASH_SIZE       1024ULL
> +#define HASH_SIZE       16384ULL
>  #define HASH_FN_SHIFT   13
>  #define HASH_FN_MASK    (HASH_SIZE - 1)
>
> @@ -87,7 +87,7 @@ typedef bool (*match_fn)(struct dma_debug_entry *, struct dma_debug_entry *);
>  struct hash_bucket {
>         struct list_head list;
>         spinlock_t lock;
> -} ____cacheline_aligned_in_smp;
> +};
>
>  /* Hash list to save the allocated dma addresses */
>  static struct hash_bucket dma_entry_hash[HASH_SIZE];

JFTR, this increases dma_entry_hash size by 327680 bytes, and pushes
a few more boards beyond their bootloader-imposed kernel size limits.

Disabling CONFIG_DMA_API_DEBUG fixes that.
Of course the real fix is to fix the bootloaders...

Gr{oetje,eeting}s,

                        Geert


--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH] dma-debug: increase HASH_SIZE

[Eric Dumazet]

On Tue, Dec 10, 2019 at 6:55 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote:

> JFTR, this increases dma_entry_hash size by 327680 bytes, and pushes
> a few more boards beyond their bootloader-imposed kernel size limits.
>
> Disabling CONFIG_DMA_API_DEBUG fixes that.
> Of course the real fix is to fix the bootloaders...
>

Hi Geert

Maybe we can make the hash size arch-dependent, or better dynamically
allocate this memory ?

Re: [PATCH] dma-debug: increase HASH_SIZE

[Geert Uytterhoeven]

Hi Eric,

On Tue, Dec 10, 2019 at 4:04 PM Eric Dumazet <edumazet@google.com> wrote:
> On Tue, Dec 10, 2019 at 6:55 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> > JFTR, this increases dma_entry_hash size by 327680 bytes, and pushes
> > a few more boards beyond their bootloader-imposed kernel size limits.
> >
> > Disabling CONFIG_DMA_API_DEBUG fixes that.
> > Of course the real fix is to fix the bootloaders...
>
> Maybe we can make the hash size arch-dependent, or better dynamically
> allocate this memory ?

Dynamically allocating would be the better solution, IMHO.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH] dma-debug: increase HASH_SIZE

[Christoph Hellwig]

On Tue, Dec 10, 2019 at 03:55:08PM +0100, Geert Uytterhoeven wrote:
> On Wed, Oct 30, 2019 at 8:13 PM Eric Dumazet <edumazet@google.com> wrote:
> > With modern NIC, it is not unusual having about ~256,000 active dma
> > mappings. Hash size of 1024 buckets is too small.
> >
> > Forcing full cache line per bucket does not seem useful,
> > especially now that we have a contention on free_entries_lock
> > for allocations and freeing of entries. Better using space
> > to fit more buckets.
> >
> > Signed-off-by: Eric Dumazet <edumazet@google.com>
> > Cc: Christoph Hellwig <hch@lst.de>
> > Cc: Marek Szyprowski <m.szyprowski@samsung.com>
> > ---
> >  kernel/dma/debug.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/kernel/dma/debug.c b/kernel/dma/debug.c
> > index 4ad74f5987ea9e95f9bb5e2d1592254e367d24fb..35e2a853bff9c482d789ab331d79aaee07753a97 100644
> > --- a/kernel/dma/debug.c
> > +++ b/kernel/dma/debug.c
> > @@ -27,7 +27,7 @@
> >
> >  #include <asm/sections.h>
> >
> > -#define HASH_SIZE       1024ULL
> > +#define HASH_SIZE       16384ULL
> >  #define HASH_FN_SHIFT   13
> >  #define HASH_FN_MASK    (HASH_SIZE - 1)
> >
> > @@ -87,7 +87,7 @@ typedef bool (*match_fn)(struct dma_debug_entry *, struct dma_debug_entry *);
> >  struct hash_bucket {
> >         struct list_head list;
> >         spinlock_t lock;
> > -} ____cacheline_aligned_in_smp;
> > +};
> >
> >  /* Hash list to save the allocated dma addresses */
> >  static struct hash_bucket dma_entry_hash[HASH_SIZE];
> 
> JFTR, this increases dma_entry_hash size by 327680 bytes, and pushes
> a few more boards beyond their bootloader-imposed kernel size limits.
> 
> Disabling CONFIG_DMA_API_DEBUG fixes that.
> Of course the real fix is to fix the bootloaders...

Can someone just send a patch to switch this to a dynamic allocation?

Re: [PATCH] dma-debug: increase HASH_SIZE

[Eric Dumazet]

On Thu, Jan 30, 2020 at 8:27 AM Christoph Hellwig <hch@lst.de> wrote:
>
> On Tue, Dec 10, 2019 at 03:55:08PM +0100, Geert Uytterhoeven wrote:

>
> Can someone just send a patch to switch this to a dynamic allocation?

Yes I can do that, thanks for the reminder.