Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN

From: Arnd Bergmann <arnd@arndb.de>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>,
	Ard Biesheuvel <ardb@kernel.org>,
	Herbert Xu <herbert@gondor.apana.org.au>,
	Will Deacon <will@kernel.org>, Marc Zyngier <maz@kernel.org>,
	Arnd Bergmann <arnd@arndb.de>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Linux Memory Management List <linux-mm@kvack.org>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	"David S. Miller" <davem@davemloft.net>
Subject: Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN
Date: Fri, 15 Apr 2022 13:09:15 +0200	[thread overview]
Message-ID: <CAK8P3a3j2_8WjZCdj3mEtGnKoRDR4qsckx9igEM70AUG+4VgRA@mail.gmail.com> (raw)
In-Reply-To: <CAHk-=wjwupOLq3xh8z02CpXe8-=Lb_z0kXXQJYQ5YaQPpi3MiA@mail.gmail.com>

On Fri, Apr 15, 2022 at 12:25 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> >
> And it could be that if I have 150k of those smallish allocations, a
> server with lots of active users might have millions. Not having
> looked at where they come from, maybe that isn't the case, but it
> *might* be.
>
> Maybe adding something like a
>
>         static int warn_every_1k = 0;
>         WARN_ON(size < 32 && (1023 & ++warn_every_1k));
>
> to kmalloc() would give us a statistical view of "lots of these small
> allocations" thing, and we could add GFP_NODMA to them. There probably
> aren't that many places that have those small allocations, and it's
> certainly safer to annotate "this is not for DMA" than have the
> requirement that all DMA allocations must be marked.

I think finding out the allocations is one of the most common examples
for ftrace. I followed the instructions from
https://www.kernel.org/doc/Documentation/trace/events.txt to
show me a histogram of all allocations under 256 bytes, which
(one kernel compile later) gives me something like

$echo 'hist:key=call_site.sym-offset:val=bytes_req:sort=bytes_req.descending
if bytes_req<256' >   \
/sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
$ make -skj30
...
$ head  /sys/kernel/debug/tracing/events/kmem/kmalloc/hist
{ call_site: [ffffffffc04e457f]
btrfs_delete_delayed_dir_index+0xbf/0x1e0 [btrfs]       } hitcount:
 146914  bytes_req:   16454368
{ call_site: [ffffffffbbe601a3] generic_file_buffered_read+0x463/0x4a0
                 } hitcount:      98187  bytes_req:   14906232
{ call_site: [ffffffffc0497b81] btrfs_buffered_write+0x131/0x7e0
[btrfs]                } hitcount:     156513  bytes_req:   10038544
{ call_site: [ffffffffc05125c9] btrfs_alloc_block_rsv+0x29/0x60
[btrfs]                 } hitcount:     155044  bytes_req:    8682464
{ call_site: [ffffffffbbfe7272] kernfs_fop_open+0xc2/0x290
                 } hitcount:      38764  bytes_req:    5892128
{ call_site: [ffffffffbbfb6ea2] load_elf_binary+0x242/0xed0
                 } hitcount:      58276  bytes_req:    3729664
{ call_site: [ffffffffc04b52d0] __btrfs_map_block+0x1f0/0xb60 [btrfs]
                 } hitcount:      29289  bytes_req:    3521656
{ call_site: [ffffffffbbf7ac7e] inotify_handle_inode_event+0x7e/0x210
                 } hitcount:      61688  bytes_req:    2986992
{ call_site: [ffffffffbbf2fa35] alloc_pipe_info+0x65/0x230
                 } hitcount:      13139  bytes_req:    2312464
{ call_site: [ffffffffbc0cd3ec] security_task_alloc+0x9c/0x100
                 } hitcount:      60475  bytes_req:    2177100
{ call_site: [ffffffffbc0cd5f6] security_prepare_creds+0x76/0xa0
                 } hitcount:     266124  bytes_req:    2128992
{ call_site: [ffffffffbbfe710e] kernfs_get_open_node+0x7e/0x120
                 } hitcount:      38764  bytes_req:    1860672
{ call_site: [ffffffffc04e1fbd] btrfs_alloc_delayed_item+0x1d/0x50
[btrfs]              } hitcount:      11859  bytes_req:    1833383
{ call_site: [ffffffffc046595d] split_item+0x8d/0x2e0 [btrfs]
                 } hitcount:      14049  bytes_req:    1716288
{ call_site: [ffffffffbbfb6dbc] load_elf_binary+0x15c/0xed0
                 } hitcount:      58276  bytes_req:    1631728
{ call_site: [ffffffffbbf40e79] __d_alloc+0x179/0x1f0
                 } hitcount:      24814  bytes_req:    1280649
{ call_site: [ffffffffbbf5203f] single_open+0x2f/0xa0
                 } hitcount:      34541  bytes_req:    1105312
{ call_site: [ffffffffc047ad0a] btrfs_wq_submit_bio+0x4a/0xe0 [btrfs]
                 } hitcount:       7746  bytes_req:    1053456
{ call_site: [ffffffffbc519e95] xhci_urb_enqueue+0xf5/0x3c0
                 } hitcount:       5511  bytes_req:     484968
{ call_site: [ffffffffc0482935] btrfs_opendir+0x25/0x70 [btrfs]
                 } hitcount:      60245  bytes_req:     481960
{ call_site: [ffffffffc04c44ff] overwrite_item+0x1cf/0x5c0 [btrfs]
                 } hitcount:       7378  bytes_req:     364305
{ call_site: [ffffffffc04c4514] overwrite_item+0x1e4/0x5c0 [btrfs]
                 } hitcount:       7378  bytes_req:     364305
{ call_site: [ffffffffc04e207f] btrfs_wq_run_delayed_node+0x2f/0x80
[btrfs]             } hitcount:       3427  bytes_req:     356408
{ call_site: [ffffffffbbe7e96d] shmem_symlink+0xbd/0x250
                 } hitcount:       5169  bytes_req:     242943
{ call_site: [ffffffffc03e0526] hid_input_field+0x56/0x290 [hid]
                 } hitcount:      11004  bytes_req:     175760

I think these are all safe for the GFP_NODMA approach you suggest, maybe
not the xhci_urb_enqueue one.

          Arnd