From: "Rafał Miłecki" <zajec5@gmail.com>
To: Felix Fietkau <nbd@nbd.name>, Andrew Lunn <andrew@lunn.ch>
Cc: Arnd Bergmann <arnd@arndb.de>,
Alexander Lobakin <alexandr.lobakin@intel.com>,
Network Development <netdev@vger.kernel.org>,
linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
Russell King <linux@armlinux.org.uk>,
"openwrt-devel@lists.openwrt.org"
<openwrt-devel@lists.openwrt.org>,
Florian Fainelli <f.fainelli@gmail.com>
Subject: Re: Optimizing kernel compilation / alignments for network performance
Date: Fri, 6 May 2022 09:47:41 +0200 [thread overview]
Message-ID: <04fa6560-e6f4-005f-cddb-7bc9b4859ba2@gmail.com> (raw)
In-Reply-To: <2a338e8e-3288-859c-d2e8-26c5712d3d06@nbd.name>
On 5.05.2022 18:46, Felix Fietkau wrote:
>
> On 05.05.22 18:04, Andrew Lunn wrote:
>>> you'll see that most used functions are:
>>> v7_dma_inv_range
>>> __irqentry_text_end
>>> l2c210_inv_range
>>> v7_dma_clean_range
>>> bcma_host_soc_read32
>>> __netif_receive_skb_core
>>> arch_cpu_idle
>>> l2c210_clean_range
>>> fib_table_lookup
>>
>> There is a lot of cache management functions here. Might sound odd,
>> but have you tried disabling SMP? These cache functions need to
>> operate across all CPUs, and the communication between CPUs can slow
>> them down. If there is only one CPU, these cache functions get simpler
>> and faster.
>>
>> It just depends on your workload. If you have 1 CPU loaded to 100% and
>> the other 3 idle, you might see an improvement. If you actually need
>> more than one CPU, it will probably be worse.
>>
>> I've also found that some Ethernet drivers invalidate or flush too
>> much. If you are sending a 64 byte TCP ACK, all you need to flush is
>> 64 bytes, not the full 1500 MTU. If you receive a TCP ACK, and then
>> recycle the buffer, all you need to invalidate is the size of the ACK,
>> so long as you can guarantee nothing has touched the memory above it.
>> But you need to be careful when implementing tricks like this, or you
>> can get subtle corruption bugs when you get it wrong.
> I just took a quick look at the driver. It allocates and maps rx buffers that can cover a packet size of BGMAC_RX_MAX_FRAME_SIZE = 9724.
> This seems rather excessive, especially since most people are going to use a MTU of 1500.
> My proposal would be to add support for making rx buffer size dependent on MTU, reallocating the ring on MTU changes.
> This should significantly reduce the time spent on flushing caches.
Oh, that's important too, it was changed by commit 8c7da63978f1 ("bgmac:
configure MTU and add support for frames beyond 8192 byte size"):
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8c7da63978f1672eb4037bbca6e7eac73f908f03
It lowered NAT speed with bgmac by 60% (362 Mbps → 140 Mbps).
I do all my testing with
#define BGMAC_RX_MAX_FRAME_SIZE 1536
next prev parent reply other threads:[~2022-05-06 7:47 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-27 12:04 Optimizing kernel compilation / alignments for network performance Rafał Miłecki
2022-04-27 12:56 ` Alexander Lobakin
2022-04-27 17:31 ` Rafał Miłecki
2022-04-29 14:18 ` Rafał Miłecki
2022-04-29 14:49 ` Arnd Bergmann
2022-05-05 15:42 ` Rafał Miłecki
2022-05-05 16:04 ` Andrew Lunn
2022-05-05 16:46 ` Felix Fietkau
2022-05-06 7:47 ` Rafał Miłecki [this message]
2022-05-06 12:42 ` Andrew Lunn
2022-05-10 10:29 ` Rafał Miłecki
2022-05-10 14:09 ` Dave Taht
2022-05-10 19:15 ` Dave Taht
2022-05-06 7:44 ` Rafał Miłecki
2022-05-06 8:45 ` Arnd Bergmann
2022-05-06 8:55 ` Rafał Miłecki
2022-05-06 9:44 ` Arnd Bergmann
2022-05-10 12:51 ` Rafał Miłecki
2022-05-10 13:19 ` Arnd Bergmann
2022-05-10 11:23 ` Rafał Miłecki
2022-05-10 13:18 ` Arnd Bergmann
2022-05-08 9:53 ` Rafał Miłecki
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=04fa6560-e6f4-005f-cddb-7bc9b4859ba2@gmail.com \
--to=zajec5@gmail.com \
--cc=alexandr.lobakin@intel.com \
--cc=andrew@lunn.ch \
--cc=arnd@arndb.de \
--cc=f.fainelli@gmail.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux@armlinux.org.uk \
--cc=nbd@nbd.name \
--cc=netdev@vger.kernel.org \
--cc=openwrt-devel@lists.openwrt.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).