From: "Rafał Miłecki" <zajec5@gmail.com> To: Network Development <netdev@vger.kernel.org>, linux-arm-kernel <linux-arm-kernel@lists.infradead.org>, Russell King <linux@armlinux.org.uk>, Andrew Lunn <andrew@lunn.ch>, Felix Fietkau <nbd@nbd.name> Cc: "openwrt-devel@lists.openwrt.org" <openwrt-devel@lists.openwrt.org>, Florian Fainelli <f.fainelli@gmail.com> Subject: Optimizing kernel compilation / alignments for network performance Date: Wed, 27 Apr 2022 14:04:54 +0200 [thread overview] Message-ID: <84f25f73-1fab-fe43-70eb-45d25b614b4c@gmail.com> (raw) Hi, I noticed years ago that kernel changes touching code - that I don't use at all - can affect network performance for me. I work with home routers based on Broadcom Northstar platform. Those are SoCs with not-so-powerful 2 x ARM Cortex-A9 CPU cores. Main task of those devices is NAT masquerade and that is what I test with iperf running on two x86 machines. *** Example of such unused code change: ce5013ff3bec ("mtd: spi-nor: Add support for XM25QH64A and XM25QH128A"). https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ce5013ff3bec05cf2a8a05c75fcd520d9914d92b It lowered my NAT speed from 381 Mb/s to 367 Mb/s (-3,5%). I first reported that issue it in the e-mail thread: ARM router NAT performance affected by random/unrelated commits https://lkml.org/lkml/2019/5/21/349 https://www.spinics.net/lists/linux-block/msg40624.html Back then it was commit 5b0890a97204 ("flow_dissector: Parse batman-adv unicast headers") https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9316a9ed6895c4ad2f0cde171d486f80c55d8283 that increased my NAT speed from 741 Mb/s to 773 Mb/s (+4,3%). *** It appears Northstar CPUs have little cache size and so any change in location of kernel symbols can affect NAT performance. That explains why changing unrelated code affects anything & it has been partially proven aligning some of cache-v7.S code. My question is: is there a way to find out & force an optimal symbols locations? Adding .align 5 to the cache-v7.S is a partial success. I'd like to find out what other functions are worth optimizing (aligning) and force that (I guess __attribute__((aligned(32))) could be used). I can't really draw any conclusions from comparing System.map before and after above commits as they relocate thousands of symbols in one go. Optimizing is pretty important for me for two reasons: 1. I want to reach maximum possible NAT masquerade performance 2. I need stable performance across random commits to detect regressions
WARNING: multiple messages have this Message-ID (diff)
From: "Rafał Miłecki" <zajec5@gmail.com> To: Network Development <netdev@vger.kernel.org>, linux-arm-kernel <linux-arm-kernel@lists.infradead.org>, Russell King <linux@armlinux.org.uk>, Andrew Lunn <andrew@lunn.ch>, Felix Fietkau <nbd@nbd.name> Cc: "openwrt-devel@lists.openwrt.org" <openwrt-devel@lists.openwrt.org>, Florian Fainelli <f.fainelli@gmail.com> Subject: Optimizing kernel compilation / alignments for network performance Date: Wed, 27 Apr 2022 14:04:54 +0200 [thread overview] Message-ID: <84f25f73-1fab-fe43-70eb-45d25b614b4c@gmail.com> (raw) Hi, I noticed years ago that kernel changes touching code - that I don't use at all - can affect network performance for me. I work with home routers based on Broadcom Northstar platform. Those are SoCs with not-so-powerful 2 x ARM Cortex-A9 CPU cores. Main task of those devices is NAT masquerade and that is what I test with iperf running on two x86 machines. *** Example of such unused code change: ce5013ff3bec ("mtd: spi-nor: Add support for XM25QH64A and XM25QH128A"). https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ce5013ff3bec05cf2a8a05c75fcd520d9914d92b It lowered my NAT speed from 381 Mb/s to 367 Mb/s (-3,5%). I first reported that issue it in the e-mail thread: ARM router NAT performance affected by random/unrelated commits https://lkml.org/lkml/2019/5/21/349 https://www.spinics.net/lists/linux-block/msg40624.html Back then it was commit 5b0890a97204 ("flow_dissector: Parse batman-adv unicast headers") https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9316a9ed6895c4ad2f0cde171d486f80c55d8283 that increased my NAT speed from 741 Mb/s to 773 Mb/s (+4,3%). *** It appears Northstar CPUs have little cache size and so any change in location of kernel symbols can affect NAT performance. That explains why changing unrelated code affects anything & it has been partially proven aligning some of cache-v7.S code. My question is: is there a way to find out & force an optimal symbols locations? Adding .align 5 to the cache-v7.S is a partial success. I'd like to find out what other functions are worth optimizing (aligning) and force that (I guess __attribute__((aligned(32))) could be used). I can't really draw any conclusions from comparing System.map before and after above commits as they relocate thousands of symbols in one go. Optimizing is pretty important for me for two reasons: 1. I want to reach maximum possible NAT masquerade performance 2. I need stable performance across random commits to detect regressions _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next reply other threads:[~2022-04-27 12:05 UTC|newest] Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-04-27 12:04 Rafał Miłecki [this message] 2022-04-27 12:04 ` Optimizing kernel compilation / alignments for network performance Rafał Miłecki 2022-04-27 12:56 ` Alexander Lobakin 2022-04-27 12:56 ` Alexander Lobakin 2022-04-27 17:31 ` Rafał Miłecki 2022-04-27 17:31 ` Rafał Miłecki 2022-04-29 14:18 ` Rafał Miłecki 2022-04-29 14:18 ` Rafał Miłecki 2022-04-29 14:49 ` Arnd Bergmann 2022-04-29 14:49 ` Arnd Bergmann 2022-05-05 15:42 ` Rafał Miłecki 2022-05-05 15:42 ` Rafał Miłecki 2022-05-05 16:04 ` Andrew Lunn 2022-05-05 16:04 ` Andrew Lunn 2022-05-05 16:46 ` Felix Fietkau 2022-05-05 16:46 ` Felix Fietkau 2022-05-06 7:47 ` Rafał Miłecki 2022-05-06 7:47 ` Rafał Miłecki 2022-05-06 12:42 ` Andrew Lunn 2022-05-06 12:42 ` Andrew Lunn 2022-05-10 10:29 ` Rafał Miłecki 2022-05-10 10:29 ` Rafał Miłecki 2022-05-10 14:09 ` Dave Taht 2022-05-10 14:09 ` Dave Taht 2022-05-10 19:15 ` Dave Taht 2022-05-10 19:15 ` Dave Taht 2022-05-06 7:44 ` Rafał Miłecki 2022-05-06 7:44 ` Rafał Miłecki 2022-05-06 8:45 ` Arnd Bergmann 2022-05-06 8:45 ` Arnd Bergmann 2022-05-06 8:55 ` Rafał Miłecki 2022-05-06 8:55 ` Rafał Miłecki 2022-05-06 9:44 ` Arnd Bergmann 2022-05-06 9:44 ` Arnd Bergmann 2022-05-10 12:51 ` Rafał Miłecki 2022-05-10 12:51 ` Rafał Miłecki 2022-05-10 13:19 ` Arnd Bergmann 2022-05-10 13:19 ` Arnd Bergmann 2022-05-10 11:23 ` Rafał Miłecki 2022-05-10 11:23 ` Rafał Miłecki 2022-05-10 13:18 ` Arnd Bergmann 2022-05-10 13:18 ` Arnd Bergmann 2022-05-08 9:53 ` Rafał Miłecki 2022-05-08 9:53 ` Rafał Miłecki
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=84f25f73-1fab-fe43-70eb-45d25b614b4c@gmail.com \ --to=zajec5@gmail.com \ --cc=andrew@lunn.ch \ --cc=f.fainelli@gmail.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux@armlinux.org.uk \ --cc=nbd@nbd.name \ --cc=netdev@vger.kernel.org \ --cc=openwrt-devel@lists.openwrt.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.