From: Tony Nguyen <anthony.l.nguyen@intel.com> To: Alexander Lobakin <alexandr.lobakin@intel.com> Cc: "David S. Miller" <davem@davemloft.net>, Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>, Jesse Brandeburg <jesse.brandeburg@intel.com>, Madhu Chittim <madhu.chittim@intel.com>, Maciej Fijalkowski <maciej.fijalkowski@intel.com>, Brett Creeley <brett@pensando.io>, <intel-wired-lan@lists.osuosl.org>, <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>, Ivan Vecera <ivecera@redhat.com> Subject: Re: [PATCH v2 net] ice: arfs: fix use-after-free when freeing @rx_cpu_rmap Date: Fri, 8 Apr 2022 08:58:19 -0700 [thread overview] Message-ID: <607fe8b6-2f30-ac0c-996a-9d8c4cfeaba5@intel.com> (raw) In-Reply-To: <20220408123120.1829671-1-alexandr.lobakin@intel.com> On 4/8/2022 5:31 AM, Alexander Lobakin wrote: > From: Alexander Lobakin <alexandr.lobakin@intel.com> > Date: Mon, 4 Apr 2022 18:15:09 +0200 > >> The CI testing bots triggered the following splat: >> >> [ 718.203054] BUG: KASAN: use-after-free in free_irq_cpu_rmap+0x53/0x80 >> [ 718.206349] Read of size 4 at addr ffff8881bd127e00 by task sh/20834 >> [ 718.212852] CPU: 28 PID: 20834 Comm: sh Kdump: loaded Tainted: G S W IOE 5.17.0-rc8_nextqueue-devqueue-02643-g23f3121aca93 #1 >> [ 718.219695] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0012.070720200218 07/07/2020 >> [ 718.223418] Call Trace: >> [ 718.227139] >> [ 718.230783] dump_stack_lvl+0x33/0x42 >> [ 718.234431] print_address_description.constprop.9+0x21/0x170 >> [ 718.238177] ? free_irq_cpu_rmap+0x53/0x80 >> [ 718.241885] ? free_irq_cpu_rmap+0x53/0x80 >> [ 718.245539] kasan_report.cold.18+0x7f/0x11b >> [ 718.249197] ? free_irq_cpu_rmap+0x53/0x80 >> [ 718.252852] free_irq_cpu_rmap+0x53/0x80 >> [ 718.256471] ice_free_cpu_rx_rmap.part.11+0x37/0x50 [ice] >> [ 718.260174] ice_remove_arfs+0x5f/0x70 [ice] >> [ 718.263810] ice_rebuild_arfs+0x3b/0x70 [ice] >> [ 718.267419] ice_rebuild+0x39c/0xb60 [ice] >> [ 718.270974] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 >> [ 718.274472] ? ice_init_phy_user_cfg+0x360/0x360 [ice] >> [ 718.278033] ? delay_tsc+0x4a/0xb0 >> [ 718.281513] ? preempt_count_sub+0x14/0xc0 >> [ 718.284984] ? delay_tsc+0x8f/0xb0 >> [ 718.288463] ice_do_reset+0x92/0xf0 [ice] >> [ 718.292014] ice_pci_err_resume+0x91/0xf0 [ice] >> [ 718.295561] pci_reset_function+0x53/0x80 >> <...> >> [ 718.393035] Allocated by task 690: >> [ 718.433497] Freed by task 20834: >> [ 718.495688] Last potentially related work creation: >> [ 718.568966] The buggy address belongs to the object at ffff8881bd127e00 >> which belongs to the cache kmalloc-96 of size 96 >> [ 718.574085] The buggy address is located 0 bytes inside of >> 96-byte region [ffff8881bd127e00, ffff8881bd127e60) >> [ 718.579265] The buggy address belongs to the page: >> [ 718.598905] Memory state around the buggy address: >> [ 718.601809] ffff8881bd127d00: fa fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc >> [ 718.604796] ffff8881bd127d80: 00 00 00 00 00 00 00 00 00 00 fc fc fc fc fc fc >> [ 718.607794] >ffff8881bd127e00: fa fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc >> [ 718.610811] ^ >> [ 718.613819] ffff8881bd127e80: 00 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc >> [ 718.617107] ffff8881bd127f00: fa fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc >> >> This is due to that free_irq_cpu_rmap() is always being called >> *after* (devm_)free_irq() and thus it tries to work with IRQ descs >> already freed. For example, on device reset the driver frees the >> rmap right before allocating a new one (the splat above). >> Make rmap creation and freeing function symmetrical with >> {request,free}_irq() calls i.e. do that on ifup/ifdown instead >> of device probe/remove/resume. These operations can be performed >> independently from the actual device aRFS configuration. >> Also, make sure ice_vsi_free_irq() clears IRQ affinity notifiers >> only when aRFS is disabled -- otherwise, CPU rmap sets and clears >> its own and they must not be touched manually. >> >> Fixes: 28bf26724fdb0 ("ice: Implement aRFS") >> Co-developed-by: Ivan Vecera <ivecera@redhat.com> >> Signed-off-by: Ivan Vecera <ivecera@redhat.com> >> Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com> > Bah, forgot to mention in v2 that it's an urgent fix. Tony, are you > okay with posting it to netdev or allowing it to go directly to > -net? It's been tested by Ivan already (I had also asked Konrad, but > he hasn't replied yet). I have another patch to send as well. I'll send that and this one to netdev today. Thanks, Tony >> --- >> From v1[0]: >> - remove the obsolete `!vsi->arfs_fltr_list` check from >> ice_free_cpu_rx_rmap() leading to a leak and trace (Ivan). >> >> [0] https://lore.kernel.org/netdev/20220404132832.1936529-1-alexandr.lobakin@intel.com >> --- > --- 8< --- > >> -- >> 2.35.1 > Thanks, > Al
WARNING: multiple messages have this Message-ID (diff)
From: Tony Nguyen <anthony.l.nguyen@intel.com> To: intel-wired-lan@osuosl.org Subject: [Intel-wired-lan] [PATCH v2 net] ice: arfs: fix use-after-free when freeing @rx_cpu_rmap Date: Fri, 8 Apr 2022 08:58:19 -0700 [thread overview] Message-ID: <607fe8b6-2f30-ac0c-996a-9d8c4cfeaba5@intel.com> (raw) In-Reply-To: <20220408123120.1829671-1-alexandr.lobakin@intel.com> On 4/8/2022 5:31 AM, Alexander Lobakin wrote: > From: Alexander Lobakin <alexandr.lobakin@intel.com> > Date: Mon, 4 Apr 2022 18:15:09 +0200 > >> The CI testing bots triggered the following splat: >> >> [ 718.203054] BUG: KASAN: use-after-free in free_irq_cpu_rmap+0x53/0x80 >> [ 718.206349] Read of size 4 at addr ffff8881bd127e00 by task sh/20834 >> [ 718.212852] CPU: 28 PID: 20834 Comm: sh Kdump: loaded Tainted: G S W IOE 5.17.0-rc8_nextqueue-devqueue-02643-g23f3121aca93 #1 >> [ 718.219695] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0012.070720200218 07/07/2020 >> [ 718.223418] Call Trace: >> [ 718.227139] >> [ 718.230783] dump_stack_lvl+0x33/0x42 >> [ 718.234431] print_address_description.constprop.9+0x21/0x170 >> [ 718.238177] ? free_irq_cpu_rmap+0x53/0x80 >> [ 718.241885] ? free_irq_cpu_rmap+0x53/0x80 >> [ 718.245539] kasan_report.cold.18+0x7f/0x11b >> [ 718.249197] ? free_irq_cpu_rmap+0x53/0x80 >> [ 718.252852] free_irq_cpu_rmap+0x53/0x80 >> [ 718.256471] ice_free_cpu_rx_rmap.part.11+0x37/0x50 [ice] >> [ 718.260174] ice_remove_arfs+0x5f/0x70 [ice] >> [ 718.263810] ice_rebuild_arfs+0x3b/0x70 [ice] >> [ 718.267419] ice_rebuild+0x39c/0xb60 [ice] >> [ 718.270974] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 >> [ 718.274472] ? ice_init_phy_user_cfg+0x360/0x360 [ice] >> [ 718.278033] ? delay_tsc+0x4a/0xb0 >> [ 718.281513] ? preempt_count_sub+0x14/0xc0 >> [ 718.284984] ? delay_tsc+0x8f/0xb0 >> [ 718.288463] ice_do_reset+0x92/0xf0 [ice] >> [ 718.292014] ice_pci_err_resume+0x91/0xf0 [ice] >> [ 718.295561] pci_reset_function+0x53/0x80 >> <...> >> [ 718.393035] Allocated by task 690: >> [ 718.433497] Freed by task 20834: >> [ 718.495688] Last potentially related work creation: >> [ 718.568966] The buggy address belongs to the object at ffff8881bd127e00 >> which belongs to the cache kmalloc-96 of size 96 >> [ 718.574085] The buggy address is located 0 bytes inside of >> 96-byte region [ffff8881bd127e00, ffff8881bd127e60) >> [ 718.579265] The buggy address belongs to the page: >> [ 718.598905] Memory state around the buggy address: >> [ 718.601809] ffff8881bd127d00: fa fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc >> [ 718.604796] ffff8881bd127d80: 00 00 00 00 00 00 00 00 00 00 fc fc fc fc fc fc >> [ 718.607794] >ffff8881bd127e00: fa fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc >> [ 718.610811] ^ >> [ 718.613819] ffff8881bd127e80: 00 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc >> [ 718.617107] ffff8881bd127f00: fa fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc >> >> This is due to that free_irq_cpu_rmap() is always being called >> *after* (devm_)free_irq() and thus it tries to work with IRQ descs >> already freed. For example, on device reset the driver frees the >> rmap right before allocating a new one (the splat above). >> Make rmap creation and freeing function symmetrical with >> {request,free}_irq() calls i.e. do that on ifup/ifdown instead >> of device probe/remove/resume. These operations can be performed >> independently from the actual device aRFS configuration. >> Also, make sure ice_vsi_free_irq() clears IRQ affinity notifiers >> only when aRFS is disabled -- otherwise, CPU rmap sets and clears >> its own and they must not be touched manually. >> >> Fixes: 28bf26724fdb0 ("ice: Implement aRFS") >> Co-developed-by: Ivan Vecera <ivecera@redhat.com> >> Signed-off-by: Ivan Vecera <ivecera@redhat.com> >> Signed-off-by: Alexander Lobakin <alexandr.lobakin@intel.com> > Bah, forgot to mention in v2 that it's an urgent fix. Tony, are you > okay with posting it to netdev or allowing it to go directly to > -net? It's been tested by Ivan already (I had also asked Konrad, but > he hasn't replied yet). I have another patch to send as well. I'll send that and this one to netdev today. Thanks, Tony >> --- >> From v1[0]: >> - remove the obsolete `!vsi->arfs_fltr_list` check from >> ice_free_cpu_rx_rmap() leading to a leak and trace (Ivan). >> >> [0] https://lore.kernel.org/netdev/20220404132832.1936529-1-alexandr.lobakin at intel.com >> --- > --- 8< --- > >> -- >> 2.35.1 > Thanks, > Al
next prev parent reply other threads:[~2022-04-08 16:01 UTC|newest] Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-04-04 16:15 [PATCH v2 net] ice: arfs: fix use-after-free when freeing @rx_cpu_rmap Alexander Lobakin 2022-04-04 16:15 ` [Intel-wired-lan] " Alexander Lobakin 2022-04-04 16:21 ` Ivan Vecera 2022-04-04 16:21 ` [Intel-wired-lan] " Ivan Vecera 2022-04-08 12:31 ` Alexander Lobakin 2022-04-08 12:31 ` [Intel-wired-lan] " Alexander Lobakin 2022-04-08 15:58 ` Tony Nguyen [this message] 2022-04-08 15:58 ` Tony Nguyen
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=607fe8b6-2f30-ac0c-996a-9d8c4cfeaba5@intel.com \ --to=anthony.l.nguyen@intel.com \ --cc=alexandr.lobakin@intel.com \ --cc=brett@pensando.io \ --cc=davem@davemloft.net \ --cc=intel-wired-lan@lists.osuosl.org \ --cc=ivecera@redhat.com \ --cc=jesse.brandeburg@intel.com \ --cc=kuba@kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=maciej.fijalkowski@intel.com \ --cc=madhu.chittim@intel.com \ --cc=netdev@vger.kernel.org \ --cc=pabeni@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.