netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH RFC net-next] mlx5: fix for crash on net-next
@ 2021-02-22 19:50 Jesper Dangaard Brouer
  2021-02-23  8:28 ` Tariq Toukan
  0 siblings, 1 reply; 3+ messages in thread
From: Jesper Dangaard Brouer @ 2021-02-22 19:50 UTC (permalink / raw)
  To: Jesper Dangaard Brouer, netdev, Saeed Mahameed
  Cc: Jakub Kicinski, David S. Miller, saeedm, Tariq Toukan, eranbe, maximmi

Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17')

There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
where it seems that num_tc_x_num_ch can become zero in a modulo operation:

	if (unlikely(txq_ix >= num_tc_x_num_ch))
		txq_ix %= num_tc_x_num_ch;

I think error were introduced in:
 - 214baf22870c ("net/mlx5e: Support HTB offload")

The modulo operation was introduced in:
 - 145e5637d941 ("net/mlx5e: Add TX PTP port object support")

The crash looks like this:

[   12.112849] divide error: 0000 [#1] PREEMPT SMP PTI
[   12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575
[   12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
[   12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core]
[   12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f
[   12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297
[   12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007
[   12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740
[   12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]---

This is an RFC, because I don't think this is a proper fix, but
at least it allows me to boot with mlx5 driver enabled.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en_tx.c |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index bdbffe484fce..4dc26c1f2b53 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -131,6 +131,7 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
 	int ch_ix;
 
 	/* Sync with mlx5e_update_num_tc_x_num_ch - avoid refetching. */
+	// XXX this reading of num_tc_x_num_ch looks strange
 	num_tc_x_num_ch = READ_ONCE(priv->num_tc_x_num_ch);
 	if (unlikely(dev->real_num_tx_queues > num_tc_x_num_ch)) {
 		/* Order maj_id before defcls - pairs with mlx5e_htb_root_add. */
@@ -142,19 +143,21 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
 				return txq_ix;
 		}
 
-		if (unlikely(priv->channels.port_ptp))
+		txq_ix = netdev_pick_tx(dev, skb, NULL);
+
+		if (unlikely(priv->channels.port_ptp)) {
 			if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) &&
 			    mlx5e_use_ptpsq(skb))
 				return mlx5e_select_ptpsq(dev, skb);
 
-		txq_ix = netdev_pick_tx(dev, skb, NULL);
 		/* Fix netdev_pick_tx() not to choose ptp_channel and HTB txqs.
 		 * If they are selected, switch to regular queues.
 		 * Driver to select these queues only at mlx5e_select_ptpsq()
 		 * and mlx5e_select_htb_queue().
 		 */
-		if (unlikely(txq_ix >= num_tc_x_num_ch))
-			txq_ix %= num_tc_x_num_ch;
+			if (unlikely(txq_ix >= num_tc_x_num_ch))
+				txq_ix %= num_tc_x_num_ch;
+		}
 	} else {
 		txq_ix = netdev_pick_tx(dev, skb, NULL);
 	}



^ permalink raw reply related	[flat|nested] 3+ messages in thread

* Re: [PATCH RFC net-next] mlx5: fix for crash on net-next
  2021-02-22 19:50 [PATCH RFC net-next] mlx5: fix for crash on net-next Jesper Dangaard Brouer
@ 2021-02-23  8:28 ` Tariq Toukan
  2021-02-23  9:16   ` Jesper Dangaard Brouer
  0 siblings, 1 reply; 3+ messages in thread
From: Tariq Toukan @ 2021-02-23  8:28 UTC (permalink / raw)
  To: Jesper Dangaard Brouer, netdev, Saeed Mahameed
  Cc: Jakub Kicinski, David S. Miller, saeedm, Tariq Toukan, eranbe,
	maximmi, Tariq Toukan, Saeed Mahameed



On 2/22/2021 9:50 PM, Jesper Dangaard Brouer wrote:
> Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17')
> 
> There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
> where it seems that num_tc_x_num_ch can become zero in a modulo operation:
> 
> 	if (unlikely(txq_ix >= num_tc_x_num_ch))
> 		txq_ix %= num_tc_x_num_ch;
> 
> I think error were introduced in:
>   - 214baf22870c ("net/mlx5e: Support HTB offload")
> 
> The modulo operation was introduced in:
>   - 145e5637d941 ("net/mlx5e: Add TX PTP port object support")
> 
> The crash looks like this:
> 
> [   12.112849] divide error: 0000 [#1] PREEMPT SMP PTI
> [   12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575
> [   12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
> [   12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core]
> [   12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f
> [   12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297
> [   12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007
> [   12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740
> [   12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]---
> 
> This is an RFC, because I don't think this is a proper fix, but
> at least it allows me to boot with mlx5 driver enabled.
> 
> Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
> ---

Hi Jesper,

Thanks for your report and patch.

We'll check it and post the proper fix.
I wonder what's special in your configuration / use case, as I'm not 
aware of such failure.
Can you please share more info about the reproduction?

Regards,
Tariq

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH RFC net-next] mlx5: fix for crash on net-next
  2021-02-23  8:28 ` Tariq Toukan
@ 2021-02-23  9:16   ` Jesper Dangaard Brouer
  0 siblings, 0 replies; 3+ messages in thread
From: Jesper Dangaard Brouer @ 2021-02-23  9:16 UTC (permalink / raw)
  To: Tariq Toukan
  Cc: netdev, Saeed Mahameed, Jakub Kicinski, David S. Miller, saeedm,
	Tariq Toukan, eranbe, maximmi, Tariq Toukan, brouer

On Tue, 23 Feb 2021 10:28:35 +0200
Tariq Toukan <ttoukan.linux@gmail.com> wrote:

> On 2/22/2021 9:50 PM, Jesper Dangaard Brouer wrote:
> > Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17')
> > 
> > There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
> > where it seems that num_tc_x_num_ch can become zero in a modulo operation:
> > 
> > 	if (unlikely(txq_ix >= num_tc_x_num_ch))
> > 		txq_ix %= num_tc_x_num_ch;
> > 
> > I think error were introduced in:
> >   - 214baf22870c ("net/mlx5e: Support HTB offload")
> > 
> > The modulo operation was introduced in:
> >   - 145e5637d941 ("net/mlx5e: Add TX PTP port object support")
> > 
> > The crash looks like this:
> > 
> > [   12.112849] divide error: 0000 [#1] PREEMPT SMP PTI
> > [   12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575
> > [   12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
> > [   12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core]
> > [   12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f
> > [   12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297
> > [   12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007
> > [   12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740
> > [   12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]---
> > 
> > This is an RFC, because I don't think this is a proper fix, but
> > at least it allows me to boot with mlx5 driver enabled.
> > 
> > Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
> > ---  
> 
> Hi Jesper,
> 
> Thanks for your report and patch.
> 
> We'll check it and post the proper fix.
> I wonder what's special in your configuration / use case, as I'm not 
> aware of such failure.
> Can you please share more info about the reproduction?

The machine simply crash under boot, that is how I reproduced.

You should look at why: num_tc_x_num_ch can be zero in:

 u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
		       struct net_device *sb_dev)
 {
   [...]
   num_tc_x_num_ch = READ_ONCE(priv->num_tc_x_num_ch);


Here are some info on the mlx5 NIC I'm using:

$ ethtool -i mlx5p1
driver: mlx5_core
version: 5.11.0-net-next-alloc_pages_bul
firmware-version: 16.23.1020 (MT_0000000009)
expansion-rom-version: 
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

03:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
        Subsystem: Mellanox Technologies Device 0002
        Physical Slot: 0
        Flags: bus master, fast devsel, latency 0, IRQ 26
        Memory at f0000000 (64-bit, prefetchable) [size=32M]
        Expansion ROM at fba00000 [disabled] [size=1M]
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [48] Vital Product Data
        Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [c0] Vendor Specific Information: Len=18 <?>
        Capabilities: [40] Power Management version 3
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [1c0] Secondary PCI Express
        Capabilities: [230] Access Control Services
        Capabilities: [320] Lane Margining at the Receiver <?>
        Capabilities: [370] Physical Layer 16.0 GT/s <?>
        Capabilities: [420] Data Link Feature <?>
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core

03:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
        Subsystem: Mellanox Technologies Device 0002
        Physical Slot: 0
        Flags: bus master, fast devsel, latency 0, IRQ 86
        Memory at ee000000 (64-bit, prefetchable) [size=32M]
        Expansion ROM at fb900000 [disabled] [size=1M]
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [48] Vital Product Data
        Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [c0] Vendor Specific Information: Len=18 <?>
        Capabilities: [40] Power Management version 3
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [230] Access Control Services
        Capabilities: [420] Data Link Feature <?>
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2021-02-23  9:18 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-22 19:50 [PATCH RFC net-next] mlx5: fix for crash on net-next Jesper Dangaard Brouer
2021-02-23  8:28 ` Tariq Toukan
2021-02-23  9:16   ` Jesper Dangaard Brouer

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).