* [PATCH RFC net-next] mlx5: fix for crash on net-next
@ 2021-02-22 19:50 Jesper Dangaard Brouer
2021-02-23 8:28 ` Tariq Toukan
0 siblings, 1 reply; 3+ messages in thread
From: Jesper Dangaard Brouer @ 2021-02-22 19:50 UTC (permalink / raw)
To: Jesper Dangaard Brouer, netdev, Saeed Mahameed
Cc: Jakub Kicinski, David S. Miller, saeedm, Tariq Toukan, eranbe, maximmi
Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17')
There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
where it seems that num_tc_x_num_ch can become zero in a modulo operation:
if (unlikely(txq_ix >= num_tc_x_num_ch))
txq_ix %= num_tc_x_num_ch;
I think error were introduced in:
- 214baf22870c ("net/mlx5e: Support HTB offload")
The modulo operation was introduced in:
- 145e5637d941 ("net/mlx5e: Add TX PTP port object support")
The crash looks like this:
[ 12.112849] divide error: 0000 [#1] PREEMPT SMP PTI
[ 12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575
[ 12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
[ 12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core]
[ 12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f
[ 12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297
[ 12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007
[ 12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740
[ 12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]---
This is an RFC, because I don't think this is a proper fix, but
at least it allows me to boot with mlx5 driver enabled.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index bdbffe484fce..4dc26c1f2b53 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -131,6 +131,7 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
int ch_ix;
/* Sync with mlx5e_update_num_tc_x_num_ch - avoid refetching. */
+ // XXX this reading of num_tc_x_num_ch looks strange
num_tc_x_num_ch = READ_ONCE(priv->num_tc_x_num_ch);
if (unlikely(dev->real_num_tx_queues > num_tc_x_num_ch)) {
/* Order maj_id before defcls - pairs with mlx5e_htb_root_add. */
@@ -142,19 +143,21 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
return txq_ix;
}
- if (unlikely(priv->channels.port_ptp))
+ txq_ix = netdev_pick_tx(dev, skb, NULL);
+
+ if (unlikely(priv->channels.port_ptp)) {
if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) &&
mlx5e_use_ptpsq(skb))
return mlx5e_select_ptpsq(dev, skb);
- txq_ix = netdev_pick_tx(dev, skb, NULL);
/* Fix netdev_pick_tx() not to choose ptp_channel and HTB txqs.
* If they are selected, switch to regular queues.
* Driver to select these queues only at mlx5e_select_ptpsq()
* and mlx5e_select_htb_queue().
*/
- if (unlikely(txq_ix >= num_tc_x_num_ch))
- txq_ix %= num_tc_x_num_ch;
+ if (unlikely(txq_ix >= num_tc_x_num_ch))
+ txq_ix %= num_tc_x_num_ch;
+ }
} else {
txq_ix = netdev_pick_tx(dev, skb, NULL);
}
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [PATCH RFC net-next] mlx5: fix for crash on net-next
2021-02-22 19:50 [PATCH RFC net-next] mlx5: fix for crash on net-next Jesper Dangaard Brouer
@ 2021-02-23 8:28 ` Tariq Toukan
2021-02-23 9:16 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 3+ messages in thread
From: Tariq Toukan @ 2021-02-23 8:28 UTC (permalink / raw)
To: Jesper Dangaard Brouer, netdev, Saeed Mahameed
Cc: Jakub Kicinski, David S. Miller, saeedm, Tariq Toukan, eranbe,
maximmi, Tariq Toukan, Saeed Mahameed
On 2/22/2021 9:50 PM, Jesper Dangaard Brouer wrote:
> Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17')
>
> There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
> where it seems that num_tc_x_num_ch can become zero in a modulo operation:
>
> if (unlikely(txq_ix >= num_tc_x_num_ch))
> txq_ix %= num_tc_x_num_ch;
>
> I think error were introduced in:
> - 214baf22870c ("net/mlx5e: Support HTB offload")
>
> The modulo operation was introduced in:
> - 145e5637d941 ("net/mlx5e: Add TX PTP port object support")
>
> The crash looks like this:
>
> [ 12.112849] divide error: 0000 [#1] PREEMPT SMP PTI
> [ 12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575
> [ 12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
> [ 12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core]
> [ 12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f
> [ 12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297
> [ 12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007
> [ 12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740
> [ 12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]---
>
> This is an RFC, because I don't think this is a proper fix, but
> at least it allows me to boot with mlx5 driver enabled.
>
> Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
> ---
Hi Jesper,
Thanks for your report and patch.
We'll check it and post the proper fix.
I wonder what's special in your configuration / use case, as I'm not
aware of such failure.
Can you please share more info about the reproduction?
Regards,
Tariq
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH RFC net-next] mlx5: fix for crash on net-next
2021-02-23 8:28 ` Tariq Toukan
@ 2021-02-23 9:16 ` Jesper Dangaard Brouer
0 siblings, 0 replies; 3+ messages in thread
From: Jesper Dangaard Brouer @ 2021-02-23 9:16 UTC (permalink / raw)
To: Tariq Toukan
Cc: netdev, Saeed Mahameed, Jakub Kicinski, David S. Miller, saeedm,
Tariq Toukan, eranbe, maximmi, Tariq Toukan, brouer
On Tue, 23 Feb 2021 10:28:35 +0200
Tariq Toukan <ttoukan.linux@gmail.com> wrote:
> On 2/22/2021 9:50 PM, Jesper Dangaard Brouer wrote:
> > Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17')
> >
> > There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
> > where it seems that num_tc_x_num_ch can become zero in a modulo operation:
> >
> > if (unlikely(txq_ix >= num_tc_x_num_ch))
> > txq_ix %= num_tc_x_num_ch;
> >
> > I think error were introduced in:
> > - 214baf22870c ("net/mlx5e: Support HTB offload")
> >
> > The modulo operation was introduced in:
> > - 145e5637d941 ("net/mlx5e: Add TX PTP port object support")
> >
> > The crash looks like this:
> >
> > [ 12.112849] divide error: 0000 [#1] PREEMPT SMP PTI
> > [ 12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575
> > [ 12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
> > [ 12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core]
> > [ 12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f
> > [ 12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297
> > [ 12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007
> > [ 12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740
> > [ 12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]---
> >
> > This is an RFC, because I don't think this is a proper fix, but
> > at least it allows me to boot with mlx5 driver enabled.
> >
> > Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
> > ---
>
> Hi Jesper,
>
> Thanks for your report and patch.
>
> We'll check it and post the proper fix.
> I wonder what's special in your configuration / use case, as I'm not
> aware of such failure.
> Can you please share more info about the reproduction?
The machine simply crash under boot, that is how I reproduced.
You should look at why: num_tc_x_num_ch can be zero in:
u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
struct net_device *sb_dev)
{
[...]
num_tc_x_num_ch = READ_ONCE(priv->num_tc_x_num_ch);
Here are some info on the mlx5 NIC I'm using:
$ ethtool -i mlx5p1
driver: mlx5_core
version: 5.11.0-net-next-alloc_pages_bul
firmware-version: 16.23.1020 (MT_0000000009)
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
03:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
Subsystem: Mellanox Technologies Device 0002
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 26
Memory at f0000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at fba00000 [disabled] [size=1M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [1c0] Secondary PCI Express
Capabilities: [230] Access Control Services
Capabilities: [320] Lane Margining at the Receiver <?>
Capabilities: [370] Physical Layer 16.0 GT/s <?>
Capabilities: [420] Data Link Feature <?>
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
03:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
Subsystem: Mellanox Technologies Device 0002
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 86
Memory at ee000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at fb900000 [disabled] [size=1M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [40] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [230] Access Control Services
Capabilities: [420] Data Link Feature <?>
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-02-23 9:18 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-22 19:50 [PATCH RFC net-next] mlx5: fix for crash on net-next Jesper Dangaard Brouer
2021-02-23 8:28 ` Tariq Toukan
2021-02-23 9:16 ` Jesper Dangaard Brouer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.