* [bug] dpaa2-eth: "Wrong SWA type" and null deref in dpaa2_eth_free_tx_fd()
@ 2023-08-30 17:10 Daniel Klauer
2023-10-04 15:50 ` Ioana Ciornei
0 siblings, 1 reply; 5+ messages in thread
From: Daniel Klauer @ 2023-08-30 17:10 UTC (permalink / raw)
To: Ioana Ciornei; +Cc: netdev
Hi,
while doing Ethernet tests with raw packet sockets on our custom LX2160A board with Linux v6.1.50 (plus some patches for board support, but none for dpaa2-eth), I noticed the following crash:
[ 26.290737] Wrong SWA type
[ 26.290760] WARNING: CPU: 7 PID: 0 at drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c:1117 dpaa2_eth_free_tx_fd.isra.0+0x36c/0x380 [fsl_dpaa2_eth]
followed by
[ 26.323016] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028
[ 26.324122] Mem abort info:
[ 26.324475] ESR = 0x0000000096000004
[ 26.324948] EC = 0x25: DABT (current EL), IL = 32 bits
[ 26.325618] SET = 0, FnV = 0
[ 26.326004] EA = 0, S1PTW = 0
[ 26.326406] FSC = 0x04: level 0 translation fault
[ 26.327021] Data abort info:
[ 26.327385] ISV = 0, ISS = 0x00000004
[ 26.327869] CM = 0, WnR = 0
[ 26.328244] user pgtable: 4k pages, 48-bit VAs, pgdp=00000020861cf000
[ 26.329055] [0000000000000028] pgd=0000000000000000, p4d=0000000000000000
[ 26.329912] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
[ 26.330702] Modules linked in: tag_dsa marvell mv88e6xxx aes_ce_blk caam_jr aes_ce_cipher caamhash_desc crct10dif_ce ghash_ce fsl_dpaa2_eth caamalg_desc xhci_plat_hcd sha256_generic gf128mul libsha256 libaes xhci_hcd crypto_engine pcs_lynx sha2_ce sha1_ce usbcore libdes sha256_arm64 cfg80211 dp83867 sha1_generic fsl_mc_dpio xgmac_mdio dpaa2_console dwc3 ahci ahci_qoriq udc_core caam libahci_platform roles error libahci usb_common libata at24 lm90 qoriq_thermal nvmem_layerscape_sfp sfp mdio_i2c
[ 26.336237] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G W 6.1.50-00121-g10168a070f4d #11
[ 26.337396] Hardware name: mpxlx2160a (DT)
[ 26.337956] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 26.338833] pc : dpaa2_eth_free_tx_fd.isra.0+0xd4/0x380 [fsl_dpaa2_eth]
[ 26.339673] lr : dpaa2_eth_free_tx_fd.isra.0+0xb4/0x380 [fsl_dpaa2_eth]
[ 26.340512] sp : ffff800008cf3d70
[ 26.340931] x29: ffff800008cf3d70 x28: ffff002002900000 x27: 0000000000000000
[ 26.341832] x26: 0000000000000001 x25: 0000000000000001 x24: 0000000000000000
[ 26.342732] x23: 0000000000002328 x22: ffff002009742728 x21: 00000020884fffc2
[ 26.343633] x20: ffff002009740840 x19: ffff0020084fffc2 x18: 0000000000000018
[ 26.344534] x17: ffff8026b3a9a000 x16: ffff800008cf0000 x15: fffffffffffed3f8
[ 26.345435] x14: 0000000000000000 x13: ffff800008bad028 x12: 0000000000000966
[ 26.346335] x11: 0000000000000322 x10: ffff800008c09b58 x9 : ffff800008bad028
[ 26.347236] x8 : 0001000000000000 x7 : ffff0020095e6480 x6 : 00000020884fffc2
[ 26.348137] x5 : ffff0020095e6480 x4 : 0000000000000000 x3 : 0000000000000000
[ 26.349037] x2 : 00000000e7e00000 x1 : 0000000000000001 x0 : 0000000049759e0c
[ 26.349938] Call trace:
[ 26.350247] dpaa2_eth_free_tx_fd.isra.0+0xd4/0x380 [fsl_dpaa2_eth]
[ 26.351044] dpaa2_eth_tx_conf+0x84/0xc0 [fsl_dpaa2_eth]
[ 26.351720] dpaa2_eth_poll+0xec/0x3a4 [fsl_dpaa2_eth]
[ 26.352375] __napi_poll+0x34/0x180
[ 26.352816] net_rx_action+0x128/0x2b4
[ 26.353290] _stext+0x124/0x2a0
[ 26.353687] ____do_softirq+0xc/0x14
[ 26.354139] call_on_irq_stack+0x24/0x40
[ 26.354635] do_softirq_own_stack+0x18/0x2c
[ 26.355164] __irq_exit_rcu+0xc4/0xf0
[ 26.355628] irq_exit_rcu+0xc/0x14
[ 26.356059] el1_interrupt+0x34/0x60
[ 26.356511] el1h_64_irq_handler+0x14/0x20
[ 26.357028] el1h_64_irq+0x64/0x68
[ 26.357458] cpuidle_enter_state+0x12c/0x314
[ 26.357997] cpuidle_enter+0x34/0x4c
[ 26.358450] do_idle+0x208/0x270
[ 26.358860] cpu_startup_entry+0x24/0x30
[ 26.359356] secondary_start_kernel+0x128/0x14c
[ 26.359928] __secondary_switched+0x64/0x68
[ 26.360460] Code: 7100081f 54000d00 71000c1f 540000c0 (3940a360)
[ 26.361228] ---[ end trace 0000000000000000 ]---
It happens when receiving big Ethernet frames on a AF_PACKET + SOCK_RAW socket, for example MTU 9000. It does not happen with the standard MTU 1500. It does not happen when just sending.
It's 100% reproducible here, however it seems to depend on the data rate/load: Once it happened after receiving the first 80 frames, another time after the first 300 frames, etc., and if I only send 5 frames per second, it does not happen at all.
Please let me know if I should provide more info or do more tests. I can provide a test program if needed.
Kind regards,
Daniel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [bug] dpaa2-eth: "Wrong SWA type" and null deref in dpaa2_eth_free_tx_fd()
2023-08-30 17:10 [bug] dpaa2-eth: "Wrong SWA type" and null deref in dpaa2_eth_free_tx_fd() Daniel Klauer
@ 2023-10-04 15:50 ` Ioana Ciornei
2023-10-06 14:03 ` Daniel Klauer
0 siblings, 1 reply; 5+ messages in thread
From: Ioana Ciornei @ 2023-10-04 15:50 UTC (permalink / raw)
To: Daniel Klauer; +Cc: netdev
On Wed, Aug 30, 2023 at 07:10:05PM +0200, Daniel Klauer wrote:
> Hi,
>
Hi Daniel,
> while doing Ethernet tests with raw packet sockets on our custom
> LX2160A board with Linux v6.1.50 (plus some patches for board support,
> but none for dpaa2-eth), I noticed the following crash:
>
Did you happen to test with any other newer kernel?
> [ 26.290737] Wrong SWA type
> [ 26.290760] WARNING: CPU: 7 PID: 0 at drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c:1117 dpaa2_eth_free_tx_fd.isra.0+0x36c/0x380 [fsl_dpaa2_eth]
>
> followed by
>
> [ 26.323016] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028
> [ 26.324122] Mem abort info:
> [ 26.324475] ESR = 0x0000000096000004
> [ 26.324948] EC = 0x25: DABT (current EL), IL = 32 bits
> [ 26.325618] SET = 0, FnV = 0
> [ 26.326004] EA = 0, S1PTW = 0
> [ 26.326406] FSC = 0x04: level 0 translation fault
> [ 26.327021] Data abort info:
> [ 26.327385] ISV = 0, ISS = 0x00000004
> [ 26.327869] CM = 0, WnR = 0
> [ 26.328244] user pgtable: 4k pages, 48-bit VAs, pgdp=00000020861cf000
> [ 26.329055] [0000000000000028] pgd=0000000000000000, p4d=0000000000000000
> [ 26.329912] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
> [ 26.330702] Modules linked in: tag_dsa marvell mv88e6xxx aes_ce_blk caam_jr aes_ce_cipher caamhash_desc crct10dif_ce ghash_ce fsl_dpaa2_eth caamalg_desc xhci_plat_hcd sha256_generic gf128mul libsha256 libaes xhci_hcd crypto_engine pcs_lynx sha2_ce sha1_ce usbcore libdes sha256_arm64 cfg80211 dp83867 sha1_generic fsl_mc_dpio xgmac_mdio dpaa2_console dwc3 ahci ahci_qoriq udc_core caam libahci_platform roles error libahci usb_common libata at24 lm90 qoriq_thermal nvmem_layerscape_sfp sfp mdio_i2c
> [ 26.336237] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G W 6.1.50-00121-g10168a070f4d #11
> [ 26.337396] Hardware name: mpxlx2160a (DT)
> [ 26.337956] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [ 26.338833] pc : dpaa2_eth_free_tx_fd.isra.0+0xd4/0x380 [fsl_dpaa2_eth]
> [ 26.339673] lr : dpaa2_eth_free_tx_fd.isra.0+0xb4/0x380 [fsl_dpaa2_eth]
> [ 26.340512] sp : ffff800008cf3d70
> [ 26.340931] x29: ffff800008cf3d70 x28: ffff002002900000 x27: 0000000000000000
> [ 26.341832] x26: 0000000000000001 x25: 0000000000000001 x24: 0000000000000000
> [ 26.342732] x23: 0000000000002328 x22: ffff002009742728 x21: 00000020884fffc2
> [ 26.343633] x20: ffff002009740840 x19: ffff0020084fffc2 x18: 0000000000000018
> [ 26.344534] x17: ffff8026b3a9a000 x16: ffff800008cf0000 x15: fffffffffffed3f8
> [ 26.345435] x14: 0000000000000000 x13: ffff800008bad028 x12: 0000000000000966
> [ 26.346335] x11: 0000000000000322 x10: ffff800008c09b58 x9 : ffff800008bad028
> [ 26.347236] x8 : 0001000000000000 x7 : ffff0020095e6480 x6 : 00000020884fffc2
> [ 26.348137] x5 : ffff0020095e6480 x4 : 0000000000000000 x3 : 0000000000000000
> [ 26.349037] x2 : 00000000e7e00000 x1 : 0000000000000001 x0 : 0000000049759e0c
> [ 26.349938] Call trace:
> [ 26.350247] dpaa2_eth_free_tx_fd.isra.0+0xd4/0x380 [fsl_dpaa2_eth]
> [ 26.351044] dpaa2_eth_tx_conf+0x84/0xc0 [fsl_dpaa2_eth]
> [ 26.351720] dpaa2_eth_poll+0xec/0x3a4 [fsl_dpaa2_eth]
> [ 26.352375] __napi_poll+0x34/0x180
> [ 26.352816] net_rx_action+0x128/0x2b4
> [ 26.353290] _stext+0x124/0x2a0
> [ 26.353687] ____do_softirq+0xc/0x14
> [ 26.354139] call_on_irq_stack+0x24/0x40
> [ 26.354635] do_softirq_own_stack+0x18/0x2c
> [ 26.355164] __irq_exit_rcu+0xc4/0xf0
> [ 26.355628] irq_exit_rcu+0xc/0x14
> [ 26.356059] el1_interrupt+0x34/0x60
> [ 26.356511] el1h_64_irq_handler+0x14/0x20
> [ 26.357028] el1h_64_irq+0x64/0x68
> [ 26.357458] cpuidle_enter_state+0x12c/0x314
> [ 26.357997] cpuidle_enter+0x34/0x4c
> [ 26.358450] do_idle+0x208/0x270
> [ 26.358860] cpu_startup_entry+0x24/0x30
> [ 26.359356] secondary_start_kernel+0x128/0x14c
> [ 26.359928] __secondary_switched+0x64/0x68
> [ 26.360460] Code: 7100081f 54000d00 71000c1f 540000c0 (3940a360)
> [ 26.361228] ---[ end trace 0000000000000000 ]---
>
> It happens when receiving big Ethernet frames on a AF_PACKET +
> SOCK_RAW socket, for example MTU 9000. It does not happen with the
> standard MTU 1500. It does not happen when just sending.
>
Are the transmitted frames also big?
> It's 100% reproducible here, however it seems to depend on the data
> rate/load: Once it happened after receiving the first 80 frames,
> another time after the first 300 frames, etc., and if I only send 5
> frames per second, it does not happen at all.
>
> Please let me know if I should provide more info or do more tests. I
> can provide a test program if needed.
>
If you can provide a test program, that would be great. It would help in
reproducing and debugging the issue on my side.
Ioana
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [bug] dpaa2-eth: "Wrong SWA type" and null deref in dpaa2_eth_free_tx_fd()
2023-10-04 15:50 ` Ioana Ciornei
@ 2023-10-06 14:03 ` Daniel Klauer
2023-10-13 15:52 ` Ioana Ciornei
0 siblings, 1 reply; 5+ messages in thread
From: Daniel Klauer @ 2023-10-06 14:03 UTC (permalink / raw)
To: Ioana Ciornei; +Cc: netdev
[-- Attachment #1: Type: text/plain, Size: 13339 bytes --]
On 04.10.23 17:50, Ioana Ciornei wrote:
> On Wed, Aug 30, 2023 at 07:10:05PM +0200, Daniel Klauer wrote:
>> Hi,
>>
>
> Hi Daniel,
>
>> while doing Ethernet tests with raw packet sockets on our custom
>> LX2160A board with Linux v6.1.50 (plus some patches for board support,
>> but none for dpaa2-eth), I noticed the following crash:
>>
>
> Did you happen to test with any other newer kernel?
Today I tested with v6.5.5, it also shows the same crash:
[ 160.013619] fsl_dpaa2_eth dpni.0 eth6: entered promiscuous mode
[ 163.100294] fsl_dpaa2_eth dpni.0 eth6: Link is Up - 1Gbps/Full - flow control off
[ 163.544566] ------------[ cut here ]------------
[ 163.545163] Wrong SWA type
[ 163.545188] WARNING: CPU: 12 PID: 0 at drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c:1148 dpaa2_eth_free_tx_fd+0x3bc/0x3c8 [fsl_dpaa2_eth]
[ 163.547145] Modules linked in: marvell tag_dsa xhci_plat_hcd xhci_hcd aes_ce_blk usbcore aes_ce_cipher crct10dif_ce caam_jr ghash_ce gf128mul dwc3 fsl_dpaa2_eth caamhash_desc libaes caamalg_desc sha2_ce crypto_engine sha256_arm64 pcs_lynx mv88e6xxx libdes udc_core sha1_ce ahci_qoriq roles fsl_mc_dpio dp83867 libahci_platform ahci usb_common sha1_generic dpaa2_console sfp xgmac_mdio caam libahci at24 error libata mdio_i2c lm90 qoriq_thermal nvmem_layerscape_sfp
[ 163.552482] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G O 6.5.5-00118-g8e92933b7fa0 #1
[ 163.553643] Hardware name: mpxlx2160a (DT)
[ 163.554204] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 163.555082] pc : dpaa2_eth_free_tx_fd+0x3bc/0x3c8 [fsl_dpaa2_eth]
[ 163.555858] lr : dpaa2_eth_free_tx_fd+0x3bc/0x3c8 [fsl_dpaa2_eth]
[ 163.556635] sp : ffff800080d7bd70
[ 163.557054] x29: ffff800080d7bd70 x28: 00000020874fbfc2 x27: ffff00200ba1b000
[ 163.557958] x26: ffff00200ba60880 x25: 0000000000000001 x24: ffff00200320d800
[ 163.558860] x23: ffff00200ba43d70 x22: 0000000000002328 x21: ffff00200ba40880
[ 163.559762] x20: 0000000000000000 x19: ffff0020074fbfc2 x18: 0000000000000018
[ 163.560665] x17: ffff80263bae3000 x16: ffff800080d78000 x15: fffffffffffee2d0
[ 163.561567] x14: ffff800080bfb388 x13: ffff800080bfb3e0 x12: 0000000000000a38
[ 163.562469] x11: 0000000000000368 x10: ffff800080c585a0 x9 : ffff800080bfb3e0
[ 163.563372] x8 : 00000000ffffefff x7 : ffff800080c533e0 x6 : 00000000000051c0
[ 163.564274] x5 : ffff0026bc5b98c8 x4 : 0000000000000000 x3 : 0000000000000027
[ 163.565176] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00200080e040
[ 163.566078] Call trace:
[ 163.566389] dpaa2_eth_free_tx_fd+0x3bc/0x3c8 [fsl_dpaa2_eth]
[ 163.567121] dpaa2_eth_tx_conf+0x74/0xac [fsl_dpaa2_eth]
[ 163.567798] dpaa2_eth_poll+0xec/0x3e0 [fsl_dpaa2_eth]
[ 163.568454] __napi_poll+0x34/0x184
[ 163.568902] net_rx_action+0x11c/0x258
[ 163.569377] __do_softirq+0x11c/0x284
[ 163.569843] ____do_softirq+0xc/0x14
[ 163.570296] call_on_irq_stack+0x24/0x34
[ 163.570793] do_softirq_own_stack+0x18/0x20
[ 163.571322] irq_exit_rcu+0xd0/0xe8
[ 163.571765] el1_interrupt+0x34/0x60
[ 163.572222] el1h_64_irq_handler+0x14/0x1c
[ 163.572742] el1h_64_irq+0x64/0x68
[ 163.573172] cpuidle_enter_state+0x130/0x2fc
[ 163.573712] cpuidle_enter+0x34/0x48
[ 163.574166] do_idle+0x1c8/0x230
[ 163.574578] cpu_startup_entry+0x20/0x28
[ 163.575076] secondary_start_kernel+0x128/0x148
[ 163.575650] __secondary_switched+0x6c/0x70
[ 163.576183] ---[ end trace 0000000000000000 ]---
[ 163.576778] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028
[ 163.577884] Mem abort info:
[ 163.578244] ESR = 0x0000000096000004
[ 163.578718] EC = 0x25: DABT (current EL), IL = 32 bits
[ 163.579388] SET = 0, FnV = 0
[ 163.579774] EA = 0, S1PTW = 0
[ 163.580171] FSC = 0x04: level 0 translation fault
[ 163.580787] Data abort info:
[ 163.581151] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
[ 163.581842] CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[ 163.582485] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 163.583156] user pgtable: 4k pages, 48-bit VAs, pgdp=00000020874aa000
[ 163.583968] [0000000000000028] pgd=0000000000000000, p4d=0000000000000000
[ 163.584825] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
[ 163.585616] Modules linked in: marvell tag_dsa xhci_plat_hcd xhci_hcd aes_ce_blk usbcore aes_ce_cipher crct10dif_ce caam_jr ghash_ce gf128mul dwc3 fsl_dpaa2_eth caamhash_desc libaes caamalg_desc sha2_ce crypto_engine sha256_arm64 pcs_lynx mv88e6xxx libdes udc_core sha1_ce ahci_qoriq roles fsl_mc_dpio dp83867 libahci_platform ahci usb_common sha1_generic dpaa2_console sfp xgmac_mdio caam libahci at24 error libata mdio_i2c lm90 qoriq_thermal nvmem_layerscape_sfp
[ 163.590941] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G W O 6.5.5-00118-g8e92933b7fa0 #1
[ 163.592102] Hardware name: mpxlx2160a (DT)
[ 163.592662] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 163.593540] pc : dpaa2_eth_free_tx_fd+0xe8/0x3c8 [fsl_dpaa2_eth]
[ 163.594305] lr : dpaa2_eth_free_tx_fd+0xc0/0x3c8 [fsl_dpaa2_eth]
[ 163.595069] sp : ffff800080d7bd70
[ 163.595487] x29: ffff800080d7bd70 x28: 00000020874fbfc2 x27: 0000000000000000
[ 163.596390] x26: 0000000000000001 x25: 0000000000000001 x24: ffff00200320d800
[ 163.597292] x23: ffff00200ba43d70 x22: 0000000000002328 x21: ffff00200ba40880
[ 163.598194] x20: 0000000000000000 x19: ffff0020074fbfc2 x18: 0000000000000018
[ 163.599096] x17: ffff80263bae3000 x16: ffff800080d78000 x15: fffffffffffee2d0
[ 163.599998] x14: ffff800080bfb388 x13: ffff800080bfb3e0 x12: 0000000000000a38
[ 163.600900] x11: 0000000000000368 x10: ffff800080c585a0 x9 : ffff800080bfb3e0
[ 163.601803] x8 : 0001000000000000 x7 : ffff002002dae480 x6 : 00000020874fbfc2
[ 163.602705] x5 : ffff002002dae480 x4 : 0000000000000000 x3 : 0000000000000000
[ 163.603607] x2 : 00000000e7e00000 x1 : 0000000000000001 x0 : 0000000001010101
[ 163.604509] Call trace:
[ 163.604819] dpaa2_eth_free_tx_fd+0xe8/0x3c8 [fsl_dpaa2_eth]
[ 163.605539] dpaa2_eth_tx_conf+0x74/0xac [fsl_dpaa2_eth]
[ 163.606216] dpaa2_eth_poll+0xec/0x3e0 [fsl_dpaa2_eth]
[ 163.606871] __napi_poll+0x34/0x184
[ 163.607314] net_rx_action+0x11c/0x258
[ 163.607789] __do_softirq+0x11c/0x284
[ 163.608252] ____do_softirq+0xc/0x14
[ 163.608705] call_on_irq_stack+0x24/0x34
[ 163.609201] do_softirq_own_stack+0x18/0x20
[ 163.609730] irq_exit_rcu+0xd0/0xe8
[ 163.610172] el1_interrupt+0x34/0x60
[ 163.610627] el1h_64_irq_handler+0x14/0x1c
[ 163.611147] el1h_64_irq+0x64/0x68
[ 163.611578] cpuidle_enter_state+0x130/0x2fc
[ 163.612117] cpuidle_enter+0x34/0x48
[ 163.612570] do_idle+0x1c8/0x230
[ 163.612981] cpu_startup_entry+0x20/0x28
[ 163.613479] secondary_start_kernel+0x128/0x148
[ 163.614052] __secondary_switched+0x6c/0x70
[ 163.614585] Code: 7100081f 54000de0 7100101f 540000c0 (3940a360)
[ 163.615355] ---[ end trace 0000000000000000 ]---
[ 163.615938] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[ 163.616803] SMP: stopping secondary CPUs
[ 163.617309] Kernel Offset: disabled
[ 163.617750] CPU features: 0x40000000,12010000,0800420b
[ 163.618399] Memory Limit: none
[ 163.618787] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---
>
>> [ 26.290737] Wrong SWA type
>> [ 26.290760] WARNING: CPU: 7 PID: 0 at drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c:1117 dpaa2_eth_free_tx_fd.isra.0+0x36c/0x380 [fsl_dpaa2_eth]
>>
>> followed by
>>
>> [ 26.323016] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028
>> [ 26.324122] Mem abort info:
>> [ 26.324475] ESR = 0x0000000096000004
>> [ 26.324948] EC = 0x25: DABT (current EL), IL = 32 bits
>> [ 26.325618] SET = 0, FnV = 0
>> [ 26.326004] EA = 0, S1PTW = 0
>> [ 26.326406] FSC = 0x04: level 0 translation fault
>> [ 26.327021] Data abort info:
>> [ 26.327385] ISV = 0, ISS = 0x00000004
>> [ 26.327869] CM = 0, WnR = 0
>> [ 26.328244] user pgtable: 4k pages, 48-bit VAs, pgdp=00000020861cf000
>> [ 26.329055] [0000000000000028] pgd=0000000000000000, p4d=0000000000000000
>> [ 26.329912] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
>> [ 26.330702] Modules linked in: tag_dsa marvell mv88e6xxx aes_ce_blk caam_jr aes_ce_cipher caamhash_desc crct10dif_ce ghash_ce fsl_dpaa2_eth caamalg_desc xhci_plat_hcd sha256_generic gf128mul libsha256 libaes xhci_hcd crypto_engine pcs_lynx sha2_ce sha1_ce usbcore libdes sha256_arm64 cfg80211 dp83867 sha1_generic fsl_mc_dpio xgmac_mdio dpaa2_console dwc3 ahci ahci_qoriq udc_core caam libahci_platform roles error libahci usb_common libata at24 lm90 qoriq_thermal nvmem_layerscape_sfp sfp mdio_i2c
>> [ 26.336237] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G W 6.1.50-00121-g10168a070f4d #11
>> [ 26.337396] Hardware name: mpxlx2160a (DT)
>> [ 26.337956] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
>> [ 26.338833] pc : dpaa2_eth_free_tx_fd.isra.0+0xd4/0x380 [fsl_dpaa2_eth]
>> [ 26.339673] lr : dpaa2_eth_free_tx_fd.isra.0+0xb4/0x380 [fsl_dpaa2_eth]
>> [ 26.340512] sp : ffff800008cf3d70
>> [ 26.340931] x29: ffff800008cf3d70 x28: ffff002002900000 x27: 0000000000000000
>> [ 26.341832] x26: 0000000000000001 x25: 0000000000000001 x24: 0000000000000000
>> [ 26.342732] x23: 0000000000002328 x22: ffff002009742728 x21: 00000020884fffc2
>> [ 26.343633] x20: ffff002009740840 x19: ffff0020084fffc2 x18: 0000000000000018
>> [ 26.344534] x17: ffff8026b3a9a000 x16: ffff800008cf0000 x15: fffffffffffed3f8
>> [ 26.345435] x14: 0000000000000000 x13: ffff800008bad028 x12: 0000000000000966
>> [ 26.346335] x11: 0000000000000322 x10: ffff800008c09b58 x9 : ffff800008bad028
>> [ 26.347236] x8 : 0001000000000000 x7 : ffff0020095e6480 x6 : 00000020884fffc2
>> [ 26.348137] x5 : ffff0020095e6480 x4 : 0000000000000000 x3 : 0000000000000000
>> [ 26.349037] x2 : 00000000e7e00000 x1 : 0000000000000001 x0 : 0000000049759e0c
>> [ 26.349938] Call trace:
>> [ 26.350247] dpaa2_eth_free_tx_fd.isra.0+0xd4/0x380 [fsl_dpaa2_eth]
>> [ 26.351044] dpaa2_eth_tx_conf+0x84/0xc0 [fsl_dpaa2_eth]
>> [ 26.351720] dpaa2_eth_poll+0xec/0x3a4 [fsl_dpaa2_eth]
>> [ 26.352375] __napi_poll+0x34/0x180
>> [ 26.352816] net_rx_action+0x128/0x2b4
>> [ 26.353290] _stext+0x124/0x2a0
>> [ 26.353687] ____do_softirq+0xc/0x14
>> [ 26.354139] call_on_irq_stack+0x24/0x40
>> [ 26.354635] do_softirq_own_stack+0x18/0x2c
>> [ 26.355164] __irq_exit_rcu+0xc4/0xf0
>> [ 26.355628] irq_exit_rcu+0xc/0x14
>> [ 26.356059] el1_interrupt+0x34/0x60
>> [ 26.356511] el1h_64_irq_handler+0x14/0x20
>> [ 26.357028] el1h_64_irq+0x64/0x68
>> [ 26.357458] cpuidle_enter_state+0x12c/0x314
>> [ 26.357997] cpuidle_enter+0x34/0x4c
>> [ 26.358450] do_idle+0x208/0x270
>> [ 26.358860] cpu_startup_entry+0x24/0x30
>> [ 26.359356] secondary_start_kernel+0x128/0x14c
>> [ 26.359928] __secondary_switched+0x64/0x68
>> [ 26.360460] Code: 7100081f 54000d00 71000c1f 540000c0 (3940a360)
>> [ 26.361228] ---[ end trace 0000000000000000 ]---
>>
>> It happens when receiving big Ethernet frames on a AF_PACKET +
>> SOCK_RAW socket, for example MTU 9000. It does not happen with the
>> standard MTU 1500. It does not happen when just sending.
>>
>
> Are the transmitted frames also big?
Yes, size 9000 for both sendto() and recvfrom().
>
>> It's 100% reproducible here, however it seems to depend on the data
>> rate/load: Once it happened after receiving the first 80 frames,
>> another time after the first 300 frames, etc., and if I only send 5
>> frames per second, it does not happen at all.
>>
>> Please let me know if I should provide more info or do more tests. I
>> can provide a test program if needed.
>>
>
> If you can provide a test program, that would be great. It would help in
> reproducing and debugging the issue on my side.
OK, I've attached a test program, send_and_recv.c, reduced as far as I could get it. If I run it:
ip link set up dev eth6
ip link set mtu 9000 dev eth6
./send_and_recv eth6
it triggers the crash, sometimes very quickly, sometimes it takes a few seconds. In my case eth6 is DPAA2 MAC7, configured for MAC_LINK_TYPE_PHY + SGMII. However the same issue happens with MAC17 with MAC_LINK_TYPE_FIXED + RGMII and with MAC3/MAC4 XFI ports. The link must be up for the test; I just connect it to an external switch with nothing else attached.
Other things I've tried: sending only (and receiving on another machine), receiving only (while sending from another machine), and sending and receiving on separate interfaces on the LX2160A machine, but these cases did not seem to trigger the crash. So it looks like it's only in case both sending and receiving happens on the same interface.
Another detail I noticed: While the null pointer crash always happens, the "Wrong SWA type" warning does not always appear. It shows up only if certain byte values are written into the Ethernet frame payload (see comments in the test program). So perhaps that is a separate issue (and maybe my own fault for using a custom/invalid ethertype value).
>
> Ioana
>
>
[-- Attachment #2: send_and_recv.c --]
[-- Type: text/x-csrc, Size: 7200 bytes --]
#define _DEFAULT_SOURCE 1 // ensure net/if.h defines struct ifreq
#include <arpa/inet.h>
#include <errno.h>
#include <inttypes.h>
#include <linux/if_ether.h>
#include <linux/if_packet.h>
#include <net/ethernet.h>
#include <net/if.h>
#include <pthread.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/select.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>
#define BUFFER_SIZE 9000
#define CUSTOM_ETHER_TYPE 0x1234
#define SOCKET_PROTOCOL htons(ETH_P_ALL)
static void enable_promisc(int fd, unsigned int interface_index)
{
struct packet_mreq mr = {};
mr.mr_ifindex = interface_index;
mr.mr_type = PACKET_MR_PROMISC;
if (setsockopt(fd, SOL_PACKET, PACKET_ADD_MEMBERSHIP, &mr, sizeof(mr)) < 0) {
perror("setsockopt(SOL_PACKET/PACKET_ADD_MEMBERSHIP/PACKET_MR_PROMISC) failed");
exit(1);
}
}
static void bind_to_interface(int fd, unsigned int interface_index)
{
struct sockaddr_ll addr = {};
addr.sll_family = AF_PACKET;
addr.sll_protocol = SOCKET_PROTOCOL;
addr.sll_ifindex = interface_index;
if (bind(fd, (const struct sockaddr *)&addr, sizeof(addr)) < 0) {
perror("bind() failed");
exit(1);
}
}
static void get_interface_mac_addr(int fd, const char *interface, uint8_t *addr)
{
struct ifreq ifreq = {};
strncpy(ifreq.ifr_name, interface, IFNAMSIZ - 1);
if (ioctl(fd, SIOCGIFHWADDR, &ifreq) < 0) {
perror("ioctl(SIOCGIFHWADDR) failed");
exit(1);
}
memcpy(addr, ifreq.ifr_hwaddr.sa_data, IFHWADDRLEN);
}
static int setup_socket(const char *interface, unsigned int *interface_index)
{
*interface_index = if_nametoindex(interface);
if (*interface_index == 0) {
perror("if_nametoindex() failed");
exit(1);
}
int fd = socket(PF_PACKET, SOCK_RAW, SOCKET_PROTOCOL);
if (fd < 0) {
perror("socket() failed");
exit(1);
}
enable_promisc(fd, *interface_index);
bind_to_interface(fd, *interface_index);
return fd;
}
static void *do_send(void *arg)
{
const char *interface = arg;
unsigned int interface_index;
int fd = setup_socket(interface, &interface_index);
uint8_t srcaddr[6] = {};
get_interface_mac_addr(fd, interface, srcaddr);
// Destination address does not really matter, we are using promisc mode anyways for receiving
const uint8_t dstaddr[6] = {0x02, 0x1b, 0xfa, 0, 0, 0};
// build ethernet frame:
// 1. ethernet header
// 2. first 8 bytes of ethernet payload will be used for u64 counter (see below)
// 3. fill rest of ethernet payload with certain byte values:
// 0 or 0xFF does not trigger the "Wrong SWA type" message, but for example 1 or 0xA5 does.
// (but the null deref crash happens either way)
uint8_t buffer[BUFFER_SIZE] = {};
struct ether_header header = {};
memcpy(header.ether_dhost, dstaddr, sizeof header.ether_dhost);
memcpy(header.ether_shost, srcaddr, sizeof header.ether_shost);
header.ether_type = htons(CUSTOM_ETHER_TYPE);
memcpy(buffer, &header, sizeof(header));
memset(buffer + sizeof(struct ether_header) + sizeof(uint64_t), 1, BUFFER_SIZE - sizeof(struct ether_header) - sizeof(uint64_t));
uint64_t n_frames = 0;
while (1) {
fd_set fds;
FD_ZERO(&fds);
FD_SET(fd, &fds);
const int selected = select(fd + 1, NULL, &fds, NULL, NULL);
if (selected < 0) {
perror("select() failed");
return NULL;
}
if (FD_ISSET(fd, &fds)) {
n_frames++;
// put counter into frame payload
uint8_t *payload = buffer + sizeof(struct ether_header);
memcpy(payload, &n_frames, sizeof(n_frames));
struct sockaddr_ll dst = {};
dst.sll_family = AF_PACKET;
dst.sll_protocol = CUSTOM_ETHER_TYPE;
dst.sll_ifindex = interface_index;
dst.sll_halen = 6;
memcpy(dst.sll_addr, dstaddr, 6);
/*
printf("send frame %" PRIu64 ": size=%i"
" dst=%02x:%02x:%02x:%02x:%02x:%02x"
" src=%02x:%02x:%02x:%02x:%02x:%02x"
" ethertype=0x%04X "
"payload=[%02x %02x %02x %02x %02x %02x %02x %02x...]\n",
n_frames,
BUFFER_SIZE,
header.ether_dhost[0],
header.ether_dhost[1],
header.ether_dhost[2],
header.ether_dhost[3],
header.ether_dhost[4],
header.ether_dhost[5],
header.ether_shost[0],
header.ether_shost[1],
header.ether_shost[2],
header.ether_shost[3],
header.ether_shost[4],
header.ether_shost[5],
ntohs(header.ether_type),
payload[0],
payload[1],
payload[2],
payload[3],
payload[4],
payload[5],
payload[6],
payload[7]
);
*/
const ssize_t result = sendto(fd, buffer, BUFFER_SIZE, 0, (const struct sockaddr *)&dst, sizeof(dst));
if (result < 0) {
perror("sendto() failed");
return NULL;
}
const size_t size = result;
if (size != BUFFER_SIZE) {
fprintf(stderr, "sendto() sent only %zu of %i bytes", size, BUFFER_SIZE);
return NULL;
}
}
//usleep(1000000);
}
return NULL;
}
static int do_recv(const char *interface)
{
unsigned int interface_index;
int fd = setup_socket(interface, &interface_index);
uint8_t buffer[BUFFER_SIZE] = {};
uint64_t n_frames = 0;
while (1) {
fd_set fds;
FD_ZERO(&fds);
FD_SET(fd, &fds);
const int selected = select(fd + 1, &fds, NULL, NULL, NULL);
if (selected < 0) {
perror("select() failed");
return 1;
}
if (FD_ISSET(fd, &fds)) {
n_frames++;
struct sockaddr_ll srcaddr = {};
socklen_t srcaddrlen = sizeof(srcaddr);
const ssize_t result = recvfrom(fd, buffer, BUFFER_SIZE, 0, (struct sockaddr *)&srcaddr, &srcaddrlen);
if (result < 0) {
perror("recvfrom() failed");
return 1;
}
if (srcaddrlen != sizeof(srcaddr)) {
fprintf(stderr, "unexpected src sockaddr from recvfrom(): expected %zu bytes, actual %u\n", sizeof(srcaddr), srcaddrlen);
return 1;
}
const size_t size = result;
if (size < sizeof(struct ether_header)) {
fprintf(stderr, "packet too small for Ethernet header: %zu bytes", size);
return 1;
}
struct ether_header header;
memcpy(&header, buffer, sizeof(header));
const uint8_t *payload = buffer + sizeof(struct ether_header);
printf("recv frame %" PRIu64 ": ifindex=%i pkttype=%i size=%zu"
" dst=%02x:%02x:%02x:%02x:%02x:%02x"
" src=%02x:%02x:%02x:%02x:%02x:%02x"
" ethertype=0x%04X "
"payload=[%02x %02x %02x %02x %02x %02x %02x %02x...]\n",
n_frames,
srcaddr.sll_ifindex,
srcaddr.sll_pkttype,
size,
header.ether_dhost[0],
header.ether_dhost[1],
header.ether_dhost[2],
header.ether_dhost[3],
header.ether_dhost[4],
header.ether_dhost[5],
header.ether_shost[0],
header.ether_shost[1],
header.ether_shost[2],
header.ether_shost[3],
header.ether_shost[4],
header.ether_shost[5],
ntohs(header.ether_type),
payload[0],
payload[1],
payload[2],
payload[3],
payload[4],
payload[5],
payload[6],
payload[7]
);
}
}
return 0;
}
int main(int argc, char **argv)
{
if (argc < 2) {
fprintf(stderr, "usage: %s <interface>\n", argv[0]);
return 1;
}
char *interface = argv[1];
pthread_t thread;
int err = pthread_create(&thread, NULL, do_send, interface);
if (err != 0) {
errno = err;
perror("pthread_create() failed");
return 1;
}
do_recv(interface);
return 0;
}
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [bug] dpaa2-eth: "Wrong SWA type" and null deref in dpaa2_eth_free_tx_fd()
2023-10-06 14:03 ` Daniel Klauer
@ 2023-10-13 15:52 ` Ioana Ciornei
2023-10-27 9:12 ` Daniel Klauer
0 siblings, 1 reply; 5+ messages in thread
From: Ioana Ciornei @ 2023-10-13 15:52 UTC (permalink / raw)
To: Daniel Klauer; +Cc: netdev
[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]
On Fri, Oct 06, 2023 at 04:03:19PM +0200, Daniel Klauer wrote:
> On 04.10.23 17:50, Ioana Ciornei wrote:
> > On Wed, Aug 30, 2023 at 07:10:05PM +0200, Daniel Klauer wrote:
> >> Hi,
> >>
>
(...)
> >
> >> It's 100% reproducible here, however it seems to depend on the data
> >> rate/load: Once it happened after receiving the first 80 frames,
> >> another time after the first 300 frames, etc., and if I only send 5
> >> frames per second, it does not happen at all.
> >>
> >> Please let me know if I should provide more info or do more tests. I
> >> can provide a test program if needed.
> >>
> >
> > If you can provide a test program, that would be great. It would help in
> > reproducing and debugging the issue on my side.
>
> OK, I've attached a test program, send_and_recv.c, reduced as far as I could get it. If I run it:
>
> ip link set up dev eth6
> ip link set mtu 9000 dev eth6
> ./send_and_recv eth6
>
Thanks for the test program! I was able to reproduce the issue fairly easily.
I still do not know the root cause but it seems to always happen with
frames which are not 64 bytes aligned. I am afraid that the memory is
somehow corrupted between Tx and Tx conf.
The driver already has a PTR_ALIGN call in dpaa2_eth_build_single_fd()
but if there is not enough space in the skb's headroom then it will just
go ahead without it.
Attached you will find 2 patches which make the 64 bytes alignment a
must. With these patches applied onto net-next I do not see the issue
anymore.
Could you please also test on your side?
In the meantime, I will search internally for some more information on
the Tx alignment restrictions in DPAA2 and whether or not they are only
"nice to have" as the comment below suggests.
/* If there's enough room to align the FD address, do it.
* It will help hardware optimize accesses.
*/
Ioana
[-- Attachment #2: 0001-dpaa2-eth-increase-the-needed-headroom-to-account-fo.patch --]
[-- Type: text/x-diff, Size: 1810 bytes --]
From a8574f7e8e9aa01047f0e72a960209534a8257e8 Mon Sep 17 00:00:00 2001
From: Ioana Ciornei <ioana.ciornei@nxp.com>
Date: Fri, 13 Oct 2023 17:19:31 +0300
Subject: [PATCH 1/2] dpaa2-eth: increase the needed headroom to account for
alignment
Increase the needed headroom to account for a 64 byte alignment
restriction which, with this patch, we make mandatory on the Tx path.
The case in which the amount of headroom needed is not available is
already handled by the driver which instead sends the a S/G frame with
the first buffer only holding the SW and HW annotation areas.
Signed-off-by: Ioana Ciornei <ioana.ciornei@nxp.com>
---
drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c
index 15bab41cee48..5442d0e5fd66 100644
--- a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c
+++ b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c
@@ -1081,6 +1081,8 @@ static int dpaa2_eth_build_single_fd(struct dpaa2_eth_priv *priv,
DPAA2_ETH_TX_BUF_ALIGN);
if (aligned_start >= skb->head)
buffer_start = aligned_start;
+ else
+ return -ENOMEM;
/* Store a backpointer to the skb at the beginning of the buffer
* (in the private data area) such that we can release it
@@ -1412,7 +1414,7 @@ static netdev_tx_t __dpaa2_eth_tx(struct sk_buff *skb,
percpu_extras = this_cpu_ptr(priv->percpu_extras);
fd = (this_cpu_ptr(priv->fd))->array;
- needed_headroom = dpaa2_eth_needed_headroom(skb);
+ needed_headroom = dpaa2_eth_needed_headroom(skb) + DPAA2_ETH_TX_BUF_ALIGN;
/* We'll be holding a back-reference to the skb until Tx Confirmation;
* we don't want that overwritten by a concurrent Tx with a cloned skb.
--
2.25.1
[-- Attachment #3: 0002-dpaa2-eth-set-needed_headroom-to-the-maximum-value-n.patch --]
[-- Type: text/x-diff, Size: 1231 bytes --]
From 2eb596d0486adc5954f6d9263f153afddb3e470b Mon Sep 17 00:00:00 2001
From: Ioana Ciornei <ioana.ciornei@nxp.com>
Date: Fri, 13 Oct 2023 17:37:23 +0300
Subject: [PATCH 2/2] dpaa2-eth: set needed_headroom to the maximum value
needed
Set the needed_headroom net device field to the maximum value that the
driver would need on the Tx path:
- 64 bytes for the software annotation area
- 64 bytes for the hardware annotation area
- 64 bytes to account for a 64 byte aligned buffer address
Signed-off-by: Ioana Ciornei <ioana.ciornei@nxp.com>
---
drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c
index 5442d0e5fd66..9f029f3fe909 100644
--- a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c
+++ b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c
@@ -4969,6 +4969,8 @@ static int dpaa2_eth_probe(struct fsl_mc_device *dpni_dev)
if (err)
goto err_dl_port_add;
+ net_dev->needed_headroom = DPAA2_ETH_SWA_SIZE + DPAA2_ETH_TX_HWA_SIZE + DPAA2_ETH_TX_BUF_ALIGN;
+
err = register_netdev(net_dev);
if (err < 0) {
dev_err(dev, "register_netdev() failed\n");
--
2.25.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [bug] dpaa2-eth: "Wrong SWA type" and null deref in dpaa2_eth_free_tx_fd()
2023-10-13 15:52 ` Ioana Ciornei
@ 2023-10-27 9:12 ` Daniel Klauer
0 siblings, 0 replies; 5+ messages in thread
From: Daniel Klauer @ 2023-10-27 9:12 UTC (permalink / raw)
To: Ioana Ciornei; +Cc: netdev
On 13.10.23 17:52, Ioana Ciornei wrote:
> Thanks for the test program! I was able to reproduce the issue fairly easily.
>
> I still do not know the root cause but it seems to always happen with
> frames which are not 64 bytes aligned. I am afraid that the memory is
> somehow corrupted between Tx and Tx conf.
>
> The driver already has a PTR_ALIGN call in dpaa2_eth_build_single_fd()
> but if there is not enough space in the skb's headroom then it will just
> go ahead without it.
>
> Attached you will find 2 patches which make the 64 bytes alignment a
> must. With these patches applied onto net-next I do not see the issue
> anymore.
>
> Could you please also test on your side?
After applying the two patches, the test program runs without crashing.
The "wrong SWA type" warning is gone too. As before I've tested this with
our v6.1 and v6.5 kernels. Thanks!
>
> In the meantime, I will search internally for some more information on
> the Tx alignment restrictions in DPAA2 and whether or not they are only
> "nice to have" as the comment below suggests.
>
> /* If there's enough room to align the FD address, do it.
> * It will help hardware optimize accesses.
> */
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-10-27 9:35 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-08-30 17:10 [bug] dpaa2-eth: "Wrong SWA type" and null deref in dpaa2_eth_free_tx_fd() Daniel Klauer
2023-10-04 15:50 ` Ioana Ciornei
2023-10-06 14:03 ` Daniel Klauer
2023-10-13 15:52 ` Ioana Ciornei
2023-10-27 9:12 ` Daniel Klauer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.