From: Ioana Ciornei <ioana.ciornei@nxp.com>
To: Jeremy Linton <jeremy.linton@arm.com>
Cc: Hamza Mahfooz <someguy@effective-light.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Christoph Hellwig <hch@lst.de>,
Marek Szyprowski <m.szyprowski@samsung.com>,
Robin Murphy <robin.murphy@arm.com>,
"iommu@lists.linux-foundation.org"
<iommu@lists.linux-foundation.org>,
Dan Williams <dan.j.williams@intel.com>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>
Subject: Re: DPAA2 triggers, [PATCH] dma debug: report -EEXIST errors in add_dma_entry
Date: Tue, 14 Sep 2021 15:45:06 +0000 [thread overview]
Message-ID: <20210914154504.z6vqxuh3byqwgfzx@skbuf> (raw)
In-Reply-To: <fd67fbac-64bf-f0ea-01e1-5938ccfab9d0@arm.com>
On Wed, Sep 08, 2021 at 10:33:26PM -0500, Jeremy Linton wrote:
> +DPAA2, netdev maintainers
> Hi,
>
> On 5/18/21 7:54 AM, Hamza Mahfooz wrote:
> > Since, overlapping mappings are not supported by the DMA API we should
> > report an error if active_cacheline_insert returns -EEXIST.
>
> It seems this patch found a victim. I was trying to run iperf3 on a
> honeycomb (5.14.0, fedora 35) and the console is blasting this error message
> at 100% cpu. So, I changed it to a WARN_ONCE() to get the call trace, which
> is attached below.
>
>
> [ 151.839693] cacheline tracking EEXIST, overlapping mappings aren't
> supported
> ...
> [ 151.924397] Hardware name: SolidRun Ltd. SolidRun CEX7 Platform, BIOS
> EDK II Aug 9 2021
> [ 151.932481] pstate: 40400005 (nZcv daif +PAN -UAO -TCO BTYPE=--)
> [ 151.938483] pc : add_dma_entry+0x218/0x240
> [ 151.942575] lr : add_dma_entry+0x218/0x240
> [ 151.946666] sp : ffff8000101e2f20
> [ 151.949975] x29: ffff8000101e2f20 x28: ffffaf317ac85000 x27:
> ffff3d0366ecb3a0
> [ 151.957116] x26: 0000040000000000 x25: 0000000000000001 x24:
> ffffaf317bbe8908
> [ 151.964257] x23: 0000000000000001 x22: ffffaf317bbe8810 x21:
> 0000000000000000
> [ 151.971397] x20: 0000000082e48000 x19: ffffaf317be6e000 x18:
> ffffffffffffffff
> [ 151.978537] x17: 646574726f707075 x16: 732074276e657261 x15:
> ffff8000901e2c2f
> [ 151.985676] x14: 0000000000000000 x13: 0000000000000000 x12:
> 0000000000000000
> [ 151.992816] x11: ffffaf317bb4c4c0 x10: 00000000ffffe000 x9 :
> ffffaf3179708060
> [ 151.999956] x8 : 00000000ffffdfff x7 : ffffaf317bb4c4c0 x6 :
> 0000000000000001
> [ 152.007096] x5 : ffff3d0a9af66e30 x4 : 0000000000000000 x3 :
> 0000000000000027
> [ 152.014236] x2 : 0000000000000023 x1 : ffff3d0360aac000 x0 :
> 0000000000000040
> [ 152.021376] Call trace:
> [ 152.023816] add_dma_entry+0x218/0x240
> [ 152.027561] debug_dma_map_sg+0x118/0x17c
> [ 152.031566] dma_map_sg_attrs+0x70/0xb0
> [ 152.035397] dpaa2_eth_build_sg_fd+0xac/0x2f0 [fsl_dpaa2_eth]
> [ 152.041150] __dpaa2_eth_tx+0x3ec/0x570 [fsl_dpaa2_eth]
> [ 152.046377] dpaa2_eth_tx+0x74/0x110 [fsl_dpaa2_eth]
> [ 152.051342] dev_hard_start_xmit+0xe8/0x1a4
> [ 152.055523] sch_direct_xmit+0x8c/0x1e0
> [ 152.059355] __dev_xmit_skb+0x484/0x6a0
> [ 152.063186] __dev_queue_xmit+0x380/0x744
> [ 152.067190] dev_queue_xmit+0x20/0x2c
> [ 152.070848] neigh_hh_output+0xb4/0x130
> [ 152.074679] ip_finish_output2+0x494/0x8f0
> [ 152.078770] __ip_finish_output+0x12c/0x230
> [ 152.082948] ip_finish_output+0x40/0xe0
> [ 152.086778] ip_output+0xe4/0x2d4
> [ 152.090088] __ip_queue_xmit+0x1b4/0x5c0
> [ 152.094006] ip_queue_xmit+0x20/0x30
> [ 152.097576] __tcp_transmit_skb+0x3b8/0x7b4
> [ 152.101755] tcp_write_xmit+0x350/0x8e0
> [ 152.105586] __tcp_push_pending_frames+0x48/0x110
> [ 152.110286] tcp_rcv_established+0x338/0x690
> [ 152.114550] tcp_v4_do_rcv+0x1c0/0x29c
> [ 152.118294] tcp_v4_rcv+0xd14/0xe3c
> [ 152.121777] ip_protocol_deliver_rcu+0x88/0x340
> [ 152.126302] ip_local_deliver_finish+0xc0/0x184
> [ 152.130827] ip_local_deliver+0x7c/0x23c
> [ 152.134744] ip_rcv_finish+0xb4/0x100
> [ 152.138400] ip_rcv+0x54/0x210
> [ 152.141449] deliver_skb+0x74/0xdc
> [ 152.144846] __netif_receive_skb_core.constprop.0+0x250/0x81c
> [ 152.150588] __netif_receive_skb_list_core+0x94/0x264
> [ 152.155635] netif_receive_skb_list_internal+0x1d0/0x3bc
> [ 152.160942] netif_receive_skb_list+0x38/0x70
> [ 152.165295] dpaa2_eth_poll+0x168/0x350 [fsl_dpaa2_eth]
> [ 152.170521] __napi_poll.constprop.0+0x40/0x19c
> [ 152.175047] net_rx_action+0x2c4/0x360
> [ 152.178792] __do_softirq+0x1b0/0x394
> [ 152.182450] run_ksoftirqd+0x68/0xa0
> [ 152.186023] smpboot_thread_fn+0x13c/0x270
> [ 152.190115] kthread+0x138/0x140
>
I got some time to look at this and I am not sure if it's an actual
problem or not.
First of all, I added some more debug prints when any overlapping
happens so that I can actually see the entries.
[ 245.927020] fsl_dpaa2_eth dpni.3: scather-gather idx 0 P=20a7320000 N=20a7320 D=20a7320000 L=30 DMA_BIDIRECTIONAL dma map error check not applicable·
[ 245.927048] fsl_dpaa2_eth dpni.3: scather-gather idx 1 P=20a7320030 N=20a7320 D=20a7320030 L=5a8 DMA_BIDIRECTIONAL dma map error check not applicable
[ 245.927062] DMA-API: cacheline tracking EEXIST, overlapping mappings aren't supported
The first line is the dump of the dma_debug_entry which is already present
in the radix tree and the second one is the entry which just triggered
the EEXIST.
As we can see, they are not actually overlapping, at least from my
understanding. The first one starts at 0x20a7320000 with a size 0x30
and the second one at 0x20a7320030.
I wanted to see where these mappings are originating so I added some
traces around the dma_[un]map_single, dma_[un]map_sg operations in
dpaa2-eth.
I can see the following:
- There are two S/G skbs being sent one after another (no cleanup of
the Tx confirmation is done between, so not kfree or dma_unmap is
happening).
- Skb#1 has 3 frags and skb#2 just 2 frags.
- The skb#1 3rd frag will get into the same cacheline as the skb#2 2nd frag.
skb#1:
245.926981: dpaa2_eth:dpaa2_dma_map_sg: [eth4] scl=0x0xffff4cf3ac188200 num_sg=3
245.926984: dpaa2_eth:dpaa2_dma_map_sg_entry: [eth4] size=0x42 dma_addr=0x20b66e58fe
245.926987: dpaa2_eth:dpaa2_dma_map_sg_entry: [eth4] size=0x578 dma_addr=0x20ab9c7a88
245.926989: dpaa2_eth:dpaa2_dma_map_sg_entry: [eth4] size=0x30 dma_addr=0x20a7320000
^
|
|
This skb frag will land in the same cacheline number with the 2nd frag from skb#2.
skb#2:
245.934933: dpaa2_eth:dpaa2_dma_map_sg: [eth4] scl=0x0xffff4cf3ac188300 num_sg=2
245.934936: dpaa2_eth:dpaa2_dma_map_sg_entry: [eth4] size=0x42 dma_addr=0x20b66e60fe
245.934939: dpaa2_eth:dpaa2_dma_map_sg_entry: [eth4] size=0x5a8 dma_addr=0x20a7320030
^
|
These frags are allocated by the stack, transformed into a scatterlist
by skb_to_sgvec and then DMA mapped with dma_map_sg. It was not the
dpaa2-eth's decision to use two fragments from the same page (that will
also end un in the same cacheline) in two different in-flight skbs.
Is this behavior normal?
Ioana
WARNING: multiple messages have this Message-ID (diff)
From: Ioana Ciornei <ioana.ciornei@nxp.com>
To: Jeremy Linton <jeremy.linton@arm.com>
Cc: Hamza Mahfooz <someguy@effective-light.com>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"iommu@lists.linux-foundation.org"
<iommu@lists.linux-foundation.org>,
Dan Williams <dan.j.williams@intel.com>,
Robin Murphy <robin.murphy@arm.com>,
Christoph Hellwig <hch@lst.de>
Subject: Re: DPAA2 triggers, [PATCH] dma debug: report -EEXIST errors in add_dma_entry
Date: Tue, 14 Sep 2021 15:45:06 +0000 [thread overview]
Message-ID: <20210914154504.z6vqxuh3byqwgfzx@skbuf> (raw)
In-Reply-To: <fd67fbac-64bf-f0ea-01e1-5938ccfab9d0@arm.com>
On Wed, Sep 08, 2021 at 10:33:26PM -0500, Jeremy Linton wrote:
> +DPAA2, netdev maintainers
> Hi,
>
> On 5/18/21 7:54 AM, Hamza Mahfooz wrote:
> > Since, overlapping mappings are not supported by the DMA API we should
> > report an error if active_cacheline_insert returns -EEXIST.
>
> It seems this patch found a victim. I was trying to run iperf3 on a
> honeycomb (5.14.0, fedora 35) and the console is blasting this error message
> at 100% cpu. So, I changed it to a WARN_ONCE() to get the call trace, which
> is attached below.
>
>
> [ 151.839693] cacheline tracking EEXIST, overlapping mappings aren't
> supported
> ...
> [ 151.924397] Hardware name: SolidRun Ltd. SolidRun CEX7 Platform, BIOS
> EDK II Aug 9 2021
> [ 151.932481] pstate: 40400005 (nZcv daif +PAN -UAO -TCO BTYPE=--)
> [ 151.938483] pc : add_dma_entry+0x218/0x240
> [ 151.942575] lr : add_dma_entry+0x218/0x240
> [ 151.946666] sp : ffff8000101e2f20
> [ 151.949975] x29: ffff8000101e2f20 x28: ffffaf317ac85000 x27:
> ffff3d0366ecb3a0
> [ 151.957116] x26: 0000040000000000 x25: 0000000000000001 x24:
> ffffaf317bbe8908
> [ 151.964257] x23: 0000000000000001 x22: ffffaf317bbe8810 x21:
> 0000000000000000
> [ 151.971397] x20: 0000000082e48000 x19: ffffaf317be6e000 x18:
> ffffffffffffffff
> [ 151.978537] x17: 646574726f707075 x16: 732074276e657261 x15:
> ffff8000901e2c2f
> [ 151.985676] x14: 0000000000000000 x13: 0000000000000000 x12:
> 0000000000000000
> [ 151.992816] x11: ffffaf317bb4c4c0 x10: 00000000ffffe000 x9 :
> ffffaf3179708060
> [ 151.999956] x8 : 00000000ffffdfff x7 : ffffaf317bb4c4c0 x6 :
> 0000000000000001
> [ 152.007096] x5 : ffff3d0a9af66e30 x4 : 0000000000000000 x3 :
> 0000000000000027
> [ 152.014236] x2 : 0000000000000023 x1 : ffff3d0360aac000 x0 :
> 0000000000000040
> [ 152.021376] Call trace:
> [ 152.023816] add_dma_entry+0x218/0x240
> [ 152.027561] debug_dma_map_sg+0x118/0x17c
> [ 152.031566] dma_map_sg_attrs+0x70/0xb0
> [ 152.035397] dpaa2_eth_build_sg_fd+0xac/0x2f0 [fsl_dpaa2_eth]
> [ 152.041150] __dpaa2_eth_tx+0x3ec/0x570 [fsl_dpaa2_eth]
> [ 152.046377] dpaa2_eth_tx+0x74/0x110 [fsl_dpaa2_eth]
> [ 152.051342] dev_hard_start_xmit+0xe8/0x1a4
> [ 152.055523] sch_direct_xmit+0x8c/0x1e0
> [ 152.059355] __dev_xmit_skb+0x484/0x6a0
> [ 152.063186] __dev_queue_xmit+0x380/0x744
> [ 152.067190] dev_queue_xmit+0x20/0x2c
> [ 152.070848] neigh_hh_output+0xb4/0x130
> [ 152.074679] ip_finish_output2+0x494/0x8f0
> [ 152.078770] __ip_finish_output+0x12c/0x230
> [ 152.082948] ip_finish_output+0x40/0xe0
> [ 152.086778] ip_output+0xe4/0x2d4
> [ 152.090088] __ip_queue_xmit+0x1b4/0x5c0
> [ 152.094006] ip_queue_xmit+0x20/0x30
> [ 152.097576] __tcp_transmit_skb+0x3b8/0x7b4
> [ 152.101755] tcp_write_xmit+0x350/0x8e0
> [ 152.105586] __tcp_push_pending_frames+0x48/0x110
> [ 152.110286] tcp_rcv_established+0x338/0x690
> [ 152.114550] tcp_v4_do_rcv+0x1c0/0x29c
> [ 152.118294] tcp_v4_rcv+0xd14/0xe3c
> [ 152.121777] ip_protocol_deliver_rcu+0x88/0x340
> [ 152.126302] ip_local_deliver_finish+0xc0/0x184
> [ 152.130827] ip_local_deliver+0x7c/0x23c
> [ 152.134744] ip_rcv_finish+0xb4/0x100
> [ 152.138400] ip_rcv+0x54/0x210
> [ 152.141449] deliver_skb+0x74/0xdc
> [ 152.144846] __netif_receive_skb_core.constprop.0+0x250/0x81c
> [ 152.150588] __netif_receive_skb_list_core+0x94/0x264
> [ 152.155635] netif_receive_skb_list_internal+0x1d0/0x3bc
> [ 152.160942] netif_receive_skb_list+0x38/0x70
> [ 152.165295] dpaa2_eth_poll+0x168/0x350 [fsl_dpaa2_eth]
> [ 152.170521] __napi_poll.constprop.0+0x40/0x19c
> [ 152.175047] net_rx_action+0x2c4/0x360
> [ 152.178792] __do_softirq+0x1b0/0x394
> [ 152.182450] run_ksoftirqd+0x68/0xa0
> [ 152.186023] smpboot_thread_fn+0x13c/0x270
> [ 152.190115] kthread+0x138/0x140
>
I got some time to look at this and I am not sure if it's an actual
problem or not.
First of all, I added some more debug prints when any overlapping
happens so that I can actually see the entries.
[ 245.927020] fsl_dpaa2_eth dpni.3: scather-gather idx 0 P=20a7320000 N=20a7320 D=20a7320000 L=30 DMA_BIDIRECTIONAL dma map error check not applicable·
[ 245.927048] fsl_dpaa2_eth dpni.3: scather-gather idx 1 P=20a7320030 N=20a7320 D=20a7320030 L=5a8 DMA_BIDIRECTIONAL dma map error check not applicable
[ 245.927062] DMA-API: cacheline tracking EEXIST, overlapping mappings aren't supported
The first line is the dump of the dma_debug_entry which is already present
in the radix tree and the second one is the entry which just triggered
the EEXIST.
As we can see, they are not actually overlapping, at least from my
understanding. The first one starts at 0x20a7320000 with a size 0x30
and the second one at 0x20a7320030.
I wanted to see where these mappings are originating so I added some
traces around the dma_[un]map_single, dma_[un]map_sg operations in
dpaa2-eth.
I can see the following:
- There are two S/G skbs being sent one after another (no cleanup of
the Tx confirmation is done between, so not kfree or dma_unmap is
happening).
- Skb#1 has 3 frags and skb#2 just 2 frags.
- The skb#1 3rd frag will get into the same cacheline as the skb#2 2nd frag.
skb#1:
245.926981: dpaa2_eth:dpaa2_dma_map_sg: [eth4] scl=0x0xffff4cf3ac188200 num_sg=3
245.926984: dpaa2_eth:dpaa2_dma_map_sg_entry: [eth4] size=0x42 dma_addr=0x20b66e58fe
245.926987: dpaa2_eth:dpaa2_dma_map_sg_entry: [eth4] size=0x578 dma_addr=0x20ab9c7a88
245.926989: dpaa2_eth:dpaa2_dma_map_sg_entry: [eth4] size=0x30 dma_addr=0x20a7320000
^
|
|
This skb frag will land in the same cacheline number with the 2nd frag from skb#2.
skb#2:
245.934933: dpaa2_eth:dpaa2_dma_map_sg: [eth4] scl=0x0xffff4cf3ac188300 num_sg=2
245.934936: dpaa2_eth:dpaa2_dma_map_sg_entry: [eth4] size=0x42 dma_addr=0x20b66e60fe
245.934939: dpaa2_eth:dpaa2_dma_map_sg_entry: [eth4] size=0x5a8 dma_addr=0x20a7320030
^
|
These frags are allocated by the stack, transformed into a scatterlist
by skb_to_sgvec and then DMA mapped with dma_map_sg. It was not the
dpaa2-eth's decision to use two fragments from the same page (that will
also end un in the same cacheline) in two different in-flight skbs.
Is this behavior normal?
Ioana
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
next prev parent reply other threads:[~2021-09-14 15:45 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-18 12:54 [PATCH] dma debug: report -EEXIST errors in add_dma_entry Hamza Mahfooz
2021-05-18 12:54 ` Hamza Mahfooz
2021-06-22 7:41 ` Christoph Hellwig
2021-06-22 7:41 ` Christoph Hellwig
2021-09-09 3:33 ` DPAA2 triggers, " Jeremy Linton
2021-09-09 3:33 ` Jeremy Linton
2021-09-09 21:16 ` Ioana Ciornei
2021-09-09 21:16 ` Ioana Ciornei
2021-09-10 10:23 ` Christoph Hellwig
2021-09-10 10:23 ` Christoph Hellwig
2021-09-14 15:45 ` Ioana Ciornei [this message]
2021-09-14 15:45 ` Ioana Ciornei
2021-09-30 13:37 ` Karsten Graul
2021-09-30 13:37 ` Karsten Graul
2021-10-01 12:52 ` Gerald Schaefer
2021-10-01 12:52 ` Gerald Schaefer
2021-10-06 13:10 ` Gerald Schaefer
2021-10-06 13:10 ` Gerald Schaefer
2021-10-06 13:21 ` Gerald Schaefer
2021-10-06 13:21 ` Gerald Schaefer
2021-10-06 14:23 ` Robin Murphy
2021-10-06 14:23 ` Robin Murphy
2021-10-06 15:06 ` Gerald Schaefer
2021-10-06 15:06 ` Gerald Schaefer
2021-10-07 10:59 ` Karsten Graul
2021-10-07 10:59 ` Karsten Graul
2021-10-07 16:40 ` Gerald Schaefer
2021-10-07 16:40 ` Gerald Schaefer
2021-10-11 11:47 ` Christoph Hellwig
2021-10-11 11:47 ` Christoph Hellwig
2021-10-01 4:19 ` Christoph Hellwig
2021-10-01 4:19 ` Christoph Hellwig
2021-10-01 9:21 ` Ioana Ciornei
2021-10-01 9:21 ` Ioana Ciornei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210914154504.z6vqxuh3byqwgfzx@skbuf \
--to=ioana.ciornei@nxp.com \
--cc=dan.j.williams@intel.com \
--cc=hch@lst.de \
--cc=iommu@lists.linux-foundation.org \
--cc=jeremy.linton@arm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=m.szyprowski@samsung.com \
--cc=netdev@vger.kernel.org \
--cc=robin.murphy@arm.com \
--cc=someguy@effective-light.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.