* BUG cxgb3: Check and handle the dma mapping errors @ 2013-08-05 2:59 Alexey Kardashevskiy 2013-08-05 18:41 ` Jay Fenlason 0 siblings, 1 reply; 6+ messages in thread From: Alexey Kardashevskiy @ 2013-08-05 2:59 UTC (permalink / raw) To: Santosh Rastapur, Divy Le Ray Cc: Jay Fenlason, David S. Miller, netdev, Linux Kernel Mailing List Hi! Recently I started getting multiple errors like this: cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr c000001fbdaaa882 npages 1 cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr c000001fbdaaa882 npages 1 cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr c000001fbdaaa882 npages 1 cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr c000001fbdaaa882 npages 1 cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr c000001fbdaaa882 npages 1 cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr c000001fbdaaa882 npages 1 cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr c000001fbdaaa882 npages 1 ... and so on This is all happening on a PPC64 "powernv" platform machine. To trigger the error state, it is enough to _flood_ ping CXGB3 card from another machine (which has Emulex 10Gb NIC + Cisco switch). Just do "ping -f 172.20.1.2" and wait 10-15 seconds. The messages are coming from arch/powerpc/kernel/iommu.c and basically mean that the driver requested more pages than the DMA window has which is normally 1GB (there could be another possible source of errors - ppc_md.tce_build callback - but on powernv platform it always succeeds). The patch after which it broke is: commit f83331bab149e29fa2c49cf102c0cd8c3f1ce9f9 Author: Santosh Rastapur <santosh@chelsio.com> Date: Tue May 21 04:21:29 2013 +0000 cxgb3: Check and handle the dma mapping errors Any quick ideas? Thanks! -- Alexey ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BUG cxgb3: Check and handle the dma mapping errors 2013-08-05 2:59 BUG cxgb3: Check and handle the dma mapping errors Alexey Kardashevskiy @ 2013-08-05 18:41 ` Jay Fenlason 2013-08-06 0:15 ` Alexey Kardashevskiy 2013-08-07 16:55 ` Divy Le ray 0 siblings, 2 replies; 6+ messages in thread From: Jay Fenlason @ 2013-08-05 18:41 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Santosh Rastapur, Divy Le Ray, David S. Miller, netdev, Linux Kernel Mailing List On Mon, Aug 05, 2013 at 12:59:04PM +1000, Alexey Kardashevskiy wrote: > Hi! > > Recently I started getting multiple errors like this: > > cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr > c000001fbdaaa882 npages 1 > cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr > c000001fbdaaa882 npages 1 > cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr > c000001fbdaaa882 npages 1 > cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr > c000001fbdaaa882 npages 1 > cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr > c000001fbdaaa882 npages 1 > cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr > c000001fbdaaa882 npages 1 > cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr > c000001fbdaaa882 npages 1 > ... and so on > > This is all happening on a PPC64 "powernv" platform machine. To trigger the > error state, it is enough to _flood_ ping CXGB3 card from another machine > (which has Emulex 10Gb NIC + Cisco switch). Just do "ping -f 172.20.1.2" > and wait 10-15 seconds. > > > The messages are coming from arch/powerpc/kernel/iommu.c and basically > mean that the driver requested more pages than the DMA window has which is > normally 1GB (there could be another possible source of errors - > ppc_md.tce_build callback - but on powernv platform it always succeeds). > > > The patch after which it broke is: > commit f83331bab149e29fa2c49cf102c0cd8c3f1ce9f9 > Author: Santosh Rastapur <santosh@chelsio.com> > Date: Tue May 21 04:21:29 2013 +0000 > cxgb3: Check and handle the dma mapping errors > > Any quick ideas? Thanks! That patch adds error checking to detect failed dma mapping requests. Before it, the code always assumed that dma mapping requests succeded, whether they actually do or not, so the fact that the older kernel does not log errors only means that the failures are being ignored, and any appearance of working is through pure luck. The machine could have just crashed at that point. What is the observed behavior of the system by the machine initiating the ping flood? Do the older and newer kernels differ in the percentage of pings that do not receive replies? O the newer kernel, when the mapping errors are detected, the packet that it is trying to transmit is dropped, but I'm not at all sure what happens on the older kernel after the dma mapping fails. As I mentioned earlier, I'm surprised it does not crash. Perhaps the folks from Chelsio have a better idea what happens after a dma mapping error is ignored? -- JF ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BUG cxgb3: Check and handle the dma mapping errors 2013-08-05 18:41 ` Jay Fenlason @ 2013-08-06 0:15 ` Alexey Kardashevskiy 2013-08-07 16:55 ` Divy Le ray 1 sibling, 0 replies; 6+ messages in thread From: Alexey Kardashevskiy @ 2013-08-06 0:15 UTC (permalink / raw) To: Jay Fenlason Cc: Santosh Rastapur, Divy Le Ray, David S. Miller, netdev, Linux Kernel Mailing List On 08/06/2013 04:41 AM, Jay Fenlason wrote: > On Mon, Aug 05, 2013 at 12:59:04PM +1000, Alexey Kardashevskiy wrote: >> Hi! >> >> Recently I started getting multiple errors like this: >> >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> ... and so on >> >> This is all happening on a PPC64 "powernv" platform machine. To trigger the >> error state, it is enough to _flood_ ping CXGB3 card from another machine >> (which has Emulex 10Gb NIC + Cisco switch). Just do "ping -f 172.20.1.2" >> and wait 10-15 seconds. >> >> >> The messages are coming from arch/powerpc/kernel/iommu.c and basically >> mean that the driver requested more pages than the DMA window has which is >> normally 1GB (there could be another possible source of errors - >> ppc_md.tce_build callback - but on powernv platform it always succeeds). >> >> >> The patch after which it broke is: >> commit f83331bab149e29fa2c49cf102c0cd8c3f1ce9f9 >> Author: Santosh Rastapur <santosh@chelsio.com> >> Date: Tue May 21 04:21:29 2013 +0000 >> cxgb3: Check and handle the dma mapping errors >> >> Any quick ideas? Thanks! > > That patch adds error checking to detect failed dma mapping requests. > Before it, the code always assumed that dma mapping requests succeded, > whether they actually do or not, so the fact that the older kernel > does not log errors only means that the failures are being ignored, > and any appearance of working is through pure luck. The machine could > have just crashed at that point. >From what I see, the patch adds map_skb() function which is called in two new places, so the patch does not just mechanically replace skb_frag_dma_map() to map_skb() or something like that. > What is the observed behavior of the system by the machine initiating > the ping flood? Do the older and newer kernels differ in the > percentage of pings that do not receive replies? The other machine stops receiving replies. It is using different adapter, not Chelsio and the kernel version does not really matter. > O the newer kernel, > when the mapping errors are detected, the packet that it is trying to > transmit is dropped, but I'm not at all sure what happens on the older > kernel after the dma mapping fails. As I mentioned earlier, I'm > surprised it does not crash. Perhaps the folks from Chelsio have a > better idea what happens after a dma mapping error is ignored? Any kernel cannot avoid platform's iommu_alloc() on ppc64/powernv so if there was a problem, we would have seen messages (and yes, kernel would have crashed). -- Alexey ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BUG cxgb3: Check and handle the dma mapping errors 2013-08-05 18:41 ` Jay Fenlason 2013-08-06 0:15 ` Alexey Kardashevskiy @ 2013-08-07 16:55 ` Divy Le ray 2013-08-08 5:38 ` Alexey Kardashevskiy 1 sibling, 1 reply; 6+ messages in thread From: Divy Le ray @ 2013-08-07 16:55 UTC (permalink / raw) To: Jay Fenlason Cc: Alexey Kardashevskiy, Santosh Rastapur, David S. Miller, netdev, Linux Kernel Mailing List On 08/05/2013 11:41 AM, Jay Fenlason wrote: > On Mon, Aug 05, 2013 at 12:59:04PM +1000, Alexey Kardashevskiy wrote: >> Hi! >> >> Recently I started getting multiple errors like this: >> >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> ... and so on >> >> This is all happening on a PPC64 "powernv" platform machine. To trigger the >> error state, it is enough to _flood_ ping CXGB3 card from another machine >> (which has Emulex 10Gb NIC + Cisco switch). Just do "ping -f 172.20.1.2" >> and wait 10-15 seconds. >> >> >> The messages are coming from arch/powerpc/kernel/iommu.c and basically >> mean that the driver requested more pages than the DMA window has which is >> normally 1GB (there could be another possible source of errors - >> ppc_md.tce_build callback - but on powernv platform it always succeeds). >> >> >> The patch after which it broke is: >> commit f83331bab149e29fa2c49cf102c0cd8c3f1ce9f9 >> Author: Santosh Rastapur <santosh@chelsio.com> >> Date: Tue May 21 04:21:29 2013 +0000 >> cxgb3: Check and handle the dma mapping errors >> >> Any quick ideas? Thanks! > That patch adds error checking to detect failed dma mapping requests. > Before it, the code always assumed that dma mapping requests succeded, > whether they actually do or not, so the fact that the older kernel > does not log errors only means that the failures are being ignored, > and any appearance of working is through pure luck. The machine could > have just crashed at that point. > > What is the observed behavior of the system by the machine initiating > the ping flood? Do the older and newer kernels differ in the > percentage of pings that do not receive replies? O the newer kernel, > when the mapping errors are detected, the packet that it is trying to > transmit is dropped, but I'm not at all sure what happens on the older > kernel after the dma mapping fails. As I mentioned earlier, I'm > surprised it does not crash. Perhaps the folks from Chelsio have a > better idea what happens after a dma mapping error is ignored? Hi, It should definitely not be ignored. It should not happen this reliably either. I wonder if we are not hitting a leak of iommu entries. Cheers, Divy ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: BUG cxgb3: Check and handle the dma mapping errors 2013-08-07 16:55 ` Divy Le ray @ 2013-08-08 5:38 ` Alexey Kardashevskiy 2013-08-13 2:42 ` Alexey Kardashevskiy 0 siblings, 1 reply; 6+ messages in thread From: Alexey Kardashevskiy @ 2013-08-08 5:38 UTC (permalink / raw) To: Divy Le ray Cc: Jay Fenlason, Santosh Rastapur, David S. Miller, netdev, Linux Kernel Mailing List [-- Attachment #1: Type: text/plain, Size: 3477 bytes --] On 08/08/2013 02:55 AM, Divy Le ray wrote: > On 08/05/2013 11:41 AM, Jay Fenlason wrote: >> On Mon, Aug 05, 2013 at 12:59:04PM +1000, Alexey Kardashevskiy wrote: >>> Hi! >>> >>> Recently I started getting multiple errors like this: >>> >>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>> c000001fbdaaa882 npages 1 >>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>> c000001fbdaaa882 npages 1 >>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>> c000001fbdaaa882 npages 1 >>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>> c000001fbdaaa882 npages 1 >>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>> c000001fbdaaa882 npages 1 >>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>> c000001fbdaaa882 npages 1 >>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>> c000001fbdaaa882 npages 1 >>> ... and so on >>> >>> This is all happening on a PPC64 "powernv" platform machine. To trigger the >>> error state, it is enough to _flood_ ping CXGB3 card from another machine >>> (which has Emulex 10Gb NIC + Cisco switch). Just do "ping -f 172.20.1.2" >>> and wait 10-15 seconds. >>> >>> >>> The messages are coming from arch/powerpc/kernel/iommu.c and basically >>> mean that the driver requested more pages than the DMA window has which is >>> normally 1GB (there could be another possible source of errors - >>> ppc_md.tce_build callback - but on powernv platform it always succeeds). >>> >>> >>> The patch after which it broke is: >>> commit f83331bab149e29fa2c49cf102c0cd8c3f1ce9f9 >>> Author: Santosh Rastapur <santosh@chelsio.com> >>> Date: Tue May 21 04:21:29 2013 +0000 >>> cxgb3: Check and handle the dma mapping errors >>> >>> Any quick ideas? Thanks! >> That patch adds error checking to detect failed dma mapping requests. >> Before it, the code always assumed that dma mapping requests succeded, >> whether they actually do or not, so the fact that the older kernel >> does not log errors only means that the failures are being ignored, >> and any appearance of working is through pure luck. The machine could >> have just crashed at that point. >> >> What is the observed behavior of the system by the machine initiating >> the ping flood? Do the older and newer kernels differ in the >> percentage of pings that do not receive replies? O the newer kernel, >> when the mapping errors are detected, the packet that it is trying to >> transmit is dropped, but I'm not at all sure what happens on the older >> kernel after the dma mapping fails. As I mentioned earlier, I'm >> surprised it does not crash. Perhaps the folks from Chelsio have a >> better idea what happens after a dma mapping error is ignored? > > Hi, > > It should definitely not be ignored. It should not happen this reliably > either. > I wonder if we are not hitting a leak of iommu entries. Yes we do. I did some more tests with socklib from here http://junkcode.samba.org/ftp/unpacked/junkcode/socklib/ The test is basically sock_source sending packets to sock_sink. If block size is >=512 bytes, there is no leak, if I set packet size to <=256 bytes, it starts leaking, smaller block size means faster leak. The type of the other adapter does not really matter, can be the same Emulex adapter. I am attaching a small patch which I made in order to detect the leak. Without the patch, no leak happens, I double checked. -- Alexey [-- Attachment #2: cxgb3_debug.patch --] [-- Type: text/x-patch, Size: 5126 bytes --] commit 8327d4ca1d63a96454897ef7e5603bf78c9a76c5 Author: Alexey Kardashevskiy <aik@ozlabs.ru> AuthorDate: Thu Aug 8 15:33:55 2013 +1000 Commit: Alexey Kardashevskiy <aik@ozlabs.ru> CommitDate: Thu Aug 8 15:33:55 2013 +1000 cxgb3: debug patch Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> diff --git a/drivers/net/ethernet/chelsio/cxgb3/sge.c b/drivers/net/ethernet/chelsio/cxgb3/sge.c index 687ec4a..4165c0b 100644 --- a/drivers/net/ethernet/chelsio/cxgb3/sge.c +++ b/drivers/net/ethernet/chelsio/cxgb3/sge.c @@ -46,6 +46,15 @@ #include "firmware_exports.h" #include "cxgb3_offload.h" + +unsigned long _cxgb3_maps, _cxgb3_unmaps; +#define _FLUSH() printk("+%lu -%lu diff=%lu\n", _cxgb3_maps, _cxgb3_unmaps, _cxgb3_maps - _cxgb3_unmaps) +#define _INC(n) _cxgb3_maps += (n); if (printk_ratelimit()) _FLUSH() +#define _DEC(n) _cxgb3_unmaps += (n); if (printk_ratelimit()) _FLUSH() +static int backtraced = 0; +#define _CXBUG() _FLUSH(); if (!backtraced) { dump_stack(); backtraced = 1; } + + #define USE_GTS 0 #define SGE_RX_SM_BUF_SIZE 1536 @@ -246,6 +255,7 @@ static inline void unmap_skb(struct sk_buff *skb, struct sge_txq *q, if (frag_idx == 0 && skb_headlen(skb)) { pci_unmap_single(pdev, be64_to_cpu(sgp->addr[0]), skb_headlen(skb), PCI_DMA_TODEVICE); + _DEC(1); j = 1; } @@ -256,6 +266,7 @@ static inline void unmap_skb(struct sk_buff *skb, struct sge_txq *q, pci_unmap_page(pdev, be64_to_cpu(sgp->addr[j]), skb_frag_size(&skb_shinfo(skb)->frags[frag_idx]), PCI_DMA_TODEVICE); + _DEC(1); j ^= 1; if (j == 0) { sgp++; @@ -355,15 +366,19 @@ static void clear_rx_desc(struct pci_dev *pdev, const struct sge_fl *q, if (q->use_pages && d->pg_chunk.page) { (*d->pg_chunk.p_cnt)--; if (!*d->pg_chunk.p_cnt) + { pci_unmap_page(pdev, d->pg_chunk.mapping, q->alloc_size, PCI_DMA_FROMDEVICE); + _DEC(1); + } put_page(d->pg_chunk.page); d->pg_chunk.page = NULL; } else { pci_unmap_single(pdev, dma_unmap_addr(d, dma_addr), q->buf_size, PCI_DMA_FROMDEVICE); + _DEC(1); kfree_skb(d->skb); d->skb = NULL; } @@ -416,7 +431,11 @@ static inline int add_one_rx_buf(void *va, unsigned int len, mapping = pci_map_single(pdev, va, len, PCI_DMA_FROMDEVICE); if (unlikely(pci_dma_mapping_error(pdev, mapping))) + { + _CXBUG(); return -ENOMEM; + } + _INC(1); dma_unmap_addr_set(sd, dma_addr, mapping); @@ -458,8 +477,10 @@ static int alloc_pg_chunk(struct adapter *adapter, struct sge_fl *q, if (unlikely(pci_dma_mapping_error(adapter->pdev, mapping))) { __free_pages(q->pg_chunk.page, order); q->pg_chunk.page = NULL; + _CXBUG(); return -EIO; } + _INC(1); q->pg_chunk.mapping = mapping; } sd->pg_chunk = q->pg_chunk; @@ -816,6 +837,7 @@ recycle: use_orig_buf: pci_unmap_single(adap->pdev, dma_unmap_addr(sd, dma_addr), fl->buf_size, PCI_DMA_FROMDEVICE); + _DEC(1); skb = sd->skb; skb_put(skb, len); __refill_fl(adap, fl); @@ -887,10 +909,13 @@ recycle: PCI_DMA_FROMDEVICE); (*sd->pg_chunk.p_cnt)--; if (!*sd->pg_chunk.p_cnt && sd->pg_chunk.page != fl->pg_chunk.page) + { pci_unmap_page(adap->pdev, sd->pg_chunk.mapping, fl->alloc_size, PCI_DMA_FROMDEVICE); + _DEC(1); + } if (!skb) { __skb_put(newskb, SGE_RX_PULL_LEN); memcpy(newskb->data, sd->pg_chunk.va, SGE_RX_PULL_LEN); @@ -972,7 +997,7 @@ static int map_skb(struct pci_dev *pdev, const struct sk_buff *skb, PCI_DMA_TODEVICE); if (pci_dma_mapping_error(pdev, *addr)) goto out_err; - + _INC(1); si = skb_shinfo(skb); end = &si->frags[si->nr_frags]; @@ -981,16 +1006,23 @@ static int map_skb(struct pci_dev *pdev, const struct sk_buff *skb, DMA_TO_DEVICE); if (pci_dma_mapping_error(pdev, *addr)) goto unwind; + _INC(1); } return 0; unwind: + _CXBUG(); while (fp-- > si->frags) + { dma_unmap_page(&pdev->dev, *--addr, skb_frag_size(fp), DMA_TO_DEVICE); + _DEC(1); + } pci_unmap_single(pdev, addr[-1], skb_headlen(skb), PCI_DMA_TODEVICE); + _DEC(1); out_err: + _CXBUG(); return -ENOMEM; } @@ -1584,13 +1616,18 @@ static void deferred_unmap_destructor(struct sk_buff *skb) p = dui->addr; if (skb_tail_pointer(skb) - skb_transport_header(skb)) + { pci_unmap_single(dui->pdev, *p++, skb_tail_pointer(skb) - skb_transport_header(skb), PCI_DMA_TODEVICE); - + _DEC(1); + } si = skb_shinfo(skb); for (i = 0; i < si->nr_frags; i++) + { pci_unmap_page(dui->pdev, *p++, skb_frag_size(&si->frags[i]), PCI_DMA_TODEVICE); + _DEC(1); + } } static void setup_deferred_unmapping(struct sk_buff *skb, struct pci_dev *pdev, @@ -2143,11 +2180,13 @@ static void lro_add_page(struct adapter *adap, struct sge_qset *qs, (*sd->pg_chunk.p_cnt)--; if (!*sd->pg_chunk.p_cnt && sd->pg_chunk.page != fl->pg_chunk.page) + { pci_unmap_page(adap->pdev, sd->pg_chunk.mapping, fl->alloc_size, PCI_DMA_FROMDEVICE); - + _DEC(1); + } if (!skb) { put_page(sd->pg_chunk.page); if (complete) ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: BUG cxgb3: Check and handle the dma mapping errors 2013-08-08 5:38 ` Alexey Kardashevskiy @ 2013-08-13 2:42 ` Alexey Kardashevskiy 0 siblings, 0 replies; 6+ messages in thread From: Alexey Kardashevskiy @ 2013-08-13 2:42 UTC (permalink / raw) To: Divy Le ray Cc: Jay Fenlason, Santosh Rastapur, David S. Miller, netdev, Linux Kernel Mailing List Ping, anyone? On 08/08/2013 03:38 PM, Alexey Kardashevskiy wrote: > On 08/08/2013 02:55 AM, Divy Le ray wrote: >> On 08/05/2013 11:41 AM, Jay Fenlason wrote: >>> On Mon, Aug 05, 2013 at 12:59:04PM +1000, Alexey Kardashevskiy wrote: >>>> Hi! >>>> >>>> Recently I started getting multiple errors like this: >>>> >>>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>>> c000001fbdaaa882 npages 1 >>>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>>> c000001fbdaaa882 npages 1 >>>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>>> c000001fbdaaa882 npages 1 >>>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>>> c000001fbdaaa882 npages 1 >>>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>>> c000001fbdaaa882 npages 1 >>>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>>> c000001fbdaaa882 npages 1 >>>> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >>>> c000001fbdaaa882 npages 1 >>>> ... and so on >>>> >>>> This is all happening on a PPC64 "powernv" platform machine. To trigger the >>>> error state, it is enough to _flood_ ping CXGB3 card from another machine >>>> (which has Emulex 10Gb NIC + Cisco switch). Just do "ping -f 172.20.1.2" >>>> and wait 10-15 seconds. >>>> >>>> >>>> The messages are coming from arch/powerpc/kernel/iommu.c and basically >>>> mean that the driver requested more pages than the DMA window has which is >>>> normally 1GB (there could be another possible source of errors - >>>> ppc_md.tce_build callback - but on powernv platform it always succeeds). >>>> >>>> >>>> The patch after which it broke is: >>>> commit f83331bab149e29fa2c49cf102c0cd8c3f1ce9f9 >>>> Author: Santosh Rastapur <santosh@chelsio.com> >>>> Date: Tue May 21 04:21:29 2013 +0000 >>>> cxgb3: Check and handle the dma mapping errors >>>> >>>> Any quick ideas? Thanks! >>> That patch adds error checking to detect failed dma mapping requests. >>> Before it, the code always assumed that dma mapping requests succeded, >>> whether they actually do or not, so the fact that the older kernel >>> does not log errors only means that the failures are being ignored, >>> and any appearance of working is through pure luck. The machine could >>> have just crashed at that point. >>> >>> What is the observed behavior of the system by the machine initiating >>> the ping flood? Do the older and newer kernels differ in the >>> percentage of pings that do not receive replies? O the newer kernel, >>> when the mapping errors are detected, the packet that it is trying to >>> transmit is dropped, but I'm not at all sure what happens on the older >>> kernel after the dma mapping fails. As I mentioned earlier, I'm >>> surprised it does not crash. Perhaps the folks from Chelsio have a >>> better idea what happens after a dma mapping error is ignored? >> >> Hi, >> >> It should definitely not be ignored. It should not happen this reliably >> either. >> I wonder if we are not hitting a leak of iommu entries. > > Yes we do. I did some more tests with socklib from here > http://junkcode.samba.org/ftp/unpacked/junkcode/socklib/ > > The test is basically sock_source sending packets to sock_sink. If block > size is >=512 bytes, there is no leak, if I set packet size to <=256 bytes, > it starts leaking, smaller block size means faster leak. The type of the > other adapter does not really matter, can be the same Emulex adapter. > > I am attaching a small patch which I made in order to detect the leak. > Without the patch, no leak happens, I double checked. > > -- Alexey ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-08-13 2:42 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-08-05 2:59 BUG cxgb3: Check and handle the dma mapping errors Alexey Kardashevskiy 2013-08-05 18:41 ` Jay Fenlason 2013-08-06 0:15 ` Alexey Kardashevskiy 2013-08-07 16:55 ` Divy Le ray 2013-08-08 5:38 ` Alexey Kardashevskiy 2013-08-13 2:42 ` Alexey Kardashevskiy
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).