* PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-16 13:58 Chris Pringle 2009-06-16 14:16 ` Michael S. Zick ` (2 more replies) 0 siblings, 3 replies; 36+ messages in thread From: Chris Pringle @ 2009-06-16 13:58 UTC (permalink / raw) To: linux-kernel Hello All, We're developing on a Freescale MPC8272 and are having some nasty problems with PCI bus mastering and data corruption. We have some custom hardware that is bus mastering, reading data from the CPUs memory for it's own use. Most of the time, the data is correct, however occasionally we are seeing data that appears to be from somewhere else in memory (usually memory that has already been read by the PCI device). The problem looks like stale data on the PCI bridge prefetch buffers or a cache coherency problem, but we've been unable to come up with a solution to our problem. It is my understanding that it shouldn't be a cache coherency problem as the CPU cache should be snooped as the data is read from memory. Even if it were an issue, the pci_map_sg* functions should have sorted out any cache coherency issues before the DMA operation started. I've not been able to find anything on the Freescale data sheet that provides any way of flushing the prefetch cache on the PCI bridge. We've done a bit of experimenting, and found that turning off prefetch appears to solve (or possibly mask?) the problem (at the expensive of massive performance problems). I've also tried DMA'ing two adjacent userspace buffers in memory (from the same page), and see corruption on the second buffer. If I populate both buffers, then DMA them both, the data is fine. If I populate the first, DMA the first, then populate the second and DMA the second, corruption occurs at the start of the second buffer. If I add 8-32 bytes of padding between the buffers, the problem goes away. The PCI spec says that the PCI bridge is supposed to flush any data from it's prefetch buffers that are not read by the bus master, so technically, this isn't supposed to happen. I've tried making sure that buffers are cache line (and page) aligned, and are multiples of cache lines, but it's made no difference. PIO mode works fine, and I've checked the data with the CPU just before, and immediately after the DMA and the driver sees no data integrity issues. There are memory write barriers just before the DMA start, so all the registers should be correct before the DMA starts. For background info, the device doing the bus mastering is a Xilinx Virtex 5 FPGA. We've monitored the data as it comes off the PCI bus using ChipScope - so the firmware should not be manipulating the data in any way. We have some hardware/firmware/drivers that has a lot of common code that runs on an x86 platform (as opposed to powerpc), and that works without any issues whatsoever. Has anyone got any ideas what this might be? Does anyone of know issues with PCI bridges on the PowerPC platform? Is there extra things that need to be done from the driver when DMAing on PowerPC (I've looked at other drivers and there's nothing obvious). The chip errata doesn't have anything on it that looks like it could cause this. I'm really hoping this is something that we're doing wrong in the driver or the firmware, but we've been through both the firmware and drivers countless times and are unable to see anything wrong. Any thoughts/ideas would be much appreciated! Regards, Chris -- ______________________________ Chris Pringle Software Engineer Miranda Technologies Ltd. Hithercroft Road Wallingford Oxfordshire OX10 9DG UK Tel. +44 1491 820206 Fax. +44 1491 820001 www.miranda.com ____________________________ Miranda Technologies Limited Registered in England and Wales CN 02017053 Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 13:58 PowerPC PCI DMA issues (prefetch/coherency?) Chris Pringle @ 2009-06-16 14:16 ` Michael S. Zick 2009-06-16 14:59 ` Hu Gang 2009-06-16 16:21 ` Scott Wood 2 siblings, 0 replies; 36+ messages in thread From: Michael S. Zick @ 2009-06-16 14:16 UTC (permalink / raw) To: Chris Pringle; +Cc: linux-kernel On Tue June 16 2009, Chris Pringle wrote: > Hello All, > > We're developing on a Freescale MPC8272 and are having some nasty > problems with PCI bus mastering and data corruption. > > We have some custom hardware that is bus mastering, reading data from > the CPUs memory for it's own use. Most of the time, the data is correct, > however occasionally we are seeing data that appears to be from > somewhere else in memory (usually memory that has already been read by > the PCI device). The problem looks like stale data on the PCI bridge > prefetch buffers or a cache coherency problem, but we've been unable to > come up with a solution to our problem. It is my understanding that it > shouldn't be a cache coherency problem as the CPU cache should be > snooped as the data is read from memory. Even if it were an issue, the > pci_map_sg* functions should have sorted out any cache coherency issues > before the DMA operation started. > > I've not been able to find anything on the Freescale data sheet that > provides any way of flushing the prefetch cache on the PCI bridge. We've > done a bit of experimenting, and found that turning off prefetch appears > to solve (or possibly mask?) the problem (at the expensive of massive > performance problems). I've also tried DMA'ing two adjacent userspace > buffers in memory (from the same page), and see corruption on the second > buffer. If I populate both buffers, then DMA them both, the data is > fine. If I populate the first, DMA the first, then populate the second > and DMA the second, corruption occurs at the start of the second buffer. > If I add 8-32 bytes of padding between the buffers, the problem goes away. > > The PCI spec says that the PCI bridge is supposed to flush any data from > it's prefetch buffers that are not read by the bus master, so > technically, this isn't supposed to happen. > > I've tried making sure that buffers are cache line (and page) aligned, > and are multiples of cache lines, but it's made no difference. PIO mode > works fine, and I've checked the data with the CPU just before, and > immediately after the DMA and the driver sees no data integrity issues. > There are memory write barriers just before the DMA start, so all the > registers should be correct before the DMA starts. > > For background info, the device doing the bus mastering is a Xilinx > Virtex 5 FPGA. We've monitored the data as it comes off the PCI bus > using ChipScope - so the firmware should not be manipulating the data in > any way. > > We have some hardware/firmware/drivers that has a lot of common code > that runs on an x86 platform (as opposed to powerpc), and that works > without any issues whatsoever. > > Has anyone got any ideas what this might be? Does anyone of know issues > with PCI bridges on the PowerPC platform? Is there extra things that > need to be done from the driver when DMAing on PowerPC (I've looked at > other drivers and there's nothing obvious). The chip errata doesn't have > anything on it that looks like it could cause this. > > I'm really hoping this is something that we're doing wrong in the driver > or the firmware, but we've been through both the firmware and drivers > countless times and are unable to see anything wrong. > > Any thoughts/ideas would be much appreciated! > Did you actually check the load image for proper alignment? Like in: gen2-32 compressed # objdump -x vmlinux.bin - - - - - Sections: Idx Name Size VMA LMA File off Algn - - - - - 21 .data.page_aligned 00001800 c068a000 0068a000 0058b000 2**12 CONTENTS, ALLOC, LOAD, DATA 22 .data.cacheline_aligned 000026c0 c068b800 0068b800 0058c800 2**6 CONTENTS, ALLOC, LOAD, DATA 23 .data.read_mostly 00001e98 c068dec0 0068dec0 0058eec0 2**6 CONTENTS, ALLOC, LOAD, DATA = = = = I had to make this change to the x86 loader script to get alignment for VIA: diff --git a/arch/x86/kernel/vmlinux_32.lds.S b/arch/x86/kernel/vmlinux_32.lds.S index 62ad500..26f68a5 100644 --- a/arch/x86/kernel/vmlinux_32.lds.S +++ b/arch/x86/kernel/vmlinux_32.lds.S @@ -82,7 +82,7 @@ SECTIONS *(.data.idt) } - . = ALIGN(32); + . = ALIGN(L1_CACHE_BYTES); .data.cacheline_aligned : AT(ADDR(.data.cacheline_aligned) - LOAD_OFFSET) { *(.data.cacheline_aligned) } = = = = Eyeball your loader script if you haven't already done so. Mike > Regards, > Chris > ^ permalink raw reply related [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 13:58 PowerPC PCI DMA issues (prefetch/coherency?) Chris Pringle 2009-06-16 14:16 ` Michael S. Zick @ 2009-06-16 14:59 ` Hu Gang 2009-06-16 16:21 ` Scott Wood 2 siblings, 0 replies; 36+ messages in thread From: Hu Gang @ 2009-06-16 14:59 UTC (permalink / raw) To: Chris Pringle; +Cc: linux-kernel On Tue, 16 Jun 2009 14:58:27 +0100 Chris Pringle <chris.pringle@oxtel.com> wrote: > Hello All, > > We're developing on a Freescale MPC8272 and are having some nasty > problems with PCI bus mastering and data corruption. > > We have some custom hardware that is bus mastering, reading data from > the CPUs memory for it's own use. Most of the time, the data is correct, > however occasionally we are seeing data that appears to be from > somewhere else in memory (usually memory that has already been read by > the PCI device). The problem looks like stale data on the PCI bridge > prefetch buffers or a cache coherency problem, but we've been unable to > come up with a solution to our problem. It is my understanding that it > shouldn't be a cache coherency problem as the CPU cache should be > snooped as the data is read from memory. Even if it were an issue, the > pci_map_sg* functions should have sorted out any cache coherency issues > before the DMA operation started. > > I've not been able to find anything on the Freescale data sheet that > provides any way of flushing the prefetch cache on the PCI bridge. We've > done a bit of experimenting, and found that turning off prefetch appears > to solve (or possibly mask?) the problem (at the expensive of massive > performance problems). I've also tried DMA'ing two adjacent userspace > buffers in memory (from the same page), and see corruption on the second > buffer. If I populate both buffers, then DMA them both, the data is > fine. If I populate the first, DMA the first, then populate the second > and DMA the second, corruption occurs at the start of the second buffer. > If I add 8-32 bytes of padding between the buffers, the problem goes away. > > The PCI spec says that the PCI bridge is supposed to flush any data from > it's prefetch buffers that are not read by the bus master, so > technically, this isn't supposed to happen. > > I've tried making sure that buffers are cache line (and page) aligned, > and are multiples of cache lines, but it's made no difference. PIO mode > works fine, and I've checked the data with the CPU just before, and > immediately after the DMA and the driver sees no data integrity issues. > There are memory write barriers just before the DMA start, so all the > registers should be correct before the DMA starts. > > For background info, the device doing the bus mastering is a Xilinx > Virtex 5 FPGA. We've monitored the data as it comes off the PCI bus > using ChipScope - so the firmware should not be manipulating the data in > any way. > > We have some hardware/firmware/drivers that has a lot of common code > that runs on an x86 platform (as opposed to powerpc), and that works > without any issues whatsoever. > > Has anyone got any ideas what this might be? Does anyone of know issues > with PCI bridges on the PowerPC platform? Is there extra things that > need to be done from the driver when DMAing on PowerPC (I've looked at > other drivers and there's nothing obvious). The chip errata doesn't have > anything on it that looks like it could cause this. > > I'm really hoping this is something that we're doing wrong in the driver > or the firmware, but we've been through both the firmware and drivers > countless times and are unable to see anything wrong. > > Any thoughts/ideas would be much appreciated! > > Regards, > Chris Can you try allocate the data buffer using dma_alloc_coherent, that can making the data coherency. I got the similar issues in the arm cpu with a dma device base on spartan3, using the dma_alloc_coherent making it works well. hope this help. thanks. -- steve ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 13:58 PowerPC PCI DMA issues (prefetch/coherency?) Chris Pringle 2009-06-16 14:16 ` Michael S. Zick 2009-06-16 14:59 ` Hu Gang @ 2009-06-16 16:21 ` Scott Wood 2009-06-16 16:34 ` Chris Pringle 2 siblings, 1 reply; 36+ messages in thread From: Scott Wood @ 2009-06-16 16:21 UTC (permalink / raw) To: Chris Pringle; +Cc: linux-kernel On Tue, Jun 16, 2009 at 02:58:27PM +0100, Chris Pringle wrote: > We're developing on a Freescale MPC8272 and are having some nasty > problems with PCI bus mastering and data corruption. What kernel version? What firmware? Custom board, or one in upstream (which one)? > We have some custom hardware that is bus mastering, reading data from > the CPUs memory for it's own use. Most of the time, the data is correct, > however occasionally we are seeing data that appears to be from > somewhere else in memory (usually memory that has already been read by > the PCI device). The problem looks like stale data on the PCI bridge > prefetch buffers or a cache coherency problem, but we've been unable to > come up with a solution to our problem. It is my understanding that it > shouldn't be a cache coherency problem as the CPU cache should be > snooped as the data is read from memory. Even if it were an issue, the > pci_map_sg* functions should have sorted out any cache coherency issues > before the DMA operation started. Cache coherency on PCI DMA requires that the memory be mapped with the M attribute on this chip, but that should be happening based on detection of the core. Also make sure that you park the bus on PCI and raise its arbitration priority, as done at the end of fixup_pci in arch/powerpc/boot/cuboot-pq2.c. BTW, you may want to post to linuxppc-dev@lists.ozlabs.org for powerpc-specific issues, especially this kind of hardware issue. -Scott ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 16:21 ` Scott Wood @ 2009-06-16 16:34 ` Chris Pringle 2009-06-16 16:46 ` Scott Wood 0 siblings, 1 reply; 36+ messages in thread From: Chris Pringle @ 2009-06-16 16:34 UTC (permalink / raw) To: Scott Wood; +Cc: linux-kernel Scott Wood wrote: > On Tue, Jun 16, 2009 at 02:58:27PM +0100, Chris Pringle wrote: > >> We're developing on a Freescale MPC8272 and are having some nasty >> problems with PCI bus mastering and data corruption. >> > > What kernel version? What firmware? Custom board, or one in upstream > (which one)? > The kernel version is 2.6.26. Firmware is custom on a custom board. > Cache coherency on PCI DMA requires that the memory be mapped with the M > attribute on this chip, but that should be happening based on detection of > the core. > I'm not sure where to look to verify this? > Also make sure that you park the bus on PCI and raise its arbitration > priority, as done at the end of fixup_pci in arch/powerpc/boot/cuboot-pq2.c. > Since this is a reasonably recent kernel, I'd guess that both of these things are correct. I've had a quick look in that file and there is code in there raising arbitartion priority and parking the bus. > BTW, you may want to post to linuxppc-dev@lists.ozlabs.org for > powerpc-specific issues, especially this kind of hardware issue. > > I've just posted there :-) Interestingly, I've just turned off cache snooping and the problem has got much worse. This has surprised me as I thought that part of the job done by pci_map_sg was to flush the CPU cache - so I wasn't expecting this to make any difference. The memory corruption with the cache snooping turned off is almost identical to that which I'm seeing with it turned on, only much, much worse. -- ______________________________ Chris Pringle Software Engineer Miranda Technologies Ltd. Hithercroft Road Wallingford Oxfordshire OX10 9DG UK Tel. +44 1491 820206 Fax. +44 1491 820001 www.miranda.com ____________________________ Miranda Technologies Limited Registered in England and Wales CN 02017053 Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 16:34 ` Chris Pringle @ 2009-06-16 16:46 ` Scott Wood 0 siblings, 0 replies; 36+ messages in thread From: Scott Wood @ 2009-06-16 16:46 UTC (permalink / raw) To: Chris Pringle; +Cc: linux-kernel, linuxppc-dev@ozlabs.org list Chris Pringle wrote: > The kernel version is 2.6.26. Firmware is custom on a custom board. >> Cache coherency on PCI DMA requires that the memory be mapped with the M >> attribute on this chip, but that should be happening based on >> detection of >> the core. >> > I'm not sure where to look to verify this? Check asm/cputable.h for CPU_FTR_NEED_COHERENT. Make sure that CONFIG_8260 is one of the #ifdefs that turns that on. It looks like that was in place by 2.6.26 in arch/powerpc. I'm not sure what to look for in arch/ppc. >> Also make sure that you park the bus on PCI and raise its arbitration >> priority, as done at the end of fixup_pci in >> arch/powerpc/boot/cuboot-pq2.c. >> > Since this is a reasonably recent kernel, Not really, there was a fair amount of 82xx work in the mid-2.6.20s. The addition of CPU_FTR_NEED_COHERENT to 82xx was somewhere in that time. Can you try 2.6.30? > I'd guess that both of these > things are correct. I've had a quick look in that file and there is code > in there raising arbitartion priority and parking the bus. Just because the code is there doesn't mean you're using it -- are you using cuImage? Are you using arch/ppc or arch/powerpc? Typically this would be done by firmware; it's only in cuboot because u-boot wasn't doing it. >> BTW, you may want to post to linuxppc-dev@lists.ozlabs.org for >> powerpc-specific issues, especially this kind of hardware issue. >> >> > I've just posted there :-) Doh, sorry. :-) > Interestingly, I've just turned off cache snooping and the problem has > got much worse. This has surprised me as I thought that part of the job > done by pci_map_sg was to flush the CPU cache It only flushes the cache on hardware that doesn't do coherent DMA. -Scott ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-16 16:46 ` Scott Wood 0 siblings, 0 replies; 36+ messages in thread From: Scott Wood @ 2009-06-16 16:46 UTC (permalink / raw) To: Chris Pringle; +Cc: linuxppc-dev@ozlabs.org list, linux-kernel Chris Pringle wrote: > The kernel version is 2.6.26. Firmware is custom on a custom board. >> Cache coherency on PCI DMA requires that the memory be mapped with the M >> attribute on this chip, but that should be happening based on >> detection of >> the core. >> > I'm not sure where to look to verify this? Check asm/cputable.h for CPU_FTR_NEED_COHERENT. Make sure that CONFIG_8260 is one of the #ifdefs that turns that on. It looks like that was in place by 2.6.26 in arch/powerpc. I'm not sure what to look for in arch/ppc. >> Also make sure that you park the bus on PCI and raise its arbitration >> priority, as done at the end of fixup_pci in >> arch/powerpc/boot/cuboot-pq2.c. >> > Since this is a reasonably recent kernel, Not really, there was a fair amount of 82xx work in the mid-2.6.20s. The addition of CPU_FTR_NEED_COHERENT to 82xx was somewhere in that time. Can you try 2.6.30? > I'd guess that both of these > things are correct. I've had a quick look in that file and there is code > in there raising arbitartion priority and parking the bus. Just because the code is there doesn't mean you're using it -- are you using cuImage? Are you using arch/ppc or arch/powerpc? Typically this would be done by firmware; it's only in cuboot because u-boot wasn't doing it. >> BTW, you may want to post to linuxppc-dev@lists.ozlabs.org for >> powerpc-specific issues, especially this kind of hardware issue. >> >> > I've just posted there :-) Doh, sorry. :-) > Interestingly, I've just turned off cache snooping and the problem has > got much worse. This has surprised me as I thought that part of the job > done by pci_map_sg was to flush the CPU cache It only flushes the cache on hardware that doesn't do coherent DMA. -Scott ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 16:46 ` Scott Wood @ 2009-06-16 16:57 ` Chris Pringle -1 siblings, 0 replies; 36+ messages in thread From: Chris Pringle @ 2009-06-16 16:57 UTC (permalink / raw) To: Scott Wood; +Cc: linux-kernel, linuxppc-dev@ozlabs.org list Scott Wood wrote: > > Check asm/cputable.h for CPU_FTR_NEED_COHERENT. Make sure that > CONFIG_8260 is one of the #ifdefs that turns that on. It looks like > that was in place by 2.6.26 in arch/powerpc. I'm not sure what to > look for in arch/ppc. I've just checked that and it's definitely switched on in CPU_FTR_COMMON (CONFIG_8260 is also being used). > >>> Also make sure that you park the bus on PCI and raise its arbitration >>> priority, as done at the end of fixup_pci in >>> arch/powerpc/boot/cuboot-pq2.c. >>> >> Since this is a reasonably recent kernel, > > Not really, there was a fair amount of 82xx work in the mid-2.6.20s. > The addition of CPU_FTR_NEED_COHERENT to 82xx was somewhere in that time. > > Can you try 2.6.30? I'll give it a try, but that won't be a quick thing to do - will hopefully manage to get that done tomorrow if it patches without too many issues. I should point out that we've got the low latency patches on this kernel too; I guess it'd be worth trying it without them before I move kernels. > >> I'd guess that both of these things are correct. I've had a quick >> look in that file and there is code in there raising arbitartion >> priority and parking the bus. > > Just because the code is there doesn't mean you're using it -- are you > using cuImage? Are you using arch/ppc or arch/powerpc? > > Typically this would be done by firmware; it's only in cuboot because > u-boot wasn't doing it. Just checked this is being called and it is. We're using arch/powerpc. > > >> Interestingly, I've just turned off cache snooping and the problem >> has got much worse. This has surprised me as I thought that part of >> the job done by pci_map_sg was to flush the CPU cache > > It only flushes the cache on hardware that doesn't do coherent DMA. > Ah right - that would explain what we're seeing then... Doh. Thought I might have been onto something then. Is there any way to force a cache flush? That'd at least prove it was a caching issue if it resolved the problem. Thanks, Chris -- ______________________________ Chris Pringle Software Engineer Miranda Technologies Ltd. Hithercroft Road Wallingford Oxfordshire OX10 9DG UK Tel. +44 1491 820206 Fax. +44 1491 820001 www.miranda.com ____________________________ Miranda Technologies Limited Registered in England and Wales CN 02017053 Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-16 16:57 ` Chris Pringle 0 siblings, 0 replies; 36+ messages in thread From: Chris Pringle @ 2009-06-16 16:57 UTC (permalink / raw) To: Scott Wood; +Cc: linuxppc-dev@ozlabs.org list, linux-kernel Scott Wood wrote: > > Check asm/cputable.h for CPU_FTR_NEED_COHERENT. Make sure that > CONFIG_8260 is one of the #ifdefs that turns that on. It looks like > that was in place by 2.6.26 in arch/powerpc. I'm not sure what to > look for in arch/ppc. I've just checked that and it's definitely switched on in CPU_FTR_COMMON (CONFIG_8260 is also being used). > >>> Also make sure that you park the bus on PCI and raise its arbitration >>> priority, as done at the end of fixup_pci in >>> arch/powerpc/boot/cuboot-pq2.c. >>> >> Since this is a reasonably recent kernel, > > Not really, there was a fair amount of 82xx work in the mid-2.6.20s. > The addition of CPU_FTR_NEED_COHERENT to 82xx was somewhere in that time. > > Can you try 2.6.30? I'll give it a try, but that won't be a quick thing to do - will hopefully manage to get that done tomorrow if it patches without too many issues. I should point out that we've got the low latency patches on this kernel too; I guess it'd be worth trying it without them before I move kernels. > >> I'd guess that both of these things are correct. I've had a quick >> look in that file and there is code in there raising arbitartion >> priority and parking the bus. > > Just because the code is there doesn't mean you're using it -- are you > using cuImage? Are you using arch/ppc or arch/powerpc? > > Typically this would be done by firmware; it's only in cuboot because > u-boot wasn't doing it. Just checked this is being called and it is. We're using arch/powerpc. > > >> Interestingly, I've just turned off cache snooping and the problem >> has got much worse. This has surprised me as I thought that part of >> the job done by pci_map_sg was to flush the CPU cache > > It only flushes the cache on hardware that doesn't do coherent DMA. > Ah right - that would explain what we're seeing then... Doh. Thought I might have been onto something then. Is there any way to force a cache flush? That'd at least prove it was a caching issue if it resolved the problem. Thanks, Chris -- ______________________________ Chris Pringle Software Engineer Miranda Technologies Ltd. Hithercroft Road Wallingford Oxfordshire OX10 9DG UK Tel. +44 1491 820206 Fax. +44 1491 820001 www.miranda.com ____________________________ Miranda Technologies Limited Registered in England and Wales CN 02017053 Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 16:57 ` Chris Pringle @ 2009-06-16 17:03 ` Scott Wood -1 siblings, 0 replies; 36+ messages in thread From: Scott Wood @ 2009-06-16 17:03 UTC (permalink / raw) To: Chris Pringle; +Cc: linux-kernel, linuxppc-dev@ozlabs.org list Chris Pringle wrote: > Ah right - that would explain what we're seeing then... Doh. Thought I > might have been onto something then. Is there any way to force a cache > flush? That'd at least prove it was a caching issue if it resolved the > problem. You could enable CONFIG_NOT_COHERENT_CACHE. -Scott ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-16 17:03 ` Scott Wood 0 siblings, 0 replies; 36+ messages in thread From: Scott Wood @ 2009-06-16 17:03 UTC (permalink / raw) To: Chris Pringle; +Cc: linuxppc-dev@ozlabs.org list, linux-kernel Chris Pringle wrote: > Ah right - that would explain what we're seeing then... Doh. Thought I > might have been onto something then. Is there any way to force a cache > flush? That'd at least prove it was a caching issue if it resolved the > problem. You could enable CONFIG_NOT_COHERENT_CACHE. -Scott ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 17:03 ` Scott Wood (?) @ 2009-06-16 17:43 ` Arnd Bergmann 2009-06-16 17:49 ` Scott Wood -1 siblings, 1 reply; 36+ messages in thread From: Arnd Bergmann @ 2009-06-16 17:43 UTC (permalink / raw) To: linuxppc-dev; +Cc: Scott Wood, Chris Pringle, linux-kernel On Tuesday 16 June 2009, Scott Wood wrote: > Chris Pringle wrote: > > Ah right - that would explain what we're seeing then... Doh. Thought I > > might have been onto something then. Is there any way to force a cache > > flush? That'd at least prove it was a caching issue if it resolved the > > problem. > > You could enable CONFIG_NOT_COHERENT_CACHE. If the whole system is noncoherent, that is the right solution. If the device is the only one, you can also use dma_alloc_noncoherent() and flush explicitly with dma_cache_sync(). Arnd <>< ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 17:43 ` Arnd Bergmann @ 2009-06-16 17:49 ` Scott Wood 0 siblings, 0 replies; 36+ messages in thread From: Scott Wood @ 2009-06-16 17:49 UTC (permalink / raw) To: Arnd Bergmann; +Cc: linuxppc-dev, Chris Pringle, linux-kernel Arnd Bergmann wrote: > On Tuesday 16 June 2009, Scott Wood wrote: >> Chris Pringle wrote: >>> Ah right - that would explain what we're seeing then... Doh. Thought I >>> might have been onto something then. Is there any way to force a cache >>> flush? That'd at least prove it was a caching issue if it resolved the >>> problem. >> You could enable CONFIG_NOT_COHERENT_CACHE. > > If the whole system is noncoherent, that is the right solution. I meant it more as a test than a permanent solution... > If the > device is the only one, you can also use dma_alloc_noncoherent() and > flush explicitly with dma_cache_sync(). I don't see how that would help -- aren't those also controlled by CONFIG_NOT_COHERENT_CACHE? -Scott ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-16 17:49 ` Scott Wood 0 siblings, 0 replies; 36+ messages in thread From: Scott Wood @ 2009-06-16 17:49 UTC (permalink / raw) To: Arnd Bergmann; +Cc: Chris Pringle, linuxppc-dev, linux-kernel Arnd Bergmann wrote: > On Tuesday 16 June 2009, Scott Wood wrote: >> Chris Pringle wrote: >>> Ah right - that would explain what we're seeing then... Doh. Thought I >>> might have been onto something then. Is there any way to force a cache >>> flush? That'd at least prove it was a caching issue if it resolved the >>> problem. >> You could enable CONFIG_NOT_COHERENT_CACHE. > > If the whole system is noncoherent, that is the right solution. I meant it more as a test than a permanent solution... > If the > device is the only one, you can also use dma_alloc_noncoherent() and > flush explicitly with dma_cache_sync(). I don't see how that would help -- aren't those also controlled by CONFIG_NOT_COHERENT_CACHE? -Scott ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 17:49 ` Scott Wood @ 2009-06-16 18:02 ` Arnd Bergmann -1 siblings, 0 replies; 36+ messages in thread From: Arnd Bergmann @ 2009-06-16 18:02 UTC (permalink / raw) To: Scott Wood; +Cc: linuxppc-dev, Chris Pringle, linux-kernel On Tuesday 16 June 2009, Scott Wood wrote: > > If the > > device is the only one, you can also use dma_alloc_noncoherent() and > > flush explicitly with dma_cache_sync(). > > I don't see how that would help -- aren't those also controlled by > CONFIG_NOT_COHERENT_CACHE? Ah, yes you are right. PowerPC implements dma_alloc_noncoherent as dma_alloc_coherent, so dma_cache_sync() is actually a NOP (or should be). Actually there seems to be a bug in here: Since dma_alloc_noncoherent gives you a coherent mapping (or NULL) on noncoherent machines, dma_cache_sync() is redundant and should not actually flush the cache, or we should change dma_alloc_noncoherent to do a simple alloc_pages on CONFIG_NON_COHERENT_CACHE and leave dma_cache_sync() as it is. Arnd <>< ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-16 18:02 ` Arnd Bergmann 0 siblings, 0 replies; 36+ messages in thread From: Arnd Bergmann @ 2009-06-16 18:02 UTC (permalink / raw) To: Scott Wood; +Cc: Chris Pringle, linuxppc-dev, linux-kernel On Tuesday 16 June 2009, Scott Wood wrote: > > If the > > device is the only one, you can also use dma_alloc_noncoherent() and > > flush explicitly with dma_cache_sync(). > > I don't see how that would help -- aren't those also controlled by > CONFIG_NOT_COHERENT_CACHE? Ah, yes you are right. PowerPC implements dma_alloc_noncoherent as dma_alloc_coherent, so dma_cache_sync() is actually a NOP (or should be). Actually there seems to be a bug in here: Since dma_alloc_noncoherent gives you a coherent mapping (or NULL) on noncoherent machines, dma_cache_sync() is redundant and should not actually flush the cache, or we should change dma_alloc_noncoherent to do a simple alloc_pages on CONFIG_NON_COHERENT_CACHE and leave dma_cache_sync() as it is. Arnd <>< ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 18:02 ` Arnd Bergmann (?) @ 2009-06-17 0:18 ` Benjamin Herrenschmidt 2009-06-17 0:37 ` FUJITA Tomonori -1 siblings, 1 reply; 36+ messages in thread From: Benjamin Herrenschmidt @ 2009-06-17 0:18 UTC (permalink / raw) To: Arnd Bergmann; +Cc: Scott Wood, Chris Pringle, linuxppc-dev, linux-kernel On Tue, 2009-06-16 at 20:02 +0200, Arnd Bergmann wrote: > On Tuesday 16 June 2009, Scott Wood wrote: > > > If the > > > device is the only one, you can also use dma_alloc_noncoherent() and > > > flush explicitly with dma_cache_sync(). > > > > I don't see how that would help -- aren't those also controlled by > > CONFIG_NOT_COHERENT_CACHE? > > Ah, yes you are right. PowerPC implements dma_alloc_noncoherent as > dma_alloc_coherent, so dma_cache_sync() is actually a NOP (or should be). But we still need to sync the result of dma_map_* when used multiple times for a single mapping. Cheers, Ben. > Actually there seems to be a bug in here: Since dma_alloc_noncoherent > gives you a coherent mapping (or NULL) on noncoherent machines, > dma_cache_sync() is redundant and should not actually flush the > cache, or we should change dma_alloc_noncoherent to do a simple > alloc_pages on CONFIG_NON_COHERENT_CACHE and leave dma_cache_sync() > as it is. > > Arnd <>< > _______________________________________________ > Linuxppc-dev mailing list > Linuxppc-dev@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/linuxppc-dev ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-17 0:18 ` Benjamin Herrenschmidt @ 2009-06-17 0:37 ` FUJITA Tomonori 0 siblings, 0 replies; 36+ messages in thread From: FUJITA Tomonori @ 2009-06-17 0:37 UTC (permalink / raw) To: benh; +Cc: arnd, scottwood, chris.pringle, linuxppc-dev, linux-kernel On Wed, 17 Jun 2009 10:18:45 +1000 Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote: > On Tue, 2009-06-16 at 20:02 +0200, Arnd Bergmann wrote: > > On Tuesday 16 June 2009, Scott Wood wrote: > > > > If the > > > > device is the only one, you can also use dma_alloc_noncoherent() and > > > > flush explicitly with dma_cache_sync(). > > > > > > I don't see how that would help -- aren't those also controlled by > > > CONFIG_NOT_COHERENT_CACHE? > > > > Ah, yes you are right. PowerPC implements dma_alloc_noncoherent as > > dma_alloc_coherent, so dma_cache_sync() is actually a NOP (or should be). > > But we still need to sync the result of dma_map_* when used multiple > times for a single mapping. We have dma_sync_{single|sg}_for_{cpu|device} API for the above purpose. dma_cache_sync is supposed to be used only with the buffers that dma_alloc_noncoherent() returns. On architecutures that maps dma_alloc_noncoherent to dma_alloc_coherent, dma_cache_sync() is supposed to be NOP. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-17 0:37 ` FUJITA Tomonori 0 siblings, 0 replies; 36+ messages in thread From: FUJITA Tomonori @ 2009-06-17 0:37 UTC (permalink / raw) To: benh; +Cc: scottwood, chris.pringle, linuxppc-dev, linux-kernel, arnd On Wed, 17 Jun 2009 10:18:45 +1000 Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote: > On Tue, 2009-06-16 at 20:02 +0200, Arnd Bergmann wrote: > > On Tuesday 16 June 2009, Scott Wood wrote: > > > > If the > > > > device is the only one, you can also use dma_alloc_noncoherent() and > > > > flush explicitly with dma_cache_sync(). > > > > > > I don't see how that would help -- aren't those also controlled by > > > CONFIG_NOT_COHERENT_CACHE? > > > > Ah, yes you are right. PowerPC implements dma_alloc_noncoherent as > > dma_alloc_coherent, so dma_cache_sync() is actually a NOP (or should be). > > But we still need to sync the result of dma_map_* when used multiple > times for a single mapping. We have dma_sync_{single|sg}_for_{cpu|device} API for the above purpose. dma_cache_sync is supposed to be used only with the buffers that dma_alloc_noncoherent() returns. On architecutures that maps dma_alloc_noncoherent to dma_alloc_coherent, dma_cache_sync() is supposed to be NOP. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-17 0:37 ` FUJITA Tomonori @ 2009-06-17 0:56 ` Leon Woestenberg -1 siblings, 0 replies; 36+ messages in thread From: Leon Woestenberg @ 2009-06-17 0:56 UTC (permalink / raw) To: FUJITA Tomonori Cc: benh, arnd, scottwood, chris.pringle, linuxppc-dev, linux-kernel Hello all, On Wed, Jun 17, 2009 at 2:37 AM, FUJITA Tomonori<fujita.tomonori@lab.ntt.co.jp> wrote: > On Wed, 17 Jun 2009 10:18:45 +1000 > Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote: > >> On Tue, 2009-06-16 at 20:02 +0200, Arnd Bergmann wrote: >> > On Tuesday 16 June 2009, Scott Wood wrote: >> > > > If the >> > > > device is the only one, you can also use dma_alloc_noncoherent() and >> > > > flush explicitly with dma_cache_sync(). >> > > >> > > I don't see how that would help -- aren't those also controlled by >> > > CONFIG_NOT_COHERENT_CACHE? >> > >> > Ah, yes you are right. PowerPC implements dma_alloc_noncoherent as >> > dma_alloc_coherent, so dma_cache_sync() is actually a NOP (or should be). >> >> But we still need to sync the result of dma_map_* when used multiple >> times for a single mapping. > > We have dma_sync_{single|sg}_for_{cpu|device} API for the above > purpose. > > dma_cache_sync is supposed to be used only with the buffers that > dma_alloc_noncoherent() returns. On architecutures that maps > dma_alloc_noncoherent to dma_alloc_coherent, dma_cache_sync() is > supposed to be NOP. This discussion raised some doubt with me about my use case: I my case (note I am not the poster) I am using (what LDD3 calls) streaming mappings: I use pci_map_sg(), have the device perform either DMA master reads or writes to the bus address using PCIe. After that, I use pci_unmap_sg(). My assumption is that pci_unmap_sg() either makes the cache coherent or invalidated and thus I do not need to take further actions. This is on a MPC83xx or 85xx system. Is this assumption correct? Regards, Leon/ -- Leon ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-17 0:56 ` Leon Woestenberg 0 siblings, 0 replies; 36+ messages in thread From: Leon Woestenberg @ 2009-06-17 0:56 UTC (permalink / raw) To: FUJITA Tomonori Cc: arnd, linux-kernel, scottwood, chris.pringle, linuxppc-dev Hello all, On Wed, Jun 17, 2009 at 2:37 AM, FUJITA Tomonori<fujita.tomonori@lab.ntt.co.jp> wrote: > On Wed, 17 Jun 2009 10:18:45 +1000 > Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote: > >> On Tue, 2009-06-16 at 20:02 +0200, Arnd Bergmann wrote: >> > On Tuesday 16 June 2009, Scott Wood wrote: >> > > > If the >> > > > device is the only one, you can also use dma_alloc_noncoherent() and >> > > > flush explicitly with dma_cache_sync(). >> > > >> > > I don't see how that would help -- aren't those also controlled by >> > > CONFIG_NOT_COHERENT_CACHE? >> > >> > Ah, yes you are right. PowerPC implements dma_alloc_noncoherent as >> > dma_alloc_coherent, so dma_cache_sync() is actually a NOP (or should be). >> >> But we still need to sync the result of dma_map_* when used multiple >> times for a single mapping. > > We have dma_sync_{single|sg}_for_{cpu|device} API for the above > purpose. > > dma_cache_sync is supposed to be used only with the buffers that > dma_alloc_noncoherent() returns. On architecutures that maps > dma_alloc_noncoherent to dma_alloc_coherent, dma_cache_sync() is > supposed to be NOP. This discussion raised some doubt with me about my use case: I my case (note I am not the poster) I am using (what LDD3 calls) streaming mappings: I use pci_map_sg(), have the device perform either DMA master reads or writes to the bus address using PCIe. After that, I use pci_unmap_sg(). My assumption is that pci_unmap_sg() either makes the cache coherent or invalidated and thus I do not need to take further actions. This is on a MPC83xx or 85xx system. Is this assumption correct? Regards, Leon/ -- Leon ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-17 0:56 ` Leon Woestenberg @ 2009-06-17 1:08 ` Benjamin Herrenschmidt -1 siblings, 0 replies; 36+ messages in thread From: Benjamin Herrenschmidt @ 2009-06-17 1:08 UTC (permalink / raw) To: Leon Woestenberg Cc: FUJITA Tomonori, arnd, scottwood, chris.pringle, linuxppc-dev, linux-kernel On Wed, 2009-06-17 at 02:56 +0200, Leon Woestenberg wrote: > I use pci_map_sg(), have the device perform either DMA master reads or > writes to the bus address using PCIe. > After that, I use pci_unmap_sg(). > > My assumption is that pci_unmap_sg() either makes the cache coherent > or invalidated and thus I do not need to take further actions. > This is on a MPC83xx or 85xx system. > > Is this assumption correct? It is, as far as I understand things :-) Those APIs are fun. You should only need the explicit sync calls if you are going to peek or poke at the DMA mapped memory before you unmap it. Cheers, Ben. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-17 1:08 ` Benjamin Herrenschmidt 0 siblings, 0 replies; 36+ messages in thread From: Benjamin Herrenschmidt @ 2009-06-17 1:08 UTC (permalink / raw) To: Leon Woestenberg Cc: arnd, linux-kernel, FUJITA Tomonori, scottwood, chris.pringle, linuxppc-dev On Wed, 2009-06-17 at 02:56 +0200, Leon Woestenberg wrote: > I use pci_map_sg(), have the device perform either DMA master reads or > writes to the bus address using PCIe. > After that, I use pci_unmap_sg(). > > My assumption is that pci_unmap_sg() either makes the cache coherent > or invalidated and thus I do not need to take further actions. > This is on a MPC83xx or 85xx system. > > Is this assumption correct? It is, as far as I understand things :-) Those APIs are fun. You should only need the explicit sync calls if you are going to peek or poke at the DMA mapped memory before you unmap it. Cheers, Ben. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-17 1:08 ` Benjamin Herrenschmidt @ 2009-06-17 1:13 ` Leon Woestenberg -1 siblings, 0 replies; 36+ messages in thread From: Leon Woestenberg @ 2009-06-17 1:13 UTC (permalink / raw) To: Benjamin Herrenschmidt Cc: FUJITA Tomonori, arnd, scottwood, chris.pringle, linuxppc-dev, linux-kernel Hello Benjamin, On Wed, Jun 17, 2009 at 3:08 AM, Benjamin Herrenschmidt<benh@kernel.crashing.org> wrote: > On Wed, 2009-06-17 at 02:56 +0200, Leon Woestenberg wrote: >> I use pci_map_sg(), have the device perform either DMA master reads or >> writes to the bus address using PCIe. >> After that, I use pci_unmap_sg(). >> >> My assumption is that pci_unmap_sg() either makes the cache coherent >> or invalidated and thus I do not need to take further actions. >> This is on a MPC83xx or 85xx system. >> >> Is this assumption correct? > > It is, as far as I understand things :-) Those APIs are fun. > > You should only need the explicit sync calls if you are going to peek or > poke at the DMA mapped memory before you unmap it. > Thanks, then at least I understand things in the corr^H^H^H^Hsame way :-) And btw, I hadn't seen any data mismatch yet. Regards, -- Leon 'likewise' Woestenberg ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-17 1:13 ` Leon Woestenberg 0 siblings, 0 replies; 36+ messages in thread From: Leon Woestenberg @ 2009-06-17 1:13 UTC (permalink / raw) To: Benjamin Herrenschmidt Cc: arnd, linux-kernel, FUJITA Tomonori, scottwood, chris.pringle, linuxppc-dev Hello Benjamin, On Wed, Jun 17, 2009 at 3:08 AM, Benjamin Herrenschmidt<benh@kernel.crashing.org> wrote: > On Wed, 2009-06-17 at 02:56 +0200, Leon Woestenberg wrote: >> I use pci_map_sg(), have the device perform either DMA master reads or >> writes to the bus address using PCIe. >> After that, I use pci_unmap_sg(). >> >> My assumption is that pci_unmap_sg() either makes the cache coherent >> or invalidated and thus I do not need to take further actions. >> This is on a MPC83xx or 85xx system. >> >> Is this assumption correct? > > It is, as far as I understand things :-) Those APIs are fun. > > You should only need the explicit sync calls if you are going to peek or > poke at the DMA mapped memory before you unmap it. > Thanks, then at least I understand things in the corr^H^H^H^Hsame way :-) And btw, I hadn't seen any data mismatch yet. Regards, -- Leon 'likewise' Woestenberg ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-17 0:37 ` FUJITA Tomonori @ 2009-06-17 1:07 ` Benjamin Herrenschmidt -1 siblings, 0 replies; 36+ messages in thread From: Benjamin Herrenschmidt @ 2009-06-17 1:07 UTC (permalink / raw) To: FUJITA Tomonori Cc: arnd, scottwood, chris.pringle, linuxppc-dev, linux-kernel On Wed, 2009-06-17 at 09:37 +0900, FUJITA Tomonori wrote: > > dma_cache_sync is supposed to be used only with the buffers that > dma_alloc_noncoherent() returns. On architecutures that maps > dma_alloc_noncoherent to dma_alloc_coherent, dma_cache_sync() is > supposed to be NOP. > Or at least a sync() on powerpc but yeah, I see. We should probably do that. Cheers, Ben. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-17 1:07 ` Benjamin Herrenschmidt 0 siblings, 0 replies; 36+ messages in thread From: Benjamin Herrenschmidt @ 2009-06-17 1:07 UTC (permalink / raw) To: FUJITA Tomonori Cc: scottwood, chris.pringle, linuxppc-dev, linux-kernel, arnd On Wed, 2009-06-17 at 09:37 +0900, FUJITA Tomonori wrote: > > dma_cache_sync is supposed to be used only with the buffers that > dma_alloc_noncoherent() returns. On architecutures that maps > dma_alloc_noncoherent to dma_alloc_coherent, dma_cache_sync() is > supposed to be NOP. > Or at least a sync() on powerpc but yeah, I see. We should probably do that. Cheers, Ben. ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 17:03 ` Scott Wood @ 2009-06-17 7:58 ` Chris Pringle -1 siblings, 0 replies; 36+ messages in thread From: Chris Pringle @ 2009-06-17 7:58 UTC (permalink / raw) To: Scott Wood; +Cc: linux-kernel, linuxppc-dev@ozlabs.org list > You could enable CONFIG_NOT_COHERENT_CACHE. > I've just tried this (I had to edit Kconfig in power/platforms to make the build system accept it), and interestingly it's making no difference. I'm using streaming mappings, and are using the pci_map_sg functions to ensure the memory is mapped/flushed correctly. I've also explicitly put in a pci_dma_sync_sg_for_device, however that's also not made any difference. Turning the cpu cache snoop off has the same affect as it did without CONFIG_NOT_COHERENT_CACHE; it gets much worse. Any other ideas? Will back off the low latency patches next, and give 2.6.30 a try - see if that makes any difference. -- ______________________________ Chris Pringle Software Engineer Miranda Technologies Ltd. Hithercroft Road Wallingford Oxfordshire OX10 9DG UK Tel. +44 1491 820206 Fax. +44 1491 820001 www.miranda.com ____________________________ Miranda Technologies Limited Registered in England and Wales CN 02017053 Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-17 7:58 ` Chris Pringle 0 siblings, 0 replies; 36+ messages in thread From: Chris Pringle @ 2009-06-17 7:58 UTC (permalink / raw) To: Scott Wood; +Cc: linuxppc-dev@ozlabs.org list, linux-kernel > You could enable CONFIG_NOT_COHERENT_CACHE. > I've just tried this (I had to edit Kconfig in power/platforms to make the build system accept it), and interestingly it's making no difference. I'm using streaming mappings, and are using the pci_map_sg functions to ensure the memory is mapped/flushed correctly. I've also explicitly put in a pci_dma_sync_sg_for_device, however that's also not made any difference. Turning the cpu cache snoop off has the same affect as it did without CONFIG_NOT_COHERENT_CACHE; it gets much worse. Any other ideas? Will back off the low latency patches next, and give 2.6.30 a try - see if that makes any difference. -- ______________________________ Chris Pringle Software Engineer Miranda Technologies Ltd. Hithercroft Road Wallingford Oxfordshire OX10 9DG UK Tel. +44 1491 820206 Fax. +44 1491 820001 www.miranda.com ____________________________ Miranda Technologies Limited Registered in England and Wales CN 02017053 Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-17 7:58 ` Chris Pringle (?) @ 2009-06-17 13:18 ` Chris Pringle 2009-06-18 11:24 ` Chris Pringle -1 siblings, 1 reply; 36+ messages in thread From: Chris Pringle @ 2009-06-17 13:18 UTC (permalink / raw) To: Scott Wood; +Cc: linuxppc-dev@ozlabs.org list, linux-kernel Chris Pringle wrote: >> You could enable CONFIG_NOT_COHERENT_CACHE. >> > I've just tried this (I had to edit Kconfig in power/platforms to make > the build system accept it), and interestingly it's making no > difference. I'm using streaming mappings, and are using the pci_map_sg > functions to ensure the memory is mapped/flushed correctly. I've also > explicitly put in a pci_dma_sync_sg_for_device, however that's also > not made any difference. Turning the cpu cache snoop off has the same > affect as it did without CONFIG_NOT_COHERENT_CACHE; it gets much > worse. Any other ideas? > > Will back off the low latency patches next, and give 2.6.30 a try - > see if that makes any difference. > Low latency patches made no difference. Tried it with 2.6.30 and it now works. There are a couple of commits contributing to the fix, including one introduced between 2.6.29-rc8 and 2.6.29 proper in powerpc/kernel/head_32.S (couple of commits with the name "Fix Respect _PAGE_COHERENT on classic ppc32 SW TLB load machines"). I've tried backporting this to 2.6.29-rc8 and it then worked. Backporting to 2.6.26 made no difference however, so I suspect there are other things fixed which are also contributing. I'm going to move to 2.6.29/2.6.30 which should resolve our issue. Thanks to all who have contributed to this thread. Chris -- ______________________________ Chris Pringle Software Engineer Miranda Technologies Ltd. Hithercroft Road Wallingford Oxfordshire OX10 9DG UK Tel. +44 1491 820206 Fax. +44 1491 820001 www.miranda.com ____________________________ Miranda Technologies Limited Registered in England and Wales CN 02017053 Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-17 13:18 ` Chris Pringle @ 2009-06-18 11:24 ` Chris Pringle 2009-06-22 14:31 ` Sergej.Stepanov 0 siblings, 1 reply; 36+ messages in thread From: Chris Pringle @ 2009-06-18 11:24 UTC (permalink / raw) To: Scott Wood; +Cc: linuxppc-dev@ozlabs.org list, linux-kernel Chris Pringle wrote: > Chris Pringle wrote: >>> You could enable CONFIG_NOT_COHERENT_CACHE. >>> >> I've just tried this (I had to edit Kconfig in power/platforms to >> make the build system accept it), and interestingly it's making no >> difference. I'm using streaming mappings, and are using the >> pci_map_sg functions to ensure the memory is mapped/flushed >> correctly. I've also explicitly put in a pci_dma_sync_sg_for_device, >> however that's also not made any difference. Turning the cpu cache >> snoop off has the same affect as it did without >> CONFIG_NOT_COHERENT_CACHE; it gets much worse. Any other ideas? >> >> Will back off the low latency patches next, and give 2.6.30 a try - >> see if that makes any difference. >> > Low latency patches made no difference. Tried it with 2.6.30 and it > now works. There are a couple of commits contributing to the fix, > including one introduced between 2.6.29-rc8 and 2.6.29 proper in > powerpc/kernel/head_32.S (couple of commits with the name "Fix Respect > _PAGE_COHERENT on classic ppc32 SW TLB load machines"). I've tried > backporting this to 2.6.29-rc8 and it then worked. Backporting to > 2.6.26 made no difference however, so I suspect there are other things > fixed which are also contributing. > > I'm going to move to 2.6.29/2.6.30 which should resolve our issue. > > Thanks to all who have contributed to this thread. > The other part of the fix is in asm-powerpc/pgtable32.h. _PAGE_BASE needs _PAGE_COHERENT in order to work correctly, and in fact there is now a comment in there to that affect in 2.6.29. Backporting that change has made it work on 2.6.26. Both this patch, and the fix to head_32.S are needed for it to work correctly on older kernels. Chris -- ______________________________ Chris Pringle Software Engineer Miranda Technologies Ltd. Hithercroft Road Wallingford Oxfordshire OX10 9DG UK Tel. +44 1491 820206 Fax. +44 1491 820001 www.miranda.com ____________________________ Miranda Technologies Limited Registered in England and Wales CN 02017053 Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ ^ permalink raw reply [flat|nested] 36+ messages in thread
* AW: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-18 11:24 ` Chris Pringle @ 2009-06-22 14:31 ` Sergej.Stepanov 0 siblings, 0 replies; 36+ messages in thread From: Sergej.Stepanov @ 2009-06-22 14:31 UTC (permalink / raw) To: chris.pringle, scottwood; +Cc: linuxppc-dev, linux-kernel >The other part of the fix is in asm-powerpc/pgtable32.h. _PAGE_BASE >needs _PAGE_COHERENT in order to work correctly, and in fact there is >now a comment in there to that affect in 2.6.29. Backporting that change >has made it work on 2.6.26. Both this patch, and the fix to head_32.S >are needed for it to work correctly on older kernels. > >Chris Hello Chris, sorry for dummy, but if it possible, could you, please, send a corresponding summary patch of backporting you've done for older kernels? or just summary of that changes once again? Many thanks Sergej. ^ permalink raw reply [flat|nested] 36+ messages in thread
* AW: PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-22 14:31 ` Sergej.Stepanov 0 siblings, 0 replies; 36+ messages in thread From: Sergej.Stepanov @ 2009-06-22 14:31 UTC (permalink / raw) To: chris.pringle, scottwood; +Cc: linuxppc-dev, linux-kernel >The other part of the fix is in asm-powerpc/pgtable32.h. _PAGE_BASE >needs _PAGE_COHERENT in order to work correctly, and in fact there is >now a comment in there to that affect in 2.6.29. Backporting that change >has made it work on 2.6.26. Both this patch, and the fix to head_32.S >are needed for it to work correctly on older kernels. > >Chris Hello Chris, sorry for dummy, but if it possible, could you, please, send a correspondin= g summary patch of backporting you've done for older kernels? or just summary of that changes once again? Many thanks Sergej.= ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: AW: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-22 14:31 ` Sergej.Stepanov (?) @ 2009-06-29 8:11 ` Chris Pringle -1 siblings, 0 replies; 36+ messages in thread From: Chris Pringle @ 2009-06-29 8:11 UTC (permalink / raw) To: Sergej.Stepanov; +Cc: scottwood, linuxppc-dev, linux-kernel [-- Attachment #1: Type: text/plain, Size: 1463 bytes --] Hi Sergej, I've attached the patch used to fix this issue. Both the patch to pgtable32.h and head_32.S are required in order to make it work. The change to pgtable32.h ensures that all pages are marked cache coherent (results in setting the M bit). The change to head_32.S ensures that the M bit is not unconditionally masked out - it should only be masked out if CPU_FTR_NEED_COHERENT is not set. Hope this helps. Cheers, Chris Sergej.Stepanov@ids.de wrote: >> The other part of the fix is in asm-powerpc/pgtable32.h. _PAGE_BASE >> needs _PAGE_COHERENT in order to work correctly, and in fact there is >> now a comment in there to that affect in 2.6.29. Backporting that change >> has made it work on 2.6.26. Both this patch, and the fix to head_32.S >> are needed for it to work correctly on older kernels. >> >> Chris >> > > Hello Chris, > > sorry for dummy, but if it possible, could you, please, send a corresponding summary patch of backporting you've done for older kernels? > or just summary of that changes once again? > > Many thanks > > Sergej. -- ______________________________ Chris Pringle Software Engineer Miranda Technologies Ltd. Hithercroft Road Wallingford Oxfordshire OX10 9DG UK Tel. +44 1491 820206 Fax. +44 1491 820001 www.miranda.com ____________________________ Miranda Technologies Limited Registered in England and Wales CN 02017053 Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ [-- Attachment #2: dma-cache-coherency-fix.patch --] [-- Type: text/x-patch, Size: 2590 bytes --] diff -r -U3 ./arch/powerpc/kernel/head_32.S ../../kernel.WORKS/linux-2.6.26/arch/powerpc/kernel/head_32.S --- ./arch/powerpc/kernel/head_32.S 2008-07-13 22:51:29.000000000 +0100 +++ ../../kernel.WORKS/linux-2.6.26/arch/powerpc/kernel/head_32.S 2009-06-17 18:18:04.000000000 +0100 @@ -501,8 +501,11 @@ and r1,r1,r2 /* writable if _RW and _DIRTY */ rlwimi r3,r3,32-1,30,30 /* _PAGE_USER -> PP msb */ rlwimi r3,r3,32-1,31,31 /* _PAGE_USER -> PP lsb */ - ori r1,r1,0xe14 /* clear out reserved bits and M */ + ori r1,r1,0xe04 /* clear out reserved bits */ andc r1,r3,r1 /* PP = user? (rw&dirty? 2: 3): 0 */ +BEGIN_FTR_SECTION + rlwinm r1,r1,0,~_PAGE_COHERENT /* clear M (coherence not required) */ +END_FTR_SECTION_IFCLR(CPU_FTR_NEED_COHERENT) mtspr SPRN_RPA,r1 mfspr r3,SPRN_IMISS tlbli r3 @@ -575,8 +578,12 @@ and r1,r1,r2 /* writable if _RW and _DIRTY */ rlwimi r3,r3,32-1,30,30 /* _PAGE_USER -> PP msb */ rlwimi r3,r3,32-1,31,31 /* _PAGE_USER -> PP lsb */ - ori r1,r1,0xe14 /* clear out reserved bits and M */ + ori r1,r1,0xe04 /* clear out reserved bits */ andc r1,r3,r1 /* PP = user? (rw&dirty? 2: 3): 0 */ +BEGIN_FTR_SECTION + rlwinm r1,r1,0,~_PAGE_COHERENT /* clear M (coherence not required) */ +END_FTR_SECTION_IFCLR(CPU_FTR_NEED_COHERENT) + /*ori r1,r1,0x10*/ mtspr SPRN_RPA,r1 mfspr r3,SPRN_DMISS tlbld r3 @@ -643,8 +650,12 @@ stw r3,0(r2) /* update PTE (accessed/dirty bits) */ /* Convert linux-style PTE to low word of PPC-style PTE */ rlwimi r3,r3,32-1,30,30 /* _PAGE_USER -> PP msb */ - li r1,0xe15 /* clear out reserved bits and M */ + li r1,0xe05 /* clear out reserved bits & PP lsb */ andc r1,r3,r1 /* PP = user? 2: 0 */ +BEGIN_FTR_SECTION + rlwinm r1,r1,0,~_PAGE_COHERENT /* clear M (coherence not required) */ +END_FTR_SECTION_IFCLR(CPU_FTR_NEED_COHERENT) + /*ori r1,r1,0x10*/ mtspr SPRN_RPA,r1 mfspr r3,SPRN_DMISS tlbld r3 diff -r -U3 ./include/asm-powerpc/pgtable-ppc32.h ../../kernel.WORKS/linux-2.6.26/include/asm-powerpc/pgtable-ppc32.h --- ./include/asm-powerpc/pgtable-ppc32.h 2008-07-13 22:51:29.000000000 +0100 +++ ../../kernel.WORKS/linux-2.6.26/include/asm-powerpc/pgtable-ppc32.h 2009-06-18 12:11:57.000000000 +0100 @@ -408,7 +408,7 @@ #ifdef CONFIG_44x #define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED) #else -#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED) +#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_COHERENT) #endif #define _PAGE_WRENABLE (_PAGE_RW | _PAGE_DIRTY | _PAGE_HWWRITE) #define _PAGE_KERNEL (_PAGE_BASE | _PAGE_SHARED | _PAGE_WRENABLE) ^ permalink raw reply [flat|nested] 36+ messages in thread
* PowerPC PCI DMA issues (prefetch/coherency?) @ 2009-06-16 14:08 Chris Pringle 2009-06-16 15:48 ` Roderick Colenbrander 0 siblings, 1 reply; 36+ messages in thread From: Chris Pringle @ 2009-06-16 14:08 UTC (permalink / raw) To: linuxppc-dev Hello All, We're developing on a Freescale MPC8272 and are having some nasty problems with PCI bus mastering and data corruption. We have some custom hardware that is bus mastering, reading data from the CPUs memory for it's own use. Most of the time, the data is correct, however occasionally we are seeing data that appears to be from somewhere else in memory (usually memory that has already been read by the PCI device). The problem looks like stale data on the PCI bridge prefetch buffers or a cache coherency problem, but we've been unable to come up with a solution to our problem. It is my understanding that it shouldn't be a cache coherency problem as the CPU cache should be snooped as the data is read from memory. Even if it were an issue, the pci_map_sg* functions should have sorted out any cache coherency issues before the DMA operation started. I've not been able to find anything on the Freescale data sheet that provides any way of flushing the prefetch cache on the PCI bridge. We've done a bit of experimenting, and found that turning off prefetch appears to solve (or possibly mask?) the problem (at the expensive of massive performance problems). I've also tried DMA'ing two adjacent userspace buffers in memory (from the same page), and see corruption on the second buffer. If I populate both buffers, then DMA them both, the data is fine. If I populate the first, DMA the first, then populate the second and DMA the second, corruption occurs at the start of the second buffer. If I add 8-32 bytes of padding between the buffers, the problem goes away. The PCI spec says that the PCI bridge is supposed to flush any data from it's prefetch buffers that are not read by the bus master, so technically, this isn't supposed to happen. I've tried making sure that buffers are cache line (and page) aligned, and are multiples of cache lines, but it's made no difference. PIO mode works fine, and I've checked the data with the CPU just before, and immediately after the DMA and the driver sees no data integrity issues. There are memory write barriers just before the DMA start, so all the registers should be correct before the DMA starts. For background info, the device doing the bus mastering is a Xilinx Virtex 5 FPGA. We've monitored the data as it comes off the PCI bus using ChipScope - so the firmware should not be manipulating the data in any way. We have some hardware/firmware/drivers that has a lot of common code that runs on an x86 platform (as opposed to powerpc), and that works without any issues whatsoever. Has anyone got any ideas what this might be? Does anyone of know issues with PCI bridges on the PowerPC platform? Is there extra things that need to be done from the driver when DMAing on PowerPC (I've looked at other drivers and there's nothing obvious). The chip errata doesn't have anything on it that looks like it could cause this. I'm really hoping this is something that we're doing wrong in the driver or the firmware, but we've been through both the firmware and drivers countless times and are unable to see anything wrong. Any thoughts/ideas would be much appreciated! Regards, Chris -- ______________________________ Chris Pringle Software Engineer Miranda Technologies Ltd. Hithercroft Road Wallingford Oxfordshire OX10 9DG UK Tel. +44 1491 820206 Fax. +44 1491 820001 www.miranda.com ____________________________ Miranda Technologies Limited Registered in England and Wales CN 02017053 Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ ^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: PowerPC PCI DMA issues (prefetch/coherency?) 2009-06-16 14:08 Chris Pringle @ 2009-06-16 15:48 ` Roderick Colenbrander 0 siblings, 0 replies; 36+ messages in thread From: Roderick Colenbrander @ 2009-06-16 15:48 UTC (permalink / raw) To: Chris Pringle; +Cc: linuxppc-dev Hi Chris, The last couple of months I had the 'pleasure' to work with a Xilinx ML510 ATX board which contains a Virtex-5 FXT. In my case I had to set up the plbv46 pci soft-core to function properly with PCI add-on boards and onboard pci devices. I have had a lot of issues including ones with DMA but in the end it worked properly. I'm not sure how in your case the Virtex-5 is wired to the PCI bus but in my case it was connected to the crossbar switch of the PowerPC 440 which is inside the Virtex-5 FXT. Not all Xilinx reference designs were interfaced properly to this (sometimes the MPLB->SPLB connection from the soft-core to the crossbar wasn't connected) and even then the PCI soft-core wasn't configured properly by Xilinx and this MIGHT be similar to your issue. In the explanation of the issue I will use the terminology used by Xilinx. The CPU side is the 'IPIF' side and the and the PCI side is the 'PCI' side. Both domains have their own address domains and hence translation from IPIFtoPCI and PCItoIPIF is needed. From what I understood is that the 'incoming window' (ipif->pci) and outgoing window (pci>ipif) should use different address ranges else the bridge gets confused. I don't have access to my design right now but by head Xilinx doesn't set C_IPIFBAR2PCIBAR_0 properly in a lot of cases and just sets it to 0. Which means that cpu -> ipif (lets say the soft core is accessible from 0xa0000000 from the cpu) 0xa0001234 translates to 0x00001234 on the PCI side but PCI cards expect that when they write low memory addresses that they write to system memory. What should be done is to set C_IPIFBAR2PCIBAR_0 to 0xa0000000 in this case, so that the incoming window is 0xa0000000-0xffffffff and the outgoing window 0x00000000-0x9fffffff or just set it half way at 0x80000000. I'm not saying this is your issue but it could be it. Regards, Roderick Colenbrander On Tue, Jun 16, 2009 at 4:08 PM, Chris Pringle<chris.pringle@oxtel.com> wrote: > Hello All > > We're developing on a Freescale MPC8272 and are having some nasty problems > with PCI bus mastering and data corruption. > > We have some custom hardware that is bus mastering, reading data from the > CPUs memory for it's own use. Most of the time, the data is correct, however > occasionally we are seeing data that appears to be from somewhere else in > memory (usually memory that has already been read by the PCI device). The > problem looks like stale data on the PCI bridge prefetch buffers or a cache > coherency problem, but we've been unable to come up with a solution to our > problem. It is my understanding that it shouldn't be a cache coherency > problem as the CPU cache should be snooped as the data is read from memory. > Even if it were an issue, the pci_map_sg* functions should have sorted out > any cache coherency issues before the DMA operation started. > > I've not been able to find anything on the Freescale data sheet that > provides any way of flushing the prefetch cache on the PCI bridge. We've > done a bit of experimenting, and found that turning off prefetch appears to > solve (or possibly mask?) the problem (at the expensive of massive > performance problems). I've also tried DMA'ing two adjacent userspace > buffers in memory (from the same page), and see corruption on the second > buffer. If I populate both buffers, then DMA them both, the data is fine. If > I populate the first, DMA the first, then populate the second and DMA the > second, corruption occurs at the start of the second buffer. If I add 8-32 > bytes of padding between the buffers, the problem goes away. > > The PCI spec says that the PCI bridge is supposed to flush any data from > it's prefetch buffers that are not read by the bus master, so technically, > this isn't supposed to happen. > > I've tried making sure that buffers are cache line (and page) aligned, and > are multiples of cache lines, but it's made no difference. PIO mode works > fine, and I've checked the data with the CPU just before, and immediately > after the DMA and the driver sees no data integrity issues. There are memory > write barriers just before the DMA start, so all the registers should be > correct before the DMA starts. > > For background info, the device doing the bus mastering is a Xilinx Virtex 5 > FPGA. We've monitored the data as it comes off the PCI bus using ChipScope - > so the firmware should not be manipulating the data in any way. > > We have some hardware/firmware/drivers that has a lot of common code that > runs on an x86 platform (as opposed to powerpc), and that works without any > issues whatsoever. > > Has anyone got any ideas what this might be? Does anyone of know issues with > PCI bridges on the PowerPC platform? Is there extra things that need to be > done from the driver when DMAing on PowerPC (I've looked at other drivers > and there's nothing obvious). The chip errata doesn't have anything on it > that looks like it could cause this. > > I'm really hoping this is something that we're doing wrong in the driver or > the firmware, but we've been through both the firmware and drivers countless > times and are unable to see anything wrong. > > Any thoughts/ideas would be much appreciated! > > Regards, > Chris > > -- > > ______________________________ > Chris Pringle > Software Engineer > > Miranda Technologies Ltd. > Hithercroft Road > Wallingford > Oxfordshire OX10 9DG > UK > > Tel. +44 1491 820206 > Fax. +44 1491 820001 > www.miranda.com > > ____________________________ > > Miranda Technologies Limited > Registered in England and Wales CN 02017053 > Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 > 1FJ > _______________________________________________ > Linuxppc-dev mailing list > Linuxppc-dev@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/linuxppc-dev > ^ permalink raw reply [flat|nested] 36+ messages in thread
end of thread, other threads:[~2009-06-29 8:11 UTC | newest] Thread overview: 36+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2009-06-16 13:58 PowerPC PCI DMA issues (prefetch/coherency?) Chris Pringle 2009-06-16 14:16 ` Michael S. Zick 2009-06-16 14:59 ` Hu Gang 2009-06-16 16:21 ` Scott Wood 2009-06-16 16:34 ` Chris Pringle 2009-06-16 16:46 ` Scott Wood 2009-06-16 16:46 ` Scott Wood 2009-06-16 16:57 ` Chris Pringle 2009-06-16 16:57 ` Chris Pringle 2009-06-16 17:03 ` Scott Wood 2009-06-16 17:03 ` Scott Wood 2009-06-16 17:43 ` Arnd Bergmann 2009-06-16 17:49 ` Scott Wood 2009-06-16 17:49 ` Scott Wood 2009-06-16 18:02 ` Arnd Bergmann 2009-06-16 18:02 ` Arnd Bergmann 2009-06-17 0:18 ` Benjamin Herrenschmidt 2009-06-17 0:37 ` FUJITA Tomonori 2009-06-17 0:37 ` FUJITA Tomonori 2009-06-17 0:56 ` Leon Woestenberg 2009-06-17 0:56 ` Leon Woestenberg 2009-06-17 1:08 ` Benjamin Herrenschmidt 2009-06-17 1:08 ` Benjamin Herrenschmidt 2009-06-17 1:13 ` Leon Woestenberg 2009-06-17 1:13 ` Leon Woestenberg 2009-06-17 1:07 ` Benjamin Herrenschmidt 2009-06-17 1:07 ` Benjamin Herrenschmidt 2009-06-17 7:58 ` Chris Pringle 2009-06-17 7:58 ` Chris Pringle 2009-06-17 13:18 ` Chris Pringle 2009-06-18 11:24 ` Chris Pringle 2009-06-22 14:31 ` AW: " Sergej.Stepanov 2009-06-22 14:31 ` Sergej.Stepanov 2009-06-29 8:11 ` Chris Pringle 2009-06-16 14:08 Chris Pringle 2009-06-16 15:48 ` Roderick Colenbrander
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.