[RFC] consistent_sync and non L1 cache line aligned buffers

[RFC] consistent_sync and non L1 cache line aligned buffers

[Eugene Surovegin]

Hi!

I think this is a known problem.

There are drivers or even subsystems which use stack allocated DMA buffers.
To make things worse, those buffers usually non L1 cache line aligned
(start and/or end).

When they use pci_map_* with PCI_DMA_FROMDEVICE, consistent_sync calls
invalidate_dcache_range for the buffer.

invalidate_dcache_range works in L1_CACHE_LINE chunks, so if start and/or
end of the buffer are not aligned we may corrupt data located in the same
cache line (usually stack variable(s) declared before or after buffer
declaration).

According to MV kernel, there are USB devices that use such buffers.

After spending last weekend with RISCWatch :) I can say that SCSI subsystem
is also guilty of this behavior (drivers/scsi/scsi_scan.c::scan_scsis,
scsi_result0).

Unfortunately, I don't know how many similar places of code are still
waiting to be found :(.
To be safe I think it's better to modify consistent_sync to handle such
"bad" buffers.

If start and/or end of the buffer are not properly aligned I use "dcbf" to
flush corresponding cache line(s) and then call invalidate_dcache_range.

This change doesn't affect performance of consistent_sync noticeably (like
in the variant I found in MV kernel, where invalidate_dcache_range was
changed to flush_dcache_range if USB was enabled)

I don't know whether we should "ifdef" this for CONFIG_4xx and I know this
fix is ugly :)
I'm not even sure that such hacks should be included in the kernel :)))
(but I will definitely use it in my tree)

Comments/suggestions are welcome!

Thanks,

Eugene


===== arch/ppc/mm/cachemap.c 1.13 vs edited =====
--- 1.13/arch/ppc/mm/cachemap.c	Thu Feb 27 11:40:16 2003
+++ edited/arch/ppc/mm/cachemap.c	Mon Jul 14 20:49:28 2003
@@ -150,6 +150,21 @@
  	case PCI_DMA_NONE:
  		BUG();
  	case PCI_DMA_FROMDEVICE:	/* invalidate only */
+
+		/* Handle cases when the buffer start and/or end
+		   are not L1 cache line aligned.
+		   Some drivers/subsystems (e.g. USB, SCSI) do DMA
+		   from the stack allocated buffers, to prevent
+		   corruption of the other stack variables located
+		   near the buffer, we flush (instead of invalidate)
+		   these "dangerous" areas                     --ebs
+		*/
+		if (unlikely(start & (L1_CACHE_LINE_SIZE - 1)))
+			__asm__ __volatile__("dcbf 0,%0" : : "r" (start));
+
+		if (unlikely(end & (L1_CACHE_LINE_SIZE - 1)))
+			__asm__ __volatile__("dcbf 0,%0" : : "r" (end));
+
  		invalidate_dcache_range(start, end);
  		break;
  	case PCI_DMA_TODEVICE:		/* writeback only */


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: [RFC] consistent_sync and non L1 cache line aligned buffers

[Tom Rini]

On Mon, Jul 14, 2003 at 09:32:07PM -0700, Eugene Surovegin wrote:

> I think this is a known problem.

Yes, fortunatly.

> According to MV kernel, there are USB devices that use such buffers.
>
> After spending last weekend with RISCWatch :) I can say that SCSI subsystem
> is also guilty of this behavior (drivers/scsi/scsi_scan.c::scan_scsis,
> scsi_result0).

Owch.

> Unfortunately, I don't know how many similar places of code are still
> waiting to be found :(.
> To be safe I think it's better to modify consistent_sync to handle such
> "bad" buffers.
>
> If start and/or end of the buffer are not properly aligned I use "dcbf" to
> flush corresponding cache line(s) and then call invalidate_dcache_range.
>
> This change doesn't affect performance of consistent_sync noticeably (like
> in the variant I found in MV kernel, where invalidate_dcache_range was
> changed to flush_dcache_range if USB was enabled)

Good to know.

> I don't know whether we should "ifdef" this for CONFIG_4xx and I know this
> fix is ugly :)
> I'm not even sure that such hacks should be included in the kernel :)))
> (but I will definitely use it in my tree)
>
> Comments/suggestions are welcome!

Well, one thing that is worth noting is that the USB people knew this
was a problem, and it was / should have been fixed in the 2.5 cycle.
Similarly, SCSI was cleaned up a lot, so perhaps this has been fixed
there.  I think it's generally known that doing DMA off of the stack is
a bad idea, and should be fixed when found.

--
Tom Rini
http://gate.crashing.org/~trini/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: [RFC] consistent_sync and non L1 cache line aligned buffers

[Matt Porter]

On Mon, Jul 14, 2003 at 09:32:07PM -0700, Eugene Surovegin wrote:
>
> Hi!
>
> I think this is a known problem.
>
> There are drivers or even subsystems which use stack allocated DMA buffers.
> To make things worse, those buffers usually non L1 cache line aligned
> (start and/or end).
>
> When they use pci_map_* with PCI_DMA_FROMDEVICE, consistent_sync calls
> invalidate_dcache_range for the buffer.
>
> invalidate_dcache_range works in L1_CACHE_LINE chunks, so if start and/or
> end of the buffer are not aligned we may corrupt data located in the same
> cache line (usually stack variable(s) declared before or after buffer
> declaration).
>
> According to MV kernel, there are USB devices that use such buffers.

Well, the USB subsystem itself in 2.4 has stack DMA buffers completely
intertwined, yes.  Fixing it in the USB stack was nontrivial enough
to force it to their 2.5/2.6 code.

> After spending last weekend with RISCWatch :) I can say that SCSI subsystem
> is also guilty of this behavior (drivers/scsi/scsi_scan.c::scan_scsis,
> scsi_result0).

I went through the SCSI subsystem a while back and found a few including
that one too. :)  These should be passed up since they are clear
violations of the DMA API.  I dropped that task since (as you know)
there are other issues with using SCSI drivers on "non-coherents" at
the moment.

> Unfortunately, I don't know how many similar places of code are still
> waiting to be found :(.

<snip>

> Comments/suggestions are welcome!

I'll agree that it's a better hack, but since the offending areas in
the SCSI subsystem are easily located, it seems wiser to fix upstream.
Just my US 2 cents.

We still need someone with interest AND time to properly fix the
consistent alloc from irq issue. :)  All of the patches post to date
are incomplete bandaids.

Regards,
--
Matt Porter
mporter@kernel.crashing.org

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: [RFC] consistent_sync and non L1 cache line aligned buffers

[Eugene Surovegin]

At 08:46 AM 7/15/2003, Tom Rini wrote:
>Well, one thing that is worth noting is that the USB people knew this
>was a problem, and it was / should have been fixed in the 2.5 cycle.
>Similarly, SCSI was cleaned up a lot, so perhaps this has been fixed
>there.  I think it's generally known that doing DMA off of the stack is
>a bad idea, and should be fixed when found.

I agree this is VERY bad idea but the fact is that there is a code which
does such nasty things.

I truly hope all this will/was fixed in 2.5 but frankly I wouldn't be so
sure :)

Unfortunately, for production 2.5 is unusable and will be for some time.
A lot of people (I think majority) still use 2.4. And 2.4 (as of
2.4.22-pre6) is still broken in this respect...

Eugene


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: [RFC] consistent_sync and non L1 cache line aligned buffers

[Tom Rini]

On Tue, Jul 15, 2003 at 09:20:24AM -0700, Eugene Surovegin wrote:
> At 08:46 AM 7/15/2003, Tom Rini wrote:
> >Well, one thing that is worth noting is that the USB people knew this
> >was a problem, and it was / should have been fixed in the 2.5 cycle.
> >Similarly, SCSI was cleaned up a lot, so perhaps this has been fixed
> >there.  I think it's generally known that doing DMA off of the stack is
> >a bad idea, and should be fixed when found.
>
> I agree this is VERY bad idea but the fact is that there is a code which
> does such nasty things.
>
> I truly hope all this will/was fixed in 2.5 but frankly I wouldn't be so
> sure :)

Well, I would be, of the USB code.  SCSI might have had it fixed, and
others that we haven't found yet may or may not.  But the important
point is that doing this is a driver bug and it's OK to beat driver
authors over the head with patches to fix the behavior. :)

> Unfortunately, for production 2.5 is unusable and will be for some time.
> A lot of people (I think majority) still use 2.4. And 2.4 (as of
> 2.4.22-pre6) is still broken in this respect...

Yes, the changes to USB and SCSI probably won't be backported for some
time, if ever.  So there is still a question of should we workaround
this in 2.4 (Or, more to the point, do we leave it up to every
$(EMBEDDED VENDOR) to do it, or bite the bullet and commit it to the 2.4
mainline.

--
Tom Rini
http://gate.crashing.org/~trini/

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: [RFC] consistent_sync and non L1 cache line aligned buffers

[Eugene Surovegin]

At 09:17 AM 7/15/2003, Matt Porter wrote:
>I'll agree that it's a better hack, but since the offending areas in
>the SCSI subsystem are easily located, it seems wiser to fix upstream.

Matt, the problem is it wasn't that *easy* to locate this, at least for me :)
I'm not sure that this is the only place..

>We still need someone with interest AND time to properly fix the
>consistent alloc from irq issue. :)  All of the patches post to date
>are incomplete bandaids.

Uhh, I switched to solution which uses pre allocated consistent memory (10
pages are enough for sym53c8xx_2).
It's still not a generic solution, but at least it's safe :)

Eugene


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

PPCBoot on Ebony board

[Brian Padalino]

I am trying to transfer PPCBoot that I had compiled over a tftp session in
the 440GP 1.18 ROM Monitor (02/11/02) that came on the Ebony board.  Is that
the wrong method of trying it?  I tried using both serial ports and Kermit
with Xon/Xoff handshaking, but it just ended up timing out.

I keep getting the error:
  Loading file "C:\ppc\bootp\boot.img" ...
  Sending tftp boot request ...
  Not a valid boot image file

Which stems from an improper magic number at the top of the file to say it's
a valid boot image.

I hate to sound like such a newbie, but can anyone help in getting PPCBoot
on this new board?  Any help would be very appreciated.

Thank you,
Brian Padalino


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: PPCBoot on Ebony board

[Chris Zimman]

On Tue, Jul 15, 2003 at 02:11:07PM -0400, Brian Padalino wrote:
>
> I am trying to transfer PPCBoot that I had compiled over a tftp session in
> the 440GP 1.18 ROM Monitor (02/11/02) that came on the Ebony board.  Is that
> the wrong method of trying it?  I tried using both serial ports and Kermit
> with Xon/Xoff handshaking, but it just ended up timing out.
>
> I keep getting the error:
>   Loading file "C:\ppc\bootp\boot.img" ...
>   Sending tftp boot request ...
>   Not a valid boot image file
>
> Which stems from an improper magic number at the top of the file to say it's
> a valid boot image.

This is not going to work.  You want to program PPCBoot into flash via a
flash programmer or via JTAG.

--Chris

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: [RFC] consistent_sync and non L1 cache line aligned buffers

[Matt Porter]

On Tue, Jul 15, 2003 at 09:27:10AM -0700, Eugene Surovegin wrote:
>
> At 09:17 AM 7/15/2003, Matt Porter wrote:
> >I'll agree that it's a better hack, but since the offending areas in
> >the SCSI subsystem are easily located, it seems wiser to fix upstream.
>
> Matt, the problem is it wasn't that *easy* to locate this, at least for me :)
> I'm not sure that this is the only place..

I didn't mean to trivialize the difficulty of finding this from the path
of tracking the symptom to the source. :)  I merely was pointing out
that now that you know the source of the problem, it's not *too* difficult
to look for buffers allocated on the stack by simple inspection of the
SCSI code.  I only jumped in on this because I felt a little guilty that
when I noticed this sometime back I got distracted and never tried to
send a patch to the maintainers. :-/

> >We still need someone with interest AND time to properly fix the
> >consistent alloc from irq issue. :)  All of the patches post to date
> >are incomplete bandaids.
>
> Uhh, I switched to solution which uses pre allocated consistent memory (10
> pages are enough for sym53c8xx_2).
> It's still not a generic solution, but at least it's safe :)

Are you doing this in the sym_2 driver or in the ppc consistent_*
implementations?  I only ask because I finally convinced myself
recently that attempting to make all the locking safe in the VM
subsystem was too much work.  I think Paul suggested at one point
that we might just preallocate a pool for atomic consistent allocations
anyway.

Regards,
--
Matt Porter
mporter@kernel.crashing.org

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: PPCBoot on Ebony board

[Brian Padalino]

I found the flash programmer utility, but I am actually a bit scared to
flash the flash with  the version of PPCBoot that I made.  I am working
without a JTAG interface, so flashing using JTAG is out of the question, and
how would I re-flash if the PPCBoot binary I have doesn't work properly?  Is
the board running out of EEPROM right now with the IBM Open Shell always
there?  I have sifted through the documentation a bit, but haven't found a
straight answer as to exactly how the board is setup right out of the box.

Any sort of help is appreciated.  I am extremely new to ICE, JTAG and such
embedded systems -- so please, be patient with me (if you can).

Sincerely,
Brian Padalino


-----Original Message-----
From: owner-linuxppc-embedded@lists.linuxppc.org
[mailto:owner-linuxppc-embedded@lists.linuxppc.org]On Behalf Of Chris
Zimman
Sent: Tuesday, July 15, 2003 5:33 PM
To: Brian Padalino
Cc: linuxppc-embedded@lists.linuxppc.org
Subject: Re: PPCBoot on Ebony board



On Tue, Jul 15, 2003 at 02:11:07PM -0400, Brian Padalino wrote:
>
> I am trying to transfer PPCBoot that I had compiled over a tftp session in
> the 440GP 1.18 ROM Monitor (02/11/02) that came on the Ebony board.  Is
that
> the wrong method of trying it?  I tried using both serial ports and Kermit
> with Xon/Xoff handshaking, but it just ended up timing out.
>
> I keep getting the error:
>   Loading file "C:\ppc\bootp\boot.img" ...
>   Sending tftp boot request ...
>   Not a valid boot image file
>
> Which stems from an improper magic number at the top of the file to say
it's
> a valid boot image.

This is not going to work.  You want to program PPCBoot into flash via a
flash programmer or via JTAG.

--Chris


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: PPCBoot on Ebony board

[Chris Zimman]

On Wed, Jul 16, 2003 at 07:59:28AM -0400, Brian Padalino wrote:
> I found the flash programmer utility, but I am actually a bit scared to
> flash the flash with  the version of PPCBoot that I made.  I am working
> without a JTAG interface, so flashing using JTAG is out of the question, and
> how would I re-flash if the PPCBoot binary I have doesn't work properly?  Is
> the board running out of EEPROM right now with the IBM Open Shell always
> there?  I have sifted through the documentation a bit, but haven't found a
> straight answer as to exactly how the board is setup right out of the box.

If you built a standard U-Boot 440GP config, odds are that it will work fine.
If you have a flash programmer, you can always save a copy of the OSOpen
image and then reflash with that if U-Boot doesn't come up right.

I'm not sure if your board is using the EEPROM, but that's only used for
strapping anyway.

> Any sort of help is appreciated.  I am extremely new to ICE, JTAG and such
> embedded systems -- so please, be patient with me (if you can).

The first thing I would recommend you do is to get a BDI2000 JTAG
debugger.  If you're going to be doing any serious work on U-Boot,
Linux, etc. this is one of the most valuable tools you can have.

--Chris

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: PPCBoot on Ebony board

[Roland Dreier]

    Brian> I found the flash programmer utility, but I am actually a
    Brian> bit scared to flash the flash with the version of PPCBoot
    Brian> that I made.  I am working without a JTAG interface, so
    Brian> flashing using JTAG is out of the question, and how would I
    Brian> re-flash if the PPCBoot binary I have doesn't work
    Brian> properly?  Is the board running out of EEPROM right now
    Brian> with the IBM Open Shell always there?  I have sifted
    Brian> through the documentation a bit, but haven't found a
    Brian> straight answer as to exactly how the board is setup right
    Brian> out of the box.

The Ebony OpenBIOS runs out of the socketed EEPROM on the board.  Just
pop the current EEPROM, put a sticker on it that says "Original
Bootloader" and use a new EEPROM for PPCBoot testing.

 - Roland

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: PPCBoot on Ebony board

[Brian Padalino]

I am actually stuck with an OCDemon Wiggler JTAG interface.  I was quoted as
being $2300 for the BDI2000.

That is a bit out of my range for this project.  Know anyone selling a used
one?

I guess I will just try to get gdb to work (in cygwin) with this wiggler.

Thanks for everyone's help!

Brian

-----Original Message-----
From: owner-linuxppc-embedded@lists.linuxppc.org
[mailto:owner-linuxppc-embedded@lists.linuxppc.org]On Behalf Of Chris
Zimman
Sent: Wednesday, July 16, 2003 10:30 AM
To: Brian Padalino
Cc: linuxppc-embedded@lists.linuxppc.org
Subject: Re: PPCBoot on Ebony board



On Wed, Jul 16, 2003 at 07:59:28AM -0400, Brian Padalino wrote:
> I found the flash programmer utility, but I am actually a bit scared to
> flash the flash with  the version of PPCBoot that I made.  I am working
> without a JTAG interface, so flashing using JTAG is out of the question,
and
> how would I re-flash if the PPCBoot binary I have doesn't work properly?
Is
> the board running out of EEPROM right now with the IBM Open Shell always
> there?  I have sifted through the documentation a bit, but haven't found a
> straight answer as to exactly how the board is setup right out of the box.

If you built a standard U-Boot 440GP config, odds are that it will work
fine.
If you have a flash programmer, you can always save a copy of the OSOpen
image and then reflash with that if U-Boot doesn't come up right.

I'm not sure if your board is using the EEPROM, but that's only used for
strapping anyway.

> Any sort of help is appreciated.  I am extremely new to ICE, JTAG and such
> embedded systems -- so please, be patient with me (if you can).

The first thing I would recommend you do is to get a BDI2000 JTAG
debugger.  If you're going to be doing any serious work on U-Boot,
Linux, etc. this is one of the most valuable tools you can have.

--Chris


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/