Disabling DMA with ICH10?

Disabling DMA with ICH10?

[Lennart Baruschka]

Hello,

I'm new to this list, so please forgive me if this is the wrong place to
ask. I'm trying to build a 2.6.30.5 kernel causing as little DMA traffic
as possible in order to reduce latencies on a real-time system (using
RTAI). 

1. Is there a way to completely disable DMA usage system-wide?
If not:
2. My system uses an ICH10 chipset, the harddrive is connected to the
Intel SATA controller. Is there a way to disable DMA and switch back to
PIO? I tried compiling with libata support and without SCSI support, but
the kernel is unable to mount root (no matter if /dev/sda1
or /dev/hda1), then.
3. Another source seems to be the ATL1E driver for ethernet. Is it
possible to either disable DMA for this? 

If disabling DMA is impossible for some reason for one or both drivers,
is it at least possible to reduce the burst sizes so the bus gets free
for a short time during which I could access it from my real time kernel
module?

Please point me in the right direction; any suggestion will be greatly
appreciated!

Cheers,
Lennart

Re: Disabling DMA with ICH10?

[Alejandro Riveira Fernández]

El Mon, 28 Sep 2009 23:31:21 +0200
Lennart Baruschka <FunFlyer@gmx.net> escribió:

> Hello,
> 
> I'm new to this list, so please forgive me if this is the wrong place to
> ask. I'm trying to build a 2.6.30.5 kernel causing as little DMA traffic
> as possible in order to reduce latencies on a real-time system (using
> RTAI). 
> 
> 1. Is there a way to completely disable DMA usage system-wide?
> If not:
> 2. My system uses an ICH10 chipset, the harddrive is connected to the
> Intel SATA controller. Is there a way to disable DMA and switch back to
> PIO? I tried compiling with libata support and without SCSI support, but
> the kernel is unable to mount root (no matter if /dev/sda1
> or /dev/hda1), then.
  
  AFAIK libata needs scsi support. To disable dma in libata I get from
  Documentation/kernel-parameters.txt --> libata.dma = 0


> 3. Another source seems to be the ATL1E driver for ethernet. Is it
> possible to either disable DMA for this? 
  
  That I dunno

> 
> If disabling DMA is impossible for some reason for one or both drivers,
> is it at least possible to reduce the burst sizes so the bus gets free
> for a short time during which I could access it from my real time kernel
> module?
> 
> Please point me in the right direction; any suggestion will be greatly
> appreciated!
> 
> Cheers,
> Lennart
   
  Saludos,
  Alejandro 

Re: Disabling DMA with ICH10?

[Alan Cox]

> 1. Is there a way to completely disable DMA usage system-wide?

No

> 2. My system uses an ICH10 chipset, the harddrive is connected to the
> Intel SATA controller. Is there a way to disable DMA and switch back to
> PIO? I tried compiling with libata support and without SCSI support, but
> the kernel is unable to mount root (no matter if /dev/sda1
> or /dev/hda1), then.

If you disable DMA you will make your performance and latency worse not
better as the PIO transfers will stall the bus and thus the processor.

> If disabling DMA is impossible for some reason for one or both drivers,
> is it at least possible to reduce the burst sizes so the bus gets free
> for a short time during which I could access it from my real time kernel
> module?

You want the disk controller in AHCI mode to minimise CPU stalls and
then any burst size configuration and the like is down to the PCI
configuration. On some systems where memory or bus bandwidth is an issue
then forcing the disk to run at a lower speed can help, at least for
PATA. What effect it will have on SATA I am less sure.

If you really are that latency sensitive then the more normal approach
would be to lock one core for real time use, load the critical code into
that core CPU cache and run from cache. If you are utterly pushing the
limit you might even do crazy stuff like use on thread on the core to
execute RT stuff and the other to issue any I/O accesses that might stall.

And then on many systems things like BIOS SMI glue will still clobber you
8(

Alan

Re: Disabling DMA with ICH10?

[Lennart Baruschka]

Hi, 

On Mon, 2009-09-28 at 23:40 +0100, Alan Cox wrote:
> > 2. My system uses an ICH10 chipset, the harddrive is connected to the
> > Intel SATA controller. Is there a way to disable DMA and switch back to
> > PIO? I tried compiling with libata support and without SCSI support, but
> > the kernel is unable to mount root (no matter if /dev/sda1
> > or /dev/hda1), then.
> 
> If you disable DMA you will make your performance and latency worse not
> better as the PIO transfers will stall the bus and thus the processor.

I thought that PIO transfers (which I understand to be
write32()/read32()'s) unlike DMA transfers could be interrupted by an
high-priority interrupt. Is that wrong?



> If you really are that latency sensitive then the more normal approach
> would be to lock one core for real time use, load the critical code into
> that core CPU cache and run from cache. If you are utterly pushing the
> limit you might even do crazy stuff like use on thread on the core to
> execute RT stuff and the other to issue any I/O accesses that might stall.

Actually, that's what I do - except for locking the page, yet. I do need
to access the PCI bus in real time, though. So I wonder what happens
when the RT CPU is getting data from the PCI device, doing some
calculations on it and then writing back some data to the device,
__while at the same time__ another (non-RT) CPU starts a DMA transfer. I
figured the DMA would block the PCI bus, having my interrupt wait for it
to finish. That's why I'm trying to avoid DMA. 

Please let me know at what point I am mistaken.


Cheers
Lennart

-- 
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser

Re: Disabling DMA with ICH10?

[Arjan van de Ven]

On Tue, 29 Sep 2009 09:54:16 +0200
"Lennart Baruschka" <FunFlyer@gmx.net> wrote:

> Hi, 
> 
> On Mon, 2009-09-28 at 23:40 +0100, Alan Cox wrote:
> > > 2. My system uses an ICH10 chipset, the harddrive is connected to
> > > the Intel SATA controller. Is there a way to disable DMA and
> > > switch back to PIO? I tried compiling with libata support and
> > > without SCSI support, but the kernel is unable to mount root (no
> > > matter if /dev/sda1 or /dev/hda1), then.
> > 
> > If you disable DMA you will make your performance and latency worse
> > not better as the PIO transfers will stall the bus and thus the
> > processor.
> 
> I thought that PIO transfers (which I understand to be
> write32()/read32()'s) unlike DMA transfers could be interrupted by an
> high-priority interrupt. Is that wrong?

a single PIO instruction will NOT be interrupted by an interrupt.
And can easily take several microseconds (remember: 8Mhz bus emulation)

while DMA is going on, the CPU on the other hand is just happy
executing instructions, fetching stuff from memory etc etc.


> Actually, that's what I do - except for locking the page, yet. I do
> need to access the PCI bus in real time, though. So I wonder what
> happens when the RT CPU is getting data from the PCI device, doing
> some calculations on it and then writing back some data to the device,
> __while at the same time__ another (non-RT) CPU starts a DMA
> transfer. I figured the DMA would block the PCI bus, having my
> interrupt wait for it to finish. That's why I'm trying to avoid DMA. 

the PCI bus is time sliced, with typical transfer sizes being like 256
byte bursts...



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: Disabling DMA with ICH10?

[Krzysztof Halasa]

Arjan van de Ven <arjan@infradead.org> writes:

> a single PIO instruction will NOT be interrupted by an interrupt.
> And can easily take several microseconds (remember: 8Mhz bus
> emulation)

Not that long, there is no emulation there (except for port #80 (hex?)
and similar). Normal 33 MHz access. But it's still slow, of course -
every access takes at least 4(?) bus cycles (IIRC: address, initiator
ready, device selected, target ready). No bursts (at least on typical
x86 hw).

Of course ICH10 AHCI isn't connected with PCI 33/32 bus.

Anyway: PIO = bad idea.
-- 
Krzysztof Halasa

Re: Disabling DMA with ICH10?

[Alan Cox]

On Tue, 29 Sep 2009 14:29:27 +0200
Krzysztof Halasa <khc@pm.waw.pl> wrote:

> Arjan van de Ven <arjan@infradead.org> writes:
> 
> > a single PIO instruction will NOT be interrupted by an interrupt.
> > And can easily take several microseconds (remember: 8Mhz bus
> > emulation)
> 
> Not that long, there is no emulation there (except for port #80 (hex?)
> and similar). Normal 33 MHz access.

ATA accesses go across the cable and while some of them are snooped,
cached and other magic is done there are cases it turns into a
transaction back and forth with the drive - that *is* slow. PIO 0 in fact
is ISA speed

Re: Disabling DMA with ICH10?

[Krzysztof Halasa]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

>> Not that long, there is no emulation there (except for port #80 (hex?)
>> and similar). Normal 33 MHz access.
>
> ATA accesses go across the cable and while some of them are snooped,
> cached and other magic is done there are cases it turns into a
> transaction back and forth with the drive - that *is* slow. PIO 0 in fact
> is ISA speed

Sure, PATA inserts wait states, the 4 bus cycles per access (or
something like that) is the minimum for PIO on PCI. But it's still
normal 33 MHz access (with as many wait states as the PATA mode needs),
not ISA 8 MHz emulation (fastest PIO would be faster than ISA).
-- 
Krzysztof Halasa

Re: Disabling DMA with ICH10?

[Alan Cox]

On Tue, 29 Sep 2009 22:35:54 +0200
Krzysztof Halasa <khc@pm.waw.pl> wrote:

> Alan Cox <alan@lxorguk.ukuu.org.uk> writes:
> 
> >> Not that long, there is no emulation there (except for port #80 (hex?)
> >> and similar). Normal 33 MHz access.
> >
> > ATA accesses go across the cable and while some of them are snooped,
> > cached and other magic is done there are cases it turns into a
> > transaction back and forth with the drive - that *is* slow. PIO 0 in fact
> > is ISA speed
> 
> Sure, PATA inserts wait states, the 4 bus cycles per access (or
> something like that) is the minimum for PIO on PCI. But it's still
> normal 33 MHz access (with as many wait states as the PATA mode needs),
> not ISA 8 MHz emulation (fastest PIO would be faster than ISA)

You are missing the point. The sequence is


INB foo

	CPU -> PCI device		read this register
	PCI device -> Disk
			trundle whirr whirrr
			clunk clunk thud
	Disk -> PCI device
	PCI device -> CPU... "5"

INB completes

The INB is not interruptible mid instruction and stalls the CPU for the
full period of the message passing back and forth across the bus. So PIO
0 on PCI stalls the bus for the equivalent of an ISA access, and PIO4
while a good deal faster is still a very long stall in hard real time
terms.

Re: Disabling DMA with ICH10?

[Krzysztof Halasa]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> INB foo
>
> 	CPU -> PCI device		read this register
> 	PCI device -> Disk
> 			trundle whirr whirrr
> 			clunk clunk thud
> 	Disk -> PCI device
> 	PCI device -> CPU... "5"
>
> INB completes
>
> The INB is not interruptible mid instruction and stalls the CPU for the
> full period of the message passing back and forth across the bus. So PIO
> 0 on PCI stalls the bus for the equivalent of an ISA access, and PIO4
> while a good deal faster is still a very long stall in hard real time
> terms.

Sure :-)

That's why I wrote it's a bad idea. I just wanted to correct the
statement about ISA speed - it's not (necessarily) ISA speed, though
it's slow and uses 100% CPU (core) time. (And PIO on PCI* in general
is not much faster, though usually faster than ISA).
-- 
Krzysztof Halasa

Re: Disabling DMA with ICH10?

[Alan Cox]

> I thought that PIO transfers (which I understand to be
> write32()/read32()'s) unlike DMA transfers could be interrupted by an
> high-priority interrupt. Is that wrong?

Yes. An I/O cycle will stall that CPU thread (and sometimes more) until
it completes. Not only that but the only way to do PIO only at the
hardware level is to take the controller out of AHCI mode into legacy
mode which means doing a lot of PIO accesses routed to the drive.

> Actually, that's what I do - except for locking the page, yet. I do need
> to access the PCI bus in real time, though. So I wonder what happens

I didnt think any system with an ICH10 even had a legacy PCI bus.

> when the RT CPU is getting data from the PCI device, doing some
> calculations on it and then writing back some data to the device,
> __while at the same time__ another (non-RT) CPU starts a DMA transfer. I
> figured the DMA would block the PCI bus,

It depends upon how you have configured the bus, devices and chipset what
priority rules are active and also what size bursts are used for the
transfers. In particular read up on the PCI latency setting for legacy
PCI bus devices.

Alan