PCI DMA Interrupt latency

PCI DMA Interrupt latency

[Burkhard Schölpen]

Hello,

I'm writing a driver for a custom pci card with an FPGA 
(Xilinx Spartan 2 (XC2S150-6) with PCI 32 LogiCore), 
which  can act as a pci bus master. The device is designed 
to do DMA transfers with high bandwidth. One task is to 
send image data to a printer which already works quite 
well, but sometimes there are randomly occuring 
problems concerning the timing between two DMA 
transfers. The issue seems to be something like interrupt 
latency in hardware. Measuring some signals with an 
oscilloscope shows, that the delay from generating the 
interrupt, which signals a finished transfer, to the time 
when the interrupt register on the card is reset (i.e. the 
beginning of the ISR) sometimes increases to more 
than 500 microseconds, which is dimensions too high. 

I already tried with other hardware deactivated, which 
could cause traffic on the pci bus or generate many interrupts 
(except hard disk). I also increased the priority of the IRQ 
used by the pci board (with a tool called irqtune) to the 
maximum possible value. Another consideration is, that 
another driver could lock all interrupts for too long (but for 
500 us??).  

As my experience on DMA stuff is not yet too 
great, I would be very glad if somebody could give me some 
advice to solve this problem. Below there is some further 
information about my environment and how I set up the DMA 
transfers.

Kind regards,
Burkhard Schölpen


Here is how I set up dma transfers from RAM to the pci device:

while (down_interruptible(my_device->write_semaphore));
my_device->dma_write_complete = 0;
my_device->dma_direction = PCI_DMA_TODEVICE;

writel (cpu_to_le32 (virt_to_phys(dma_buffer)), MY_DMA_ADDR_REGISTER);
writel (cpu_to_le32 (my_device->dma_size/4), MY_DMA_COUNT_REGISTER); //triggers dma transfer

if (wait_event_interruptible(write_wait_queue, my_device->dma_write_complete))
{
  //handle error...
}
//test, if MY_DMA_COUNT_REGISTER contains 0
up(my_device->write_semaphore);

Inside the Interrupt-handler I do the following:

my_device->dma_write_complete = 1;
wake_up_interruptible(&write_wait_queue);
return IRQ_HANDLED;

Here is some information about the PC:
- Gigabyte GA-8I945GMF mainboard with Pentium D processor
- custom pci board with Xilinx FPGA Spartan 2 (XC2S150-6) with PCI 32 LogiCore
- Debian Linux with 2.6.13.4 SMP kernel


______________________________________________________________
Verschicken Sie romantische, coole und witzige Bilder per SMS!
Jetzt bei WEB.DE FreeMail: http://f.web.de/?mc=021193

Re: PCI DMA Interrupt latency

[Paul Fulghum]

Burkhard Schölpen wrote:
> The issue seems to be something like interrupt 
> latency in hardware. Measuring some signals with an 
> oscilloscope shows, that the delay from generating the 
> interrupt, which signals a finished transfer, to the time 
> when the interrupt register on the card is reset (i.e. the 
> beginning of the ISR) sometimes increases to more 
> than 500 microseconds, which is dimensions too high. 
> 
> ... Another consideration is, that 
> another driver could lock all interrupts for too long (but for 
> 500 us??).  

I also have implemented bus master devices based
on the Spartan 2 + Xilinx PCI core and wrote the
associated Linux driver.

My observations of interrupt latency using
a similar setup to what you describe showed
50usec is common but rare events on
the order of milliseconds are possible.
You are probably correct that the
problem as a poorly behaved driver.

If you have complete control over every system
your device is installed in, you can
find and eliminate any device that
causes high interrupt latency.

If you don't have that control, your hardware
needs to tolerate interrupt latency of
that magnitude.

You describe an implementation with a single
DMA buffer. You could switch to multiple
buffers (ring structure or 2 alternating buffers)
with an interrupt triggered after each
buffer is exhausted (I use a ring of buffers).
The remaining buffers allow the transfer to
continue while waiting for ISR execution.

Or, more crudely, trigger the interrupt when
the single buffer reaches some low water mark
and poll in the ISR for actual completion.

-- 
Paul Fulghum
Microgate Systems, Ltd.

Re: PCI DMA Interrupt latency

[Lee Revell]

On Tue, 2006-01-10 at 17:59 -0600, Robert Hancock wrote:
> Most likely some driver is disabling interrupts for that period, which 
> is really longer than it should be. However, if your card/driver require 
> such tight interrupt latency to function correctly, that seems too 
> fragile and may not be reliable. Some kind of ringbuffer arrangement 
> would likely work better, so that the interrupt does not have to be 
> serviced so soon.
> 

You can easily tell if this is the case by applying these patches:

http://people.redhat.com/mingo/latency-tracing-patches/latency-tracing-patches-2.6.15-rc7.tar.gz

It says -rc7 but they apply cleanly and work with 2.6.15 final.

Lee

Re: PCI DMA Interrupt latency

[Robert Hancock]

Burkhard Schölpen wrote:
> Hello,
> 
> I'm writing a driver for a custom pci card with an FPGA 
> (Xilinx Spartan 2 (XC2S150-6) with PCI 32 LogiCore), 
> which  can act as a pci bus master. The device is designed 
> to do DMA transfers with high bandwidth. One task is to 
> send image data to a printer which already works quite 
> well, but sometimes there are randomly occuring 
> problems concerning the timing between two DMA 
> transfers. The issue seems to be something like interrupt 
> latency in hardware. Measuring some signals with an 
> oscilloscope shows, that the delay from generating the 
> interrupt, which signals a finished transfer, to the time 
> when the interrupt register on the card is reset (i.e. the 
> beginning of the ISR) sometimes increases to more 
> than 500 microseconds, which is dimensions too high. 

Most likely some driver is disabling interrupts for that period, which 
is really longer than it should be. However, if your card/driver require 
such tight interrupt latency to function correctly, that seems too 
fragile and may not be reliable. Some kind of ringbuffer arrangement 
would likely work better, so that the interrupt does not have to be 
serviced so soon.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/