Using IIO for high-speed DAQ

Using IIO for high-speed DAQ

[Matthias Klumpp]

Hello!
I would like to use IIO for high-speed data acquisition. In order to
do that I implemented a new IIO driver for the ADC chip I am using
(MAX1133, maximum sampling frequency 200ksps).

The initial approach with triggered buffers was way too slow,
achieving only a maximum sampling frequency of 4ksps.
I now changed the driver to use DMA, using the recent code landed in
IIO. The DMA dummy driver was very helpful as a template for writing
the new code and seeing how all parts fit together (why isn't this
included in the kernel?).

I am now achieving sampling frequencies of 6.6ksps, which is far
better but still way below what the chip can do. As a bonus, I got rid
of the need to trigger scans explicitly, which is quite nice for my
application.

In any case, 6.6ksps is still pretty slow, and I would ideally like to
sample with at least 10ksps, and sampling 200ksps would of course be
the ideal goal.

The question is, can IIO do that, and if so, how? Using iio_readdev
from libiio on my device prints a warning about high-speed mode not
being enabled, but I couldn't find much information on how to enable
that properly.
Maybe Lars-Peter Clausen could comment on that.

In general, if I can't use IIO, I would be stuck with rolling my own
driver without the benefit of using the IIO subsystem, or tap into
COMEDI which is a complete overkill for this usecase.

In any case, it would be awesome if someone could give me some
pointers on what to do here (I know Analog Devices uses IIO for an
oscilloscope application, so I guess high sampling rates are possible,
the question is how one can achieve that and if it makes sense at
all).

Kind regards,
    Matthias

Re: Using IIO for high-speed DAQ

[Jonathan Cameron]

On 13/01/17 14:13, Matthias Klumpp wrote:
> Hello!
> I would like to use IIO for high-speed data acquisition. In order to
> do that I implemented a new IIO driver for the ADC chip I am using
> (MAX1133, maximum sampling frequency 200ksps).
> 
> The initial approach with triggered buffers was way too slow,
> achieving only a maximum sampling frequency of 4ksps.
Any idea where the bottle neck was specifically?  I'm guessing it
might have been on the userspace side, but not certain.
People have the ti_am335x ADC on the beaglebone black running
at 200ksps without using the DMA buffer interface (through the
kfifo) Now this uses dma internally and the board has a fifo
so the data turns up in blocks so it's not really comparable.

https://www.spinics.net/lists/linux-iio/msg27422.html

> I now changed the driver to use DMA, using the recent code landed in
> IIO. The DMA dummy driver was very helpful as a template for writing
> the new code and seeing how all parts fit together (why isn't this
> included in the kernel?).
It originally got stalled on some other changes occurring in the dummy
driver (I think it was moving out of staging at the time).
Since then I know Lars has wanted to change some stuff in the
DMA buffers that has been making him not want to publicise that too
much!

When you say use DMA, this part is an SPI device - so are you hand
rolling DMA transfers from the spi controller?

We've discussed in the past (long time ago now!) how to use spi
controllers that support streaming modes but nothing has really come
of it yet. What SPI controller are you using?

Last time I was trying to do similar things (was a while ago now)
I fairly quickly hit the limitation that the round trip time on SPI
transfers even when using DMA was a lot longer than the theoretical.
Any idea where you will be limited with that?

> 
> I am now achieving sampling frequencies of 6.6ksps, which is far
> better but still way below what the chip can do. As a bonus, I got rid
> of the need to trigger scans explicitly, which is quite nice for my
> application.
> 
> In any case, 6.6ksps is still pretty slow, and I would ideally like to
> sample with at least 10ksps, and sampling 200ksps would of course be
> the ideal goal.
> 
> The question is, can IIO do that, and if so, how? Using iio_readdev
> from libiio on my device prints a warning about high-speed mode not
> being enabled, but I couldn't find much information on how to enable
> that properly.
> Maybe Lars-Peter Clausen could comment on that.
> 
> In general, if I can't use IIO, I would be stuck with rolling my own
> driver without the benefit of using the IIO subsystem, or tap into
> COMEDI which is a complete overkill for this usecase.
> 
> In any case, it would be awesome if someone could give me some
> pointers on what to do here (I know Analog Devices uses IIO for an
> oscilloscope application, so I guess high sampling rates are possible,
> the question is how one can achieve that and if it makes sense at
> all).
Lars would indeed be the person I'd ask about this.  I've cc'd
him directly as often emails get buried in the list and missed
by people for at least a little while.

Keep us informed of how you get on. Will be useful info for others.

Jonathan
> 
> Kind regards,
>     Matthias
> --
> To unsubscribe from this list: send the line "unsubscribe linux-iio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Using IIO for high-speed DAQ

[Lars-Peter Clausen]

On 01/14/2017 05:12 PM, Jonathan Cameron wrote:
> On 13/01/17 14:13, Matthias Klumpp wrote:
>> Hello!
>> I would like to use IIO for high-speed data acquisition. In order to
>> do that I implemented a new IIO driver for the ADC chip I am using
>> (MAX1133, maximum sampling frequency 200ksps).
>>
>> The initial approach with triggered buffers was way too slow,
>> achieving only a maximum sampling frequency of 4ksps.
> Any idea where the bottle neck was specifically?  I'm guessing it
> might have been on the userspace side, but not certain.

This is less of a software restriction and more of a hardware design issue.
Devices like the MAX1133 are not designed to be interfaced to a general
purpose operating system and be able to operate at datasheet performance.
The device requires to do a SPI transfer for each sample that is
transferred. In a interrupt driven environment the context switch overhead
that is introduced by this approach makes it impractical to be used at
higher sampling rates.

You either need to dedicate one full CPU to the capture process that does
nothing else but configuring the SPI controller and spinning on the SPI
transfer completion event. The data can then be transfered through a mailbox
like system to a different CPU running the application.

Or you need offloading support in hardware somewhere between the device and
the applications processor. The offloading block needs to be able to handle
flow control and group multiple sample into a larger block which is then in
bulk transferred to the application processor. This can for example be
implemented with a small FPGA or CPLD between the converter and the
application processor. In rare cases the SPI controller built into the
application processor SoC is capable of doing hardware flow control.

The IIO framework itself does not impose a limit on the maximum sampling
rate. It's all a matter of what the hardware is capable of handling. We have
systems where we do 3-digit MSPS continuous transfers and GSPS oneshot
transfers.

- Lars

Re: Using IIO for high-speed DAQ

[Matthias Klumpp]

Thank you all for the replies!

2017-01-14 18:03 GMT+01:00 Lars-Peter Clausen <lars@metafoo.de>:
> On 01/14/2017 05:12 PM, Jonathan Cameron wrote:
>> On 13/01/17 14:13, Matthias Klumpp wrote:
>>> Hello!
>>> I would like to use IIO for high-speed data acquisition. In order to
>>> do that I implemented a new IIO driver for the ADC chip I am using
>>> (MAX1133, maximum sampling frequency 200ksps).
>>>
>>> The initial approach with triggered buffers was way too slow,
>>> achieving only a maximum sampling frequency of 4ksps.
>> Any idea where the bottle neck was specifically?  I'm guessing it
>> might have been on the userspace side, but not certain.
>
> This is less of a software restriction and more of a hardware design issue.
> Devices like the MAX1133 are not designed to be interfaced to a general
> purpose operating system and be able to operate at datasheet performance.
> The device requires to do a SPI transfer for each sample that is
> transferred. In a interrupt driven environment the context switch overhead
> that is introduced by this approach makes it impractical to be used at
> higher sampling rates.

I feared that this was the case...

> You either need to dedicate one full CPU to the capture process that does
> nothing else but configuring the SPI controller and spinning on the SPI
> transfer completion event. The data can then be transfered through a mailbox
> like system to a different CPU running the application.
>
> Or you need offloading support in hardware somewhere between the device and
> the applications processor. The offloading block needs to be able to handle
> flow control and group multiple sample into a larger block which is then in
> bulk transferred to the application processor. This can for example be
> implemented with a small FPGA or CPLD between the converter and the
> application processor. In rare cases the SPI controller built into the
> application processor SoC is capable of doing hardware flow control.

Currently I think that doing any of these will require way more time
than feasible. The chip is used as part of a device which is supposed
to do real-time data analysis for a research experiment (Neuroscience
stuff), and I think it would make much more sense to just invest the
time (and money) into buying a different chip and wire it up with the
other components than rewriting the world.

Initially the system was designed very differently and due to recent
changes ended up the way it was, and I hoped to be able to save some
time by using the preexisting MAX1133 chip directly (not the best
idea, apparently).

> The IIO framework itself does not impose a limit on the maximum sampling
> rate. It's all a matter of what the hardware is capable of handling. We have
> systems where we do 3-digit MSPS continuous transfers and GSPS oneshot
> transfers.

Since this is the first time I touched this area: Is there a group of
chips which you would recommend that samples with 200ksps, has a 16bit
bit-depth and bipolar mode and is accessible at that speed via IIO?
I'll look around a bit to find a solution for this issue.

Thank you a lot for the work you put into this! The slides from your
FOSDEM/LinuxFoundation talks have been incredibly helpful in
understanding the bigger picture.

In any case, now that I have it anyway, I'll submit the MAX1133
non-DMA version of the driver as RFC after I polished it a little
(could still be useful for someone).

Regards,
    Matthias

Re: Using IIO for high-speed DAQ

[Lars-Peter Clausen]

On 01/14/2017 06:18 PM, Matthias Klumpp wrote:
>> The IIO framework itself does not impose a limit on the maximum sampling
>> rate. It's all a matter of what the hardware is capable of handling. We have
>> systems where we do 3-digit MSPS continuous transfers and GSPS oneshot
>> transfers.
> 
> Since this is the first time I touched this area: Is there a group of
> chips which you would recommend that samples with 200ksps, has a 16bit
> bit-depth and bipolar mode and is accessible at that speed via IIO?
> I'll look around a bit to find a solution for this issue.

Unfortunately this is often as  much about the AP as it is about the
converter. In the setups we have we use a AP+FPGA combination where the AP
runs Linux with IIO and the FPGA is responsible for handling the SPI flow
control during the acquisition phase. For this we've developed the
SPI-Engine framework[1]. It is compatible with most ADI single-channel SAR
and Sigma-Delta ADCs[2].

- Lars

[1] https://wiki.analog.com/resources/fpga/peripherals/spi_engine
[2]
http://www.analog.com/en/products/analog-to-digital-converters/precision-adc-10msps/single-channel-ad-converters.html

Re: Using IIO for high-speed DAQ

[Matthias Klumpp]

2017-01-14 18:49 GMT+01:00 Lars-Peter Clausen <lars@metafoo.de>:
> On 01/14/2017 06:18 PM, Matthias Klumpp wrote:
>>> The IIO framework itself does not impose a limit on the maximum sampling
>>> rate. It's all a matter of what the hardware is capable of handling. We have
>>> systems where we do 3-digit MSPS continuous transfers and GSPS oneshot
>>> transfers.
>>
>> Since this is the first time I touched this area: Is there a group of
>> chips which you would recommend that samples with 200ksps, has a 16bit
>> bit-depth and bipolar mode and is accessible at that speed via IIO?
>> I'll look around a bit to find a solution for this issue.
>
> Unfortunately this is often as  much about the AP as it is about the
> converter. In the setups we have we use a AP+FPGA combination where the AP
> runs Linux with IIO and the FPGA is responsible for handling the SPI flow
> control during the acquisition phase. For this we've developed the
> SPI-Engine framework[1]. It is compatible with most ADI single-channel SAR
> and Sigma-Delta ADCs[2].
> [...]

Thanks!
I was already looking into this, but also tried to access the BCM2835
on my Raspberry Pi test device directly (someone wrote a nice helper
library for that at [1]).
Turns out that using this library, I can easily achieve my desired
speed of 200ksps (it also turned out that with a few changes on the
setup, 6ksps can be just enough...).

Maybe the kernel abstraction is causing the delays I was experiencing,
or maybe my kernel driver is simply not optimized enough (it could
read multiple samples in batches).

If I find the bottleneck, I'll let you know...
Meanwhile, I am glad that I apparently got around making an FPGA-based
controller, that would have been a lot more work.

Regards,
    Matthias

[1]: http://www.airspayce.com/mikem/bcm2835/

Re: Using IIO for high-speed DAQ

[Lars-Peter Clausen]

On 01/16/2017 03:23 PM, Matthias Klumpp wrote:
> 2017-01-14 18:49 GMT+01:00 Lars-Peter Clausen <lars@metafoo.de>:
>> On 01/14/2017 06:18 PM, Matthias Klumpp wrote:
>>>> The IIO framework itself does not impose a limit on the maximum sampling
>>>> rate. It's all a matter of what the hardware is capable of handling. We have
>>>> systems where we do 3-digit MSPS continuous transfers and GSPS oneshot
>>>> transfers.
>>>
>>> Since this is the first time I touched this area: Is there a group of
>>> chips which you would recommend that samples with 200ksps, has a 16bit
>>> bit-depth and bipolar mode and is accessible at that speed via IIO?
>>> I'll look around a bit to find a solution for this issue.
>>
>> Unfortunately this is often as  much about the AP as it is about the
>> converter. In the setups we have we use a AP+FPGA combination where the AP
>> runs Linux with IIO and the FPGA is responsible for handling the SPI flow
>> control during the acquisition phase. For this we've developed the
>> SPI-Engine framework[1]. It is compatible with most ADI single-channel SAR
>> and Sigma-Delta ADCs[2].
>> [...]
> 
> Thanks!
> I was already looking into this, but also tried to access the BCM2835
> on my Raspberry Pi test device directly (someone wrote a nice helper
> library for that at [1]).
> Turns out that using this library, I can easily achieve my desired
> speed of 200ksps (it also turned out that with a few changes on the
> setup, 6ksps can be just enough...).

This is basically the dedicated CPU approach I described. Your application
is spinning on the SPI registers and not using interrupts. This means a lot
of CPU time is used for busy-waiting. If you are issuing one transfer after
another CPU utilization will go to 100%.

One issue with doing this in Linux userspace is that your application will
still get interrupted occasionally, either to run interrupts or potentially
schedule other applications for a short duration on the CPU. When this
happens there is a good chance that you'll loose a few sample and have a
discontinuity in your signal. Which will show itself in strange glitches
when you are processing the data.

Re: Using IIO for high-speed DAQ

[Matthias Klumpp]

2017-01-14 17:12 GMT+01:00 Jonathan Cameron <jic23@kernel.org>:
> On 13/01/17 14:13, Matthias Klumpp wrote:
>> Hello!
>> I would like to use IIO for high-speed data acquisition. In order to
>> do that I implemented a new IIO driver for the ADC chip I am using
>> (MAX1133, maximum sampling frequency 200ksps).
>>
>> The initial approach with triggered buffers was way too slow,
>> achieving only a maximum sampling frequency of 4ksps.
> Any idea where the bottle neck was specifically?  I'm guessing it
> might have been on the userspace side, but not certain.

The userspace side shouldn't be a problem, it is incredibly fast
(written to meet realtime deadlines even).

> [...]
>
> When you say use DMA, this part is an SPI device - so are you hand
> rolling DMA transfers from the spi controller?
>
> We've discussed in the past (long time ago now!) how to use spi
> controllers that support streaming modes but nothing has really come
> of it yet. What SPI controller are you using?
>
> Last time I was trying to do similar things (was a while ago now)
> I fairly quickly hit the limitation that the round trip time on SPI
> transfers even when using DMA was a lot longer than the theoretical.
> Any idea where you will be limited with that?

This is the problem - for each data acquisition, I make one spi_sync
call at time and I am really just hand-rolling SPI transfers from the
controller.
I think the gain in speed I saw came purely from the fact that I got
rid of the timer trigger, not so much from the DMA...

>> [...]
> Lars would indeed be the person I'd ask about this.  I've cc'd
> him directly as often emails get buried in the list and missed
> by people for at least a little while.
>
> Keep us informed of how you get on. Will be useful info for others.

Thank you a lot for your reply and pointers! This project I am working
on started quite small and now became way bigger than expected, with
me touching a lot of areas I have less knowledge of, so any feedback
has tremendous value.

Kind regards,
    Matthias