[PATCH v3 0/6] iio: Add DMA buffer support

[PATCH v3 0/6] iio: Add DMA buffer support

[Lars-Peter Clausen]

Changes since v2:
	* Forgot to commit some of the changes for v2, sorry for the noise

Changes since v1:
	* Typo fixes and some comment rewording
	* Dropped the dummy DMA driver until the dummy IIO driver has moved out
	  of staging

Original cover letter below:

This series contains phase 1 of adding DMA buffer support to the IIO
framework. It adds the necessary in-kernel infrastructure to support DMA
buffers, but does not yet modify the userspace ABI.

The traditional approach used in IIO to implement buffered capture requires
the generation of at least one interrupt per sample. In the interrupt
handler the driver reads the sample from the device and copies it to a
software buffer. This approach has a rather large per sample overhead
associated with it. And while it works fine for samplerates in the range of
up to 1000 samples per second it starts to consume a rather large share of
the available CPU processing time once we go beyond that, this is
especially true on an embedded system with limited processing power. The
regular interrupt also causes increased power consumption by not allowing
the hardware into deeper sleep states, which is something that becomes more
and more important on mobile battery powered devices.

And while the recently added watermark support mitigates some of the issues
by allowing the device to generate interrupts at a rate lower than the data
output rate, this still requires a storage buffer inside the device and
even if it exists it is only a few 100 samples deep at most.

DMA support on the other hand allows to capture multiple millions or even
more samples without any CPU interaction. This allows the CPU to either go
to sleep for longer periods or focus on other tasks which increases overall
system performance and power consumption. In addition to that some devices
might not even offer a way to read the data other than using DMA, which
makes DMA mandatory to use for them.

For the DMA buffer support that is introduced in this series sample data is
grouped in so called blocks. A block is the basic unit at which data is
exchanged between the application and the hardware. The application is
responsible for allocating the memory associated with the block and then
passes the block to the hardware. When the hardware has captured the amount
of samples equal to size of a block it will notify the application, which
can then read the data from the block and process it. The block size can be
freely chosen (within the constraints of the hardware). This allows to make
a trade-off between latency and management overhead. The larger the block
size the lower the per sample overhead but the latency between when the
data was captured and when the application will be able to access it
increases, in a similar way smaller block sizes have a larger per sample
management overhead but a lower latency. The ideal block size thus depends
on system and application requirements.

The series starts with some minor modifications to the watermark handling
code, primarily for dealing with fixed watermarks. It then adds support for
enable()/disable() callbacks to the iio_buffer_access_ops. This allows the
DMA buffer support to start and stop the DMA controller when the buffer is
enabled and disabled.

The next patch after that contains the core of this series. It adds a
generic infrastructure for managing DMA buffers and while this is not a
complete buffer implementation on its own it contains support for all the
management work that is necessary to support DMA buffers, i.e. implementing
the IIO buffer callbacks. The only thing not part of this infrastructure
is configuring and setting up the DMA controller hardware itself. This is
left to be implemented by a device driver which wants to support DMA buffer
support and can be hooked up into the generic infrastructure by two simple
callbacks. This minimizes the per driver code that is necessary to
implement a DMA buffer.

On top of that the series adds support for DMA buffers to the IIO dummy
simple driver. This code can be used as a template for implementing new DMA
buffer drivers as well as for testing the generic DMA buffer infrastructure
in absence of hardware with DMA capabilities.

Finally this series introduces a fully functional and completely hardware
independent DMA buffer implementation which binds the previously introduced
generic IIO DMA buffer infrastructure to the DMAengine framework. Which is
used setup the actual hardware transfers. This reduces the amount of driver
specific code for implementing a DMA buffer to two lines. One for
registering the buffer and one for unregistering it. This makes using the
DMAengine based buffer implementation the recommended way of implementing a
DMA buffer for a device.

The DMA buffer support has been successfully used and tested at Analog
Devices for a while now with a wide variety of high speed data acquisition
and processing systems with sample rates going up to multiple giga samples
per second. Andrey Yurovsky has also successfully added DMA buffer support
to the existing AT91 ADC driver allowing faster and more efficient data
capture.

The next step after this series and phase 2 is to extend the DMA buffer
support with some userspace ABI additions, this includes support for giving
userspace a more fine grained control over buffer allocation and management
as well as support for mmap() and splice(). mmap() allows to access the
buffer data from userspace without having to copy it which additionally
reduces the management overhead. While splice() allows to exchange data
with other kernel buffers without having to copy it, this can for example
be used to send the data to a network or storage disk controller.

For more background information on using IIO for high-speed data
acquisition and using the DMA buffer support please also refer to [1] and
[2] as well as [3].

- Lars

[1] http://events.linuxfoundation.org/sites/events/files/slides/iio_high_speed.pdf
[2] https://wiki.analog.com/resources/tools-software/linux-software/libiio_internals#high-speed_mmap_interface
[3] https://wiki.analog.com/resources/tools-software/linux-software/libiio_internals#zerocopy



Lars-Peter Clausen (6):
  iio: Set device watermark based on watermark of all attached buffers
  iio:iio_buffer_init(): Only set watermark if not already set
  iio: Add support for indicating fixed watermarks
  iio: Add buffer enable/disable callbacks
  iio: Add generic DMA buffer infrastructure
  iio: Add a DMAengine framework based buffer

 drivers/iio/buffer/Kconfig                         |  20 +
 drivers/iio/buffer/Makefile                        |   2 +
 drivers/iio/buffer/industrialio-buffer-dma.c       | 683 +++++++++++++++++++++
 drivers/iio/buffer/industrialio-buffer-dmaengine.c | 213 +++++++
 drivers/iio/industrialio-buffer.c                  |  58 +-
 include/linux/iio/buffer-dma.h                     | 152 +++++
 include/linux/iio/buffer-dmaengine.h               |  18 +
 include/linux/iio/buffer.h                         |  16 +
 8 files changed, 1156 insertions(+), 6 deletions(-)
 create mode 100644 drivers/iio/buffer/industrialio-buffer-dma.c
 create mode 100644 drivers/iio/buffer/industrialio-buffer-dmaengine.c
 create mode 100644 include/linux/iio/buffer-dma.h
 create mode 100644 include/linux/iio/buffer-dmaengine.h

-- 
2.1.4

[PATCH v3 1/6] iio: Set device watermark based on watermark of all attached buffers

[Lars-Peter Clausen]

Currently the watermark of the device is only set based on the watermark
that is set for the user space buffer. This doesn't consider the watermarks
set on any attached in-kernel buffers.

Change this so that the watermark of the device should be the minimum of
the watermarks over all attached buffers.

Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
---
 drivers/iio/industrialio-buffer.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/drivers/iio/industrialio-buffer.c b/drivers/iio/industrialio-buffer.c
index d7e908a..7340922 100644
--- a/drivers/iio/industrialio-buffer.c
+++ b/drivers/iio/industrialio-buffer.c
@@ -610,6 +610,7 @@ static void iio_free_scan_mask(struct iio_dev *indio_dev,
 
 struct iio_device_config {
 	unsigned int mode;
+	unsigned int watermark;
 	const unsigned long *scan_mask;
 	unsigned int scan_bytes;
 	bool scan_timestamp;
@@ -642,10 +643,14 @@ static int iio_verify_update(struct iio_dev *indio_dev,
 		if (buffer == remove_buffer)
 			continue;
 		modes &= buffer->access->modes;
+		config->watermark = min(config->watermark, buffer->watermark);
 	}
 
-	if (insert_buffer)
+	if (insert_buffer) {
 		modes &= insert_buffer->access->modes;
+		config->watermark = min(config->watermark,
+			insert_buffer->watermark);
+	}
 
 	/* Definitely possible for devices to support both of these. */
 	if ((modes & INDIO_BUFFER_TRIGGERED) && indio_dev->trig) {
@@ -743,6 +748,10 @@ static int iio_enable_buffers(struct iio_dev *indio_dev,
 		}
 	}
 
+	if (indio_dev->info->hwfifo_set_watermark)
+		indio_dev->info->hwfifo_set_watermark(indio_dev,
+			config->watermark);
+
 	indio_dev->currentmode = config->mode;
 
 	if (indio_dev->setup_ops->postenable) {
@@ -974,9 +983,6 @@ static ssize_t iio_buffer_store_watermark(struct device *dev,
 	}
 
 	buffer->watermark = val;
-
-	if (indio_dev->info->hwfifo_set_watermark)
-		indio_dev->info->hwfifo_set_watermark(indio_dev, val);
 out:
 	mutex_unlock(&indio_dev->mlock);
 
-- 
2.1.4

[PATCH v3 2/6] iio:iio_buffer_init(): Only set watermark if not already set

[Lars-Peter Clausen]

Only initialize the watermark field if it is still 0. This allows drivers
to provide a custom default watermark value. E.g. some driver might have a
fixed watermark or can only support watermarks within a certain range and
the initial value for the watermark should be within this range.

Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
---
 drivers/iio/industrialio-buffer.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/iio/industrialio-buffer.c b/drivers/iio/industrialio-buffer.c
index 7340922..5f2c8c8 100644
--- a/drivers/iio/industrialio-buffer.c
+++ b/drivers/iio/industrialio-buffer.c
@@ -193,7 +193,8 @@ void iio_buffer_init(struct iio_buffer *buffer)
 	INIT_LIST_HEAD(&buffer->buffer_list);
 	init_waitqueue_head(&buffer->pollq);
 	kref_init(&buffer->ref);
-	buffer->watermark = 1;
+	if (!buffer->watermark)
+		buffer->watermark = 1;
 }
 EXPORT_SYMBOL(iio_buffer_init);
 
-- 
2.1.4

[PATCH v3 3/6] iio: Add support for indicating fixed watermarks

[Lars-Peter Clausen]

For buffers which have a fixed wake-up watermark the watermark attribute
should be read-only. Add a new FIXED_WATERMARK flag to the
struct iio_buffer_access_funcs, which can be set by a buffer
implementation.

Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
---
 drivers/iio/industrialio-buffer.c | 5 +++++
 include/linux/iio/buffer.h        | 8 ++++++++
 2 files changed, 13 insertions(+)

diff --git a/drivers/iio/industrialio-buffer.c b/drivers/iio/industrialio-buffer.c
index 5f2c8c8..98a6447 100644
--- a/drivers/iio/industrialio-buffer.c
+++ b/drivers/iio/industrialio-buffer.c
@@ -998,6 +998,8 @@ static DEVICE_ATTR(enable, S_IRUGO | S_IWUSR,
 		   iio_buffer_show_enable, iio_buffer_store_enable);
 static DEVICE_ATTR(watermark, S_IRUGO | S_IWUSR,
 		   iio_buffer_show_watermark, iio_buffer_store_watermark);
+static struct device_attribute dev_attr_watermark_ro = __ATTR(watermark,
+	S_IRUGO, iio_buffer_show_watermark, NULL);
 
 static struct attribute *iio_buffer_attrs[] = {
 	&dev_attr_length.attr,
@@ -1040,6 +1042,9 @@ int iio_buffer_alloc_sysfs_and_mask(struct iio_dev *indio_dev)
 	if (!buffer->access->set_length)
 		attr[0] = &dev_attr_length_ro.attr;
 
+	if (buffer->access->flags & INDIO_BUFFER_FLAG_FIXED_WATERMARK)
+		attr[2] = &dev_attr_watermark_ro.attr;
+
 	if (buffer->attrs)
 		memcpy(&attr[ARRAY_SIZE(iio_buffer_attrs)], buffer->attrs,
 		       sizeof(struct attribute *) * attrcount);
diff --git a/include/linux/iio/buffer.h b/include/linux/iio/buffer.h
index 1600c55..4d99a53 100644
--- a/include/linux/iio/buffer.h
+++ b/include/linux/iio/buffer.h
@@ -18,6 +18,12 @@
 struct iio_buffer;
 
 /**
+ * INDIO_BUFFER_FLAG_FIXED_WATERMARK - Watermark level of the buffer can not be
+ *   configured. It has a fixed value which will be buffer specific.
+ */
+#define INDIO_BUFFER_FLAG_FIXED_WATERMARK BIT(0)
+
+/**
  * struct iio_buffer_access_funcs - access functions for buffers.
  * @store_to:		actually store stuff to the buffer
  * @read_first_n:	try to get a specified number of bytes (must exist)
@@ -30,6 +36,7 @@ struct iio_buffer;
  * @release:		called when the last reference to the buffer is dropped,
  *			should free all resources allocated by the buffer.
  * @modes:		Supported operating modes by this buffer type
+ * @flags:		A bitmask combination of INDIO_BUFFER_FLAG_*
  *
  * The purpose of this structure is to make the buffer element
  * modular as event for a given driver, different usecases may require
@@ -54,6 +61,7 @@ struct iio_buffer_access_funcs {
 	void (*release)(struct iio_buffer *buffer);
 
 	unsigned int modes;
+	unsigned int flags;
 };
 
 /**
-- 
2.1.4

[PATCH v3 4/6] iio: Add buffer enable/disable callbacks

[Lars-Peter Clausen]

This patch adds a enable and disable callback that is called when the
buffer is enabled/disabled. This can be used by buffer implementations that
need to do some setup or teardown work. E.g. a DMA based buffer can use
this to start/stop the DMA transfer.

Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
---
 drivers/iio/industrialio-buffer.c | 36 +++++++++++++++++++++++++++++++++++-
 include/linux/iio/buffer.h        |  8 ++++++++
 2 files changed, 43 insertions(+), 1 deletion(-)

diff --git a/drivers/iio/industrialio-buffer.c b/drivers/iio/industrialio-buffer.c
index 98a6447..a4b164a 100644
--- a/drivers/iio/industrialio-buffer.c
+++ b/drivers/iio/industrialio-buffer.c
@@ -568,6 +568,22 @@ static void iio_buffer_deactivate_all(struct iio_dev *indio_dev)
 		iio_buffer_deactivate(buffer);
 }
 
+static int iio_buffer_enable(struct iio_buffer *buffer,
+	struct iio_dev *indio_dev)
+{
+	if (!buffer->access->enable)
+		return 0;
+	return buffer->access->enable(buffer, indio_dev);
+}
+
+static int iio_buffer_disable(struct iio_buffer *buffer,
+	struct iio_dev *indio_dev)
+{
+	if (!buffer->access->disable)
+		return 0;
+	return buffer->access->disable(buffer, indio_dev);
+}
+
 static void iio_buffer_update_bytes_per_datum(struct iio_dev *indio_dev,
 	struct iio_buffer *buffer)
 {
@@ -719,6 +735,7 @@ static int iio_verify_update(struct iio_dev *indio_dev,
 static int iio_enable_buffers(struct iio_dev *indio_dev,
 	struct iio_device_config *config)
 {
+	struct iio_buffer *buffer;
 	int ret;
 
 	indio_dev->active_scan_mask = config->scan_mask;
@@ -753,6 +770,12 @@ static int iio_enable_buffers(struct iio_dev *indio_dev,
 		indio_dev->info->hwfifo_set_watermark(indio_dev,
 			config->watermark);
 
+	list_for_each_entry(buffer, &indio_dev->buffer_list, buffer_list) {
+		ret = iio_buffer_enable(buffer, indio_dev);
+		if (ret)
+			goto err_disable_buffers;
+	}
+
 	indio_dev->currentmode = config->mode;
 
 	if (indio_dev->setup_ops->postenable) {
@@ -760,12 +783,16 @@ static int iio_enable_buffers(struct iio_dev *indio_dev,
 		if (ret) {
 			dev_dbg(&indio_dev->dev,
 			       "Buffer not started: postenable failed (%d)\n", ret);
-			goto err_run_postdisable;
+			goto err_disable_buffers;
 		}
 	}
 
 	return 0;
 
+err_disable_buffers:
+	list_for_each_entry_continue_reverse(buffer, &indio_dev->buffer_list,
+					     buffer_list)
+		iio_buffer_disable(buffer, indio_dev);
 err_run_postdisable:
 	indio_dev->currentmode = INDIO_DIRECT_MODE;
 	if (indio_dev->setup_ops->postdisable)
@@ -778,6 +805,7 @@ err_undo_config:
 
 static int iio_disable_buffers(struct iio_dev *indio_dev)
 {
+	struct iio_buffer *buffer;
 	int ret = 0;
 	int ret2;
 
@@ -798,6 +826,12 @@ static int iio_disable_buffers(struct iio_dev *indio_dev)
 			ret = ret2;
 	}
 
+	list_for_each_entry(buffer, &indio_dev->buffer_list, buffer_list) {
+		ret2 = iio_buffer_disable(buffer, indio_dev);
+		if (ret2 && !ret)
+			ret = ret2;
+	}
+
 	indio_dev->currentmode = INDIO_DIRECT_MODE;
 
 	if (indio_dev->setup_ops->postdisable) {
diff --git a/include/linux/iio/buffer.h b/include/linux/iio/buffer.h
index 4d99a53..2ec3ad5 100644
--- a/include/linux/iio/buffer.h
+++ b/include/linux/iio/buffer.h
@@ -33,6 +33,11 @@ struct iio_buffer;
  *			storage.
  * @set_bytes_per_datum:set number of bytes per datum
  * @set_length:		set number of datums in buffer
+ * @enable:             called if the buffer is attached to a device and the
+ *                      device starts sampling. Calls are balanced with
+ *                      @disable.
+ * @disable:            called if the buffer is attached to a device and the
+ *                      device stops sampling. Calles are balanced with @enable.
  * @release:		called when the last reference to the buffer is dropped,
  *			should free all resources allocated by the buffer.
  * @modes:		Supported operating modes by this buffer type
@@ -58,6 +63,9 @@ struct iio_buffer_access_funcs {
 	int (*set_bytes_per_datum)(struct iio_buffer *buffer, size_t bpd);
 	int (*set_length)(struct iio_buffer *buffer, int length);
 
+	int (*enable)(struct iio_buffer *buffer, struct iio_dev *indio_dev);
+	int (*disable)(struct iio_buffer *buffer, struct iio_dev *indio_dev);
+
 	void (*release)(struct iio_buffer *buffer);
 
 	unsigned int modes;
-- 
2.1.4

[PATCH v3 5/6] iio: Add generic DMA buffer infrastructure

[Lars-Peter Clausen]

The traditional approach used in IIO to implement buffered capture requires
the generation of at least one interrupt per sample. In the interrupt
handler the driver reads the sample from the device and copies it to a
software buffer. This approach has a rather large per sample overhead
associated with it. And while it works fine for samplerates in the range of
up to 1000 samples per second it starts to consume a rather large share of
the available CPU processing time once we go beyond that, this is
especially true on an embedded system with limited processing power. The
regular interrupt also causes increased power consumption by not allowing
the hardware into deeper sleep states, which is something that becomes more
and more important on mobile battery powered devices.

And while the recently added watermark support mitigates some of the issues
by allowing the device to generate interrupts at a rate lower than the data
output rate, this still requires a storage buffer inside the device and
even if it exists it is only a few 100 samples deep at most.

DMA support on the other hand allows to capture multiple millions or even
more samples without any CPU interaction. This allows the CPU to either go
to sleep for longer periods or focus on other tasks which increases overall
system performance and power consumption. In addition to that some devices
might not even offer a way to read the data other than using DMA, which
makes DMA mandatory to use for them.

The tasks involved in implementing a DMA buffer can be divided into two
categories. The first category is memory buffer management (allocation,
mapping, etc.) and hooking this up the IIO buffer callbacks like read(),
enable(), disable(), etc. The second category of tasks is to setup the
DMA hardware and manage the DMA transfers. Tasks from the first category
will be very similar for all IIO drivers supporting DMA buffers, while the
tasks from the second category will be hardware specific.

This patch implements a generic infrastructure that take care of the former
tasks. It provides a set of functions that implement the standard IIO
buffer iio_buffer_access_funcs callbacks. These can either be used as is or
be overloaded and augmented with driver specific code where necessary.

For the DMA buffer support infrastructure that is introduced in this series
sample data is grouped by so called blocks. A block is the basic unit at
which data is exchanged between the application and the hardware. The
application is responsible for allocating the memory associated with the
block and then passes the block to the hardware. When the hardware has
captured the amount of samples equal to size of a block it will notify the
application, which can then read the data from the block and process it.
The block size can freely chosen (within the constraints of the hardware).
This allows to make a trade-off between latency and management overhead.
The larger the block size the lower the per sample overhead but the latency
between when the data was captured and when the application will be able to
access it increases, in a similar way smaller block sizes have a larger per
sample management overhead but a lower latency. The ideal block size thus
depends on system and application requirements.

For the time being the infrastructure only implements a simple double
buffered scheme which allocates two blocks each with half the size of the
configured buffer size. This provides basic support for capturing
continuous uninterrupted data over the existing file-IO ABI. Future
extensions to the DMA buffer infrastructure will give applications a more
fine grained control over how many blocks are allocated and the size of
each block. But this requires userspace ABI additions which are
intentionally not part of this patch and will be added separately.

Tasks of the second category need to be implemented by a device specific
driver. They can be hooked up into the generic infrastructure using two
simple callbacks, submit() and abort().

The submit() callback is used to schedule DMA transfers for blocks. Once a
DMA transfer has been completed it is expected that the buffer driver calls
iio_dma_buffer_block_done() to notify. The abort() callback is used for
stopping all pending and active DMA transfers when the buffer is disabled.

Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
---
 drivers/iio/buffer/Kconfig                   |   9 +
 drivers/iio/buffer/Makefile                  |   1 +
 drivers/iio/buffer/industrialio-buffer-dma.c | 683 +++++++++++++++++++++++++++
 include/linux/iio/buffer-dma.h               | 152 ++++++
 4 files changed, 845 insertions(+)
 create mode 100644 drivers/iio/buffer/industrialio-buffer-dma.c
 create mode 100644 include/linux/iio/buffer-dma.h

diff --git a/drivers/iio/buffer/Kconfig b/drivers/iio/buffer/Kconfig
index 0a7b2fd..b2fda1a 100644
--- a/drivers/iio/buffer/Kconfig
+++ b/drivers/iio/buffer/Kconfig
@@ -9,6 +9,15 @@ config IIO_BUFFER_CB
 	  Should be selected by any drivers that do in-kernel push
 	  usage.  That is, those where the data is pushed to the consumer.
 
+config IIO_BUFFER_DMA
+	tristate
+	help
+	  Provides the generic IIO DMA buffer infrastructure that can be used by
+	  drivers for devices with DMA support to implement the IIO buffer.
+
+	  Should be selected by drivers that want to use the generic DMA buffer
+	  infrastructure.
+
 config IIO_KFIFO_BUF
 	tristate "Industrial I/O buffering based on kfifo"
 	help
diff --git a/drivers/iio/buffer/Makefile b/drivers/iio/buffer/Makefile
index 4d193b9..bda3f11 100644
--- a/drivers/iio/buffer/Makefile
+++ b/drivers/iio/buffer/Makefile
@@ -4,5 +4,6 @@
 
 # When adding new entries keep the list in alphabetical order
 obj-$(CONFIG_IIO_BUFFER_CB) += industrialio-buffer-cb.o
+obj-$(CONFIG_IIO_BUFFER_DMA) += industrialio-buffer-dma.o
 obj-$(CONFIG_IIO_TRIGGERED_BUFFER) += industrialio-triggered-buffer.o
 obj-$(CONFIG_IIO_KFIFO_BUF) += kfifo_buf.o
diff --git a/drivers/iio/buffer/industrialio-buffer-dma.c b/drivers/iio/buffer/industrialio-buffer-dma.c
new file mode 100644
index 0000000..212cbed
--- /dev/null
+++ b/drivers/iio/buffer/industrialio-buffer-dma.c
@@ -0,0 +1,683 @@
+/*
+ * Copyright 2013-2015 Analog Devices Inc.
+ *  Author: Lars-Peter Clausen <lars@metafoo.de>
+ *
+ * Licensed under the GPL-2.
+ */
+
+#include <linux/slab.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/workqueue.h>
+#include <linux/mutex.h>
+#include <linux/sched.h>
+#include <linux/poll.h>
+#include <linux/iio/buffer.h>
+#include <linux/iio/buffer-dma.h>
+#include <linux/dma-mapping.h>
+#include <linux/sizes.h>
+
+/*
+ * For DMA buffers the storage is sub-divided into so called blocks. Each block
+ * has its own memory buffer. The size of the block is the granularity at which
+ * memory is exchanged between the hardware and the application. Increasing the
+ * basic unit of data exchange from one sample to one block decreases the
+ * management overhead that is associated with each sample. E.g. if we say the
+ * management overhead for one exchange is x and the unit of exchange is one
+ * sample the overhead will be x for each sample. Whereas when using a block
+ * which contains n samples the overhead per sample is reduced to x/n. This
+ * allows to achieve much higher samplerates than what can be sustained with
+ * the one sample approach.
+ *
+ * Blocks are exchanged between the DMA controller and the application via the
+ * means of two queues. The incoming queue and the outgoing queue. Blocks on the
+ * incoming queue are waiting for the DMA controller to pick them up and fill
+ * them with data. Block on the outgoing queue have been filled with data and
+ * are waiting for the application to dequeue them and read the data.
+ *
+ * A block can be in one of the following states:
+ *  * Owned by the application. In this state the application can read data from
+ *    the block.
+ *  * On the incoming list: Blocks on the incoming list are queued up to be
+ *    processed by the DMA controller.
+ *  * Owned by the DMA controller: The DMA controller is processing the block
+ *    and filling it with data.
+ *  * On the outgoing list: Blocks on the outgoing list have been successfully
+ *    processed by the DMA controller and contain data. They can be dequeued by
+ *    the application.
+ *  * Dead: A block that is dead has been marked as to be freed. It might still
+ *    be owned by either the application or the DMA controller at the moment.
+ *    But once they are done processing it instead of going to either the
+ *    incoming or outgoing queue the block will be freed.
+ *
+ * In addition to this blocks are reference counted and the memory associated
+ * with both the block structure as well as the storage memory for the block
+ * will be freed when the last reference to the block is dropped. This means a
+ * block must not be accessed without holding a reference.
+ *
+ * The iio_dma_buffer implementation provides a generic infrastructure for
+ * managing the blocks.
+ *
+ * A driver for a specific piece of hardware that has DMA capabilities need to
+ * implement the submit() callback from the iio_dma_buffer_ops structure. This
+ * callback is supposed to initiate the DMA transfer copying data from the
+ * converter to the memory region of the block. Once the DMA transfer has been
+ * completed the driver must call iio_dma_buffer_block_done() for the completed
+ * block.
+ *
+ * Prior to this it must set the bytes_used field of the block contains
+ * the actual number of bytes in the buffer. Typically this will be equal to the
+ * size of the block, but if the DMA hardware has certain alignment requirements
+ * for the transfer length it might choose to use less than the full size. In
+ * either case it is expected that bytes_used is a multiple of the bytes per
+ * datum, i.e. the block must not contain partial samples.
+ *
+ * The driver must call iio_dma_buffer_block_done() for each block it has
+ * received through its submit_block() callback, even if it does not actually
+ * perform a DMA transfer for the block, e.g. because the buffer was disabled
+ * before the block transfer was started. In this case it should set bytes_used
+ * to 0.
+ *
+ * In addition it is recommended that a driver implements the abort() callback.
+ * It will be called when the buffer is disabled and can be used to cancel
+ * pending and stop active transfers.
+ *
+ * The specific driver implementation should use the default callback
+ * implementations provided by this module for the iio_buffer_access_funcs
+ * struct. It may overload some callbacks with custom variants if the hardware
+ * has special requirements that are not handled by the generic functions. If a
+ * driver chooses to overload a callback it has to ensure that the generic
+ * callback is called from within the custom callback.
+ */
+
+static void iio_buffer_block_release(struct kref *kref)
+{
+	struct iio_dma_buffer_block *block = container_of(kref,
+		struct iio_dma_buffer_block, kref);
+
+	WARN_ON(block->state != IIO_BLOCK_STATE_DEAD);
+
+	dma_free_coherent(block->queue->dev, PAGE_ALIGN(block->size),
+					block->vaddr, block->phys_addr);
+
+	iio_buffer_put(&block->queue->buffer);
+	kfree(block);
+}
+
+static void iio_buffer_block_get(struct iio_dma_buffer_block *block)
+{
+	kref_get(&block->kref);
+}
+
+static void iio_buffer_block_put(struct iio_dma_buffer_block *block)
+{
+	kref_put(&block->kref, iio_buffer_block_release);
+}
+
+/*
+ * dma_free_coherent can sleep, hence we need to take some special care to be
+ * able to drop a reference from an atomic context.
+ */
+static LIST_HEAD(iio_dma_buffer_dead_blocks);
+static DEFINE_SPINLOCK(iio_dma_buffer_dead_blocks_lock);
+
+static void iio_dma_buffer_cleanup_worker(struct work_struct *work)
+{
+	struct iio_dma_buffer_block *block, *_block;
+	LIST_HEAD(block_list);
+
+	spin_lock_irq(&iio_dma_buffer_dead_blocks_lock);
+	list_splice_tail_init(&iio_dma_buffer_dead_blocks, &block_list);
+	spin_unlock_irq(&iio_dma_buffer_dead_blocks_lock);
+
+	list_for_each_entry_safe(block, _block, &block_list, head)
+		iio_buffer_block_release(&block->kref);
+}
+static DECLARE_WORK(iio_dma_buffer_cleanup_work, iio_dma_buffer_cleanup_worker);
+
+static void iio_buffer_block_release_atomic(struct kref *kref)
+{
+	struct iio_dma_buffer_block *block;
+	unsigned long flags;
+
+	block = container_of(kref, struct iio_dma_buffer_block, kref);
+
+	spin_lock_irqsave(&iio_dma_buffer_dead_blocks_lock, flags);
+	list_add_tail(&block->head, &iio_dma_buffer_dead_blocks);
+	spin_unlock_irqrestore(&iio_dma_buffer_dead_blocks_lock, flags);
+
+	schedule_work(&iio_dma_buffer_cleanup_work);
+}
+
+/*
+ * Version of iio_buffer_block_put() that can be called from atomic context
+ */
+static void iio_buffer_block_put_atomic(struct iio_dma_buffer_block *block)
+{
+	kref_put(&block->kref, iio_buffer_block_release_atomic);
+}
+
+static struct iio_dma_buffer_queue *iio_buffer_to_queue(struct iio_buffer *buf)
+{
+	return container_of(buf, struct iio_dma_buffer_queue, buffer);
+}
+
+static struct iio_dma_buffer_block *iio_dma_buffer_alloc_block(
+	struct iio_dma_buffer_queue *queue, size_t size)
+{
+	struct iio_dma_buffer_block *block;
+
+	block = kzalloc(sizeof(*block), GFP_KERNEL);
+	if (!block)
+		return NULL;
+
+	block->vaddr = dma_alloc_coherent(queue->dev, PAGE_ALIGN(size),
+		&block->phys_addr, GFP_KERNEL);
+	if (!block->vaddr) {
+		kfree(block);
+		return NULL;
+	}
+
+	block->size = size;
+	block->state = IIO_BLOCK_STATE_DEQUEUED;
+	block->queue = queue;
+	INIT_LIST_HEAD(&block->head);
+	kref_init(&block->kref);
+
+	iio_buffer_get(&queue->buffer);
+
+	return block;
+}
+
+static void _iio_dma_buffer_block_done(struct iio_dma_buffer_block *block)
+{
+	struct iio_dma_buffer_queue *queue = block->queue;
+
+	/*
+	 * The buffer has already been freed by the application, just drop the
+	 * reference.
+	 */
+	if (block->state != IIO_BLOCK_STATE_DEAD) {
+		block->state = IIO_BLOCK_STATE_DONE;
+		list_add_tail(&block->head, &queue->outgoing);
+	}
+}
+
+/**
+ * iio_dma_buffer_block_done() - Indicate that a block has been completed
+ * @block: The completed block
+ *
+ * Should be called when the DMA controller has finished handling the block to
+ * pass back ownership of the block to the queue.
+ */
+void iio_dma_buffer_block_done(struct iio_dma_buffer_block *block)
+{
+	struct iio_dma_buffer_queue *queue = block->queue;
+	unsigned long flags;
+
+	spin_lock_irqsave(&queue->list_lock, flags);
+	_iio_dma_buffer_block_done(block);
+	spin_unlock_irqrestore(&queue->list_lock, flags);
+
+	iio_buffer_block_put_atomic(block);
+	wake_up_interruptible_poll(&queue->buffer.pollq, POLLIN | POLLRDNORM);
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_block_done);
+
+/**
+ * iio_dma_buffer_block_list_abort() - Indicate that a list block has been
+ *   aborted
+ * @queue: Queue for which to complete blocks.
+ * @list: List of aborted blocks. All blocks in this list must be from @queue.
+ *
+ * Typically called from the abort() callback after the DMA controller has been
+ * stopped. This will set bytes_used to 0 for each block in the list and then
+ * hand the blocks back to the queue.
+ */
+void iio_dma_buffer_block_list_abort(struct iio_dma_buffer_queue *queue,
+	struct list_head *list)
+{
+	struct iio_dma_buffer_block *block, *_block;
+	unsigned long flags;
+
+	spin_lock_irqsave(&queue->list_lock, flags);
+	list_for_each_entry_safe(block, _block, list, head) {
+		list_del(&block->head);
+		block->bytes_used = 0;
+		_iio_dma_buffer_block_done(block);
+		iio_buffer_block_put_atomic(block);
+	}
+	spin_unlock_irqrestore(&queue->list_lock, flags);
+
+	wake_up_interruptible_poll(&queue->buffer.pollq, POLLIN | POLLRDNORM);
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_block_list_abort);
+
+static bool iio_dma_block_reusable(struct iio_dma_buffer_block *block)
+{
+	/*
+	 * If the core owns the block it can be re-used. This should be the
+	 * default case when enabling the buffer, unless the DMA controller does
+	 * not support abort and has not given back the block yet.
+	 */
+	switch (block->state) {
+	case IIO_BLOCK_STATE_DEQUEUED:
+	case IIO_BLOCK_STATE_QUEUED:
+	case IIO_BLOCK_STATE_DONE:
+		return true;
+	default:
+		return false;
+	}
+}
+
+/**
+ * iio_dma_buffer_request_update() - DMA buffer request_update callback
+ * @buffer: The buffer which to request an update
+ *
+ * Should be used as the iio_dma_buffer_request_update() callback for
+ * iio_buffer_access_ops struct for DMA buffers.
+ */
+int iio_dma_buffer_request_update(struct iio_buffer *buffer)
+{
+	struct iio_dma_buffer_queue *queue = iio_buffer_to_queue(buffer);
+	struct iio_dma_buffer_block *block;
+	bool try_reuse = false;
+	size_t size;
+	int ret = 0;
+	int i;
+
+	/*
+	 * Split the buffer into two even parts. This is used as a double
+	 * buffering scheme with usually one block at a time being used by the
+	 * DMA and the other one by the application.
+	 */
+	size = DIV_ROUND_UP(queue->buffer.bytes_per_datum *
+		queue->buffer.length, 2);
+
+	mutex_lock(&queue->lock);
+
+	/* Allocations are page aligned */
+	if (PAGE_ALIGN(queue->fileio.block_size) == PAGE_ALIGN(size))
+		try_reuse = true;
+
+	queue->fileio.block_size = size;
+	queue->fileio.active_block = NULL;
+
+	spin_lock_irq(&queue->list_lock);
+	for (i = 0; i < 2; i++) {
+		block = queue->fileio.blocks[i];
+
+		/* If we can't re-use it free it */
+		if (block && (!iio_dma_block_reusable(block) || !try_reuse))
+			block->state = IIO_BLOCK_STATE_DEAD;
+	}
+
+	/*
+	 * At this point all blocks are either owned by the core or marked as
+	 * dead. This means we can reset the lists without having to fear
+	 * corrution.
+	 */
+	INIT_LIST_HEAD(&queue->outgoing);
+	spin_unlock_irq(&queue->list_lock);
+
+	INIT_LIST_HEAD(&queue->incoming);
+
+	for (i = 0; i < 2; i++) {
+		if (queue->fileio.blocks[i]) {
+			block = queue->fileio.blocks[i];
+			if (block->state == IIO_BLOCK_STATE_DEAD) {
+				/* Could not reuse it */
+				iio_buffer_block_put(block);
+				block = NULL;
+			} else {
+				block->size = size;
+			}
+		} else {
+			block = NULL;
+		}
+
+		if (!block) {
+			block = iio_dma_buffer_alloc_block(queue, size);
+			if (!block) {
+				ret = -ENOMEM;
+				goto out_unlock;
+			}
+			queue->fileio.blocks[i] = block;
+		}
+
+		block->state = IIO_BLOCK_STATE_QUEUED;
+		list_add_tail(&block->head, &queue->incoming);
+	}
+
+out_unlock:
+	mutex_unlock(&queue->lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_request_update);
+
+static void iio_dma_buffer_submit_block(struct iio_dma_buffer_queue *queue,
+	struct iio_dma_buffer_block *block)
+{
+	int ret;
+
+	/*
+	 * If the hardware has already been removed we put the block into
+	 * limbo. It will neither be on the incoming nor outgoing list, nor will
+	 * it ever complete. It will just wait to be freed eventually.
+	 */
+	if (!queue->ops)
+		return;
+
+	block->state = IIO_BLOCK_STATE_ACTIVE;
+	iio_buffer_block_get(block);
+	ret = queue->ops->submit(queue, block);
+	if (ret) {
+		/*
+		 * This is a bit of a problem and there is not much we can do
+		 * other then wait for the buffer to be disabled and re-enabled
+		 * and try again. But it should not really happen unless we run
+		 * out of memory or something similar.
+		 *
+		 * TODO: Implement support in the IIO core to allow buffers to
+		 * notify consumers that something went wrong and the buffer
+		 * should be disabled.
+		 */
+		iio_buffer_block_put(block);
+	}
+}
+
+/**
+ * iio_dma_buffer_enable() - Enable DMA buffer
+ * @buffer: IIO buffer to enable
+ * @indio_dev: IIO device the buffer is attached to
+ *
+ * Needs to be called when the device that the buffer is attached to starts
+ * sampling. Typically should be the iio_buffer_access_ops enable callback.
+ *
+ * This will allocate the DMA buffers and start the DMA transfers.
+ */
+int iio_dma_buffer_enable(struct iio_buffer *buffer,
+	struct iio_dev *indio_dev)
+{
+	struct iio_dma_buffer_queue *queue = iio_buffer_to_queue(buffer);
+	struct iio_dma_buffer_block *block, *_block;
+
+	mutex_lock(&queue->lock);
+	queue->active = true;
+	list_for_each_entry_safe(block, _block, &queue->incoming, head) {
+		list_del(&block->head);
+		iio_dma_buffer_submit_block(queue, block);
+	}
+	mutex_unlock(&queue->lock);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_enable);
+
+/**
+ * iio_dma_buffer_disable() - Disable DMA buffer
+ * @buffer: IIO DMA buffer to disable
+ * @indio_dev: IIO device the buffer is attached to
+ *
+ * Needs to be called when the device that the buffer is attached to stops
+ * sampling. Typically should be the iio_buffer_access_ops disable callback.
+ */
+int iio_dma_buffer_disable(struct iio_buffer *buffer,
+	struct iio_dev *indio_dev)
+{
+	struct iio_dma_buffer_queue *queue = iio_buffer_to_queue(buffer);
+
+	mutex_lock(&queue->lock);
+	queue->active = false;
+
+	if (queue->ops && queue->ops->abort)
+		queue->ops->abort(queue);
+	mutex_unlock(&queue->lock);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_disable);
+
+static void iio_dma_buffer_enqueue(struct iio_dma_buffer_queue *queue,
+	struct iio_dma_buffer_block *block)
+{
+	if (block->state == IIO_BLOCK_STATE_DEAD) {
+		iio_buffer_block_put(block);
+	} else if (queue->active) {
+		iio_dma_buffer_submit_block(queue, block);
+	} else {
+		block->state = IIO_BLOCK_STATE_QUEUED;
+		list_add_tail(&block->head, &queue->incoming);
+	}
+}
+
+static struct iio_dma_buffer_block *iio_dma_buffer_dequeue(
+	struct iio_dma_buffer_queue *queue)
+{
+	struct iio_dma_buffer_block *block;
+
+	spin_lock_irq(&queue->list_lock);
+	block = list_first_entry_or_null(&queue->outgoing, struct
+		iio_dma_buffer_block, head);
+	if (block != NULL) {
+		list_del(&block->head);
+		block->state = IIO_BLOCK_STATE_DEQUEUED;
+	}
+	spin_unlock_irq(&queue->list_lock);
+
+	return block;
+}
+
+/**
+ * iio_dma_buffer_read() - DMA buffer read callback
+ * @buffer: Buffer to read form
+ * @n: Number of bytes to read
+ * @user_buffer: Userspace buffer to copy the data to
+ *
+ * Should be used as the read_first_n callback for iio_buffer_access_ops
+ * struct for DMA buffers.
+ */
+int iio_dma_buffer_read(struct iio_buffer *buffer, size_t n,
+	char __user *user_buffer)
+{
+	struct iio_dma_buffer_queue *queue = iio_buffer_to_queue(buffer);
+	struct iio_dma_buffer_block *block;
+	int ret;
+
+	if (n < buffer->bytes_per_datum)
+		return -EINVAL;
+
+	mutex_lock(&queue->lock);
+
+	if (!queue->fileio.active_block) {
+		block = iio_dma_buffer_dequeue(queue);
+		if (block == NULL) {
+			ret = 0;
+			goto out_unlock;
+		}
+		queue->fileio.pos = 0;
+		queue->fileio.active_block = block;
+	} else {
+		block = queue->fileio.active_block;
+	}
+
+	n = rounddown(n, buffer->bytes_per_datum);
+	if (n > block->bytes_used - queue->fileio.pos)
+		n = block->bytes_used - queue->fileio.pos;
+
+	if (copy_to_user(user_buffer, block->vaddr + queue->fileio.pos, n)) {
+		ret = -EFAULT;
+		goto out_unlock;
+	}
+
+	queue->fileio.pos += n;
+
+	if (queue->fileio.pos == block->bytes_used) {
+		queue->fileio.active_block = NULL;
+		iio_dma_buffer_enqueue(queue, block);
+	}
+
+	ret = n;
+
+out_unlock:
+	mutex_unlock(&queue->lock);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_read);
+
+/**
+ * iio_dma_buffer_data_available() - DMA buffer data_available callback
+ * @buf: Buffer to check for data availability
+ *
+ * Should be used as the data_available callback for iio_buffer_access_ops
+ * struct for DMA buffers.
+ */
+size_t iio_dma_buffer_data_available(struct iio_buffer *buf)
+{
+	struct iio_dma_buffer_queue *queue = iio_buffer_to_queue(buf);
+	struct iio_dma_buffer_block *block;
+	size_t data_available = 0;
+
+	/*
+	 * For counting the available bytes we'll use the size of the block not
+	 * the number of actual bytes available in the block. Otherwise it is
+	 * possible that we end up with a value that is lower than the watermark
+	 * but won't increase since all blocks are in use.
+	 */
+
+	mutex_lock(&queue->lock);
+	if (queue->fileio.active_block)
+		data_available += queue->fileio.active_block->size;
+
+	spin_lock_irq(&queue->list_lock);
+	list_for_each_entry(block, &queue->outgoing, head)
+		data_available += block->size;
+	spin_unlock_irq(&queue->list_lock);
+	mutex_unlock(&queue->lock);
+
+	return data_available;
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_data_available);
+
+/**
+ * iio_dma_buffer_set_bytes_per_datum() - DMA buffer set_bytes_per_datum callback
+ * @buffer: Buffer to set the bytes-per-datum for
+ * @bpd: The new bytes-per-datum value
+ *
+ * Should be used as the set_bytes_per_datum callback for iio_buffer_access_ops
+ * struct for DMA buffers.
+ */
+int iio_dma_buffer_set_bytes_per_datum(struct iio_buffer *buffer, size_t bpd)
+{
+	buffer->bytes_per_datum = bpd;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_set_bytes_per_datum);
+
+/**
+ * iio_dma_buffer_set_length - DMA buffer set_length callback
+ * @buffer: Buffer to set the length for
+ * @length: The new buffer length
+ *
+ * Should be used as the set_length callback for iio_buffer_access_ops
+ * struct for DMA buffers.
+ */
+int iio_dma_buffer_set_length(struct iio_buffer *buffer, int length)
+{
+	/* Avoid an invalid state */
+	if (length < 2)
+		length = 2;
+	buffer->length = length;
+	buffer->watermark = length / 2;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_set_length);
+
+/**
+ * iio_dma_buffer_init() - Initialize DMA buffer queue
+ * @queue: Buffer to initialize
+ * @dev: DMA device
+ * @ops: DMA buffer queue callback operations
+ *
+ * The DMA device will be used by the queue to do DMA memory allocations. So it
+ * should refer to the device that will perform the DMA to ensure that
+ * allocations are done from a memory region that can be accessed by the device.
+ */
+int iio_dma_buffer_init(struct iio_dma_buffer_queue *queue,
+	struct device *dev, const struct iio_dma_buffer_ops *ops)
+{
+	iio_buffer_init(&queue->buffer);
+	queue->buffer.length = PAGE_SIZE;
+	queue->buffer.watermark = queue->buffer.length / 2;
+	queue->dev = dev;
+	queue->ops = ops;
+
+	INIT_LIST_HEAD(&queue->incoming);
+	INIT_LIST_HEAD(&queue->outgoing);
+
+	mutex_init(&queue->lock);
+	spin_lock_init(&queue->list_lock);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_init);
+
+/**
+ * iio_dma_buffer_exit() - Cleanup DMA buffer queue
+ * @queue: Buffer to cleanup
+ *
+ * After this function has completed it is safe to free any resources that are
+ * associated with the buffer and are accessed inside the callback operations.
+ */
+void iio_dma_buffer_exit(struct iio_dma_buffer_queue *queue)
+{
+	unsigned int i;
+
+	mutex_lock(&queue->lock);
+
+	spin_lock_irq(&queue->list_lock);
+	for (i = 0; i < ARRAY_SIZE(queue->fileio.blocks); i++) {
+		if (!queue->fileio.blocks[i])
+			continue;
+		queue->fileio.blocks[i]->state = IIO_BLOCK_STATE_DEAD;
+	}
+	INIT_LIST_HEAD(&queue->outgoing);
+	spin_unlock_irq(&queue->list_lock);
+
+	INIT_LIST_HEAD(&queue->incoming);
+
+	for (i = 0; i < ARRAY_SIZE(queue->fileio.blocks); i++) {
+		if (!queue->fileio.blocks[i])
+			continue;
+		iio_buffer_block_put(queue->fileio.blocks[i]);
+		queue->fileio.blocks[i] = NULL;
+	}
+	queue->fileio.active_block = NULL;
+	queue->ops = NULL;
+
+	mutex_unlock(&queue->lock);
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_exit);
+
+/**
+ * iio_dma_buffer_release() - Release final buffer resources
+ * @queue: Buffer to release
+ *
+ * Frees resources that can't yet be freed in iio_dma_buffer_exit(). Should be
+ * called in the buffers release callback implementation right before freeing
+ * the memory associated with the buffer.
+ */
+void iio_dma_buffer_release(struct iio_dma_buffer_queue *queue)
+{
+	mutex_destroy(&queue->lock);
+}
+EXPORT_SYMBOL_GPL(iio_dma_buffer_release);
+
+MODULE_AUTHOR("Lars-Peter Clausen <lars@metafoo.de>");
+MODULE_DESCRIPTION("DMA buffer for the IIO framework");
+MODULE_LICENSE("GPL v2");
diff --git a/include/linux/iio/buffer-dma.h b/include/linux/iio/buffer-dma.h
new file mode 100644
index 0000000..767467d
--- /dev/null
+++ b/include/linux/iio/buffer-dma.h
@@ -0,0 +1,152 @@
+/*
+ * Copyright 2013-2015 Analog Devices Inc.
+ *  Author: Lars-Peter Clausen <lars@metafoo.de>
+ *
+ * Licensed under the GPL-2.
+ */
+
+#ifndef __INDUSTRIALIO_DMA_BUFFER_H__
+#define __INDUSTRIALIO_DMA_BUFFER_H__
+
+#include <linux/list.h>
+#include <linux/kref.h>
+#include <linux/spinlock.h>
+#include <linux/mutex.h>
+#include <linux/iio/buffer.h>
+
+struct iio_dma_buffer_queue;
+struct iio_dma_buffer_ops;
+struct device;
+
+struct iio_buffer_block {
+	u32 size;
+	u32 bytes_used;
+};
+
+/**
+ * enum iio_block_state - State of a struct iio_dma_buffer_block
+ * @IIO_BLOCK_STATE_DEQUEUED: Block is not queued
+ * @IIO_BLOCK_STATE_QUEUED: Block is on the incoming queue
+ * @IIO_BLOCK_STATE_ACTIVE: Block is currently being processed by the DMA
+ * @IIO_BLOCK_STATE_DONE: Block is on the outgoing queue
+ * @IIO_BLOCK_STATE_DEAD: Block has been marked as to be freed
+ */
+enum iio_block_state {
+	IIO_BLOCK_STATE_DEQUEUED,
+	IIO_BLOCK_STATE_QUEUED,
+	IIO_BLOCK_STATE_ACTIVE,
+	IIO_BLOCK_STATE_DONE,
+	IIO_BLOCK_STATE_DEAD,
+};
+
+/**
+ * struct iio_dma_buffer_block - IIO buffer block
+ * @head: List head
+ * @size: Total size of the block in bytes
+ * @bytes_used: Number of bytes that contain valid data
+ * @vaddr: Virutal address of the blocks memory
+ * @phys_addr: Physical address of the blocks memory
+ * @queue: Parent DMA buffer queue
+ * @kref: kref used to manage the lifetime of block
+ * @state: Current state of the block
+ */
+struct iio_dma_buffer_block {
+	/* May only be accessed by the owner of the block */
+	struct list_head head;
+	size_t bytes_used;
+
+	/*
+	 * Set during allocation, constant thereafter. May be accessed read-only
+	 * by anybody holding a reference to the block.
+	 */
+	void *vaddr;
+	dma_addr_t phys_addr;
+	size_t size;
+	struct iio_dma_buffer_queue *queue;
+
+	/* Must not be accessed outside the core. */
+	struct kref kref;
+	/*
+	 * Must not be accessed outside the core. Access needs to hold
+	 * queue->list_lock if the block is not owned by the core.
+	 */
+	enum iio_block_state state;
+};
+
+/**
+ * struct iio_dma_buffer_queue_fileio - FileIO state for the DMA buffer
+ * @blocks: Buffer blocks used for fileio
+ * @active_block: Block being used in read()
+ * @pos: Read offset in the active block
+ * @block_size: Size of each block
+ */
+struct iio_dma_buffer_queue_fileio {
+	struct iio_dma_buffer_block *blocks[2];
+	struct iio_dma_buffer_block *active_block;
+	size_t pos;
+	size_t block_size;
+};
+
+/**
+ * struct iio_dma_buffer_queue - DMA buffer base structure
+ * @buffer: IIO buffer base structure
+ * @dev: Parent device
+ * @ops: DMA buffer callbacks
+ * @lock: Protects the incoming list, active and the fields in the fileio
+ *   substruct
+ * @list_lock: Protects lists that contain blocks which can be modified in
+ *   atomic context as well as blocks on those lists. This is the outgoing queue
+ *   list and typically also a list of active blocks in the part that handles
+ *   the DMA controller
+ * @incoming: List of buffers on the incoming queue
+ * @outgoing: List of buffers on the outgoing queue
+ * @active: Whether the buffer is currently active
+ * @fileio: FileIO state
+ */
+struct iio_dma_buffer_queue {
+	struct iio_buffer buffer;
+	struct device *dev;
+	const struct iio_dma_buffer_ops *ops;
+
+	struct mutex lock;
+	spinlock_t list_lock;
+	struct list_head incoming;
+	struct list_head outgoing;
+
+	bool active;
+
+	struct iio_dma_buffer_queue_fileio fileio;
+};
+
+/**
+ * struct iio_dma_buffer_ops - DMA buffer callback operations
+ * @submit: Called when a block is submitted to the DMA controller
+ * @abort: Should abort all pending transfers
+ */
+struct iio_dma_buffer_ops {
+	int (*submit)(struct iio_dma_buffer_queue *queue,
+		struct iio_dma_buffer_block *block);
+	void (*abort)(struct iio_dma_buffer_queue *queue);
+};
+
+void iio_dma_buffer_block_done(struct iio_dma_buffer_block *block);
+void iio_dma_buffer_block_list_abort(struct iio_dma_buffer_queue *queue,
+	struct list_head *list);
+
+int iio_dma_buffer_enable(struct iio_buffer *buffer,
+	struct iio_dev *indio_dev);
+int iio_dma_buffer_disable(struct iio_buffer *buffer,
+	struct iio_dev *indio_dev);
+int iio_dma_buffer_read(struct iio_buffer *buffer, size_t n,
+	char __user *user_buffer);
+size_t iio_dma_buffer_data_available(struct iio_buffer *buffer);
+int iio_dma_buffer_set_bytes_per_datum(struct iio_buffer *buffer, size_t bpd);
+int iio_dma_buffer_set_length(struct iio_buffer *buffer, int length);
+int iio_dma_buffer_request_update(struct iio_buffer *buffer);
+
+int iio_dma_buffer_init(struct iio_dma_buffer_queue *queue,
+	struct device *dma_dev, const struct iio_dma_buffer_ops *ops);
+void iio_dma_buffer_exit(struct iio_dma_buffer_queue *queue);
+void iio_dma_buffer_release(struct iio_dma_buffer_queue *queue);
+
+#endif
-- 
2.1.4

[PATCH v3 6/6] iio: Add a DMAengine framework based buffer

[Lars-Peter Clausen]

Add a generic fully device independent DMA buffer implementation that uses
the DMAegnine framework to perform the DMA transfers. This can be used by
converter drivers that whish to provide a DMA buffer for converters that
are connected to a DMA core that implements the DMAengine API.

Apart from allocating the buffer using iio_dmaengine_buffer_alloc() and
freeing it using iio_dmaengine_buffer_free() no additional converter driver
specific code is required when using this DMA buffer implementation.

Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
---
 drivers/iio/buffer/Kconfig                         |  11 ++
 drivers/iio/buffer/Makefile                        |   1 +
 drivers/iio/buffer/industrialio-buffer-dmaengine.c | 213 +++++++++++++++++++++
 include/linux/iio/buffer-dmaengine.h               |  18 ++
 4 files changed, 243 insertions(+)
 create mode 100644 drivers/iio/buffer/industrialio-buffer-dmaengine.c
 create mode 100644 include/linux/iio/buffer-dmaengine.h

diff --git a/drivers/iio/buffer/Kconfig b/drivers/iio/buffer/Kconfig
index b2fda1a..4ffd3db 100644
--- a/drivers/iio/buffer/Kconfig
+++ b/drivers/iio/buffer/Kconfig
@@ -18,6 +18,17 @@ config IIO_BUFFER_DMA
 	  Should be selected by drivers that want to use the generic DMA buffer
 	  infrastructure.
 
+config IIO_BUFFER_DMAENGINE
+	tristate
+	select IIO_BUFFER_DMA
+	help
+	  Provides a bonding of the generic IIO DMA buffer infrastructure with the
+	  DMAengine framework. This can be used by converter drivers with a DMA port
+	  connected to an external DMA controller which is supported by the
+	  DMAengine framework.
+
+	  Should be selected by drivers that want to use this functionality.
+
 config IIO_KFIFO_BUF
 	tristate "Industrial I/O buffering based on kfifo"
 	help
diff --git a/drivers/iio/buffer/Makefile b/drivers/iio/buffer/Makefile
index bda3f11..85beaae 100644
--- a/drivers/iio/buffer/Makefile
+++ b/drivers/iio/buffer/Makefile
@@ -5,5 +5,6 @@
 # When adding new entries keep the list in alphabetical order
 obj-$(CONFIG_IIO_BUFFER_CB) += industrialio-buffer-cb.o
 obj-$(CONFIG_IIO_BUFFER_DMA) += industrialio-buffer-dma.o
+obj-$(CONFIG_IIO_BUFFER_DMAENGINE) += industrialio-buffer-dmaengine.o
 obj-$(CONFIG_IIO_TRIGGERED_BUFFER) += industrialio-triggered-buffer.o
 obj-$(CONFIG_IIO_KFIFO_BUF) += kfifo_buf.o
diff --git a/drivers/iio/buffer/industrialio-buffer-dmaengine.c b/drivers/iio/buffer/industrialio-buffer-dmaengine.c
new file mode 100644
index 0000000..ebdb838
--- /dev/null
+++ b/drivers/iio/buffer/industrialio-buffer-dmaengine.c
@@ -0,0 +1,213 @@
+/*
+ * Copyright 2014-2015 Analog Devices Inc.
+ *  Author: Lars-Peter Clausen <lars@metafoo.de>
+ *
+ * Licensed under the GPL-2 or later.
+ */
+
+#include <linux/slab.h>
+#include <linux/kernel.h>
+#include <linux/dmaengine.h>
+#include <linux/dma-mapping.h>
+#include <linux/spinlock.h>
+#include <linux/err.h>
+
+#include <linux/iio/iio.h>
+#include <linux/iio/buffer.h>
+#include <linux/iio/buffer-dma.h>
+#include <linux/iio/buffer-dmaengine.h>
+
+/*
+ * The IIO DMAengine buffer combines the generic IIO DMA buffer infrastructure
+ * with the DMAengine framework. The generic IIO DMA buffer infrastructure is
+ * used to manage the buffer memory and implement the IIO buffer operations
+ * while the DMAengine framework is used to perform the DMA transfers. Combined
+ * this results in a device independent fully functional DMA buffer
+ * implementation that can be used by device drivers for peripherals which are
+ * connected to a DMA controller which has a DMAengine driver implementation.
+ */
+
+struct dmaengine_buffer {
+	struct iio_dma_buffer_queue queue;
+
+	struct dma_chan *chan;
+	struct list_head active;
+
+	size_t align;
+	size_t max_size;
+};
+
+static struct dmaengine_buffer *iio_buffer_to_dmaengine_buffer(
+		struct iio_buffer *buffer)
+{
+	return container_of(buffer, struct dmaengine_buffer, queue.buffer);
+}
+
+static void iio_dmaengine_buffer_block_done(void *data)
+{
+	struct iio_dma_buffer_block *block = data;
+	unsigned long flags;
+
+	spin_lock_irqsave(&block->queue->list_lock, flags);
+	list_del(&block->head);
+	spin_unlock_irqrestore(&block->queue->list_lock, flags);
+	iio_dma_buffer_block_done(block);
+}
+
+static int iio_dmaengine_buffer_submit_block(struct iio_dma_buffer_queue *queue,
+	struct iio_dma_buffer_block *block)
+{
+	struct dmaengine_buffer *dmaengine_buffer =
+		iio_buffer_to_dmaengine_buffer(&queue->buffer);
+	struct dma_async_tx_descriptor *desc;
+	dma_cookie_t cookie;
+
+	block->bytes_used = min(block->size, dmaengine_buffer->max_size);
+	block->bytes_used = rounddown(block->bytes_used,
+			dmaengine_buffer->align);
+
+	desc = dmaengine_prep_slave_single(dmaengine_buffer->chan,
+		block->phys_addr, block->bytes_used, DMA_DEV_TO_MEM,
+		DMA_PREP_INTERRUPT);
+	if (!desc)
+		return -ENOMEM;
+
+	desc->callback = iio_dmaengine_buffer_block_done;
+	desc->callback_param = block;
+
+	cookie = dmaengine_submit(desc);
+	if (dma_submit_error(cookie))
+		return dma_submit_error(cookie);
+
+	spin_lock_irq(&dmaengine_buffer->queue.list_lock);
+	list_add_tail(&block->head, &dmaengine_buffer->active);
+	spin_unlock_irq(&dmaengine_buffer->queue.list_lock);
+
+	dma_async_issue_pending(dmaengine_buffer->chan);
+
+	return 0;
+}
+
+static void iio_dmaengine_buffer_abort(struct iio_dma_buffer_queue *queue)
+{
+	struct dmaengine_buffer *dmaengine_buffer =
+		iio_buffer_to_dmaengine_buffer(&queue->buffer);
+
+	dmaengine_terminate_all(dmaengine_buffer->chan);
+	/* FIXME: There is a slight chance of a race condition here.
+	 * dmaengine_terminate_all() does not guarantee that all transfer
+	 * callbacks have finished running. Need to introduce a
+	 * dmaengine_terminate_all_sync().
+	 */
+	iio_dma_buffer_block_list_abort(queue, &dmaengine_buffer->active);
+}
+
+static void iio_dmaengine_buffer_release(struct iio_buffer *buf)
+{
+	struct dmaengine_buffer *dmaengine_buffer =
+		iio_buffer_to_dmaengine_buffer(buf);
+
+	iio_dma_buffer_release(&dmaengine_buffer->queue);
+	kfree(dmaengine_buffer);
+}
+
+static const struct iio_buffer_access_funcs iio_dmaengine_buffer_ops = {
+	.read_first_n = iio_dma_buffer_read,
+	.set_bytes_per_datum = iio_dma_buffer_set_bytes_per_datum,
+	.set_length = iio_dma_buffer_set_length,
+	.request_update = iio_dma_buffer_request_update,
+	.enable = iio_dma_buffer_enable,
+	.disable = iio_dma_buffer_disable,
+	.data_available = iio_dma_buffer_data_available,
+	.release = iio_dmaengine_buffer_release,
+
+	.modes = INDIO_BUFFER_HARDWARE,
+	.flags = INDIO_BUFFER_FLAG_FIXED_WATERMARK,
+};
+
+static const struct iio_dma_buffer_ops iio_dmaengine_default_ops = {
+	.submit = iio_dmaengine_buffer_submit_block,
+	.abort = iio_dmaengine_buffer_abort,
+};
+
+/**
+ * iio_dmaengine_buffer_alloc() - Allocate new buffer which uses DMAengine
+ * @dev: Parent device for the buffer
+ * @channel: DMA channel name, typically "rx".
+ *
+ * This allocates a new IIO buffer which internally uses the DMAengine framework
+ * to perform its transfers. The parent device will be used to request the DMA
+ * channel.
+ *
+ * Once done using the buffer iio_dmaengine_buffer_free() should be used to
+ * release it.
+ */
+struct iio_buffer *iio_dmaengine_buffer_alloc(struct device *dev,
+	const char *channel)
+{
+	struct dmaengine_buffer *dmaengine_buffer;
+	unsigned int width, src_width, dest_width;
+	struct dma_slave_caps caps;
+	struct dma_chan *chan;
+	int ret;
+
+	dmaengine_buffer = kzalloc(sizeof(*dmaengine_buffer), GFP_KERNEL);
+	if (!dmaengine_buffer)
+		return ERR_PTR(-ENOMEM);
+
+	chan = dma_request_slave_channel_reason(dev, channel);
+	if (IS_ERR(chan)) {
+		ret = PTR_ERR(chan);
+		goto err_free;
+	}
+
+	ret = dma_get_slave_caps(chan, &caps);
+	if (ret < 0)
+		goto err_free;
+
+	/* Needs to be aligned to the maximum of the minimums */
+	if (caps.src_addr_widths)
+		src_width = __ffs(caps.src_addr_widths);
+	else
+		src_width = 1;
+	if (caps.dst_addr_widths)
+		dest_width = __ffs(caps.dst_addr_widths);
+	else
+		dest_width = 1;
+	width = max(src_width, dest_width);
+
+	INIT_LIST_HEAD(&dmaengine_buffer->active);
+	dmaengine_buffer->chan = chan;
+	dmaengine_buffer->align = width;
+	dmaengine_buffer->max_size = dma_get_max_seg_size(chan->device->dev);
+
+	iio_dma_buffer_init(&dmaengine_buffer->queue, chan->device->dev,
+		&iio_dmaengine_default_ops);
+
+	dmaengine_buffer->queue.buffer.access = &iio_dmaengine_buffer_ops;
+
+	return &dmaengine_buffer->queue.buffer;
+
+err_free:
+	kfree(dmaengine_buffer);
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL(iio_dmaengine_buffer_alloc);
+
+/**
+ * iio_dmaengine_buffer_free() - Free dmaengine buffer
+ * @buffer: Buffer to free
+ *
+ * Frees a buffer previously allocated with iio_dmaengine_buffer_alloc().
+ */
+void iio_dmaengine_buffer_free(struct iio_buffer *buffer)
+{
+	struct dmaengine_buffer *dmaengine_buffer =
+		iio_buffer_to_dmaengine_buffer(buffer);
+
+	iio_dma_buffer_exit(&dmaengine_buffer->queue);
+	dma_release_channel(dmaengine_buffer->chan);
+
+	iio_buffer_put(buffer);
+}
+EXPORT_SYMBOL_GPL(iio_dmaengine_buffer_free);
diff --git a/include/linux/iio/buffer-dmaengine.h b/include/linux/iio/buffer-dmaengine.h
new file mode 100644
index 0000000..5dcddf4
--- /dev/null
+++ b/include/linux/iio/buffer-dmaengine.h
@@ -0,0 +1,18 @@
+/*
+ * Copyright 2014-2015 Analog Devices Inc.
+ *  Author: Lars-Peter Clausen <lars@metafoo.de>
+ *
+ * Licensed under the GPL-2 or later.
+ */
+
+#ifndef __IIO_DMAENGINE_H__
+#define __IIO_DMAENGINE_H__
+
+struct iio_buffer;
+struct device;
+
+struct iio_buffer *iio_dmaengine_buffer_alloc(struct device *dev,
+	const char *channel);
+void iio_dmaengine_buffer_free(struct iio_buffer *buffer);
+
+#endif
-- 
2.1.4

Re: [PATCH v3 1/6] iio: Set device watermark based on watermark of all attached buffers

[Jonathan Cameron]

On 13/10/15 17:10, Lars-Peter Clausen wrote:
> Currently the watermark of the device is only set based on the watermark
> that is set for the user space buffer. This doesn't consider the watermarks
> set on any attached in-kernel buffers.
> 
> Change this so that the watermark of the device should be the minimum of
> the watermarks over all attached buffers.
> 
> Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
Applied to the togreg branch of iio.git - initially pushed out as testing.

These have missed the coming merge window, but in someways I'm happier having
these have a very long bake in linux-next (and hopefully some user get added
before they mainline!)

Jonathan
> ---
>  drivers/iio/industrialio-buffer.c | 14 ++++++++++----
>  1 file changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/iio/industrialio-buffer.c b/drivers/iio/industrialio-buffer.c
> index d7e908a..7340922 100644
> --- a/drivers/iio/industrialio-buffer.c
> +++ b/drivers/iio/industrialio-buffer.c
> @@ -610,6 +610,7 @@ static void iio_free_scan_mask(struct iio_dev *indio_dev,
>  
>  struct iio_device_config {
>  	unsigned int mode;
> +	unsigned int watermark;
>  	const unsigned long *scan_mask;
>  	unsigned int scan_bytes;
>  	bool scan_timestamp;
> @@ -642,10 +643,14 @@ static int iio_verify_update(struct iio_dev *indio_dev,
>  		if (buffer == remove_buffer)
>  			continue;
>  		modes &= buffer->access->modes;
> +		config->watermark = min(config->watermark, buffer->watermark);
>  	}
>  
> -	if (insert_buffer)
> +	if (insert_buffer) {
>  		modes &= insert_buffer->access->modes;
> +		config->watermark = min(config->watermark,
> +			insert_buffer->watermark);
> +	}
>  
>  	/* Definitely possible for devices to support both of these. */
>  	if ((modes & INDIO_BUFFER_TRIGGERED) && indio_dev->trig) {
> @@ -743,6 +748,10 @@ static int iio_enable_buffers(struct iio_dev *indio_dev,
>  		}
>  	}
>  
> +	if (indio_dev->info->hwfifo_set_watermark)
> +		indio_dev->info->hwfifo_set_watermark(indio_dev,
> +			config->watermark);
> +
>  	indio_dev->currentmode = config->mode;
>  
>  	if (indio_dev->setup_ops->postenable) {
> @@ -974,9 +983,6 @@ static ssize_t iio_buffer_store_watermark(struct device *dev,
>  	}
>  
>  	buffer->watermark = val;
> -
> -	if (indio_dev->info->hwfifo_set_watermark)
> -		indio_dev->info->hwfifo_set_watermark(indio_dev, val);
>  out:
>  	mutex_unlock(&indio_dev->mlock);
>  
> 

Re: [PATCH v3 2/6] iio:iio_buffer_init(): Only set watermark if not already set

[Jonathan Cameron]

On 13/10/15 17:10, Lars-Peter Clausen wrote:
> Only initialize the watermark field if it is still 0. This allows drivers
> to provide a custom default watermark value. E.g. some driver might have a
> fixed watermark or can only support watermarks within a certain range and
> the initial value for the watermark should be within this range.
> 
> Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
Applied.
> ---
>  drivers/iio/industrialio-buffer.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iio/industrialio-buffer.c b/drivers/iio/industrialio-buffer.c
> index 7340922..5f2c8c8 100644
> --- a/drivers/iio/industrialio-buffer.c
> +++ b/drivers/iio/industrialio-buffer.c
> @@ -193,7 +193,8 @@ void iio_buffer_init(struct iio_buffer *buffer)
>  	INIT_LIST_HEAD(&buffer->buffer_list);
>  	init_waitqueue_head(&buffer->pollq);
>  	kref_init(&buffer->ref);
> -	buffer->watermark = 1;
> +	if (!buffer->watermark)
> +		buffer->watermark = 1;
>  }
>  EXPORT_SYMBOL(iio_buffer_init);
>  
> 

Re: [PATCH v3 3/6] iio: Add support for indicating fixed watermarks

[Jonathan Cameron]

On 13/10/15 17:10, Lars-Peter Clausen wrote:
> For buffers which have a fixed wake-up watermark the watermark attribute
> should be read-only. Add a new FIXED_WATERMARK flag to the
> struct iio_buffer_access_funcs, which can be set by a buffer
> implementation.
> 
> Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
Applied.
> ---
>  drivers/iio/industrialio-buffer.c | 5 +++++
>  include/linux/iio/buffer.h        | 8 ++++++++
>  2 files changed, 13 insertions(+)
> 
> diff --git a/drivers/iio/industrialio-buffer.c b/drivers/iio/industrialio-buffer.c
> index 5f2c8c8..98a6447 100644
> --- a/drivers/iio/industrialio-buffer.c
> +++ b/drivers/iio/industrialio-buffer.c
> @@ -998,6 +998,8 @@ static DEVICE_ATTR(enable, S_IRUGO | S_IWUSR,
>  		   iio_buffer_show_enable, iio_buffer_store_enable);
>  static DEVICE_ATTR(watermark, S_IRUGO | S_IWUSR,
>  		   iio_buffer_show_watermark, iio_buffer_store_watermark);
> +static struct device_attribute dev_attr_watermark_ro = __ATTR(watermark,
> +	S_IRUGO, iio_buffer_show_watermark, NULL);
>  
>  static struct attribute *iio_buffer_attrs[] = {
>  	&dev_attr_length.attr,
> @@ -1040,6 +1042,9 @@ int iio_buffer_alloc_sysfs_and_mask(struct iio_dev *indio_dev)
>  	if (!buffer->access->set_length)
>  		attr[0] = &dev_attr_length_ro.attr;
>  
> +	if (buffer->access->flags & INDIO_BUFFER_FLAG_FIXED_WATERMARK)
> +		attr[2] = &dev_attr_watermark_ro.attr;
> +
>  	if (buffer->attrs)
>  		memcpy(&attr[ARRAY_SIZE(iio_buffer_attrs)], buffer->attrs,
>  		       sizeof(struct attribute *) * attrcount);
> diff --git a/include/linux/iio/buffer.h b/include/linux/iio/buffer.h
> index 1600c55..4d99a53 100644
> --- a/include/linux/iio/buffer.h
> +++ b/include/linux/iio/buffer.h
> @@ -18,6 +18,12 @@
>  struct iio_buffer;
>  
>  /**
> + * INDIO_BUFFER_FLAG_FIXED_WATERMARK - Watermark level of the buffer can not be
> + *   configured. It has a fixed value which will be buffer specific.
> + */
> +#define INDIO_BUFFER_FLAG_FIXED_WATERMARK BIT(0)
> +
> +/**
>   * struct iio_buffer_access_funcs - access functions for buffers.
>   * @store_to:		actually store stuff to the buffer
>   * @read_first_n:	try to get a specified number of bytes (must exist)
> @@ -30,6 +36,7 @@ struct iio_buffer;
>   * @release:		called when the last reference to the buffer is dropped,
>   *			should free all resources allocated by the buffer.
>   * @modes:		Supported operating modes by this buffer type
> + * @flags:		A bitmask combination of INDIO_BUFFER_FLAG_*
>   *
>   * The purpose of this structure is to make the buffer element
>   * modular as event for a given driver, different usecases may require
> @@ -54,6 +61,7 @@ struct iio_buffer_access_funcs {
>  	void (*release)(struct iio_buffer *buffer);
>  
>  	unsigned int modes;
> +	unsigned int flags;
>  };
>  
>  /**
> 

Re: [PATCH v3 4/6] iio: Add buffer enable/disable callbacks

[Jonathan Cameron]

On 13/10/15 17:10, Lars-Peter Clausen wrote:
> This patch adds a enable and disable callback that is called when the
> buffer is enabled/disabled. This can be used by buffer implementations that
> need to do some setup or teardown work. E.g. a DMA based buffer can use
> this to start/stop the DMA transfer.
> 
> Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
Applied.
> ---
>  drivers/iio/industrialio-buffer.c | 36 +++++++++++++++++++++++++++++++++++-
>  include/linux/iio/buffer.h        |  8 ++++++++
>  2 files changed, 43 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iio/industrialio-buffer.c b/drivers/iio/industrialio-buffer.c
> index 98a6447..a4b164a 100644
> --- a/drivers/iio/industrialio-buffer.c
> +++ b/drivers/iio/industrialio-buffer.c
> @@ -568,6 +568,22 @@ static void iio_buffer_deactivate_all(struct iio_dev *indio_dev)
>  		iio_buffer_deactivate(buffer);
>  }
>  
> +static int iio_buffer_enable(struct iio_buffer *buffer,
> +	struct iio_dev *indio_dev)
> +{
> +	if (!buffer->access->enable)
> +		return 0;
> +	return buffer->access->enable(buffer, indio_dev);
> +}
> +
> +static int iio_buffer_disable(struct iio_buffer *buffer,
> +	struct iio_dev *indio_dev)
> +{
> +	if (!buffer->access->disable)
> +		return 0;
> +	return buffer->access->disable(buffer, indio_dev);
> +}
> +
>  static void iio_buffer_update_bytes_per_datum(struct iio_dev *indio_dev,
>  	struct iio_buffer *buffer)
>  {
> @@ -719,6 +735,7 @@ static int iio_verify_update(struct iio_dev *indio_dev,
>  static int iio_enable_buffers(struct iio_dev *indio_dev,
>  	struct iio_device_config *config)
>  {
> +	struct iio_buffer *buffer;
>  	int ret;
>  
>  	indio_dev->active_scan_mask = config->scan_mask;
> @@ -753,6 +770,12 @@ static int iio_enable_buffers(struct iio_dev *indio_dev,
>  		indio_dev->info->hwfifo_set_watermark(indio_dev,
>  			config->watermark);
>  
> +	list_for_each_entry(buffer, &indio_dev->buffer_list, buffer_list) {
> +		ret = iio_buffer_enable(buffer, indio_dev);
> +		if (ret)
> +			goto err_disable_buffers;
> +	}
> +
>  	indio_dev->currentmode = config->mode;
>  
>  	if (indio_dev->setup_ops->postenable) {
> @@ -760,12 +783,16 @@ static int iio_enable_buffers(struct iio_dev *indio_dev,
>  		if (ret) {
>  			dev_dbg(&indio_dev->dev,
>  			       "Buffer not started: postenable failed (%d)\n", ret);
> -			goto err_run_postdisable;
> +			goto err_disable_buffers;
>  		}
>  	}
>  
>  	return 0;
>  
> +err_disable_buffers:
> +	list_for_each_entry_continue_reverse(buffer, &indio_dev->buffer_list,
> +					     buffer_list)
> +		iio_buffer_disable(buffer, indio_dev);
>  err_run_postdisable:
>  	indio_dev->currentmode = INDIO_DIRECT_MODE;
>  	if (indio_dev->setup_ops->postdisable)
> @@ -778,6 +805,7 @@ err_undo_config:
>  
>  static int iio_disable_buffers(struct iio_dev *indio_dev)
>  {
> +	struct iio_buffer *buffer;
>  	int ret = 0;
>  	int ret2;
>  
> @@ -798,6 +826,12 @@ static int iio_disable_buffers(struct iio_dev *indio_dev)
>  			ret = ret2;
>  	}
>  
> +	list_for_each_entry(buffer, &indio_dev->buffer_list, buffer_list) {
> +		ret2 = iio_buffer_disable(buffer, indio_dev);
> +		if (ret2 && !ret)
> +			ret = ret2;
> +	}
> +
>  	indio_dev->currentmode = INDIO_DIRECT_MODE;
>  
>  	if (indio_dev->setup_ops->postdisable) {
> diff --git a/include/linux/iio/buffer.h b/include/linux/iio/buffer.h
> index 4d99a53..2ec3ad5 100644
> --- a/include/linux/iio/buffer.h
> +++ b/include/linux/iio/buffer.h
> @@ -33,6 +33,11 @@ struct iio_buffer;
>   *			storage.
>   * @set_bytes_per_datum:set number of bytes per datum
>   * @set_length:		set number of datums in buffer
> + * @enable:             called if the buffer is attached to a device and the
> + *                      device starts sampling. Calls are balanced with
> + *                      @disable.
> + * @disable:            called if the buffer is attached to a device and the
> + *                      device stops sampling. Calles are balanced with @enable.
>   * @release:		called when the last reference to the buffer is dropped,
>   *			should free all resources allocated by the buffer.
>   * @modes:		Supported operating modes by this buffer type
> @@ -58,6 +63,9 @@ struct iio_buffer_access_funcs {
>  	int (*set_bytes_per_datum)(struct iio_buffer *buffer, size_t bpd);
>  	int (*set_length)(struct iio_buffer *buffer, int length);
>  
> +	int (*enable)(struct iio_buffer *buffer, struct iio_dev *indio_dev);
> +	int (*disable)(struct iio_buffer *buffer, struct iio_dev *indio_dev);
> +
>  	void (*release)(struct iio_buffer *buffer);
>  
>  	unsigned int modes;
> 

Re: [PATCH v3 5/6] iio: Add generic DMA buffer infrastructure

[Jonathan Cameron]

On 13/10/15 17:10, Lars-Peter Clausen wrote:
> The traditional approach used in IIO to implement buffered capture requires
> the generation of at least one interrupt per sample. In the interrupt
> handler the driver reads the sample from the device and copies it to a
> software buffer. This approach has a rather large per sample overhead
> associated with it. And while it works fine for samplerates in the range of
> up to 1000 samples per second it starts to consume a rather large share of
> the available CPU processing time once we go beyond that, this is
> especially true on an embedded system with limited processing power. The
> regular interrupt also causes increased power consumption by not allowing
> the hardware into deeper sleep states, which is something that becomes more
> and more important on mobile battery powered devices.
> 
> And while the recently added watermark support mitigates some of the issues
> by allowing the device to generate interrupts at a rate lower than the data
> output rate, this still requires a storage buffer inside the device and
> even if it exists it is only a few 100 samples deep at most.
> 
> DMA support on the other hand allows to capture multiple millions or even
> more samples without any CPU interaction. This allows the CPU to either go
> to sleep for longer periods or focus on other tasks which increases overall
> system performance and power consumption. In addition to that some devices
> might not even offer a way to read the data other than using DMA, which
> makes DMA mandatory to use for them.
> 
> The tasks involved in implementing a DMA buffer can be divided into two
> categories. The first category is memory buffer management (allocation,
> mapping, etc.) and hooking this up the IIO buffer callbacks like read(),
> enable(), disable(), etc. The second category of tasks is to setup the
> DMA hardware and manage the DMA transfers. Tasks from the first category
> will be very similar for all IIO drivers supporting DMA buffers, while the
> tasks from the second category will be hardware specific.
> 
> This patch implements a generic infrastructure that take care of the former
> tasks. It provides a set of functions that implement the standard IIO
> buffer iio_buffer_access_funcs callbacks. These can either be used as is or
> be overloaded and augmented with driver specific code where necessary.
> 
> For the DMA buffer support infrastructure that is introduced in this series
> sample data is grouped by so called blocks. A block is the basic unit at
> which data is exchanged between the application and the hardware. The
> application is responsible for allocating the memory associated with the
> block and then passes the block to the hardware. When the hardware has
> captured the amount of samples equal to size of a block it will notify the
> application, which can then read the data from the block and process it.
> The block size can freely chosen (within the constraints of the hardware).
> This allows to make a trade-off between latency and management overhead.
> The larger the block size the lower the per sample overhead but the latency
> between when the data was captured and when the application will be able to
> access it increases, in a similar way smaller block sizes have a larger per
> sample management overhead but a lower latency. The ideal block size thus
> depends on system and application requirements.
> 
> For the time being the infrastructure only implements a simple double
> buffered scheme which allocates two blocks each with half the size of the
> configured buffer size. This provides basic support for capturing
> continuous uninterrupted data over the existing file-IO ABI. Future
> extensions to the DMA buffer infrastructure will give applications a more
> fine grained control over how many blocks are allocated and the size of
> each block. But this requires userspace ABI additions which are
> intentionally not part of this patch and will be added separately.
> 
> Tasks of the second category need to be implemented by a device specific
> driver. They can be hooked up into the generic infrastructure using two
> simple callbacks, submit() and abort().
> 
> The submit() callback is used to schedule DMA transfers for blocks. Once a
> DMA transfer has been completed it is expected that the buffer driver calls
> iio_dma_buffer_block_done() to notify. The abort() callback is used for
> stopping all pending and active DMA transfers when the buffer is disabled.
> 
> Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
I like this stuff a lot.  Good to have it in and am looking forward to seeing
just how many devices can actually support this stuff.

Anyhow,

Applied.
> ---
>  drivers/iio/buffer/Kconfig                   |   9 +
>  drivers/iio/buffer/Makefile                  |   1 +
>  drivers/iio/buffer/industrialio-buffer-dma.c | 683 +++++++++++++++++++++++++++
>  include/linux/iio/buffer-dma.h               | 152 ++++++
>  4 files changed, 845 insertions(+)
>  create mode 100644 drivers/iio/buffer/industrialio-buffer-dma.c
>  create mode 100644 include/linux/iio/buffer-dma.h
> 
> diff --git a/drivers/iio/buffer/Kconfig b/drivers/iio/buffer/Kconfig
> index 0a7b2fd..b2fda1a 100644
> --- a/drivers/iio/buffer/Kconfig
> +++ b/drivers/iio/buffer/Kconfig
> @@ -9,6 +9,15 @@ config IIO_BUFFER_CB
>  	  Should be selected by any drivers that do in-kernel push
>  	  usage.  That is, those where the data is pushed to the consumer.
>  
> +config IIO_BUFFER_DMA
> +	tristate
> +	help
> +	  Provides the generic IIO DMA buffer infrastructure that can be used by
> +	  drivers for devices with DMA support to implement the IIO buffer.
> +
> +	  Should be selected by drivers that want to use the generic DMA buffer
> +	  infrastructure.
> +
>  config IIO_KFIFO_BUF
>  	tristate "Industrial I/O buffering based on kfifo"
>  	help
> diff --git a/drivers/iio/buffer/Makefile b/drivers/iio/buffer/Makefile
> index 4d193b9..bda3f11 100644
> --- a/drivers/iio/buffer/Makefile
> +++ b/drivers/iio/buffer/Makefile
> @@ -4,5 +4,6 @@
>  
>  # When adding new entries keep the list in alphabetical order
>  obj-$(CONFIG_IIO_BUFFER_CB) += industrialio-buffer-cb.o
> +obj-$(CONFIG_IIO_BUFFER_DMA) += industrialio-buffer-dma.o
>  obj-$(CONFIG_IIO_TRIGGERED_BUFFER) += industrialio-triggered-buffer.o
>  obj-$(CONFIG_IIO_KFIFO_BUF) += kfifo_buf.o
> diff --git a/drivers/iio/buffer/industrialio-buffer-dma.c b/drivers/iio/buffer/industrialio-buffer-dma.c
> new file mode 100644
> index 0000000..212cbed
> --- /dev/null
> +++ b/drivers/iio/buffer/industrialio-buffer-dma.c
> @@ -0,0 +1,683 @@
> +/*
> + * Copyright 2013-2015 Analog Devices Inc.
> + *  Author: Lars-Peter Clausen <lars@metafoo.de>
> + *
> + * Licensed under the GPL-2.
> + */
> +
> +#include <linux/slab.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/device.h>
> +#include <linux/workqueue.h>
> +#include <linux/mutex.h>
> +#include <linux/sched.h>
> +#include <linux/poll.h>
> +#include <linux/iio/buffer.h>
> +#include <linux/iio/buffer-dma.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/sizes.h>
> +
> +/*
> + * For DMA buffers the storage is sub-divided into so called blocks. Each block
> + * has its own memory buffer. The size of the block is the granularity at which
> + * memory is exchanged between the hardware and the application. Increasing the
> + * basic unit of data exchange from one sample to one block decreases the
> + * management overhead that is associated with each sample. E.g. if we say the
> + * management overhead for one exchange is x and the unit of exchange is one
> + * sample the overhead will be x for each sample. Whereas when using a block
> + * which contains n samples the overhead per sample is reduced to x/n. This
> + * allows to achieve much higher samplerates than what can be sustained with
> + * the one sample approach.
> + *
> + * Blocks are exchanged between the DMA controller and the application via the
> + * means of two queues. The incoming queue and the outgoing queue. Blocks on the
> + * incoming queue are waiting for the DMA controller to pick them up and fill
> + * them with data. Block on the outgoing queue have been filled with data and
> + * are waiting for the application to dequeue them and read the data.
> + *
> + * A block can be in one of the following states:
> + *  * Owned by the application. In this state the application can read data from
> + *    the block.
> + *  * On the incoming list: Blocks on the incoming list are queued up to be
> + *    processed by the DMA controller.
> + *  * Owned by the DMA controller: The DMA controller is processing the block
> + *    and filling it with data.
> + *  * On the outgoing list: Blocks on the outgoing list have been successfully
> + *    processed by the DMA controller and contain data. They can be dequeued by
> + *    the application.
> + *  * Dead: A block that is dead has been marked as to be freed. It might still
> + *    be owned by either the application or the DMA controller at the moment.
> + *    But once they are done processing it instead of going to either the
> + *    incoming or outgoing queue the block will be freed.
> + *
> + * In addition to this blocks are reference counted and the memory associated
> + * with both the block structure as well as the storage memory for the block
> + * will be freed when the last reference to the block is dropped. This means a
> + * block must not be accessed without holding a reference.
> + *
> + * The iio_dma_buffer implementation provides a generic infrastructure for
> + * managing the blocks.
> + *
> + * A driver for a specific piece of hardware that has DMA capabilities need to
> + * implement the submit() callback from the iio_dma_buffer_ops structure. This
> + * callback is supposed to initiate the DMA transfer copying data from the
> + * converter to the memory region of the block. Once the DMA transfer has been
> + * completed the driver must call iio_dma_buffer_block_done() for the completed
> + * block.
> + *
> + * Prior to this it must set the bytes_used field of the block contains
> + * the actual number of bytes in the buffer. Typically this will be equal to the
> + * size of the block, but if the DMA hardware has certain alignment requirements
> + * for the transfer length it might choose to use less than the full size. In
> + * either case it is expected that bytes_used is a multiple of the bytes per
> + * datum, i.e. the block must not contain partial samples.
> + *
> + * The driver must call iio_dma_buffer_block_done() for each block it has
> + * received through its submit_block() callback, even if it does not actually
> + * perform a DMA transfer for the block, e.g. because the buffer was disabled
> + * before the block transfer was started. In this case it should set bytes_used
> + * to 0.
> + *
> + * In addition it is recommended that a driver implements the abort() callback.
> + * It will be called when the buffer is disabled and can be used to cancel
> + * pending and stop active transfers.
> + *
> + * The specific driver implementation should use the default callback
> + * implementations provided by this module for the iio_buffer_access_funcs
> + * struct. It may overload some callbacks with custom variants if the hardware
> + * has special requirements that are not handled by the generic functions. If a
> + * driver chooses to overload a callback it has to ensure that the generic
> + * callback is called from within the custom callback.
> + */
> +
> +static void iio_buffer_block_release(struct kref *kref)
> +{
> +	struct iio_dma_buffer_block *block = container_of(kref,
> +		struct iio_dma_buffer_block, kref);
> +
> +	WARN_ON(block->state != IIO_BLOCK_STATE_DEAD);
> +
> +	dma_free_coherent(block->queue->dev, PAGE_ALIGN(block->size),
> +					block->vaddr, block->phys_addr);
> +
> +	iio_buffer_put(&block->queue->buffer);
> +	kfree(block);
> +}
> +
> +static void iio_buffer_block_get(struct iio_dma_buffer_block *block)
> +{
> +	kref_get(&block->kref);
> +}
> +
> +static void iio_buffer_block_put(struct iio_dma_buffer_block *block)
> +{
> +	kref_put(&block->kref, iio_buffer_block_release);
> +}
> +
> +/*
> + * dma_free_coherent can sleep, hence we need to take some special care to be
> + * able to drop a reference from an atomic context.
> + */
> +static LIST_HEAD(iio_dma_buffer_dead_blocks);
> +static DEFINE_SPINLOCK(iio_dma_buffer_dead_blocks_lock);
> +
> +static void iio_dma_buffer_cleanup_worker(struct work_struct *work)
> +{
> +	struct iio_dma_buffer_block *block, *_block;
> +	LIST_HEAD(block_list);
> +
> +	spin_lock_irq(&iio_dma_buffer_dead_blocks_lock);
> +	list_splice_tail_init(&iio_dma_buffer_dead_blocks, &block_list);
> +	spin_unlock_irq(&iio_dma_buffer_dead_blocks_lock);
> +
> +	list_for_each_entry_safe(block, _block, &block_list, head)
> +		iio_buffer_block_release(&block->kref);
> +}
> +static DECLARE_WORK(iio_dma_buffer_cleanup_work, iio_dma_buffer_cleanup_worker);
> +
> +static void iio_buffer_block_release_atomic(struct kref *kref)
> +{
> +	struct iio_dma_buffer_block *block;
> +	unsigned long flags;
> +
> +	block = container_of(kref, struct iio_dma_buffer_block, kref);
> +
> +	spin_lock_irqsave(&iio_dma_buffer_dead_blocks_lock, flags);
> +	list_add_tail(&block->head, &iio_dma_buffer_dead_blocks);
> +	spin_unlock_irqrestore(&iio_dma_buffer_dead_blocks_lock, flags);
> +
> +	schedule_work(&iio_dma_buffer_cleanup_work);
> +}
> +
> +/*
> + * Version of iio_buffer_block_put() that can be called from atomic context
> + */
> +static void iio_buffer_block_put_atomic(struct iio_dma_buffer_block *block)
> +{
> +	kref_put(&block->kref, iio_buffer_block_release_atomic);
> +}
> +
> +static struct iio_dma_buffer_queue *iio_buffer_to_queue(struct iio_buffer *buf)
> +{
> +	return container_of(buf, struct iio_dma_buffer_queue, buffer);
> +}
> +
> +static struct iio_dma_buffer_block *iio_dma_buffer_alloc_block(
> +	struct iio_dma_buffer_queue *queue, size_t size)
> +{
> +	struct iio_dma_buffer_block *block;
> +
> +	block = kzalloc(sizeof(*block), GFP_KERNEL);
> +	if (!block)
> +		return NULL;
> +
> +	block->vaddr = dma_alloc_coherent(queue->dev, PAGE_ALIGN(size),
> +		&block->phys_addr, GFP_KERNEL);
> +	if (!block->vaddr) {
> +		kfree(block);
> +		return NULL;
> +	}
> +
> +	block->size = size;
> +	block->state = IIO_BLOCK_STATE_DEQUEUED;
> +	block->queue = queue;
> +	INIT_LIST_HEAD(&block->head);
> +	kref_init(&block->kref);
> +
> +	iio_buffer_get(&queue->buffer);
> +
> +	return block;
> +}
> +
> +static void _iio_dma_buffer_block_done(struct iio_dma_buffer_block *block)
> +{
> +	struct iio_dma_buffer_queue *queue = block->queue;
> +
> +	/*
> +	 * The buffer has already been freed by the application, just drop the
> +	 * reference.
> +	 */
> +	if (block->state != IIO_BLOCK_STATE_DEAD) {
> +		block->state = IIO_BLOCK_STATE_DONE;
> +		list_add_tail(&block->head, &queue->outgoing);
> +	}
> +}
> +
> +/**
> + * iio_dma_buffer_block_done() - Indicate that a block has been completed
> + * @block: The completed block
> + *
> + * Should be called when the DMA controller has finished handling the block to
> + * pass back ownership of the block to the queue.
> + */
> +void iio_dma_buffer_block_done(struct iio_dma_buffer_block *block)
> +{
> +	struct iio_dma_buffer_queue *queue = block->queue;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&queue->list_lock, flags);
> +	_iio_dma_buffer_block_done(block);
> +	spin_unlock_irqrestore(&queue->list_lock, flags);
> +
> +	iio_buffer_block_put_atomic(block);
> +	wake_up_interruptible_poll(&queue->buffer.pollq, POLLIN | POLLRDNORM);
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_block_done);
> +
> +/**
> + * iio_dma_buffer_block_list_abort() - Indicate that a list block has been
> + *   aborted
> + * @queue: Queue for which to complete blocks.
> + * @list: List of aborted blocks. All blocks in this list must be from @queue.
> + *
> + * Typically called from the abort() callback after the DMA controller has been
> + * stopped. This will set bytes_used to 0 for each block in the list and then
> + * hand the blocks back to the queue.
> + */
> +void iio_dma_buffer_block_list_abort(struct iio_dma_buffer_queue *queue,
> +	struct list_head *list)
> +{
> +	struct iio_dma_buffer_block *block, *_block;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&queue->list_lock, flags);
> +	list_for_each_entry_safe(block, _block, list, head) {
> +		list_del(&block->head);
> +		block->bytes_used = 0;
> +		_iio_dma_buffer_block_done(block);
> +		iio_buffer_block_put_atomic(block);
> +	}
> +	spin_unlock_irqrestore(&queue->list_lock, flags);
> +
> +	wake_up_interruptible_poll(&queue->buffer.pollq, POLLIN | POLLRDNORM);
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_block_list_abort);
> +
> +static bool iio_dma_block_reusable(struct iio_dma_buffer_block *block)
> +{
> +	/*
> +	 * If the core owns the block it can be re-used. This should be the
> +	 * default case when enabling the buffer, unless the DMA controller does
> +	 * not support abort and has not given back the block yet.
> +	 */
> +	switch (block->state) {
> +	case IIO_BLOCK_STATE_DEQUEUED:
> +	case IIO_BLOCK_STATE_QUEUED:
> +	case IIO_BLOCK_STATE_DONE:
> +		return true;
> +	default:
> +		return false;
> +	}
> +}
> +
> +/**
> + * iio_dma_buffer_request_update() - DMA buffer request_update callback
> + * @buffer: The buffer which to request an update
> + *
> + * Should be used as the iio_dma_buffer_request_update() callback for
> + * iio_buffer_access_ops struct for DMA buffers.
> + */
> +int iio_dma_buffer_request_update(struct iio_buffer *buffer)
> +{
> +	struct iio_dma_buffer_queue *queue = iio_buffer_to_queue(buffer);
> +	struct iio_dma_buffer_block *block;
> +	bool try_reuse = false;
> +	size_t size;
> +	int ret = 0;
> +	int i;
> +
> +	/*
> +	 * Split the buffer into two even parts. This is used as a double
> +	 * buffering scheme with usually one block at a time being used by the
> +	 * DMA and the other one by the application.
> +	 */
> +	size = DIV_ROUND_UP(queue->buffer.bytes_per_datum *
> +		queue->buffer.length, 2);
> +
> +	mutex_lock(&queue->lock);
> +
> +	/* Allocations are page aligned */
> +	if (PAGE_ALIGN(queue->fileio.block_size) == PAGE_ALIGN(size))
> +		try_reuse = true;
> +
> +	queue->fileio.block_size = size;
> +	queue->fileio.active_block = NULL;
> +
> +	spin_lock_irq(&queue->list_lock);
> +	for (i = 0; i < 2; i++) {
> +		block = queue->fileio.blocks[i];
> +
> +		/* If we can't re-use it free it */
> +		if (block && (!iio_dma_block_reusable(block) || !try_reuse))
> +			block->state = IIO_BLOCK_STATE_DEAD;
> +	}
> +
> +	/*
> +	 * At this point all blocks are either owned by the core or marked as
> +	 * dead. This means we can reset the lists without having to fear
> +	 * corrution.
> +	 */
> +	INIT_LIST_HEAD(&queue->outgoing);
> +	spin_unlock_irq(&queue->list_lock);
> +
> +	INIT_LIST_HEAD(&queue->incoming);
> +
> +	for (i = 0; i < 2; i++) {
> +		if (queue->fileio.blocks[i]) {
> +			block = queue->fileio.blocks[i];
> +			if (block->state == IIO_BLOCK_STATE_DEAD) {
> +				/* Could not reuse it */
> +				iio_buffer_block_put(block);
> +				block = NULL;
> +			} else {
> +				block->size = size;
> +			}
> +		} else {
> +			block = NULL;
> +		}
> +
> +		if (!block) {
> +			block = iio_dma_buffer_alloc_block(queue, size);
> +			if (!block) {
> +				ret = -ENOMEM;
> +				goto out_unlock;
> +			}
> +			queue->fileio.blocks[i] = block;
> +		}
> +
> +		block->state = IIO_BLOCK_STATE_QUEUED;
> +		list_add_tail(&block->head, &queue->incoming);
> +	}
> +
> +out_unlock:
> +	mutex_unlock(&queue->lock);
> +
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_request_update);
> +
> +static void iio_dma_buffer_submit_block(struct iio_dma_buffer_queue *queue,
> +	struct iio_dma_buffer_block *block)
> +{
> +	int ret;
> +
> +	/*
> +	 * If the hardware has already been removed we put the block into
> +	 * limbo. It will neither be on the incoming nor outgoing list, nor will
> +	 * it ever complete. It will just wait to be freed eventually.
> +	 */
> +	if (!queue->ops)
> +		return;
> +
> +	block->state = IIO_BLOCK_STATE_ACTIVE;
> +	iio_buffer_block_get(block);
> +	ret = queue->ops->submit(queue, block);
> +	if (ret) {
> +		/*
> +		 * This is a bit of a problem and there is not much we can do
> +		 * other then wait for the buffer to be disabled and re-enabled
> +		 * and try again. But it should not really happen unless we run
> +		 * out of memory or something similar.
> +		 *
> +		 * TODO: Implement support in the IIO core to allow buffers to
> +		 * notify consumers that something went wrong and the buffer
> +		 * should be disabled.
> +		 */
> +		iio_buffer_block_put(block);
> +	}
> +}
> +
> +/**
> + * iio_dma_buffer_enable() - Enable DMA buffer
> + * @buffer: IIO buffer to enable
> + * @indio_dev: IIO device the buffer is attached to
> + *
> + * Needs to be called when the device that the buffer is attached to starts
> + * sampling. Typically should be the iio_buffer_access_ops enable callback.
> + *
> + * This will allocate the DMA buffers and start the DMA transfers.
> + */
> +int iio_dma_buffer_enable(struct iio_buffer *buffer,
> +	struct iio_dev *indio_dev)
> +{
> +	struct iio_dma_buffer_queue *queue = iio_buffer_to_queue(buffer);
> +	struct iio_dma_buffer_block *block, *_block;
> +
> +	mutex_lock(&queue->lock);
> +	queue->active = true;
> +	list_for_each_entry_safe(block, _block, &queue->incoming, head) {
> +		list_del(&block->head);
> +		iio_dma_buffer_submit_block(queue, block);
> +	}
> +	mutex_unlock(&queue->lock);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_enable);
> +
> +/**
> + * iio_dma_buffer_disable() - Disable DMA buffer
> + * @buffer: IIO DMA buffer to disable
> + * @indio_dev: IIO device the buffer is attached to
> + *
> + * Needs to be called when the device that the buffer is attached to stops
> + * sampling. Typically should be the iio_buffer_access_ops disable callback.
> + */
> +int iio_dma_buffer_disable(struct iio_buffer *buffer,
> +	struct iio_dev *indio_dev)
> +{
> +	struct iio_dma_buffer_queue *queue = iio_buffer_to_queue(buffer);
> +
> +	mutex_lock(&queue->lock);
> +	queue->active = false;
> +
> +	if (queue->ops && queue->ops->abort)
> +		queue->ops->abort(queue);
> +	mutex_unlock(&queue->lock);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_disable);
> +
> +static void iio_dma_buffer_enqueue(struct iio_dma_buffer_queue *queue,
> +	struct iio_dma_buffer_block *block)
> +{
> +	if (block->state == IIO_BLOCK_STATE_DEAD) {
> +		iio_buffer_block_put(block);
> +	} else if (queue->active) {
> +		iio_dma_buffer_submit_block(queue, block);
> +	} else {
> +		block->state = IIO_BLOCK_STATE_QUEUED;
> +		list_add_tail(&block->head, &queue->incoming);
> +	}
> +}
> +
> +static struct iio_dma_buffer_block *iio_dma_buffer_dequeue(
> +	struct iio_dma_buffer_queue *queue)
> +{
> +	struct iio_dma_buffer_block *block;
> +
> +	spin_lock_irq(&queue->list_lock);
> +	block = list_first_entry_or_null(&queue->outgoing, struct
> +		iio_dma_buffer_block, head);
> +	if (block != NULL) {
> +		list_del(&block->head);
> +		block->state = IIO_BLOCK_STATE_DEQUEUED;
> +	}
> +	spin_unlock_irq(&queue->list_lock);
> +
> +	return block;
> +}
> +
> +/**
> + * iio_dma_buffer_read() - DMA buffer read callback
> + * @buffer: Buffer to read form
> + * @n: Number of bytes to read
> + * @user_buffer: Userspace buffer to copy the data to
> + *
> + * Should be used as the read_first_n callback for iio_buffer_access_ops
> + * struct for DMA buffers.
> + */
> +int iio_dma_buffer_read(struct iio_buffer *buffer, size_t n,
> +	char __user *user_buffer)
> +{
> +	struct iio_dma_buffer_queue *queue = iio_buffer_to_queue(buffer);
> +	struct iio_dma_buffer_block *block;
> +	int ret;
> +
> +	if (n < buffer->bytes_per_datum)
> +		return -EINVAL;
> +
> +	mutex_lock(&queue->lock);
> +
> +	if (!queue->fileio.active_block) {
> +		block = iio_dma_buffer_dequeue(queue);
> +		if (block == NULL) {
> +			ret = 0;
> +			goto out_unlock;
> +		}
> +		queue->fileio.pos = 0;
> +		queue->fileio.active_block = block;
> +	} else {
> +		block = queue->fileio.active_block;
> +	}
> +
> +	n = rounddown(n, buffer->bytes_per_datum);
> +	if (n > block->bytes_used - queue->fileio.pos)
> +		n = block->bytes_used - queue->fileio.pos;
> +
> +	if (copy_to_user(user_buffer, block->vaddr + queue->fileio.pos, n)) {
> +		ret = -EFAULT;
> +		goto out_unlock;
> +	}
> +
> +	queue->fileio.pos += n;
> +
> +	if (queue->fileio.pos == block->bytes_used) {
> +		queue->fileio.active_block = NULL;
> +		iio_dma_buffer_enqueue(queue, block);
> +	}
> +
> +	ret = n;
> +
> +out_unlock:
> +	mutex_unlock(&queue->lock);
> +
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_read);
> +
> +/**
> + * iio_dma_buffer_data_available() - DMA buffer data_available callback
> + * @buf: Buffer to check for data availability
> + *
> + * Should be used as the data_available callback for iio_buffer_access_ops
> + * struct for DMA buffers.
> + */
> +size_t iio_dma_buffer_data_available(struct iio_buffer *buf)
> +{
> +	struct iio_dma_buffer_queue *queue = iio_buffer_to_queue(buf);
> +	struct iio_dma_buffer_block *block;
> +	size_t data_available = 0;
> +
> +	/*
> +	 * For counting the available bytes we'll use the size of the block not
> +	 * the number of actual bytes available in the block. Otherwise it is
> +	 * possible that we end up with a value that is lower than the watermark
> +	 * but won't increase since all blocks are in use.
> +	 */
> +
> +	mutex_lock(&queue->lock);
> +	if (queue->fileio.active_block)
> +		data_available += queue->fileio.active_block->size;
> +
> +	spin_lock_irq(&queue->list_lock);
> +	list_for_each_entry(block, &queue->outgoing, head)
> +		data_available += block->size;
> +	spin_unlock_irq(&queue->list_lock);
> +	mutex_unlock(&queue->lock);
> +
> +	return data_available;
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_data_available);
> +
> +/**
> + * iio_dma_buffer_set_bytes_per_datum() - DMA buffer set_bytes_per_datum callback
> + * @buffer: Buffer to set the bytes-per-datum for
> + * @bpd: The new bytes-per-datum value
> + *
> + * Should be used as the set_bytes_per_datum callback for iio_buffer_access_ops
> + * struct for DMA buffers.
> + */
> +int iio_dma_buffer_set_bytes_per_datum(struct iio_buffer *buffer, size_t bpd)
> +{
> +	buffer->bytes_per_datum = bpd;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_set_bytes_per_datum);
> +
> +/**
> + * iio_dma_buffer_set_length - DMA buffer set_length callback
> + * @buffer: Buffer to set the length for
> + * @length: The new buffer length
> + *
> + * Should be used as the set_length callback for iio_buffer_access_ops
> + * struct for DMA buffers.
> + */
> +int iio_dma_buffer_set_length(struct iio_buffer *buffer, int length)
> +{
> +	/* Avoid an invalid state */
> +	if (length < 2)
> +		length = 2;
> +	buffer->length = length;
> +	buffer->watermark = length / 2;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_set_length);
> +
> +/**
> + * iio_dma_buffer_init() - Initialize DMA buffer queue
> + * @queue: Buffer to initialize
> + * @dev: DMA device
> + * @ops: DMA buffer queue callback operations
> + *
> + * The DMA device will be used by the queue to do DMA memory allocations. So it
> + * should refer to the device that will perform the DMA to ensure that
> + * allocations are done from a memory region that can be accessed by the device.
> + */
> +int iio_dma_buffer_init(struct iio_dma_buffer_queue *queue,
> +	struct device *dev, const struct iio_dma_buffer_ops *ops)
> +{
> +	iio_buffer_init(&queue->buffer);
> +	queue->buffer.length = PAGE_SIZE;
> +	queue->buffer.watermark = queue->buffer.length / 2;
> +	queue->dev = dev;
> +	queue->ops = ops;
> +
> +	INIT_LIST_HEAD(&queue->incoming);
> +	INIT_LIST_HEAD(&queue->outgoing);
> +
> +	mutex_init(&queue->lock);
> +	spin_lock_init(&queue->list_lock);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_init);
> +
> +/**
> + * iio_dma_buffer_exit() - Cleanup DMA buffer queue
> + * @queue: Buffer to cleanup
> + *
> + * After this function has completed it is safe to free any resources that are
> + * associated with the buffer and are accessed inside the callback operations.
> + */
> +void iio_dma_buffer_exit(struct iio_dma_buffer_queue *queue)
> +{
> +	unsigned int i;
> +
> +	mutex_lock(&queue->lock);
> +
> +	spin_lock_irq(&queue->list_lock);
> +	for (i = 0; i < ARRAY_SIZE(queue->fileio.blocks); i++) {
> +		if (!queue->fileio.blocks[i])
> +			continue;
> +		queue->fileio.blocks[i]->state = IIO_BLOCK_STATE_DEAD;
> +	}
> +	INIT_LIST_HEAD(&queue->outgoing);
> +	spin_unlock_irq(&queue->list_lock);
> +
> +	INIT_LIST_HEAD(&queue->incoming);
> +
> +	for (i = 0; i < ARRAY_SIZE(queue->fileio.blocks); i++) {
> +		if (!queue->fileio.blocks[i])
> +			continue;
> +		iio_buffer_block_put(queue->fileio.blocks[i]);
> +		queue->fileio.blocks[i] = NULL;
> +	}
> +	queue->fileio.active_block = NULL;
> +	queue->ops = NULL;
> +
> +	mutex_unlock(&queue->lock);
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_exit);
> +
> +/**
> + * iio_dma_buffer_release() - Release final buffer resources
> + * @queue: Buffer to release
> + *
> + * Frees resources that can't yet be freed in iio_dma_buffer_exit(). Should be
> + * called in the buffers release callback implementation right before freeing
> + * the memory associated with the buffer.
> + */
> +void iio_dma_buffer_release(struct iio_dma_buffer_queue *queue)
> +{
> +	mutex_destroy(&queue->lock);
> +}
> +EXPORT_SYMBOL_GPL(iio_dma_buffer_release);
> +
> +MODULE_AUTHOR("Lars-Peter Clausen <lars@metafoo.de>");
> +MODULE_DESCRIPTION("DMA buffer for the IIO framework");
> +MODULE_LICENSE("GPL v2");
> diff --git a/include/linux/iio/buffer-dma.h b/include/linux/iio/buffer-dma.h
> new file mode 100644
> index 0000000..767467d
> --- /dev/null
> +++ b/include/linux/iio/buffer-dma.h
> @@ -0,0 +1,152 @@
> +/*
> + * Copyright 2013-2015 Analog Devices Inc.
> + *  Author: Lars-Peter Clausen <lars@metafoo.de>
> + *
> + * Licensed under the GPL-2.
> + */
> +
> +#ifndef __INDUSTRIALIO_DMA_BUFFER_H__
> +#define __INDUSTRIALIO_DMA_BUFFER_H__
> +
> +#include <linux/list.h>
> +#include <linux/kref.h>
> +#include <linux/spinlock.h>
> +#include <linux/mutex.h>
> +#include <linux/iio/buffer.h>
> +
> +struct iio_dma_buffer_queue;
> +struct iio_dma_buffer_ops;
> +struct device;
> +
> +struct iio_buffer_block {
> +	u32 size;
> +	u32 bytes_used;
> +};
> +
> +/**
> + * enum iio_block_state - State of a struct iio_dma_buffer_block
> + * @IIO_BLOCK_STATE_DEQUEUED: Block is not queued
> + * @IIO_BLOCK_STATE_QUEUED: Block is on the incoming queue
> + * @IIO_BLOCK_STATE_ACTIVE: Block is currently being processed by the DMA
> + * @IIO_BLOCK_STATE_DONE: Block is on the outgoing queue
> + * @IIO_BLOCK_STATE_DEAD: Block has been marked as to be freed
> + */
> +enum iio_block_state {
> +	IIO_BLOCK_STATE_DEQUEUED,
> +	IIO_BLOCK_STATE_QUEUED,
> +	IIO_BLOCK_STATE_ACTIVE,
> +	IIO_BLOCK_STATE_DONE,
> +	IIO_BLOCK_STATE_DEAD,
> +};
> +
> +/**
> + * struct iio_dma_buffer_block - IIO buffer block
> + * @head: List head
> + * @size: Total size of the block in bytes
> + * @bytes_used: Number of bytes that contain valid data
> + * @vaddr: Virutal address of the blocks memory
> + * @phys_addr: Physical address of the blocks memory
> + * @queue: Parent DMA buffer queue
> + * @kref: kref used to manage the lifetime of block
> + * @state: Current state of the block
> + */
> +struct iio_dma_buffer_block {
> +	/* May only be accessed by the owner of the block */
> +	struct list_head head;
> +	size_t bytes_used;
> +
> +	/*
> +	 * Set during allocation, constant thereafter. May be accessed read-only
> +	 * by anybody holding a reference to the block.
> +	 */
> +	void *vaddr;
> +	dma_addr_t phys_addr;
> +	size_t size;
> +	struct iio_dma_buffer_queue *queue;
> +
> +	/* Must not be accessed outside the core. */
> +	struct kref kref;
> +	/*
> +	 * Must not be accessed outside the core. Access needs to hold
> +	 * queue->list_lock if the block is not owned by the core.
> +	 */
> +	enum iio_block_state state;
> +};
> +
> +/**
> + * struct iio_dma_buffer_queue_fileio - FileIO state for the DMA buffer
> + * @blocks: Buffer blocks used for fileio
> + * @active_block: Block being used in read()
> + * @pos: Read offset in the active block
> + * @block_size: Size of each block
> + */
> +struct iio_dma_buffer_queue_fileio {
> +	struct iio_dma_buffer_block *blocks[2];
> +	struct iio_dma_buffer_block *active_block;
> +	size_t pos;
> +	size_t block_size;
> +};
> +
> +/**
> + * struct iio_dma_buffer_queue - DMA buffer base structure
> + * @buffer: IIO buffer base structure
> + * @dev: Parent device
> + * @ops: DMA buffer callbacks
> + * @lock: Protects the incoming list, active and the fields in the fileio
> + *   substruct
> + * @list_lock: Protects lists that contain blocks which can be modified in
> + *   atomic context as well as blocks on those lists. This is the outgoing queue
> + *   list and typically also a list of active blocks in the part that handles
> + *   the DMA controller
> + * @incoming: List of buffers on the incoming queue
> + * @outgoing: List of buffers on the outgoing queue
> + * @active: Whether the buffer is currently active
> + * @fileio: FileIO state
> + */
> +struct iio_dma_buffer_queue {
> +	struct iio_buffer buffer;
> +	struct device *dev;
> +	const struct iio_dma_buffer_ops *ops;
> +
> +	struct mutex lock;
> +	spinlock_t list_lock;
> +	struct list_head incoming;
> +	struct list_head outgoing;
> +
> +	bool active;
> +
> +	struct iio_dma_buffer_queue_fileio fileio;
> +};
> +
> +/**
> + * struct iio_dma_buffer_ops - DMA buffer callback operations
> + * @submit: Called when a block is submitted to the DMA controller
> + * @abort: Should abort all pending transfers
> + */
> +struct iio_dma_buffer_ops {
> +	int (*submit)(struct iio_dma_buffer_queue *queue,
> +		struct iio_dma_buffer_block *block);
> +	void (*abort)(struct iio_dma_buffer_queue *queue);
> +};
> +
> +void iio_dma_buffer_block_done(struct iio_dma_buffer_block *block);
> +void iio_dma_buffer_block_list_abort(struct iio_dma_buffer_queue *queue,
> +	struct list_head *list);
> +
> +int iio_dma_buffer_enable(struct iio_buffer *buffer,
> +	struct iio_dev *indio_dev);
> +int iio_dma_buffer_disable(struct iio_buffer *buffer,
> +	struct iio_dev *indio_dev);
> +int iio_dma_buffer_read(struct iio_buffer *buffer, size_t n,
> +	char __user *user_buffer);
> +size_t iio_dma_buffer_data_available(struct iio_buffer *buffer);
> +int iio_dma_buffer_set_bytes_per_datum(struct iio_buffer *buffer, size_t bpd);
> +int iio_dma_buffer_set_length(struct iio_buffer *buffer, int length);
> +int iio_dma_buffer_request_update(struct iio_buffer *buffer);
> +
> +int iio_dma_buffer_init(struct iio_dma_buffer_queue *queue,
> +	struct device *dma_dev, const struct iio_dma_buffer_ops *ops);
> +void iio_dma_buffer_exit(struct iio_dma_buffer_queue *queue);
> +void iio_dma_buffer_release(struct iio_dma_buffer_queue *queue);
> +
> +#endif
> 

Re: [PATCH v3 6/6] iio: Add a DMAengine framework based buffer

[Jonathan Cameron]

On 13/10/15 17:10, Lars-Peter Clausen wrote:
> Add a generic fully device independent DMA buffer implementation that uses
> the DMAegnine framework to perform the DMA transfers. This can be used by
> converter drivers that whish to provide a DMA buffer for converters that
> are connected to a DMA core that implements the DMAengine API.
> 
> Apart from allocating the buffer using iio_dmaengine_buffer_alloc() and
> freeing it using iio_dmaengine_buffer_free() no additional converter driver
> specific code is required when using this DMA buffer implementation.
> 
> Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
Applied to the togreg branch of iio.git - initially pushed out as testing
for the autobuilders to play with it (not that they are likely to actually build
much of it given no users :)

J
> ---
>  drivers/iio/buffer/Kconfig                         |  11 ++
>  drivers/iio/buffer/Makefile                        |   1 +
>  drivers/iio/buffer/industrialio-buffer-dmaengine.c | 213 +++++++++++++++++++++
>  include/linux/iio/buffer-dmaengine.h               |  18 ++
>  4 files changed, 243 insertions(+)
>  create mode 100644 drivers/iio/buffer/industrialio-buffer-dmaengine.c
>  create mode 100644 include/linux/iio/buffer-dmaengine.h
> 
> diff --git a/drivers/iio/buffer/Kconfig b/drivers/iio/buffer/Kconfig
> index b2fda1a..4ffd3db 100644
> --- a/drivers/iio/buffer/Kconfig
> +++ b/drivers/iio/buffer/Kconfig
> @@ -18,6 +18,17 @@ config IIO_BUFFER_DMA
>  	  Should be selected by drivers that want to use the generic DMA buffer
>  	  infrastructure.
>  
> +config IIO_BUFFER_DMAENGINE
> +	tristate
> +	select IIO_BUFFER_DMA
> +	help
> +	  Provides a bonding of the generic IIO DMA buffer infrastructure with the
> +	  DMAengine framework. This can be used by converter drivers with a DMA port
> +	  connected to an external DMA controller which is supported by the
> +	  DMAengine framework.
> +
> +	  Should be selected by drivers that want to use this functionality.
> +
>  config IIO_KFIFO_BUF
>  	tristate "Industrial I/O buffering based on kfifo"
>  	help
> diff --git a/drivers/iio/buffer/Makefile b/drivers/iio/buffer/Makefile
> index bda3f11..85beaae 100644
> --- a/drivers/iio/buffer/Makefile
> +++ b/drivers/iio/buffer/Makefile
> @@ -5,5 +5,6 @@
>  # When adding new entries keep the list in alphabetical order
>  obj-$(CONFIG_IIO_BUFFER_CB) += industrialio-buffer-cb.o
>  obj-$(CONFIG_IIO_BUFFER_DMA) += industrialio-buffer-dma.o
> +obj-$(CONFIG_IIO_BUFFER_DMAENGINE) += industrialio-buffer-dmaengine.o
>  obj-$(CONFIG_IIO_TRIGGERED_BUFFER) += industrialio-triggered-buffer.o
>  obj-$(CONFIG_IIO_KFIFO_BUF) += kfifo_buf.o
> diff --git a/drivers/iio/buffer/industrialio-buffer-dmaengine.c b/drivers/iio/buffer/industrialio-buffer-dmaengine.c
> new file mode 100644
> index 0000000..ebdb838
> --- /dev/null
> +++ b/drivers/iio/buffer/industrialio-buffer-dmaengine.c
> @@ -0,0 +1,213 @@
> +/*
> + * Copyright 2014-2015 Analog Devices Inc.
> + *  Author: Lars-Peter Clausen <lars@metafoo.de>
> + *
> + * Licensed under the GPL-2 or later.
> + */
> +
> +#include <linux/slab.h>
> +#include <linux/kernel.h>
> +#include <linux/dmaengine.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/spinlock.h>
> +#include <linux/err.h>
> +
> +#include <linux/iio/iio.h>
> +#include <linux/iio/buffer.h>
> +#include <linux/iio/buffer-dma.h>
> +#include <linux/iio/buffer-dmaengine.h>
> +
> +/*
> + * The IIO DMAengine buffer combines the generic IIO DMA buffer infrastructure
> + * with the DMAengine framework. The generic IIO DMA buffer infrastructure is
> + * used to manage the buffer memory and implement the IIO buffer operations
> + * while the DMAengine framework is used to perform the DMA transfers. Combined
> + * this results in a device independent fully functional DMA buffer
> + * implementation that can be used by device drivers for peripherals which are
> + * connected to a DMA controller which has a DMAengine driver implementation.
> + */
> +
> +struct dmaengine_buffer {
> +	struct iio_dma_buffer_queue queue;
> +
> +	struct dma_chan *chan;
> +	struct list_head active;
> +
> +	size_t align;
> +	size_t max_size;
> +};
> +
> +static struct dmaengine_buffer *iio_buffer_to_dmaengine_buffer(
> +		struct iio_buffer *buffer)
> +{
> +	return container_of(buffer, struct dmaengine_buffer, queue.buffer);
> +}
> +
> +static void iio_dmaengine_buffer_block_done(void *data)
> +{
> +	struct iio_dma_buffer_block *block = data;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&block->queue->list_lock, flags);
> +	list_del(&block->head);
> +	spin_unlock_irqrestore(&block->queue->list_lock, flags);
> +	iio_dma_buffer_block_done(block);
> +}
> +
> +static int iio_dmaengine_buffer_submit_block(struct iio_dma_buffer_queue *queue,
> +	struct iio_dma_buffer_block *block)
> +{
> +	struct dmaengine_buffer *dmaengine_buffer =
> +		iio_buffer_to_dmaengine_buffer(&queue->buffer);
> +	struct dma_async_tx_descriptor *desc;
> +	dma_cookie_t cookie;
> +
> +	block->bytes_used = min(block->size, dmaengine_buffer->max_size);
> +	block->bytes_used = rounddown(block->bytes_used,
> +			dmaengine_buffer->align);
> +
> +	desc = dmaengine_prep_slave_single(dmaengine_buffer->chan,
> +		block->phys_addr, block->bytes_used, DMA_DEV_TO_MEM,
> +		DMA_PREP_INTERRUPT);
> +	if (!desc)
> +		return -ENOMEM;
> +
> +	desc->callback = iio_dmaengine_buffer_block_done;
> +	desc->callback_param = block;
> +
> +	cookie = dmaengine_submit(desc);
> +	if (dma_submit_error(cookie))
> +		return dma_submit_error(cookie);
> +
> +	spin_lock_irq(&dmaengine_buffer->queue.list_lock);
> +	list_add_tail(&block->head, &dmaengine_buffer->active);
> +	spin_unlock_irq(&dmaengine_buffer->queue.list_lock);
> +
> +	dma_async_issue_pending(dmaengine_buffer->chan);
> +
> +	return 0;
> +}
> +
> +static void iio_dmaengine_buffer_abort(struct iio_dma_buffer_queue *queue)
> +{
> +	struct dmaengine_buffer *dmaengine_buffer =
> +		iio_buffer_to_dmaengine_buffer(&queue->buffer);
> +
> +	dmaengine_terminate_all(dmaengine_buffer->chan);
> +	/* FIXME: There is a slight chance of a race condition here.
> +	 * dmaengine_terminate_all() does not guarantee that all transfer
> +	 * callbacks have finished running. Need to introduce a
> +	 * dmaengine_terminate_all_sync().
> +	 */
> +	iio_dma_buffer_block_list_abort(queue, &dmaengine_buffer->active);
> +}
> +
> +static void iio_dmaengine_buffer_release(struct iio_buffer *buf)
> +{
> +	struct dmaengine_buffer *dmaengine_buffer =
> +		iio_buffer_to_dmaengine_buffer(buf);
> +
> +	iio_dma_buffer_release(&dmaengine_buffer->queue);
> +	kfree(dmaengine_buffer);
> +}
> +
> +static const struct iio_buffer_access_funcs iio_dmaengine_buffer_ops = {
> +	.read_first_n = iio_dma_buffer_read,
> +	.set_bytes_per_datum = iio_dma_buffer_set_bytes_per_datum,
> +	.set_length = iio_dma_buffer_set_length,
> +	.request_update = iio_dma_buffer_request_update,
> +	.enable = iio_dma_buffer_enable,
> +	.disable = iio_dma_buffer_disable,
> +	.data_available = iio_dma_buffer_data_available,
> +	.release = iio_dmaengine_buffer_release,
> +
> +	.modes = INDIO_BUFFER_HARDWARE,
> +	.flags = INDIO_BUFFER_FLAG_FIXED_WATERMARK,
> +};
> +
> +static const struct iio_dma_buffer_ops iio_dmaengine_default_ops = {
> +	.submit = iio_dmaengine_buffer_submit_block,
> +	.abort = iio_dmaengine_buffer_abort,
> +};
> +
> +/**
> + * iio_dmaengine_buffer_alloc() - Allocate new buffer which uses DMAengine
> + * @dev: Parent device for the buffer
> + * @channel: DMA channel name, typically "rx".
> + *
> + * This allocates a new IIO buffer which internally uses the DMAengine framework
> + * to perform its transfers. The parent device will be used to request the DMA
> + * channel.
> + *
> + * Once done using the buffer iio_dmaengine_buffer_free() should be used to
> + * release it.
> + */
> +struct iio_buffer *iio_dmaengine_buffer_alloc(struct device *dev,
> +	const char *channel)
> +{
> +	struct dmaengine_buffer *dmaengine_buffer;
> +	unsigned int width, src_width, dest_width;
> +	struct dma_slave_caps caps;
> +	struct dma_chan *chan;
> +	int ret;
> +
> +	dmaengine_buffer = kzalloc(sizeof(*dmaengine_buffer), GFP_KERNEL);
> +	if (!dmaengine_buffer)
> +		return ERR_PTR(-ENOMEM);
> +
> +	chan = dma_request_slave_channel_reason(dev, channel);
> +	if (IS_ERR(chan)) {
> +		ret = PTR_ERR(chan);
> +		goto err_free;
> +	}
> +
> +	ret = dma_get_slave_caps(chan, &caps);
> +	if (ret < 0)
> +		goto err_free;
> +
> +	/* Needs to be aligned to the maximum of the minimums */
> +	if (caps.src_addr_widths)
> +		src_width = __ffs(caps.src_addr_widths);
> +	else
> +		src_width = 1;
> +	if (caps.dst_addr_widths)
> +		dest_width = __ffs(caps.dst_addr_widths);
> +	else
> +		dest_width = 1;
> +	width = max(src_width, dest_width);
> +
> +	INIT_LIST_HEAD(&dmaengine_buffer->active);
> +	dmaengine_buffer->chan = chan;
> +	dmaengine_buffer->align = width;
> +	dmaengine_buffer->max_size = dma_get_max_seg_size(chan->device->dev);
> +
> +	iio_dma_buffer_init(&dmaengine_buffer->queue, chan->device->dev,
> +		&iio_dmaengine_default_ops);
> +
> +	dmaengine_buffer->queue.buffer.access = &iio_dmaengine_buffer_ops;
> +
> +	return &dmaengine_buffer->queue.buffer;
> +
> +err_free:
> +	kfree(dmaengine_buffer);
> +	return ERR_PTR(ret);
> +}
> +EXPORT_SYMBOL(iio_dmaengine_buffer_alloc);
> +
> +/**
> + * iio_dmaengine_buffer_free() - Free dmaengine buffer
> + * @buffer: Buffer to free
> + *
> + * Frees a buffer previously allocated with iio_dmaengine_buffer_alloc().
> + */
> +void iio_dmaengine_buffer_free(struct iio_buffer *buffer)
> +{
> +	struct dmaengine_buffer *dmaengine_buffer =
> +		iio_buffer_to_dmaengine_buffer(buffer);
> +
> +	iio_dma_buffer_exit(&dmaengine_buffer->queue);
> +	dma_release_channel(dmaengine_buffer->chan);
> +
> +	iio_buffer_put(buffer);
> +}
> +EXPORT_SYMBOL_GPL(iio_dmaengine_buffer_free);
> diff --git a/include/linux/iio/buffer-dmaengine.h b/include/linux/iio/buffer-dmaengine.h
> new file mode 100644
> index 0000000..5dcddf4
> --- /dev/null
> +++ b/include/linux/iio/buffer-dmaengine.h
> @@ -0,0 +1,18 @@
> +/*
> + * Copyright 2014-2015 Analog Devices Inc.
> + *  Author: Lars-Peter Clausen <lars@metafoo.de>
> + *
> + * Licensed under the GPL-2 or later.
> + */
> +
> +#ifndef __IIO_DMAENGINE_H__
> +#define __IIO_DMAENGINE_H__
> +
> +struct iio_buffer;
> +struct device;
> +
> +struct iio_buffer *iio_dmaengine_buffer_alloc(struct device *dev,
> +	const char *channel);
> +void iio_dmaengine_buffer_free(struct iio_buffer *buffer);
> +
> +#endif
>