[Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

Hi All,

This topic would be around the way the various subsystems interact, in the
rough area of 'sensors' (I haven't yet had much of an issue with subsystem
crossing with output devices but maybe that's just over the next hill!)

Scope may well be wider but includes:
* input (some of it)
* hwmon
* iio
* comedi(?)
* thermal
* power/battery
* gpio - the blur between gpios and beaglescope / PLC type I/O.
* v4l - when does a device jump from being a multi pixel thermopile
to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
sensors (effectively 9ish pixel cameras)
* Lots of random things we haven't seen yet.

All these make use of devices that are at their hearts ADCs. Whilst it
would nice if the underlying devices always fell into one clear
category that is not always the case. Hardware descriptions (e.g.
device tree) are an important area of contention.

The examples that follow are mostly based around IIO interactions with
hwmon / input as that's my area of expertise. However, I know that
similar issues occur at other boundaries.

Guenter Roeck has been working on hwmon / thermal interfacing recently.
We also regularly push drivers one way or the other around that boundary.

Dmitry and I often have to decide on the IIO / input scope boundary
(and we've moved it over time) + a number of cross over drivers have
turned up recently.

The other subsystems I encounter have been 'quieter' in their
interactions with me but many of the issues below apply to them as well.

A rough list of related topics that might be worth face to face
discussion includes:

1) Are we happy with the somewhat adhoc divisions of subsystems as they
stand?

I am personally happy enough with this, but it's worth checking
everyone else is. This probably causes more pain for new submitters
than it does for old hands. Perhaps we can come up with a checklist style
doc to save people having to ask.
 
We have had quite a few cases where a whole driver is submitted to the
'wrong' subsystem. Not the best first interaction with the kernel
world for new contributors. (This sort of area is attracts newbies
as it's relatively simple and the hardware is fairly cheap and more
than once their first experience with review as exactly this!)

If we aren't happy, what do we do about it?

I'm hoping we are happy or, at least, resigned to the current state
of affairs, but think the question should be asked from time to time
as the answer may well change + the discussion will highlight pinch
points we can work on.

2) Bridging drivers - there will always be cross over cases.
E.g. Generic ADCs used for battery voltage monitoring.  We have a
number of bridges in mainline already and others out of tree.  How
important are these and what features would make them more useful?

Is it ever sensible to have two drivers for the same part because
the use cases are so different? (I hope there is always a better way
but maybe not)

This covers both generic bridges (e.g. iio-hwmon) and device
specific bridges (a good example of a touch screen driver turned
up in my inbox yesterday).

3) Bindings (device tree and similar).  When it is appropriate to use a 
device tree to describe the overlap (bridging of channels)?
Sometimes there is obvious real hardware involved (a thermistor on a
generic ADC input for example), but there are many grey areas.

Finally bashing out an answer to the issue device tree maintainers
have with the iio-hwmon bindings would be great. 
On a less biased note, that example is pretty fiddly to define but would
act as a good basis to work from more generally.

We know we got it wrong (back in the dark ages), but it isn't obvious how
to get it right!

IIO-input has been partly held outside mainline for years by this issue.

Also some practical design decisions to make around how to implement
these mappings to work best with deferred probing etc.

4) Should we drop all the bindings for bridging between such subsystems
and do it all from userspace?

I think we may end up with a hybrid of the two, but need to be able to
make it work in 'standard' cases without userspace being involved. That
hybrid solution may well be devicetree overlay based... Unclear so far.
+ plenty of crazy things that 'might' make sense where there isn't even
the pretence of representing real hardware.

5) Complex device interaction usecases.  At the moment the ones I've come
across are mostly contained in IIO.  The moment we start sticking in
MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
mix it can get fiddly.  Swapping war stories may well be worthwhile on
this. This stuff also turns up in ASoC for example so probably lessons
to be learned from there.

The analog devices software defined radios are another possible case
study.

There of course may well be lessons to be learned from similar
interactions elsewhere in the kernel.

There is a lot of history in how we ended up where we are (it all made
absolute sense at the time). Sitting back and taking the time
to discuss the future would be great.  Whilst this might be solvable
by email we've made no definitive progress for years
(and what has been made has been on a case by case basis deep in
driver reviews.)

I threw comedi in the list above but, at the moment, I think the more
likely direction there is a single userspace library abstracting
the underlying subsystem (Analog Devices are working in that
direction - perhaps Lars can offer more on that?).

GPIO is another interesting case - a lot of hardware is capable of
parallel sampling, some at high speeds. It's another area that
is probably too specialist for this discussion, but if people want
to dive into the details it might be interesting.

I think we have only a small amount of fuzz around the v4l boundary,
but wanted to leave the door open if anyone wants to discuss that
one further as it's come up a few times over recent years.

The SoC world is a major case of one device, many uses.  Some SoCs
are turning up with multiple ADC units, sometimes with different
designs, sometimes simply so that the same hardware can be used for different things.

A few suggestions for people:

Guenter Roeck (hmwon)
Dmitry Torokhov (input)
Me (Jonathan Cameron - IIO)

Someone thermal related (not sure who would be most interested)
Someone power supply related? (Guenter, any suggestions?)

Someone device tree related (Mark Rutland? Rob Herring?)

Linus Waleij (gpio)

Mark Brown - as some of concepts of IIO bridging and how far it
can be taken (everything via a bridge driver) came from
discussions with him on SoC ADC handling.

Lar-Peter Clausen for generally doing insane things with ADCs + having
one foot in the userspace side of things.

Thanks,

Jonathan Cameron
IIO Maintainer

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Hans Verkuil]

On 07/20/2016 11:18 PM, Jonathan Cameron wrote:
> Hi All,
> 
> This topic would be around the way the various subsystems interact, in the
> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> crossing with output devices but maybe that's just over the next hill!)

This happens regularly with drm/kms vs v4l, especially for embedded devices
where video inputs and outputs are often connected internally (i.e. paths
straight from video capture to video output without needing to go to memory
first).

> 
> Scope may well be wider but includes:
> * input (some of it)
> * hwmon
> * iio
> * comedi(?)
> * thermal
> * power/battery
> * gpio - the blur between gpios and beaglescope / PLC type I/O.
> * v4l - when does a device jump from being a multi pixel thermopile
> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
> sensors (effectively 9ish pixel cameras)

See for an example of that this patch series that adds support for v4l-touch
devices:

http://www.spinics.net/lists/linux-input/msg45918.html

> * Lots of random things we haven't seen yet.
> 
> All these make use of devices that are at their hearts ADCs. Whilst it
> would nice if the underlying devices always fell into one clear
> category that is not always the case. Hardware descriptions (e.g.
> device tree) are an important area of contention.
> 
> The examples that follow are mostly based around IIO interactions with
> hwmon / input as that's my area of expertise. However, I know that
> similar issues occur at other boundaries.
> 
> Guenter Roeck has been working on hwmon / thermal interfacing recently.
> We also regularly push drivers one way or the other around that boundary.
> 
> Dmitry and I often have to decide on the IIO / input scope boundary
> (and we've moved it over time) + a number of cross over drivers have
> turned up recently.
> 
> The other subsystems I encounter have been 'quieter' in their
> interactions with me but many of the issues below apply to them as well.
> 
> A rough list of related topics that might be worth face to face
> discussion includes:
> 
> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
> stand?
> 
> I am personally happy enough with this, but it's worth checking
> everyone else is. This probably causes more pain for new submitters
> than it does for old hands. Perhaps we can come up with a checklist style
> doc to save people having to ask.
>  
> We have had quite a few cases where a whole driver is submitted to the
> 'wrong' subsystem. Not the best first interaction with the kernel
> world for new contributors. (This sort of area is attracts newbies
> as it's relatively simple and the hardware is fairly cheap and more
> than once their first experience with review as exactly this!)
> 
> If we aren't happy, what do we do about it?
> 
> I'm hoping we are happy or, at least, resigned to the current state
> of affairs, but think the question should be asked from time to time
> as the answer may well change + the discussion will highlight pinch
> points we can work on.
> 
> 2) Bridging drivers - there will always be cross over cases.
> E.g. Generic ADCs used for battery voltage monitoring.  We have a
> number of bridges in mainline already and others out of tree.  How
> important are these and what features would make them more useful?
> 
> Is it ever sensible to have two drivers for the same part because
> the use cases are so different? (I hope there is always a better way
> but maybe not)

One example of this is video output: both drm/kms and v4l can do video
output. Video output using v4l is specifically geared towards frame-based
video output whereas drm/kms is for desktop environments/GPUs. Luckily there
are very few v4l video output drivers and the driver duplication occurs
only for the i2c video transmitter devices. I know of two that are duplicated:
adv7511 and (I think) saa7127. The rarity is the reason nobody ever created
a shim or something similar to allow for one i2c driver to serve both
subsystems. Technically this would be perfectly possible, it's just work.

> This covers both generic bridges (e.g. iio-hwmon) and device
> specific bridges (a good example of a touch screen driver turned
> up in my inbox yesterday).
> 
> 3) Bindings (device tree and similar).  When it is appropriate to use a 
> device tree to describe the overlap (bridging of channels)?
> Sometimes there is obvious real hardware involved (a thermistor on a
> generic ADC input for example), but there are many grey areas.
> 
> Finally bashing out an answer to the issue device tree maintainers
> have with the iio-hwmon bindings would be great. 
> On a less biased note, that example is pretty fiddly to define but would
> act as a good basis to work from more generally.
> 
> We know we got it wrong (back in the dark ages), but it isn't obvious how
> to get it right!
> 
> IIO-input has been partly held outside mainline for years by this issue.
> 
> Also some practical design decisions to make around how to implement
> these mappings to work best with deferred probing etc.
> 
> 4) Should we drop all the bindings for bridging between such subsystems
> and do it all from userspace?
> 
> I think we may end up with a hybrid of the two, but need to be able to
> make it work in 'standard' cases without userspace being involved. That
> hybrid solution may well be devicetree overlay based... Unclear so far.
> + plenty of crazy things that 'might' make sense where there isn't even
> the pretence of representing real hardware.
> 
> 5) Complex device interaction usecases.  At the moment the ones I've come
> across are mostly contained in IIO.  The moment we start sticking in
> MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
> mix it can get fiddly.  Swapping war stories may well be worthwhile on
> this. This stuff also turns up in ASoC for example so probably lessons
> to be learned from there.

V4L has a long history of this: if you have multiple video scalers in your
pipeline, how do you control them? E.g. there can be scalers in the video
sensor and in the SoC. We came up with the media controller device to
expose the topology of the hardware and to be able to control each bit
directly from userspace.

While the media controller is quite powerful and can be used to expose
connections across subsystems, it also requires more work from userspace,
and that's an area where not enough attention has been given.

> The analog devices software defined radios are another possible case
> study.

Note that the V4L2 subsystem has support for several SDR devices. Probably
another area where drivers can end up in different subsystems.

> 
> There of course may well be lessons to be learned from similar
> interactions elsewhere in the kernel.
> 
> There is a lot of history in how we ended up where we are (it all made
> absolute sense at the time). Sitting back and taking the time
> to discuss the future would be great.  Whilst this might be solvable
> by email we've made no definitive progress for years
> (and what has been made has been on a case by case basis deep in
> driver reviews.)
> 
> I threw comedi in the list above but, at the moment, I think the more
> likely direction there is a single userspace library abstracting
> the underlying subsystem (Analog Devices are working in that
> direction - perhaps Lars can offer more on that?).

When you get complex systems you probably have to move in that direction.
I'd be interesting in such work going on in other subsystems. As mentioned
above, the media controller needs something like that and it would be
good to learn from others.

> GPIO is another interesting case - a lot of hardware is capable of
> parallel sampling, some at high speeds. It's another area that
> is probably too specialist for this discussion, but if people want
> to dive into the details it might be interesting.
> 
> I think we have only a small amount of fuzz around the v4l boundary,
> but wanted to leave the door open if anyone wants to discuss that
> one further as it's come up a few times over recent years.

There is more of that than we wish, but quite often it is just dropped
in the end as being too much work to solve and not quite important enough
to spend the time and money on.

> The SoC world is a major case of one device, many uses.  Some SoCs
> are turning up with multiple ADC units, sometimes with different
> designs, sometimes simply so that the same hardware can be used for different things.
> 
> A few suggestions for people:
> 
> Guenter Roeck (hmwon)
> Dmitry Torokhov (input)
> Me (Jonathan Cameron - IIO)
> 
> Someone thermal related (not sure who would be most interested)
> Someone power supply related? (Guenter, any suggestions?)
> 
> Someone device tree related (Mark Rutland? Rob Herring?)
> 
> Linus Waleij (gpio)
> 
> Mark Brown - as some of concepts of IIO bridging and how far it
> can be taken (everything via a bridge driver) came from
> discussions with him on SoC ADC handling.
> 
> Lar-Peter Clausen for generally doing insane things with ADCs + having
> one foot in the userspace side of things.

Laurent Pinchart or myself for v4l/mc

Regards,

	Hans

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

On Wed, Jul 20, 2016 at 10:18:11PM +0100, Jonathan Cameron wrote:

> 4) Should we drop all the bindings for bridging between such subsystems
> and do it all from userspace?

This clearly doesn't seem sensible, we have plenty of hardware that's
obviously wired up in a fixed function format even if it could be
general purpose and several duplicated ABIs (eg, hwmon and power_supply
have clear overlap).

> I think we may end up with a hybrid of the two, but need to be able to
> make it work in 'standard' cases without userspace being involved. That
> hybrid solution may well be devicetree overlay based... Unclear so far.
> + plenty of crazy things that 'might' make sense where there isn't even
> the pretence of representing real hardware.

I'm definitely seeing people choosing their kernel subsystem on the
basis of libiio.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Guenter Roeck]

On 07/20/2016 02:18 PM, Jonathan Cameron wrote:
> Hi All,
>
> This topic would be around the way the various subsystems interact, in the
> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> crossing with output devices but maybe that's just over the next hill!)
>
> Scope may well be wider but includes:
> * input (some of it)
> * hwmon
> * iio
> * comedi(?)
> * thermal
> * power/battery
> * gpio - the blur between gpios and beaglescope / PLC type I/O.
> * v4l - when does a device jump from being a multi pixel thermopile
> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
> sensors (effectively 9ish pixel cameras)
> * Lots of random things we haven't seen yet.
>
> All these make use of devices that are at their hearts ADCs. Whilst it
> would nice if the underlying devices always fell into one clear
> category that is not always the case. Hardware descriptions (e.g.
> device tree) are an important area of contention.
>
> The examples that follow are mostly based around IIO interactions with
> hwmon / input as that's my area of expertise. However, I know that
> similar issues occur at other boundaries.
>
> Guenter Roeck has been working on hwmon / thermal interfacing recently.
> We also regularly push drivers one way or the other around that boundary.
>
> Dmitry and I often have to decide on the IIO / input scope boundary
> (and we've moved it over time) + a number of cross over drivers have
> turned up recently.
>
> The other subsystems I encounter have been 'quieter' in their
> interactions with me but many of the issues below apply to them as well.
>
> A rough list of related topics that might be worth face to face
> discussion includes:
>
> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
> stand?
>
> I am personally happy enough with this, but it's worth checking

Same here.

> everyone else is. This probably causes more pain for new submitters
> than it does for old hands. Perhaps we can come up with a checklist style
> doc to save people having to ask.
>
I think that would be very useful.

> We have had quite a few cases where a whole driver is submitted to the
> 'wrong' subsystem. Not the best first interaction with the kernel

... especially to avoid that situation.

> world for new contributors. (This sort of area is attracts newbies
> as it's relatively simple and the hardware is fairly cheap and more
> than once their first experience with review as exactly this!)
>
> If we aren't happy, what do we do about it?
>
> I'm hoping we are happy or, at least, resigned to the current state
> of affairs, but think the question should be asked from time to time
> as the answer may well change + the discussion will highlight pinch
> points we can work on.
>
> 2) Bridging drivers - there will always be cross over cases.
> E.g. Generic ADCs used for battery voltage monitoring.  We have a
> number of bridges in mainline already and others out of tree.  How
> important are these and what features would make them more useful?
>
> Is it ever sensible to have two drivers for the same part because
> the use cases are so different? (I hope there is always a better way
> but maybe not)
>
> This covers both generic bridges (e.g. iio-hwmon) and device
> specific bridges (a good example of a touch screen driver turned
> up in my inbox yesterday).
>
> 3) Bindings (device tree and similar).  When it is appropriate to use a
> device tree to describe the overlap (bridging of channels)?
> Sometimes there is obvious real hardware involved (a thermistor on a
> generic ADC input for example), but there are many grey areas.
>
> Finally bashing out an answer to the issue device tree maintainers
> have with the iio-hwmon bindings would be great.

Agreed, it would be great if we can come up with a fix for that.

> On a less biased note, that example is pretty fiddly to define but would
> act as a good basis to work from more generally.
>
> We know we got it wrong (back in the dark ages), but it isn't obvious how
> to get it right!
>
> IIO-input has been partly held outside mainline for years by this issue.
>
> Also some practical design decisions to make around how to implement
> these mappings to work best with deferred probing etc.
>
> 4) Should we drop all the bindings for bridging between such subsystems
> and do it all from userspace?
>
I think that would be a terrible idea. My hope is that we should be able
to present a system to the user as it is intended to be, and that should include
the use case for given hardware. An ADC is not just an ADC, it has a use case.
The use case may be obvious if it is used as input device, but if it is
used to measure a voltage (or current, or temperature) it should be possible
to express that as well, without having to have userspace involved.

> I think we may end up with a hybrid of the two, but need to be able to
> make it work in 'standard' cases without userspace being involved. That
> hybrid solution may well be devicetree overlay based... Unclear so far.
> + plenty of crazy things that 'might' make sense where there isn't even
> the pretence of representing real hardware.
>
Not really sure if/how that would require overlays as long as the use case
is static. If an ADC is used to measure a voltage on a given system, it is
unlikely to be used for something else in the same system.

> 5) Complex device interaction usecases.  At the moment the ones I've come
> across are mostly contained in IIO.  The moment we start sticking in
> MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
> mix it can get fiddly.  Swapping war stories may well be worthwhile on
> this. This stuff also turns up in ASoC for example so probably lessons
> to be learned from there.
>
> The analog devices software defined radios are another possible case
> study.
>
> There of course may well be lessons to be learned from similar
> interactions elsewhere in the kernel.
>
> There is a lot of history in how we ended up where we are (it all made
> absolute sense at the time). Sitting back and taking the time
> to discuss the future would be great.  Whilst this might be solvable
> by email we've made no definitive progress for years
> (and what has been made has been on a case by case basis deep in
> driver reviews.)
>
> I threw comedi in the list above but, at the moment, I think the more
> likely direction there is a single userspace library abstracting
> the underlying subsystem (Analog Devices are working in that
> direction - perhaps Lars can offer more on that?).
>
> GPIO is another interesting case - a lot of hardware is capable of
> parallel sampling, some at high speeds. It's another area that
> is probably too specialist for this discussion, but if people want
> to dive into the details it might be interesting.
>
> I think we have only a small amount of fuzz around the v4l boundary,
> but wanted to leave the door open if anyone wants to discuss that
> one further as it's come up a few times over recent years.
>
> The SoC world is a major case of one device, many uses.  Some SoCs
> are turning up with multiple ADC units, sometimes with different
> designs, sometimes simply so that the same hardware can be used for different things.
>
> A few suggestions for people:
>
> Guenter Roeck (hmwon)
> Dmitry Torokhov (input)
> Me (Jonathan Cameron - IIO)
>
> Someone thermal related (not sure who would be most interested)
> Someone power supply related? (Guenter, any suggestions?)

Not really, other than the power subsystem maintainers. Question here
might be if it would make sense to register battery charger drivers as
hardware monitoring devices - with the new hwmon API that should be quite
straightforward.

I keep thinking about power, but regulators are probably at least as
relevant when it comes to interaction with hardware monitoring.
Many PMBus chips have both hardware monitoring and voltage/current
regulator functionality. The PMBus core implements some basic
functionality along that line, and can register a chip as regulator
if requested. Would it make sense to register regulators as hardware
monitoring devices ? I don't really know, and never thought about it
until now. Maybe Mark Brown and/or Liam Girdwood have some thoughts
on this.

Thanks,
Guenter

>
> Someone device tree related (Mark Rutland? Rob Herring?)
>
> Linus Waleij (gpio)
>
> Mark Brown - as some of concepts of IIO bridging and how far it
> can be taken (everything via a bridge driver) came from
> discussions with him on SoC ADC handling.
>
> Lar-Peter Clausen for generally doing insane things with ADCs + having
> one foot in the userspace side of things.
>
> Thanks,
>
> Jonathan Cameron
> IIO Maintainer
>

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Torokhov]

On Thu, Jul 21, 2016 at 08:29:34PM -0700, Guenter Roeck wrote:
> On 07/20/2016 02:18 PM, Jonathan Cameron wrote:
> >Hi All,
> >
> >This topic would be around the way the various subsystems interact, in the
> >rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> >crossing with output devices but maybe that's just over the next hill!)
> >
> >Scope may well be wider but includes:
> >* input (some of it)
> >* hwmon
> >* iio
> >* comedi(?)
> >* thermal
> >* power/battery
> >* gpio - the blur between gpios and beaglescope / PLC type I/O.
> >* v4l - when does a device jump from being a multi pixel thermopile
> >to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
> >sensors (effectively 9ish pixel cameras)
> >* Lots of random things we haven't seen yet.
> >
> >All these make use of devices that are at their hearts ADCs. Whilst it
> >would nice if the underlying devices always fell into one clear
> >category that is not always the case. Hardware descriptions (e.g.
> >device tree) are an important area of contention.
> >
> >The examples that follow are mostly based around IIO interactions with
> >hwmon / input as that's my area of expertise. However, I know that
> >similar issues occur at other boundaries.
> >
> >Guenter Roeck has been working on hwmon / thermal interfacing recently.
> >We also regularly push drivers one way or the other around that boundary.
> >
> >Dmitry and I often have to decide on the IIO / input scope boundary
> >(and we've moved it over time) + a number of cross over drivers have
> >turned up recently.
> >
> >The other subsystems I encounter have been 'quieter' in their
> >interactions with me but many of the issues below apply to them as well.
> >
> >A rough list of related topics that might be worth face to face
> >discussion includes:
> >
> >1) Are we happy with the somewhat adhoc divisions of subsystems as they
> >stand?
> >
> >I am personally happy enough with this, but it's worth checking
> 
> Same here.
> 
> >everyone else is. This probably causes more pain for new submitters
> >than it does for old hands. Perhaps we can come up with a checklist style
> >doc to save people having to ask.
> >
> I think that would be very useful.
> 
> >We have had quite a few cases where a whole driver is submitted to the
> >'wrong' subsystem. Not the best first interaction with the kernel
> 
> ... especially to avoid that situation.
> 
> >world for new contributors. (This sort of area is attracts newbies
> >as it's relatively simple and the hardware is fairly cheap and more
> >than once their first experience with review as exactly this!)
> >
> >If we aren't happy, what do we do about it?
> >
> >I'm hoping we are happy or, at least, resigned to the current state
> >of affairs, but think the question should be asked from time to time
> >as the answer may well change + the discussion will highlight pinch
> >points we can work on.
> >
> >2) Bridging drivers - there will always be cross over cases.
> >E.g. Generic ADCs used for battery voltage monitoring.  We have a
> >number of bridges in mainline already and others out of tree.  How
> >important are these and what features would make them more useful?
> >
> >Is it ever sensible to have two drivers for the same part because
> >the use cases are so different? (I hope there is always a better way
> >but maybe not)
> >
> >This covers both generic bridges (e.g. iio-hwmon) and device
> >specific bridges (a good example of a touch screen driver turned
> >up in my inbox yesterday).
> >
> >3) Bindings (device tree and similar).  When it is appropriate to use a
> >device tree to describe the overlap (bridging of channels)?
> >Sometimes there is obvious real hardware involved (a thermistor on a
> >generic ADC input for example), but there are many grey areas.
> >
> >Finally bashing out an answer to the issue device tree maintainers
> >have with the iio-hwmon bindings would be great.
> 
> Agreed, it would be great if we can come up with a fix for that.
> 
> >On a less biased note, that example is pretty fiddly to define but would
> >act as a good basis to work from more generally.
> >
> >We know we got it wrong (back in the dark ages), but it isn't obvious how
> >to get it right!
> >
> >IIO-input has been partly held outside mainline for years by this issue.
> >
> >Also some practical design decisions to make around how to implement
> >these mappings to work best with deferred probing etc.
> >
> >4) Should we drop all the bindings for bridging between such subsystems
> >and do it all from userspace?
> >
> I think that would be a terrible idea. My hope is that we should be able
> to present a system to the user as it is intended to be, and that should include
> the use case for given hardware. An ADC is not just an ADC, it has a use case.
> The use case may be obvious if it is used as input device, but if it is
> used to measure a voltage (or current, or temperature) it should be possible
> to express that as well, without having to have userspace involved.

Totally agree. We might want to allow reconfiguring from userspace,
but having initial configuration conveyed through DT/ACPI/whatever
according to the original system purpose makes sense to me.

Thanks.

-- 
Dmitry

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

On Thu, Jul 21, 2016 at 08:29:34PM -0700, Guenter Roeck wrote:

> if requested. Would it make sense to register regulators as hardware
> monitoring devices ? I don't really know, and never thought about it
> until now. Maybe Mark Brown and/or Liam Girdwood have some thoughts
> on this.

It has been discussed before but it never seemed sensible to me.  hwmon
is a read only interface monitoring the actual state of the system while
regulators are write only and if things are going wrong may not reflect
the actual state of the system.  Exporting write only information via
hwmon seems to break expectations for users of it, but perhaps that's
not what people are expecting from hwmon these days.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Linus Walleij]

On Wed, Jul 20, 2016 at 11:18 PM, Jonathan Cameron <jic23@kernel.org> wrote:

> This topic would be around the way the various subsystems interact, in the
> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> crossing with output devices but maybe that's just over the next hill!)
>
> Scope may well be wider but includes:
> * input (some of it)
> * hwmon
> * iio
> * comedi(?)
> * thermal
> * power/battery
> * gpio - the blur between gpios and beaglescope / PLC type I/O.

I'm in. The new chardev ABI was designed to fit play/well with IIO,
for example I added a 64bit timestamp also to GPIO events so that
people could do what I guess they refer to as sensor fusion mixing
and blending sensor input events with GPIO events.
(https://en.wikipedia.org/wiki/Sensor_fusion)

What I wanted to bring up ASAP was when I noticed in linux-next
commit bc2b7dab629a51e8beb5fda4222c62a23b729f26
"iio:core: timestamping clock selection support"
that we should probably move this code to e.g. drivers/base
or drivers/lib so I can reuse the same ABI in sysfs for
GPIO event timestamping.

I think timestamping events uniformly is something that is
important to all userspaces.

> Dmitry and I often have to decide on the IIO / input scope boundary
> (and we've moved it over time) + a number of cross over drivers have
> turned up recently.

(snip from elsewhere)

> Is it ever sensible to have two drivers for the same part because
> the use cases are so different? (I hope there is always a better way
> but maybe not)

I have one on my knee as I want to make a proper IIO driver
for MPU-3050 gyroscope which has a (very simplified) driver in
drivers/input/misc/mpu3050.c that was merged before the advent
of the IIO subsystem.

For the new driver I have these options:

- Keep both drivers and consider them for different usecases.
  of the same hardware that require different Kconfig (seems
  very unelegant to me)

- Add some input event calls into the IIO driver so we can
  phase over the older uses to the new driver.

- Drop the input event handling since gyroscopes should not
  be handled by input anyway

- Develop a generic IIO-gyroscope-to-input event bridge.
  (Phew seems complex.)

(elsewhere)
> IIO-input has been partly held outside mainline for years by this issue.

Is there such a thing? gitweb?

> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
> stand?

IETF has this motto "rough consensus and running code", and
for me that is the working assumption to most things
driver-subsystem-division-of-responsibilities-related.

> We have had quite a few cases where a whole driver is submitted to the
> 'wrong' subsystem. Not the best first interaction with the kernel
> world for new contributors. (This sort of area is attracts newbies
> as it's relatively simple and the hardware is fairly cheap and more
> than once their first experience with review as exactly this!)

What is sometimes lacking is users who are clever enough to
take the latest kernel (or preferably even -next) and git grep for
a few strings related to the hardware they want to support, and
be revealed the fact that there is already a driver for it.

If there isn't, their problem is that of finding the right subsystem
to pigeon-hole their hardware into. It typically requires a kernel
old grumpy person saying "no, THIS thing goes THERE, THAT
thing goes THERE". Unfortunately I think this is pretty hard to
codify.

> If we aren't happy, what do we do about it?

More helpful kernel generalists...

> GPIO is another interesting case - a lot of hardware is capable of
> parallel sampling, some at high speeds. It's another area that
> is probably too specialist for this discussion, but if people want
> to dive into the details it might be interesting.

I currently have (for v4.8) an ABI which makes i possible to e.g.
read 32 simultaneous lines with one userspace/kernelspace
swap, if it ends up as a readl() from a register.

The userspace ABI has a limitation to 64 values, so it would be
possible to have some driver read 64 GPIO lines and return them
in a u8 vector back to userspace in one go.

For an oscilloscope or signal analyzer usecase with more than
64 channels needing to be sampled simultaneously, we may need
to add another "large GPIO read" ABI, and also probably a new
driver API interface to achieve it. I've never seen hardware that can
read more values in parallel than 32 though, I chose 64 to get some
headroom.

> Linus Waleij (gpio)

Sure, if travels etc work out and I get invited I'd come running.

Yours,
Linus Walleij

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 22/07/16 06:18, Torokhov wrote:
> On Thu, Jul 21, 2016 at 08:29:34PM -0700, Guenter Roeck wrote:
>> On 07/20/2016 02:18 PM, Jonathan Cameron wrote:
>>> Hi All,
>>>
>>> This topic would be around the way the various subsystems interact, in the
>>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>>> crossing with output devices but maybe that's just over the next hill!)
>>>
>>> Scope may well be wider but includes:
>>> * input (some of it)
>>> * hwmon
>>> * iio
>>> * comedi(?)
>>> * thermal
>>> * power/battery
>>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
>>> * v4l - when does a device jump from being a multi pixel thermopile
>>> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
>>> sensors (effectively 9ish pixel cameras)
>>> * Lots of random things we haven't seen yet.
>>>
>>> All these make use of devices that are at their hearts ADCs. Whilst it
>>> would nice if the underlying devices always fell into one clear
>>> category that is not always the case. Hardware descriptions (e.g.
>>> device tree) are an important area of contention.
>>>
>>> The examples that follow are mostly based around IIO interactions with
>>> hwmon / input as that's my area of expertise. However, I know that
>>> similar issues occur at other boundaries.
>>>
>>> Guenter Roeck has been working on hwmon / thermal interfacing recently.
>>> We also regularly push drivers one way or the other around that boundary.
>>>
>>> Dmitry and I often have to decide on the IIO / input scope boundary
>>> (and we've moved it over time) + a number of cross over drivers have
>>> turned up recently.
>>>
>>> The other subsystems I encounter have been 'quieter' in their
>>> interactions with me but many of the issues below apply to them as well.
>>>
>>> A rough list of related topics that might be worth face to face
>>> discussion includes:
>>>
>>> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
>>> stand?
>>>
>>> I am personally happy enough with this, but it's worth checking
>>
>> Same here.
>>
>>> everyone else is. This probably causes more pain for new submitters
>>> than it does for old hands. Perhaps we can come up with a checklist style
>>> doc to save people having to ask.
>>>
>> I think that would be very useful.
>>
>>> We have had quite a few cases where a whole driver is submitted to the
>>> 'wrong' subsystem. Not the best first interaction with the kernel
>>
>> ... especially to avoid that situation.
>>
>>> world for new contributors. (This sort of area is attracts newbies
>>> as it's relatively simple and the hardware is fairly cheap and more
>>> than once their first experience with review as exactly this!)
>>>
>>> If we aren't happy, what do we do about it?
>>>
>>> I'm hoping we are happy or, at least, resigned to the current state
>>> of affairs, but think the question should be asked from time to time
>>> as the answer may well change + the discussion will highlight pinch
>>> points we can work on.
>>>
>>> 2) Bridging drivers - there will always be cross over cases.
>>> E.g. Generic ADCs used for battery voltage monitoring.  We have a
>>> number of bridges in mainline already and others out of tree.  How
>>> important are these and what features would make them more useful?
>>>
>>> Is it ever sensible to have two drivers for the same part because
>>> the use cases are so different? (I hope there is always a better way
>>> but maybe not)
>>>
>>> This covers both generic bridges (e.g. iio-hwmon) and device
>>> specific bridges (a good example of a touch screen driver turned
>>> up in my inbox yesterday).
>>>
>>> 3) Bindings (device tree and similar).  When it is appropriate to use a
>>> device tree to describe the overlap (bridging of channels)?
>>> Sometimes there is obvious real hardware involved (a thermistor on a
>>> generic ADC input for example), but there are many grey areas.
>>>
>>> Finally bashing out an answer to the issue device tree maintainers
>>> have with the iio-hwmon bindings would be great.
>>
>> Agreed, it would be great if we can come up with a fix for that.
>>
>>> On a less biased note, that example is pretty fiddly to define but would
>>> act as a good basis to work from more generally.
>>>
>>> We know we got it wrong (back in the dark ages), but it isn't obvious how
>>> to get it right!
>>>
>>> IIO-input has been partly held outside mainline for years by this issue.
>>>
>>> Also some practical design decisions to make around how to implement
>>> these mappings to work best with deferred probing etc.
>>>
>>> 4) Should we drop all the bindings for bridging between such subsystems
>>> and do it all from userspace?
>>>
>> I think that would be a terrible idea. My hope is that we should be able
>> to present a system to the user as it is intended to be, and that should include
>> the use case for given hardware. An ADC is not just an ADC, it has a use case.
>> The use case may be obvious if it is used as input device, but if it is
>> used to measure a voltage (or current, or temperature) it should be possible
>> to express that as well, without having to have userspace involved.
> 
> Totally agree. We might want to allow reconfiguring from userspace,
> but having initial configuration conveyed through DT/ACPI/whatever
> according to the original system purpose makes sense to me.
Agreed. There may be exceptions, but in general we need bindings or
similar.  The trick is how to make them both generic enough that they
describe the hardware, and specific enough that we can easily use the
bindings to make the relevant connections.

iio-hwmon always got grief for being Linux specific. Going as generic
as this channel is a temperature sensor, this one is a power line
etc makes for fiddly bindings as it's not clear what driver's problem
they are.  There is probably a middle ground we need to tread here.

There are plenty of single devices that have multiple uses as well
some of which require very different handling.  e.g. Accelerometer in
a laptop.  Does screen orientation, but may also handle parking
a hard disk head during freefall, or even various augmented reality
stuff (e.g. human input).

Jonathan
> 
> Thanks.
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 22/07/16 14:04, Linus Walleij wrote:
> On Wed, Jul 20, 2016 at 11:18 PM, Jonathan Cameron <jic23@kernel.org> wrote:
> 
>> This topic would be around the way the various subsystems interact, in the
>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>> crossing with output devices but maybe that's just over the next hill!)
>>
>> Scope may well be wider but includes:
>> * input (some of it)
>> * hwmon
>> * iio
>> * comedi(?)
>> * thermal
>> * power/battery
>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
> 
> I'm in. The new chardev ABI was designed to fit play/well with IIO,
> for example I added a 64bit timestamp also to GPIO events so that
> people could do what I guess they refer to as sensor fusion mixing
> and blending sensor input events with GPIO events.
> (https://en.wikipedia.org/wiki/Sensor_fusion)
Great.
> 
> What I wanted to bring up ASAP was when I noticed in linux-next
> commit bc2b7dab629a51e8beb5fda4222c62a23b729f26
> "iio:core: timestamping clock selection support"
> that we should probably move this code to e.g. drivers/base
> or drivers/lib so I can reuse the same ABI in sysfs for
> GPIO event timestamping.
> 
> I think timestamping events uniformly is something that is
> important to all userspaces.
Seems sensible to me. 
Should be easy enough to move IIO over to a shared implementation without
ABI breakage.

> 
>> Dmitry and I often have to decide on the IIO / input scope boundary
>> (and we've moved it over time) + a number of cross over drivers have
>> turned up recently.
> 
> (snip from elsewhere)
> 
>> Is it ever sensible to have two drivers for the same part because
>> the use cases are so different? (I hope there is always a better way
>> but maybe not)
> 
> I have one on my knee as I want to make a proper IIO driver
> for MPU-3050 gyroscope which has a (very simplified) driver in
> drivers/input/misc/mpu3050.c that was merged before the advent
> of the IIO subsystem.
> 
> For the new driver I have these options:
> 
> - Keep both drivers and consider them for different usecases.
>   of the same hardware that require different Kconfig (seems
>   very unelegant to me)
Agreed. 
> 
> - Add some input event calls into the IIO driver so we can
>   phase over the older uses to the new driver.
I'd push back on this probably, though if the old driver has
is in heavy use we'd have little choice about maintaining the ABI for
a good long time.  So whilst it would feel nasty, it might be the only
way to keep an 'identical' ABI in place.
> 
> - Drop the input event handling since gyroscopes should not
>   be handled by input anyway
Would break userspace unfortunately...
> 
> - Develop a generic IIO-gyroscope-to-input event bridge.
>   (Phew seems complex.)

Not that hard actually - see the below.  The infrastructure to
do this stuff went in a long time back, but we are only just seeing
mainline users showing up.  Given we get bug fixes for that core
code from time to time, presumably there are non mainline users.
The core rewrite that led to that was very useful in making us tidy
some corners up anyway.

> 
> (elsewhere)
>> IIO-input has been partly held outside mainline for years by this issue.
> 
> Is there such a thing? gitweb?
Umm. Hides in old emails...
http://www.spinics.net/lists/linux-iio/msg06881.html
Hohum. Nearly 4 years since I revised that oops.

> 
>> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
>> stand?
> 
> IETF has this motto "rough consensus and running code", and
> for me that is the working assumption to most things
> driver-subsystem-division-of-responsibilities-related.
> 
>> We have had quite a few cases where a whole driver is submitted to the
>> 'wrong' subsystem. Not the best first interaction with the kernel
>> world for new contributors. (This sort of area is attracts newbies
>> as it's relatively simple and the hardware is fairly cheap and more
>> than once their first experience with review as exactly this!)
> 
> What is sometimes lacking is users who are clever enough to
> take the latest kernel (or preferably even -next) and git grep for
> a few strings related to the hardware they want to support, and
> be revealed the fact that there is already a driver for it.
> 
> If there isn't, their problem is that of finding the right subsystem
> to pigeon-hole their hardware into. It typically requires a kernel
> old grumpy person saying "no, THIS thing goes THERE, THAT
> thing goes THERE". Unfortunately I think this is pretty hard to
> codify.
> 
>> If we aren't happy, what do we do about it?
> 
> More helpful kernel generalists...
:)
> 
>> GPIO is another interesting case - a lot of hardware is capable of
>> parallel sampling, some at high speeds. It's another area that
>> is probably too specialist for this discussion, but if people want
>> to dive into the details it might be interesting.
> 
> I currently have (for v4.8) an ABI which makes i possible to e.g.
> read 32 simultaneous lines with one userspace/kernelspace
> swap, if it ends up as a readl() from a register.
Cool. I've kind of lost track of where you were going with that.
> 
> The userspace ABI has a limitation to 64 values, so it would be
> possible to have some driver read 64 GPIO lines and return them
> in a u8 vector back to userspace in one go.
> 
> For an oscilloscope or signal analyzer usecase with more than
> 64 channels needing to be sampled simultaneously, we may need
> to add another "large GPIO read" ABI, and also probably a new
> driver API interface to achieve it. I've never seen hardware that can
> read more values in parallel than 32 though, I chose 64 to get some
> headroom.
Very nice.

That gets a long way for SoC gpio type cases, but there is also the
class of external fpga or similar devices with their own buffering.
This stuff fits well with the work Lars did on DMA buffers in IIO.
The PRU stuff in beagle logic falls into the same category.

Also plenty of simple devices have a few inputs on them sampled in
parallel with their 'ADC' channels, some of these have hardware fifos
etc to make life more interesting.  Reading isn't low latency, but
it is high frequency.

Now neither of these are really GPIOs and probably shouldn't be
treated as such.  However, clearly there is a blurred line between the
two. Particularly when we consider triggered polling of GPIOs.
At somepoint I want to look at buffer fusion for IIO.  Can do it
now with a driver consuming multiple IIO buffers, but it would be
ugly. (By this I merely mean fusing the data streams from multiple
devices running of one sampling trigger into a single stream, not
an clever data processing)
> 
>> Linus Waleij (gpio)
> 
> Sure, if travels etc work out and I get invited I'd come running.
Fingers crossed then!

Jonathan
> 
> Yours,
> Linus Walleij
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 22/07/16 05:29, Guenter Roeck wrote:
> On 07/20/2016 02:18 PM, Jonathan Cameron wrote:
>> Hi All,
>>
>> This topic would be around the way the various subsystems interact, in the
>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>> crossing with output devices but maybe that's just over the next hill!)
>>
>> Scope may well be wider but includes:
>> * input (some of it)
>> * hwmon
>> * iio
>> * comedi(?)
>> * thermal
>> * power/battery
>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
>> * v4l - when does a device jump from being a multi pixel thermopile
>> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
>> sensors (effectively 9ish pixel cameras)
>> * Lots of random things we haven't seen yet.
>>
>> All these make use of devices that are at their hearts ADCs. Whilst it
>> would nice if the underlying devices always fell into one clear
>> category that is not always the case. Hardware descriptions (e.g.
>> device tree) are an important area of contention.
>>
>> The examples that follow are mostly based around IIO interactions with
>> hwmon / input as that's my area of expertise. However, I know that
>> similar issues occur at other boundaries.
>>
>> Guenter Roeck has been working on hwmon / thermal interfacing recently.
>> We also regularly push drivers one way or the other around that boundary.
>>
>> Dmitry and I often have to decide on the IIO / input scope boundary
>> (and we've moved it over time) + a number of cross over drivers have
>> turned up recently.
>>
>> The other subsystems I encounter have been 'quieter' in their
>> interactions with me but many of the issues below apply to them as well.
>>
>> A rough list of related topics that might be worth face to face
>> discussion includes:
>>
>> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
>> stand?
>>
>> I am personally happy enough with this, but it's worth checking
> 
> Same here.
> 
>> everyone else is. This probably causes more pain for new submitters
>> than it does for old hands. Perhaps we can come up with a checklist style
>> doc to save people having to ask.
>>
> I think that would be very useful.
This is one where we can perhaps bash out a rough version in advance
to be talked over /refined in person.
> 
>> We have had quite a few cases where a whole driver is submitted to the
>> 'wrong' subsystem. Not the best first interaction with the kernel
> 
> ... especially to avoid that situation.
Agreed. I feel guilty every time it happens....
> 
>> world for new contributors. (This sort of area is attracts newbies
>> as it's relatively simple and the hardware is fairly cheap and more
>> than once their first experience with review as exactly this!)
>>
>> If we aren't happy, what do we do about it?
>>
>> I'm hoping we are happy or, at least, resigned to the current state
>> of affairs, but think the question should be asked from time to time
>> as the answer may well change + the discussion will highlight pinch
>> points we can work on.
>>
>> 2) Bridging drivers - there will always be cross over cases.
>> E.g. Generic ADCs used for battery voltage monitoring.  We have a
>> number of bridges in mainline already and others out of tree.  How
>> important are these and what features would make them more useful?
>>
>> Is it ever sensible to have two drivers for the same part because
>> the use cases are so different? (I hope there is always a better way
>> but maybe not)
>>
>> This covers both generic bridges (e.g. iio-hwmon) and device
>> specific bridges (a good example of a touch screen driver turned
>> up in my inbox yesterday).
>>
>> 3) Bindings (device tree and similar).  When it is appropriate to use a
>> device tree to describe the overlap (bridging of channels)?
>> Sometimes there is obvious real hardware involved (a thermistor on a
>> generic ADC input for example), but there are many grey areas.
>>
>> Finally bashing out an answer to the issue device tree maintainers
>> have with the iio-hwmon bindings would be great.
> 
> Agreed, it would be great if we can come up with a fix for that.
> 
>> On a less biased note, that example is pretty fiddly to define but would
>> act as a good basis to work from more generally.
>>
>> We know we got it wrong (back in the dark ages), but it isn't obvious how
>> to get it right!
>>
>> IIO-input has been partly held outside mainline for years by this issue.
>>
>> Also some practical design decisions to make around how to implement
>> these mappings to work best with deferred probing etc.
>>
>> 4) Should we drop all the bindings for bridging between such subsystems
>> and do it all from userspace?
>>
> I think that would be a terrible idea. My hope is that we should be able
> to present a system to the user as it is intended to be, and that should include
> the use case for given hardware. An ADC is not just an ADC, it has a use case.
> The use case may be obvious if it is used as input device, but if it is
> used to measure a voltage (or current, or temperature) it should be possible
> to express that as well, without having to have userspace involved.
> 
>> I think we may end up with a hybrid of the two, but need to be able to
>> make it work in 'standard' cases without userspace being involved. That
>> hybrid solution may well be devicetree overlay based... Unclear so far.
>> + plenty of crazy things that 'might' make sense where there isn't even
>> the pretence of representing real hardware.
>>
> Not really sure if/how that would require overlays as long as the use case
> is static. If an ADC is used to measure a voltage on a given system, it is
> unlikely to be used for something else in the same system.
So you'd think ;)  Lots of IIO usecases include ADCs measuring voltages on
screw terminals and similar.  A surprising number of industrial systems
are build around generic boards, wired in weird and wonderful ways.  They
still want their 'hyrda' to work sensibly with standard kernels and present
data in a way that makes sense.  So probably a generic board with the
mappings loaded via a device tree overlay.  They just mess around with the 
overlay.


> 
>> 5) Complex device interaction usecases.  At the moment the ones I've come
>> across are mostly contained in IIO.  The moment we start sticking in
>> MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
>> mix it can get fiddly.  Swapping war stories may well be worthwhile on
>> this. This stuff also turns up in ASoC for example so probably lessons
>> to be learned from there.
>>
>> The analog devices software defined radios are another possible case
>> study.
>>
>> There of course may well be lessons to be learned from similar
>> interactions elsewhere in the kernel.
>>
>> There is a lot of history in how we ended up where we are (it all made
>> absolute sense at the time). Sitting back and taking the time
>> to discuss the future would be great.  Whilst this might be solvable
>> by email we've made no definitive progress for years
>> (and what has been made has been on a case by case basis deep in
>> driver reviews.)
>>
>> I threw comedi in the list above but, at the moment, I think the more
>> likely direction there is a single userspace library abstracting
>> the underlying subsystem (Analog Devices are working in that
>> direction - perhaps Lars can offer more on that?).
>>
>> GPIO is another interesting case - a lot of hardware is capable of
>> parallel sampling, some at high speeds. It's another area that
>> is probably too specialist for this discussion, but if people want
>> to dive into the details it might be interesting.
>>
>> I think we have only a small amount of fuzz around the v4l boundary,
>> but wanted to leave the door open if anyone wants to discuss that
>> one further as it's come up a few times over recent years.
>>
>> The SoC world is a major case of one device, many uses.  Some SoCs
>> are turning up with multiple ADC units, sometimes with different
>> designs, sometimes simply so that the same hardware can be used for different things.
>>
>> A few suggestions for people:
>>
>> Guenter Roeck (hmwon)
>> Dmitry Torokhov (input)
>> Me (Jonathan Cameron - IIO)
>>
>> Someone thermal related (not sure who would be most interested)
>> Someone power supply related? (Guenter, any suggestions?)
> 
> Not really, other than the power subsystem maintainers. Question here
> might be if it would make sense to register battery charger drivers as
> hardware monitoring devices - with the new hwmon API that should be quite
> straightforward.
Anyone want to nominate themselves from that area?
> 
> I keep thinking about power, but regulators are probably at least as
> relevant when it comes to interaction with hardware monitoring.
> Many PMBus chips have both hardware monitoring and voltage/current
> regulator functionality. The PMBus core implements some basic
> functionality along that line, and can register a chip as regulator
> if requested. Would it make sense to register regulators as hardware
> monitoring devices ? I don't really know, and never thought about it
> until now. Maybe Mark Brown and/or Liam Girdwood have some thoughts
> on this.
> 
> Thanks,
> Guenter
> 
>>
>> Someone device tree related (Mark Rutland? Rob Herring?)
>>
>> Linus Waleij (gpio)
>>
>> Mark Brown - as some of concepts of IIO bridging and how far it
>> can be taken (everything via a bridge driver) came from
>> discussions with him on SoC ADC handling.
>>
>> Lar-Peter Clausen for generally doing insane things with ADCs + having
>> one foot in the userspace side of things.
>>
>> Thanks,
>>
>> Jonathan Cameron
>> IIO Maintainer
>>
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 21/07/16 09:39, Hans Verkuil wrote:
> 
> 
> On 07/20/2016 11:18 PM, Jonathan Cameron wrote:
>> Hi All,
>>
>> This topic would be around the way the various subsystems interact, in the
>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>> crossing with output devices but maybe that's just over the next hill!)
> 
> This happens regularly with drm/kms vs v4l, especially for embedded devices
> where video inputs and outputs are often connected internally (i.e. paths
> straight from video capture to video output without needing to go to memory
> first).
> 
>>
>> Scope may well be wider but includes:
>> * input (some of it)
>> * hwmon
>> * iio
>> * comedi(?)
>> * thermal
>> * power/battery
>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
>> * v4l - when does a device jump from being a multi pixel thermopile
>> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
>> sensors (effectively 9ish pixel cameras)
> 
> See for an example of that this patch series that adds support for v4l-touch
> devices:
> 
> http://www.spinics.net/lists/linux-input/msg45918.html
Certainly interesting - never occurred to me that you could get data where
that made sense from touch devices.
> 
>> * Lots of random things we haven't seen yet.
>>
>> All these make use of devices that are at their hearts ADCs. Whilst it
>> would nice if the underlying devices always fell into one clear
>> category that is not always the case. Hardware descriptions (e.g.
>> device tree) are an important area of contention.
>>
>> The examples that follow are mostly based around IIO interactions with
>> hwmon / input as that's my area of expertise. However, I know that
>> similar issues occur at other boundaries.
>>
>> Guenter Roeck has been working on hwmon / thermal interfacing recently.
>> We also regularly push drivers one way or the other around that boundary.
>>
>> Dmitry and I often have to decide on the IIO / input scope boundary
>> (and we've moved it over time) + a number of cross over drivers have
>> turned up recently.
>>
>> The other subsystems I encounter have been 'quieter' in their
>> interactions with me but many of the issues below apply to them as well.
>>
>> A rough list of related topics that might be worth face to face
>> discussion includes:
>>
>> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
>> stand?
>>
>> I am personally happy enough with this, but it's worth checking
>> everyone else is. This probably causes more pain for new submitters
>> than it does for old hands. Perhaps we can come up with a checklist style
>> doc to save people having to ask.
>>  
>> We have had quite a few cases where a whole driver is submitted to the
>> 'wrong' subsystem. Not the best first interaction with the kernel
>> world for new contributors. (This sort of area is attracts newbies
>> as it's relatively simple and the hardware is fairly cheap and more
>> than once their first experience with review as exactly this!)
>>
>> If we aren't happy, what do we do about it?
>>
>> I'm hoping we are happy or, at least, resigned to the current state
>> of affairs, but think the question should be asked from time to time
>> as the answer may well change + the discussion will highlight pinch
>> points we can work on.
>>
>> 2) Bridging drivers - there will always be cross over cases.
>> E.g. Generic ADCs used for battery voltage monitoring.  We have a
>> number of bridges in mainline already and others out of tree.  How
>> important are these and what features would make them more useful?
>>
>> Is it ever sensible to have two drivers for the same part because
>> the use cases are so different? (I hope there is always a better way
>> but maybe not)
> 
> One example of this is video output: both drm/kms and v4l can do video
> output. Video output using v4l is specifically geared towards frame-based
> video output whereas drm/kms is for desktop environments/GPUs. Luckily there
> are very few v4l video output drivers and the driver duplication occurs
> only for the i2c video transmitter devices. I know of two that are duplicated:
> adv7511 and (I think) saa7127. The rarity is the reason nobody ever created
> a shim or something similar to allow for one i2c driver to serve both
> subsystems. Technically this would be perfectly possible, it's just work.
So the balance here (at least when we aren't doing the work) is how
much maintainers push back to get this sort of work done before taking
a driver.  Always balances to be struck.
> 
>> This covers both generic bridges (e.g. iio-hwmon) and device
>> specific bridges (a good example of a touch screen driver turned
>> up in my inbox yesterday).
>>
>> 3) Bindings (device tree and similar).  When it is appropriate to use a 
>> device tree to describe the overlap (bridging of channels)?
>> Sometimes there is obvious real hardware involved (a thermistor on a
>> generic ADC input for example), but there are many grey areas.
>>
>> Finally bashing out an answer to the issue device tree maintainers
>> have with the iio-hwmon bindings would be great. 
>> On a less biased note, that example is pretty fiddly to define but would
>> act as a good basis to work from more generally.
>>
>> We know we got it wrong (back in the dark ages), but it isn't obvious how
>> to get it right!
>>
>> IIO-input has been partly held outside mainline for years by this issue.
>>
>> Also some practical design decisions to make around how to implement
>> these mappings to work best with deferred probing etc.
>>
>> 4) Should we drop all the bindings for bridging between such subsystems
>> and do it all from userspace?
>>
>> I think we may end up with a hybrid of the two, but need to be able to
>> make it work in 'standard' cases without userspace being involved. That
>> hybrid solution may well be devicetree overlay based... Unclear so far.
>> + plenty of crazy things that 'might' make sense where there isn't even
>> the pretence of representing real hardware.
>>
>> 5) Complex device interaction usecases.  At the moment the ones I've come
>> across are mostly contained in IIO.  The moment we start sticking in
>> MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
>> mix it can get fiddly.  Swapping war stories may well be worthwhile on
>> this. This stuff also turns up in ASoC for example so probably lessons
>> to be learned from there.
> 
> V4L has a long history of this: if you have multiple video scalers in your
> pipeline, how do you control them? E.g. there can be scalers in the video
> sensor and in the SoC. We came up with the media controller device to
> expose the topology of the hardware and to be able to control each bit
> directly from userspace.
> 
> While the media controller is quite powerful and can be used to expose
> connections across subsystems, it also requires more work from userspace,
> and that's an area where not enough attention has been given.
Agreed.  When I've discussed this with Lars-Peter who has been looking
at expanding the scope of the media control framework into IIO he has
suggested that it makes a lot of sense for DSP type configurations, but
perhaps not simple stuff like what is on the end of a wire...

Definitely another thing to include in any discussion though.
> 
>> The analog devices software defined radios are another possible case
>> study.
> 
> Note that the V4L2 subsystem has support for several SDR devices. Probably
> another area where drivers can end up in different subsystems.
> 
>>
>> There of course may well be lessons to be learned from similar
>> interactions elsewhere in the kernel.
>>
>> There is a lot of history in how we ended up where we are (it all made
>> absolute sense at the time). Sitting back and taking the time
>> to discuss the future would be great.  Whilst this might be solvable
>> by email we've made no definitive progress for years
>> (and what has been made has been on a case by case basis deep in
>> driver reviews.)
>>
>> I threw comedi in the list above but, at the moment, I think the more
>> likely direction there is a single userspace library abstracting
>> the underlying subsystem (Analog Devices are working in that
>> direction - perhaps Lars can offer more on that?).
> 
> When you get complex systems you probably have to move in that direction.
> I'd be interesting in such work going on in other subsystems. As mentioned
> above, the media controller needs something like that and it would be
> good to learn from others.
> 
>> GPIO is another interesting case - a lot of hardware is capable of
>> parallel sampling, some at high speeds. It's another area that
>> is probably too specialist for this discussion, but if people want
>> to dive into the details it might be interesting.
>>
>> I think we have only a small amount of fuzz around the v4l boundary,
>> but wanted to leave the door open if anyone wants to discuss that
>> one further as it's come up a few times over recent years.
> 
> There is more of that than we wish, but quite often it is just dropped
> in the end as being too much work to solve and not quite important enough
> to spend the time and money on.
Sounds like I underestimate how fiddly the boundaries between subsystems
I haven't often interacted with are!  Thanks for the info.
> 
>> The SoC world is a major case of one device, many uses.  Some SoCs
>> are turning up with multiple ADC units, sometimes with different
>> designs, sometimes simply so that the same hardware can be used for different things.
>>
>> A few suggestions for people:
>>
>> Guenter Roeck (hmwon)
>> Dmitry Torokhov (input)
>> Me (Jonathan Cameron - IIO)
>>
>> Someone thermal related (not sure who would be most interested)
>> Someone power supply related? (Guenter, any suggestions?)
>>
>> Someone device tree related (Mark Rutland? Rob Herring?)
>>
>> Linus Waleij (gpio)
>>
>> Mark Brown - as some of concepts of IIO bridging and how far it
>> can be taken (everything via a bridge driver) came from
>> discussions with him on SoC ADC handling.
>>
>> Lar-Peter Clausen for generally doing insane things with ADCs + having
>> one foot in the userspace side of things.
> 
> Laurent Pinchart or myself for v4l/mc
Cool.
> 
> Regards,
> 
> 	Hans
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Sebastian Reichel]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

Hi,

On Fri, Jul 22, 2016 at 09:31:15PM +0200, Jonathan Cameron wrote:
> >> Someone power supply related? (Guenter, any suggestions?)
> > Not really, other than the power subsystem maintainers. Question
> > here might be if it would make sense to register battery charger
> > drivers as hardware monitoring devices - with the new hwmon API
> > that should be quite straightforward.
> Anyone want to nominate themselves from that area?

Sorry, travelling to USA is not something I can financially afford
until I have found an employer. Since [0] I basically maintained
the power supply subsystem alone, so Dmitry and David are probably
not the best candidates either.

If he's there, Krzysztof Kozlowski (added to CC) may be a good candidate,
though.

[0] https://lkml.org/lkml/2014/7/16/621

-- Sebastian

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Vinod Koul]

Hi Jonathan,

On Wed, Jul 20, 2016 at 10:18:11PM +0100, Jonathan Cameron wrote:
> 
> 5) Complex device interaction usecases.  At the moment the ones I've come
> across are mostly contained in IIO.  The moment we start sticking in
> MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
> mix it can get fiddly.  Swapping war stories may well be worthwhile on
> this. This stuff also turns up in ASoC for example so probably lessons
> to be learned from there.
> 
> The analog devices software defined radios are another possible case
> study.
> 
> There of course may well be lessons to be learned from similar
> interactions elsewhere in the kernel.
> 
> There is a lot of history in how we ended up where we are (it all made
> absolute sense at the time). Sitting back and taking the time
> to discuss the future would be great.  Whilst this might be solvable
> by email we've made no definitive progress for years
> (and what has been made has been on a case by case basis deep in
> driver reviews.)
> 
> I threw comedi in the list above but, at the moment, I think the more
> likely direction there is a single userspace library abstracting
> the underlying subsystem (Analog Devices are working in that
> direction - perhaps Lars can offer more on that?).
> 
> GPIO is another interesting case - a lot of hardware is capable of
> parallel sampling, some at high speeds. It's another area that
> is probably too specialist for this discussion, but if people want
> to dive into the details it might be interesting.
> 
> I think we have only a small amount of fuzz around the v4l boundary,
> but wanted to leave the door open if anyone wants to discuss that
> one further as it's come up a few times over recent years.
> 
> The SoC world is a major case of one device, many uses.  Some SoCs
> are turning up with multiple ADC units, sometimes with different
> designs, sometimes simply so that the same hardware can be used for different things.

I would be interested in this discussion. Complex device seem to be getting
more complex over here as well!

Thanks
-- 
~Vinod

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Mauro Carvalho Chehab]

Em Wed, 20 Jul 2016 22:18:11 +0100
Jonathan Cameron <jic23@kernel.org> escreveu:

> Hi All,
> 
> This topic would be around the way the various subsystems interact, in the
> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> crossing with output devices but maybe that's just over the next hill!)

Yeah, there is a gray area here as devices become more complex.
So, I'm interested on such topic.

> Scope may well be wider but includes:
> * input (some of it)
> * hwmon
> * iio
> * comedi(?)
> * thermal
> * power/battery
> * gpio - the blur between gpios and beaglescope / PLC type I/O.
> * v4l - when does a device jump from being a multi pixel thermopile
> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
> sensors (effectively 9ish pixel cameras)

For devices that provide images somehow, I'd say that the best would be
to implement them as via the V4L2 API. We'll likely need to discuss it
case by case, specially for early cross-subsystem drivers, though, as
it is not trivial to identify the subsystem boundaries, and sometimes
multiple APIs from different subsystems is needed. Also, it is not
clear where such drivers would fit at the Kernel tree.

One interesting case is an input device driver with multi-finger
support (drivers/input/touchscreen/sur40.c). I suspect we'll have
more cases like that over time.

> * Lots of random things we haven't seen yet.

Thanks,
Mauro

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Laurent Pinchart]

Hi Jonathan,

On Wednesday 20 Jul 2016 22:18:11 Jonathan Cameron wrote:
> Hi All,
> 
> This topic would be around the way the various subsystems interact, in the
> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> crossing with output devices but maybe that's just over the next hill!)

It's already here :-) We have an API to control camera flash (as in light, not 
memory) in V4L2 and another one in the LED subsystem.

> Scope may well be wider but includes:
> * input (some of it)
> * hwmon
> * iio
> * comedi(?)
> * thermal
> * power/battery
> * gpio - the blur between gpios and beaglescope / PLC type I/O.
> * v4l - when does a device jump from being a multi pixel thermopile
> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
> sensors (effectively 9ish pixel cameras)
> * Lots of random things we haven't seen yet.
> 
> All these make use of devices that are at their hearts ADCs. Whilst it
> would nice if the underlying devices always fell into one clear
> category that is not always the case. Hardware descriptions (e.g.
> device tree) are an important area of contention.
> 
> The examples that follow are mostly based around IIO interactions with
> hwmon / input as that's my area of expertise. However, I know that
> similar issues occur at other boundaries.
> 
> Guenter Roeck has been working on hwmon / thermal interfacing recently.
> We also regularly push drivers one way or the other around that boundary.
> 
> Dmitry and I often have to decide on the IIO / input scope boundary
> (and we've moved it over time) + a number of cross over drivers have
> turned up recently.
> 
> The other subsystems I encounter have been 'quieter' in their
> interactions with me but many of the issues below apply to them as well.
> 
> A rough list of related topics that might be worth face to face
> discussion includes:
> 
> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
> stand?
> 
> I am personally happy enough with this, but it's worth checking
> everyone else is. This probably causes more pain for new submitters
> than it does for old hands. Perhaps we can come up with a checklist style
> doc to save people having to ask.
> 
> We have had quite a few cases where a whole driver is submitted to the
> 'wrong' subsystem. Not the best first interaction with the kernel
> world for new contributors. (This sort of area is attracts newbies
> as it's relatively simple and the hardware is fairly cheap and more
> than once their first experience with review as exactly this!)
> 
> If we aren't happy, what do we do about it?
> 
> I'm hoping we are happy or, at least, resigned to the current state
> of affairs, but think the question should be asked from time to time
> as the answer may well change + the discussion will highlight pinch
> points we can work on.
> 
> 2) Bridging drivers - there will always be cross over cases.
> E.g. Generic ADCs used for battery voltage monitoring.  We have a
> number of bridges in mainline already and others out of tree.  How
> important are these and what features would make them more useful?
> 
> Is it ever sensible to have two drivers for the same part because
> the use cases are so different? (I hope there is always a better way
> but maybe not)

I'd love to say no, but we have already hit that problem. There are two 
drivers for the ADV7511 HDMI encoder, one in the media subsystem for V4L2 
output devices (drivers/media/i2c/adv7511.c), and one in the DRM/KMS subsystem 
for display devices (drivers/gpu/drm/i2c/adv7511.c, being moved to 
drivers/gpu/drm/bridge/). Try compiling them both in the kernel and see how 
they race to probe the same device (enjoy the show, popcorns not included).

> This covers both generic bridges (e.g. iio-hwmon) and device
> specific bridges (a good example of a touch screen driver turned
> up in my inbox yesterday).
> 
> 3) Bindings (device tree and similar).  When it is appropriate to use a
> device tree to describe the overlap (bridging of channels)?
> Sometimes there is obvious real hardware involved (a thermistor on a
> generic ADC input for example), but there are many grey areas.
> 
> Finally bashing out an answer to the issue device tree maintainers
> have with the iio-hwmon bindings would be great.
> On a less biased note, that example is pretty fiddly to define but would
> act as a good basis to work from more generally.
> 
> We know we got it wrong (back in the dark ages), but it isn't obvious how
> to get it right!
> 
> IIO-input has been partly held outside mainline for years by this issue.
> 
> Also some practical design decisions to make around how to implement
> these mappings to work best with deferred probing etc.
> 
> 4) Should we drop all the bindings for bridging between such subsystems
> and do it all from userspace?
> 
> I think we may end up with a hybrid of the two, but need to be able to
> make it work in 'standard' cases without userspace being involved. That
> hybrid solution may well be devicetree overlay based... Unclear so far.
> + plenty of crazy things that 'might' make sense where there isn't even
> the pretence of representing real hardware.
> 
> 5) Complex device interaction usecases.  At the moment the ones I've come
> across are mostly contained in IIO.  The moment we start sticking in
> MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
> mix it can get fiddly.  Swapping war stories may well be worthwhile on
> this. This stuff also turns up in ASoC for example so probably lessons
> to be learned from there.
> 
> The analog devices software defined radios are another possible case
> study.
> 
> There of course may well be lessons to be learned from similar
> interactions elsewhere in the kernel.
> 
> There is a lot of history in how we ended up where we are (it all made
> absolute sense at the time). Sitting back and taking the time
> to discuss the future would be great.  Whilst this might be solvable
> by email we've made no definitive progress for years
> (and what has been made has been on a case by case basis deep in
> driver reviews.)
> 
> I threw comedi in the list above but, at the moment, I think the more
> likely direction there is a single userspace library abstracting
> the underlying subsystem (Analog Devices are working in that
> direction - perhaps Lars can offer more on that?).
> 
> GPIO is another interesting case - a lot of hardware is capable of
> parallel sampling, some at high speeds. It's another area that
> is probably too specialist for this discussion, but if people want
> to dive into the details it might be interesting.
> 
> I think we have only a small amount of fuzz around the v4l boundary,
> but wanted to leave the door open if anyone wants to discuss that
> one further as it's come up a few times over recent years.

Don't forget to take system integration into account. If I given you a high-
speed ADC you will not think about V4L2 as your subsystem of choice. If the 
system designer has connected that ADC to a CPLD that generates fake 
horizontal and vertical sync signals, and connected the output to the camera 
interface of the SoC, you will be left with no choice but use the V4L2 API. 
That's largely a userspace issue in this case, but it implies that V4L2 need 
to define an "image format" for the ADC data.

> The SoC world is a major case of one device, many uses.  Some SoCs
> are turning up with multiple ADC units, sometimes with different
> designs, sometimes simply so that the same hardware can be used for
> different things.
> 
> A few suggestions for people:
> 
> Guenter Roeck (hmwon)
> Dmitry Torokhov (input)
> Me (Jonathan Cameron - IIO)
> 
> Someone thermal related (not sure who would be most interested)
> Someone power supply related? (Guenter, any suggestions?)
> 
> Someone device tree related (Mark Rutland? Rob Herring?)
> 
> Linus Waleij (gpio)
> 
> Mark Brown - as some of concepts of IIO bridging and how far it
> can be taken (everything via a bridge driver) came from
> discussions with him on SoC ADC handling.
> 
> Lar-Peter Clausen for generally doing insane things with ADCs + having
> one foot in the userspace side of things.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Laurent Pinchart]

On Thursday 28 Jul 2016 08:58:46 Mauro Carvalho Chehab wrote:
> Em Wed, 20 Jul 2016 22:18:11 +0100 Jonathan Cameron escreveu:
> > Hi All,
> > 
> > This topic would be around the way the various subsystems interact, in the
> > rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> > crossing with output devices but maybe that's just over the next hill!)
> 
> Yeah, there is a gray area here as devices become more complex.
> So, I'm interested on such topic.
> 
> > Scope may well be wider but includes:
> > * input (some of it)
> > * hwmon
> > * iio
> > * comedi(?)
> > * thermal
> > * power/battery
> > * gpio - the blur between gpios and beaglescope / PLC type I/O.
> > * v4l - when does a device jump from being a multi pixel thermopile
> > to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
> > sensors (effectively 9ish pixel cameras)
> 
> For devices that provide images somehow, I'd say that the best would be
> to implement them as via the V4L2 API.

This requires defining what an image is. Furthermore, we very low image 
resolution devices, we will need to deal with high frame rates (1k-10k is a 
range we need to consider). The V4L2 API will likely show performance issues.

> We'll likely need to discuss it case by case, specially for early cross-
> subsystem drivers, though, as it is not trivial to identify the subsystem
> boundaries, and sometimes multiple APIs from different subsystems is needed.
> Also, it is not clear where such drivers would fit at the Kernel tree.
> 
> One interesting case is an input device driver with multi-finger
> support (drivers/input/touchscreen/sur40.c). I suspect we'll have
> more cases like that over time.
> 
> > * Lots of random things we haven't seen yet.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Laurent Pinchart]

Hi Linus,

On Friday 22 Jul 2016 14:04:07 Linus Walleij wrote:
> On Wed, Jul 20, 2016 at 11:18 PM, Jonathan Cameron <jic23@kernel.org> wrote:
> > This topic would be around the way the various subsystems interact, in the
> > rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> > crossing with output devices but maybe that's just over the next hill!)
> > 
> > Scope may well be wider but includes:
> > * input (some of it)
> > * hwmon
> > * iio
> > * comedi(?)
> > * thermal
> > * power/battery
> > * gpio - the blur between gpios and beaglescope / PLC type I/O.
> 
> I'm in. The new chardev ABI was designed to fit play/well with IIO,
> for example I added a 64bit timestamp also to GPIO events so that
> people could do what I guess they refer to as sensor fusion mixing
> and blending sensor input events with GPIO events.
> (https://en.wikipedia.org/wiki/Sensor_fusion)
> 
> What I wanted to bring up ASAP was when I noticed in linux-next
> commit bc2b7dab629a51e8beb5fda4222c62a23b729f26
> "iio:core: timestamping clock selection support"
> that we should probably move this code to e.g. drivers/base
> or drivers/lib so I can reuse the same ABI in sysfs for
> GPIO event timestamping.
> 
> I think timestamping events uniformly is something that is
> important to all userspaces.
> 
> > Dmitry and I often have to decide on the IIO / input scope boundary
> > (and we've moved it over time) + a number of cross over drivers have
> > turned up recently.
> 
> (snip from elsewhere)
> 
> > Is it ever sensible to have two drivers for the same part because
> > the use cases are so different? (I hope there is always a better way
> > but maybe not)
> 
> I have one on my knee as I want to make a proper IIO driver
> for MPU-3050 gyroscope which has a (very simplified) driver in
> drivers/input/misc/mpu3050.c that was merged before the advent
> of the IIO subsystem.
> 
> For the new driver I have these options:
> 
> - Keep both drivers and consider them for different usecases.
>   of the same hardware that require different Kconfig (seems
>   very unelegant to me)
> 
> - Add some input event calls into the IIO driver so we can
>   phase over the older uses to the new driver.
> 
> - Drop the input event handling since gyroscopes should not
>   be handled by input anyway
> 
> - Develop a generic IIO-gyroscope-to-input event bridge.
>   (Phew seems complex.)
> 
> (elsewhere)
> 
> > IIO-input has been partly held outside mainline for years by this issue.
> 
> Is there such a thing? gitweb?
> 
> > 1) Are we happy with the somewhat adhoc divisions of subsystems as they
> > stand?
> 
> IETF has this motto "rough consensus and running code", and
> for me that is the working assumption to most things
> driver-subsystem-division-of-responsibilities-related.
> 
> > We have had quite a few cases where a whole driver is submitted to the
> > 'wrong' subsystem. Not the best first interaction with the kernel
> > world for new contributors. (This sort of area is attracts newbies
> > as it's relatively simple and the hardware is fairly cheap and more
> > than once their first experience with review as exactly this!)
> 
> What is sometimes lacking is users who are clever enough to
> take the latest kernel (or preferably even -next) and git grep for
> a few strings related to the hardware they want to support, and
> be revealed the fact that there is already a driver for it.
> 
> If there isn't, their problem is that of finding the right subsystem
> to pigeon-hole their hardware into. It typically requires a kernel
> old grumpy person saying "no, THIS thing goes THERE, THAT
> thing goes THERE". Unfortunately I think this is pretty hard to
> codify.
> 
> > If we aren't happy, what do we do about it?
> 
> More helpful kernel generalists...
> 
> > GPIO is another interesting case - a lot of hardware is capable of
> > parallel sampling, some at high speeds. It's another area that
> > is probably too specialist for this discussion, but if people want
> > to dive into the details it might be interesting.
> 
> I currently have (for v4.8) an ABI which makes i possible to e.g.
> read 32 simultaneous lines with one userspace/kernelspace
> swap, if it ends up as a readl() from a register.
> 
> The userspace ABI has a limitation to 64 values, so it would be
> possible to have some driver read 64 GPIO lines and return them
> in a u8 vector back to userspace in one go.
> 
> For an oscilloscope or signal analyzer usecase with more than
> 64 channels needing to be sampled simultaneously, we may need
> to add another "large GPIO read" ABI, and also probably a new
> driver API interface to achieve it. I've never seen hardware that can
> read more values in parallel than 32 though, I chose 64 to get some
> headroom.

Such a use case would put strong real-time constraints on the system, I wonder 
how useful a GPIO-based signal analyzer would be. Are you aware of any GPIO 
hardware that can be programmed to sample at a precise point in time (or 
perhaps even periodically) ?

> > Linus Waleij (gpio)
> 
> Sure, if travels etc work out and I get invited I'd come running.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Laurent Pinchart]

Hi Hans,

On Thursday 21 Jul 2016 09:39:04 Hans Verkuil wrote:
> On 07/20/2016 11:18 PM, Jonathan Cameron wrote:
> > Hi All,
> > 
> > This topic would be around the way the various subsystems interact, in the
> > rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> > crossing with output devices but maybe that's just over the next hill!)
> 
> This happens regularly with drm/kms vs v4l, especially for embedded devices
> where video inputs and outputs are often connected internally (i.e. paths
> straight from video capture to video output without needing to go to memory
> first).
> 
> > Scope may well be wider but includes:
> > * input (some of it)
> > * hwmon
> > * iio
> > * comedi(?)
> > * thermal
> > * power/battery
> > * gpio - the blur between gpios and beaglescope / PLC type I/O.
> > * v4l - when does a device jump from being a multi pixel thermopile
> > to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
> > sensors (effectively 9ish pixel cameras)
> 
> See for an example of that this patch series that adds support for v4l-touch
> devices:
> 
> http://www.spinics.net/lists/linux-input/msg45918.html
> 
> > * Lots of random things we haven't seen yet.
> > 
> > All these make use of devices that are at their hearts ADCs. Whilst it
> > would nice if the underlying devices always fell into one clear
> > category that is not always the case. Hardware descriptions (e.g.
> > device tree) are an important area of contention.
> > 
> > The examples that follow are mostly based around IIO interactions with
> > hwmon / input as that's my area of expertise. However, I know that
> > similar issues occur at other boundaries.
> > 
> > Guenter Roeck has been working on hwmon / thermal interfacing recently.
> > We also regularly push drivers one way or the other around that boundary.
> > 
> > Dmitry and I often have to decide on the IIO / input scope boundary
> > (and we've moved it over time) + a number of cross over drivers have
> > turned up recently.
> > 
> > The other subsystems I encounter have been 'quieter' in their
> > interactions with me but many of the issues below apply to them as well.
> > 
> > A rough list of related topics that might be worth face to face
> > discussion includes:
> > 
> > 1) Are we happy with the somewhat adhoc divisions of subsystems as they
> > stand?
> > 
> > I am personally happy enough with this, but it's worth checking
> > everyone else is. This probably causes more pain for new submitters
> > than it does for old hands. Perhaps we can come up with a checklist style
> > doc to save people having to ask.
> > 
> > We have had quite a few cases where a whole driver is submitted to the
> > 'wrong' subsystem. Not the best first interaction with the kernel
> > world for new contributors. (This sort of area is attracts newbies
> > as it's relatively simple and the hardware is fairly cheap and more
> > than once their first experience with review as exactly this!)
> > 
> > If we aren't happy, what do we do about it?
> > 
> > I'm hoping we are happy or, at least, resigned to the current state
> > of affairs, but think the question should be asked from time to time
> > as the answer may well change + the discussion will highlight pinch
> > points we can work on.
> > 
> > 2) Bridging drivers - there will always be cross over cases.
> > E.g. Generic ADCs used for battery voltage monitoring.  We have a
> > number of bridges in mainline already and others out of tree.  How
> > important are these and what features would make them more useful?
> > 
> > Is it ever sensible to have two drivers for the same part because
> > the use cases are so different? (I hope there is always a better way
> > but maybe not)
> 
> One example of this is video output: both drm/kms and v4l can do video
> output. Video output using v4l is specifically geared towards frame-based
> video output whereas drm/kms is for desktop environments/GPUs. Luckily there
> are very few v4l video output drivers and the driver duplication occurs
> only for the i2c video transmitter devices. I know of two that are
> duplicated: adv7511 and (I think) saa7127. The rarity is the reason nobody
> ever created a shim or something similar to allow for one i2c driver to
> serve both subsystems. Technically this would be perfectly possible, it's
> just work.
> > This covers both generic bridges (e.g. iio-hwmon) and device
> > specific bridges (a good example of a touch screen driver turned
> > up in my inbox yesterday).
> > 
> > 3) Bindings (device tree and similar).  When it is appropriate to use a
> > device tree to describe the overlap (bridging of channels)?
> > Sometimes there is obvious real hardware involved (a thermistor on a
> > generic ADC input for example), but there are many grey areas.
> > 
> > Finally bashing out an answer to the issue device tree maintainers
> > have with the iio-hwmon bindings would be great.
> > On a less biased note, that example is pretty fiddly to define but would
> > act as a good basis to work from more generally.
> > 
> > We know we got it wrong (back in the dark ages), but it isn't obvious how
> > to get it right!
> > 
> > IIO-input has been partly held outside mainline for years by this issue.
> > 
> > Also some practical design decisions to make around how to implement
> > these mappings to work best with deferred probing etc.
> > 
> > 4) Should we drop all the bindings for bridging between such subsystems
> > and do it all from userspace?
> > 
> > I think we may end up with a hybrid of the two, but need to be able to
> > make it work in 'standard' cases without userspace being involved. That
> > hybrid solution may well be devicetree overlay based... Unclear so far.
> > + plenty of crazy things that 'might' make sense where there isn't even
> > the pretence of representing real hardware.
> > 
> > 5) Complex device interaction usecases.  At the moment the ones I've come
> > across are mostly contained in IIO.  The moment we start sticking in
> > MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
> > mix it can get fiddly.  Swapping war stories may well be worthwhile on
> > this. This stuff also turns up in ASoC for example so probably lessons
> > to be learned from there.
> 
> V4L has a long history of this: if you have multiple video scalers in your
> pipeline, how do you control them? E.g. there can be scalers in the video
> sensor and in the SoC. We came up with the media controller device to
> expose the topology of the hardware and to be able to control each bit
> directly from userspace.
> 
> While the media controller is quite powerful and can be used to expose
> connections across subsystems, it also requires more work from userspace,
> and that's an area where not enough attention has been given.
> 
> > The analog devices software defined radios are another possible case
> > study.
> 
> Note that the V4L2 subsystem has support for several SDR devices. Probably
> another area where drivers can end up in different subsystems.
> 
> > There of course may well be lessons to be learned from similar
> > interactions elsewhere in the kernel.
> > 
> > There is a lot of history in how we ended up where we are (it all made
> > absolute sense at the time). Sitting back and taking the time
> > to discuss the future would be great.  Whilst this might be solvable
> > by email we've made no definitive progress for years
> > (and what has been made has been on a case by case basis deep in
> > driver reviews.)
> > 
> > I threw comedi in the list above but, at the moment, I think the more
> > likely direction there is a single userspace library abstracting
> > the underlying subsystem (Analog Devices are working in that
> > direction - perhaps Lars can offer more on that?).
> 
> When you get complex systems you probably have to move in that direction.
> I'd be interesting in such work going on in other subsystems. As mentioned
> above, the media controller needs something like that and it would be
> good to learn from others.
> 
> > GPIO is another interesting case - a lot of hardware is capable of
> > parallel sampling, some at high speeds. It's another area that
> > is probably too specialist for this discussion, but if people want
> > to dive into the details it might be interesting.
> > 
> > I think we have only a small amount of fuzz around the v4l boundary,
> > but wanted to leave the door open if anyone wants to discuss that
> > one further as it's come up a few times over recent years.
> 
> There is more of that than we wish, but quite often it is just dropped
> in the end as being too much work to solve and not quite important enough
> to spend the time and money on.
> 
> > The SoC world is a major case of one device, many uses.  Some SoCs
> > are turning up with multiple ADC units, sometimes with different
> > designs, sometimes simply so that the same hardware can be used for
> > different things.
> > 
> > A few suggestions for people:
> > 
> > Guenter Roeck (hmwon)
> > Dmitry Torokhov (input)
> > Me (Jonathan Cameron - IIO)
> > 
> > Someone thermal related (not sure who would be most interested)
> > Someone power supply related? (Guenter, any suggestions?)
> > 
> > Someone device tree related (Mark Rutland? Rob Herring?)
> > 
> > Linus Waleij (gpio)
> > 
> > Mark Brown - as some of concepts of IIO bridging and how far it
> > can be taken (everything via a bridge driver) came from
> > discussions with him on SoC ADC handling.
> > 
> > Lar-Peter Clausen for generally doing insane things with ADCs + having
> > one foot in the userspace side of things.
> 
> Laurent Pinchart or myself for v4l/mc

I'd love to *not* take part in the discussion, but that would imply living in 
a world where the problem doesn't exist, and I don't know of any :-) As we 
have to solve it, I'm interested in this topic.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Lars-Peter Clausen]

On 07/28/2016 06:46 PM, Laurent Pinchart wrote:
> Hi Linus,
> 
> On Friday 22 Jul 2016 14:04:07 Linus Walleij wrote:
>> On Wed, Jul 20, 2016 at 11:18 PM, Jonathan Cameron <jic23@kernel.org> wrote:
>>> This topic would be around the way the various subsystems interact, in the
>>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>>> crossing with output devices but maybe that's just over the next hill!)
>>>
>>> Scope may well be wider but includes:
>>> * input (some of it)
>>> * hwmon
>>> * iio
>>> * comedi(?)
>>> * thermal
>>> * power/battery
>>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
>>
>> I'm in. The new chardev ABI was designed to fit play/well with IIO,
>> for example I added a 64bit timestamp also to GPIO events so that
>> people could do what I guess they refer to as sensor fusion mixing
>> and blending sensor input events with GPIO events.
>> (https://en.wikipedia.org/wiki/Sensor_fusion)
>>
>> What I wanted to bring up ASAP was when I noticed in linux-next
>> commit bc2b7dab629a51e8beb5fda4222c62a23b729f26
>> "iio:core: timestamping clock selection support"
>> that we should probably move this code to e.g. drivers/base
>> or drivers/lib so I can reuse the same ABI in sysfs for
>> GPIO event timestamping.
>>
>> I think timestamping events uniformly is something that is
>> important to all userspaces.
>>
>>> Dmitry and I often have to decide on the IIO / input scope boundary
>>> (and we've moved it over time) + a number of cross over drivers have
>>> turned up recently.
>>
>> (snip from elsewhere)
>>
>>> Is it ever sensible to have two drivers for the same part because
>>> the use cases are so different? (I hope there is always a better way
>>> but maybe not)
>>
>> I have one on my knee as I want to make a proper IIO driver
>> for MPU-3050 gyroscope which has a (very simplified) driver in
>> drivers/input/misc/mpu3050.c that was merged before the advent
>> of the IIO subsystem.
>>
>> For the new driver I have these options:
>>
>> - Keep both drivers and consider them for different usecases.
>>   of the same hardware that require different Kconfig (seems
>>   very unelegant to me)
>>
>> - Add some input event calls into the IIO driver so we can
>>   phase over the older uses to the new driver.
>>
>> - Drop the input event handling since gyroscopes should not
>>   be handled by input anyway
>>
>> - Develop a generic IIO-gyroscope-to-input event bridge.
>>   (Phew seems complex.)
>>
>> (elsewhere)
>>
>>> IIO-input has been partly held outside mainline for years by this issue.
>>
>> Is there such a thing? gitweb?
>>
>>> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
>>> stand?
>>
>> IETF has this motto "rough consensus and running code", and
>> for me that is the working assumption to most things
>> driver-subsystem-division-of-responsibilities-related.
>>
>>> We have had quite a few cases where a whole driver is submitted to the
>>> 'wrong' subsystem. Not the best first interaction with the kernel
>>> world for new contributors. (This sort of area is attracts newbies
>>> as it's relatively simple and the hardware is fairly cheap and more
>>> than once their first experience with review as exactly this!)
>>
>> What is sometimes lacking is users who are clever enough to
>> take the latest kernel (or preferably even -next) and git grep for
>> a few strings related to the hardware they want to support, and
>> be revealed the fact that there is already a driver for it.
>>
>> If there isn't, their problem is that of finding the right subsystem
>> to pigeon-hole their hardware into. It typically requires a kernel
>> old grumpy person saying "no, THIS thing goes THERE, THAT
>> thing goes THERE". Unfortunately I think this is pretty hard to
>> codify.
>>
>>> If we aren't happy, what do we do about it?
>>
>> More helpful kernel generalists...
>>
>>> GPIO is another interesting case - a lot of hardware is capable of
>>> parallel sampling, some at high speeds. It's another area that
>>> is probably too specialist for this discussion, but if people want
>>> to dive into the details it might be interesting.
>>
>> I currently have (for v4.8) an ABI which makes i possible to e.g.
>> read 32 simultaneous lines with one userspace/kernelspace
>> swap, if it ends up as a readl() from a register.
>>
>> The userspace ABI has a limitation to 64 values, so it would be
>> possible to have some driver read 64 GPIO lines and return them
>> in a u8 vector back to userspace in one go.
>>
>> For an oscilloscope or signal analyzer usecase with more than
>> 64 channels needing to be sampled simultaneously, we may need
>> to add another "large GPIO read" ABI, and also probably a new
>> driver API interface to achieve it. I've never seen hardware that can
>> read more values in parallel than 32 though, I chose 64 to get some
>> headroom.
> 
> Such a use case would put strong real-time constraints on the system, I wonder 
> how useful a GPIO-based signal analyzer would be. Are you aware of any GPIO 
> hardware that can be programmed to sample at a precise point in time (or 
> perhaps even periodically) ?

I have such hardware. It's custom hardware, non-public at the moment.

But it works like this, for each pin you can configure whether it can be
controlled independently, like a normal GPIO, or whether the values are
sampled/updated to/from a continuous data stream that is regularly
clocked (basically a 1-bit ADC/DAC). This falls right on the edge
between GPIO and IIO and I'm really torn where to put it. For now I went
with IIO since it predates the chardev GPIO interface and because of all
the libiio magic.

Another similar application that uses regular GPIO hardware (IIRC) that
falls into the same category is the beaglelogic project
(https://github.com/abhishek-kakkar/BeagleLogic/wiki). And there have
been discussions to create a IIO driver for a similar setup in the past.

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Lars-Peter Clausen]

On 07/28/2016 06:39 PM, Laurent Pinchart wrote:
[...]
>> I think we have only a small amount of fuzz around the v4l boundary,
>> but wanted to leave the door open if anyone wants to discuss that
>> one further as it's come up a few times over recent years.
> 
> Don't forget to take system integration into account. If I given you a high-
> speed ADC you will not think about V4L2 as your subsystem of choice. If the 
> system designer has connected that ADC to a CPLD that generates fake 
> horizontal and vertical sync signals, and connected the output to the camera 
> interface of the SoC, you will be left with no choice but use the V4L2 API. 
> That's largely a userspace issue in this case, but it implies that V4L2 need 
> to define an "image format" for the ADC data.

I think this hits the core of this discussion. Todays hardware is
getting more and more generic. It is a lot more economical to produce a
single general purpose high-volume device than a handful of low or
medium volume specialized devices. Even if the raw production cost of
the general purpose part will be higher (since it contains more logic)
the overall per unit price will be lower since the per part contribution
of the one-time design cost is lower in a high volume run.

So new hardware often tends to be general purpose and can be used in
many different applications.

But our kernel frameworks are designed around application specific tasks.

* ALSA is for audio data capture/playback
* V4L2 is for video data capture/playback
* DRM is for video display
* IIO is for sensor data capture/playback

When you capture data over a particular interface there is a specific
meaning associated to the data rather than the data just being data,
which is how the hardware might see it.

On the kernel side we have started to address this by having generic
frameworks like DMAengine. I've used the same DMA core with the same
DMAengine driver exposed to userspace for all four different types
listed above depending on the application.

This works as long as you know that your hardware is generic and you
design the driver to be generic. But it breaks if your hardware has a
primary function that is application specific.

E.g. a CSI-2 receiver will most likely receive video data, so we write a
V4L2 driver for it. A I2S receiver will most likely receive audio data,
so we write a ALSA driver for it. But now somebody might decide to hook
up a gyroscope to any of these interfaces because that might be the best
way to feed data into the particular SoC used in that system. And than
things start to fall apart.

And this is not just hypothetical I've seen questions how to make this
work repeatedly. And I also expect that in a time constraint environment
people will go ahead with a custom solution where they capture audio
data through V4L2 and just ignore all data type hints V4L2 provides and
re-interprets the data since their specialized application knows what
the data layout looks like.

A similar issue is that there are quite a few pieces of hardware that
are multi-use. E.g. general purpose serial data cores that support SPI,
I2S, and similar. At the moment we have to write two different drivers
for them using compatible strings to decide which function they should have.

So going forward we might have to address this by creating a more
generic interface that allows us to exchange data between a peripheral
and a application without assigning any kind of meaning to the data
itself. And then have that meaning provided through side channels. E.g.
a V4L2 device could say this over there is my data capture device and
the data layout is the following. Similar for the other frameworks that
allow capture/playback.

With vb2 (former V4L2 buffer handling code) now being independent from
the V4L2 framework this might be a prime candidate as a starting point.
I've been meaning to re-write the IIO DMA buffer code on top of vb2 to
reduce the amount of custom code.

This of course would be a very grand task and maybe we'll loose
ourselves in endless discussions about the details and all the corner
cases that need to be considered. But if we want to find a solution that
keeps up with the current trend that the hardware landscape seems to be
going in we might have no other choice. Otherwise I'd say it is
inevitable that we see more and more hardware which has multiple
drivers, each driver handling a different type of application.

Such a grand unified media framework would also help for applications
where multiple streams of different data types need to be synchronized
e.g. audio and video.

- Lars

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Lars-Peter Clausen]

On 07/20/2016 11:18 PM, Jonathan Cameron wrote:
> Hi All,
> 
> This topic would be around the way the various subsystems interact, in the
> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> crossing with output devices but maybe that's just over the next hill!)

I've been on the other side of that hill and I can tell you there is
also some output overlap. E.g. between regulator drivers and IIO DAC
drivers. There are regulator drivers for general-purpose DACs that are
no any different than other DACs supported by IIO. E.g. ad5398. Other
overlaps might be backlight brightness control or lens control. In those
cases you have some kind of hardware aperture connected to the output
and the application is no longer ambiguous. But we should still avoid
writing application specific drivers for this case and rather make sure
that we have proper in-kernel frameworks so a generic driver can be
controlled through the application specific userspace API.

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2363 bytes --]

On Thu, Jul 28, 2016 at 08:53:26PM +0200, Lars-Peter Clausen wrote:

> But our kernel frameworks are designed around application specific tasks.

> * ALSA is for audio data capture/playback
> * V4L2 is for video data capture/playback
> * DRM is for video display
> * IIO is for sensor data capture/playback

Plus as we were discussing elsewhere hwmon, power_supply, input and
probably others that aren't springing to mind right now that also
represent sensor values out to userspace.

> And this is not just hypothetical I've seen questions how to make this
> work repeatedly. And I also expect that in a time constraint environment
> people will go ahead with a custom solution where they capture audio
> data through V4L2 and just ignore all data type hints V4L2 provides and
> re-interprets the data since their specialized application knows what
> the data layout looks like.

I've also seen cases (indeed I've got one myself right now) where the
opposite case applies and it's a lot easier to put a chip into a
hardware class it wasn't really designed for because the userspace
interfaces we offer are much more appropriate for that application.  In
that case it's a chip that really wants to be hwmon but libiio is
*really* nice.

> So going forward we might have to address this by creating a more
> generic interface that allows us to exchange data between a peripheral
> and a application without assigning any kind of meaning to the data
> itself. And then have that meaning provided through side channels. E.g.
> a V4L2 device could say this over there is my data capture device and
> the data layout is the following. Similar for the other frameworks that
> allow capture/playback.

Those side channels are the most tricky bit I fear.

> This of course would be a very grand task and maybe we'll loose
> ourselves in endless discussions about the details and all the corner
> cases that need to be considered. But if we want to find a solution that
> keeps up with the current trend that the hardware landscape seems to be
> going in we might have no other choice. Otherwise I'd say it is
> inevitable that we see more and more hardware which has multiple
> drivers, each driver handling a different type of application.

It's a worthwhile goal to bring things together.  I wonder how hard
it'd be to get ALSA to interact with the v4l buffer style...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Lars-Peter Clausen]

On 07/22/2016 05:29 AM, Guenter Roeck wrote:
[...]
>>
>> 4) Should we drop all the bindings for bridging between such subsystems
>> and do it all from userspace?
>>
> I think that would be a terrible idea. My hope is that we should be able
> to present a system to the user as it is intended to be, and that should
> include
> the use case for given hardware. An ADC is not just an ADC, it has a use
> case.
> The use case may be obvious if it is used as input device, but if it is
> used to measure a voltage (or current, or temperature) it should be
> possible
> to express that as well, without having to have userspace involved.

We need to differentiate between two different things. One is figuring
out the function. If you have a general purpose ADC that is hooked up to
some other hardware construct that usually gives you the function. E.g.
a GP ADC connected to a resistor ladder with push-buttons is a input
device. The same ADC connected to a thermocouple creates a temperature
sensor. This can easily be implemented by describing the additional
hardware properly in the DT and having a construct in the DT that
establishes that this ADC is connected to this other thing. We can then
use the in-kernel consumer API to get the raw ADC readings and translate
them to a application specific result.

This is all OK and I don't think anybody has any problem with this. It
is well within the scope of what devicetree is meant for and it has no
operating system specific bits per se.

The other thing is choosing the userspace interface through which the
data is exposed when the application falls in an area where it could be
exposed through one of multiple APIs. E.g. a general purpose temperature
sensor could be either IIO or hwmon. A accelerometer could either be
input or IIO. Some user might prefer one interface over the other
because they have existing application code that has support for this
interface. Another user wants to use another interface because they have
code for this interface.

This is the kind of bridge setup I'd rather very much avoid putting in
the DT. Strictly speaking at the DT level you don't know what kind of
application the users are going to run so you can't make that decission
at that level. Sure if you build a very tightly coupled embedded
application you get kernel, DT and userspace out of one hand and you
know at the DT level which interface your application will require, but
this is still a layering violation and should be avoided if possible.

Saying you want a IIO or hwmon interface in the DT is also very Linux
specific. Another reason why we should avoid this.

In my opinion we should make the default interface what we think to the
best of our knowledge is the appropriate interface for this type of
device and not offer an option to overwrite this through the DT. Where
things are on the edge we might let the driver author decide since they
might have a valid usecase for that interface.

In addition to that I think we should offer the possibility to
instantiate bridges from userspace to handle the case where somebody
really wants to use the other interface. Theses bridges should solely be
about changing the delivery method and not about changing the data type.

E.g. a userspace bridge should not be used to create a input device for
a resistor ladder connected to a GP ADC as this would change the meaning
of the data. On the other hand creating a hwmon to IIO bridge for a
temperature sensor should be OK since both report the same temperature
value.

I'm not sure what I think of this yet, but for completeness I want to
mention that we could also get a similar result by implementing the
bridge in userspace by LD_PRELOAD'ing. Maybe this is where is belongs?

Another thing that is up for discussion is what should do with the ADC
device in case there is additional hardware connected. Should we still
expose the userspace interface to raw ADC device? If not how would we
figure out that we shouldn't considering that the consumer is probed
after the provider.

- Lars

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 28/07/16 17:46, Laurent Pinchart wrote:
> Hi Linus,
> 
> On Friday 22 Jul 2016 14:04:07 Linus Walleij wrote:
>> On Wed, Jul 20, 2016 at 11:18 PM, Jonathan Cameron <jic23@kernel.org> wrote:
>>> This topic would be around the way the various subsystems interact, in the
>>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>>> crossing with output devices but maybe that's just over the next hill!)
>>>
>>> Scope may well be wider but includes:
>>> * input (some of it)
>>> * hwmon
>>> * iio
>>> * comedi(?)
>>> * thermal
>>> * power/battery
>>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
>>
>> I'm in. The new chardev ABI was designed to fit play/well with IIO,
>> for example I added a 64bit timestamp also to GPIO events so that
>> people could do what I guess they refer to as sensor fusion mixing
>> and blending sensor input events with GPIO events.
>> (https://en.wikipedia.org/wiki/Sensor_fusion)
>>
>> What I wanted to bring up ASAP was when I noticed in linux-next
>> commit bc2b7dab629a51e8beb5fda4222c62a23b729f26
>> "iio:core: timestamping clock selection support"
>> that we should probably move this code to e.g. drivers/base
>> or drivers/lib so I can reuse the same ABI in sysfs for
>> GPIO event timestamping.
>>
>> I think timestamping events uniformly is something that is
>> important to all userspaces.
>>
>>> Dmitry and I often have to decide on the IIO / input scope boundary
>>> (and we've moved it over time) + a number of cross over drivers have
>>> turned up recently.
>>
>> (snip from elsewhere)
>>
>>> Is it ever sensible to have two drivers for the same part because
>>> the use cases are so different? (I hope there is always a better way
>>> but maybe not)
>>
>> I have one on my knee as I want to make a proper IIO driver
>> for MPU-3050 gyroscope which has a (very simplified) driver in
>> drivers/input/misc/mpu3050.c that was merged before the advent
>> of the IIO subsystem.
>>
>> For the new driver I have these options:
>>
>> - Keep both drivers and consider them for different usecases.
>>   of the same hardware that require different Kconfig (seems
>>   very unelegant to me)
>>
>> - Add some input event calls into the IIO driver so we can
>>   phase over the older uses to the new driver.
>>
>> - Drop the input event handling since gyroscopes should not
>>   be handled by input anyway
>>
>> - Develop a generic IIO-gyroscope-to-input event bridge.
>>   (Phew seems complex.)
>>
>> (elsewhere)
>>
>>> IIO-input has been partly held outside mainline for years by this issue.
>>
>> Is there such a thing? gitweb?
>>
>>> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
>>> stand?
>>
>> IETF has this motto "rough consensus and running code", and
>> for me that is the working assumption to most things
>> driver-subsystem-division-of-responsibilities-related.
>>
>>> We have had quite a few cases where a whole driver is submitted to the
>>> 'wrong' subsystem. Not the best first interaction with the kernel
>>> world for new contributors. (This sort of area is attracts newbies
>>> as it's relatively simple and the hardware is fairly cheap and more
>>> than once their first experience with review as exactly this!)
>>
>> What is sometimes lacking is users who are clever enough to
>> take the latest kernel (or preferably even -next) and git grep for
>> a few strings related to the hardware they want to support, and
>> be revealed the fact that there is already a driver for it.
>>
>> If there isn't, their problem is that of finding the right subsystem
>> to pigeon-hole their hardware into. It typically requires a kernel
>> old grumpy person saying "no, THIS thing goes THERE, THAT
>> thing goes THERE". Unfortunately I think this is pretty hard to
>> codify.
>>
>>> If we aren't happy, what do we do about it?
>>
>> More helpful kernel generalists...
>>
>>> GPIO is another interesting case - a lot of hardware is capable of
>>> parallel sampling, some at high speeds. It's another area that
>>> is probably too specialist for this discussion, but if people want
>>> to dive into the details it might be interesting.
>>
>> I currently have (for v4.8) an ABI which makes i possible to e.g.
>> read 32 simultaneous lines with one userspace/kernelspace
>> swap, if it ends up as a readl() from a register.
>>
>> The userspace ABI has a limitation to 64 values, so it would be
>> possible to have some driver read 64 GPIO lines and return them
>> in a u8 vector back to userspace in one go.
>>
>> For an oscilloscope or signal analyzer usecase with more than
>> 64 channels needing to be sampled simultaneously, we may need
>> to add another "large GPIO read" ABI, and also probably a new
>> driver API interface to achieve it. I've never seen hardware that can
>> read more values in parallel than 32 though, I chose 64 to get some
>> headroom.
> 
> Such a use case would put strong real-time constraints on the system, I wonder 
> how useful a GPIO-based signal analyzer would be. Are you aware of any GPIO 
> hardware that can be programmed to sample at a precise point in time (or 
> perhaps even periodically) ?
There are cross over cases from the other side where
people are putting devices that do do sequenced input sampling to use
to do gpio tasks.  A lot of sensors have spare pins and someone always
decides they might as well be gpi(o) pins rather than waste them.

Sometimes they have to be polled separately. Sometimes they have the
even more cunning plan of sticking them in 'spare bits' of ADC channels,
or in additional registers that can be dropped into a hardware fifo.
All in then takes is some designer running low on pins on the SoC to
decide they'll be handy for something random.

A simple gpio driven logic analyzer case probably needs some
hardware support. There are latched capture units out there that
provide this sort of precise timing as well, though they are well
out of the realms of what you'd normally call a general purpose input.
TIs universal parallel ports for example (I think), or something
like beaglelogic where the timing is pushed to a coprocessor.

It's also possible to design a wide variety of algorithms to work
with uneven sampling of data, as long as you know 'when' it was
sampled.

So more fiddly corner cases :)

Jonathan
> 
>>> Linus Waleij (gpio)
>>
>> Sure, if travels etc work out and I get invited I'd come running.
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 28/07/16 17:42, Laurent Pinchart wrote:
> On Thursday 28 Jul 2016 08:58:46 Mauro Carvalho Chehab wrote:
>> Em Wed, 20 Jul 2016 22:18:11 +0100 Jonathan Cameron escreveu:
>>> Hi All,
>>>
>>> This topic would be around the way the various subsystems interact, in the
>>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>>> crossing with output devices but maybe that's just over the next hill!)
>>
>> Yeah, there is a gray area here as devices become more complex.
>> So, I'm interested on such topic.
>>
>>> Scope may well be wider but includes:
>>> * input (some of it)
>>> * hwmon
>>> * iio
>>> * comedi(?)
>>> * thermal
>>> * power/battery
>>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
>>> * v4l - when does a device jump from being a multi pixel thermopile
>>> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
>>> sensors (effectively 9ish pixel cameras)
>>
>> For devices that provide images somehow, I'd say that the best would be
>> to implement them as via the V4L2 API.
> 
> This requires defining what an image is. Furthermore, we very low image 
> resolution devices, we will need to deal with high frame rates (1k-10k is a 
> range we need to consider). The V4L2 API will likely show performance issues.
An interesting point I'd never thought about before. Funnily enough the devices
around this boundary that I've encountered have all been relatively slow.
Doubt we'll be lucky enough that that will last!

Jonathan
> 
>> We'll likely need to discuss it case by case, specially for early cross-
>> subsystem drivers, though, as it is not trivial to identify the subsystem
>> boundaries, and sometimes multiple APIs from different subsystems is needed.
>> Also, it is not clear where such drivers would fit at the Kernel tree.
>>
>> One interesting case is an input device driver with multi-finger
>> support (drivers/input/touchscreen/sur40.c). I suspect we'll have
>> more cases like that over time.
>>
>>> * Lots of random things we haven't seen yet.
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 28/07/16 17:39, Laurent Pinchart wrote:
> Hi Jonathan,
> 
> On Wednesday 20 Jul 2016 22:18:11 Jonathan Cameron wrote:
>> Hi All,
>>
>> This topic would be around the way the various subsystems interact, in the
>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>> crossing with output devices but maybe that's just over the next hill!)
> 
> It's already here :-) We have an API to control camera flash (as in light, not 
> memory) in V4L2 and another one in the LED subsystem.
I wonder how many devices are doing it with a gpio or some of the more
configurable PWM units..
> 
>> Scope may well be wider but includes:
>> * input (some of it)
>> * hwmon
>> * iio
>> * comedi(?)
>> * thermal
>> * power/battery
>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
>> * v4l - when does a device jump from being a multi pixel thermopile
>> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
>> sensors (effectively 9ish pixel cameras)
>> * Lots of random things we haven't seen yet.
>>
>> All these make use of devices that are at their hearts ADCs. Whilst it
>> would nice if the underlying devices always fell into one clear
>> category that is not always the case. Hardware descriptions (e.g.
>> device tree) are an important area of contention.
>>
>> The examples that follow are mostly based around IIO interactions with
>> hwmon / input as that's my area of expertise. However, I know that
>> similar issues occur at other boundaries.
>>
>> Guenter Roeck has been working on hwmon / thermal interfacing recently.
>> We also regularly push drivers one way or the other around that boundary.
>>
>> Dmitry and I often have to decide on the IIO / input scope boundary
>> (and we've moved it over time) + a number of cross over drivers have
>> turned up recently.
>>
>> The other subsystems I encounter have been 'quieter' in their
>> interactions with me but many of the issues below apply to them as well.
>>
>> A rough list of related topics that might be worth face to face
>> discussion includes:
>>
>> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
>> stand?
>>
>> I am personally happy enough with this, but it's worth checking
>> everyone else is. This probably causes more pain for new submitters
>> than it does for old hands. Perhaps we can come up with a checklist style
>> doc to save people having to ask.
>>
>> We have had quite a few cases where a whole driver is submitted to the
>> 'wrong' subsystem. Not the best first interaction with the kernel
>> world for new contributors. (This sort of area is attracts newbies
>> as it's relatively simple and the hardware is fairly cheap and more
>> than once their first experience with review as exactly this!)
>>
>> If we aren't happy, what do we do about it?
>>
>> I'm hoping we are happy or, at least, resigned to the current state
>> of affairs, but think the question should be asked from time to time
>> as the answer may well change + the discussion will highlight pinch
>> points we can work on.
>>
>> 2) Bridging drivers - there will always be cross over cases.
>> E.g. Generic ADCs used for battery voltage monitoring.  We have a
>> number of bridges in mainline already and others out of tree.  How
>> important are these and what features would make them more useful?
>>
>> Is it ever sensible to have two drivers for the same part because
>> the use cases are so different? (I hope there is always a better way
>> but maybe not)
> 
> I'd love to say no, but we have already hit that problem. There are two 
> drivers for the ADV7511 HDMI encoder, one in the media subsystem for V4L2 
> output devices (drivers/media/i2c/adv7511.c), and one in the DRM/KMS subsystem 
> for display devices (drivers/gpu/drm/i2c/adv7511.c, being moved to 
> drivers/gpu/drm/bridge/). Try compiling them both in the kernel and see how 
> they race to probe the same device (enjoy the show, popcorns not included).
Nasty.
> 
>> This covers both generic bridges (e.g. iio-hwmon) and device
>> specific bridges (a good example of a touch screen driver turned
>> up in my inbox yesterday).
>>
>> 3) Bindings (device tree and similar).  When it is appropriate to use a
>> device tree to describe the overlap (bridging of channels)?
>> Sometimes there is obvious real hardware involved (a thermistor on a
>> generic ADC input for example), but there are many grey areas.
>>
>> Finally bashing out an answer to the issue device tree maintainers
>> have with the iio-hwmon bindings would be great.
>> On a less biased note, that example is pretty fiddly to define but would
>> act as a good basis to work from more generally.
>>
>> We know we got it wrong (back in the dark ages), but it isn't obvious how
>> to get it right!
>>
>> IIO-input has been partly held outside mainline for years by this issue.
>>
>> Also some practical design decisions to make around how to implement
>> these mappings to work best with deferred probing etc.
>>
>> 4) Should we drop all the bindings for bridging between such subsystems
>> and do it all from userspace?
>>
>> I think we may end up with a hybrid of the two, but need to be able to
>> make it work in 'standard' cases without userspace being involved. That
>> hybrid solution may well be devicetree overlay based... Unclear so far.
>> + plenty of crazy things that 'might' make sense where there isn't even
>> the pretence of representing real hardware.
>>
>> 5) Complex device interaction usecases.  At the moment the ones I've come
>> across are mostly contained in IIO.  The moment we start sticking in
>> MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
>> mix it can get fiddly.  Swapping war stories may well be worthwhile on
>> this. This stuff also turns up in ASoC for example so probably lessons
>> to be learned from there.
>>
>> The analog devices software defined radios are another possible case
>> study.
>>
>> There of course may well be lessons to be learned from similar
>> interactions elsewhere in the kernel.
>>
>> There is a lot of history in how we ended up where we are (it all made
>> absolute sense at the time). Sitting back and taking the time
>> to discuss the future would be great.  Whilst this might be solvable
>> by email we've made no definitive progress for years
>> (and what has been made has been on a case by case basis deep in
>> driver reviews.)
>>
>> I threw comedi in the list above but, at the moment, I think the more
>> likely direction there is a single userspace library abstracting
>> the underlying subsystem (Analog Devices are working in that
>> direction - perhaps Lars can offer more on that?).
>>
>> GPIO is another interesting case - a lot of hardware is capable of
>> parallel sampling, some at high speeds. It's another area that
>> is probably too specialist for this discussion, but if people want
>> to dive into the details it might be interesting.
>>
>> I think we have only a small amount of fuzz around the v4l boundary,
>> but wanted to leave the door open if anyone wants to discuss that
>> one further as it's come up a few times over recent years.
> 
> Don't forget to take system integration into account. If I given you a high-
> speed ADC you will not think about V4L2 as your subsystem of choice. If the 
> system designer has connected that ADC to a CPLD that generates fake 
> horizontal and vertical sync signals, and connected the output to the camera 
> interface of the SoC, you will be left with no choice but use the V4L2 API. 
> That's largely a userspace issue in this case, but it implies that V4L2 need 
> to define an "image format" for the ADC data.
This one was somewhat of a thought experiment (and the sort of dirty hack
we'd do in my day job :)

Absolutely.  Sometimes they'll obligingly connect it to a camera specific
interface sometimes the won't.  I think I saw an example of a camera hanging
of one of TI's universal parallel ports at one point for example - not
sure how the framing worked there.

The many generic high speed serial interfaces are also an option.

If it's all wrapped up and presented as a camera device, then we don't need
to care that it's a generic ADC really as the interesting handling will
probably all be done in the fpga / cpld anyway.

The possibility that gets more interesting from my point of view is where
the pixel stream simply hits an ADC.  Grabbing from the day job, a line
scan camera would be the classic.  There'd probably still be some sort
of line framing but lots of places to hide that including in the analog
stream itself.

Even if they do, there always comes a point where it's easier to just declare
that we are better off with a monolithic device driver hiding what
is inside.

Anyhow, I kind of hope no one ever does this *crosses fingers*

Possibly more likely around the blurry boundary of what is a video
capture device and what is simply taking a small number of specially
related signals and sampling them...
> 
>> The SoC world is a major case of one device, many uses.  Some SoCs
>> are turning up with multiple ADC units, sometimes with different
>> designs, sometimes simply so that the same hardware can be used for
>> different things.
>>
>> A few suggestions for people:
>>
>> Guenter Roeck (hmwon)
>> Dmitry Torokhov (input)
>> Me (Jonathan Cameron - IIO)
>>
>> Someone thermal related (not sure who would be most interested)
>> Someone power supply related? (Guenter, any suggestions?)
>>
>> Someone device tree related (Mark Rutland? Rob Herring?)
>>
>> Linus Waleij (gpio)
>>
>> Mark Brown - as some of concepts of IIO bridging and how far it
>> can be taken (everything via a bridge driver) came from
>> discussions with him on SoC ADC handling.
>>
>> Lar-Peter Clausen for generally doing insane things with ADCs + having
>> one foot in the userspace side of things.
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 28/07/16 12:58, Mauro Carvalho Chehab wrote:
> Em Wed, 20 Jul 2016 22:18:11 +0100
> Jonathan Cameron <jic23@kernel.org> escreveu:
> 
>> Hi All,
>>
>> This topic would be around the way the various subsystems interact, in the
>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>> crossing with output devices but maybe that's just over the next hill!)
> 
> Yeah, there is a gray area here as devices become more complex.
> So, I'm interested on such topic.
> 
>> Scope may well be wider but includes:
>> * input (some of it)
>> * hwmon
>> * iio
>> * comedi(?)
>> * thermal
>> * power/battery
>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
>> * v4l - when does a device jump from being a multi pixel thermopile
>> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
>> sensors (effectively 9ish pixel cameras)
> 
> For devices that provide images somehow, I'd say that the best would be
> to implement them as via the V4L2 API. We'll likely need to discuss it
> case by case, specially for early cross-subsystem drivers, though, as
> it is not trivial to identify the subsystem boundaries, and sometimes
> multiple APIs from different subsystems is needed. Also, it is not
> clear where such drivers would fit at the Kernel tree.
Absolutely.  We can come up with some rule's of thumb to at least cut
down on the number of drivers getting written / pitched for totally
the wrong subsystem, but there are always going to be corner cases.
> 
> One interesting case is an input device driver with multi-finger
> support (drivers/input/touchscreen/sur40.c). I suspect we'll have
> more cases like that over time.
Becomes a case of whether we almost 'limit' devices by putting them
in a subsystem dictated by their usecase, or find a way to bridge
between subsystems so the common use case and interfaces are supported
but the more 'interesting' stuff is available underneath - possibly
for other uses.

> 
>> * Lots of random things we haven't seen yet.
I've already seen a few in this thread!

Jonathan
> 
> Thanks,
> Mauro
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 28/07/16 22:30, Lars-Peter Clausen wrote:
> On 07/22/2016 05:29 AM, Guenter Roeck wrote:
> [...]
>>>
>>> 4) Should we drop all the bindings for bridging between such subsystems
>>> and do it all from userspace?
>>>
>> I think that would be a terrible idea. My hope is that we should be able
>> to present a system to the user as it is intended to be, and that should
>> include
>> the use case for given hardware. An ADC is not just an ADC, it has a use
>> case.
>> The use case may be obvious if it is used as input device, but if it is
>> used to measure a voltage (or current, or temperature) it should be
>> possible
>> to express that as well, without having to have userspace involved.
> 
> We need to differentiate between two different things. One is figuring
> out the function. If you have a general purpose ADC that is hooked up to
> some other hardware construct that usually gives you the function. E.g.
> a GP ADC connected to a resistor ladder with push-buttons is a input
> device. The same ADC connected to a thermocouple creates a temperature
> sensor. This can easily be implemented by describing the additional
> hardware properly in the DT and having a construct in the DT that
> establishes that this ADC is connected to this other thing. We can then
> use the in-kernel consumer API to get the raw ADC readings and translate
> them to a application specific result.
> 
> This is all OK and I don't think anybody has any problem with this. It
> is well within the scope of what devicetree is meant for and it has no
> operating system specific bits per se.
> 
> The other thing is choosing the userspace interface through which the
> data is exposed when the application falls in an area where it could be
> exposed through one of multiple APIs. E.g. a general purpose temperature
> sensor could be either IIO or hwmon. A accelerometer could either be
> input or IIO. Some user might prefer one interface over the other
> because they have existing application code that has support for this
> interface. Another user wants to use another interface because they have
> code for this interface.
> 
> This is the kind of bridge setup I'd rather very much avoid putting in
> the DT. Strictly speaking at the DT level you don't know what kind of
> application the users are going to run so you can't make that decission
> at that level. Sure if you build a very tightly coupled embedded
> application you get kernel, DT and userspace out of one hand and you
> know at the DT level which interface your application will require, but
> this is still a layering violation and should be avoided if possible.
> 
> Saying you want a IIO or hwmon interface in the DT is also very Linux
> specific. Another reason why we should avoid this.
> 
> In my opinion we should make the default interface what we think to the
> best of our knowledge is the appropriate interface for this type of
> device and not offer an option to overwrite this through the DT. Where
> things are on the edge we might let the driver author decide since they
> might have a valid usecase for that interface.
> 
> In addition to that I think we should offer the possibility to
> instantiate bridges from userspace to handle the case where somebody
> really wants to use the other interface. Theses bridges should solely be
> about changing the delivery method and not about changing the data type.
> 
> E.g. a userspace bridge should not be used to create a input device for
> a resistor ladder connected to a GP ADC as this would change the meaning
> of the data. On the other hand creating a hwmon to IIO bridge for a
> temperature sensor should be OK since both report the same temperature
> value.
That's the sort of balance I think we need to reach as well.
> 
> I'm not sure what I think of this yet, but for completeness I want to
> mention that we could also get a similar result by implementing the
> bridge in userspace by LD_PRELOAD'ing. Maybe this is where is belongs?
> 
> Another thing that is up for discussion is what should do with the ADC
> device in case there is additional hardware connected. Should we still
> expose the userspace interface to raw ADC device? If not how would we
> figure out that we shouldn't considering that the consumer is probed
> after the provider.
Is there any harm in doing so? 

There was an effort a few years back to separate the backend and front
end of IIO explicitly to allow the dropping of the userspace interface.
In the extreme case could we 'solve' the exposing the wrong thing case
by requiring every channel to be mapped to a client?  So if you
have a general purpose channel, it has to be mapped as such -
interfaces are effectively only provided on demand. Once an interface
has been provided in the extreme case we could lock out any more
from using that channel.

Doing this for all subsystems with fuzzy edge cases would be a pain
however!

Jonathan
> 
> - Lars
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Alexandre Belloni]

Hi,

On 20/07/2016 at 22:18:11 +0100, Jonathan Cameron wrote :
> A few suggestions for people:
> 
> Guenter Roeck (hmwon)
> Dmitry Torokhov (input)
> Me (Jonathan Cameron - IIO)
> 
> Someone thermal related (not sure who would be most interested)
> Someone power supply related? (Guenter, any suggestions?)
> 
> Someone device tree related (Mark Rutland? Rob Herring?)
> 
> Linus Waleij (gpio)
> 
> Mark Brown - as some of concepts of IIO bridging and how far it
> can be taken (everything via a bridge driver) came from
> discussions with him on SoC ADC handling.
> 
> Lar-Peter Clausen for generally doing insane things with ADCs + having
> one foot in the userspace side of things.
> 

I'm pretty interested in that discussion as I've been writing/using some
of the ADC/touch drivers (at91, mxs) and I've been involved in the sunxi
ones.

Also, I've recently been seeing people using GP ADCs wanting to get an
ALSA interface and others using audio ADCs wanting the IIO interface.

Regards,

-- 
Alexandre Belloni, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Guenter Roeck]

On 07/28/2016 02:30 PM, Lars-Peter Clausen wrote:
> On 07/22/2016 05:29 AM, Guenter Roeck wrote:
> [...]
>>>
>>> 4) Should we drop all the bindings for bridging between such subsystems
>>> and do it all from userspace?
>>>
>> I think that would be a terrible idea. My hope is that we should be able
>> to present a system to the user as it is intended to be, and that should
>> include
>> the use case for given hardware. An ADC is not just an ADC, it has a use
>> case.
>> The use case may be obvious if it is used as input device, but if it is
>> used to measure a voltage (or current, or temperature) it should be
>> possible
>> to express that as well, without having to have userspace involved.
>
> We need to differentiate between two different things. One is figuring
> out the function. If you have a general purpose ADC that is hooked up to
> some other hardware construct that usually gives you the function. E.g.
> a GP ADC connected to a resistor ladder with push-buttons is a input
> device. The same ADC connected to a thermocouple creates a temperature
> sensor. This can easily be implemented by describing the additional
> hardware properly in the DT and having a construct in the DT that
> establishes that this ADC is connected to this other thing. We can then
> use the in-kernel consumer API to get the raw ADC readings and translate
> them to a application specific result.
>
> This is all OK and I don't think anybody has any problem with this. It
> is well within the scope of what devicetree is meant for and it has no
> operating system specific bits per se.
>
> The other thing is choosing the userspace interface through which the
> data is exposed when the application falls in an area where it could be
> exposed through one of multiple APIs. E.g. a general purpose temperature
> sensor could be either IIO or hwmon. A accelerometer could either be
> input or IIO. Some user might prefer one interface over the other
> because they have existing application code that has support for this
> interface. Another user wants to use another interface because they have
> code for this interface.
>
> This is the kind of bridge setup I'd rather very much avoid putting in
> the DT. Strictly speaking at the DT level you don't know what kind of
> application the users are going to run so you can't make that decission
> at that level. Sure if you build a very tightly coupled embedded
> application you get kernel, DT and userspace out of one hand and you
> know at the DT level which interface your application will require, but
> this is still a layering violation and should be avoided if possible.
>
> Saying you want a IIO or hwmon interface in the DT is also very Linux
> specific. Another reason why we should avoid this.
>

That ambiguity has not been the case at least in the use cases I was
involved in. It just happened that some hardware engineers decided that
they wanted to use a common ADC to measure voltages on a board (rather,
on all the boards built by that company). So the ADCs are connected to board
voltages, and the intent was very clear - to be used for hardware monitoring.

I am sure it can be expressed that an ADC is connected to, say, a regulator
output. The question is how to express the intent to use that connection
for the purpose of hardware monitoring. I think it should be possible
for the board designer to make a statement about the intended use
of the hardware.

Guenter

> In my opinion we should make the default interface what we think to the
> best of our knowledge is the appropriate interface for this type of
> device and not offer an option to overwrite this through the DT. Where
> things are on the edge we might let the driver author decide since they
> might have a valid usecase for that interface.
>
> In addition to that I think we should offer the possibility to
> instantiate bridges from userspace to handle the case where somebody
> really wants to use the other interface. Theses bridges should solely be
> about changing the delivery method and not about changing the data type.
>
> E.g. a userspace bridge should not be used to create a input device for
> a resistor ladder connected to a GP ADC as this would change the meaning
> of the data. On the other hand creating a hwmon to IIO bridge for a
> temperature sensor should be OK since both report the same temperature
> value.
>
> I'm not sure what I think of this yet, but for completeness I want to
> mention that we could also get a similar result by implementing the
> bridge in userspace by LD_PRELOAD'ing. Maybe this is where is belongs?
>
> Another thing that is up for discussion is what should do with the ADC
> device in case there is additional hardware connected. Should we still
> expose the userspace interface to raw ADC device? If not how would we
> figure out that we shouldn't considering that the consumer is probed
> after the provider.
>
> - Lars
>
>

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 29 July 2016 01:56:51 BST, Guenter Roeck <linux@roeck-us.net> wrote:
>On 07/28/2016 02:30 PM, Lars-Peter Clausen wrote:
>> On 07/22/2016 05:29 AM, Guenter Roeck wrote:
>> [...]
>>>>
>>>> 4) Should we drop all the bindings for bridging between such
>subsystems
>>>> and do it all from userspace?
>>>>
>>> I think that would be a terrible idea. My hope is that we should be
>able
>>> to present a system to the user as it is intended to be, and that
>should
>>> include
>>> the use case for given hardware. An ADC is not just an ADC, it has a
>use
>>> case.
>>> The use case may be obvious if it is used as input device, but if it
>is
>>> used to measure a voltage (or current, or temperature) it should be
>>> possible
>>> to express that as well, without having to have userspace involved.
>>
>> We need to differentiate between two different things. One is
>figuring
>> out the function. If you have a general purpose ADC that is hooked up
>to
>> some other hardware construct that usually gives you the function.
>E.g.
>> a GP ADC connected to a resistor ladder with push-buttons is a input
>> device. The same ADC connected to a thermocouple creates a
>temperature
>> sensor. This can easily be implemented by describing the additional
>> hardware properly in the DT and having a construct in the DT that
>> establishes that this ADC is connected to this other thing. We can
>then
>> use the in-kernel consumer API to get the raw ADC readings and
>translate
>> them to a application specific result.
>>
>> This is all OK and I don't think anybody has any problem with this.
>It
>> is well within the scope of what devicetree is meant for and it has
>no
>> operating system specific bits per se.
>>
>> The other thing is choosing the userspace interface through which the
>> data is exposed when the application falls in an area where it could
>be
>> exposed through one of multiple APIs. E.g. a general purpose
>temperature
>> sensor could be either IIO or hwmon. A accelerometer could either be
>> input or IIO. Some user might prefer one interface over the other
>> because they have existing application code that has support for this
>> interface. Another user wants to use another interface because they
>have
>> code for this interface.
>>
>> This is the kind of bridge setup I'd rather very much avoid putting
>in
>> the DT. Strictly speaking at the DT level you don't know what kind of
>> application the users are going to run so you can't make that
>decission
>> at that level. Sure if you build a very tightly coupled embedded
>> application you get kernel, DT and userspace out of one hand and you
>> know at the DT level which interface your application will require,
>but
>> this is still a layering violation and should be avoided if possible.
>>
>> Saying you want a IIO or hwmon interface in the DT is also very Linux
>> specific. Another reason why we should avoid this.
>>
>
>That ambiguity has not been the case at least in the use cases I was
>involved in. It just happened that some hardware engineers decided that
>they wanted to use a common ADC to measure voltages on a board (rather,
>on all the boards built by that company). So the ADCs are connected to
>board
>voltages, and the intent was very clear - to be used for hardware
>monitoring.
>
>I am sure it can be expressed that an ADC is connected to, say, a
>regulator
>output. The question is how to express the intent to use that
>connection
>for the purpose of hardware monitoring. I think it should be possible
>for the board designer to make a statement about the intended use
>of the hardware.

Arguably, this is another case of describing wiring. If it's wired to monitor
something like a power line, it's that we should try to figure out how to describe.

Maybe your suggestion of making it linked to a regulator is one way of doing it. 
'This adc channel is measuring this regulated channel'. Always a hwmon case
 to my mind.  Going to end up with sets of ad-hoc looking rules though to make that
work if we can do it at all.

This was the device tree maintainer suggestion IIRC on how we should have
done the IIO-HWMON bindings..

Jonathan
>
>Guenter
>
>> In my opinion we should make the default interface what we think to
>the
>> best of our knowledge is the appropriate interface for this type of
>> device and not offer an option to overwrite this through the DT.
>Where
>> things are on the edge we might let the driver author decide since
>they
>> might have a valid usecase for that interface.
>>
>> In addition to that I think we should offer the possibility to
>> instantiate bridges from userspace to handle the case where somebody
>> really wants to use the other interface. Theses bridges should solely
>be
>> about changing the delivery method and not about changing the data
>type.
>>
>> E.g. a userspace bridge should not be used to create a input device
>for
>> a resistor ladder connected to a GP ADC as this would change the
>meaning
>> of the data. On the other hand creating a hwmon to IIO bridge for a
>> temperature sensor should be OK since both report the same
>temperature
>> value.
>>
>> I'm not sure what I think of this yet, but for completeness I want to
>> mention that we could also get a similar result by implementing the
>> bridge in userspace by LD_PRELOAD'ing. Maybe this is where is
>belongs?
>>
>> Another thing that is up for discussion is what should do with the
>ADC
>> device in case there is additional hardware connected. Should we
>still
>> expose the userspace interface to raw ADC device? If not how would we
>> figure out that we shouldn't considering that the consumer is probed
>> after the provider.
>>
>> - Lars
>>
>>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Jonathan Cameron]

On 29 July 2016 00:38:12 BST, Alexandre Belloni <alexandre.belloni@free-electrons.com> wrote:
>Hi,
>
>On 20/07/2016 at 22:18:11 +0100, Jonathan Cameron wrote :
>> A few suggestions for people:
>> 
>> Guenter Roeck (hmwon)
>> Dmitry Torokhov (input)
>> Me (Jonathan Cameron - IIO)
>> 
>> Someone thermal related (not sure who would be most interested)
>> Someone power supply related? (Guenter, any suggestions?)
>> 
>> Someone device tree related (Mark Rutland? Rob Herring?)
>> 
>> Linus Waleij (gpio)
>> 
>> Mark Brown - as some of concepts of IIO bridging and how far it
>> can be taken (everything via a bridge driver) came from
>> discussions with him on SoC ADC handling.
>> 
>> Lar-Peter Clausen for generally doing insane things with ADCs +
>having
>> one foot in the userspace side of things.
>> 
>
>I'm pretty interested in that discussion as I've been writing/using
>some
>of the ADC/touch drivers (at91, mxs) and I've been involved in the
>sunxi
>ones.
>
>Also, I've recently been seeing people using GP ADCs wanting to get an
>ALSA interface and others using audio ADCs wanting the IIO interface.

So here is another case where it looks like we ultimately need bridges.

Hopefully the hardware usecase is obvious enough here that bindings
will be straight forward anyway...

Fits in well with Lars' point about everything is going more generic and
we need to be ready for it. Sounds like we are a bit late already...

I am beginning to wonder why I opened this can of worms! 

Jonathan
>
>Regards,

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Hans Verkuil]

On 07/28/2016 06:42 PM, Laurent Pinchart wrote:
> On Thursday 28 Jul 2016 08:58:46 Mauro Carvalho Chehab wrote:
>> Em Wed, 20 Jul 2016 22:18:11 +0100 Jonathan Cameron escreveu:
>>> Hi All,
>>>
>>> This topic would be around the way the various subsystems interact, in the
>>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>>> crossing with output devices but maybe that's just over the next hill!)
>>
>> Yeah, there is a gray area here as devices become more complex.
>> So, I'm interested on such topic.
>>
>>> Scope may well be wider but includes:
>>> * input (some of it)
>>> * hwmon
>>> * iio
>>> * comedi(?)
>>> * thermal
>>> * power/battery
>>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
>>> * v4l - when does a device jump from being a multi pixel thermopile
>>> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
>>> sensors (effectively 9ish pixel cameras)
>>
>> For devices that provide images somehow, I'd say that the best would be
>> to implement them as via the V4L2 API.
> 
> This requires defining what an image is. Furthermore, we very low image 
> resolution devices, we will need to deal with high frame rates (1k-10k is a 
> range we need to consider). The V4L2 API will likely show performance issues.

The main problems there are the limit to the number of buffers (32) in v4l2 and
that the number of qbuf/dqbuf ioctls is large for these high framerates. You would
probably want to introduce mqbuf/mdqbuf ioctls to queue/dequeue multiple buffers.
The core internals of v4l2 (vb2) are all pretty low overhead.

Regards,

	Hans

> 
>> We'll likely need to discuss it case by case, specially for early cross-
>> subsystem drivers, though, as it is not trivial to identify the subsystem
>> boundaries, and sometimes multiple APIs from different subsystems is needed.
>> Also, it is not clear where such drivers would fit at the Kernel tree.
>>
>> One interesting case is an input device driver with multi-finger
>> support (drivers/input/touchscreen/sur40.c). I suspect we'll have
>> more cases like that over time.
>>
>>> * Lots of random things we haven't seen yet.
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Hans Verkuil]

On 07/29/2016 12:26 AM, Jonathan Cameron wrote:
> On 28/07/16 17:39, Laurent Pinchart wrote:
>> Hi Jonathan,
>>
>> On Wednesday 20 Jul 2016 22:18:11 Jonathan Cameron wrote:
>>> Hi All,
>>>
>>> This topic would be around the way the various subsystems interact, in the
>>> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
>>> crossing with output devices but maybe that's just over the next hill!)
>>
>> It's already here :-) We have an API to control camera flash (as in light, not 
>> memory) in V4L2 and another one in the LED subsystem.
> I wonder how many devices are doing it with a gpio or some of the more
> configurable PWM units..
>>
>>> Scope may well be wider but includes:
>>> * input (some of it)
>>> * hwmon
>>> * iio
>>> * comedi(?)
>>> * thermal
>>> * power/battery
>>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
>>> * v4l - when does a device jump from being a multi pixel thermopile
>>> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
>>> sensors (effectively 9ish pixel cameras)
>>> * Lots of random things we haven't seen yet.
>>>
>>> All these make use of devices that are at their hearts ADCs. Whilst it
>>> would nice if the underlying devices always fell into one clear
>>> category that is not always the case. Hardware descriptions (e.g.
>>> device tree) are an important area of contention.
>>>
>>> The examples that follow are mostly based around IIO interactions with
>>> hwmon / input as that's my area of expertise. However, I know that
>>> similar issues occur at other boundaries.
>>>
>>> Guenter Roeck has been working on hwmon / thermal interfacing recently.
>>> We also regularly push drivers one way or the other around that boundary.
>>>
>>> Dmitry and I often have to decide on the IIO / input scope boundary
>>> (and we've moved it over time) + a number of cross over drivers have
>>> turned up recently.
>>>
>>> The other subsystems I encounter have been 'quieter' in their
>>> interactions with me but many of the issues below apply to them as well.
>>>
>>> A rough list of related topics that might be worth face to face
>>> discussion includes:
>>>
>>> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
>>> stand?
>>>
>>> I am personally happy enough with this, but it's worth checking
>>> everyone else is. This probably causes more pain for new submitters
>>> than it does for old hands. Perhaps we can come up with a checklist style
>>> doc to save people having to ask.
>>>
>>> We have had quite a few cases where a whole driver is submitted to the
>>> 'wrong' subsystem. Not the best first interaction with the kernel
>>> world for new contributors. (This sort of area is attracts newbies
>>> as it's relatively simple and the hardware is fairly cheap and more
>>> than once their first experience with review as exactly this!)
>>>
>>> If we aren't happy, what do we do about it?
>>>
>>> I'm hoping we are happy or, at least, resigned to the current state
>>> of affairs, but think the question should be asked from time to time
>>> as the answer may well change + the discussion will highlight pinch
>>> points we can work on.
>>>
>>> 2) Bridging drivers - there will always be cross over cases.
>>> E.g. Generic ADCs used for battery voltage monitoring.  We have a
>>> number of bridges in mainline already and others out of tree.  How
>>> important are these and what features would make them more useful?
>>>
>>> Is it ever sensible to have two drivers for the same part because
>>> the use cases are so different? (I hope there is always a better way
>>> but maybe not)
>>
>> I'd love to say no, but we have already hit that problem. There are two 
>> drivers for the ADV7511 HDMI encoder, one in the media subsystem for V4L2 
>> output devices (drivers/media/i2c/adv7511.c), and one in the DRM/KMS subsystem 
>> for display devices (drivers/gpu/drm/i2c/adv7511.c, being moved to 
>> drivers/gpu/drm/bridge/). Try compiling them both in the kernel and see how 
>> they race to probe the same device (enjoy the show, popcorns not included).
> Nasty.
>>
>>> This covers both generic bridges (e.g. iio-hwmon) and device
>>> specific bridges (a good example of a touch screen driver turned
>>> up in my inbox yesterday).
>>>
>>> 3) Bindings (device tree and similar).  When it is appropriate to use a
>>> device tree to describe the overlap (bridging of channels)?
>>> Sometimes there is obvious real hardware involved (a thermistor on a
>>> generic ADC input for example), but there are many grey areas.
>>>
>>> Finally bashing out an answer to the issue device tree maintainers
>>> have with the iio-hwmon bindings would be great.
>>> On a less biased note, that example is pretty fiddly to define but would
>>> act as a good basis to work from more generally.
>>>
>>> We know we got it wrong (back in the dark ages), but it isn't obvious how
>>> to get it right!
>>>
>>> IIO-input has been partly held outside mainline for years by this issue.
>>>
>>> Also some practical design decisions to make around how to implement
>>> these mappings to work best with deferred probing etc.
>>>
>>> 4) Should we drop all the bindings for bridging between such subsystems
>>> and do it all from userspace?
>>>
>>> I think we may end up with a hybrid of the two, but need to be able to
>>> make it work in 'standard' cases without userspace being involved. That
>>> hybrid solution may well be devicetree overlay based... Unclear so far.
>>> + plenty of crazy things that 'might' make sense where there isn't even
>>> the pretence of representing real hardware.
>>>
>>> 5) Complex device interaction usecases.  At the moment the ones I've come
>>> across are mostly contained in IIO.  The moment we start sticking in
>>> MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
>>> mix it can get fiddly.  Swapping war stories may well be worthwhile on
>>> this. This stuff also turns up in ASoC for example so probably lessons
>>> to be learned from there.
>>>
>>> The analog devices software defined radios are another possible case
>>> study.
>>>
>>> There of course may well be lessons to be learned from similar
>>> interactions elsewhere in the kernel.
>>>
>>> There is a lot of history in how we ended up where we are (it all made
>>> absolute sense at the time). Sitting back and taking the time
>>> to discuss the future would be great.  Whilst this might be solvable
>>> by email we've made no definitive progress for years
>>> (and what has been made has been on a case by case basis deep in
>>> driver reviews.)
>>>
>>> I threw comedi in the list above but, at the moment, I think the more
>>> likely direction there is a single userspace library abstracting
>>> the underlying subsystem (Analog Devices are working in that
>>> direction - perhaps Lars can offer more on that?).
>>>
>>> GPIO is another interesting case - a lot of hardware is capable of
>>> parallel sampling, some at high speeds. It's another area that
>>> is probably too specialist for this discussion, but if people want
>>> to dive into the details it might be interesting.
>>>
>>> I think we have only a small amount of fuzz around the v4l boundary,
>>> but wanted to leave the door open if anyone wants to discuss that
>>> one further as it's come up a few times over recent years.
>>
>> Don't forget to take system integration into account. If I given you a high-
>> speed ADC you will not think about V4L2 as your subsystem of choice. If the 
>> system designer has connected that ADC to a CPLD that generates fake 
>> horizontal and vertical sync signals, and connected the output to the camera 
>> interface of the SoC, you will be left with no choice but use the V4L2 API. 
>> That's largely a userspace issue in this case, but it implies that V4L2 need 
>> to define an "image format" for the ADC data.
> This one was somewhat of a thought experiment (and the sort of dirty hack
> we'd do in my day job :)
> 
> Absolutely.  Sometimes they'll obligingly connect it to a camera specific
> interface sometimes the won't.  I think I saw an example of a camera hanging
> of one of TI's universal parallel ports at one point for example - not
> sure how the framing worked there.

One of the first v4l drivers was for a parallel port webcam.

And I just heard of a device that uses spi for the video transfer. HW designers
are evil...

Regards,

	Hans

> 
> The many generic high speed serial interfaces are also an option.
> 
> If it's all wrapped up and presented as a camera device, then we don't need
> to care that it's a generic ADC really as the interesting handling will
> probably all be done in the fpga / cpld anyway.
> 
> The possibility that gets more interesting from my point of view is where
> the pixel stream simply hits an ADC.  Grabbing from the day job, a line
> scan camera would be the classic.  There'd probably still be some sort
> of line framing but lots of places to hide that including in the analog
> stream itself.
> 
> Even if they do, there always comes a point where it's easier to just declare
> that we are better off with a monolithic device driver hiding what
> is inside.
> 
> Anyhow, I kind of hope no one ever does this *crosses fingers*
> 
> Possibly more likely around the blurry boundary of what is a video
> capture device and what is simply taking a small number of specially
> related signals and sampling them...
>>
>>> The SoC world is a major case of one device, many uses.  Some SoCs
>>> are turning up with multiple ADC units, sometimes with different
>>> designs, sometimes simply so that the same hardware can be used for
>>> different things.
>>>
>>> A few suggestions for people:
>>>
>>> Guenter Roeck (hmwon)
>>> Dmitry Torokhov (input)
>>> Me (Jonathan Cameron - IIO)
>>>
>>> Someone thermal related (not sure who would be most interested)
>>> Someone power supply related? (Guenter, any suggestions?)
>>>
>>> Someone device tree related (Mark Rutland? Rob Herring?)
>>>
>>> Linus Waleij (gpio)
>>>
>>> Mark Brown - as some of concepts of IIO bridging and how far it
>>> can be taken (everything via a bridge driver) came from
>>> discussions with him on SoC ADC handling.
>>>
>>> Lar-Peter Clausen for generally doing insane things with ADCs + having
>>> one foot in the userspace side of things.
>>
> 
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
> 

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Vinod Koul]

On Thu, Jul 28, 2016 at 08:53:26PM +0200, Lars-Peter Clausen wrote:

> So new hardware often tends to be general purpose and can be used in
> many different applications.
> 
> But our kernel frameworks are designed around application specific tasks.
> 
> * ALSA is for audio data capture/playback
> * V4L2 is for video data capture/playback
> * DRM is for video display
> * IIO is for sensor data capture/playback

Spot on. Even if you look at kernel source code we have directory based on
end usage..

Also these subsystems expose a user interface for sending/receiving data.
The interface is application specific. One can argue that designing a generic
data interface is kernel which can take care of high volume, bursty or
continuous data, then potentially we can unify these

> When you capture data over a particular interface there is a specific
> meaning associated to the data rather than the data just being data,
> which is how the hardware might see it.
> 
> On the kernel side we have started to address this by having generic
> frameworks like DMAengine. I've used the same DMA core with the same
> DMAengine driver exposed to userspace for all four different types
> listed above depending on the application.

Yes DMAengine framework is generic for application usage. We don't define
the purpose of DMA but we care about nature, memcpy, slave, interleaved...

You can use these for audio, video, RAID or anything else..

> This of course would be a very grand task and maybe we'll loose
> ourselves in endless discussions about the details and all the corner
> cases that need to be considered. But if we want to find a solution that
> keeps up with the current trend that the hardware landscape seems to be
> going in we might have no other choice. Otherwise I'd say it is
> inevitable that we see more and more hardware which has multiple
> drivers, each driver handling a different type of application.

Given that SoCs are becoming complex and HW folks are merging stuff, I
wont be surprised if we get a converged hardware which does multiple
jobs, so we would need these to be unified.

-- 
~Vinod

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Laurent Pinchart]

On Thursday 28 Jul 2016 23:09:59 Jonathan Cameron wrote:
> On 28/07/16 17:42, Laurent Pinchart wrote:
> > On Thursday 28 Jul 2016 08:58:46 Mauro Carvalho Chehab wrote:
> >> Em Wed, 20 Jul 2016 22:18:11 +0100 Jonathan Cameron escreveu:
> >>> Hi All,
> >>> 
> >>> This topic would be around the way the various subsystems interact, in
> >>> the rough area of 'sensors' (I haven't yet had much of an issue with
> >>> subsystem crossing with output devices but maybe that's just over the
> >>> next hill!)
> >> 
> >> Yeah, there is a gray area here as devices become more complex.
> >> So, I'm interested on such topic.
> >> 
> >>> Scope may well be wider but includes:
> >>> * input (some of it)
> >>> * hwmon
> >>> * iio
> >>> * comedi(?)
> >>> * thermal
> >>> * power/battery
> >>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
> >>> * v4l - when does a device jump from being a multi pixel thermopile
> >>> to a thermal camera? Smart fingerprint scanners etc.  Gesture
> >>> recognition sensors (effectively 9ish pixel cameras)
> >> 
> >> For devices that provide images somehow, I'd say that the best would be
> >> to implement them as via the V4L2 API.
> > 
> > This requires defining what an image is. Furthermore, we very low image
> > resolution devices, we will need to deal with high frame rates (1k-10k is
> > a range we need to consider). The V4L2 API will likely show performance
> > issues.
>
> An interesting point I'd never thought about before. Funnily enough the
> devices around this boundary that I've encountered have all been relatively
> slow. Doubt we'll be lucky enough that that will last!

For the sake of completeness, I'd like to add that always-on image sensors 
(for instance https://globenewswire.com/news-release/2016/01/19/802804/0/en/Himax-Launches-Ultra-Low-Power-CMOS-Image-Sensor-for-Always-On-Computer-Vision-Applications.html) will also bring 
interesting challenges. The frame rate won't be particularly high, but the low 
resolution will allow transmitting image data over I2C (or rather I3C, a 
faster I2C-compatible serial bus, see http://mipi.org/specifications/i3c℠-sensor-specification) in a continuous fashion.

> >> We'll likely need to discuss it case by case, specially for early cross-
> >> subsystem drivers, though, as it is not trivial to identify the subsystem
> >> boundaries, and sometimes multiple APIs from different subsystems is
> >> needed. Also, it is not clear where such drivers would fit at the Kernel
> >> tree.
> >> 
> >> One interesting case is an input device driver with multi-finger
> >> support (drivers/input/touchscreen/sur40.c). I suspect we'll have
> >> more cases like that over time.
> >> 
> >>> * Lots of random things we haven't seen yet.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Laurent Pinchart]

Hi Jonathan,

On Thursday 28 Jul 2016 23:26:26 Jonathan Cameron wrote:
> On 28/07/16 17:39, Laurent Pinchart wrote:
> > On Wednesday 20 Jul 2016 22:18:11 Jonathan Cameron wrote:
> >> Hi All,
> >> 
> >> This topic would be around the way the various subsystems interact, in
> >> the rough area of 'sensors' (I haven't yet had much of an issue with
> >> subsystem crossing with output devices but maybe that's just over the
> >> next hill!)
> > 
> > It's already here :-) We have an API to control camera flash (as in light,
> > not memory) in V4L2 and another one in the LED subsystem.
> 
> I wonder how many devices are doing it with a gpio or some of the more
> configurable PWM units..

At least on phone platforms the device usually has a dedicated chip for this, 
as there are safety implications that require a hardware timeout (at the 
highest current leaving the LED flash always on could damage the LED, cause 
the phone to overhead and catch fire, and possibly even damage your eyes).

> >> Scope may well be wider but includes:
> >> * input (some of it)
> >> * hwmon
> >> * iio
> >> * comedi(?)
> >> * thermal
> >> * power/battery
> >> * gpio - the blur between gpios and beaglescope / PLC type I/O.
> >> * v4l - when does a device jump from being a multi pixel thermopile
> >> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
> >> sensors (effectively 9ish pixel cameras)
> >> * Lots of random things we haven't seen yet.
> >> 
> >> All these make use of devices that are at their hearts ADCs. Whilst it
> >> would nice if the underlying devices always fell into one clear
> >> category that is not always the case. Hardware descriptions (e.g.
> >> device tree) are an important area of contention.
> >> 
> >> The examples that follow are mostly based around IIO interactions with
> >> hwmon / input as that's my area of expertise. However, I know that
> >> similar issues occur at other boundaries.
> >> 
> >> Guenter Roeck has been working on hwmon / thermal interfacing recently.
> >> We also regularly push drivers one way or the other around that boundary.
> >> 
> >> Dmitry and I often have to decide on the IIO / input scope boundary
> >> (and we've moved it over time) + a number of cross over drivers have
> >> turned up recently.
> >> 
> >> The other subsystems I encounter have been 'quieter' in their
> >> interactions with me but many of the issues below apply to them as well.
> >> 
> >> A rough list of related topics that might be worth face to face
> >> discussion includes:
> >> 
> >> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
> >> stand?
> >> 
> >> I am personally happy enough with this, but it's worth checking
> >> everyone else is. This probably causes more pain for new submitters
> >> than it does for old hands. Perhaps we can come up with a checklist style
> >> doc to save people having to ask.
> >> 
> >> We have had quite a few cases where a whole driver is submitted to the
> >> 'wrong' subsystem. Not the best first interaction with the kernel
> >> world for new contributors. (This sort of area is attracts newbies
> >> as it's relatively simple and the hardware is fairly cheap and more
> >> than once their first experience with review as exactly this!)
> >> 
> >> If we aren't happy, what do we do about it?
> >> 
> >> I'm hoping we are happy or, at least, resigned to the current state
> >> of affairs, but think the question should be asked from time to time
> >> as the answer may well change + the discussion will highlight pinch
> >> points we can work on.
> >> 
> >> 2) Bridging drivers - there will always be cross over cases.
> >> E.g. Generic ADCs used for battery voltage monitoring.  We have a
> >> number of bridges in mainline already and others out of tree.  How
> >> important are these and what features would make them more useful?
> >> 
> >> Is it ever sensible to have two drivers for the same part because
> >> the use cases are so different? (I hope there is always a better way
> >> but maybe not)
> > 
> > I'd love to say no, but we have already hit that problem. There are two
> > drivers for the ADV7511 HDMI encoder, one in the media subsystem for V4L2
> > output devices (drivers/media/i2c/adv7511.c), and one in the DRM/KMS
> > subsystem for display devices (drivers/gpu/drm/i2c/adv7511.c, being moved
> > to drivers/gpu/drm/bridge/). Try compiling them both in the kernel and
> > see how they race to probe the same device (enjoy the show, popcorns not
> > included).
>
> Nasty.
> 
> >> This covers both generic bridges (e.g. iio-hwmon) and device
> >> specific bridges (a good example of a touch screen driver turned
> >> up in my inbox yesterday).
> >> 
> >> 3) Bindings (device tree and similar).  When it is appropriate to use a
> >> device tree to describe the overlap (bridging of channels)?
> >> Sometimes there is obvious real hardware involved (a thermistor on a
> >> generic ADC input for example), but there are many grey areas.
> >> 
> >> Finally bashing out an answer to the issue device tree maintainers
> >> have with the iio-hwmon bindings would be great.
> >> On a less biased note, that example is pretty fiddly to define but would
> >> act as a good basis to work from more generally.
> >> 
> >> We know we got it wrong (back in the dark ages), but it isn't obvious how
> >> to get it right!
> >> 
> >> IIO-input has been partly held outside mainline for years by this issue.
> >> 
> >> Also some practical design decisions to make around how to implement
> >> these mappings to work best with deferred probing etc.
> >> 
> >> 4) Should we drop all the bindings for bridging between such subsystems
> >> and do it all from userspace?
> >> 
> >> I think we may end up with a hybrid of the two, but need to be able to
> >> make it work in 'standard' cases without userspace being involved. That
> >> hybrid solution may well be devicetree overlay based... Unclear so far.
> >> + plenty of crazy things that 'might' make sense where there isn't even
> >> the pretence of representing real hardware.
> >> 
> >> 5) Complex device interaction usecases.  At the moment the ones I've come
> >> across are mostly contained in IIO.  The moment we start sticking in
> >> MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
> >> mix it can get fiddly.  Swapping war stories may well be worthwhile on
> >> this. This stuff also turns up in ASoC for example so probably lessons
> >> to be learned from there.
> >> 
> >> The analog devices software defined radios are another possible case
> >> study.
> >> 
> >> There of course may well be lessons to be learned from similar
> >> interactions elsewhere in the kernel.
> >> 
> >> There is a lot of history in how we ended up where we are (it all made
> >> absolute sense at the time). Sitting back and taking the time
> >> to discuss the future would be great.  Whilst this might be solvable
> >> by email we've made no definitive progress for years
> >> (and what has been made has been on a case by case basis deep in
> >> driver reviews.)
> >> 
> >> I threw comedi in the list above but, at the moment, I think the more
> >> likely direction there is a single userspace library abstracting
> >> the underlying subsystem (Analog Devices are working in that
> >> direction - perhaps Lars can offer more on that?).
> >> 
> >> GPIO is another interesting case - a lot of hardware is capable of
> >> parallel sampling, some at high speeds. It's another area that
> >> is probably too specialist for this discussion, but if people want
> >> to dive into the details it might be interesting.
> >> 
> >> I think we have only a small amount of fuzz around the v4l boundary,
> >> but wanted to leave the door open if anyone wants to discuss that
> >> one further as it's come up a few times over recent years.
> > 
> > Don't forget to take system integration into account. If I given you a
> > high-speed ADC you will not think about V4L2 as your subsystem of
> > choice. If the system designer has connected that ADC to a CPLD that
> > generates fake horizontal and vertical sync signals, and connected the
> > output to the camera interface of the SoC, you will be left with no
> > choice but use the V4L2 API. That's largely a userspace issue in this
> > case, but it implies that V4L2 need to define an "image format" for the
> > ADC data.
> 
> This one was somewhat of a thought experiment (and the sort of dirty hack
> we'd do in my day job :)

It's a use case a customer of mine implemented in a real product :-)

> Absolutely.  Sometimes they'll obligingly connect it to a camera specific
> interface sometimes the won't.  I think I saw an example of a camera hanging
> of one of TI's universal parallel ports at one point for example - not sure
> how the framing worked there.
> 
> The many generic high speed serial interfaces are also an option.
> 
> If it's all wrapped up and presented as a camera device, then we don't need
> to care that it's a generic ADC really as the interesting handling will
> probably all be done in the fpga / cpld anyway.
> 
> The possibility that gets more interesting from my point of view is where
> the pixel stream simply hits an ADC.  Grabbing from the day job, a line
> scan camera would be the classic.  There'd probably still be some sort
> of line framing but lots of places to hide that including in the analog
> stream itself.
> 
> Even if they do, there always comes a point where it's easier to just
> declare that we are better off with a monolithic device driver hiding what
> is inside.
> 
> Anyhow, I kind of hope no one ever does this *crosses fingers*
> 
> Possibly more likely around the blurry boundary of what is a video
> capture device and what is simply taking a small number of specially
> related signals and sampling them...
> 
> >> The SoC world is a major case of one device, many uses.  Some SoCs
> >> are turning up with multiple ADC units, sometimes with different
> >> designs, sometimes simply so that the same hardware can be used for
> >> different things.
> >> 
> >> A few suggestions for people:
> >> 
> >> Guenter Roeck (hmwon)
> >> Dmitry Torokhov (input)
> >> Me (Jonathan Cameron - IIO)
> >> 
> >> Someone thermal related (not sure who would be most interested)
> >> Someone power supply related? (Guenter, any suggestions?)
> >> 
> >> Someone device tree related (Mark Rutland? Rob Herring?)
> >> 
> >> Linus Waleij (gpio)
> >> 
> >> Mark Brown - as some of concepts of IIO bridging and how far it
> >> can be taken (everything via a bridge driver) came from
> >> discussions with him on SoC ADC handling.
> >> 
> >> Lar-Peter Clausen for generally doing insane things with ADCs + having
> >> one foot in the userspace side of things.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Laurent Pinchart]

Hi Hans,

On Friday 29 Jul 2016 09:28:06 Hans Verkuil wrote:
> On 07/28/2016 06:42 PM, Laurent Pinchart wrote:
> > On Thursday 28 Jul 2016 08:58:46 Mauro Carvalho Chehab wrote:
> >> Em Wed, 20 Jul 2016 22:18:11 +0100 Jonathan Cameron escreveu:
> >>> Hi All,
> >>> 
> >>> This topic would be around the way the various subsystems interact, in
> >>> the rough area of 'sensors' (I haven't yet had much of an issue with
> >>> subsystem crossing with output devices but maybe that's just over the
> >>> next hill!)
> >> 
> >> Yeah, there is a gray area here as devices become more complex.
> >> So, I'm interested on such topic.
> >> 
> >>> Scope may well be wider but includes:
> >>> * input (some of it)
> >>> * hwmon
> >>> * iio
> >>> * comedi(?)
> >>> * thermal
> >>> * power/battery
> >>> * gpio - the blur between gpios and beaglescope / PLC type I/O.
> >>> * v4l - when does a device jump from being a multi pixel thermopile
> >>> to a thermal camera? Smart fingerprint scanners etc.  Gesture
> >>> recognition sensors (effectively 9ish pixel cameras)
> >> 
> >> For devices that provide images somehow, I'd say that the best would be
> >> to implement them as via the V4L2 API.
> > 
> > This requires defining what an image is. Furthermore, we very low image
> > resolution devices, we will need to deal with high frame rates (1k-10k is
> > a range we need to consider). The V4L2 API will likely show performance
> > issues.
>
> The main problems there are the limit to the number of buffers (32) in v4l2
> and that the number of qbuf/dqbuf ioctls is large for these high
> framerates. You would probably want to introduce mqbuf/mdqbuf ioctls to
> queue/dequeue multiple buffers. The core internals of v4l2 (vb2) are all
> pretty low overhead.

Or, just brainstorming, using some kind of ring-buffer based API that doesn't 
require context switches between userspace and kernelspace. That could also be 
somehow useful for larger images, as low-latency video streaming could benefit 
from the ability to start encoding the image before it's fully captured (or 
the other way around, we could start consuming it before it's fully decoded).

> >> We'll likely need to discuss it case by case, specially for early cross-
> >> subsystem drivers, though, as it is not trivial to identify the subsystem
> >> boundaries, and sometimes multiple APIs from different subsystems is
> >> needed. Also, it is not clear where such drivers would fit at the Kernel
> >> tree.
> >> 
> >> One interesting case is an input device driver with multi-finger
> >> support (drivers/input/touchscreen/sur40.c). I suspect we'll have
> >> more cases like that over time.
> >> 
> >>> * Lots of random things we haven't seen yet.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Laurent Pinchart]

Hi Lars,

On Thursday 28 Jul 2016 20:53:26 Lars-Peter Clausen wrote:
> On 07/28/2016 06:39 PM, Laurent Pinchart wrote:
> [...]
> 
> >> I think we have only a small amount of fuzz around the v4l boundary,
> >> but wanted to leave the door open if anyone wants to discuss that
> >> one further as it's come up a few times over recent years.
> > 
> > Don't forget to take system integration into account. If I given you a
> > high- speed ADC you will not think about V4L2 as your subsystem of
> > choice. If the system designer has connected that ADC to a CPLD that
> > generates fake horizontal and vertical sync signals, and connected the
> > output to the camera interface of the SoC, you will be left with no
> > choice but use the V4L2 API. That's largely a userspace issue in this
> > case, but it implies that V4L2 need to define an "image format" for the
> > ADC data.
> 
> I think this hits the core of this discussion. Todays hardware is
> getting more and more generic. It is a lot more economical to produce a
> single general purpose high-volume device than a handful of low or
> medium volume specialized devices. Even if the raw production cost of
> the general purpose part will be higher (since it contains more logic)
> the overall per unit price will be lower since the per part contribution
> of the one-time design cost is lower in a high volume run.
> 
> So new hardware often tends to be general purpose and can be used in
> many different applications.
> 
> But our kernel frameworks are designed around application specific tasks.
> 
> * ALSA is for audio data capture/playback
> * V4L2 is for video data capture/playback
> * DRM is for video display
> * IIO is for sensor data capture/playback
> 
> When you capture data over a particular interface there is a specific
> meaning associated to the data rather than the data just being data,
> which is how the hardware might see it.
> 
> On the kernel side we have started to address this by having generic
> frameworks like DMAengine. I've used the same DMA core with the same
> DMAengine driver exposed to userspace for all four different types
> listed above depending on the application.
> 
> This works as long as you know that your hardware is generic and you
> design the driver to be generic. But it breaks if your hardware has a
> primary function that is application specific.
> 
> E.g. a CSI-2 receiver will most likely receive video data, so we write a
> V4L2 driver for it. A I2S receiver will most likely receive audio data,
> so we write a ALSA driver for it. But now somebody might decide to hook
> up a gyroscope to any of these interfaces because that might be the best
> way to feed data into the particular SoC used in that system. And than
> things start to fall apart.
> 
> And this is not just hypothetical I've seen questions how to make this
> work repeatedly. And I also expect that in a time constraint environment
> people will go ahead with a custom solution where they capture audio
> data through V4L2 and just ignore all data type hints V4L2 provides and
> re-interprets the data since their specialized application knows what
> the data layout looks like.
> 
> A similar issue is that there are quite a few pieces of hardware that
> are multi-use. E.g. general purpose serial data cores that support SPI,
> I2S, and similar. At the moment we have to write two different drivers
> for them using compatible strings to decide which function they should have.
> 
> So going forward we might have to address this by creating a more
> generic interface that allows us to exchange data between a peripheral
> and a application without assigning any kind of meaning to the data
> itself. And then have that meaning provided through side channels. E.g.
> a V4L2 device could say this over there is my data capture device and
> the data layout is the following. Similar for the other frameworks that
> allow capture/playback.
> 
> With vb2 (former V4L2 buffer handling code) now being independent from
> the V4L2 framework this might be a prime candidate as a starting point.
> I've been meaning to re-write the IIO DMA buffer code on top of vb2 to
> reduce the amount of custom code.
> 
> This of course would be a very grand task and maybe we'll loose
> ourselves in endless discussions about the details and all the corner
> cases that need to be considered. But if we want to find a solution that
> keeps up with the current trend that the hardware landscape seems to be
> going in we might have no other choice. Otherwise I'd say it is
> inevitable that we see more and more hardware which has multiple
> drivers, each driver handling a different type of application.

You have a very good point here. It might be time decoupling the control and 
data parts of our userspace APIs. The buffer queued-based method of data 
passing is very similar between V4L2 and IIO as far as I understand, there 
might not be any good reason to have two APIs there apart from historical 
reasons. When it comes to controlling the device, though, video and I/O are 
pretty different.

If we were designing this from scratch today tt could thus make sense to 
standardize cross-subsystem methods for passing large quantities of data 
between kernelspace and userspace (buffer queues and ring buffers come to 
mind), with a set of common API elements (most likely IOCTLs). As we have to 
deal with our history I'm not sure what latitude we still have to fix the 
problem.

> Such a grand unified media framework would also help for applications
> where multiple streams of different data types need to be synchronized
> e.g. audio and video.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Linus Walleij]

On Thu, Jul 28, 2016 at 6:46 PM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> On Friday 22 Jul 2016 14:04:07 Linus Walleij wrote:

>> For an oscilloscope or signal analyzer usecase with more than
>> 64 channels needing to be sampled simultaneously, we may need
>> to add another "large GPIO read" ABI, and also probably a new
>> driver API interface to achieve it. I've never seen hardware that can
>> read more values in parallel than 32 though, I chose 64 to get some
>> headroom.
>
> Such a use case would put strong real-time constraints on the system, I wonder
> how useful a GPIO-based signal analyzer would be. Are you aware of any GPIO
> hardware that can be programmed to sample at a precise point in time (or
> perhaps even periodically) ?

Nope. I would expect to close the gap to IIO and reuse their trigger system
in that case, instead of creating a new one. IIO uses either hardware
trigger interrupts (usually a fixed frequency, some supply hardware
timestamps with the interrupts) or alternatively hrtimers, which in turn
would I guess be most reliable using RT patches.

Yours,
Linus Walleij

Re: [Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

[Linus Walleij]

On Thu, Jul 28, 2016 at 8:08 PM, Lars-Peter Clausen <lars@metafoo.de> wrote:
> On 07/28/2016 06:46 PM, Laurent Pinchart wrote:
>> Are you aware of any GPIO
>> hardware that can be programmed to sample at a precise point in time (or
>> perhaps even periodically) ?
>
> I have such hardware. It's custom hardware, non-public at the moment.
>
> But it works like this, for each pin you can configure whether it can be
> controlled independently, like a normal GPIO, or whether the values are
> sampled/updated to/from a continuous data stream that is regularly
> clocked (basically a 1-bit ADC/DAC). This falls right on the edge
> between GPIO and IIO and I'm really torn where to put it. For now I went
> with IIO since it predates the chardev GPIO interface and because of all
> the libiio magic.

I guess either GPIO should reuse the IIO triggers, or we create something
in IIO that uses GPIO from inside the kernel and add get_multiple() as
similar to set_multiple() to the API and thus make a generic GPIO sampler
driver in IIO that can reuse the IIO triggers, while using a plain GPIO
driver as back-end. If the APIs can be made precise enough.

We can even add get_multiple_with_timestamp() if need be to be
able to contain it in gpiolib.

Yours,
Linus Walleij