[RFC] HID and multitouch

[RFC] HID and multitouch

[Stéphane Chatty]

Dear Jiri,

over the last few months I have been working on HID quirks for a  
bunch of multitouch devices: NTrig, two flavours of Stantum devices,  
3M, Acer T230H. For each of these I have working code, which I should  
be able to submit real soon now.

However, people are asking me questions about why a specific driver  
is needed for each device. Answers are starting to emerge, and it  
looks like they might lead to some work on the Linux HID code itself:

- the one-to-one mapping between HID and evdev is stretched to its  
limits with these devices that have several times the same group of  
HID fields in the same event: an event is made of fingers, each  
finger is made of an X, a Y and a few other fields. We need a  
mechanism to emit SYN_MT_REPORT between fingers.

- furthermore, some fingers have to be discarded because they are  
placeholders with no valid data, and we know if to dump them only  
when they have been fully parsed. Therefore, we need a mechanism to  
cache finger data until the decision is made. Currently, the caching  
mechanism has to be device-dependent because the fields are not  
exactly the same and they come in a device-dependent order.

- for all these devices, it is very tempting to have a touchscreen  
emulation so as to use them with existing software (X.org,  
particularly). Rafi and I share the idea that this should be done by  
creating two evdev nodes: a backward-compatible touchscreen, and a  
multitouch one. Most of the corresponding code could be shared among  
drivers. However, the algorithm for producing the touchscreen events  
differs from one device to another and we need to support this.


Any opinions or suggestions?

Cheers

Stéphane

--
To unsubscribe from this list: send the line "unsubscribe linux-input" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] HID and multitouch

[Mohamed Ikbel Boulabiar]

Hi,

For the first issue, it seems that the mapping should be put outside
the kernel itself.
Or in a more precise and possible solution, to create a small
component which will do the mapping.

For simple devices it can be done in textual mapping files, but for
others like multi-touch screens, a module doing dynamic mapping can be
a solution.

Rewriting the component to push it from static one2one mapping to
dynamic one may be a transparent solution for other components in next
layers.

Cheers,

Mohamed-Ikbel



On Sun, Dec 6, 2009 at 11:58 AM, Stéphane Chatty <chatty@enac.fr> wrote:
> Dear Jiri,
>
> over the last few months I have been working on HID quirks for a bunch of
> multitouch devices: NTrig, two flavours of Stantum devices, 3M, Acer T230H.
> For each of these I have working code, which I should be able to submit real
> soon now.
>
> However, people are asking me questions about why a specific driver is
> needed for each device. Answers are starting to emerge, and it looks like
> they might lead to some work on the Linux HID code itself:
>
> - the one-to-one mapping between HID and evdev is stretched to its limits
> with these devices that have several times the same group of HID fields in
> the same event: an event is made of fingers, each finger is made of an X, a
> Y and a few other fields. We need a mechanism to emit SYN_MT_REPORT between
> fingers.
>
> - furthermore, some fingers have to be discarded because they are
> placeholders with no valid data, and we know if to dump them only when they
> have been fully parsed. Therefore, we need a mechanism to cache finger data
> until the decision is made. Currently, the caching mechanism has to be
> device-dependent because the fields are not exactly the same and they come
> in a device-dependent order.
>
> - for all these devices, it is very tempting to have a touchscreen emulation
> so as to use them with existing software (X.org, particularly). Rafi and I
> share the idea that this should be done by creating two evdev nodes: a
> backward-compatible touchscreen, and a multitouch one. Most of the
> corresponding code could be shared among drivers. However, the algorithm for
> producing the touchscreen events differs from one device to another and we
> need to support this.
>
>
> Any opinions or suggestions?
>
> Cheers
>
> Stéphane
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-input" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-input" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] HID and multitouch

[Jiri Kosina]

On Sun, 6 Dec 2009, Stéphane Chatty wrote:

> over the last few months I have been working on HID quirks for a bunch 
> of multitouch devices: NTrig, two flavours of Stantum devices, 3M, Acer 
> T230H. For each of these I have working code, which I should be able to 
> submit real soon now.

Hi Stéphane,

that's an excellent news, thanks!

> However, people are asking me questions about why a specific driver is 
> needed for each device. Answers are starting to emerge, and it looks 
> like they might lead to some work on the Linux HID code itself:
> 
> - the one-to-one mapping between HID and evdev is stretched to its 
> limits with these devices that have several times the same group of HID 
> fields in the same event: an event is made of fingers, each finger is 
> made of an X, a Y and a few other fields. We need a mechanism to emit 
> SYN_MT_REPORT between fingers.

This should be rather striaghtforward, right? I don't see any principal 
issue here. If you provide such patch, I will happuly merge it.

Introducing wrapper around hidinput_hid_event() which drivers for 
multitouch devices will be using should be sufficient, shouldn't it?

> - furthermore, some fingers have to be discarded because they are 
> placeholders with no valid data, and we know if to dump them only when 
> they have been fully parsed. Therefore, we need a mechanism to cache 
> finger data until the decision is made. Currently, the caching mechanism 
> has to be device-dependent because the fields are not exactly the same 
> and they come in a device-dependent order.

What kind of caching is required here? Is there any upper bound 
approximation on the amount of data that need to be cached?

Would simple infrastructure, introducing linked list of [usage, value] 
tuples, together with "flush()" method, that will again only call 
hidinput_hid_event multiple times, be enough to be common for all drivers 
needing this?

Thanks,

-- 
Jiri Kosina
SUSE Labs, Novell Inc.
--
To unsubscribe from this list: send the line "unsubscribe linux-input" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] HID and multitouch

[Rafi Rubin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>> - furthermore, some fingers have to be discarded because they are 
>> placeholders with no valid data, and we know if to dump them only when 
>> they have been fully parsed. Therefore, we need a mechanism to cache 
>> finger data until the decision is made. Currently, the caching mechanism 
>> has to be device-dependent because the fields are not exactly the same 
>> and they come in a device-dependent order.
> 
> What kind of caching is required here? Is there any upper bound 
> approximation on the amount of data that need to be cached?
> 
> Would simple infrastructure, introducing linked list of [usage, value] 
> tuples, together with "flush()" method, that will again only call 
> hidinput_hid_event multiple times, be enough to be common for all drivers 
> needing this?

I can only answer for the ntrig.  To get a correct output of active fingers, we'd have to cache a total of twice the maximum number
of fingers.  The first copy is for the previous state and the second is for the current to check if id's have shifted.

On the upshot, all fingers seem to be transmitted on every input event.  So we don't have to stall input events for longer than the
io and processing time to perform such a correction.


My impression is that most or all of the other devices perform finger tracking in which case they need to buffer at most just the
maximum number of fingers, though if using dynamic structures, might be able to shrink that to the current max contact id (max id of
active inputs).  It sounds like the devices are sending full data, when they just want to transmit a sparse array.


I'm not sure there's a significant benefit to a dynamic data structure for holding the incoming points.  I don't think any of these
devices are capable of reporting more than 5 or 10 points.  With the current firmware, the ntrig only reports 4+pen, though it
reports them as 6 points (id 4 is always transmitted and always 0, and 5 is the pen).  Anyway, we're talking small arrays with just
a little state in each cell.

Rafi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksnqDkACgkQwuRiAT9o60/zCwCfY/eoK2Ys7tBFDg0ITUuJ12WS
iE8AoNhBd+7jeXDK5ViKasbL9icdMcTh
=1Tdr
-----END PGP SIGNATURE-----

Re: [RFC] HID and multitouch

[Stéphane Chatty]

Le 15 déc. 09 à 15:05, Jiri Kosina a écrit :
>>
>> - the one-to-one mapping between HID and evdev is stretched to its
>> limits with these devices that have several times the same group  
>> of HID
>> fields in the same event: an event is made of fingers, each finger is
>> made of an X, a Y and a few other fields. We need a mechanism to emit
>> SYN_MT_REPORT between fingers.
>
> This should be rather striaghtforward, right? I don't see any  
> principal
> issue here. If you provide such patch, I will happuly merge it.
>
> Introducing wrapper around hidinput_hid_event() which drivers for
> multitouch devices will be using should be sufficient, shouldn't it?
>

OK. As soon as I have some time (january?) I'll have a look at this.


>> - furthermore, some fingers have to be discarded because they are
>> placeholders with no valid data, and we know if to dump them only  
>> when
>> they have been fully parsed. Therefore, we need a mechanism to cache
>> finger data until the decision is made. Currently, the caching  
>> mechanism
>> has to be device-dependent because the fields are not exactly the  
>> same
>> and they come in a device-dependent order.
>
> What kind of caching is required here? Is there any upper bound
> approximation on the amount of data that need to be cached?

Usually, we are speaking about x, y, w, h, id, pressure, each between  
16 and 32 bits. If we want to perform single touch emulation (useful  
for using the panels with current applications), some devices require  
that we cache all fingers. In that case, multiply this by 6 to 10.

>
> Would simple infrastructure, introducing linked list of [usage, value]
> tuples, together with "flush()" method, that will again only call
> hidinput_hid_event multiple times, be enough to be common for all  
> drivers
> needing this?


Er, I guess yes. Less comfortable that the current device-specific  
struct, but probably OK.

St.

--
To unsubscribe from this list: send the line "unsubscribe linux-input" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] HID and multitouch

[Jiri Kosina]

On Tue, 15 Dec 2009, Rafi Rubin wrote:

> My impression is that most or all of the other devices perform finger 
> tracking in which case they need to buffer at most just the maximum 
> number of fingers, though if using dynamic structures, might be able to 
> shrink that to the current max contact id (max id of active inputs).  
> It sounds like the devices are sending full data, when they just want to 
> transmit a sparse array.
> 
> I'm not sure there's a significant benefit to a dynamic data structure 
> for holding the incoming points.  I don't think any of these devices are 
> capable of reporting more than 5 or 10 points.  With the current 
> firmware, the ntrig only reports 4+pen, though it reports them as 6 
> points (id 4 is always transmitted and always 0, and 5 is the pen).  
> Anyway, we're talking small arrays with just a little state in each 
> cell.

Thanks for information. If other devices behave similar to n-trig in this 
respect, we can easily have static caching data structure for that then.

-- 
Jiri Kosina
SUSE Labs, Novell Inc.