[PATCH v2 0/4] Input: support virtual objects on touchscreens

[PATCH v2 0/4] Input: support virtual objects on touchscreens

[Javier Carrasco]

Some touchscreens are shipped with a physical layer on top of them where
a number of buttons and a resized touchscreen surface might be available.

In order to generate proper key events by overlay buttons and adjust the
touch events to a clipped surface, these patches offer a documented,
device-tree-based solution by means of helper functions.
An implementation for a specific touchscreen driver is also included.

The functions in ts-virtobj provide a simple workflow to acquire
physical objects from the device tree, map them into the device driver
structures as virtual objects and generate events according to
the object descriptions.

This feature has been tested with a JT240MHQS-E3 display, which consists
of an st1624 as the base touchscreen and an overlay with two buttons and
a frame that clips its effective surface mounted on it.

Signed-off-by: Javier Carrasco <javier.carrasco@wolfvision.net>
---
Changes in v2:
- PATCH 1/4: remove preprocessor directives (the module is selected by
  the drivers that support the feature). Typo in the commit message.
- PATCH 2/4: more detailed documentation. Images and examples were added.
- PATCH 3/4: select ts-virtobj automatically.
- Link to v1: https://lore.kernel.org/r/20230510-feature-ts_virtobj_patch-v1-0-5ae5e81bc264@wolfvision.net

---
Javier Carrasco (4):
      Input: ts-virtobj - Add touchscreen virtual object handling
      dt-bindings: touchscreen: add virtual-touchscreen and virtual-buttons properties
      Input: st1232 - add virtual touchscreen and buttons handling
      dt-bindings: input: touchscreen: st1232: add example with ts-virtobj

 .../input/touchscreen/sitronix,st1232.yaml         |  40 +++
 .../bindings/input/touchscreen/touchscreen.yaml    | 139 ++++++++
 MAINTAINERS                                        |   7 +
 drivers/input/touchscreen/Kconfig                  |  10 +
 drivers/input/touchscreen/Makefile                 |   1 +
 drivers/input/touchscreen/st1232.c                 |  87 +++--
 drivers/input/touchscreen/ts-virtobj.c             | 356 +++++++++++++++++++++
 include/linux/input/ts-virtobj.h                   |  43 +++
 8 files changed, 665 insertions(+), 18 deletions(-)
---
base-commit: ac9a78681b921877518763ba0e89202254349d1b
change-id: 20230510-feature-ts_virtobj_patch-e267540aae74

Best regards,
-- 
Javier Carrasco <javier.carrasco@wolfvision.net>

[PATCH v2 1/4] Input: ts-virtobj - Add touchscreen virtual object handling

[Javier Carrasco]

Some touchscreens provide mechanical overlays with different objects
like buttons or clipped touchscreen surfaces.

In order to support these objects, add a series of helper functions
to the input subsystem to transform them into virtual objects via
device tree nodes.

These virtual objects consume the raw touch events and report the
expected input events depending on the object properties.

Signed-off-by: Javier Carrasco <javier.carrasco@wolfvision.net>
---
 MAINTAINERS                            |   7 +
 drivers/input/touchscreen/Kconfig      |   9 +
 drivers/input/touchscreen/Makefile     |   1 +
 drivers/input/touchscreen/ts-virtobj.c | 356 +++++++++++++++++++++++++++++++++
 include/linux/input/ts-virtobj.h       |  43 ++++
 5 files changed, 416 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 7e0b87d5aa2e..296f71bcfe92 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -21434,6 +21434,13 @@ W:	https://github.com/srcres258/linux-doc
 T:	git git://github.com/srcres258/linux-doc.git doc-zh-tw
 F:	Documentation/translations/zh_TW/
 
+TOUCHSCREEN VIRTUAL OBJECTS
+M:	Javier Carrasco <javier.carrasco@wolfvision.net>
+L:	linux-input@vger.kernel.org
+S:	Maintained
+F:	drivers/input/touchscreen/ts-virtobj.c
+F:	include/linux/input/ts-virtobj.h
+
 TTY LAYER
 M:	Greg Kroah-Hartman <gregkh@linuxfoundation.org>
 M:	Jiri Slaby <jirislaby@kernel.org>
diff --git a/drivers/input/touchscreen/Kconfig b/drivers/input/touchscreen/Kconfig
index 143ff43c67ae..0bdbe0a77b34 100644
--- a/drivers/input/touchscreen/Kconfig
+++ b/drivers/input/touchscreen/Kconfig
@@ -1388,4 +1388,13 @@ config TOUCHSCREEN_HIMAX_HX83112B
 	  To compile this driver as a module, choose M here: the
 	  module will be called himax_hx83112b.
 
+config TOUCHSCREEN_TS_VIRTOBJ
+	tristate "Touchscreen Virtual Objects"
+	help
+	  Say Y here if you are using a touchscreen driver that supports
+	  printed overlays with keys or a clipped touchscreen area.
+
+	  Should be selected by the touchscren drivers that support
+	  this feature.
+
 endif
diff --git a/drivers/input/touchscreen/Makefile b/drivers/input/touchscreen/Makefile
index 159cd5136fdb..dc315d58a03b 100644
--- a/drivers/input/touchscreen/Makefile
+++ b/drivers/input/touchscreen/Makefile
@@ -117,3 +117,4 @@ obj-$(CONFIG_TOUCHSCREEN_RASPBERRYPI_FW)	+= raspberrypi-ts.o
 obj-$(CONFIG_TOUCHSCREEN_IQS5XX)	+= iqs5xx.o
 obj-$(CONFIG_TOUCHSCREEN_ZINITIX)	+= zinitix.o
 obj-$(CONFIG_TOUCHSCREEN_HIMAX_HX83112B)	+= himax_hx83112b.o
+obj-$(CONFIG_TOUCHSCREEN_TS_VIRTOBJ)	+= ts-virtobj.o
diff --git a/drivers/input/touchscreen/ts-virtobj.c b/drivers/input/touchscreen/ts-virtobj.c
new file mode 100644
index 000000000000..07e5611b7b3d
--- /dev/null
+++ b/drivers/input/touchscreen/ts-virtobj.c
@@ -0,0 +1,356 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ *  Helper functions for virtual objects on touchscreens
+ *
+ *  Copyright (c) 2023 Javier Carrasco <javier.carrasco@wolfvision.net>
+ */
+
+#include <linux/property.h>
+#include <linux/input.h>
+#include <linux/input/mt.h>
+#include <linux/module.h>
+#include <linux/input/ts-virtobj.h>
+
+enum ts_virtobj_valid_objects {
+	TOUCHSCREEN,
+	BUTTON,
+};
+
+static const char *const ts_virtobj_names[] = {
+	[TOUCHSCREEN] = "virtual-touchscreen",
+	[BUTTON] = "virtual-buttons",
+};
+
+struct ts_virtobj_shape {
+	u32 x_origin;
+	u32 y_origin;
+	u32 x_size;
+	u32 y_size;
+};
+
+struct ts_virtobj_button {
+	struct ts_virtobj_shape shape;
+	u32 key;
+	bool pressed;
+	int slot;
+};
+
+static int ts_virtobj_get_shape_properties(struct fwnode_handle *child_node,
+					   struct ts_virtobj_shape *shape)
+{
+	int rc;
+
+	rc = fwnode_property_read_u32(child_node, "x-origin", &shape->x_origin);
+	if (rc < 0)
+		return rc;
+
+	rc = fwnode_property_read_u32(child_node, "y-origin", &shape->y_origin);
+	if (rc < 0)
+		return rc;
+
+	rc = fwnode_property_read_u32(child_node, "x-size", &shape->x_size);
+	if (rc < 0)
+		return rc;
+
+	rc = fwnode_property_read_u32(child_node, "y-size", &shape->y_size);
+	if (rc < 0)
+		return rc;
+
+	return 0;
+}
+
+static int ts_virtobj_get_button_properties(struct device *dev,
+					    struct fwnode_handle *child_node,
+					    struct ts_virtobj_button *btn)
+{
+	struct fwnode_handle *child_btn;
+	int rc;
+	int j = 0;
+
+	fwnode_for_each_child_node(child_node, child_btn) {
+		rc = ts_virtobj_get_shape_properties(child_btn, &btn[j].shape);
+		if (rc < 0)
+			goto button_prop_cleanup;
+
+		rc = fwnode_property_read_u32(child_btn, "linux,code",
+					      &btn[j].key);
+		if (rc < 0)
+			goto button_prop_cleanup;
+
+		dev_info(dev, "Added button at (%u, %u), size %ux%u, code=%u\n",
+			 btn[j].shape.x_origin, btn[j].shape.y_origin,
+			 btn[j].shape.x_size, btn[j].shape.y_size, btn[j].key);
+		j++;
+	}
+
+	return 0;
+
+button_prop_cleanup:
+	fwnode_handle_put(child_btn);
+	return rc;
+}
+
+void ts_virtobj_set_button_caps(struct ts_virtobj_map *map,
+				struct input_dev *dev)
+{
+	int i;
+
+	for (i = 0; i < map->button_count; i++)
+		input_set_capability(dev, EV_KEY, map->buttons[i].key);
+}
+EXPORT_SYMBOL(ts_virtobj_set_button_caps);
+
+static int ts_virtobj_count_buttons(struct device *dev)
+{
+	struct fwnode_handle *child_node;
+	struct fwnode_handle *child_button;
+	int count = 0;
+
+	child_node = device_get_named_child_node(dev, ts_virtobj_names[BUTTON]);
+	if (!child_node)
+		return 0;
+
+	fwnode_for_each_child_node(child_node, child_button)
+		count++;
+	fwnode_handle_put(child_node);
+
+	return count;
+}
+
+static int ts_virtobj_map_touchscreen(struct device *dev,
+				      struct ts_virtobj_map *map)
+{
+	struct fwnode_handle *child;
+	int rc = 0;
+
+	child = device_get_named_child_node(dev, ts_virtobj_names[TOUCHSCREEN]);
+	if (!child)
+		goto touchscreen_ret;
+
+	map->touchscreen =
+		devm_kzalloc(dev, sizeof(*map->touchscreen), GFP_KERNEL);
+	if (!map->touchscreen) {
+		rc = -ENOMEM;
+		goto touchscreen_handle;
+	}
+	rc = ts_virtobj_get_shape_properties(child, map->touchscreen);
+	if (rc < 0)
+		goto touchscreen_free;
+
+	map->virtual_touchscreen = true;
+	dev_info(dev, "Added virtual touchscreen at (%u, %u), size %u x %u\n",
+		 map->touchscreen->x_origin, map->touchscreen->y_origin,
+		 map->touchscreen->x_size, map->touchscreen->y_size);
+
+	rc = 0;
+	goto touchscreen_handle;
+
+touchscreen_free:
+	devm_kfree(dev, map->touchscreen);
+touchscreen_handle:
+	fwnode_handle_put(child);
+touchscreen_ret:
+	return rc;
+}
+
+static int ts_virtobj_map_buttons(struct device *dev,
+				  struct ts_virtobj_map *map,
+				  struct input_dev *input)
+{
+	struct fwnode_handle *child;
+	u32 button_count;
+	int rc = 0;
+
+	button_count = ts_virtobj_count_buttons(dev);
+	if (button_count) {
+		map->buttons = devm_kcalloc(dev, button_count,
+					    sizeof(*map->buttons), GFP_KERNEL);
+		if (!map->buttons) {
+			rc = -ENOMEM;
+			goto map_buttons_ret;
+		}
+		child = device_get_named_child_node(dev,
+						    ts_virtobj_names[BUTTON]);
+		if (unlikely(!child))
+			goto map_buttons_free;
+
+		rc = ts_virtobj_get_button_properties(dev, child, map->buttons);
+		if (rc < 0)
+			goto map_buttons_free;
+
+		map->button_count = button_count;
+	}
+
+	return 0;
+
+map_buttons_free:
+	devm_kfree(dev, map->buttons);
+map_buttons_ret:
+	return rc;
+}
+
+static bool ts_virtobj_defined_objects(struct device *dev)
+{
+	struct fwnode_handle *child;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(ts_virtobj_names); i++) {
+		child = device_get_named_child_node(dev, ts_virtobj_names[i]);
+		if (child) {
+			fwnode_handle_put(child);
+			return true;
+		}
+		fwnode_handle_put(child);
+	}
+
+	return false;
+}
+
+struct ts_virtobj_map *ts_virtobj_map_objects(struct device *dev,
+					      struct input_dev *input)
+{
+	struct ts_virtobj_map *map = NULL;
+	int rc;
+
+	if (!ts_virtobj_defined_objects(dev))
+		return NULL;
+
+	map = devm_kzalloc(dev, sizeof(*map), GFP_KERNEL);
+	if (!map) {
+		rc = -ENOMEM;
+		goto objects_err;
+	}
+	rc = ts_virtobj_map_touchscreen(dev, map);
+	if (rc < 0)
+		goto objects_free;
+
+	rc = ts_virtobj_map_buttons(dev, map, input);
+	if (rc < 0)
+		goto objects_free;
+
+	return map;
+
+objects_free:
+	devm_kfree(dev, map);
+objects_err:
+	return ERR_PTR(rc);
+}
+EXPORT_SYMBOL(ts_virtobj_map_objects);
+
+void ts_virtobj_get_touchscreen_abs(struct ts_virtobj_map *map, u16 *x, u16 *y)
+{
+	*x = map->touchscreen->x_size - 1;
+	*y = map->touchscreen->y_size - 1;
+}
+EXPORT_SYMBOL(ts_virtobj_get_touchscreen_abs);
+
+static bool ts_virtobj_shape_event(struct ts_virtobj_shape *shape, u32 x, u32 y)
+{
+	if (!shape)
+		return false;
+
+	if (x >= shape->x_origin && x < (shape->x_origin + shape->x_size) &&
+	    y >= shape->y_origin && y < (shape->y_origin + shape->y_size))
+		return true;
+
+	return false;
+}
+
+static bool ts_virtobj_touchscreen_event(struct ts_virtobj_shape *touchscreen,
+					 u32 *x, u32 *y)
+{
+	if (ts_virtobj_shape_event(touchscreen, *x, *y)) {
+		*x -= touchscreen->x_origin;
+		*y -= touchscreen->y_origin;
+		return true;
+	}
+
+	return false;
+}
+
+bool ts_virtobj_mapped_touchscreen(struct ts_virtobj_map *map)
+{
+	if (!map || !map->virtual_touchscreen)
+		return false;
+
+	return true;
+}
+EXPORT_SYMBOL(ts_virtobj_mapped_touchscreen);
+
+bool ts_virtobj_mapped_buttons(struct ts_virtobj_map *map)
+{
+	if (!map || !map->button_count)
+		return false;
+
+	return true;
+}
+EXPORT_SYMBOL(ts_virtobj_mapped_buttons);
+
+bool ts_virtobj_mt_on_touchscreen(struct ts_virtobj_map *map, u32 *x, u32 *y)
+{
+	if (!ts_virtobj_mapped_touchscreen(map))
+		return true;
+
+	if (!ts_virtobj_touchscreen_event(map->touchscreen, x, y))
+		return false;
+
+	return true;
+}
+EXPORT_SYMBOL(ts_virtobj_mt_on_touchscreen);
+
+bool ts_virtobj_button_press(struct ts_virtobj_map *map,
+			     struct input_dev *input, u32 x, u32 y, u32 slot)
+{
+	int i;
+
+	if (!ts_virtobj_mapped_buttons(map))
+		return false;
+
+	for (i = 0; i < map->button_count; i++) {
+		if (ts_virtobj_shape_event(&map->buttons[i].shape, x, y)) {
+			input_report_key(input, map->buttons[i].key, 1);
+			map->buttons[i].pressed = true;
+			map->buttons[i].slot = slot;
+			return true;
+		}
+	}
+
+	return false;
+}
+EXPORT_SYMBOL(ts_virtobj_button_press);
+
+bool ts_virtobj_is_button_slot(struct ts_virtobj_map *map, int slot)
+{
+	int i;
+
+	if (!map || !map->button_count)
+		return false;
+
+	for (i = 0; i < map->button_count; i++) {
+		if (map->buttons[i].pressed && map->buttons[i].slot == slot)
+			return true;
+	}
+
+	return false;
+}
+EXPORT_SYMBOL(ts_virtobj_is_button_slot);
+
+void ts_virtobj_button_release(struct ts_virtobj_map *map,
+			       struct input_dev *input, u32 slot)
+{
+	int i;
+
+	if (!map || !map->button_count)
+		return;
+
+	for (i = 0; i < map->button_count; i++) {
+		if (map->buttons[i].pressed && map->buttons[i].slot == slot) {
+			input_report_key(input, map->buttons[i].key, 0);
+			map->buttons[i].pressed = false;
+		}
+	}
+}
+EXPORT_SYMBOL(ts_virtobj_button_release);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Helper functions for virtual objects on touchscreens");
diff --git a/include/linux/input/ts-virtobj.h b/include/linux/input/ts-virtobj.h
new file mode 100644
index 000000000000..f221b46c4d09
--- /dev/null
+++ b/include/linux/input/ts-virtobj.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2023 Javier Carrasco <javier.carrasco@wolfvision.net>
+ */
+
+#ifndef _TS_VIRTOBJ
+#define _TS_VIRTOBJ
+
+#include <linux/types.h>
+
+struct input_dev;
+struct device;
+
+struct ts_virtobj_map {
+	struct ts_virtobj_shape *touchscreen;
+	bool virtual_touchscreen;
+	struct ts_virtobj_button *buttons;
+	u32 button_count;
+};
+
+struct ts_virtobj_map *ts_virtobj_map_objects(struct device *dev,
+					      struct input_dev *input);
+
+void ts_virtobj_get_touchscreen_abs(struct ts_virtobj_map *map, u16 *x, u16 *y);
+
+bool ts_virtobj_mapped_touchscreen(struct ts_virtobj_map *map);
+
+bool ts_virtobj_mapped_buttons(struct ts_virtobj_map *map);
+
+bool ts_virtobj_mt_on_touchscreen(struct ts_virtobj_map *map, u32 *x, u32 *y);
+
+bool ts_virtobj_button_press(struct ts_virtobj_map *map,
+			     struct input_dev *input, u32 x, u32 y, u32 slot);
+
+bool ts_virtobj_is_button_slot(struct ts_virtobj_map *map, int slot);
+
+void ts_virtobj_button_release(struct ts_virtobj_map *map,
+			       struct input_dev *input, u32 slot);
+
+void ts_virtobj_set_button_caps(struct ts_virtobj_map *map,
+				struct input_dev *dev);
+
+#endif

-- 
2.39.2

[PATCH v2 2/4] dt-bindings: touchscreen: add virtual-touchscreen and virtual-buttons properties

[Javier Carrasco]

The virtual-touchscreen object defines an area within the touchscreen
where touch events are reported and their coordinates get converted to
the virtual origin. This object avoids getting events from areas that
are physically hidden by overlay frames.

For touchscreens where overlay buttons on the touchscreen surface are
provided, the virtual-buttons object contains a node for every button
and the key event that should be reported when pressed.

Signed-off-by: Javier Carrasco <javier.carrasco@wolfvision.net>
---
 .../bindings/input/touchscreen/touchscreen.yaml    | 139 +++++++++++++++++++++
 1 file changed, 139 insertions(+)

diff --git a/Documentation/devicetree/bindings/input/touchscreen/touchscreen.yaml b/Documentation/devicetree/bindings/input/touchscreen/touchscreen.yaml
index 895592da9626..866bfb45a8cf 100644
--- a/Documentation/devicetree/bindings/input/touchscreen/touchscreen.yaml
+++ b/Documentation/devicetree/bindings/input/touchscreen/touchscreen.yaml
@@ -80,6 +80,145 @@ properties:
   touchscreen-y-plate-ohms:
     description: Resistance of the Y-plate in Ohms
 
+  virtual-touchscreen:
+    description: Clipped touchscreen area
+
+      This object can be used to describe a frame that restricts the area
+      within touch events are reported, ignoring the events that occur outside
+      this area. This is of special interest if the touchscreen is shipped
+      with a physical overlay on top of it with a frame that hides some part
+      of the original touchscreen area.
+
+      The x-origin and y-origin properties of this object define the offset of
+      a new origin from where the touchscreen events are referenced.
+      This offset is applied to the events accordingly. The x-size and y-size
+      properties define the size of the virtual-touchscreen (effective area).
+
+      The following example shows the new touchscreen area and the new origin
+      (0',0') for the touch events generated by the device.
+
+                   Touchscreen (full area)
+         ┌────────────────────────────────────────┐
+         │    ┌───────────────────────────────┐   │
+         │    │                               │   │
+         │    ├ y-size                        │   │
+         │    │                               │   │
+         │    │      virtual-touchscreen      │   │
+         │    │                               │   │
+         │    │                               │   │
+         │    │            x-size             │   │
+         │   ┌└──────────────┴────────────────┘   │
+         │(0',0')                                 │
+        ┌└────────────────────────────────────────┘
+      (0,0)
+
+     where (0',0') = (0+x-origin,0+y-origin)
+
+    type: object
+
+    properties:
+      x-origin:
+        description: horizontal origin of the clipped area
+        $ref: /schemas/types.yaml#/definitions/uint32
+
+      y-origin:
+        description: vertical origin of the clipped area
+        $ref: /schemas/types.yaml#/definitions/uint32
+
+      x-size:
+        description: horizontal resolution of the clipped area
+        $ref: /schemas/types.yaml#/definitions/uint32
+
+      y-size:
+        description: vertical resolution of the clipped area
+        $ref: /schemas/types.yaml#/definitions/uint32
+
+  virtual-buttons:
+    description: list of nodes defining the buttons on the touchscreen
+
+      This object can be used to describe buttons on the touchscreen area,
+      reporting the touch events on their surface as key events instead of
+      the original touch events.
+
+      This is of special interest if the touchscreen is shipped with a
+      physical overlay on top of it where a number of buttons with some
+      predefined functionality are printed. In that case a specific behavior
+      is expected from those buttons instead of raw touch events.
+
+      The virtual-buttons properties define a per-button area as well as an
+      origin relative to the real touchscreen origin. Touch events within the
+      button area are reported as the key event defined in the linux,code
+      property. Given that the key events do not provide coordinates, the
+      button origin is only used to place the button area on the touchscreen
+      surface. Any event outside the virtual-buttons object is reported as a
+      touch event with no coordinate transformation.
+
+      The following example shows a touchscreen with a single button on it
+
+              Touchscreen (full area)
+        ┌───────────────────────────────────┐
+        │                                   │
+        │                                   │
+        │   ┌─────────┐                     │
+        │   │button 0 │                     │
+        │   │KEY_POWER│                     │
+        │   └─────────┘                     │
+        │                                   │
+        │                                   │
+       ┌└───────────────────────────────────┘
+     (0,0)
+
+      The virtual-buttons object can  be combined with the virtual-touchscreen
+      object as shown in the following example. In that case only the events
+      within the virtual-touchscreen object are reported as touch events.
+
+                  Touchscreen (full area)
+        ┌─────────┬──────────────────────────────┐
+        │         │                              │
+        │         │    ┌───────────────────────┐ │
+        │ button 0│    │                       │ │
+        │KEY_POWER│    │                       │ │
+        │         │    │                       │ │
+        ├─────────┤    │  virtual-touchscreen  │ │
+        │         │    │                       │ │
+        │         │    │                       │ │
+        │ button 1│    │                       │ │
+        │ KEY_INFO│   ┌└───────────────────────┘ │
+        │         │(0',0')                       │
+       ┌└─────────┴──────────────────────────────┘
+     (0,0)
+
+    type: object
+
+    patternProperties:
+      '^button-':
+        type: object
+        description:
+          Each button (key) is represented as a sub-node.
+
+        properties:
+          label:
+            $ref: /schemas/types.yaml#/definitions/string
+            description: descriptive name of the button
+
+          linux,code: true
+
+          x-origin:
+            description: horizontal origin of the button area
+            $ref: /schemas/types.yaml#/definitions/uint32
+
+          y-origin:
+            description: vertical origin of the button area
+            $ref: /schemas/types.yaml#/definitions/uint32
+
+          x-size:
+            description: horizontal resolution of the button area
+            $ref: /schemas/types.yaml#/definitions/uint32
+
+          y-size:
+            description: vertical resolution of the button area
+            $ref: /schemas/types.yaml#/definitions/uint32
+
 dependencies:
   touchscreen-size-x: [ touchscreen-size-y ]
   touchscreen-size-y: [ touchscreen-size-x ]

-- 
2.39.2

[PATCH v2 3/4] Input: st1232 - add virtual touchscreen and buttons handling

[Javier Carrasco]

Use ts-virtobj to support overlay objects such as buttons and resized
frames defined in the device tree.

If a virtual-touchscreen is provided, ignore touch events outside of
the area defined by its properties. Otherwise, transform the event
coordinates to the virtual-touchscreen x and y-axis if required.

If buttons are provided, register an additional device to report key
events separately. A key event will be generated if the coordinates
of a touch event are within the area defined by the button properties.

Signed-off-by: Javier Carrasco <javier.carrasco@wolfvision.net>
---
 drivers/input/touchscreen/Kconfig  |  1 +
 drivers/input/touchscreen/st1232.c | 87 ++++++++++++++++++++++++++++++--------
 2 files changed, 70 insertions(+), 18 deletions(-)

diff --git a/drivers/input/touchscreen/Kconfig b/drivers/input/touchscreen/Kconfig
index 0bdbe0a77b34..d975ca308439 100644
--- a/drivers/input/touchscreen/Kconfig
+++ b/drivers/input/touchscreen/Kconfig
@@ -1215,6 +1215,7 @@ config TOUCHSCREEN_SIS_I2C
 config TOUCHSCREEN_ST1232
 	tristate "Sitronix ST1232 or ST1633 touchscreen controllers"
 	depends on I2C
+	select TOUCHSCREEN_TS_VIRTOBJ
 	help
 	  Say Y here if you want to support the Sitronix ST1232
 	  or ST1633 touchscreen controller.
diff --git a/drivers/input/touchscreen/st1232.c b/drivers/input/touchscreen/st1232.c
index f49566dc96f8..b8139b7daa40 100644
--- a/drivers/input/touchscreen/st1232.c
+++ b/drivers/input/touchscreen/st1232.c
@@ -22,6 +22,7 @@
 #include <linux/pm_qos.h>
 #include <linux/slab.h>
 #include <linux/types.h>
+#include <linux/input/ts-virtobj.h>
 
 #define ST1232_TS_NAME	"st1232-ts"
 #define ST1633_TS_NAME	"st1633-ts"
@@ -56,6 +57,8 @@ struct st1232_ts_data {
 	struct touchscreen_properties prop;
 	struct dev_pm_qos_request low_latency_req;
 	struct gpio_desc *reset_gpio;
+	struct input_dev *virtual_keypad;
+	struct ts_virtobj_map *map;
 	const struct st_chip_info *chip_info;
 	int read_buf_len;
 	u8 *read_buf;
@@ -129,10 +132,12 @@ static int st1232_ts_read_resolution(struct st1232_ts_data *ts, u16 *max_x,
 
 static int st1232_ts_parse_and_report(struct st1232_ts_data *ts)
 {
-	struct input_dev *input = ts->input_dev;
+	struct input_dev *tscreen = ts->input_dev;
+	__maybe_unused struct input_dev *keypad = ts->virtual_keypad;
 	struct input_mt_pos pos[ST_TS_MAX_FINGERS];
 	u8 z[ST_TS_MAX_FINGERS];
 	int slots[ST_TS_MAX_FINGERS];
+	__maybe_unused bool button_pressed[ST_TS_MAX_FINGERS];
 	int n_contacts = 0;
 	int i;
 
@@ -143,6 +148,15 @@ static int st1232_ts_parse_and_report(struct st1232_ts_data *ts)
 			unsigned int x = ((buf[0] & 0x70) << 4) | buf[1];
 			unsigned int y = ((buf[0] & 0x07) << 8) | buf[2];
 
+			/* forward button presses to the keypad input device */
+			if (ts_virtobj_is_button_slot(ts->map, i) ||
+			    ts_virtobj_button_press(ts->map, keypad, x, y, i)) {
+				button_pressed[i] = true;
+				continue;
+			}
+			/* Ignore events out of the virtual screen if defined */
+			if (!ts_virtobj_mt_on_touchscreen(ts->map, &x, &y))
+				continue;
 			touchscreen_set_mt_pos(&pos[n_contacts],
 					       &ts->prop, x, y);
 
@@ -154,18 +168,25 @@ static int st1232_ts_parse_and_report(struct st1232_ts_data *ts)
 		}
 	}
 
-	input_mt_assign_slots(input, slots, pos, n_contacts, 0);
+	input_mt_assign_slots(tscreen, slots, pos, n_contacts, 0);
 	for (i = 0; i < n_contacts; i++) {
-		input_mt_slot(input, slots[i]);
-		input_mt_report_slot_state(input, MT_TOOL_FINGER, true);
-		input_report_abs(input, ABS_MT_POSITION_X, pos[i].x);
-		input_report_abs(input, ABS_MT_POSITION_Y, pos[i].y);
+		input_mt_slot(tscreen, slots[i]);
+		input_mt_report_slot_state(tscreen, MT_TOOL_FINGER, true);
+		input_report_abs(tscreen, ABS_MT_POSITION_X, pos[i].x);
+		input_report_abs(tscreen, ABS_MT_POSITION_Y, pos[i].y);
 		if (ts->chip_info->have_z)
-			input_report_abs(input, ABS_MT_TOUCH_MAJOR, z[i]);
+			input_report_abs(tscreen, ABS_MT_TOUCH_MAJOR, z[i]);
+	}
+	input_mt_sync_frame(tscreen);
+	input_sync(tscreen);
+
+	if (ts_virtobj_mapped_buttons(ts->map)) {
+		for (i = 0; i < ts->chip_info->max_fingers; i++)
+			if (ts_virtobj_is_button_slot(ts->map, i) &&
+			    !button_pressed[i])
+				ts_virtobj_button_release(ts->map, keypad, i);
+		input_sync(keypad);
 	}
-
-	input_mt_sync_frame(input);
-	input_sync(input);
 
 	return n_contacts;
 }
@@ -226,6 +247,7 @@ static int st1232_ts_probe(struct i2c_client *client)
 	const struct st_chip_info *match;
 	struct st1232_ts_data *ts;
 	struct input_dev *input_dev;
+	struct input_dev __maybe_unused *virtual_keypad;
 	u16 max_x, max_y;
 	int error;
 
@@ -292,18 +314,28 @@ static int st1232_ts_probe(struct i2c_client *client)
 	if (error)
 		return error;
 
-	/* Read resolution from the chip */
-	error = st1232_ts_read_resolution(ts, &max_x, &max_y);
-	if (error) {
-		dev_err(&client->dev,
-			"Failed to read resolution: %d\n", error);
-		return error;
-	}
-
 	if (ts->chip_info->have_z)
 		input_set_abs_params(input_dev, ABS_MT_TOUCH_MAJOR, 0,
 				     ts->chip_info->max_area, 0, 0);
 
+	/* map virtual objects if defined in the device tree */
+	ts->map = ts_virtobj_map_objects(&ts->client->dev, ts->input_dev);
+	if (IS_ERR(ts->map))
+		return PTR_ERR(ts->map);
+
+	if (ts_virtobj_mapped_touchscreen(ts->map)) {
+		/* Read resolution from the virtual touchscreen if defined*/
+		ts_virtobj_get_touchscreen_abs(ts->map, &max_x, &max_y);
+	} else {
+		/* Read resolution from the chip */
+		error = st1232_ts_read_resolution(ts, &max_x, &max_y);
+		if (error) {
+			dev_err(&client->dev,
+				"Failed to read resolution: %d\n", error);
+			return error;
+		}
+	}
+
 	input_set_abs_params(input_dev, ABS_MT_POSITION_X,
 			     0, max_x, 0, 0);
 	input_set_abs_params(input_dev, ABS_MT_POSITION_Y,
@@ -335,6 +367,25 @@ static int st1232_ts_probe(struct i2c_client *client)
 		return error;
 	}
 
+	/* Register keypad input device if virtual buttons were defined */
+	if (ts_virtobj_mapped_buttons(ts->map)) {
+		virtual_keypad = devm_input_allocate_device(&client->dev);
+		if (!virtual_keypad)
+			return -ENOMEM;
+
+		ts->virtual_keypad = virtual_keypad;
+		virtual_keypad->name = "st1232-keypad";
+		virtual_keypad->id.bustype = BUS_I2C;
+		ts_virtobj_set_button_caps(ts->map, virtual_keypad);
+		error = input_register_device(ts->virtual_keypad);
+		if (error) {
+			dev_err(&client->dev,
+				"Unable to register %s input device\n",
+				virtual_keypad->name);
+			return error;
+		}
+	}
+
 	i2c_set_clientdata(client, ts);
 
 	return 0;

-- 
2.39.2

[PATCH v2 4/4] dt-bindings: input: touchscreen: st1232: add example with ts-virtobj

[Javier Carrasco]

The st1232 driver supports the virtual-touchscreen and virtual-buttons
objects defined in the generic touchscreen bindings and implemented in
the ts-virtobj module. Add an example where nodes for a virtual
touchscreen and virtual buttons are defined.

Signed-off-by: Javier Carrasco <javier.carrasco@wolfvision.net>
---
 .../input/touchscreen/sitronix,st1232.yaml         | 40 ++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/Documentation/devicetree/bindings/input/touchscreen/sitronix,st1232.yaml b/Documentation/devicetree/bindings/input/touchscreen/sitronix,st1232.yaml
index 1d8ca19fd37a..97a2c063b47c 100644
--- a/Documentation/devicetree/bindings/input/touchscreen/sitronix,st1232.yaml
+++ b/Documentation/devicetree/bindings/input/touchscreen/sitronix,st1232.yaml
@@ -48,3 +48,43 @@ examples:
                     gpios = <&gpio1 166 0>;
             };
     };
+  - |
+    #include <dt-bindings/input/linux-event-codes.h>
+    i2c {
+            #address-cells = <1>;
+            #size-cells = <0>;
+
+            touchscreen@55 {
+                    compatible = "sitronix,st1232";
+                    reg = <0x55>;
+                    interrupts = <2 0>;
+                    gpios = <&gpio1 166 0>;
+
+                    virtual-touchscreen {
+                            x-origin = <0>;
+                            x-size = <240>;
+                            y-origin = <40>;
+                            y-size = <280>;
+                    };
+
+                    virtual-buttons {
+                            button-light {
+                                    label = "Camera light";
+                                    linux,code = <KEY_LIGHTS_TOGGLE>;
+                                    x-origin = <40>;
+                                    x-size = <40>;
+                                    y-origin = <0>;
+                                    y-size = <40>;
+                            };
+
+                            button-suspend {
+                                    label = "Suspend";
+                                    linux,code = <KEY_SUSPEND>;
+                                    x-origin = <160>;
+                                    x-size = <40>;
+                                    y-origin = <0>;
+                                    y-size = <40>;
+                            };
+                    };
+            };
+    };

-- 
2.39.2

Re: [PATCH v2 2/4] dt-bindings: touchscreen: add virtual-touchscreen and virtual-buttons properties

[Krzysztof Kozlowski]

On 15/05/2023 17:00, Javier Carrasco wrote:
> The virtual-touchscreen object defines an area within the touchscreen
> where touch events are reported and their coordinates get converted to
> the virtual origin. This object avoids getting events from areas that
> are physically hidden by overlay frames.
> 
> For touchscreens where overlay buttons on the touchscreen surface are
> provided, the virtual-buttons object contains a node for every button
> and the key event that should be reported when pressed.
> 
> Signed-off-by: Javier Carrasco <javier.carrasco@wolfvision.net>
> ---


Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>

Best regards,
Krzysztof

Re: [PATCH v2 2/4] dt-bindings: touchscreen: add virtual-touchscreen and virtual-buttons properties

[Krzysztof Kozlowski]

On 16/05/2023 10:10, Krzysztof Kozlowski wrote:
> On 15/05/2023 17:00, Javier Carrasco wrote:
>> The virtual-touchscreen object defines an area within the touchscreen
>> where touch events are reported and their coordinates get converted to
>> the virtual origin. This object avoids getting events from areas that
>> are physically hidden by overlay frames.
>>
>> For touchscreens where overlay buttons on the touchscreen surface are
>> provided, the virtual-buttons object contains a node for every button
>> and the key event that should be reported when pressed.
>>
>> Signed-off-by: Javier Carrasco <javier.carrasco@wolfvision.net>
>> ---
> 
> 
> Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>

Apologies, second thoughts - why calling all this binding and properties
"virtual"? That's the word which immediately raises questions, because
bindings are only for real things, not virtual.

Touchscreen is just clipped, not virtual, so maybe "clipped-area"
instead of virtual-touchscreen? Buttons are real, so maybe just "buttons"?

Best regards,
Krzysztof

Re: [PATCH v2 2/4] dt-bindings: touchscreen: add virtual-touchscreen and virtual-buttons properties

[Javier Carrasco]

On 16.05.23 10:13, Krzysztof Kozlowski wrote:
> On 16/05/2023 10:10, Krzysztof Kozlowski wrote:
>> On 15/05/2023 17:00, Javier Carrasco wrote:
>>> The virtual-touchscreen object defines an area within the touchscreen
>>> where touch events are reported and their coordinates get converted to
>>> the virtual origin. This object avoids getting events from areas that
>>> are physically hidden by overlay frames.
>>>
>>> For touchscreens where overlay buttons on the touchscreen surface are
>>> provided, the virtual-buttons object contains a node for every button
>>> and the key event that should be reported when pressed.
>>>
>>> Signed-off-by: Javier Carrasco <javier.carrasco@wolfvision.net>
>>> ---
>>
>>
>> Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
> 
> Apologies, second thoughts - why calling all this binding and properties
> "virtual"? That's the word which immediately raises questions, because
> bindings are only for real things, not virtual.
> 
> Touchscreen is just clipped, not virtual, so maybe "clipped-area"
> instead of virtual-touchscreen? Buttons are real, so maybe just "buttons"?
> 
> Best regards,
> Krzysztof
> 
I guess it is a matter of perspective. For a user the buttons and the
clipped area are 100% real, but for a driver developer they are virtual
in the sense that there is not an "active" hardware behind apart from
the original touchscreen.

I just wanted to avoid misunderstandings when implementing this feature
for other drivers. One might wonder if the touchscreen now has
mechanical keys attached to it. With the "virtual-" prefix it is clear
that the objects are not additional pieces of hardware or extensions of
the touchscreen functionality.

For the virtual-touchscreen your point is stronger because there is
indeed a real touchscreen hardware no matter the area you define, but my
approach was keeping homogeneous names for the different objects in case
some new ones might appear in the future: every object that gets on top
of the touchscreen area is virtual, so add a new object type and name it
virtual-xxx.

I have nothing against about doing some renaming and I will do it if it
is required, but with the documentation I think it is now more clear
what everything means and in the end it might make more sense for the
drivers so they can differentiate between real and virtual devices.

Re: [PATCH v2 2/4] dt-bindings: touchscreen: add virtual-touchscreen and virtual-buttons properties

[Krzysztof Kozlowski]

On 16/05/2023 11:03, Javier Carrasco wrote:
> On 16.05.23 10:13, Krzysztof Kozlowski wrote:
>> On 16/05/2023 10:10, Krzysztof Kozlowski wrote:
>>> On 15/05/2023 17:00, Javier Carrasco wrote:
>>>> The virtual-touchscreen object defines an area within the touchscreen
>>>> where touch events are reported and their coordinates get converted to
>>>> the virtual origin. This object avoids getting events from areas that
>>>> are physically hidden by overlay frames.
>>>>
>>>> For touchscreens where overlay buttons on the touchscreen surface are
>>>> provided, the virtual-buttons object contains a node for every button
>>>> and the key event that should be reported when pressed.
>>>>
>>>> Signed-off-by: Javier Carrasco <javier.carrasco@wolfvision.net>
>>>> ---
>>>
>>>
>>> Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
>>
>> Apologies, second thoughts - why calling all this binding and properties
>> "virtual"? That's the word which immediately raises questions, because
>> bindings are only for real things, not virtual.
>>
>> Touchscreen is just clipped, not virtual, so maybe "clipped-area"
>> instead of virtual-touchscreen? Buttons are real, so maybe just "buttons"?
>>
>> Best regards,
>> Krzysztof
>>
> I guess it is a matter of perspective. For a user the buttons and the
> clipped area are 100% real, but for a driver developer they are virtual
> in the sense that there is not an "active" hardware behind apart from
> the original touchscreen.


The feature describes the hardware, not driver. To understand what does
it mean, look from hardware point of view - does it have some virtual
area or clipped area?

> 
> I just wanted to avoid misunderstandings when implementing this feature
> for other drivers. One might wonder if the touchscreen now has
> mechanical keys attached to it. With the "virtual-" prefix it is clear
> that the objects are not additional pieces of hardware or extensions of
> the touchscreen functionality.

But what if actual physical buttons are added there? You still would
have clipped/virtual area, just without virtual buttons.

> 
> For the virtual-touchscreen your point is stronger because there is
> indeed a real touchscreen hardware no matter the area you define, but my
> approach was keeping homogeneous names for the different objects in case
> some new ones might appear in the future: every object that gets on top
> of the touchscreen area is virtual, so add a new object type and name it
> virtual-xxx.

To me, word "virtual" suggests something which does not exist. Kind of
something abstracted or symbolic. Opposite to "real". Here all this
really exists. You have physical stickers on the touchscreen.

Maybe this should be then "dedicated"? or "isolated"?

Or just "overlay-area"?

> 
> I have nothing against about doing some renaming and I will do it if it
> is required, but with the documentation I think it is now more clear
> what everything means and in the end it might make more sense for the
> drivers so they can differentiate between real and virtual devices.

Best regards,
Krzysztof

Re: [PATCH v2 2/4] dt-bindings: touchscreen: add virtual-touchscreen and virtual-buttons properties

[Javier Carrasco]

On 17.05.23 10:59, Krzysztof Kozlowski wrote:
> On 16/05/2023 11:03, Javier Carrasco wrote:
>> On 16.05.23 10:13, Krzysztof Kozlowski wrote:
>>> On 16/05/2023 10:10, Krzysztof Kozlowski wrote:
>>>> On 15/05/2023 17:00, Javier Carrasco wrote:
>>>>> The virtual-touchscreen object defines an area within the touchscreen
>>>>> where touch events are reported and their coordinates get converted to
>>>>> the virtual origin. This object avoids getting events from areas that
>>>>> are physically hidden by overlay frames.
>>>>>
>>>>> For touchscreens where overlay buttons on the touchscreen surface are
>>>>> provided, the virtual-buttons object contains a node for every button
>>>>> and the key event that should be reported when pressed.
>>>>>
>>>>> Signed-off-by: Javier Carrasco <javier.carrasco@wolfvision.net>
>>>>> ---
>>>>
>>>>
>>>> Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
>>>
>>> Apologies, second thoughts - why calling all this binding and properties
>>> "virtual"? That's the word which immediately raises questions, because
>>> bindings are only for real things, not virtual.
>>>
>>> Touchscreen is just clipped, not virtual, so maybe "clipped-area"
>>> instead of virtual-touchscreen? Buttons are real, so maybe just "buttons"?
>>>
>>> Best regards,
>>> Krzysztof
>>>
>> I guess it is a matter of perspective. For a user the buttons and the
>> clipped area are 100% real, but for a driver developer they are virtual
>> in the sense that there is not an "active" hardware behind apart from
>> the original touchscreen.
> 
> 
> The feature describes the hardware, not driver. To understand what does
> it mean, look from hardware point of view - does it have some virtual
> area or clipped area?
> 
>>
>> I just wanted to avoid misunderstandings when implementing this feature
>> for other drivers. One might wonder if the touchscreen now has
>> mechanical keys attached to it. With the "virtual-" prefix it is clear
>> that the objects are not additional pieces of hardware or extensions of
>> the touchscreen functionality.
> 
> But what if actual physical buttons are added there? You still would
> have clipped/virtual area, just without virtual buttons.
> 
>>
>> For the virtual-touchscreen your point is stronger because there is
>> indeed a real touchscreen hardware no matter the area you define, but my
>> approach was keeping homogeneous names for the different objects in case
>> some new ones might appear in the future: every object that gets on top
>> of the touchscreen area is virtual, so add a new object type and name it
>> virtual-xxx.
> 
> To me, word "virtual" suggests something which does not exist. Kind of
> something abstracted or symbolic. Opposite to "real". Here all this
> really exists. You have physical stickers on the touchscreen.
> 
> Maybe this should be then "dedicated"? or "isolated"?
> 
> Or just "overlay-area"?
> 
>>
>> I have nothing against about doing some renaming and I will do it if it
>> is required, but with the documentation I think it is now more clear
>> what everything means and in the end it might make more sense for the
>> drivers so they can differentiate between real and virtual devices.
> 
> Best regards,
> Krzysztof
> 
I think the word "overlay" is more precise and does not lead to any
misunderstanding, so I will rename everything to xxx-overlay and
overlay-xxx in the code and the documentation if that is ok.
virtual-buttons -> overlay-buttons
ts_virtobj_xxx()->ts_overlay_xxx()
etc.