linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls
@ 2020-08-04 19:29 Helen Koike
  2020-08-04 19:29 ` [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
                   ` (9 more replies)
  0 siblings, 10 replies; 51+ messages in thread
From: Helen Koike @ 2020-08-04 19:29 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Helen Koike, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	boris.brezillon, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Hello,

This is v5 of the Extended API for formats and buffers, which introduces
the following new ioctls:

int ioctl(int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
int ioctl(int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
int ioctl(int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)

int ioctl(int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp)
int ioctl(int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp)
int ioctl(int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp)
int ioctl(int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp)
int ioctl(int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp)

Please check v4 cover letter specific topic past discussions
https://patchwork.linuxtv.org/project/linux-media/cover/20200717115435.2632623-1-helen.koike@collabora.com/

Documentation
=============
I added a first version of the documentation.
I would appreciate some tips in how to organize this better, since there are
several parts of the docs which reference the "old" API, and for now
I just added a note pointing to the Extended API.

Instead of duplicating documentation from the code to the Docs (as used by
most part of v4l2 documentation), I just added a reference to let Sphinx get
the structs documentation from the code, so we can avoid duplicating.

For reviewing convenience, I uploaded the generated html docs at
https://people.collabora.com/~koike/ext-doc-v5/userspace-api/media/v4l/ext-api.html

There is room for improvements, it would be great to get your suggestions.

uAPI
====
This version have some minor changes in the uAPI structs, highlight to the
mem_offset that was returned to struct v4l2_ext_plane, memory field that now
is per plane, and some adjustments in field sizes and re-ordering to make
structs the same for 32 and 64 bits (which I verified with pahole tool).

I also updated v4l2-compliance:
https://gitlab.collabora.com/koike/v4l-utils/-/tree/ext-api/wip

We need to decide the size of the reserved fields left, how much reserved
fields do you think we should leave for each struct?

What is next
============
I would like to improve implementation (for the kernel and v4l2-compliane) and
drop the RFC tag for next version, so please review the uAPI.


List of changes in v5:
======================
* first version of Documentation
* migrate memory from v4l2_ext_buffer to v4l2_ext_plane
* return mem_offset to struct v4l2_ext_plane
* change sizes and reorder fields to avoid holes in the struct and make it
  the same for 32 and 64 bits
* removed __attribute__ ((packed)) from uapi structs
* set request_fd to signed
* add documentation
* updated some commit messages

Hans Verkuil (1):
  media: v4l2: Add extended buffer operations

Helen Koike (6):
  media: v4l2: Extend pixel formats to unify single/multi-planar
    handling (and more)
  media: videobuf2: Expose helpers to implement the _ext_fmt and
    _ext_buf hooks
  media: mediabus: Add helpers to convert a ext_pix format to/from a
    mbus_fmt
  media: vivid: Convert the capture and output drivers to
    EXT_FMT/EXT_BUF
  media: vimc: Implement the ext_fmt and ext_buf hooks
  media: docs: add documentation for the Extended API

 .../userspace-api/media/v4l/buffer.rst        |   5 +
 .../userspace-api/media/v4l/common.rst        |   1 +
 .../userspace-api/media/v4l/dev-capture.rst   |   5 +
 .../userspace-api/media/v4l/dev-output.rst    |   5 +
 .../userspace-api/media/v4l/ext-api.rst       | 107 ++
 .../userspace-api/media/v4l/format.rst        |  16 +-
 .../userspace-api/media/v4l/user-func.rst     |   5 +
 .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++
 .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++
 .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++
 .../media/v4l/vidioc-ext-querybuf.rst         |  79 ++
 .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 +++
 .../media/common/videobuf2/videobuf2-core.c   |   2 +
 .../media/common/videobuf2/videobuf2-v4l2.c   | 560 ++++++-----
 .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
 drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
 drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
 drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
 .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
 .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
 .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
 .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
 .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
 .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
 drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          | 938 ++++++++++++++++--
 include/media/v4l2-ioctl.h                    |  60 ++
 include/media/v4l2-mediabus.h                 |  42 +
 include/media/videobuf2-core.h                |   6 +-
 include/media/videobuf2-v4l2.h                |  21 +-
 include/uapi/linux/videodev2.h                | 146 +++
 31 files changed, 2363 insertions(+), 723 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst

-- 
2.28.0.rc2


^ permalink raw reply	[flat|nested] 51+ messages in thread

* [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
@ 2020-08-04 19:29 ` Helen Koike
  2020-08-14  7:49   ` Alexandre Courbot
                     ` (2 more replies)
  2020-08-04 19:29 ` [PATCH v5 2/7] media: v4l2: Add extended buffer operations Helen Koike
                   ` (8 subsequent siblings)
  9 siblings, 3 replies; 51+ messages in thread
From: Helen Koike @ 2020-08-04 19:29 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Helen Koike, Boris Brezillon, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

This is part of the multiplanar and singleplanar unification process.
v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose
HW-specific formats (like tiled or compressed formats) and allow
exchanging this information with the DRM subsystem in a consistent way.

Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
v4l2_ext_format, other types will be rejected if you use the
{G,S,TRY}_EXT_PIX_FMT ioctls.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls
in drivers, but, in the meantime, the core takes care of converting
{S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace
apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
_ext_ hooks.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v5:
- change sizes and reorder fields to avoid holes in the struct and make
  it the same for 32 and 64 bits
- removed __attribute__ ((packed)) from uapi structs
- Fix doc warning from make htmldocs
- Updated commit message with EXT_PIX prefix for the ioctls.

Changes in v4:
- Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
- Add reserved fields
- Removed num_planes from struct v4l2_ext_pix_format
- Removed flag field from struct v4l2_ext_pix_format, since the only
  defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
  where we can use modifiers, or add it back later through the reserved
  bits.
- In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
  != MOD_INVALID
- Fix type assignment in v4l_g_fmt_ext_pix()
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Move the modifier in v4l2_ext_format (was formerly placed in
  v4l2_ext_plane)
- Fix a few bugs in the converters and add a strict parameter to
  allow conversion of uninitialized/mis-initialized objects
---
 drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
 drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
 include/media/v4l2-ioctl.h           |  34 ++
 include/uapi/linux/videodev2.h       |  56 +++
 4 files changed, 615 insertions(+), 81 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index a593ea0598b55..e1829906bc086 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
 			set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
 		if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
 			       ops->vidioc_g_fmt_vid_cap_mplane ||
+			       ops->vidioc_g_ext_pix_fmt_vid_cap ||
 			       ops->vidioc_g_fmt_vid_overlay)) ||
 		    (is_tx && (ops->vidioc_g_fmt_vid_out ||
 			       ops->vidioc_g_fmt_vid_out_mplane ||
-			       ops->vidioc_g_fmt_vid_out_overlay)))
+			       ops->vidioc_g_ext_pix_fmt_vid_out ||
+			       ops->vidioc_g_fmt_vid_out_overlay))) {
 			 set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
+			 set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
+		}
 		if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
 			       ops->vidioc_s_fmt_vid_cap_mplane ||
+			       ops->vidioc_s_ext_pix_fmt_vid_cap ||
 			       ops->vidioc_s_fmt_vid_overlay)) ||
 		    (is_tx && (ops->vidioc_s_fmt_vid_out ||
 			       ops->vidioc_s_fmt_vid_out_mplane ||
-			       ops->vidioc_s_fmt_vid_out_overlay)))
+			       ops->vidioc_s_ext_pix_fmt_vid_out ||
+			       ops->vidioc_s_fmt_vid_out_overlay))) {
 			 set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
+			 set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
+		}
 		if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
 			       ops->vidioc_try_fmt_vid_cap_mplane ||
+			       ops->vidioc_try_ext_pix_fmt_vid_cap ||
 			       ops->vidioc_try_fmt_vid_overlay)) ||
 		    (is_tx && (ops->vidioc_try_fmt_vid_out ||
 			       ops->vidioc_try_fmt_vid_out_mplane ||
-			       ops->vidioc_try_fmt_vid_out_overlay)))
+			       ops->vidioc_try_ext_pix_fmt_vid_out ||
+			       ops->vidioc_try_fmt_vid_out_overlay))) {
 			 set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
+			 set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
+		}
 		SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
 		SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
 		SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
@@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
 		/* touch specific ioctls */
 		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
+		SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
+		SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
+		SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
 		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
 		SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index a556880f225a5..14a0def50f8ea 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -17,6 +17,8 @@
 
 #include <linux/videodev2.h>
 
+#include <drm/drm_fourcc.h>
+
 #include <media/v4l2-common.h>
 #include <media/v4l2-ioctl.h>
 #include <media/v4l2-ctrls.h>
@@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
 	}
 }
 
+static void v4l_print_ext_pix_format(const void *arg, bool write_only)
+{
+	const struct v4l2_ext_pix_format *pix = arg;
+	unsigned int i;
+
+	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
+		prt_names(pix->type, v4l2_type_names),
+		pix->width, pix->height,
+		(pix->pixelformat & 0xff),
+		(pix->pixelformat >>  8) & 0xff,
+		(pix->pixelformat >> 16) & 0xff,
+		(pix->pixelformat >> 24) & 0xff,
+		pix->modifier, prt_names(pix->field, v4l2_field_names),
+		pix->colorspace, pix->ycbcr_enc,
+		pix->quantization, pix->xfer_func);
+	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
+		pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
+			 i, pix->plane_fmt[i].bytesperline,
+			 pix->plane_fmt[i].sizeimage);
+}
+
 static void v4l_print_framebuffer(const void *arg, bool write_only)
 {
 	const struct v4l2_framebuffer *p = arg;
@@ -964,11 +987,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
 	switch (type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
 		if ((is_vid || is_tch) && is_rx &&
-		    (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
+		    (ops->vidioc_g_fmt_vid_cap ||
+		     ops->vidioc_g_ext_pix_fmt_vid_cap ||
+		     ops->vidioc_g_fmt_vid_cap_mplane))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
+		if ((is_vid || is_tch) && is_rx &&
+		    (ops->vidioc_g_fmt_vid_cap_mplane ||
+		     ops->vidioc_g_ext_pix_fmt_vid_cap))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
@@ -977,11 +1004,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
 		if (is_vid && is_tx &&
-		    (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
+		    (ops->vidioc_g_fmt_vid_out ||
+		     ops->vidioc_g_ext_pix_fmt_vid_out ||
+		     ops->vidioc_g_fmt_vid_out_mplane))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
+		if (is_vid && is_tx &&
+		    (ops->vidioc_g_ext_pix_fmt_vid_out ||
+		     ops->vidioc_g_fmt_vid_out_mplane))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
@@ -1061,6 +1092,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
 	       sizeof(fmt->fmt.pix) - offset);
 }
 
+int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
+				  struct v4l2_format *f, bool mplane_cap,
+				  bool strict)
+{
+	const struct v4l2_plane_ext_pix_format *pe;
+	struct v4l2_plane_pix_format *p;
+	unsigned int i;
+
+	memset(f, 0, sizeof(*f));
+
+	/*
+	 * Make sure no modifier is required before doing the
+	 * conversion.
+	 */
+	if (e->modifier && strict &&
+	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
+	    e->modifier != DRM_FORMAT_MOD_INVALID)
+		return -EINVAL;
+
+	if (!e->plane_fmt[0].sizeimage && strict)
+		return -EINVAL;
+
+	if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
+		return 0;
+
+	if (!mplane_cap) {
+		f->fmt.pix.width = e->width;
+		f->fmt.pix.height = e->height;
+		f->fmt.pix.pixelformat = e->pixelformat;
+		f->fmt.pix.field = e->field;
+		f->fmt.pix.colorspace = e->colorspace;
+		f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
+		f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+		f->fmt.pix.quantization = e->quantization;
+		pe = &e->plane_fmt[0];
+		f->fmt.pix.bytesperline = pe->bytesperline;
+		f->fmt.pix.sizeimage = pe->sizeimage;
+		f->type = e->type;
+		return 0;
+	}
+
+	f->fmt.pix_mp.width = e->width;
+	f->fmt.pix_mp.height = e->height;
+	f->fmt.pix_mp.pixelformat = e->pixelformat;
+	f->fmt.pix_mp.field = e->field;
+	f->fmt.pix_mp.colorspace = e->colorspace;
+	f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
+	f->fmt.pix_mp.quantization = e->quantization;
+	if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+	else
+		f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+
+	for (i = 0; i < VIDEO_MAX_PLANES; i++) {
+		pe = &e->plane_fmt[i];
+		p = &f->fmt.pix_mp.plane_fmt[i];
+		p->bytesperline = pe->bytesperline;
+		p->sizeimage = pe->sizeimage;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
+
+int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
+				  struct v4l2_ext_pix_format *e, bool strict)
+{
+	const struct v4l2_plane_pix_format *p;
+	struct v4l2_plane_ext_pix_format *pe;
+	unsigned int i;
+
+	memset(e, 0, sizeof(*e));
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		e->width = f->fmt.pix.width;
+		e->height = f->fmt.pix.height;
+		e->pixelformat = f->fmt.pix.pixelformat;
+		e->field = f->fmt.pix.field;
+		e->colorspace = f->fmt.pix.colorspace;
+		if (f->fmt.pix.flags)
+			pr_warn("Ignoring pixelformat flags 0x%x\n",
+				f->fmt.pix.flags);
+		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
+		e->quantization = f->fmt.pix.quantization;
+		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
+		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
+		e->type = f->type;
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
+		     !f->fmt.pix_mp.num_planes) && strict)
+			return -EINVAL;
+
+		e->width = f->fmt.pix_mp.width;
+		e->height = f->fmt.pix_mp.height;
+		e->pixelformat = f->fmt.pix_mp.pixelformat;
+		e->field = f->fmt.pix_mp.field;
+		e->colorspace = f->fmt.pix_mp.colorspace;
+		if (f->fmt.pix.flags)
+			pr_warn("Ignoring pixelformat flags 0x%x\n",
+				f->fmt.pix.flags);
+		e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
+		e->quantization = f->fmt.pix_mp.quantization;
+		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+		else
+			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
+			pe = &e->plane_fmt[i];
+			p = &f->fmt.pix_mp.plane_fmt[i];
+			pe->bytesperline = p->bytesperline;
+			pe->sizeimage = p->sizeimage;
+		}
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
+
 static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1564,6 +1723,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
 	p->xfer_func = 0;
 }
 
+static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh,
+			     struct v4l2_format *f)
+{
+	struct v4l2_ext_pix_format ef;
+	int ret;
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+		ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+		ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+		ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	if (ret)
+		return ret;
+
+	return v4l2_ext_pix_format_to_format(&ef, f,
+					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
+					     true);
+}
+
 static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1600,17 +1791,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 
 	switch (p->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-		if (unlikely(!ops->vidioc_g_fmt_vid_cap))
-			break;
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		if (vfd->vfl_type == VFL_TYPE_TOUCH)
-			v4l_pix_format_touch(&p->fmt.pix);
-		return ret;
+		if (ops->vidioc_g_fmt_vid_cap) {
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		} else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
+			ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
+		if (ops->vidioc_g_fmt_vid_cap_mplane)
+			return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
+		else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
+			return v4l_g_fmt_ext_pix(ops, file, fh, p);
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
 		return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
 	case V4L2_BUF_TYPE_VBI_CAPTURE:
@@ -1618,15 +1819,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 	case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
 		return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		if (unlikely(!ops->vidioc_g_fmt_vid_out))
-			break;
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		return ret;
+		if (ops->vidioc_g_fmt_vid_out) {
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			return ret;
+		} else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
+			return v4l_g_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
+		if (ops->vidioc_g_fmt_vid_out_mplane)
+			return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
+		else if (ops->vidioc_g_ext_pix_fmt_vid_out)
+			return v4l_g_fmt_ext_pix(ops, file, fh, p);
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
 		return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
 	case V4L2_BUF_TYPE_VBI_OUTPUT:
@@ -1645,6 +1853,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 	return -EINVAL;
 }
 
+static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh, void *arg)
+{
+	struct v4l2_ext_pix_format *ef = arg;
+	struct v4l2_format f = {
+		.type = ef->type,
+	};
+	int ret;
+
+	ret = check_fmt(file, ef->type);
+	if (ret)
+		return ret;
+
+	memset(ef, 0, sizeof(*ef));
+	ef->type = f.type;
+
+	switch (f.type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+		if (ops->vidioc_g_ext_pix_fmt_vid_cap)
+			return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		if (ops->vidioc_g_ext_pix_fmt_vid_out)
+			return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	ret = v4l_g_fmt(ops, file, fh, &f);
+	if (ret)
+		return ret;
+
+	return v4l2_format_to_ext_pix_format(&f, ef, true);
+}
+
+static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh,
+			     struct v4l2_format *f)
+{
+	struct v4l2_ext_pix_format ef;
+	int ret;
+
+	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
+	if (ret)
+		return ret;
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+		ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	if (ret)
+		return ret;
+
+	return v4l2_ext_pix_format_to_format(&ef, f,
+					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
+					     true);
+}
+
 static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1663,23 +1941,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 
 	switch (p->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-		if (unlikely(!ops->vidioc_s_fmt_vid_cap))
+		if (ops->vidioc_s_fmt_vid_cap) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
+			ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		} else {
 			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+		}
+
 		if (vfd->vfl_type == VFL_TYPE_TOUCH)
 			v4l_pix_format_touch(&p->fmt.pix);
 		return ret;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
+		if (ops->vidioc_s_fmt_vid_cap_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
+			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
 		if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
 			break;
@@ -1696,21 +1982,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
 		return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		if (unlikely(!ops->vidioc_s_fmt_vid_out))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		return ret;
+		if (ops->vidioc_s_fmt_vid_out) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			return ret;
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
+			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
+		if (ops->vidioc_s_fmt_vid_out_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
+			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
 		if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
 			break;
@@ -1750,6 +2042,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 	return -EINVAL;
 }
 
+static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh, void *arg)
+{
+	struct video_device *vfd = video_devdata(file);
+	struct v4l2_ext_pix_format *ef = arg;
+	struct v4l2_format f;
+	int ret;
+
+	ret = check_fmt(file, ef->type);
+	if (ret)
+		return ret;
+
+	switch (ef->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+		if (ops->vidioc_s_ext_pix_fmt_vid_cap)
+			return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		if (ops->vidioc_s_ext_pix_fmt_vid_out)
+			return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	ret = v4l2_ext_pix_format_to_format(ef, &f,
+					    vfd->device_caps &
+					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+					     V4L2_CAP_VIDEO_M2M_MPLANE),
+					    false);
+	if (ret)
+		return ret;
+
+	ret = v4l_s_fmt(ops, file, fh, &f);
+	if (ret)
+		return ret;
+
+	return v4l2_format_to_ext_pix_format(&f, ef, true);
+}
+
+static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct v4l2_ext_pix_format ef;
+	int ret;
+
+	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
+	if (ret)
+		return ret;
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+		ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	if (ret)
+		return ret;
+
+	return v4l2_ext_pix_format_to_format(&ef, f,
+					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
+					     true);
+}
+
+
 static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1765,23 +2133,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 
 	switch (p->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-		if (unlikely(!ops->vidioc_try_fmt_vid_cap))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		if (vfd->vfl_type == VFL_TYPE_TOUCH)
-			v4l_pix_format_touch(&p->fmt.pix);
-		return ret;
+		if (ops->vidioc_try_fmt_vid_cap) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
+		if (ops->vidioc_try_fmt_vid_cap_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			return v4l_try_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
 		if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
 			break;
@@ -1798,21 +2175,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
 		return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		if (unlikely(!ops->vidioc_try_fmt_vid_out))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		return ret;
+		if (ops->vidioc_try_fmt_vid_out) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			return ret;
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			return v4l_try_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
+		if (ops->vidioc_try_fmt_vid_out_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			return v4l_try_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
 		if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
 			break;
@@ -1852,6 +2235,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 	return -EINVAL;
 }
 
+static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh, void *arg)
+{
+	struct video_device *vfd = video_devdata(file);
+	struct v4l2_ext_pix_format *ef = arg;
+	struct v4l2_format f;
+	int ret;
+
+	ret = check_fmt(file, ef->type);
+	if (ret)
+		return ret;
+
+	switch (ef->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+		if (ops->vidioc_try_ext_pix_fmt_vid_cap)
+			return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
+								   ef);
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		if (ops->vidioc_try_ext_pix_fmt_vid_out)
+			return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
+								   ef);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	ret = v4l2_ext_pix_format_to_format(ef, &f,
+					    vfd->device_caps &
+					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+					     V4L2_CAP_VIDEO_M2M_MPLANE),
+					    false);
+	if (ret)
+		return ret;
+
+	ret = v4l_try_fmt(ops, file, fh, &f);
+	if (ret)
+		return ret;
+
+	return v4l2_format_to_ext_pix_format(&f, ef, true);
+}
+
 static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -2771,7 +3197,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
 	IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
 	IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
+	IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
 	IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
+	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
 	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
@@ -2818,6 +3246,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
 	IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
+	IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
 	IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
 	IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
 	IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
index 86878fba332b0..8bbcb74d8ee31 100644
--- a/include/media/v4l2-ioctl.h
+++ b/include/media/v4l2-ioctl.h
@@ -48,11 +48,17 @@ struct v4l2_fh;
  * @vidioc_g_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
+ *	capture
  * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_g_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
+ *	out
  * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
  * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
@@ -82,11 +88,16 @@ struct v4l2_fh;
  * @vidioc_s_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
+ *	capture
  * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_s_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
  * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
  * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
@@ -116,11 +127,16 @@ struct v4l2_fh;
  * @vidioc_try_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for
+	video capture
  * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_try_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
  * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  *	output
@@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
 	/* VIDIOC_G_FMT handlers */
 	int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
 					struct v4l2_format *f);
 	int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
 					    struct v4l2_format *f);
 	int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
@@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
 	/* VIDIOC_S_FMT handlers */
 	int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
 					struct v4l2_format *f);
 	int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
 					    struct v4l2_format *f);
 	int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
@@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
 	/* VIDIOC_TRY_FMT handlers */
 	int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
 				      struct v4l2_format *f);
+	int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
+					      struct v4l2_ext_pix_format *f);
 	int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
 					  struct v4l2_format *f);
 	int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
 				      struct v4l2_format *f);
+	int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
+					      struct v4l2_ext_pix_format *f);
 	int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
 					     struct v4l2_format *f);
 	int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
@@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
 long int video_ioctl2(struct file *file,
 		      unsigned int cmd, unsigned long int arg);
 
+int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
+				  struct v4l2_ext_pix_format *e, bool strict);
+int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
+				  struct v4l2_format *f,
+				  bool mplane_cap, bool strict);
+
 /*
  * The user space interpretation of the 'v4l2_event' differs
  * based on the 'time_t' definition on 32-bit architectures, so
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index c7b70ff53bc1d..7123c6a4d9569 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
 	__u8				reserved[7];
 } __attribute__ ((packed));
 
+/**
+ * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
+ * @sizeimage:		maximum size in bytes required for data, for which
+ *			this plane will be used.
+ *			Should be set to zero for unused planes.
+ * @bytesperline:	distance in bytes between the leftmost pixels in two
+ *			adjacent lines
+ * @reserved:		extra space reserved for future fields, must be set to 0
+ */
+struct v4l2_plane_ext_pix_format {
+	__u32 sizeimage;
+	__u32 bytesperline;
+	__u32 reserved;
+};
+
+/**
+ * struct v4l2_ext_pix_format - extended single/multiplanar format definition
+ * @type:		type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
+ *			V4L2_BUF_TYPE_VIDEO_OUTPUT
+ * @width:		image width in pixels
+ * @height:		image height in pixels
+ * @field:		enum v4l2_field; field order (for interlaced video)
+ * @pixelformat:	little endian four character code (fourcc)
+ * @modifier:		modifier applied to the format (used for tiled formats
+ *			and other kind of HW-specific formats, like compressed
+ *			formats)
+ * @colorspace:		enum v4l2_colorspace; supplemental to pixelformat
+ * @plane_fmt:		per-plane information
+ * @ycbcr_enc:		enum v4l2_ycbcr_encoding, Y'CbCr encoding
+ * @hsv_enc:		enum v4l2_hsv_encoding, HSV encoding
+ * @quantization:	enum v4l2_quantization, colorspace quantization
+ * @xfer_func:		enum v4l2_xfer_func, colorspace transfer function
+ * @reserved:		extra space reserved for future fields, must be set to 0
+ */
+struct v4l2_ext_pix_format {
+	__u32 type;
+	__u32 width;
+	__u32 height;
+	__u32 field;
+	__u64 modifier;
+	__u32 pixelformat;
+	__u32 colorspace;
+	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
+	union {
+		__u32 ycbcr_enc;
+		__u32 hsv_enc;
+	};
+	__u32 quantization;
+	__u32 xfer_func;
+	__u32 reserved[9];
+};
+
 /**
  * struct v4l2_sdr_format - SDR format definition
  * @pixelformat:	little endian four character code (fourcc)
@@ -2571,6 +2623,10 @@ struct v4l2_create_buffers {
 
 #define VIDIOC_QUERY_EXT_CTRL	_IOWR('V', 103, struct v4l2_query_ext_ctrl)
 
+#define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
+#define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
+#define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
+
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
 
-- 
2.28.0.rc2


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
  2020-08-04 19:29 ` [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
@ 2020-08-04 19:29 ` Helen Koike
  2020-08-14  7:49   ` Alexandre Courbot
                     ` (2 more replies)
  2020-08-04 19:29 ` [PATCH v5 3/7] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks Helen Koike
                   ` (7 subsequent siblings)
  9 siblings, 3 replies; 51+ messages in thread
From: Helen Koike @ 2020-08-04 19:29 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Boris Brezillon, Helen Koike, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

From: Hans Verkuil <hans.verkuil@cisco.com>

Those extended buffer ops have several purpose:
1/ Fix y2038 issues by converting the timestamp into an u64 counting
   the number of ns elapsed since 1970
2/ Unify single/multiplanar handling
3/ Add a new start offset field to each v4l2 plane buffer info struct
   to support the case where a single buffer object is storing all
   planes data, each one being placed at a different offset

New hooks are created in v4l2_ioctl_ops so that drivers can start using
these new objects.

The core takes care of converting new ioctls requests to old ones
if the driver does not support the new hooks, and vice versa.

Note that the timecode field is gone, since there doesn't seem to be
in-kernel users. We can be added back in the reserved area if needed or
use the Request API to collect more metadata information from the
frame.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v5:
- migrate memory from v4l2_ext_buffer to v4l2_ext_plane
- return mem_offset to struct v4l2_ext_plane
- change sizes and reorder fields to avoid holes in the struct and make
  it the same for 32 and 64 bits

Changes in v4:
- Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
- Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
I think we can add this later, so I removed it from this RFC to simplify it.
- Remove num_planes field from struct v4l2_ext_buffer
- Add flags field to struct v4l2_ext_create_buffers
- Reformulate struct v4l2_ext_plane
- Fix some bugs caught by v4l2-compliance
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Add reserved space to v4l2_ext_buffer so that new fields can be added
  later on
---
 drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
 drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
 include/media/v4l2-ioctl.h           |  26 ++
 include/uapi/linux/videodev2.h       |  90 +++++++
 4 files changed, 476 insertions(+), 22 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index e1829906bc086..cb21ee8eb075c 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -720,15 +720,34 @@ static void determine_valid_ioctls(struct video_device *vdev)
 		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_sdr_out);
 	}
 
+	if (is_vid || is_tch) {
+		/* ioctls valid for video and touch */
+		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
+			set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
+		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
+			set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
+		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
+			set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
+		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
+			set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
+		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
+			set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
+	}
+
 	if (is_vid || is_vbi || is_sdr || is_tch || is_meta) {
 		/* ioctls valid for video, vbi, sdr, touch and metadata */
 		SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
-		SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
-		SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
 		SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
-		SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
-		SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
-		SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
+		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
+			set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
+		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
+			set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
+		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
+			set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
+		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
+			set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
+		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
+			set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
 		SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
 		SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
 	}
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 14a0def50f8ea..7ecdd9cc1bf48 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -527,6 +527,26 @@ static void v4l_print_buffer(const void *arg, bool write_only)
 			tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
 }
 
+static void v4l_print_ext_buffer(const void *arg, bool write_only)
+{
+	const struct v4l2_ext_buffer *e = arg;
+	const struct v4l2_ext_plane *plane;
+	unsigned int i;
+
+	pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d\n",
+		e->timestamp, e->index, prt_names(e->type, v4l2_type_names),
+		e->flags, prt_names(e->field, v4l2_field_names), e->sequence);
+
+	for (i = 0; i < VIDEO_MAX_PLANES &&
+		    e->planes[i].buffer_length; i++) {
+		plane = &e->planes[i];
+		pr_debug("plane %d: buffer_length=%d, plane_length=%d offset=0x%08x, memory=%s\n",
+			 i, plane->buffer_length, plane->plane_length,
+			 plane->offset,
+			 prt_names(plane->memory, v4l2_memory_names));
+	}
+}
+
 static void v4l_print_exportbuffer(const void *arg, bool write_only)
 {
 	const struct v4l2_exportbuffer *p = arg;
@@ -546,6 +566,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
 	v4l_print_format(&p->format, write_only);
 }
 
+static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
+{
+	const struct v4l2_ext_create_buffers *p = arg;
+
+	pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
+		prt_names(p->memory, v4l2_memory_names));
+	v4l_print_ext_pix_format(&p->format, write_only);
+}
+
 static void v4l_print_streamparm(const void *arg, bool write_only)
 {
 	const struct v4l2_streamparm *p = arg;
@@ -1220,6 +1249,143 @@ int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
 }
 EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
 
+/*
+ * If mplane_cap is true, b->m.planes should have a valid pointer of a
+ * struct v4l2_plane array, and b->length with its size
+ */
+int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
+			      struct v4l2_buffer *b, bool mplane_cap)
+{
+	unsigned int planes_array_size = b->length;
+	struct v4l2_plane *planes = b->m.planes;
+	u64 nsecs;
+
+	if (!mplane_cap && e->planes[1].buffer_length != 0)
+		return -EINVAL;
+
+	memset(b, 0, sizeof(*b));
+
+	b->index = e->index;
+	b->flags = e->flags;
+	b->field = e->field;
+	b->sequence = e->sequence;
+	b->memory = e->planes[0].memory;
+	b->request_fd = e->request_fd;
+	b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
+	b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
+
+	if (mplane_cap) {
+		unsigned int i;
+
+		if (!planes || !planes_array_size)
+			return -EINVAL;
+
+		b->m.planes = planes;
+
+		if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
+			b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+		else
+			b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+
+		for (i = 0; i < VIDEO_MAX_PLANES && i < planes_array_size &&
+			    e->planes[i].buffer_length; i++) {
+
+			if (e->planes[0].memory != e->planes[i].memory)
+				return -EINVAL;
+
+			if (e->planes[i].offset)
+				return -EINVAL;
+
+			memset(&b->m.planes[i], 0, sizeof(b->m.planes[i]));
+
+			if (b->memory == V4L2_MEMORY_MMAP)
+				b->m.planes[i].m.mem_offset = e->planes[i].m.mem_offset;
+			else if (b->memory == V4L2_MEMORY_DMABUF)
+				b->m.planes[i].m.fd = e->planes[i].m.dmabuf_fd;
+			else
+				b->m.planes[i].m.userptr = e->planes[i].m.userptr;
+
+			b->m.planes[i].bytesused = e->planes[i].plane_length;
+			b->m.planes[i].length = e->planes[i].buffer_length;
+		}
+		/* In multi-planar, length contain the number of planes */
+		b->length = i;
+	} else {
+		b->type = e->type;
+		b->bytesused = e->planes[0].plane_length;
+		b->length = e->planes[0].buffer_length;
+
+		if (e->planes[0].offset)
+			return -EINVAL;
+
+		if (b->memory == V4L2_MEMORY_MMAP)
+			b->m.offset = e->planes[0].m.mem_offset;
+		else if (b->memory == V4L2_MEMORY_DMABUF)
+			b->m.fd = e->planes[0].m.dmabuf_fd;
+		else
+			b->m.userptr = e->planes[0].m.userptr;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
+
+int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
+			      struct v4l2_ext_buffer *e)
+{
+	memset(e, 0, sizeof(*e));
+
+	e->index = b->index;
+	e->flags = b->flags;
+	e->field = b->field;
+	e->sequence = b->sequence;
+	e->request_fd = b->request_fd;
+	e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
+		b->timestamp.tv_usec * NSEC_PER_USEC;
+	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
+		unsigned int i;
+
+		if (!b->m.planes)
+			return -EINVAL;
+
+		if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+		else
+			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+		/* In multi-planar, length contain the number of planes */
+		for (i = 0; i < b->length; i++) {
+			if (b->memory == V4L2_MEMORY_MMAP)
+				e->planes[i].m.mem_offset = b->m.planes[i].m.mem_offset;
+			else if (b->memory == V4L2_MEMORY_DMABUF)
+				e->planes[i].m.dmabuf_fd = b->m.planes[i].m.fd;
+			else
+				e->planes[i].m.userptr = b->m.planes[i].m.userptr;
+
+			e->planes[i].memory = b->memory;
+			e->planes[i].buffer_length = b->m.planes[i].length;
+			e->planes[i].plane_length = b->m.planes[i].bytesused;
+			if (b->m.planes[i].data_offset)
+				pr_warn("Ignoring data_offset value %d\n",
+					b->m.planes[i].data_offset);
+		}
+	} else {
+		e->type = b->type;
+		e->planes[0].memory = b->memory;
+		e->planes[0].plane_length = b->bytesused;
+		e->planes[0].buffer_length = b->length;
+		if (b->memory == V4L2_MEMORY_MMAP)
+			e->planes[0].m.mem_offset = b->m.offset;
+		else if (b->memory == V4L2_MEMORY_DMABUF)
+			e->planes[0].m.dmabuf_fd = b->m.fd;
+		else
+			e->planes[0].m.userptr = b->m.userptr;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
+
 static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -2473,31 +2639,112 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
 	return ops->vidioc_reqbufs(file, fh, p);
 }
 
+static int v4l_do_buf_op(int (*op)(struct file *, void *,
+				   struct v4l2_buffer *),
+			 int (*ext_op)(struct file *, void *,
+				       struct v4l2_ext_buffer *),
+			 struct file *file, void *fh, struct v4l2_buffer *b)
+{
+	struct v4l2_ext_buffer e;
+	int ret;
+
+	ret = check_fmt(file, b->type);
+	if (ret)
+		return ret;
+
+	if (op)
+		return op(file, fh, b);
+
+	ret = v4l2_buffer_to_ext_buffer(b, &e);
+	if (ret)
+		return ret;
+
+	ret = ext_op(file, fh, &e);
+	if (ret)
+		return ret;
+
+	v4l2_ext_buffer_to_buffer(&e, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
+	return 0;
+}
+
+static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
+				       struct v4l2_buffer *),
+			     int (*ext_op)(struct file *, void *,
+					   struct v4l2_ext_buffer *),
+			     struct file *file, void *fh,
+			     struct v4l2_ext_buffer *e)
+{
+	struct video_device *vdev = video_devdata(file);
+	struct v4l2_plane planes[VIDEO_MAX_PLANES];
+	struct v4l2_buffer b;
+	bool mplane_cap;
+	int ret;
+
+	ret = check_fmt(file, e->type);
+	if (ret)
+		return ret;
+
+	if (ext_op)
+		return ext_op(file, fh, e);
+
+	mplane_cap = !!(vdev->device_caps &
+			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+			 V4L2_CAP_VIDEO_M2M_MPLANE));
+	b.m.planes = planes;
+	b.length = VIDEO_MAX_PLANES;
+	ret = v4l2_ext_buffer_to_buffer(e, &b, mplane_cap);
+	if (ret)
+		return ret;
+
+	ret = op(file, fh, &b);
+	if (ret)
+		return ret;
+
+	v4l2_buffer_to_ext_buffer(&b, e);
+	return 0;
+}
+
 static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+			struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *p = arg;
-	int ret = check_fmt(file, p->type);
+	return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
+			     file, fh, arg);
+}
 
-	return ret ? ret : ops->vidioc_querybuf(file, fh, p);
+static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
+			    struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_querybuf,
+				 ops->vidioc_ext_querybuf, file, fh, arg);
 }
 
 static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+		    struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *p = arg;
-	int ret = check_fmt(file, p->type);
+	return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
+			     file, fh, arg);
+}
 
-	return ret ? ret : ops->vidioc_qbuf(file, fh, p);
+static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
+			struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
+				 file, fh, arg);
 }
 
 static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+		     struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *p = arg;
-	int ret = check_fmt(file, p->type);
+	return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
+			     file, fh, arg);
+}
 
-	return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
+static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
+			 struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
+				 file, fh, arg);
 }
 
 static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
@@ -2513,7 +2760,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
 
 	v4l_sanitize_format(&create->format);
 
-	ret = ops->vidioc_create_bufs(file, fh, create);
+	if (ops->vidioc_create_bufs) {
+		ret = ops->vidioc_create_bufs(file, fh, create);
+	} else {
+		struct v4l2_ext_create_buffers ecreate = {
+			.count = create->count,
+			.memory = create->memory,
+		};
+
+		ret = v4l2_format_to_ext_pix_format(&create->format,
+						    &ecreate.format, true);
+		if (ret)
+			return ret;
+
+		ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
+		if (ret)
+			return ret;
+
+		create->index = ecreate.index;
+		create->count = ecreate.count;
+		create->capabilities = ecreate.capabilities;
+	}
 
 	if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
 	    create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
@@ -2522,13 +2789,60 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
 	return ret;
 }
 
+static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh, void *arg)
+{
+	struct v4l2_ext_create_buffers *ecreate = arg;
+	struct video_device *vdev = video_devdata(file);
+	struct v4l2_create_buffers create = {
+		.count = ecreate->count,
+		.memory = ecreate->memory,
+		.flags = ecreate->flags,
+	};
+	bool mplane_cap;
+	int ret;
+
+	ret = check_fmt(file, ecreate->format.type);
+	if (ret)
+		return ret;
+
+	if (ops->vidioc_ext_create_bufs)
+		return ops->vidioc_ext_create_bufs(file, fh, ecreate);
+
+	mplane_cap = !!(vdev->device_caps &
+			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+			 V4L2_CAP_VIDEO_M2M_MPLANE));
+	ret = v4l2_ext_pix_format_to_format(&ecreate->format,
+					    &create.format, mplane_cap, true);
+	if (ret)
+		return ret;
+
+	ret = v4l_create_bufs(ops, file, fh, &create);
+	if (ret)
+		return ret;
+
+	ecreate->index = create.index;
+	ecreate->count = create.count;
+	ecreate->capabilities = create.capabilities;
+
+	return 0;
+}
+
 static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
+			   struct file *file, void *fh, void *arg)
 {
-	struct v4l2_buffer *b = arg;
-	int ret = check_fmt(file, b->type);
+	return v4l_do_buf_op(ops->vidioc_prepare_buf,
+			     ops->vidioc_ext_prepare_buf,
+			     file, fh, arg);
+}
 
-	return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
+static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh, void *arg)
+{
+	return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
+				 ops->vidioc_ext_prepare_buf,
+				 file, fh, arg);
 }
 
 static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
@@ -3202,12 +3516,15 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
+	IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, planes)),
 	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
 	IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
+	IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
+	IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
@@ -3272,7 +3589,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
 	IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
 	IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
+	IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
+	IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
 	IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
 	IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
index 8bbcb74d8ee31..75996657ad1ba 100644
--- a/include/media/v4l2-ioctl.h
+++ b/include/media/v4l2-ioctl.h
@@ -169,16 +169,26 @@ struct v4l2_fh;
  *	:ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
  * @vidioc_querybuf: pointer to the function that implements
  *	:ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
+ * @vidioc_ext_querybuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
  * @vidioc_qbuf: pointer to the function that implements
  *	:ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
+ * @vidioc_ext_qbuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
  * @vidioc_expbuf: pointer to the function that implements
  *	:ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
  * @vidioc_dqbuf: pointer to the function that implements
  *	:ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
+ * @vidioc_ext_dqbuf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
  * @vidioc_create_bufs: pointer to the function that implements
  *	:ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
+ * @vidioc_ext_create_bufs: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
  * @vidioc_prepare_buf: pointer to the function that implements
  *	:ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
+ * @vidioc_ext_prepare_buf: pointer to the function that implements
+ *	:ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
  * @vidioc_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
  * @vidioc_g_fbuf: pointer to the function that implements
@@ -439,17 +449,27 @@ struct v4l2_ioctl_ops {
 			      struct v4l2_requestbuffers *b);
 	int (*vidioc_querybuf)(struct file *file, void *fh,
 			       struct v4l2_buffer *b);
+	int (*vidioc_ext_querybuf)(struct file *file, void *fh,
+				   struct v4l2_ext_buffer *b);
 	int (*vidioc_qbuf)(struct file *file, void *fh,
 			   struct v4l2_buffer *b);
+	int (*vidioc_ext_qbuf)(struct file *file, void *fh,
+			       struct v4l2_ext_buffer *b);
 	int (*vidioc_expbuf)(struct file *file, void *fh,
 			     struct v4l2_exportbuffer *e);
 	int (*vidioc_dqbuf)(struct file *file, void *fh,
 			    struct v4l2_buffer *b);
+	int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
+				struct v4l2_ext_buffer *b);
 
 	int (*vidioc_create_bufs)(struct file *file, void *fh,
 				  struct v4l2_create_buffers *b);
+	int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
+				      struct v4l2_ext_create_buffers *b);
 	int (*vidioc_prepare_buf)(struct file *file, void *fh,
 				  struct v4l2_buffer *b);
+	int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
+				      struct v4l2_ext_buffer *b);
 
 	int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
 	int (*vidioc_g_fbuf)(struct file *file, void *fh,
@@ -758,6 +778,12 @@ int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
 				  struct v4l2_format *f,
 				  bool mplane_cap, bool strict);
 
+int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
+			      struct v4l2_buffer *b,
+			      bool mplane_cap);
+int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
+			      struct v4l2_ext_buffer *e);
+
 /*
  * The user space interpretation of the 'v4l2_event' differs
  * based on the 'time_t' definition on 32-bit architectures, so
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 7123c6a4d9569..334cafdd2be97 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -996,6 +996,41 @@ struct v4l2_plane {
 	__u32			reserved[11];
 };
 
+/**
+ * struct v4l2_ext_plane - extended plane buffer info
+ * @buffer_length:	size of the entire buffer in bytes, should fit
+ *			@offset + @plane_length
+ * @plane_length:	size of the plane in bytes.
+ * @mem_offset:		If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
+ *			that should be passed to mmap() called on the video node.
+ * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
+ *			to this plane.
+ * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
+ *			associated with this plane.
+ * @offset:		offset in the memory buffer where the plane starts.
+ * @memory:		enum v4l2_memory; the method, in which the actual video
+ *			data is passed
+ * @reserved:		extra space reserved for future fields, must be set to 0.
+ *
+ *
+ * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
+ * can have one plane for Y, and another for interleaved CbCr components.
+ * Each plane can reside in a separate memory buffer, or even in
+ * a completely separate memory node (e.g. in embedded devices).
+ */
+struct v4l2_ext_plane {
+	__u32 buffer_length;
+	__u32 plane_length;
+	union {
+		__u32 mem_offset;
+		__u64 userptr;
+		__s32 dmabuf_fd;
+	} m;
+	__u32 offset;
+	__u32 memory;
+	__u32 reserved[4];
+};
+
 /**
  * struct v4l2_buffer - video buffer info
  * @index:	id number of the buffer
@@ -1057,6 +1092,33 @@ struct v4l2_buffer {
 	};
 };
 
+/**
+ * struct v4l2_ext_buffer - extended video buffer info
+ * @index:	id number of the buffer
+ * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
+ * @flags:	buffer informational flags
+ * @field:	enum v4l2_field; field order of the image in the buffer
+ * @timestamp:	frame timestamp
+ * @sequence:	sequence count of this frame
+ * @planes:	per-plane buffer information
+ * @request_fd:	fd of the request that this buffer should use
+ * @reserved:	extra space reserved for future fields, must be set to 0
+ *
+ * Contains data exchanged by application and driver using one of the Streaming
+ * I/O methods.
+ */
+struct v4l2_ext_buffer {
+	__u32 index;
+	__u32 type;
+	__u32 field;
+	__u32 sequence;
+	__u64 flags;
+	__u64 timestamp;
+	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
+	__s32 request_fd;
+	__u32 reserved[9];
+};
+
 #ifndef __KERNEL__
 /**
  * v4l2_timeval_to_ns - Convert timeval to nanoseconds
@@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
 	__u32			reserved[6];
 };
 
+/**
+ * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
+ * @index:	on return, index of the first created buffer
+ * @count:	entry: number of requested buffers,
+ *		return: number of created buffers
+ * @memory:	enum v4l2_memory; buffer memory type
+ * @capabilities: capabilities of this buffer type.
+ * @format:	frame format, for which buffers are requested
+ * @flags:	additional buffer management attributes (ignored unless the
+ *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
+ *		and configured for MMAP streaming I/O).
+ * @reserved:	extra space reserved for future fields, must be set to 0
+ */
+struct v4l2_ext_create_buffers {
+	__u32				index;
+	__u32				count;
+	__u32				memory;
+	__u32				capabilities;
+	struct v4l2_ext_pix_format	format;
+	__u32				flags;
+	__u32 reserved[5];
+};
+
 /*
  *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
  *
@@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
 #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
 #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
 #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
+#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
+#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
+#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
 
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
-- 
2.28.0.rc2


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v5 3/7] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks
  2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
  2020-08-04 19:29 ` [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
  2020-08-04 19:29 ` [PATCH v5 2/7] media: v4l2: Add extended buffer operations Helen Koike
@ 2020-08-04 19:29 ` Helen Koike
  2020-12-14  8:52   ` Tomasz Figa
  2020-08-04 19:29 ` [PATCH v5 4/7] media: mediabus: Add helpers to convert a ext_pix format to/from a mbus_fmt Helen Koike
                   ` (6 subsequent siblings)
  9 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-08-04 19:29 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Helen Koike, Boris Brezillon, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

The VB2 layer is used by a lot of drivers. Patch it to support the
_EXT_PIX_FMT and _EXT_BUF ioctls in order to ease conversion of existing
drivers to these new APIs.

Note that internally, the VB2 core is now only using ext structs and old
APIs are supported through conversion wrappers.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v5:
- Update with new format and buffer structs
- Updated commit message with the uAPI prefix

Changes in v4:
- Update with new format and buffer structs
- Fix some bugs caught by v4l2-compliance
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch
---
 .../media/common/videobuf2/videobuf2-core.c   |   2 +
 .../media/common/videobuf2/videobuf2-v4l2.c   | 560 ++++++++++--------
 include/media/videobuf2-core.h                |   6 +-
 include/media/videobuf2-v4l2.h                |  21 +-
 4 files changed, 345 insertions(+), 244 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index f544d3393e9d6..d719b1e9c148b 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -1270,6 +1270,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb)
 		vb->planes[plane].length = 0;
 		vb->planes[plane].m.fd = 0;
 		vb->planes[plane].data_offset = 0;
+		vb->planes[plane].dbuf_offset = 0;
 
 		/* Acquire each plane's memory */
 		mem_priv = call_ptr_memop(vb, attach_dmabuf,
@@ -1313,6 +1314,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb)
 		vb->planes[plane].length = planes[plane].length;
 		vb->planes[plane].m.fd = planes[plane].m.fd;
 		vb->planes[plane].data_offset = planes[plane].data_offset;
+		vb->planes[plane].dbuf_offset = planes[plane].dbuf_offset;
 	}
 
 	if (reacquired) {
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 30caad27281e1..911681d24b3ae 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -29,6 +29,7 @@
 #include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
 #include <media/v4l2-common.h>
+#include <media/v4l2-ioctl.h>
 
 #include <media/videobuf2-v4l2.h>
 
@@ -56,72 +57,39 @@ module_param(debug, int, 0644);
 				 V4L2_BUF_FLAG_TIMECODE | \
 				 V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF)
 
-/*
- * __verify_planes_array() - verify that the planes array passed in struct
- * v4l2_buffer from userspace can be safely used
- */
-static int __verify_planes_array(struct vb2_buffer *vb, const struct v4l2_buffer *b)
-{
-	if (!V4L2_TYPE_IS_MULTIPLANAR(b->type))
-		return 0;
-
-	/* Is memory for copying plane information present? */
-	if (b->m.planes == NULL) {
-		dprintk(vb->vb2_queue, 1,
-			"multi-planar buffer passed but planes array not provided\n");
-		return -EINVAL;
-	}
-
-	if (b->length < vb->num_planes || b->length > VB2_MAX_PLANES) {
-		dprintk(vb->vb2_queue, 1,
-			"incorrect planes array length, expected %d, got %d\n",
-			vb->num_planes, b->length);
-		return -EINVAL;
-	}
-
-	return 0;
-}
-
 static int __verify_planes_array_core(struct vb2_buffer *vb, const void *pb)
 {
-	return __verify_planes_array(vb, pb);
+	return 0;
 }
 
 /*
  * __verify_length() - Verify that the bytesused value for each plane fits in
  * the plane length and that the data offset doesn't exceed the bytesused value.
  */
-static int __verify_length(struct vb2_buffer *vb, const struct v4l2_buffer *b)
+static int __verify_length(struct vb2_buffer *vb,
+			   const struct v4l2_ext_buffer *b)
 {
 	unsigned int length;
 	unsigned int bytesused;
-	unsigned int plane;
+	unsigned int i;
 
 	if (V4L2_TYPE_IS_CAPTURE(b->type))
 		return 0;
 
-	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
-		for (plane = 0; plane < vb->num_planes; ++plane) {
-			length = (b->memory == VB2_MEMORY_USERPTR ||
-				  b->memory == VB2_MEMORY_DMABUF)
-			       ? b->m.planes[plane].length
-				: vb->planes[plane].length;
-			bytesused = b->m.planes[plane].bytesused
-				  ? b->m.planes[plane].bytesused : length;
-
-			if (b->m.planes[plane].bytesused > length)
-				return -EINVAL;
-
-			if (b->m.planes[plane].data_offset > 0 &&
-			    b->m.planes[plane].data_offset >= bytesused)
-				return -EINVAL;
-		}
-	} else {
-		length = (b->memory == VB2_MEMORY_USERPTR ||
-			  b->memory == VB2_MEMORY_DMABUF)
-			? b->length : vb->planes[0].length;
+	for (i = 0; i < vb->num_planes; ++i) {
+		const struct v4l2_ext_plane *plane = &b->planes[i];
 
-		if (b->bytesused > length)
+		length = (plane->memory == VB2_MEMORY_USERPTR ||
+			  plane->memory == VB2_MEMORY_DMABUF) ?
+			 plane->buffer_length :
+			 vb->planes[i].length;
+		bytesused = plane->plane_length ?
+			    plane->plane_length : length;
+
+		if (bytesused > length)
+			return -EINVAL;
+
+		if (plane->offset + bytesused > length)
 			return -EINVAL;
 	}
 
@@ -140,21 +108,12 @@ static void __init_vb2_v4l2_buffer(struct vb2_buffer *vb)
 
 static void __copy_timestamp(struct vb2_buffer *vb, const void *pb)
 {
-	const struct v4l2_buffer *b = pb;
-	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	const struct v4l2_ext_buffer *b = pb;
 	struct vb2_queue *q = vb->vb2_queue;
 
-	if (q->is_output) {
-		/*
-		 * For output buffers copy the timestamp if needed,
-		 * and the timecode field and flag if needed.
-		 */
-		if (q->copy_timestamp)
-			vb->timestamp = v4l2_buffer_get_timestamp(b);
-		vbuf->flags |= b->flags & V4L2_BUF_FLAG_TIMECODE;
-		if (b->flags & V4L2_BUF_FLAG_TIMECODE)
-			vbuf->timecode = b->timecode;
-	}
+	/* For output buffers copy the timestamp if needed. */
+	if (q->is_output && q->copy_timestamp)
+		vb->timestamp = b->timestamp;
 };
 
 static void vb2_warn_zero_bytesused(struct vb2_buffer *vb)
@@ -173,7 +132,8 @@ static void vb2_warn_zero_bytesused(struct vb2_buffer *vb)
 		pr_warn("use the actual size instead.\n");
 }
 
-static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b)
+static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb,
+				    struct v4l2_ext_buffer *b)
 {
 	struct vb2_queue *q = vb->vb2_queue;
 	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
@@ -203,110 +163,67 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
 	vbuf->request_fd = -1;
 	vbuf->is_held = false;
 
-	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
-		switch (b->memory) {
-		case VB2_MEMORY_USERPTR:
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				planes[plane].m.userptr =
-					b->m.planes[plane].m.userptr;
-				planes[plane].length =
-					b->m.planes[plane].length;
-			}
-			break;
-		case VB2_MEMORY_DMABUF:
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				planes[plane].m.fd =
-					b->m.planes[plane].m.fd;
-				planes[plane].length =
-					b->m.planes[plane].length;
-			}
-			break;
-		default:
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				planes[plane].m.offset =
-					vb->planes[plane].m.offset;
-				planes[plane].length =
-					vb->planes[plane].length;
-			}
-			break;
+	for (plane = 0; plane < vb->num_planes; ++plane) {
+		if (b->planes[0].memory != b->planes[plane].memory) {
+			dprintk(q, 1, "planes should have the same memory type\n");
+			return -EINVAL;
 		}
+	}
 
-		/* Fill in driver-provided information for OUTPUT types */
-		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
-			/*
-			 * Will have to go up to b->length when API starts
-			 * accepting variable number of planes.
-			 *
-			 * If bytesused == 0 for the output buffer, then fall
-			 * back to the full buffer size. In that case
-			 * userspace clearly never bothered to set it and
-			 * it's a safe assumption that they really meant to
-			 * use the full plane sizes.
-			 *
-			 * Some drivers, e.g. old codec drivers, use bytesused == 0
-			 * as a way to indicate that streaming is finished.
-			 * In that case, the driver should use the
-			 * allow_zero_bytesused flag to keep old userspace
-			 * applications working.
-			 */
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				struct vb2_plane *pdst = &planes[plane];
-				struct v4l2_plane *psrc = &b->m.planes[plane];
-
-				if (psrc->bytesused == 0)
-					vb2_warn_zero_bytesused(vb);
-
-				if (vb->vb2_queue->allow_zero_bytesused)
-					pdst->bytesused = psrc->bytesused;
-				else
-					pdst->bytesused = psrc->bytesused ?
-						psrc->bytesused : pdst->length;
-				pdst->data_offset = psrc->data_offset;
-			}
+	switch (b->planes[0].memory) {
+	case VB2_MEMORY_USERPTR:
+		for (plane = 0; plane < vb->num_planes; ++plane) {
+			planes[plane].m.userptr = b->planes[plane].m.userptr;
+			planes[plane].length = b->planes[plane].buffer_length;
 		}
-	} else {
+		break;
+	case VB2_MEMORY_DMABUF:
+		for (plane = 0; plane < vb->num_planes; ++plane) {
+			planes[plane].m.fd = b->planes[plane].m.dmabuf_fd;
+			planes[plane].dbuf_offset = b->planes[plane].offset;
+			planes[plane].length = b->planes[plane].buffer_length;
+		}
+		break;
+	default:
+		for (plane = 0; plane < vb->num_planes; ++plane) {
+			planes[plane].m.offset = vb->planes[plane].m.offset;
+			planes[plane].length = vb->planes[plane].length;
+		}
+		break;
+	}
+
+	/* Fill in driver-provided information for OUTPUT types */
+	if (V4L2_TYPE_IS_OUTPUT(b->type)) {
 		/*
-		 * Single-planar buffers do not use planes array,
-		 * so fill in relevant v4l2_buffer struct fields instead.
-		 * In videobuf we use our internal V4l2_planes struct for
-		 * single-planar buffers as well, for simplicity.
+		 * Will have to go up to b->length when API starts
+		 * accepting variable number of planes.
 		 *
-		 * If bytesused == 0 for the output buffer, then fall back
-		 * to the full buffer size as that's a sensible default.
+		 * If bytesused == 0 for the output buffer, then fall
+		 * back to the full buffer size. In that case
+		 * userspace clearly never bothered to set it and
+		 * it's a safe assumption that they really meant to
+		 * use the full plane sizes.
 		 *
-		 * Some drivers, e.g. old codec drivers, use bytesused == 0 as
-		 * a way to indicate that streaming is finished. In that case,
-		 * the driver should use the allow_zero_bytesused flag to keep
-		 * old userspace applications working.
+		 * Some drivers, e.g. old codec drivers, use bytesused == 0
+		 * as a way to indicate that streaming is finished.
+		 * In that case, the driver should use the
+		 * allow_zero_bytesused flag to keep old userspace
+		 * applications working.
 		 */
-		switch (b->memory) {
-		case VB2_MEMORY_USERPTR:
-			planes[0].m.userptr = b->m.userptr;
-			planes[0].length = b->length;
-			break;
-		case VB2_MEMORY_DMABUF:
-			planes[0].m.fd = b->m.fd;
-			planes[0].length = b->length;
-			break;
-		default:
-			planes[0].m.offset = vb->planes[0].m.offset;
-			planes[0].length = vb->planes[0].length;
-			break;
-		}
+		for (plane = 0; plane < vb->num_planes; ++plane) {
+			struct vb2_plane *pdst = &planes[plane];
+			struct v4l2_ext_plane *psrc = &b->planes[plane];
 
-		planes[0].data_offset = 0;
-		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
-			if (b->bytesused == 0)
+			if (psrc->plane_length == 0)
 				vb2_warn_zero_bytesused(vb);
 
 			if (vb->vb2_queue->allow_zero_bytesused)
-				planes[0].bytesused = b->bytesused;
+				pdst->bytesused = psrc->plane_length;
 			else
-				planes[0].bytesused = b->bytesused ?
-					b->bytesused : planes[0].length;
-		} else
-			planes[0].bytesused = 0;
-
+				pdst->bytesused = psrc->plane_length ?
+						  psrc->plane_length :
+						  pdst->length;
+		}
 	}
 
 	/* Zero flags that we handle */
@@ -343,7 +260,7 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
 
 static void set_buffer_cache_hints(struct vb2_queue *q,
 				   struct vb2_buffer *vb,
-				   struct v4l2_buffer *b)
+				   struct v4l2_ext_buffer *b)
 {
 	/*
 	 * DMA exporter should take care of cache syncs, so we can avoid
@@ -388,17 +305,44 @@ static void set_buffer_cache_hints(struct vb2_queue *q,
 		vb->need_cache_sync_on_prepare = 0;
 }
 
+static enum v4l2_buf_type vb2_ext_qtype(struct vb2_queue *q)
+{
+	if (!q->is_multiplanar)
+		return q->type;
+
+	if (q->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+		return V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	else if (q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
+		return V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+	return q->type;
+}
+
+static enum v4l2_buf_type vb2_ext_type(unsigned int type, bool is_multiplanar)
+{
+	if (!is_multiplanar)
+		return type;
+
+	if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+	else if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
+		return V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+
+	return type;
+}
+
 static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
-				    struct v4l2_buffer *b, bool is_prepare,
+				    struct v4l2_ext_buffer *b, bool is_prepare,
 				    struct media_request **p_req)
 {
 	const char *opname = is_prepare ? "prepare_buf" : "qbuf";
 	struct media_request *req;
 	struct vb2_v4l2_buffer *vbuf;
 	struct vb2_buffer *vb;
+	unsigned int i;
 	int ret;
 
-	if (b->type != q->type) {
+	if (b->type != vb2_ext_qtype(q)) {
 		dprintk(q, 1, "%s: invalid buffer type\n", opname);
 		return -EINVAL;
 	}
@@ -414,16 +358,14 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
 		return -EINVAL;
 	}
 
-	if (b->memory != q->memory) {
+	for (i = 0; i < VIDEO_MAX_PLANES && b->planes[i].buffer_length; i++)
+		if (b->planes[i].memory != q->memory) {
 		dprintk(q, 1, "%s: invalid memory type\n", opname);
 		return -EINVAL;
 	}
 
 	vb = q->bufs[b->index];
 	vbuf = to_vb2_v4l2_buffer(vb);
-	ret = __verify_planes_array(vb, b);
-	if (ret)
-		return ret;
 
 	if (!is_prepare && (b->flags & V4L2_BUF_FLAG_REQUEST_FD) &&
 	    vb->state != VB2_BUF_STATE_DEQUEUED) {
@@ -487,11 +429,6 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
 		    !q->ops->buf_out_validate))
 		return -EINVAL;
 
-	if (b->request_fd < 0) {
-		dprintk(q, 1, "%s: request_fd < 0\n", opname);
-		return -EINVAL;
-	}
-
 	req = media_request_get_by_fd(mdev, b->request_fd);
 	if (IS_ERR(req)) {
 		dprintk(q, 1, "%s: invalid request_fd\n", opname);
@@ -516,64 +453,46 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
 }
 
 /*
- * __fill_v4l2_buffer() - fill in a struct v4l2_buffer with information to be
- * returned to userspace
+ * __fill_v4l2_buffer() - fill in a struct v4l2_ext_buffer with information to
+ * be returned to userspace
  */
 static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 {
-	struct v4l2_buffer *b = pb;
+	struct v4l2_ext_buffer *b = pb;
 	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
 	struct vb2_queue *q = vb->vb2_queue;
 	unsigned int plane;
 
 	/* Copy back data such as timestamp, flags, etc. */
 	b->index = vb->index;
-	b->type = vb->type;
-	b->memory = vb->memory;
-	b->bytesused = 0;
+	b->type = vb2_ext_qtype(q);
 
 	b->flags = vbuf->flags;
 	b->field = vbuf->field;
-	v4l2_buffer_set_timestamp(b, vb->timestamp);
-	b->timecode = vbuf->timecode;
+	b->timestamp = vb->timestamp;
 	b->sequence = vbuf->sequence;
-	b->reserved2 = 0;
 	b->request_fd = 0;
+	memset(b->reserved, 0, sizeof(b->reserved));
 
-	if (q->is_multiplanar) {
-		/*
-		 * Fill in plane-related data if userspace provided an array
-		 * for it. The caller has already verified memory and size.
-		 */
-		b->length = vb->num_planes;
-		for (plane = 0; plane < vb->num_planes; ++plane) {
-			struct v4l2_plane *pdst = &b->m.planes[plane];
-			struct vb2_plane *psrc = &vb->planes[plane];
-
-			pdst->bytesused = psrc->bytesused;
-			pdst->length = psrc->length;
-			if (q->memory == VB2_MEMORY_MMAP)
-				pdst->m.mem_offset = psrc->m.offset;
-			else if (q->memory == VB2_MEMORY_USERPTR)
-				pdst->m.userptr = psrc->m.userptr;
-			else if (q->memory == VB2_MEMORY_DMABUF)
-				pdst->m.fd = psrc->m.fd;
-			pdst->data_offset = psrc->data_offset;
-			memset(pdst->reserved, 0, sizeof(pdst->reserved));
-		}
-	} else {
-		/*
-		 * We use length and offset in v4l2_planes array even for
-		 * single-planar buffers, but userspace does not.
-		 */
-		b->length = vb->planes[0].length;
-		b->bytesused = vb->planes[0].bytesused;
+	/*
+	 * Fill in plane-related data if userspace provided an array
+	 * for it. The caller has already verified memory and size.
+	 */
+	for (plane = 0; plane < vb->num_planes; ++plane) {
+		struct v4l2_ext_plane *pdst = &b->planes[plane];
+		struct vb2_plane *psrc = &vb->planes[plane];
+
+		pdst->memory = vb->memory;
+		pdst->plane_length = psrc->bytesused;
+		pdst->buffer_length = psrc->length;
+		pdst->offset = psrc->dbuf_offset;
 		if (q->memory == VB2_MEMORY_MMAP)
-			b->m.offset = vb->planes[0].m.offset;
+			pdst->m.mem_offset = psrc->m.offset;
 		else if (q->memory == VB2_MEMORY_USERPTR)
-			b->m.userptr = vb->planes[0].m.userptr;
+			pdst->m.userptr = psrc->m.userptr;
 		else if (q->memory == VB2_MEMORY_DMABUF)
-			b->m.fd = vb->planes[0].m.fd;
+			pdst->m.dmabuf_fd = psrc->m.fd;
+		memset(pdst->reserved, 0, sizeof(pdst->reserved));
 	}
 
 	/*
@@ -668,6 +587,35 @@ int vb2_find_timestamp(const struct vb2_queue *q, u64 timestamp,
 }
 EXPORT_SYMBOL_GPL(vb2_find_timestamp);
 
+#define vb2_buf_to_ext_buf_op(_name, ...)				\
+({									\
+	int ret;							\
+									\
+	ret = v4l2_buffer_to_ext_buffer(b, &eb);			\
+	if (!ret)							\
+		ret = _name(__VA_ARGS__);				\
+	if (!ret)							\
+		ret = v4l2_ext_buffer_to_buffer(&eb, b,			\
+						q->is_multiplanar);	\
+	ret;								\
+})
+
+int vb2_ext_querybuf(struct vb2_queue *q, struct v4l2_ext_buffer *b)
+{
+	if (b->type != vb2_ext_qtype(q)) {
+		dprintk(q, 1, "wrong buffer type\n");
+		return -EINVAL;
+	}
+
+	if (b->index >= q->num_buffers) {
+		dprintk(q, 1, "buffer index out of range\n");
+		return -EINVAL;
+	}
+	vb2_core_querybuf(q, b->index, b);
+	return 0;
+}
+EXPORT_SYMBOL(vb2_ext_querybuf);
+
 /*
  * vb2_querybuf() - query video buffer information
  * @q:		videobuf queue
@@ -683,23 +631,9 @@ EXPORT_SYMBOL_GPL(vb2_find_timestamp);
  */
 int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b)
 {
-	struct vb2_buffer *vb;
-	int ret;
-
-	if (b->type != q->type) {
-		dprintk(q, 1, "wrong buffer type\n");
-		return -EINVAL;
-	}
+	struct v4l2_ext_buffer eb;
 
-	if (b->index >= q->num_buffers) {
-		dprintk(q, 1, "buffer index out of range\n");
-		return -EINVAL;
-	}
-	vb = q->bufs[b->index];
-	ret = __verify_planes_array(vb, b);
-	if (!ret)
-		vb2_core_querybuf(q, b->index, b);
-	return ret;
+	return vb2_buf_to_ext_buf_op(vb2_ext_querybuf, q, &eb);
 }
 EXPORT_SYMBOL(vb2_querybuf);
 
@@ -741,8 +675,8 @@ int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req)
 }
 EXPORT_SYMBOL_GPL(vb2_reqbufs);
 
-int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
-		    struct v4l2_buffer *b)
+int vb2_ext_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
+			struct v4l2_ext_buffer *b)
 {
 	int ret;
 
@@ -758,16 +692,59 @@ int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
 
 	return ret ? ret : vb2_core_prepare_buf(q, b->index, b);
 }
+EXPORT_SYMBOL_GPL(vb2_ext_prepare_buf);
+
+int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
+		    struct v4l2_buffer *b)
+{
+	struct v4l2_ext_buffer eb;
+
+	return vb2_buf_to_ext_buf_op(vb2_ext_prepare_buf, q, mdev, &eb);
+}
 EXPORT_SYMBOL_GPL(vb2_prepare_buf);
 
+int vb2_ext_create_bufs(struct vb2_queue *q,
+			struct v4l2_ext_create_buffers *create)
+{
+	unsigned int requested_sizes[VIDEO_MAX_PLANES];
+	struct v4l2_ext_pix_format *f = &create->format;
+	int ret = vb2_verify_memory_type(q, create->memory,
+				vb2_ext_type(f->type, q->is_multiplanar));
+	unsigned i;
+
+	if (create->format.type != V4L2_BUF_TYPE_VIDEO_CAPTURE &&
+	    create->format.type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
+		return -EINVAL;
+
+	fill_buf_caps(q, &create->capabilities);
+	clear_consistency_attr(q, create->memory, &create->flags);
+	create->index = q->num_buffers;
+	if (create->count == 0)
+		return ret != -EBUSY ? ret : 0;
+
+	if (!f->plane_fmt[0].sizeimage)
+		return -EINVAL;
+	for (i = 0; i < VIDEO_MAX_PLANES &&
+		    f->plane_fmt[i].sizeimage; i++)
+		requested_sizes[i] = f->plane_fmt[i].sizeimage;
+
+	return ret ? ret : vb2_core_create_bufs(q, create->memory,
+						create->flags,
+						&create->count,
+						i,
+						requested_sizes);
+}
+EXPORT_SYMBOL_GPL(vb2_ext_create_bufs);
+
 int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
 {
+	struct v4l2_format *f = &create->format;
 	unsigned requested_planes = 1;
 	unsigned requested_sizes[VIDEO_MAX_PLANES];
-	struct v4l2_format *f = &create->format;
 	int ret = vb2_verify_memory_type(q, create->memory, f->type);
 	unsigned i;
 
+
 	fill_buf_caps(q, &create->capabilities);
 	clear_consistency_attr(q, create->memory, &create->flags);
 	create->index = q->num_buffers;
@@ -777,18 +754,29 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
 	switch (f->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		requested_planes = f->fmt.pix_mp.num_planes;
-		if (requested_planes == 0 ||
-		    requested_planes > VIDEO_MAX_PLANES)
-			return -EINVAL;
-		for (i = 0; i < requested_planes; i++)
-			requested_sizes[i] =
-				f->fmt.pix_mp.plane_fmt[i].sizeimage;
-		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		requested_sizes[0] = f->fmt.pix.sizeimage;
-		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT: {
+		struct v4l2_ext_create_buffers ecreate = {
+			.count = create->count,
+			.memory = create->memory,
+			.flags = create->flags,
+		};
+
+		ret = v4l2_format_to_ext_pix_format(&create->format,
+						    &ecreate.format, true);
+		if (ret)
+			return ret;
+
+		ret = vb2_ext_create_bufs(q, &ecreate);
+		if (ret)
+			return ret;
+
+		create->index = ecreate.index;
+		create->count = ecreate.count;
+		create->flags = ecreate.flags;
+		create->capabilities = ecreate.capabilities;
+		return 0;
+	}
 	case V4L2_BUF_TYPE_VBI_CAPTURE:
 	case V4L2_BUF_TYPE_VBI_OUTPUT:
 		requested_sizes[0] = f->fmt.vbi.samples_per_line *
@@ -820,8 +808,8 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
 }
 EXPORT_SYMBOL_GPL(vb2_create_bufs);
 
-int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
-	     struct v4l2_buffer *b)
+int vb2_ext_qbuf(struct vb2_queue *q, struct media_device *mdev,
+		 struct v4l2_ext_buffer *b)
 {
 	struct media_request *req = NULL;
 	int ret;
@@ -839,9 +827,19 @@ int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
 		media_request_put(req);
 	return ret;
 }
+EXPORT_SYMBOL_GPL(vb2_ext_qbuf);
+
+int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
+	     struct v4l2_buffer *b)
+{
+	struct v4l2_ext_buffer eb;
+
+	return vb2_buf_to_ext_buf_op(vb2_ext_qbuf, q, mdev, &eb);
+}
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
-int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
+int vb2_ext_dqbuf(struct vb2_queue *q, struct v4l2_ext_buffer *b,
+		  bool nonblocking)
 {
 	int ret;
 
@@ -850,7 +848,7 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
 		return -EBUSY;
 	}
 
-	if (b->type != q->type) {
+	if (b->type != vb2_ext_qtype(q)) {
 		dprintk(q, 1, "invalid buffer type\n");
 		return -EINVAL;
 	}
@@ -870,6 +868,14 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(vb2_ext_dqbuf);
+
+int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
+{
+	struct v4l2_ext_buffer eb;
+
+	return vb2_buf_to_ext_buf_op(vb2_ext_dqbuf, q, &eb, nonblocking);
+}
 EXPORT_SYMBOL_GPL(vb2_dqbuf);
 
 int vb2_streamon(struct vb2_queue *q, enum v4l2_buf_type type)
@@ -1040,6 +1046,33 @@ int vb2_ioctl_create_bufs(struct file *file, void *priv,
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_create_bufs);
 
+int vb2_ioctl_ext_create_bufs(struct file *file, void *priv,
+			      struct v4l2_ext_create_buffers *p)
+{
+	struct video_device *vdev = video_devdata(file);
+	int res = vb2_verify_memory_type(vdev->queue, p->memory,
+		vb2_ext_type(p->format.type, vdev->queue->is_multiplanar));
+
+	p->index = vdev->queue->num_buffers;
+	fill_buf_caps(vdev->queue, &p->capabilities);
+	/*
+	 * If count == 0, then just check if memory and type are valid.
+	 * Any -EBUSY result from vb2_verify_memory_type can be mapped to 0.
+	 */
+	if (p->count == 0)
+		return res != -EBUSY ? res : 0;
+	if (res)
+		return res;
+	if (vb2_queue_is_busy(vdev, file))
+		return -EBUSY;
+
+	res = vb2_ext_create_bufs(vdev->queue, p);
+	if (res == 0)
+		vdev->queue->owner = file->private_data;
+	return res;
+}
+EXPORT_SYMBOL_GPL(vb2_ioctl_ext_create_bufs);
+
 int vb2_ioctl_prepare_buf(struct file *file, void *priv,
 			  struct v4l2_buffer *p)
 {
@@ -1051,6 +1084,17 @@ int vb2_ioctl_prepare_buf(struct file *file, void *priv,
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_prepare_buf);
 
+int vb2_ioctl_ext_prepare_buf(struct file *file, void *priv,
+			      struct v4l2_ext_buffer *p)
+{
+	struct video_device *vdev = video_devdata(file);
+
+	if (vb2_queue_is_busy(vdev, file))
+		return -EBUSY;
+	return vb2_ext_prepare_buf(vdev->queue, vdev->v4l2_dev->mdev, p);
+}
+EXPORT_SYMBOL_GPL(vb2_ioctl_ext_prepare_buf);
+
 int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
@@ -1060,6 +1104,16 @@ int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p)
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_querybuf);
 
+int vb2_ioctl_ext_querybuf(struct file *file, void *priv,
+			   struct v4l2_ext_buffer *p)
+{
+	struct video_device *vdev = video_devdata(file);
+
+	/* No need to call vb2_queue_is_busy(), anyone can query buffers. */
+	return vb2_ext_querybuf(vdev->queue, p);
+}
+EXPORT_SYMBOL_GPL(vb2_ioctl_ext_querybuf);
+
 int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
@@ -1070,6 +1124,17 @@ int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_qbuf);
 
+int vb2_ioctl_ext_qbuf(struct file *file, void *priv,
+		       struct v4l2_ext_buffer *p)
+{
+	struct video_device *vdev = video_devdata(file);
+
+	if (vb2_queue_is_busy(vdev, file))
+		return -EBUSY;
+	return vb2_ext_qbuf(vdev->queue, vdev->v4l2_dev->mdev, p);
+}
+EXPORT_SYMBOL_GPL(vb2_ioctl_ext_qbuf);
+
 int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 {
 	struct video_device *vdev = video_devdata(file);
@@ -1080,6 +1145,17 @@ int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_dqbuf);
 
+int vb2_ioctl_ext_dqbuf(struct file *file, void *priv,
+			struct v4l2_ext_buffer *p)
+{
+	struct video_device *vdev = video_devdata(file);
+
+	if (vb2_queue_is_busy(vdev, file))
+		return -EBUSY;
+	return vb2_ext_dqbuf(vdev->queue, p, file->f_flags & O_NONBLOCK);
+}
+EXPORT_SYMBOL_GPL(vb2_ioctl_ext_dqbuf);
+
 int vb2_ioctl_streamon(struct file *file, void *priv, enum v4l2_buf_type i)
 {
 	struct video_device *vdev = video_devdata(file);
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 52ef92049073e..f334767785bd9 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -152,6 +152,8 @@ struct vb2_mem_ops {
  * @mem_priv:	private data with this plane.
  * @dbuf:	dma_buf - shared buffer object.
  * @dbuf_mapped:	flag to show whether dbuf is mapped or not
+ * @dbuf_offset: offset where the plane starts. Usually 0, unless the buffer
+ *		 is shared by all planes of a multi-planar format.
  * @bytesused:	number of bytes occupied by data in the plane (payload).
  * @length:	size of this plane (NOT the payload) in bytes.
  * @min_length:	minimum required size of this plane (NOT the payload) in bytes.
@@ -175,6 +177,7 @@ struct vb2_plane {
 	void			*mem_priv;
 	struct dma_buf		*dbuf;
 	unsigned int		dbuf_mapped;
+	unsigned int		dbuf_offset;
 	unsigned int		bytesused;
 	unsigned int		length;
 	unsigned int		min_length;
@@ -446,7 +449,8 @@ struct vb2_ops {
  *			struct vb2_buffer.
  *			For V4L2 this is a &struct vb2_v4l2_buffer.
  * @fill_user_buffer:	given a &vb2_buffer fill in the userspace structure.
- *			For V4L2 this is a &struct v4l2_buffer.
+ *			For V4L2 this is a &struct v4l2_buffer or
+ *			&struct v4l2_ext_buffer.
  * @fill_vb2_buffer:	given a userspace structure, fill in the &vb2_buffer.
  *			If the userspace structure is invalid, then this op
  *			will return an error.
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index b7b5a9cb5a280..1bfde6e391bf0 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -37,7 +37,7 @@
  * @planes:	plane information (userptr/fd, length, bytesused, data_offset).
  *
  * Should contain enough information to be able to cover all the fields
- * of &struct v4l2_buffer at ``videodev2.h``.
+ * of &struct v4l2_buffer and &struct v4l2_ext_buffer at ``videodev2.h``.
  */
 struct vb2_v4l2_buffer {
 	struct vb2_buffer	vb2_buf;
@@ -77,6 +77,7 @@ int vb2_find_timestamp(const struct vb2_queue *q, u64 timestamp,
 		       unsigned int start_idx);
 
 int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b);
+int vb2_ext_querybuf(struct vb2_queue *q, struct v4l2_ext_buffer *b);
 
 /**
  * vb2_reqbufs() - Wrapper for vb2_core_reqbufs() that also verifies
@@ -97,6 +98,8 @@ int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req);
  *		&v4l2_ioctl_ops->vidioc_create_bufs handler in driver
  */
 int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create);
+int vb2_ext_create_bufs(struct vb2_queue *q,
+			struct v4l2_ext_create_buffers *create);
 
 /**
  * vb2_prepare_buf() - Pass ownership of a buffer from userspace to the kernel
@@ -122,6 +125,8 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create);
  */
 int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
 		    struct v4l2_buffer *b);
+int vb2_ext_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
+			struct v4l2_ext_buffer *b);
 
 /**
  * vb2_qbuf() - Queue a buffer from userspace
@@ -148,6 +153,8 @@ int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
  */
 int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
 	     struct v4l2_buffer *b);
+int vb2_ext_qbuf(struct vb2_queue *q, struct media_device *mdev,
+		 struct v4l2_ext_buffer *b);
 
 /**
  * vb2_expbuf() - Export a buffer as a file descriptor
@@ -185,6 +192,8 @@ int vb2_expbuf(struct vb2_queue *q, struct v4l2_exportbuffer *eb);
  * from &v4l2_ioctl_ops->vidioc_dqbuf handler in driver.
  */
 int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking);
+int vb2_ext_dqbuf(struct vb2_queue *q, struct v4l2_ext_buffer *b,
+		  bool nonblocking);
 
 /**
  * vb2_streamon - start streaming
@@ -294,11 +303,21 @@ int vb2_ioctl_reqbufs(struct file *file, void *priv,
 			  struct v4l2_requestbuffers *p);
 int vb2_ioctl_create_bufs(struct file *file, void *priv,
 			  struct v4l2_create_buffers *p);
+int vb2_ioctl_ext_create_bufs(struct file *file, void *priv,
+			      struct v4l2_ext_create_buffers *p);
 int vb2_ioctl_prepare_buf(struct file *file, void *priv,
 			  struct v4l2_buffer *p);
+int vb2_ioctl_ext_prepare_buf(struct file *file, void *priv,
+			      struct v4l2_ext_buffer *p);
 int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p);
+int vb2_ioctl_ext_querybuf(struct file *file, void *priv,
+			   struct v4l2_ext_buffer *p);
 int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p);
+int vb2_ioctl_ext_qbuf(struct file *file, void *priv,
+		       struct v4l2_ext_buffer *p);
 int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p);
+int vb2_ioctl_ext_dqbuf(struct file *file, void *priv,
+			struct v4l2_ext_buffer *p);
 int vb2_ioctl_streamon(struct file *file, void *priv, enum v4l2_buf_type i);
 int vb2_ioctl_streamoff(struct file *file, void *priv, enum v4l2_buf_type i);
 int vb2_ioctl_expbuf(struct file *file, void *priv,
-- 
2.28.0.rc2


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v5 4/7] media: mediabus: Add helpers to convert a ext_pix format to/from a mbus_fmt
  2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (2 preceding siblings ...)
  2020-08-04 19:29 ` [PATCH v5 3/7] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks Helen Koike
@ 2020-08-04 19:29 ` Helen Koike
  2020-08-14  7:49   ` Alexandre Courbot
  2020-08-04 19:29 ` [PATCH v5 5/7] media: vivid: Convert the capture and output drivers to EXT_FMT/EXT_BUF Helen Koike
                   ` (5 subsequent siblings)
  9 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-08-04 19:29 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Helen Koike, Boris Brezillon, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

Just a new version of v4l2_fill_mbus_format() and v4l2_fill_ext_pix_format()
to deal with the new v4l2_ext_pix_format struct.
This is needed to convert the VIMC driver to the EXT_FMT/EXT_BUF iocts.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v4:
- Add helper v4l2_fill_ext_pix_format()
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch
---
 include/media/v4l2-mediabus.h | 42 +++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/include/media/v4l2-mediabus.h b/include/media/v4l2-mediabus.h
index 45f88f0248c4e..8133407377f7d 100644
--- a/include/media/v4l2-mediabus.h
+++ b/include/media/v4l2-mediabus.h
@@ -119,6 +119,26 @@ v4l2_fill_pix_format(struct v4l2_pix_format *pix_fmt,
 	pix_fmt->xfer_func = mbus_fmt->xfer_func;
 }
 
+/**
+ * v4l2_fill_ext_pix_format - Ancillary routine that fills a &struct
+ *	v4l2_ext_pix_format fields from a &struct v4l2_mbus_framefmt.
+ *
+ * @pix_fmt:	pointer to &struct v4l2_ext_pix_format to be filled
+ * @mbus_fmt:	pointer to &struct v4l2_mbus_framefmt to be used as model
+ */
+static inline void
+v4l2_fill_ext_pix_format(struct v4l2_ext_pix_format *pix_fmt,
+		     const struct v4l2_mbus_framefmt *mbus_fmt)
+{
+	pix_fmt->width = mbus_fmt->width;
+	pix_fmt->height = mbus_fmt->height;
+	pix_fmt->field = mbus_fmt->field;
+	pix_fmt->colorspace = mbus_fmt->colorspace;
+	pix_fmt->ycbcr_enc = mbus_fmt->ycbcr_enc;
+	pix_fmt->quantization = mbus_fmt->quantization;
+	pix_fmt->xfer_func = mbus_fmt->xfer_func;
+}
+
 /**
  * v4l2_fill_pix_format - Ancillary routine that fills a &struct
  *	v4l2_mbus_framefmt from a &struct v4l2_pix_format and a
@@ -182,4 +202,26 @@ v4l2_fill_mbus_format_mplane(struct v4l2_mbus_framefmt *mbus_fmt,
 	mbus_fmt->xfer_func = pix_mp_fmt->xfer_func;
 }
 
+/**
+ * v4l2_fill_mbus_format_ext - Ancillary routine that fills a &struct
+ *	v4l2_mbus_framefmt from a &struct v4l2_ext_pix_format.
+ *
+ * @mbus_fmt:	pointer to &struct v4l2_mbus_framefmt to be filled
+ * @pix_fmt:	pointer to &struct v4l2_ext_pix_format to be used as model
+ * @code:	data format code (from &enum v4l2_mbus_pixelcode)
+ */
+static inline void
+v4l2_fill_mbus_format_ext(struct v4l2_mbus_framefmt *mbus_fmt,
+			  const struct v4l2_ext_pix_format *pix_fmt, u32 code)
+{
+	mbus_fmt->width = pix_fmt->width;
+	mbus_fmt->height = pix_fmt->height;
+	mbus_fmt->field = pix_fmt->field;
+	mbus_fmt->colorspace = pix_fmt->colorspace;
+	mbus_fmt->ycbcr_enc = pix_fmt->ycbcr_enc;
+	mbus_fmt->quantization = pix_fmt->quantization;
+	mbus_fmt->xfer_func = pix_fmt->xfer_func;
+	mbus_fmt->code = code;
+}
+
 #endif
-- 
2.28.0.rc2


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v5 5/7] media: vivid: Convert the capture and output drivers to EXT_FMT/EXT_BUF
  2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (3 preceding siblings ...)
  2020-08-04 19:29 ` [PATCH v5 4/7] media: mediabus: Add helpers to convert a ext_pix format to/from a mbus_fmt Helen Koike
@ 2020-08-04 19:29 ` Helen Koike
  2020-08-04 19:29 ` [PATCH v5 6/7] media: vimc: Implement the ext_fmt and ext_buf hooks Helen Koike
                   ` (4 subsequent siblings)
  9 siblings, 0 replies; 51+ messages in thread
From: Helen Koike @ 2020-08-04 19:29 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Helen Koike, Boris Brezillon, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

This should simplify things a bit as we now have a single
implementation instead of the MPLANE and !MPLANE versions.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v4:
- Update with new format and buffer structs
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch
---
 drivers/media/test-drivers/vivid/vivid-core.c |  70 ++-----
 .../test-drivers/vivid/vivid-touch-cap.c      |  26 +--
 .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
 .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +++++----------
 .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
 .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++++++------------
 .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
 7 files changed, 154 insertions(+), 337 deletions(-)

diff --git a/drivers/media/test-drivers/vivid/vivid-core.c b/drivers/media/test-drivers/vivid/vivid-core.c
index f7ee37e9508db..69b9433487150 100644
--- a/drivers/media/test-drivers/vivid/vivid-core.c
+++ b/drivers/media/test-drivers/vivid/vivid-core.c
@@ -488,63 +488,33 @@ static int vivid_enum_fmt_cap(struct file *file, void  *priv,
 }
 
 static int vivid_g_fmt_cap(struct file *file, void *priv,
-			   struct v4l2_format *f)
+			   struct v4l2_ext_pix_format *f)
 {
 	struct video_device *vdev = video_devdata(file);
 
 	if (vdev->vfl_type == VFL_TYPE_TOUCH)
 		return vivid_g_fmt_tch(file, priv, f);
-	return vidioc_g_fmt_vid_cap(file, priv, f);
+	return vivid_g_fmt_vid_cap(file, priv, f);
 }
 
 static int vivid_try_fmt_cap(struct file *file, void *priv,
-			     struct v4l2_format *f)
+			     struct v4l2_ext_pix_format *f)
 {
 	struct video_device *vdev = video_devdata(file);
 
 	if (vdev->vfl_type == VFL_TYPE_TOUCH)
 		return vivid_g_fmt_tch(file, priv, f);
-	return vidioc_try_fmt_vid_cap(file, priv, f);
+	return vivid_try_fmt_vid_cap(file, priv, f);
 }
 
 static int vivid_s_fmt_cap(struct file *file, void *priv,
-			   struct v4l2_format *f)
+			   struct v4l2_ext_pix_format *f)
 {
 	struct video_device *vdev = video_devdata(file);
 
 	if (vdev->vfl_type == VFL_TYPE_TOUCH)
 		return vivid_g_fmt_tch(file, priv, f);
-	return vidioc_s_fmt_vid_cap(file, priv, f);
-}
-
-static int vivid_g_fmt_cap_mplane(struct file *file, void *priv,
-				  struct v4l2_format *f)
-{
-	struct video_device *vdev = video_devdata(file);
-
-	if (vdev->vfl_type == VFL_TYPE_TOUCH)
-		return vivid_g_fmt_tch_mplane(file, priv, f);
-	return vidioc_g_fmt_vid_cap_mplane(file, priv, f);
-}
-
-static int vivid_try_fmt_cap_mplane(struct file *file, void *priv,
-				    struct v4l2_format *f)
-{
-	struct video_device *vdev = video_devdata(file);
-
-	if (vdev->vfl_type == VFL_TYPE_TOUCH)
-		return vivid_g_fmt_tch_mplane(file, priv, f);
-	return vidioc_try_fmt_vid_cap_mplane(file, priv, f);
-}
-
-static int vivid_s_fmt_cap_mplane(struct file *file, void *priv,
-				  struct v4l2_format *f)
-{
-	struct video_device *vdev = video_devdata(file);
-
-	if (vdev->vfl_type == VFL_TYPE_TOUCH)
-		return vivid_g_fmt_tch_mplane(file, priv, f);
-	return vidioc_s_fmt_vid_cap_mplane(file, priv, f);
+	return vivid_s_fmt_vid_cap(file, priv, f);
 }
 
 static bool vivid_is_in_use(struct video_device *vdev)
@@ -640,20 +610,14 @@ static const struct v4l2_ioctl_ops vivid_ioctl_ops = {
 	.vidioc_querycap		= vidioc_querycap,
 
 	.vidioc_enum_fmt_vid_cap	= vivid_enum_fmt_cap,
-	.vidioc_g_fmt_vid_cap		= vivid_g_fmt_cap,
-	.vidioc_try_fmt_vid_cap		= vivid_try_fmt_cap,
-	.vidioc_s_fmt_vid_cap		= vivid_s_fmt_cap,
-	.vidioc_g_fmt_vid_cap_mplane	= vivid_g_fmt_cap_mplane,
-	.vidioc_try_fmt_vid_cap_mplane	= vivid_try_fmt_cap_mplane,
-	.vidioc_s_fmt_vid_cap_mplane	= vivid_s_fmt_cap_mplane,
+	.vidioc_g_ext_pix_fmt_vid_cap	= vivid_g_fmt_cap,
+	.vidioc_try_ext_pix_fmt_vid_cap	= vivid_try_fmt_cap,
+	.vidioc_s_ext_pix_fmt_vid_cap	= vivid_s_fmt_cap,
 
 	.vidioc_enum_fmt_vid_out	= vivid_enum_fmt_vid,
-	.vidioc_g_fmt_vid_out		= vidioc_g_fmt_vid_out,
-	.vidioc_try_fmt_vid_out		= vidioc_try_fmt_vid_out,
-	.vidioc_s_fmt_vid_out		= vidioc_s_fmt_vid_out,
-	.vidioc_g_fmt_vid_out_mplane	= vidioc_g_fmt_vid_out_mplane,
-	.vidioc_try_fmt_vid_out_mplane	= vidioc_try_fmt_vid_out_mplane,
-	.vidioc_s_fmt_vid_out_mplane	= vidioc_s_fmt_vid_out_mplane,
+	.vidioc_g_ext_pix_fmt_vid_out	= vivid_g_fmt_vid_out,
+	.vidioc_try_ext_pix_fmt_vid_out	= vivid_try_fmt_vid_out,
+	.vidioc_s_ext_pix_fmt_vid_out	= vivid_s_fmt_vid_out,
 
 	.vidioc_g_selection		= vidioc_g_selection,
 	.vidioc_s_selection		= vidioc_s_selection,
@@ -698,11 +662,11 @@ static const struct v4l2_ioctl_ops vivid_ioctl_ops = {
 	.vidioc_s_fbuf			= vidioc_s_fbuf,
 
 	.vidioc_reqbufs			= vb2_ioctl_reqbufs,
-	.vidioc_create_bufs		= vb2_ioctl_create_bufs,
-	.vidioc_prepare_buf		= vb2_ioctl_prepare_buf,
-	.vidioc_querybuf		= vb2_ioctl_querybuf,
-	.vidioc_qbuf			= vb2_ioctl_qbuf,
-	.vidioc_dqbuf			= vb2_ioctl_dqbuf,
+	.vidioc_ext_create_bufs		= vb2_ioctl_ext_create_bufs,
+	.vidioc_ext_prepare_buf		= vb2_ioctl_ext_prepare_buf,
+	.vidioc_ext_querybuf		= vb2_ioctl_ext_querybuf,
+	.vidioc_ext_qbuf		= vb2_ioctl_ext_qbuf,
+	.vidioc_ext_dqbuf		= vb2_ioctl_ext_dqbuf,
 	.vidioc_expbuf			= vb2_ioctl_expbuf,
 	.vidioc_streamon		= vb2_ioctl_streamon,
 	.vidioc_streamoff		= vb2_ioctl_streamoff,
diff --git a/drivers/media/test-drivers/vivid/vivid-touch-cap.c b/drivers/media/test-drivers/vivid/vivid-touch-cap.c
index ebb00b128030c..6bbe7616e91a1 100644
--- a/drivers/media/test-drivers/vivid/vivid-touch-cap.c
+++ b/drivers/media/test-drivers/vivid/vivid-touch-cap.c
@@ -3,6 +3,8 @@
  * vivid-touch-cap.c - touch support functions.
  */
 
+#include <media/v4l2-ioctl.h>
+
 #include "vivid-core.h"
 #include "vivid-kthread-touch.h"
 #include "vivid-vid-common.h"
@@ -126,27 +128,19 @@ int vivid_enum_fmt_tch(struct file *file, void  *priv, struct v4l2_fmtdesc *f)
 	return 0;
 }
 
-int vivid_g_fmt_tch(struct file *file, void *priv, struct v4l2_format *f)
+int vivid_g_fmt_tch(struct file *file, void *priv, struct v4l2_ext_pix_format *e)
 {
 	struct vivid_dev *dev = video_drvdata(file);
+	struct v4l2_format f;
+	int ret;
 
-	if (dev->multiplanar)
-		return -ENOTTY;
-	f->fmt.pix = dev->tch_format;
-	return 0;
-}
+	ret = v4l2_ext_pix_format_to_format(e, &f, dev->multiplanar, false);
+	if (ret)
+		return ret;
 
-int vivid_g_fmt_tch_mplane(struct file *file, void *priv, struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-	struct v4l2_format sp_fmt;
+	f.fmt.pix = dev->tch_format;
 
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	sp_fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
-	sp_fmt.fmt.pix = dev->tch_format;
-	fmt_sp2mp(&sp_fmt, f);
-	return 0;
+	return v4l2_format_to_ext_pix_format(&f, e, false);
 }
 
 int vivid_g_parm_tch(struct file *file, void *priv,
diff --git a/drivers/media/test-drivers/vivid/vivid-touch-cap.h b/drivers/media/test-drivers/vivid/vivid-touch-cap.h
index 07e514046ae80..fccdf0a46824b 100644
--- a/drivers/media/test-drivers/vivid/vivid-touch-cap.h
+++ b/drivers/media/test-drivers/vivid/vivid-touch-cap.h
@@ -27,8 +27,7 @@ enum vivid_tch_test {
 extern const struct vb2_ops vivid_touch_cap_qops;
 
 int vivid_enum_fmt_tch(struct file *file, void  *priv, struct v4l2_fmtdesc *f);
-int vivid_g_fmt_tch(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_g_fmt_tch_mplane(struct file *file, void *priv, struct v4l2_format *f);
+int vivid_g_fmt_tch(struct file *file, void *priv, struct v4l2_ext_pix_format *e);
 int vivid_enum_input_tch(struct file *file, void *priv, struct v4l2_input *inp);
 int vivid_g_input_tch(struct file *file, void *priv, unsigned int *i);
 int vivid_s_input_tch(struct file *file, void *priv, unsigned int i);
diff --git a/drivers/media/test-drivers/vivid/vivid-vid-cap.c b/drivers/media/test-drivers/vivid/vivid-vid-cap.c
index e94beef008c8e..dcd616c4719bb 100644
--- a/drivers/media/test-drivers/vivid/vivid-vid-cap.c
+++ b/drivers/media/test-drivers/vivid/vivid-vid-cap.c
@@ -521,28 +521,26 @@ static unsigned vivid_quantization_cap(struct vivid_dev *dev)
 }
 
 int vivid_g_fmt_vid_cap(struct file *file, void *priv,
-					struct v4l2_format *f)
+			struct v4l2_ext_pix_format *f)
 {
 	struct vivid_dev *dev = video_drvdata(file);
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
 	unsigned p;
 
-	mp->width        = dev->fmt_cap_rect.width;
-	mp->height       = dev->fmt_cap_rect.height;
-	mp->field        = dev->field_cap;
-	mp->pixelformat  = dev->fmt_cap->fourcc;
-	mp->colorspace   = vivid_colorspace_cap(dev);
-	mp->xfer_func    = vivid_xfer_func_cap(dev);
+	f->width = dev->fmt_cap_rect.width;
+	f->height = dev->fmt_cap_rect.height;
+	f->field = dev->field_cap;
+	f->pixelformat = dev->fmt_cap->fourcc;
+	f->colorspace = vivid_colorspace_cap(dev);
+	f->xfer_func = vivid_xfer_func_cap(dev);
 	if (dev->fmt_cap->color_enc == TGP_COLOR_ENC_HSV)
-		mp->hsv_enc    = vivid_hsv_enc_cap(dev);
+		f->hsv_enc = vivid_hsv_enc_cap(dev);
 	else
-		mp->ycbcr_enc    = vivid_ycbcr_enc_cap(dev);
-	mp->quantization = vivid_quantization_cap(dev);
-	mp->num_planes = dev->fmt_cap->buffers;
-	for (p = 0; p < mp->num_planes; p++) {
-		mp->plane_fmt[p].bytesperline = tpg_g_bytesperline(&dev->tpg, p);
-		mp->plane_fmt[p].sizeimage =
-			(tpg_g_line_width(&dev->tpg, p) * mp->height) /
+		f->ycbcr_enc    = vivid_ycbcr_enc_cap(dev);
+	f->quantization = vivid_quantization_cap(dev);
+	for (p = 0; p < dev->fmt_cap->buffers; p++) {
+		f->plane_fmt[p].bytesperline = tpg_g_bytesperline(&dev->tpg, p);
+		f->plane_fmt[p].sizeimage =
+			(tpg_g_line_width(&dev->tpg, p) * f->height) /
 			dev->fmt_cap->vdownsampling[p] +
 			dev->fmt_cap->data_offset[p];
 	}
@@ -550,31 +548,30 @@ int vivid_g_fmt_vid_cap(struct file *file, void *priv,
 }
 
 int vivid_try_fmt_vid_cap(struct file *file, void *priv,
-			struct v4l2_format *f)
+			  struct v4l2_ext_pix_format *f)
 {
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
-	struct v4l2_plane_pix_format *pfmt = mp->plane_fmt;
 	struct vivid_dev *dev = video_drvdata(file);
+	struct v4l2_plane_ext_pix_format *pfmt = f->plane_fmt;
 	const struct vivid_fmt *fmt;
 	unsigned bytesperline, max_bpl;
 	unsigned factor = 1;
 	unsigned w, h;
 	unsigned p;
 
-	fmt = vivid_get_format(dev, mp->pixelformat);
+	fmt = vivid_get_format(dev, f->pixelformat);
 	if (!fmt) {
 		dprintk(dev, 1, "Fourcc format (0x%08x) unknown.\n",
-			mp->pixelformat);
-		mp->pixelformat = V4L2_PIX_FMT_YUYV;
-		fmt = vivid_get_format(dev, mp->pixelformat);
+			f->pixelformat);
+		f->pixelformat = V4L2_PIX_FMT_YUYV;
+		fmt = vivid_get_format(dev, f->pixelformat);
 	}
 
-	mp->field = vivid_field_cap(dev, mp->field);
+	f->field = vivid_field_cap(dev, f->field);
 	if (vivid_is_webcam(dev)) {
 		const struct v4l2_frmsize_discrete *sz =
 			v4l2_find_nearest_size(webcam_sizes,
 					       VIVID_WEBCAM_SIZES, width,
-					       height, mp->width, mp->height);
+					       height, f->width, f->height);
 
 		w = sz->width;
 		h = sz->height;
@@ -585,14 +582,14 @@ int vivid_try_fmt_vid_cap(struct file *file, void *priv,
 		w = dev->src_rect.width;
 		h = dev->src_rect.height;
 	}
-	if (V4L2_FIELD_HAS_T_OR_B(mp->field))
+	if (V4L2_FIELD_HAS_T_OR_B(f->field))
 		factor = 2;
 	if (vivid_is_webcam(dev) ||
 	    (!dev->has_scaler_cap && !dev->has_crop_cap && !dev->has_compose_cap)) {
-		mp->width = w;
-		mp->height = h / factor;
+		f->width = w;
+		f->height = h / factor;
 	} else {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height * factor };
+		struct v4l2_rect r = { 0, 0, f->width, f->height * factor };
 
 		v4l2_rect_set_min_size(&r, &vivid_min_rect);
 		v4l2_rect_set_max_size(&r, &vivid_max_rect);
@@ -605,16 +602,15 @@ int vivid_try_fmt_vid_cap(struct file *file, void *priv,
 		} else if (!dev->has_scaler_cap && !dev->has_crop_cap) {
 			v4l2_rect_set_min_size(&r, &dev->src_rect);
 		}
-		mp->width = r.width;
-		mp->height = r.height / factor;
+		f->width = r.width;
+		f->height = r.height / factor;
 	}
 
 	/* This driver supports custom bytesperline values */
 
-	mp->num_planes = fmt->buffers;
 	for (p = 0; p < fmt->buffers; p++) {
 		/* Calculate the minimum supported bytesperline value */
-		bytesperline = (mp->width * fmt->bit_depth[p]) >> 3;
+		bytesperline = (f->width * fmt->bit_depth[p]) >> 3;
 		/* Calculate the maximum supported bytesperline value */
 		max_bpl = (MAX_ZOOM * MAX_WIDTH * fmt->bit_depth[p]) >> 3;
 
@@ -623,31 +619,27 @@ int vivid_try_fmt_vid_cap(struct file *file, void *priv,
 		if (pfmt[p].bytesperline < bytesperline)
 			pfmt[p].bytesperline = bytesperline;
 
-		pfmt[p].sizeimage = (pfmt[p].bytesperline * mp->height) /
+		pfmt[p].sizeimage = (pfmt[p].bytesperline * f->height) /
 				fmt->vdownsampling[p] + fmt->data_offset[p];
-
-		memset(pfmt[p].reserved, 0, sizeof(pfmt[p].reserved));
 	}
 	for (p = fmt->buffers; p < fmt->planes; p++)
-		pfmt[0].sizeimage += (pfmt[0].bytesperline * mp->height *
+		pfmt[0].sizeimage += (pfmt[0].bytesperline * f->height *
 			(fmt->bit_depth[p] / fmt->vdownsampling[p])) /
 			(fmt->bit_depth[0] / fmt->vdownsampling[0]);
 
-	mp->colorspace = vivid_colorspace_cap(dev);
+	f->colorspace = vivid_colorspace_cap(dev);
 	if (fmt->color_enc == TGP_COLOR_ENC_HSV)
-		mp->hsv_enc = vivid_hsv_enc_cap(dev);
+		f->hsv_enc = vivid_hsv_enc_cap(dev);
 	else
-		mp->ycbcr_enc = vivid_ycbcr_enc_cap(dev);
-	mp->xfer_func = vivid_xfer_func_cap(dev);
-	mp->quantization = vivid_quantization_cap(dev);
-	memset(mp->reserved, 0, sizeof(mp->reserved));
+		f->ycbcr_enc = vivid_ycbcr_enc_cap(dev);
+	f->xfer_func = vivid_xfer_func_cap(dev);
+	f->quantization = vivid_quantization_cap(dev);
 	return 0;
 }
 
 int vivid_s_fmt_vid_cap(struct file *file, void *priv,
-					struct v4l2_format *f)
+			struct v4l2_ext_pix_format *f)
 {
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
 	struct vivid_dev *dev = video_drvdata(file);
 	struct v4l2_rect *crop = &dev->crop_cap;
 	struct v4l2_rect *compose = &dev->compose_cap;
@@ -665,20 +657,21 @@ int vivid_s_fmt_vid_cap(struct file *file, void *priv,
 		return -EBUSY;
 	}
 
-	if (dev->overlay_cap_owner && dev->fb_cap.fmt.pixelformat != mp->pixelformat) {
+	if (dev->overlay_cap_owner &&
+	    dev->fb_cap.fmt.pixelformat != f->pixelformat) {
 		dprintk(dev, 1, "overlay is active, can't change pixelformat\n");
 		return -EBUSY;
 	}
 
-	dev->fmt_cap = vivid_get_format(dev, mp->pixelformat);
-	if (V4L2_FIELD_HAS_T_OR_B(mp->field))
+	dev->fmt_cap = vivid_get_format(dev, f->pixelformat);
+	if (V4L2_FIELD_HAS_T_OR_B(f->field))
 		factor = 2;
 
 	/* Note: the webcam input doesn't support scaling, cropping or composing */
 
 	if (!vivid_is_webcam(dev) &&
 	    (dev->has_scaler_cap || dev->has_crop_cap || dev->has_compose_cap)) {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height };
+		struct v4l2_rect r = { 0, 0, f->width, f->height };
 
 		if (dev->has_scaler_cap) {
 			if (dev->has_compose_cap)
@@ -739,99 +732,39 @@ int vivid_s_fmt_vid_cap(struct file *file, void *priv,
 	} else if (vivid_is_webcam(dev)) {
 		/* Guaranteed to be a match */
 		for (i = 0; i < ARRAY_SIZE(webcam_sizes); i++)
-			if (webcam_sizes[i].width == mp->width &&
-					webcam_sizes[i].height == mp->height)
+			if (webcam_sizes[i].width == f->width &&
+			    webcam_sizes[i].height == f->height)
 				break;
 		dev->webcam_size_idx = i;
 		if (dev->webcam_ival_idx >= 2 * (VIVID_WEBCAM_SIZES - i))
 			dev->webcam_ival_idx = 2 * (VIVID_WEBCAM_SIZES - i) - 1;
 		vivid_update_format_cap(dev, false);
 	} else {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height };
+		struct v4l2_rect r = { 0, 0, f->width, f->height };
 
 		v4l2_rect_set_size_to(compose, &r);
 		r.height *= factor;
 		v4l2_rect_set_size_to(crop, &r);
 	}
 
-	dev->fmt_cap_rect.width = mp->width;
-	dev->fmt_cap_rect.height = mp->height;
-	tpg_s_buf_height(&dev->tpg, mp->height);
+	dev->fmt_cap_rect.width = f->width;
+	dev->fmt_cap_rect.height = f->height;
+	tpg_s_buf_height(&dev->tpg, f->height);
 	tpg_s_fourcc(&dev->tpg, dev->fmt_cap->fourcc);
 	for (p = 0; p < tpg_g_buffers(&dev->tpg); p++)
-		tpg_s_bytesperline(&dev->tpg, p, mp->plane_fmt[p].bytesperline);
-	dev->field_cap = mp->field;
+		tpg_s_bytesperline(&dev->tpg, p, f->plane_fmt[p].bytesperline);
+	dev->field_cap = f->field;
 	if (dev->field_cap == V4L2_FIELD_ALTERNATE)
 		tpg_s_field(&dev->tpg, V4L2_FIELD_TOP, true);
 	else
 		tpg_s_field(&dev->tpg, dev->field_cap, false);
 	tpg_s_crop_compose(&dev->tpg, &dev->crop_cap, &dev->compose_cap);
 	if (vivid_is_sdtv_cap(dev))
-		dev->tv_field_cap = mp->field;
+		dev->tv_field_cap = f->field;
 	tpg_update_mv_step(&dev->tpg);
 	return 0;
 }
 
-int vidioc_g_fmt_vid_cap_mplane(struct file *file, void *priv,
-					struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_g_fmt_vid_cap(file, priv, f);
-}
-
-int vidioc_try_fmt_vid_cap_mplane(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_try_fmt_vid_cap(file, priv, f);
-}
-
-int vidioc_s_fmt_vid_cap_mplane(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_s_fmt_vid_cap(file, priv, f);
-}
-
-int vidioc_g_fmt_vid_cap(struct file *file, void *priv,
-					struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_g_fmt_vid_cap);
-}
-
-int vidioc_try_fmt_vid_cap(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_try_fmt_vid_cap);
-}
-
-int vidioc_s_fmt_vid_cap(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_s_fmt_vid_cap);
-}
-
 int vivid_vid_cap_g_selection(struct file *file, void *priv,
 			      struct v4l2_selection *sel)
 {
diff --git a/drivers/media/test-drivers/vivid/vivid-vid-cap.h b/drivers/media/test-drivers/vivid/vivid-vid-cap.h
index 1e422a59eeabf..7c9fc5c787b59 100644
--- a/drivers/media/test-drivers/vivid/vivid-vid-cap.h
+++ b/drivers/media/test-drivers/vivid/vivid-vid-cap.h
@@ -17,15 +17,12 @@ extern const char * const vivid_ctrl_standard_strings[];
 
 extern const struct vb2_ops vivid_vid_cap_qops;
 
-int vivid_g_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_try_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_s_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_cap_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_cap_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_cap_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f);
+int vivid_g_fmt_vid_cap(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
+int vivid_try_fmt_vid_cap(struct file *file, void *priv,
+			  struct v4l2_ext_pix_format *f);
+int vivid_s_fmt_vid_cap(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
 int vivid_vid_cap_g_selection(struct file *file, void *priv, struct v4l2_selection *sel);
 int vivid_vid_cap_s_selection(struct file *file, void *fh, struct v4l2_selection *s);
 int vivid_vid_cap_g_pixelaspect(struct file *file, void *priv, int type, struct v4l2_fract *f);
diff --git a/drivers/media/test-drivers/vivid/vivid-vid-out.c b/drivers/media/test-drivers/vivid/vivid-vid-out.c
index ee3446e3217cc..1da6bd5a8d121 100644
--- a/drivers/media/test-drivers/vivid/vivid-vid-out.c
+++ b/drivers/media/test-drivers/vivid/vivid-vid-out.c
@@ -315,59 +315,56 @@ static enum tpg_pixel_aspect vivid_get_pixel_aspect(const struct vivid_dev *dev)
 }
 
 int vivid_g_fmt_vid_out(struct file *file, void *priv,
-					struct v4l2_format *f)
+			struct v4l2_ext_pix_format *f)
 {
 	struct vivid_dev *dev = video_drvdata(file);
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
 	const struct vivid_fmt *fmt = dev->fmt_out;
 	unsigned p;
 
-	mp->width        = dev->fmt_out_rect.width;
-	mp->height       = dev->fmt_out_rect.height;
-	mp->field        = dev->field_out;
-	mp->pixelformat  = fmt->fourcc;
-	mp->colorspace   = dev->colorspace_out;
-	mp->xfer_func    = dev->xfer_func_out;
-	mp->ycbcr_enc    = dev->ycbcr_enc_out;
-	mp->quantization = dev->quantization_out;
-	mp->num_planes = fmt->buffers;
-	for (p = 0; p < mp->num_planes; p++) {
-		mp->plane_fmt[p].bytesperline = dev->bytesperline_out[p];
-		mp->plane_fmt[p].sizeimage =
-			mp->plane_fmt[p].bytesperline * mp->height +
+	f->width        = dev->fmt_out_rect.width;
+	f->height       = dev->fmt_out_rect.height;
+	f->field        = dev->field_out;
+	f->pixelformat  = fmt->fourcc;
+	f->colorspace   = dev->colorspace_out;
+	f->xfer_func    = dev->xfer_func_out;
+	f->ycbcr_enc    = dev->ycbcr_enc_out;
+	f->quantization = dev->quantization_out;
+	for (p = 0; p < fmt->buffers; p++) {
+		f->plane_fmt[p].bytesperline = dev->bytesperline_out[p];
+		f->plane_fmt[p].sizeimage =
+			f->plane_fmt[p].bytesperline * f->height +
 			fmt->data_offset[p];
 	}
 	for (p = fmt->buffers; p < fmt->planes; p++) {
 		unsigned stride = dev->bytesperline_out[p];
 
-		mp->plane_fmt[0].sizeimage +=
-			(stride * mp->height) / fmt->vdownsampling[p];
+		f->plane_fmt[0].sizeimage +=
+			(stride * f->height) / fmt->vdownsampling[p];
 	}
 	return 0;
 }
 
 int vivid_try_fmt_vid_out(struct file *file, void *priv,
-			struct v4l2_format *f)
+			  struct v4l2_ext_pix_format *f)
 {
 	struct vivid_dev *dev = video_drvdata(file);
+	struct v4l2_plane_ext_pix_format *pfmt = f->plane_fmt;
 	struct v4l2_bt_timings *bt = &dev->dv_timings_out.bt;
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
-	struct v4l2_plane_pix_format *pfmt = mp->plane_fmt;
 	const struct vivid_fmt *fmt;
 	unsigned bytesperline, max_bpl;
 	unsigned factor = 1;
 	unsigned w, h;
 	unsigned p;
 
-	fmt = vivid_get_format(dev, mp->pixelformat);
+	fmt = vivid_get_format(dev, f->pixelformat);
 	if (!fmt) {
 		dprintk(dev, 1, "Fourcc format (0x%08x) unknown.\n",
-			mp->pixelformat);
-		mp->pixelformat = V4L2_PIX_FMT_YUYV;
-		fmt = vivid_get_format(dev, mp->pixelformat);
+			f->pixelformat);
+		f->pixelformat = V4L2_PIX_FMT_YUYV;
+		fmt = vivid_get_format(dev, f->pixelformat);
 	}
 
-	mp->field = vivid_field_out(dev, mp->field);
+	f->field = vivid_field_out(dev, f->field);
 	if (vivid_is_svid_out(dev)) {
 		w = 720;
 		h = (dev->std_out & V4L2_STD_525_60) ? 480 : 576;
@@ -375,13 +372,13 @@ int vivid_try_fmt_vid_out(struct file *file, void *priv,
 		w = dev->sink_rect.width;
 		h = dev->sink_rect.height;
 	}
-	if (V4L2_FIELD_HAS_T_OR_B(mp->field))
+	if (V4L2_FIELD_HAS_T_OR_B(f->field))
 		factor = 2;
 	if (!dev->has_scaler_out && !dev->has_crop_out && !dev->has_compose_out) {
-		mp->width = w;
-		mp->height = h / factor;
+		f->width = w;
+		f->height = h / factor;
 	} else {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height * factor };
+		struct v4l2_rect r = { 0, 0, f->width, f->height * factor };
 
 		v4l2_rect_set_min_size(&r, &vivid_min_rect);
 		v4l2_rect_set_max_size(&r, &vivid_max_rect);
@@ -394,16 +391,15 @@ int vivid_try_fmt_vid_out(struct file *file, void *priv,
 		} else if (!dev->has_scaler_out && !dev->has_compose_out) {
 			v4l2_rect_set_min_size(&r, &dev->sink_rect);
 		}
-		mp->width = r.width;
-		mp->height = r.height / factor;
+		f->width = r.width;
+		f->height = r.height / factor;
 	}
 
 	/* This driver supports custom bytesperline values */
 
-	mp->num_planes = fmt->buffers;
 	for (p = 0; p < fmt->buffers; p++) {
 		/* Calculate the minimum supported bytesperline value */
-		bytesperline = (mp->width * fmt->bit_depth[p]) >> 3;
+		bytesperline = (f->width * fmt->bit_depth[p]) >> 3;
 		/* Calculate the maximum supported bytesperline value */
 		max_bpl = (MAX_ZOOM * MAX_WIDTH * fmt->bit_depth[p]) >> 3;
 
@@ -412,42 +408,39 @@ int vivid_try_fmt_vid_out(struct file *file, void *priv,
 		if (pfmt[p].bytesperline < bytesperline)
 			pfmt[p].bytesperline = bytesperline;
 
-		pfmt[p].sizeimage = (pfmt[p].bytesperline * mp->height) /
+		pfmt[p].sizeimage = (pfmt[p].bytesperline * f->height) /
 				fmt->vdownsampling[p] + fmt->data_offset[p];
 
-		memset(pfmt[p].reserved, 0, sizeof(pfmt[p].reserved));
 	}
 	for (p = fmt->buffers; p < fmt->planes; p++)
-		pfmt[0].sizeimage += (pfmt[0].bytesperline * mp->height *
+		pfmt[0].sizeimage += (pfmt[0].bytesperline * f->height *
 			(fmt->bit_depth[p] / fmt->vdownsampling[p])) /
 			(fmt->bit_depth[0] / fmt->vdownsampling[0]);
 
-	mp->xfer_func = V4L2_XFER_FUNC_DEFAULT;
-	mp->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
-	mp->quantization = V4L2_QUANTIZATION_DEFAULT;
+	f->xfer_func = V4L2_XFER_FUNC_DEFAULT;
+	f->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
+	f->quantization = V4L2_QUANTIZATION_DEFAULT;
 	if (vivid_is_svid_out(dev)) {
-		mp->colorspace = V4L2_COLORSPACE_SMPTE170M;
+		f->colorspace = V4L2_COLORSPACE_SMPTE170M;
 	} else if (dev->dvi_d_out || !(bt->flags & V4L2_DV_FL_IS_CE_VIDEO)) {
-		mp->colorspace = V4L2_COLORSPACE_SRGB;
+		f->colorspace = V4L2_COLORSPACE_SRGB;
 		if (dev->dvi_d_out)
-			mp->quantization = V4L2_QUANTIZATION_LIM_RANGE;
+			f->quantization = V4L2_QUANTIZATION_LIM_RANGE;
 	} else if (bt->width == 720 && bt->height <= 576) {
-		mp->colorspace = V4L2_COLORSPACE_SMPTE170M;
-	} else if (mp->colorspace != V4L2_COLORSPACE_SMPTE170M &&
-		   mp->colorspace != V4L2_COLORSPACE_REC709 &&
-		   mp->colorspace != V4L2_COLORSPACE_OPRGB &&
-		   mp->colorspace != V4L2_COLORSPACE_BT2020 &&
-		   mp->colorspace != V4L2_COLORSPACE_SRGB) {
-		mp->colorspace = V4L2_COLORSPACE_REC709;
+		f->colorspace = V4L2_COLORSPACE_SMPTE170M;
+	} else if (f->colorspace != V4L2_COLORSPACE_SMPTE170M &&
+		   f->colorspace != V4L2_COLORSPACE_REC709 &&
+		   f->colorspace != V4L2_COLORSPACE_OPRGB &&
+		   f->colorspace != V4L2_COLORSPACE_BT2020 &&
+		   f->colorspace != V4L2_COLORSPACE_SRGB) {
+		f->colorspace = V4L2_COLORSPACE_REC709;
 	}
-	memset(mp->reserved, 0, sizeof(mp->reserved));
 	return 0;
 }
 
 int vivid_s_fmt_vid_out(struct file *file, void *priv,
-					struct v4l2_format *f)
+			struct v4l2_ext_pix_format *f)
 {
-	struct v4l2_pix_format_mplane *mp = &f->fmt.pix_mp;
 	struct vivid_dev *dev = video_drvdata(file);
 	struct v4l2_rect *crop = &dev->crop_out;
 	struct v4l2_rect *compose = &dev->compose_out;
@@ -461,10 +454,10 @@ int vivid_s_fmt_vid_out(struct file *file, void *priv,
 
 	if (vb2_is_busy(q) &&
 	    (vivid_is_svid_out(dev) ||
-	     mp->width != dev->fmt_out_rect.width ||
-	     mp->height != dev->fmt_out_rect.height ||
-	     mp->pixelformat != dev->fmt_out->fourcc ||
-	     mp->field != dev->field_out)) {
+	     f->width != dev->fmt_out_rect.width ||
+	     f->height != dev->fmt_out_rect.height ||
+	     f->pixelformat != dev->fmt_out->fourcc ||
+	     f->field != dev->field_out)) {
 		dprintk(dev, 1, "%s device busy\n", __func__);
 		return -EBUSY;
 	}
@@ -477,12 +470,12 @@ int vivid_s_fmt_vid_out(struct file *file, void *priv,
 	if (vb2_is_busy(q))
 		goto set_colorspace;
 
-	dev->fmt_out = vivid_get_format(dev, mp->pixelformat);
-	if (V4L2_FIELD_HAS_T_OR_B(mp->field))
+	dev->fmt_out = vivid_get_format(dev, f->pixelformat);
+	if (V4L2_FIELD_HAS_T_OR_B(f->field))
 		factor = 2;
 
 	if (dev->has_scaler_out || dev->has_crop_out || dev->has_compose_out) {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height };
+		struct v4l2_rect r = { 0, 0, f->width, f->height };
 
 		if (dev->has_scaler_out) {
 			if (dev->has_crop_out)
@@ -541,30 +534,30 @@ int vivid_s_fmt_vid_out(struct file *file, void *priv,
 			crop->height /= factor;
 		}
 	} else {
-		struct v4l2_rect r = { 0, 0, mp->width, mp->height };
+		struct v4l2_rect r = { 0, 0, f->width, f->height };
 
 		v4l2_rect_set_size_to(crop, &r);
 		r.height /= factor;
 		v4l2_rect_set_size_to(compose, &r);
 	}
 
-	dev->fmt_out_rect.width = mp->width;
-	dev->fmt_out_rect.height = mp->height;
-	for (p = 0; p < mp->num_planes; p++)
-		dev->bytesperline_out[p] = mp->plane_fmt[p].bytesperline;
+	dev->fmt_out_rect.width = f->width;
+	dev->fmt_out_rect.height = f->height;
+	for (p = 0; p < VIDEO_MAX_PLANES && f->plane_fmt[p].sizeimage; p++)
+		dev->bytesperline_out[p] = f->plane_fmt[p].bytesperline;
 	for (p = dev->fmt_out->buffers; p < dev->fmt_out->planes; p++)
 		dev->bytesperline_out[p] =
 			(dev->bytesperline_out[0] * dev->fmt_out->bit_depth[p]) /
 			dev->fmt_out->bit_depth[0];
-	dev->field_out = mp->field;
+	dev->field_out = f->field;
 	if (vivid_is_svid_out(dev))
-		dev->tv_field_out = mp->field;
+		dev->tv_field_out = f->field;
 
 set_colorspace:
-	dev->colorspace_out = mp->colorspace;
-	dev->xfer_func_out = mp->xfer_func;
-	dev->ycbcr_enc_out = mp->ycbcr_enc;
-	dev->quantization_out = mp->quantization;
+	dev->colorspace_out = f->colorspace;
+	dev->xfer_func_out = f->xfer_func;
+	dev->ycbcr_enc_out = f->ycbcr_enc;
+	dev->quantization_out = f->quantization;
 	if (dev->loop_video) {
 		vivid_send_source_change(dev, SVID);
 		vivid_send_source_change(dev, HDMI);
@@ -572,66 +565,6 @@ int vivid_s_fmt_vid_out(struct file *file, void *priv,
 	return 0;
 }
 
-int vidioc_g_fmt_vid_out_mplane(struct file *file, void *priv,
-					struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_g_fmt_vid_out(file, priv, f);
-}
-
-int vidioc_try_fmt_vid_out_mplane(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_try_fmt_vid_out(file, priv, f);
-}
-
-int vidioc_s_fmt_vid_out_mplane(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (!dev->multiplanar)
-		return -ENOTTY;
-	return vivid_s_fmt_vid_out(file, priv, f);
-}
-
-int vidioc_g_fmt_vid_out(struct file *file, void *priv,
-					struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_g_fmt_vid_out);
-}
-
-int vidioc_try_fmt_vid_out(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_try_fmt_vid_out);
-}
-
-int vidioc_s_fmt_vid_out(struct file *file, void *priv,
-			struct v4l2_format *f)
-{
-	struct vivid_dev *dev = video_drvdata(file);
-
-	if (dev->multiplanar)
-		return -ENOTTY;
-	return fmt_sp2mp_func(file, priv, f, vivid_s_fmt_vid_out);
-}
-
 int vivid_vid_out_g_selection(struct file *file, void *priv,
 			      struct v4l2_selection *sel)
 {
diff --git a/drivers/media/test-drivers/vivid/vivid-vid-out.h b/drivers/media/test-drivers/vivid/vivid-vid-out.h
index 8d56314f4ea1f..b84dc578af362 100644
--- a/drivers/media/test-drivers/vivid/vivid-vid-out.h
+++ b/drivers/media/test-drivers/vivid/vivid-vid-out.h
@@ -12,15 +12,12 @@ extern const struct vb2_ops vivid_vid_out_qops;
 
 void vivid_update_format_out(struct vivid_dev *dev);
 
-int vivid_g_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_try_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vivid_s_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_out_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_out_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_out_mplane(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_g_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_try_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
-int vidioc_s_fmt_vid_out(struct file *file, void *priv, struct v4l2_format *f);
+int vivid_g_fmt_vid_out(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
+int vivid_try_fmt_vid_out(struct file *file, void *priv,
+			  struct v4l2_ext_pix_format *f);
+int vivid_s_fmt_vid_out(struct file *file, void *priv,
+			struct v4l2_ext_pix_format *f);
 int vivid_vid_out_g_selection(struct file *file, void *priv, struct v4l2_selection *sel);
 int vivid_vid_out_s_selection(struct file *file, void *fh, struct v4l2_selection *s);
 int vivid_vid_out_g_pixelaspect(struct file *file, void *priv, int type, struct v4l2_fract *f);
-- 
2.28.0.rc2


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v5 6/7] media: vimc: Implement the ext_fmt and ext_buf hooks
  2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (4 preceding siblings ...)
  2020-08-04 19:29 ` [PATCH v5 5/7] media: vivid: Convert the capture and output drivers to EXT_FMT/EXT_BUF Helen Koike
@ 2020-08-04 19:29 ` Helen Koike
  2020-08-04 19:29 ` [PATCH v5 7/7] media: docs: add documentation for the Extended API Helen Koike
                   ` (3 subsequent siblings)
  9 siblings, 0 replies; 51+ messages in thread
From: Helen Koike @ 2020-08-04 19:29 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Helen Koike, Boris Brezillon, tfiga, hiroh, nicolas,
	Brian.Starkey, kernel, narmstrong, linux-kernel, frkoenig,
	mjourdan, stanimir.varbanov

Convert the driver to the _ext_fmt and _ext_buf API.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v4:
- Update with new format and buffer structs
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- New patch
---
 .../media/test-drivers/vimc/vimc-capture.c    | 61 +++++++++----------
 drivers/media/test-drivers/vimc/vimc-common.c |  6 +-
 drivers/media/test-drivers/vimc/vimc-common.h |  2 +-
 3 files changed, 34 insertions(+), 35 deletions(-)

diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
index c63496b17b9a5..0c8b86f0dd3ec 100644
--- a/drivers/media/test-drivers/vimc/vimc-capture.c
+++ b/drivers/media/test-drivers/vimc/vimc-capture.c
@@ -15,7 +15,7 @@
 struct vimc_cap_device {
 	struct vimc_ent_device ved;
 	struct video_device vdev;
-	struct v4l2_pix_format format;
+	struct v4l2_ext_pix_format format;
 	struct vb2_queue queue;
 	struct list_head buf_list;
 	/*
@@ -32,7 +32,7 @@ struct vimc_cap_device {
 	struct media_pad pad;
 };
 
-static const struct v4l2_pix_format fmt_default = {
+static const struct v4l2_ext_pix_format fmt_default = {
 	.width = 640,
 	.height = 480,
 	.pixelformat = V4L2_PIX_FMT_RGB24,
@@ -63,7 +63,7 @@ static int vimc_cap_querycap(struct file *file, void *priv,
 }
 
 static void vimc_cap_get_format(struct vimc_ent_device *ved,
-				struct v4l2_pix_format *fmt)
+				struct v4l2_ext_pix_format *fmt)
 {
 	struct vimc_cap_device *vcap = container_of(ved, struct vimc_cap_device,
 						    ved);
@@ -72,19 +72,18 @@ static void vimc_cap_get_format(struct vimc_ent_device *ved,
 }
 
 static int vimc_cap_g_fmt_vid_cap(struct file *file, void *priv,
-				  struct v4l2_format *f)
+				  struct v4l2_ext_pix_format *f)
 {
 	struct vimc_cap_device *vcap = video_drvdata(file);
 
-	f->fmt.pix = vcap->format;
+	*f = vcap->format;
 
 	return 0;
 }
 
 static int vimc_cap_try_fmt_vid_cap(struct file *file, void *priv,
-				    struct v4l2_format *f)
+				    struct v4l2_ext_pix_format *format)
 {
-	struct v4l2_pix_format *format = &f->fmt.pix;
 	const struct vimc_pix_map *vpix;
 
 	format->width = clamp_t(u32, format->width, VIMC_FRAME_MIN_WIDTH,
@@ -99,8 +98,9 @@ static int vimc_cap_try_fmt_vid_cap(struct file *file, void *priv,
 		vpix = vimc_pix_map_by_pixelformat(format->pixelformat);
 	}
 	/* TODO: Add support for custom bytesperline values */
-	format->bytesperline = format->width * vpix->bpp;
-	format->sizeimage = format->bytesperline * format->height;
+	format->plane_fmt[0].bytesperline = format->width * vpix->bpp;
+	format->plane_fmt[0].sizeimage = format->plane_fmt[0].bytesperline *
+					 format->height;
 
 	if (format->field == V4L2_FIELD_ANY)
 		format->field = fmt_default.field;
@@ -114,7 +114,7 @@ static int vimc_cap_try_fmt_vid_cap(struct file *file, void *priv,
 }
 
 static int vimc_cap_s_fmt_vid_cap(struct file *file, void *priv,
-				  struct v4l2_format *f)
+				  struct v4l2_ext_pix_format *f)
 {
 	struct vimc_cap_device *vcap = video_drvdata(file);
 	int ret;
@@ -136,12 +136,10 @@ static int vimc_cap_s_fmt_vid_cap(struct file *file, void *priv,
 		vcap->format.quantization, vcap->format.xfer_func,
 		vcap->format.ycbcr_enc,
 		/* new */
-		f->fmt.pix.width, f->fmt.pix.height,
-		f->fmt.pix.pixelformat,	f->fmt.pix.colorspace,
-		f->fmt.pix.quantization, f->fmt.pix.xfer_func,
-		f->fmt.pix.ycbcr_enc);
+		f->width, f->height, f->pixelformat, f->colorspace,
+		f->quantization, f->xfer_func, f->ycbcr_enc);
 
-	vcap->format = f->fmt.pix;
+	vcap->format = *f;
 
 	return 0;
 }
@@ -205,18 +203,18 @@ static const struct v4l2_file_operations vimc_cap_fops = {
 static const struct v4l2_ioctl_ops vimc_cap_ioctl_ops = {
 	.vidioc_querycap = vimc_cap_querycap,
 
-	.vidioc_g_fmt_vid_cap = vimc_cap_g_fmt_vid_cap,
-	.vidioc_s_fmt_vid_cap = vimc_cap_s_fmt_vid_cap,
-	.vidioc_try_fmt_vid_cap = vimc_cap_try_fmt_vid_cap,
+	.vidioc_g_ext_pix_fmt_vid_cap = vimc_cap_g_fmt_vid_cap,
+	.vidioc_s_ext_pix_fmt_vid_cap = vimc_cap_s_fmt_vid_cap,
+	.vidioc_try_ext_pix_fmt_vid_cap = vimc_cap_try_fmt_vid_cap,
 	.vidioc_enum_fmt_vid_cap = vimc_cap_enum_fmt_vid_cap,
 	.vidioc_enum_framesizes = vimc_cap_enum_framesizes,
 
 	.vidioc_reqbufs = vb2_ioctl_reqbufs,
-	.vidioc_create_bufs = vb2_ioctl_create_bufs,
-	.vidioc_prepare_buf = vb2_ioctl_prepare_buf,
-	.vidioc_querybuf = vb2_ioctl_querybuf,
-	.vidioc_qbuf = vb2_ioctl_qbuf,
-	.vidioc_dqbuf = vb2_ioctl_dqbuf,
+	.vidioc_ext_create_bufs = vb2_ioctl_ext_create_bufs,
+	.vidioc_ext_prepare_buf = vb2_ioctl_ext_prepare_buf,
+	.vidioc_ext_querybuf = vb2_ioctl_ext_querybuf,
+	.vidioc_ext_qbuf = vb2_ioctl_ext_qbuf,
+	.vidioc_ext_dqbuf = vb2_ioctl_ext_dqbuf,
 	.vidioc_expbuf = vb2_ioctl_expbuf,
 	.vidioc_streamon = vb2_ioctl_streamon,
 	.vidioc_streamoff = vb2_ioctl_streamoff,
@@ -298,10 +296,11 @@ static int vimc_cap_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers,
 	struct vimc_cap_device *vcap = vb2_get_drv_priv(vq);
 
 	if (*nplanes)
-		return sizes[0] < vcap->format.sizeimage ? -EINVAL : 0;
+		return sizes[0] < vcap->format.plane_fmt[0].sizeimage ?
+		       -EINVAL : 0;
 	/* We don't support multiplanes for now */
 	*nplanes = 1;
-	sizes[0] = vcap->format.sizeimage;
+	sizes[0] = vcap->format.plane_fmt[0].sizeimage;
 
 	return 0;
 }
@@ -309,7 +308,7 @@ static int vimc_cap_queue_setup(struct vb2_queue *vq, unsigned int *nbuffers,
 static int vimc_cap_buffer_prepare(struct vb2_buffer *vb)
 {
 	struct vimc_cap_device *vcap = vb2_get_drv_priv(vb->vb2_queue);
-	unsigned long size = vcap->format.sizeimage;
+	unsigned long size = vcap->format.plane_fmt[0].sizeimage;
 
 	if (vb2_plane_size(vb, 0) < size) {
 		dev_err(vcap->ved.dev, "%s: buffer too small (%lu < %lu)\n",
@@ -385,11 +384,11 @@ static void *vimc_cap_process_frame(struct vimc_ent_device *ved,
 
 	vbuf = vb2_plane_vaddr(&vimc_buf->vb2.vb2_buf, 0);
 
-	memcpy(vbuf, frame, vcap->format.sizeimage);
+	memcpy(vbuf, frame, vcap->format.plane_fmt[0].sizeimage);
 
 	/* Set it as ready */
 	vb2_set_plane_payload(&vimc_buf->vb2.vb2_buf, 0,
-			      vcap->format.sizeimage);
+			      vcap->format.plane_fmt[0].sizeimage);
 	vb2_buffer_done(&vimc_buf->vb2.vb2_buf, VB2_BUF_STATE_DONE);
 	return NULL;
 }
@@ -447,9 +446,9 @@ static struct vimc_ent_device *vimc_cap_add(struct vimc_device *vimc,
 	/* Set default frame format */
 	vcap->format = fmt_default;
 	vpix = vimc_pix_map_by_pixelformat(vcap->format.pixelformat);
-	vcap->format.bytesperline = vcap->format.width * vpix->bpp;
-	vcap->format.sizeimage = vcap->format.bytesperline *
-				 vcap->format.height;
+	vcap->format.plane_fmt[0].bytesperline = vcap->format.width * vpix->bpp;
+	vcap->format.plane_fmt[0].sizeimage = vcap->format.plane_fmt[0].bytesperline *
+					      vcap->format.height;
 
 	/* Fill the vimc_ent_device struct */
 	vcap->ved.ent = &vcap->vdev.entity;
diff --git a/drivers/media/test-drivers/vimc/vimc-common.c b/drivers/media/test-drivers/vimc/vimc-common.c
index 7b27153c0728b..854f88be3e93e 100644
--- a/drivers/media/test-drivers/vimc/vimc-common.c
+++ b/drivers/media/test-drivers/vimc/vimc-common.c
@@ -236,7 +236,7 @@ const struct vimc_pix_map *vimc_pix_map_by_pixelformat(u32 pixelformat)
 }
 
 static int vimc_get_pix_format(struct media_pad *pad,
-			       struct v4l2_pix_format *fmt)
+			       struct v4l2_ext_pix_format *fmt)
 {
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
 		struct v4l2_subdev *sd =
@@ -252,7 +252,7 @@ static int vimc_get_pix_format(struct media_pad *pad,
 		if (ret)
 			return ret;
 
-		v4l2_fill_pix_format(fmt, &sd_fmt.format);
+		v4l2_fill_ext_pix_format(fmt, &sd_fmt.format);
 		pix_map = vimc_pix_map_by_code(sd_fmt.format.code);
 		fmt->pixelformat = pix_map->pixelformat;
 	} else if (is_media_entity_v4l2_video_device(pad->entity)) {
@@ -274,7 +274,7 @@ static int vimc_get_pix_format(struct media_pad *pad,
 
 int vimc_vdev_link_validate(struct media_link *link)
 {
-	struct v4l2_pix_format source_fmt, sink_fmt;
+	struct v4l2_ext_pix_format source_fmt, sink_fmt;
 	int ret;
 
 	ret = vimc_get_pix_format(link->source, &source_fmt);
diff --git a/drivers/media/test-drivers/vimc/vimc-common.h b/drivers/media/test-drivers/vimc/vimc-common.h
index a289434e75ba0..42fa60350d87b 100644
--- a/drivers/media/test-drivers/vimc/vimc-common.h
+++ b/drivers/media/test-drivers/vimc/vimc-common.h
@@ -104,7 +104,7 @@ struct vimc_ent_device {
 	void * (*process_frame)(struct vimc_ent_device *ved,
 				const void *frame);
 	void (*vdev_get_format)(struct vimc_ent_device *ved,
-			      struct v4l2_pix_format *fmt);
+				struct v4l2_ext_pix_format *fmt);
 };
 
 /**
-- 
2.28.0.rc2


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v5 7/7] media: docs: add documentation for the Extended API
  2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (5 preceding siblings ...)
  2020-08-04 19:29 ` [PATCH v5 6/7] media: vimc: Implement the ext_fmt and ext_buf hooks Helen Koike
@ 2020-08-04 19:29 ` Helen Koike
  2020-08-14  7:49   ` Alexandre Courbot
                     ` (2 more replies)
  2020-08-04 19:34 ` [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (2 subsequent siblings)
  9 siblings, 3 replies; 51+ messages in thread
From: Helen Koike @ 2020-08-04 19:29 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: Helen Koike, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	boris.brezillon, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Add documentation and update references in current documentation for the
Extended API.

Signed-off-by: Helen Koike <helen.koike@collabora.com>
---
Changes in v5:
- new patch

 .../userspace-api/media/v4l/buffer.rst        |   5 +
 .../userspace-api/media/v4l/common.rst        |   1 +
 .../userspace-api/media/v4l/dev-capture.rst   |   5 +
 .../userspace-api/media/v4l/dev-output.rst    |   5 +
 .../userspace-api/media/v4l/ext-api.rst       | 107 +++++++++
 .../userspace-api/media/v4l/format.rst        |  16 +-
 .../userspace-api/media/v4l/user-func.rst     |   5 +
 .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++++++++
 .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++++++
 .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++++++++++++++++
 .../media/v4l/vidioc-ext-querybuf.rst         |  79 +++++++
 .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 ++++++++++
 12 files changed, 697 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst

diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
index 57e752aaf414a..c832bedd64e4c 100644
--- a/Documentation/userspace-api/media/v4l/buffer.rst
+++ b/Documentation/userspace-api/media/v4l/buffer.rst
@@ -27,6 +27,11 @@ such as pointers and sizes for each plane, are stored in
 struct :c:type:`v4l2_plane` instead. In that case,
 struct :c:type:`v4l2_buffer` contains an array of plane structures.
 
+.. note::
+
+    The :ref:`ext_api` version can also be used, and it is
+    preferable when applicable.
+
 Dequeued video buffers come with timestamps. The driver decides at which
 part of the frame and with which clock the timestamp is taken. Please
 see flags in the masks ``V4L2_BUF_FLAG_TIMESTAMP_MASK`` and
diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst
index 7d81c58a13cd7..3430e0bdad667 100644
--- a/Documentation/userspace-api/media/v4l/common.rst
+++ b/Documentation/userspace-api/media/v4l/common.rst
@@ -59,6 +59,7 @@ applicable to all devices.
     ext-ctrls-detect
     fourcc
     format
+    ext-api
     planar-apis
     selection-api
     crop
diff --git a/Documentation/userspace-api/media/v4l/dev-capture.rst b/Documentation/userspace-api/media/v4l/dev-capture.rst
index 44d3094093ab6..5077639787d92 100644
--- a/Documentation/userspace-api/media/v4l/dev-capture.rst
+++ b/Documentation/userspace-api/media/v4l/dev-capture.rst
@@ -102,6 +102,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
 requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
 :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
 
+.. note::
+
+    The :ref:`ext_api` version can also be used, and it is
+    preferable when applicable.
+
 
 Reading Images
 ==============
diff --git a/Documentation/userspace-api/media/v4l/dev-output.rst b/Documentation/userspace-api/media/v4l/dev-output.rst
index e4f2a1d8b0fc7..f8f40c708e49f 100644
--- a/Documentation/userspace-api/media/v4l/dev-output.rst
+++ b/Documentation/userspace-api/media/v4l/dev-output.rst
@@ -99,6 +99,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
 requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
 :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
 
+.. note::
+
+    The :ref:`ext_api` version can also be used, and it is
+    preferable when applicable.
+
 
 Writing Images
 ==============
diff --git a/Documentation/userspace-api/media/v4l/ext-api.rst b/Documentation/userspace-api/media/v4l/ext-api.rst
new file mode 100644
index 0000000000000..da2a82960d22f
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/ext-api.rst
@@ -0,0 +1,107 @@
+.. Permission is granted to copy, distribute and/or modify this
+.. document under the terms of the GNU Free Documentation License,
+.. Version 1.1 or any later version published by the Free Software
+.. Foundation, with no Invariant Sections, no Front-Cover Texts
+.. and no Back-Cover Texts. A copy of the license is included at
+.. Documentation/userspace-api/media/fdl-appendix.rst.
+..
+.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
+
+.. _ext_api:
+
+*************
+Extendend API
+*************
+
+Introduction
+============
+
+The Extended Format API was conceived to extend V4L2 capabilities and
+to simplify certain mechanisms.
+It unifies single- vs multi- planar handling,
+brings the concept of pixelformat + modifiers, allows planes to be placed
+in any offset inside a buffer and let userspace to change colorspace
+attributes if supported by the driver.
+
+Data format negotiation and buffer handling works very similar to the classical
+version, thus in this document we'll just mention the main differences.
+
+Data Format Negotiation
+=======================
+
+The API replaces the classical ioctls:
+
+:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>`, :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`,
+:ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>`
+(with :c:type:`v4l2_format` as the main parameter).
+
+By the extended versions:
+
+:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
+:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
+:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
+(with :c:type:`v4l2_ext_pix_format` as the main parameter).
+
+For CAPTURE and OUTPUT queues only.
+
+The ``type`` field of struct :c:type:`v4l2_ext_pix_format` only accepts 
+``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or ``V4L2_BUF_TYPE_VIDEO_OUTPUT``, and it
+supports multiplanar format through a combination of ``pixelfomat`` and
+``modifier`` fields.
+
+Only the single-planar variants of the pixel formats are valid in the
+``pixelformat`` field.
+To support multi-planar, a modifier should be set in the ``modifier`` field to
+provide more information regarding the memory layout of pixels and number of
+planes.
+
+The ``plane_fmt`` field is an array which holds information by plane using
+the :c:type:`v4l2_plane_ext_pix_format`. When this struct is filled, its
+``sizeimage`` field should be non-zero for all valid planes for a given
+``pixelformat`` + ``modifier`` combination, and zeroed for the invalid ones.
+
+Colospace attributes are not read-only as in the classical version, i.e, they
+can be set by application and drivers will adjust accordingly depending on what
+is supported by the underlying hardware.
+
+Buffers
+=======
+
+The API replaces the classical ioctls:
+
+:ref:`VIDIOC_CREATE_BUFS <VIDIOC_CREATE_BUFS>`
+(with :c:type:`v4l2_create_buffers` as the main parameter),
+:ref:`VIDIOC_QUERYBUF <VIDIOC_QUERYBUF>`, :ref:`VIDIOC_QBUF <VIDIOC_QBUF>`,
+:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`, :ref:`VIDIOC_PREPARE_BUF <VIDIOC_PREPARE_BUF>`
+(with :c:type:`v4l2_buffer` as the main parameter)
+
+By the extended versions:
+
+:ref:`VIDIOC_EXT_CREATE_BUFS <VIDIOC_EXT_CREATE_BUFS>`
+(with :c:type:`v4l2_ext_create_buffers` as the main parameter),
+:ref:`VIDIOC_EXT_QUERYBUF <VIDIOC_EXT_QUERYBUF>`,
+:ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>`,
+:ref:`VIDIOC_EXT_DQBUF <VIDIOC_EXT_QBUF>`,
+:ref:`VIDIOC_EXT_PREPARE_BUF <VIDIOC_EXT_PREPARE_BUF>`
+(with :c:type:`v4l2_ext_buffer` as the main parameter)
+
+The basic difference between :c:type:`v4l2_create_buffers` and
+:c:type:`v4l2_ext_create_buffers` is that the later have a
+:c:type:`v4l2_ext_pix_format` as the type of the ``format`` field.
+
+Comparing :c:type:`v4l2_ext_buffer` with :c:type:`v4l2_buffer`, in the
+extended version there is a unification of the single-/multi- planar handling
+through the ``planes`` field of type :c:type:`v4l2_ext_plane`.
+
+The :c:type:`v4l2_ext_plane also allows planes to be placed in a given offset
+inside a buffer.
+Planes can be placed in different locations inside the same buffer, or in
+different buffers.
+
+
+.. kernel-doc:: include/uapi/linux/videodev2.h
+   :functions: v4l2_ext_plane
+
+
+.. kernel-doc:: include/uapi/linux/videodev2.h
+   :functions: v4l2_ext_buffer
diff --git a/Documentation/userspace-api/media/v4l/format.rst b/Documentation/userspace-api/media/v4l/format.rst
index e47fc0505727c..b96d26f26793c 100644
--- a/Documentation/userspace-api/media/v4l/format.rst
+++ b/Documentation/userspace-api/media/v4l/format.rst
@@ -28,13 +28,19 @@ format and the driver selects and reports the best the hardware can do
 to satisfy the request. Of course applications can also just query the
 current selection.
 
-A single mechanism exists to negotiate all data formats using the
-aggregate struct :c:type:`v4l2_format` and the
+There are two mechanism to negociate data formats:
+
+The first one is using the aggregate struct :c:type:`v4l2_format` and the
 :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and
 :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctls. Additionally the
 :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` ioctl can be used to examine
 what the hardware *could* do, without actually selecting a new data
-format. The data formats supported by the V4L2 API are covered in the
+format.
+
+The second is through the :ref:`ext_api`, please refer to its documentation
+for more information.
+
+The data formats supported by the V4L2 API are covered in the
 respective device section in :ref:`devices`. For a closer look at
 image formats see :ref:`pixfmt`.
 
@@ -71,7 +77,9 @@ earlier versions of V4L2. Switching the logical stream or returning into
 *may* support a switch using :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`.
 
 All drivers exchanging data with applications must support the
-:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl. Implementation of the
+:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl
+or the Extended respective versions (TODO: link).
+Implementation of the
 :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is highly recommended but optional.
 
 
diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index bf77c842718e5..67b254812791c 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -20,6 +20,7 @@ Function Reference
     func-close
     func-ioctl
     vidioc-create-bufs
+    vidioc-ext-create-bufs
     vidioc-cropcap
     vidioc-dbg-g-chip-info
     vidioc-dbg-g-register
@@ -48,6 +49,7 @@ Function Reference
     vidioc-g-ext-ctrls
     vidioc-g-fbuf
     vidioc-g-fmt
+    vidioc-g-ext-pix-fmt
     vidioc-g-frequency
     vidioc-g-input
     vidioc-g-jpegcomp
@@ -62,8 +64,11 @@ Function Reference
     vidioc-log-status
     vidioc-overlay
     vidioc-prepare-buf
+    vidioc-ext-prepare-buf
     vidioc-qbuf
+    vidioc-ext-qbuf
     vidioc-querybuf
+    vidioc-ext-querybuf
     vidioc-querycap
     vidioc-queryctrl
     vidioc-query-dv-timings
diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
new file mode 100644
index 0000000000000..67f2c541e4d79
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
@@ -0,0 +1,95 @@
+.. Permission is granted to copy, distribute and/or modify this
+.. document under the terms of the GNU Free Documentation License,
+.. Version 1.1 or any later version published by the Free Software
+.. Foundation, with no Invariant Sections, no Front-Cover Texts
+.. and no Back-Cover Texts. A copy of the license is included at
+.. Documentation/userspace-api/media/fdl-appendix.rst.
+..
+.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
+
+.. _VIDIOC_EXT_CREATE_BUFS:
+
+****************************
+ioctl VIDIOC_EXT_CREATE_BUFS
+****************************
+
+Name
+====
+
+VIDIOC_EXT_CREATE_BUFS - Create buffers for Memory Mapped or User Pointer or DMA Buffer I/O
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp )
+    :name: VIDIOC_EXT_CREATE_BUFS
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_ext_create_buffers`.
+
+
+Description
+===========
+
+This ioctl is used to create buffers for :ref:`memory mapped <mmap>`
+or :ref:`user pointer <userp>` or :ref:`DMA buffer <dmabuf>` I/O.
+This ioctl can be called multiple times to
+create buffers of different sizes.
+
+To allocate the device buffers applications must initialize the relevant
+fields of the struct :c:type:`v4l2_ext_create_buffers` structure. The
+``count`` field must be set to the number of requested buffers, the
+``memory`` field specifies the requested I/O method and the ``reserved``
+array must be zeroed.
+
+The ``format`` field specifies the image format that the buffers must be
+able to handle. The application has to fill in this struct
+:c:type:`v4l2_format`. Usually this will be done using the
+:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` or
+:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctls to ensure that the
+requested format is supported by the driver.
+The driver may return an error if the size(s) are not supported by the
+hardware (usually because they are too small).
+
+The driver can create a memory buffer per plane, or a single memory buffer
+containing all the planes, with a minimum size according to the size
+defined by the ``format.plane_fmt[i].sizeimage`` field.
+Usually if the ``format.plane_fmt[i].sizeimage``
+field is less than the minimum required for the given format, then an
+error will be returned since drivers will typically not allow this. If
+it is larger, then the value will be used as-is. In other words, the
+driver may reject the requested size, but if it is accepted the driver
+will use it unchanged.
+
+When the ioctl is called with a pointer to this structure the driver
+will attempt to allocate up to the requested number of buffers and store
+the actual number allocated and the starting index in the ``count`` and
+the ``index`` fields respectively. On return ``count`` can be smaller
+than the number requested.
+
+
+.. kernel-doc:: include/uapi/linux/videodev2.h
+   :functions: v4l2_ext_create_buffers
+
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+ENOMEM
+    No memory to allocate buffers for :ref:`memory mapped <mmap>` I/O.
+
+EINVAL
+    The buffer type (``format.type`` field), requested I/O method
+    (``memory``) or format (``format`` field) is not valid.
diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
new file mode 100644
index 0000000000000..88e4b57121254
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
@@ -0,0 +1,62 @@
+.. Permission is granted to copy, distribute and/or modify this
+.. document under the terms of the GNU Free Documentation License,
+.. Version 1.1 or any later version published by the Free Software
+.. Foundation, with no Invariant Sections, no Front-Cover Texts
+.. and no Back-Cover Texts. A copy of the license is included at
+.. Documentation/userspace-api/media/fdl-appendix.rst.
+..
+.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
+
+.. _VIDIOC_EXT_PREPARE_BUF:
+
+****************************
+ioctl VIDIOC_EXT_PREPARE_BUF
+****************************
+
+Name
+====
+
+VIDIOC_EXT_PREPARE_BUF - Prepare a buffer for I/O
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp )
+    :name: VIDIOC_EXT_PREPARE_BUF
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_ext_buffer`.
+
+
+Description
+===========
+
+Applications can optionally call the :ref:`VIDIOC_EXT_PREPARE_BUF` ioctl to
+pass ownership of the buffer to the driver before actually enqueuing it,
+using the :ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>` ioctl, and to prepare it for future I/O. Such
+preparations may include cache invalidation or cleaning. Performing them
+in advance saves time during the actual I/O.
+
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+EBUSY
+    File I/O is in progress.
+
+EINVAL
+    The buffer ``type`` is not supported, or the ``index`` is out of
+    bounds, or no buffers have been allocated yet, or the ``userptr`` or
+    ``length`` are invalid.
diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
new file mode 100644
index 0000000000000..083e106cf6bad
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
@@ -0,0 +1,204 @@
+.. Permission is granted to copy, distribute and/or modify this
+.. document under the terms of the GNU Free Documentation License,
+.. Version 1.1 or any later version published by the Free Software
+.. Foundation, with no Invariant Sections, no Front-Cover Texts
+.. and no Back-Cover Texts. A copy of the license is included at
+.. Documentation/userspace-api/media/fdl-appendix.rst.
+..
+.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
+
+.. _VIDIOC_EXT_QBUF:
+
+***************************************
+ioctl VIDIOC_EXT_QBUF, VIDIOC_EXT_DQBUF
+***************************************
+
+Name
+====
+
+VIDIOC_EXT_QBUF - VIDIOC_EXT_DQBUF - Exchange a buffer with the driver
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
+    :name: VIDIOC_EXT_QBUF
+
+.. c:function:: int ioctl( int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp )
+    :name: VIDIOC_EXT_DQBUF
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_ext_buffer`.
+
+
+Description
+===========
+
+Applications call the ``VIDIOC_EXT_QBUF`` ioctl to enqueue an empty
+(capturing) or filled (output) buffer in the driver's incoming queue.
+The semantics depend on the selected I/O method.
+
+To enqueue a buffer applications set the ``type`` field of a struct
+:c:type:`v4l2_ext_buffer` to the same buffer type as was
+previously used with struct :c:type:`v4l2_ext_pix_format` ``type``.
+Applications must also set the ``index`` field. Valid index numbers
+range from zero to the number of buffers allocated with
+:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
+:c:type:`v4l2_ext_create_buffers` ``count``) minus
+one. The contents of the struct :c:type:`v4l2_ext_buffer` returned
+by a :ref:`VIDIOC_EXT_QUERYBUF` ioctl will do as well.
+When the buffer is intended for output (``type`` is
+``V4L2_BUF_TYPE_VIDEO_OUTPUT``) applications must also initialize the
+``timestamp`` fields and the ``planes[i].plane_length`` for each valid plane,
+and invalid ones must be set as zero.
+see :ref:`buffer` for details. Applications must also set ``flags`` to 0. The
+``reserved`` field must be set to 0.
+
+To enqueue a :ref:`memory mapped <mmap>` buffer applications set the
+``planes[i].memory`` field to ``V4L2_MEMORY_MMAP`` in all the valid planes,
+and invalid ones must be set as zero.
+When ``VIDIOC_EXT_QBUF`` is called
+with a pointer to this structure the driver sets the
+``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_QUEUED`` flags and clears
+the ``V4L2_BUF_FLAG_DONE`` flag in the ``flags`` field, or it returns an
+``EINVAL`` error code.
+
+To enqueue a :ref:`user pointer <userp>` buffer applications set the
+``planes[i].memory`` field to ``V4L2_MEMORY_USERPTR`` in all the valid planes,
+and invalid ones must be set as zero, the ``planes[i].m.userptr`` field to the
+address of the buffer,``planes[i].buffer_length`` to the size of the memory
+buffer, ``planes[i].plane_length`` to the size that should be used by the plane,
+and ``planes[i].offset`` of the plane in the memory buffer.
+
+When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure
+the driver sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
+``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
+``flags`` field, or it returns an error code. This ioctl locks the
+memory pages of the buffer in physical memory, they cannot be swapped
+out to disk. Buffers remain locked until dequeued, until the
+:ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
+:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
+device is closed.
+
+To enqueue a :ref:`DMABUF <dmabuf>` buffer applications set the
+``planes[i].memory`` field to ``V4L2_MEMORY_DMABUF`` in all the valid planes,
+and invalid ones must be set as zero, the ``planes[i].m.fd`` field to a
+file descriptor associated with a DMABUF buffer.,``planes[i].buffer_length`` to
+the size of the memory buffer, ``planes[i].plane_length`` to the size that
+should be used by the plane, and ``planes[i].offset`` of the plane in the memory buffer.
+When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure the driver
+sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
+``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
+``flags`` field, or it returns an error code. This ioctl locks the
+buffer. Locking a buffer means passing it to a driver for a hardware
+access (usually DMA). If an application accesses (reads/writes) a locked
+buffer then the result is undefined. Buffers remain locked until
+dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
+:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
+device is closed.
+
+The ``request_fd`` field can be used with the ``VIDIOC_EXT_QBUF`` ioctl to specify
+the file descriptor of a :ref:`request <media-request-api>`, if requests are
+in use. Setting it means that the buffer will not be passed to the driver
+until the request itself is queued. Also, the driver will apply any
+settings associated with the request for this buffer. This field will
+be ignored unless the ``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
+If the device does not support requests, then ``EBADR`` will be returned.
+If requests are supported but an invalid request file descriptor is given,
+then ``EINVAL`` will be returned.
+
+.. caution::
+   It is not allowed to mix queuing requests with queuing buffers directly.
+   ``EBUSY`` will be returned if the first buffer was queued directly and
+   then the application tries to queue a request, or vice versa. After
+   closing the file descriptor, calling
+   :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or calling :ref:`VIDIOC_EXT_CREATE_BUFS`
+   the check for this will be reset.
+
+   For :ref:`memory-to-memory devices <mem2mem>` you can specify the
+   ``request_fd`` only for output buffers, not for capture buffers. Attempting
+   to specify this for a capture buffer will result in an ``EBADR`` error.
+
+Applications call the ``VIDIOC_EXT_DQBUF`` ioctl to dequeue a filled
+(capturing) or displayed (output) buffer from the driver's outgoing
+queue. They just set the ``type``, ``planes[i].memory`` and ``reserved`` fields of
+a struct :c:type:`v4l2_ext_buffer` as above, when
+``VIDIOC_EXT_DQBUF`` is called with a pointer to this structure the driver
+fills the remaining fields or returns an error code. The driver may also
+set ``V4L2_BUF_FLAG_ERROR`` in the ``flags`` field. It indicates a
+non-critical (recoverable) streaming error. In such case the application
+may continue as normal, but should be aware that data in the dequeued
+buffer might be corrupted. When using the multi-planar API, the planes
+array must be passed in as well.
+
+If the application sets the ``memory`` field to ``V4L2_MEMORY_DMABUF`` to
+dequeue a :ref:`DMABUF <dmabuf>` buffer, the driver fills the ``m.fd`` field
+with a file descriptor numerically the same as the one given to ``VIDIOC_EXT_QBUF``
+when the buffer was enqueued. No new file descriptor is created at dequeue time
+and the value is only for the application convenience.
+
+By default ``VIDIOC_EXT_DQBUF`` blocks when no buffer is in the outgoing
+queue. When the ``O_NONBLOCK`` flag was given to the
+:ref:`open() <func-open>` function, ``VIDIOC_EXT_DQBUF`` returns
+immediately with an ``EAGAIN`` error code when no buffer is available.
+
+
+.. kernel-doc:: include/uapi/linux/videodev2.h
+   :functions: v4l2_ext_buffers
+
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+EAGAIN
+    Non-blocking I/O has been selected using ``O_NONBLOCK`` and no
+    buffer was in the outgoing queue.
+
+EINVAL
+    The buffer ``type`` is not supported, or the ``index`` is out of
+    bounds, or no buffers have been allocated yet, or the ``userptr`` or
+    ``length`` are invalid, or the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was
+    set but the the given ``request_fd`` was invalid, or ``m.fd`` was
+    an invalid DMABUF file descriptor.
+
+EIO
+    ``VIDIOC_EXT_DQBUF`` failed due to an internal error. Can also indicate
+    temporary problems like signal loss.
+
+    .. note::
+
+       The driver might dequeue an (empty) buffer despite returning
+       an error, or even stop capturing. Reusing such buffer may be unsafe
+       though and its details (e.g. ``index``) may not be returned either.
+       It is recommended that drivers indicate recoverable errors by setting
+       the ``V4L2_BUF_FLAG_ERROR`` and returning 0 instead. In that case the
+       application should be able to safely reuse the buffer and continue
+       streaming.
+
+EPIPE
+    ``VIDIOC_EXT_DQBUF`` returns this on an empty capture queue for mem2mem
+    codecs if a buffer with the ``V4L2_BUF_FLAG_LAST`` was already
+    dequeued and no new buffers are expected to become available.
+
+EBADR
+    The ``V4L2_BUF_FLAG_REQUEST_FD`` flag was set but the device does not
+    support requests for the given buffer type, or
+    the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was not set but the device requires
+    that the buffer is part of a request.
+
+EBUSY
+    The first buffer was queued via a request, but the application now tries
+    to queue it directly, or vice versa (it is not permitted to mix the two
+    APIs).
diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
new file mode 100644
index 0000000000000..f2a12017253f6
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
@@ -0,0 +1,79 @@
+.. Permission is granted to copy, distribute and/or modify this
+.. document under the terms of the GNU Free Documentation License,
+.. Version 1.1 or any later version published by the Free Software
+.. Foundation, with no Invariant Sections, no Front-Cover Texts
+.. and no Back-Cover Texts. A copy of the license is included at
+.. Documentation/userspace-api/media/fdl-appendix.rst.
+..
+.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
+
+.. _VIDIOC_EXT_QUERYBUF:
+
+*************************
+ioctl VIDIOC_EXT_QUERYBUF
+*************************
+
+Name
+====
+
+VIDIOC_EXT_QUERYBUF - Query the status of a buffer
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp )
+    :name: VIDIOC_EXT_QUERYBUF
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_ext_buffer`.
+
+
+Description
+===========
+
+This ioctl is part of the :ref:`streaming <mmap>` I/O method. It can
+be used to query the status of a buffer at any time after buffers have
+been allocated with the :ref:`VIDIOC_EXT_CREATE_BUFS` ioctl.
+
+Applications set the ``type`` field of a struct
+:c:type:`v4l2_ext_buffer` to the same buffer type as was
+previously used with struct :c:type:`v4l2_ext_pix_format` ``type``,
+and the ``index`` field. Valid index numbers range from zero to the
+number of buffers allocated with
+:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
+:c:type:`v4l2_ext_create_buffers` ``count``) minus
+one. The ``reserved`` field must be set to 0.
+
+In the ``flags`` field the ``V4L2_BUF_FLAG_MAPPED``,
+``V4L2_BUF_FLAG_PREPARED``, ``V4L2_BUF_FLAG_QUEUED`` and
+``V4L2_BUF_FLAG_DONE`` flags will be valid. The ``planes.memory`` fields will be
+set to the current I/O method for each plane.
+
+For every valid plane, an entry in ``planes`` will be filled, and zeroed for
+invalid ones. ``planes[i].buffer_length`` is the size of the memory buffer
+which contains the plane, ``planes[i].plane_length`` is the length of the plane,
+and ``planes[i].offset` is where the plane is placed in the memory buffer.
+
+The size of the ``planes`` array can be calculated by the number of sequential
+planes with ``planes[i].buffer_length`` that differs from zero up to the max
+size of the array.
+
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+EINVAL
+    The buffer ``type`` is not supported, or the ``index`` is out of
+    bounds.
diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
new file mode 100644
index 0000000000000..008e6c98a88a5
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
@@ -0,0 +1,117 @@
+.. Permission is granted to copy, distribute and/or modify this
+.. document under the terms of the GNU Free Documentation License,
+.. Version 1.1 or any later version published by the Free Software
+.. Foundation, with no Invariant Sections, no Front-Cover Texts
+.. and no Back-Cover Texts. A copy of the license is included at
+.. Documentation/userspace-api/media/fdl-appendix.rst.
+..
+.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
+
+.. _VIDIOC_G_EXT_PIX_FMT:
+
+************************************************************************
+ioctl VIDIOC_G_EXT_PIX_FMT, VIDIOC_S_EXT_PIX_FMT, VIDIOC_TRY_EXT_PIX_FMT
+************************************************************************
+
+Name
+====
+
+VIDIOC_G_EXT_PIX_FMT - VIDIOC_S_EXT_PIX_FMT - VIDIOC_TRY_EXT_PIX_FMT - Get or set the data format, try a format
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
+    :name: VIDIOC_G_EXT_PIX_FMT
+
+.. c:function:: int ioctl( int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
+    :name: VIDIOC_S_EXT_PIX_FMT
+
+.. c:function:: int ioctl( int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
+    :name: VIDIOC_TRY_EXT_PIX_FMT
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_ext_pix_format`.
+
+
+Description
+===========
+
+These ioctls are used to negotiate the format of data (typically image
+format) exchanged between driver and application.
+
+To query the current parameters applications set the ``type`` field of a
+struct :c:type:`v4l2_ext_pix_format` to  ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or
+``V4L2_BUF_TYPE_VIDEO_OUTPUT``, all the other types are invalid in this API,
+and multiplanar is supported through modifiers.
+
+When the application calls the
+:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with a pointer to this
+structure the driver fills the other members.
+When the requested buffer type is not supported drivers return
+an ``EINVAL`` error code.
+
+To change the current format parameters applications initialize all
+the fields in the struct.
+For details see the documentation of the various devices types in
+:ref:`devices`. Good practice is to query the current parameters
+first, and to modify only those parameters not suitable for the
+application. When the application calls the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with
+a pointer to a struct :c:type:`v4l2_ext_pix_format` structure the driver
+checks and adjusts the parameters against hardware abilities. Drivers
+should not return an error code unless the ``type`` field is invalid,
+this is a mechanism to fathom device capabilities and to approach
+parameters acceptable for both the application and driver. On success
+the driver may program the hardware, allocate resources and generally
+prepare for data exchange. Finally the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl returns
+the current format parameters as :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` does. Very simple,
+inflexible devices may even ignore all input and always return the
+default parameters. However all V4L2 devices exchanging data with the
+application must implement the :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` and :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
+ioctl. When the requested buffer type is not supported drivers return an
+EINVAL error code on a :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` attempt. When I/O is already in
+progress or the resource is not available for other reasons drivers
+return the ``EBUSY`` error code.
+
+The :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl is equivalent to :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` with one
+exception: it does not change driver state. It can also be called at any
+time, never returning ``EBUSY``. This function is provided to negotiate
+parameters, to learn about hardware limitations, without disabling I/O
+or possibly time consuming hardware preparations. Although strongly
+recommended drivers are not required to implement this ioctl.
+
+The format as returned by :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` must be identical to what
+:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` returns for the same input or output.
+
+
+.. kernel-doc:: include/uapi/linux/videodev2.h
+   :functions: v4l2_plane_ext_pix_format
+
+
+.. kernel-doc:: include/uapi/linux/videodev2.h
+   :functions: v4l2_ext_pix_format
+
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+EINVAL
+    The struct :c:type:`v4l2_ext_pix_format` ``type`` field is
+    invalid or the requested buffer type not supported.
+
+EBUSY
+    The device is busy and cannot change the format. This could be
+    because or the device is streaming or buffers are allocated or
+    queued to the driver. Relevant for :ref:`VIDIOC_S_EXT_PIX_FMT
+    <VIDIOC_G_EXT_PIX_FMT>` only.
-- 
2.28.0.rc2


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls
  2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (6 preceding siblings ...)
  2020-08-04 19:29 ` [PATCH v5 7/7] media: docs: add documentation for the Extended API Helen Koike
@ 2020-08-04 19:34 ` Helen Koike
  2020-08-14  7:49 ` Alexandre Courbot
  2020-11-27 15:06 ` Helen Koike
  9 siblings, 0 replies; 51+ messages in thread
From: Helen Koike @ 2020-08-04 19:34 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: tfiga, hiroh, nicolas, Brian.Starkey, kernel, boris.brezillon,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov



On 8/4/20 4:29 PM, Helen Koike wrote:
> Hello,
> 
> This is v5 of the Extended API for formats and buffers, which introduces
> the following new ioctls:
> 
> int ioctl(int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
> int ioctl(int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
> int ioctl(int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
> 
> int ioctl(int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp)
> int ioctl(int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp)
> int ioctl(int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp)
> int ioctl(int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp)
> int ioctl(int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp)
> 
> Please check v4 cover letter specific topic past discussions
> https://patchwork.linuxtv.org/project/linux-media/cover/20200717115435.2632623-1-helen.koike@collabora.com/
> 
> Documentation
> =============
> I added a first version of the documentation.
> I would appreciate some tips in how to organize this better, since there are
> several parts of the docs which reference the "old" API, and for now
> I just added a note pointing to the Extended API.
> 
> Instead of duplicating documentation from the code to the Docs (as used by
> most part of v4l2 documentation), I just added a reference to let Sphinx get
> the structs documentation from the code, so we can avoid duplicating.
> 
> For reviewing convenience, I uploaded the generated html docs at
> https://people.collabora.com/~koike/ext-doc-v5/userspace-api/media/v4l/ext-api.html
> 
> There is room for improvements, it would be great to get your suggestions.
> 
> uAPI
> ====
> This version have some minor changes in the uAPI structs, highlight to the
> mem_offset that was returned to struct v4l2_ext_plane, memory field that now
> is per plane, and some adjustments in field sizes and re-ordering to make
> structs the same for 32 and 64 bits (which I verified with pahole tool).
> 
> I also updated v4l2-compliance:
> https://gitlab.collabora.com/koike/v4l-utils/-/tree/ext-api/wip
> 
> We need to decide the size of the reserved fields left, how much reserved
> fields do you think we should leave for each struct?
> 
> What is next
> ============
> I would like to improve implementation (for the kernel and v4l2-compliane) and
> drop the RFC tag for next version, so please review the uAPI.

Ops, sorry, I forgot to add RFC tag when submitting. Please consider RFC for this one.

Regards,
Helen

> 
> 
> List of changes in v5:
> ======================
> * first version of Documentation
> * migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> * return mem_offset to struct v4l2_ext_plane
> * change sizes and reorder fields to avoid holes in the struct and make it
>   the same for 32 and 64 bits
> * removed __attribute__ ((packed)) from uapi structs
> * set request_fd to signed
> * add documentation
> * updated some commit messages
> 
> Hans Verkuil (1):
>   media: v4l2: Add extended buffer operations
> 
> Helen Koike (6):
>   media: v4l2: Extend pixel formats to unify single/multi-planar
>     handling (and more)
>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>     _ext_buf hooks
>   media: mediabus: Add helpers to convert a ext_pix format to/from a
>     mbus_fmt
>   media: vivid: Convert the capture and output drivers to
>     EXT_FMT/EXT_BUF
>   media: vimc: Implement the ext_fmt and ext_buf hooks
>   media: docs: add documentation for the Extended API
> 
>  .../userspace-api/media/v4l/buffer.rst        |   5 +
>  .../userspace-api/media/v4l/common.rst        |   1 +
>  .../userspace-api/media/v4l/dev-capture.rst   |   5 +
>  .../userspace-api/media/v4l/dev-output.rst    |   5 +
>  .../userspace-api/media/v4l/ext-api.rst       | 107 ++
>  .../userspace-api/media/v4l/format.rst        |  16 +-
>  .../userspace-api/media/v4l/user-func.rst     |   5 +
>  .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++
>  .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++
>  .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++
>  .../media/v4l/vidioc-ext-querybuf.rst         |  79 ++
>  .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 +++
>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>  .../media/common/videobuf2/videobuf2-v4l2.c   | 560 ++++++-----
>  .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
>  drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
>  drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
>  drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
>  .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
>  .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
>  .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
>  .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
>  .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
>  .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
>  drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 938 ++++++++++++++++--
>  include/media/v4l2-ioctl.h                    |  60 ++
>  include/media/v4l2-mediabus.h                 |  42 +
>  include/media/videobuf2-core.h                |   6 +-
>  include/media/videobuf2-v4l2.h                |  21 +-
>  include/uapi/linux/videodev2.h                | 146 +++
>  31 files changed, 2363 insertions(+), 723 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls
  2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (7 preceding siblings ...)
  2020-08-04 19:34 ` [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
@ 2020-08-14  7:49 ` Alexandre Courbot
  2020-11-27 15:06 ` Helen Koike
  9 siblings, 0 replies; 51+ messages in thread
From: Alexandre Courbot @ 2020-08-14  7:49 UTC (permalink / raw)
  To: Helen Koike
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Tomasz Figa,
	Hirokazu Honda, Nicolas Dufresne, Brian Starkey, kernel,
	Boris Brezillon, narmstrong, LKML, frkoenig, Maxime Jourdan,
	Stanimir Varbanov

On Wed, Aug 5, 2020 at 4:32 AM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hello,
>
> This is v5 of the Extended API for formats and buffers, which introduces
> the following new ioctls:
>
> int ioctl(int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
> int ioctl(int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
> int ioctl(int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
>
> int ioctl(int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp)
> int ioctl(int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp)
> int ioctl(int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp)
> int ioctl(int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp)
> int ioctl(int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp)
>
> Please check v4 cover letter specific topic past discussions
> https://patchwork.linuxtv.org/project/linux-media/cover/20200717115435.2632623-1-helen.koike@collabora.com/

Thanks for pushing this series! I have tried to find things to say but
only have rather superficial comments, sorry about that. On the other
hand this might mean that the series is mature enough to drop the RFC.

>
> Documentation
> =============
> I added a first version of the documentation.
> I would appreciate some tips in how to organize this better, since there are
> several parts of the docs which reference the "old" API, and for now
> I just added a note pointing to the Extended API.
>
> Instead of duplicating documentation from the code to the Docs (as used by
> most part of v4l2 documentation), I just added a reference to let Sphinx get
> the structs documentation from the code, so we can avoid duplicating.
>
> For reviewing convenience, I uploaded the generated html docs at
> https://people.collabora.com/~koike/ext-doc-v5/userspace-api/media/v4l/ext-api.html
>
> There is room for improvements, it would be great to get your suggestions.
>
> uAPI
> ====
> This version have some minor changes in the uAPI structs, highlight to the
> mem_offset that was returned to struct v4l2_ext_plane, memory field that now
> is per plane, and some adjustments in field sizes and re-ordering to make
> structs the same for 32 and 64 bits (which I verified with pahole tool).
>
> I also updated v4l2-compliance:
> https://gitlab.collabora.com/koike/v4l-utils/-/tree/ext-api/wip
>
> We need to decide the size of the reserved fields left, how much reserved
> fields do you think we should leave for each struct?

Considering that the cost of a reserved field is not much compared to
the pain of not being able to extend the ABI when we need to, I'd say
that too much is preferable over not enough. :) I don't have a precise
number to give though. Formats in particular are structs that could
potentially need to carry new information. For v4l2_buffer I suppose
the request API gives us a way to extend using controls.

>
> What is next
> ============
> I would like to improve implementation (for the kernel and v4l2-compliane) and
> drop the RFC tag for next version, so please review the uAPI.
>
>
> List of changes in v5:
> ======================
> * first version of Documentation
> * migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> * return mem_offset to struct v4l2_ext_plane
> * change sizes and reorder fields to avoid holes in the struct and make it
>   the same for 32 and 64 bits
> * removed __attribute__ ((packed)) from uapi structs
> * set request_fd to signed
> * add documentation
> * updated some commit messages
>
> Hans Verkuil (1):
>   media: v4l2: Add extended buffer operations
>
> Helen Koike (6):
>   media: v4l2: Extend pixel formats to unify single/multi-planar
>     handling (and more)
>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>     _ext_buf hooks
>   media: mediabus: Add helpers to convert a ext_pix format to/from a
>     mbus_fmt
>   media: vivid: Convert the capture and output drivers to
>     EXT_FMT/EXT_BUF
>   media: vimc: Implement the ext_fmt and ext_buf hooks
>   media: docs: add documentation for the Extended API
>
>  .../userspace-api/media/v4l/buffer.rst        |   5 +
>  .../userspace-api/media/v4l/common.rst        |   1 +
>  .../userspace-api/media/v4l/dev-capture.rst   |   5 +
>  .../userspace-api/media/v4l/dev-output.rst    |   5 +
>  .../userspace-api/media/v4l/ext-api.rst       | 107 ++
>  .../userspace-api/media/v4l/format.rst        |  16 +-
>  .../userspace-api/media/v4l/user-func.rst     |   5 +
>  .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++
>  .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++
>  .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++
>  .../media/v4l/vidioc-ext-querybuf.rst         |  79 ++
>  .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 +++
>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>  .../media/common/videobuf2/videobuf2-v4l2.c   | 560 ++++++-----
>  .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
>  drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
>  drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
>  drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
>  .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
>  .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
>  .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
>  .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
>  .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
>  .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
>  drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 938 ++++++++++++++++--
>  include/media/v4l2-ioctl.h                    |  60 ++
>  include/media/v4l2-mediabus.h                 |  42 +
>  include/media/videobuf2-core.h                |   6 +-
>  include/media/videobuf2-v4l2.h                |  21 +-
>  include/uapi/linux/videodev2.h                | 146 +++
>  31 files changed, 2363 insertions(+), 723 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>
> --
> 2.28.0.rc2
>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-08-04 19:29 ` [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
@ 2020-08-14  7:49   ` Alexandre Courbot
  2020-11-19 16:23     ` Helen Koike
  2020-09-09 11:41   ` Hans Verkuil
  2020-10-02 19:49   ` Tomasz Figa
  2 siblings, 1 reply; 51+ messages in thread
From: Alexandre Courbot @ 2020-08-14  7:49 UTC (permalink / raw)
  To: Helen Koike
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Boris Brezillon,
	Tomasz Figa, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, narmstrong, LKML, frkoenig, Maxime Jourdan,
	Stanimir Varbanov

On Wed, Aug 5, 2020 at 4:32 AM Helen Koike <helen.koike@collabora.com> wrote:
>
> This is part of the multiplanar and singleplanar unification process.
> v4l2_ext_pix_format is supposed to work for both cases.
>
> We also add the concept of modifiers already employed in DRM to expose
> HW-specific formats (like tiled or compressed formats) and allow
> exchanging this information with the DRM subsystem in a consistent way.
>
> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
> v4l2_ext_format, other types will be rejected if you use the
> {G,S,TRY}_EXT_PIX_FMT ioctls.
>
> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
> in drivers, but, in the meantime, the core takes care of converting
> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
> still work if the userspace app/lib uses the new ioctls.
> The conversion is also done the other around to allow userspace
> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
> _ext_ hooks.
>
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v5:
> - change sizes and reorder fields to avoid holes in the struct and make
>   it the same for 32 and 64 bits
> - removed __attribute__ ((packed)) from uapi structs
> - Fix doc warning from make htmldocs
> - Updated commit message with EXT_PIX prefix for the ioctls.
>
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Add reserved fields
> - Removed num_planes from struct v4l2_ext_pix_format
> - Removed flag field from struct v4l2_ext_pix_format, since the only
>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>   where we can use modifiers, or add it back later through the reserved
>   bits.
> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>   != MOD_INVALID
> - Fix type assignment in v4l_g_fmt_ext_pix()
> - Rebased on top of media/master (post 5.8-rc1)
>
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
>
> Changes in v2:
> - Move the modifier in v4l2_ext_format (was formerly placed in
>   v4l2_ext_plane)
> - Fix a few bugs in the converters and add a strict parameter to
>   allow conversion of uninitialized/mis-initialized objects
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>  include/media/v4l2-ioctl.h           |  34 ++
>  include/uapi/linux/videodev2.h       |  56 +++
>  4 files changed, 615 insertions(+), 81 deletions(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index a593ea0598b55..e1829906bc086 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
>                         set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
>                 if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
>                                ops->vidioc_g_fmt_vid_cap_mplane ||
> +                              ops->vidioc_g_ext_pix_fmt_vid_cap ||
>                                ops->vidioc_g_fmt_vid_overlay)) ||
>                     (is_tx && (ops->vidioc_g_fmt_vid_out ||
>                                ops->vidioc_g_fmt_vid_out_mplane ||
> -                              ops->vidioc_g_fmt_vid_out_overlay)))
> +                              ops->vidioc_g_ext_pix_fmt_vid_out ||
> +                              ops->vidioc_g_fmt_vid_out_overlay))) {
>                          set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
> +                        set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
> +               }
>                 if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
>                                ops->vidioc_s_fmt_vid_cap_mplane ||
> +                              ops->vidioc_s_ext_pix_fmt_vid_cap ||
>                                ops->vidioc_s_fmt_vid_overlay)) ||
>                     (is_tx && (ops->vidioc_s_fmt_vid_out ||
>                                ops->vidioc_s_fmt_vid_out_mplane ||
> -                              ops->vidioc_s_fmt_vid_out_overlay)))
> +                              ops->vidioc_s_ext_pix_fmt_vid_out ||
> +                              ops->vidioc_s_fmt_vid_out_overlay))) {
>                          set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
> +                        set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
> +               }
>                 if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
>                                ops->vidioc_try_fmt_vid_cap_mplane ||
> +                              ops->vidioc_try_ext_pix_fmt_vid_cap ||
>                                ops->vidioc_try_fmt_vid_overlay)) ||
>                     (is_tx && (ops->vidioc_try_fmt_vid_out ||
>                                ops->vidioc_try_fmt_vid_out_mplane ||
> -                              ops->vidioc_try_fmt_vid_out_overlay)))
> +                              ops->vidioc_try_ext_pix_fmt_vid_out ||
> +                              ops->vidioc_try_fmt_vid_out_overlay))) {
>                          set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
> +                        set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
> +               }
>                 SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
>                 SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
>                 SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
> @@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
>                 /* touch specific ioctls */
>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
>                 SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
> +               SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
>                 SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
> +               SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
>                 SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
> +               SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
>                 SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index a556880f225a5..14a0def50f8ea 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -17,6 +17,8 @@
>
>  #include <linux/videodev2.h>
>
> +#include <drm/drm_fourcc.h>
> +
>  #include <media/v4l2-common.h>
>  #include <media/v4l2-ioctl.h>
>  #include <media/v4l2-ctrls.h>
> @@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
>         }
>  }
>
> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
> +{
> +       const struct v4l2_ext_pix_format *pix = arg;
> +       unsigned int i;
> +
> +       pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
> +               prt_names(pix->type, v4l2_type_names),
> +               pix->width, pix->height,
> +               (pix->pixelformat & 0xff),
> +               (pix->pixelformat >>  8) & 0xff,
> +               (pix->pixelformat >> 16) & 0xff,
> +               (pix->pixelformat >> 24) & 0xff,
> +               pix->modifier, prt_names(pix->field, v4l2_field_names),
> +               pix->colorspace, pix->ycbcr_enc,
> +               pix->quantization, pix->xfer_func);
> +       for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
> +               pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
> +                        i, pix->plane_fmt[i].bytesperline,
> +                        pix->plane_fmt[i].sizeimage);
> +}
> +
>  static void v4l_print_framebuffer(const void *arg, bool write_only)
>  {
>         const struct v4l2_framebuffer *p = arg;
> @@ -964,11 +987,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>         switch (type) {
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>                 if ((is_vid || is_tch) && is_rx &&
> -                   (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
> +                   (ops->vidioc_g_fmt_vid_cap ||
> +                    ops->vidioc_g_ext_pix_fmt_vid_cap ||
> +                    ops->vidioc_g_fmt_vid_cap_mplane))
>                         return 0;
>                 break;
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -               if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
> +               if ((is_vid || is_tch) && is_rx &&
> +                   (ops->vidioc_g_fmt_vid_cap_mplane ||
> +                    ops->vidioc_g_ext_pix_fmt_vid_cap))
>                         return 0;
>                 break;
>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
> @@ -977,11 +1004,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>                 break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>                 if (is_vid && is_tx &&
> -                   (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
> +                   (ops->vidioc_g_fmt_vid_out ||
> +                    ops->vidioc_g_ext_pix_fmt_vid_out ||
> +                    ops->vidioc_g_fmt_vid_out_mplane))
>                         return 0;
>                 break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -               if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
> +               if (is_vid && is_tx &&
> +                   (ops->vidioc_g_ext_pix_fmt_vid_out ||
> +                    ops->vidioc_g_fmt_vid_out_mplane))
>                         return 0;
>                 break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
> @@ -1061,6 +1092,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>                sizeof(fmt->fmt.pix) - offset);
>  }
>
> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> +                                 struct v4l2_format *f, bool mplane_cap,
> +                                 bool strict)
> +{
> +       const struct v4l2_plane_ext_pix_format *pe;
> +       struct v4l2_plane_pix_format *p;
> +       unsigned int i;
> +
> +       memset(f, 0, sizeof(*f));
> +
> +       /*
> +        * Make sure no modifier is required before doing the
> +        * conversion.
> +        */
> +       if (e->modifier && strict &&
> +           e->modifier != DRM_FORMAT_MOD_LINEAR &&
> +           e->modifier != DRM_FORMAT_MOD_INVALID)
> +               return -EINVAL;
> +
> +       if (!e->plane_fmt[0].sizeimage && strict)
> +               return -EINVAL;
> +
> +       if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
> +               return 0;
> +
> +       if (!mplane_cap) {
> +               f->fmt.pix.width = e->width;
> +               f->fmt.pix.height = e->height;
> +               f->fmt.pix.pixelformat = e->pixelformat;
> +               f->fmt.pix.field = e->field;
> +               f->fmt.pix.colorspace = e->colorspace;
> +               f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
> +               f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +               f->fmt.pix.quantization = e->quantization;
> +               pe = &e->plane_fmt[0];
> +               f->fmt.pix.bytesperline = pe->bytesperline;
> +               f->fmt.pix.sizeimage = pe->sizeimage;
> +               f->type = e->type;
> +               return 0;
> +       }
> +
> +       f->fmt.pix_mp.width = e->width;
> +       f->fmt.pix_mp.height = e->height;
> +       f->fmt.pix_mp.pixelformat = e->pixelformat;
> +       f->fmt.pix_mp.field = e->field;
> +       f->fmt.pix_mp.colorspace = e->colorspace;
> +       f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
> +       f->fmt.pix_mp.quantization = e->quantization;
> +       if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +               f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> +       else
> +               f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> +
> +       for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> +               pe = &e->plane_fmt[i];
> +               p = &f->fmt.pix_mp.plane_fmt[i];
> +               p->bytesperline = pe->bytesperline;
> +               p->sizeimage = pe->sizeimage;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
> +
> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> +                                 struct v4l2_ext_pix_format *e, bool strict)
> +{
> +       const struct v4l2_plane_pix_format *p;
> +       struct v4l2_plane_ext_pix_format *pe;
> +       unsigned int i;
> +
> +       memset(e, 0, sizeof(*e));
> +
> +       switch (f->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +               e->width = f->fmt.pix.width;
> +               e->height = f->fmt.pix.height;
> +               e->pixelformat = f->fmt.pix.pixelformat;
> +               e->field = f->fmt.pix.field;
> +               e->colorspace = f->fmt.pix.colorspace;
> +               if (f->fmt.pix.flags)
> +                       pr_warn("Ignoring pixelformat flags 0x%x\n",
> +                               f->fmt.pix.flags);
> +               e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
> +               e->quantization = f->fmt.pix.quantization;
> +               e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> +               e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
> +               e->type = f->type;
> +               break;
> +
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +               if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
> +                    !f->fmt.pix_mp.num_planes) && strict)
> +                       return -EINVAL;
> +
> +               e->width = f->fmt.pix_mp.width;
> +               e->height = f->fmt.pix_mp.height;
> +               e->pixelformat = f->fmt.pix_mp.pixelformat;
> +               e->field = f->fmt.pix_mp.field;
> +               e->colorspace = f->fmt.pix_mp.colorspace;
> +               if (f->fmt.pix.flags)
> +                       pr_warn("Ignoring pixelformat flags 0x%x\n",
> +                               f->fmt.pix.flags);
> +               e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
> +               e->quantization = f->fmt.pix_mp.quantization;
> +               if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> +                       e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +               else
> +                       e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +
> +               for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> +                       pe = &e->plane_fmt[i];
> +                       p = &f->fmt.pix_mp.plane_fmt[i];
> +                       pe->bytesperline = p->bytesperline;
> +                       pe->sizeimage = p->sizeimage;
> +               }
> +               break;
> +
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
> +
>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -1564,6 +1723,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
>         p->xfer_func = 0;
>  }
>
> +static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +                            struct file *file, void *fh,
> +                            struct v4l2_format *f)
> +{
> +       struct v4l2_ext_pix_format ef;
> +       int ret;
> +
> +       switch (f->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +               ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +               ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
> +               break;
> +
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +               ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +               ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
> +               break;
> +
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_ext_pix_format_to_format(&ef, f,
> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +                                            true);
> +}
> +
>  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -1600,17 +1791,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>
>         switch (p->type) {
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -               if (unlikely(!ops->vidioc_g_fmt_vid_cap))
> -                       break;
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               if (vfd->vfl_type == VFL_TYPE_TOUCH)
> -                       v4l_pix_format_touch(&p->fmt.pix);
> -               return ret;
> +               if (ops->vidioc_g_fmt_vid_cap) {
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +                               v4l_pix_format_touch(&p->fmt.pix);
> +                       return ret;
> +               } else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
> +                       ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +                               v4l_pix_format_touch(&p->fmt.pix);
> +                       return ret;
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -               return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> +               if (ops->vidioc_g_fmt_vid_cap_mplane)
> +                       return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> +               else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>                 return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
>         case V4L2_BUF_TYPE_VBI_CAPTURE:
> @@ -1618,15 +1819,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>         case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
>                 return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -               if (unlikely(!ops->vidioc_g_fmt_vid_out))
> -                       break;
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               return ret;
> +               if (ops->vidioc_g_fmt_vid_out) {
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       return ret;
> +               } else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -               return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> +               if (ops->vidioc_g_fmt_vid_out_mplane)
> +                       return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> +               else if (ops->vidioc_g_ext_pix_fmt_vid_out)
> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>                 return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
>         case V4L2_BUF_TYPE_VBI_OUTPUT:
> @@ -1645,6 +1853,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>         return -EINVAL;
>  }
>
> +static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +                            struct file *file, void *fh, void *arg)
> +{
> +       struct v4l2_ext_pix_format *ef = arg;
> +       struct v4l2_format f = {
> +               .type = ef->type,
> +       };
> +       int ret;
> +
> +       ret = check_fmt(file, ef->type);
> +       if (ret)
> +               return ret;
> +
> +       memset(ef, 0, sizeof(*ef));
> +       ef->type = f.type;
> +
> +       switch (f.type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +               if (ops->vidioc_g_ext_pix_fmt_vid_cap)
> +                       return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
> +               break;
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +               if (ops->vidioc_g_ext_pix_fmt_vid_out)
> +                       return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
> +               break;
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       ret = v4l_g_fmt(ops, file, fh, &f);
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
> +static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +                            struct file *file, void *fh,
> +                            struct v4l2_format *f)
> +{
> +       struct v4l2_ext_pix_format ef;
> +       int ret;
> +
> +       ret = v4l2_format_to_ext_pix_format(f, &ef, false);
> +       if (ret)
> +               return ret;
> +
> +       switch (f->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +               ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
> +               break;
> +
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +               ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
> +               break;
> +
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_ext_pix_format_to_format(&ef, f,
> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +                                            true);
> +}
> +
>  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -1663,23 +1941,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>
>         switch (p->type) {
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -               if (unlikely(!ops->vidioc_s_fmt_vid_cap))
> +               if (ops->vidioc_s_fmt_vid_cap) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
> +                       ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
> +                       ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +               } else {
>                         break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix);
> -               ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +               }
> +
>                 if (vfd->vfl_type == VFL_TYPE_TOUCH)
>                         v4l_pix_format_touch(&p->fmt.pix);
>                 return ret;
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -               if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -                                         bytesperline);
> -               return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> +               if (ops->vidioc_s_fmt_vid_cap_mplane) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +                                                 bytesperline);
> +                       return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>                 if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
>                         break;
> @@ -1696,21 +1982,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>                 CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>                 return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -               if (unlikely(!ops->vidioc_s_fmt_vid_out))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix);
> -               ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               return ret;
> +               if (ops->vidioc_s_fmt_vid_out) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
> +                       ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       return ret;
> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -               if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -                                         bytesperline);
> -               return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
> +               if (ops->vidioc_s_fmt_vid_out_mplane) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +                                                 bytesperline);
> +                       return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>                 if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
>                         break;
> @@ -1750,6 +2042,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>         return -EINVAL;
>  }
>
> +static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +                            struct file *file, void *fh, void *arg)
> +{
> +       struct video_device *vfd = video_devdata(file);
> +       struct v4l2_ext_pix_format *ef = arg;
> +       struct v4l2_format f;
> +       int ret;
> +
> +       ret = check_fmt(file, ef->type);
> +       if (ret)
> +               return ret;
> +
> +       switch (ef->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +               if (ops->vidioc_s_ext_pix_fmt_vid_cap)
> +                       return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
> +               break;
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +               if (ops->vidioc_s_ext_pix_fmt_vid_out)
> +                       return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
> +               break;
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       ret = v4l2_ext_pix_format_to_format(ef, &f,
> +                                           vfd->device_caps &
> +                                           (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +                                            V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +                                            V4L2_CAP_VIDEO_M2M_MPLANE),
> +                                           false);
> +       if (ret)
> +               return ret;
> +
> +       ret = v4l_s_fmt(ops, file, fh, &f);
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
> +static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +                              struct file *file, void *fh,
> +                              struct v4l2_format *f)
> +{
> +       struct v4l2_ext_pix_format ef;
> +       int ret;
> +
> +       ret = v4l2_format_to_ext_pix_format(f, &ef, false);
> +       if (ret)
> +               return ret;
> +
> +       switch (f->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +               ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
> +               break;
> +
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +               ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
> +               break;
> +
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_ext_pix_format_to_format(&ef, f,
> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +                                            true);
> +}
> +
> +
>  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -1765,23 +2133,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>
>         switch (p->type) {
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -               if (unlikely(!ops->vidioc_try_fmt_vid_cap))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix);
> -               ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               if (vfd->vfl_type == VFL_TYPE_TOUCH)
> -                       v4l_pix_format_touch(&p->fmt.pix);
> -               return ret;
> +               if (ops->vidioc_try_fmt_vid_cap) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
> +                       ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +                               v4l_pix_format_touch(&p->fmt.pix);
> +                       return ret;
> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +                       ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +                               v4l_pix_format_touch(&p->fmt.pix);
> +                       return ret;
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -               if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -                                         bytesperline);
> -               return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
> +               if (ops->vidioc_try_fmt_vid_cap_mplane) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +                                                 bytesperline);
> +                       return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>                 if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
>                         break;
> @@ -1798,21 +2175,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>                 CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>                 return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -               if (unlikely(!ops->vidioc_try_fmt_vid_out))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix);
> -               ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               return ret;
> +               if (ops->vidioc_try_fmt_vid_out) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
> +                       ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       return ret;
> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -               if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -                                         bytesperline);
> -               return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
> +               if (ops->vidioc_try_fmt_vid_out_mplane) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +                                                 bytesperline);
> +                       return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>                 if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
>                         break;
> @@ -1852,6 +2235,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>         return -EINVAL;
>  }
>
> +static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +                              struct file *file, void *fh, void *arg)
> +{
> +       struct video_device *vfd = video_devdata(file);
> +       struct v4l2_ext_pix_format *ef = arg;
> +       struct v4l2_format f;
> +       int ret;
> +
> +       ret = check_fmt(file, ef->type);
> +       if (ret)
> +               return ret;
> +
> +       switch (ef->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +               if (ops->vidioc_try_ext_pix_fmt_vid_cap)
> +                       return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
> +                                                                  ef);
> +               break;
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +               if (ops->vidioc_try_ext_pix_fmt_vid_out)
> +                       return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
> +                                                                  ef);
> +               break;
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       ret = v4l2_ext_pix_format_to_format(ef, &f,
> +                                           vfd->device_caps &
> +                                           (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +                                            V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +                                            V4L2_CAP_VIDEO_M2M_MPLANE),
> +                                           false);
> +       if (ret)
> +               return ret;
> +
> +       ret = v4l_try_fmt(ops, file, fh, &f);
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
>  static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -2771,7 +3197,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>         IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
>         IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
>         IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
> +       IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>         IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
> +       IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>         IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>         IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>         IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
> @@ -2818,6 +3246,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>         IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
>         IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
>         IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
> +       IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>         IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
>         IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
>         IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
> index 86878fba332b0..8bbcb74d8ee31 100644
> --- a/include/media/v4l2-ioctl.h
> +++ b/include/media/v4l2-ioctl.h
> @@ -48,11 +48,17 @@ struct v4l2_fh;
>   * @vidioc_g_fmt_vid_cap: pointer to the function that implements
>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *     in single plane mode
> + * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *     :ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *     capture
>   * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_g_fmt_vid_out: pointer to the function that implements
>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *     in single plane mode
> + * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
> + *     :ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *     out
>   * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>   * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
> @@ -82,11 +88,16 @@ struct v4l2_fh;
>   * @vidioc_s_fmt_vid_cap: pointer to the function that implements
>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *     in single plane mode
> + * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *     :ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *     capture
>   * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_s_fmt_vid_out: pointer to the function that implements
>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *     in single plane mode
> + * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
> + *     :ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>   * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>   * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
> @@ -116,11 +127,16 @@ struct v4l2_fh;
>   * @vidioc_try_fmt_vid_cap: pointer to the function that implements
>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *     in single plane mode
> + * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *     :ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for
> +       video capture
>   * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_try_fmt_vid_out: pointer to the function that implements
>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *     in single plane mode
> + * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
> + *     :ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>   * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   *     output
> @@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
>         /* VIDIOC_G_FMT handlers */
>         int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
>                                     struct v4l2_format *f);
> +       int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +                                           struct v4l2_ext_pix_format *f);
>         int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
>                                         struct v4l2_format *f);
>         int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
>                                     struct v4l2_format *f);
> +       int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +                                           struct v4l2_ext_pix_format *f);
>         int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
>                                             struct v4l2_format *f);
>         int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
>         /* VIDIOC_S_FMT handlers */
>         int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
>                                     struct v4l2_format *f);
> +       int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +                                           struct v4l2_ext_pix_format *f);
>         int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
>                                         struct v4l2_format *f);
>         int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
>                                     struct v4l2_format *f);
> +       int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +                                           struct v4l2_ext_pix_format *f);
>         int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
>                                             struct v4l2_format *f);
>         int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
>         /* VIDIOC_TRY_FMT handlers */
>         int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
>                                       struct v4l2_format *f);
> +       int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +                                             struct v4l2_ext_pix_format *f);
>         int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
>                                           struct v4l2_format *f);
>         int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
>                                       struct v4l2_format *f);
> +       int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +                                             struct v4l2_ext_pix_format *f);
>         int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
>                                              struct v4l2_format *f);
>         int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
>  long int video_ioctl2(struct file *file,
>                       unsigned int cmd, unsigned long int arg);
>
> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> +                                 struct v4l2_ext_pix_format *e, bool strict);
> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> +                                 struct v4l2_format *f,
> +                                 bool mplane_cap, bool strict);
> +
>  /*
>   * The user space interpretation of the 'v4l2_event' differs
>   * based on the 'time_t' definition on 32-bit architectures, so
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index c7b70ff53bc1d..7123c6a4d9569 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>         __u8                            reserved[7];
>  } __attribute__ ((packed));
>
> +/**
> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
> + * @sizeimage:         maximum size in bytes required for data, for which
> + *                     this plane will be used.
> + *                     Should be set to zero for unused planes.
> + * @bytesperline:      distance in bytes between the leftmost pixels in two
> + *                     adjacent lines
> + * @reserved:          extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_plane_ext_pix_format {
> +       __u32 sizeimage;
> +       __u32 bytesperline;
> +       __u32 reserved;

Do we want to make this field a __u64 so the size of the struct is a
multiple of 64 bits?



> +};
> +
> +/**
> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
> + * @type:              type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
> + *                     V4L2_BUF_TYPE_VIDEO_OUTPUT
> + * @width:             image width in pixels
> + * @height:            image height in pixels
> + * @field:             enum v4l2_field; field order (for interlaced video)
> + * @pixelformat:       little endian four character code (fourcc)
> + * @modifier:          modifier applied to the format (used for tiled formats
> + *                     and other kind of HW-specific formats, like compressed
> + *                     formats)
> + * @colorspace:                enum v4l2_colorspace; supplemental to pixelformat
> + * @plane_fmt:         per-plane information
> + * @ycbcr_enc:         enum v4l2_ycbcr_encoding, Y'CbCr encoding
> + * @hsv_enc:           enum v4l2_hsv_encoding, HSV encoding
> + * @quantization:      enum v4l2_quantization, colorspace quantization
> + * @xfer_func:         enum v4l2_xfer_func, colorspace transfer function
> + * @reserved:          extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_ext_pix_format {
> +       __u32 type;
> +       __u32 width;
> +       __u32 height;
> +       __u32 field;
> +       __u64 modifier;
> +       __u32 pixelformat;
> +       __u32 colorspace;
> +       struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
> +       union {
> +               __u32 ycbcr_enc;
> +               __u32 hsv_enc;
> +       };
> +       __u32 quantization;
> +       __u32 xfer_func;
> +       __u32 reserved[9];
> +};
> +
>  /**
>   * struct v4l2_sdr_format - SDR format definition
>   * @pixelformat:       little endian four character code (fourcc)
> @@ -2571,6 +2623,10 @@ struct v4l2_create_buffers {
>
>  #define VIDIOC_QUERY_EXT_CTRL  _IOWR('V', 103, struct v4l2_query_ext_ctrl)
>
> +#define VIDIOC_G_EXT_PIX_FMT   _IOWR('V', 104, struct v4l2_ext_pix_format)
> +#define VIDIOC_S_EXT_PIX_FMT   _IOWR('V', 105, struct v4l2_ext_pix_format)
> +#define VIDIOC_TRY_EXT_PIX_FMT _IOWR('V', 106, struct v4l2_ext_pix_format)
> +
>  /* Reminder: when adding new ioctls please add support for them to
>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>
> --
> 2.28.0.rc2
>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-08-04 19:29 ` [PATCH v5 2/7] media: v4l2: Add extended buffer operations Helen Koike
@ 2020-08-14  7:49   ` Alexandre Courbot
  2020-09-09 12:27   ` Hans Verkuil
  2020-11-20 11:14   ` Tomasz Figa
  2 siblings, 0 replies; 51+ messages in thread
From: Alexandre Courbot @ 2020-08-14  7:49 UTC (permalink / raw)
  To: Helen Koike
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Boris Brezillon,
	Tomasz Figa, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, narmstrong, LKML, frkoenig, Maxime Jourdan,
	Stanimir Varbanov

On Wed, Aug 5, 2020 at 4:32 AM Helen Koike <helen.koike@collabora.com> wrote:
>
> From: Hans Verkuil <hans.verkuil@cisco.com>
>
> Those extended buffer ops have several purpose:
> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>    the number of ns elapsed since 1970
> 2/ Unify single/multiplanar handling
> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>    to support the case where a single buffer object is storing all
>    planes data, each one being placed at a different offset
>
> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> these new objects.
>
> The core takes care of converting new ioctls requests to old ones
> if the driver does not support the new hooks, and vice versa.
>
> Note that the timecode field is gone, since there doesn't seem to be
> in-kernel users. We can be added back in the reserved area if needed or

"We can add it back" or "It can be added back"?


> use the Request API to collect more metadata information from the
> frame.
>
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v5:
> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> - return mem_offset to struct v4l2_ext_plane
> - change sizes and reorder fields to avoid holes in the struct and make
>   it the same for 32 and 64 bits
>
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> I think we can add this later, so I removed it from this RFC to simplify it.
> - Remove num_planes field from struct v4l2_ext_buffer
> - Add flags field to struct v4l2_ext_create_buffers
> - Reformulate struct v4l2_ext_plane
> - Fix some bugs caught by v4l2-compliance
> - Rebased on top of media/master (post 5.8-rc1)
>
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
>
> Changes in v2:
> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>   later on
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>  include/media/v4l2-ioctl.h           |  26 ++
>  include/uapi/linux/videodev2.h       |  90 +++++++
>  4 files changed, 476 insertions(+), 22 deletions(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index e1829906bc086..cb21ee8eb075c 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -720,15 +720,34 @@ static void determine_valid_ioctls(struct video_device *vdev)
>                 SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_sdr_out);
>         }
>
> +       if (is_vid || is_tch) {
> +               /* ioctls valid for video and touch */
> +               if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
> +                       set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
> +               if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
> +                       set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
> +               if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
> +                       set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
> +               if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
> +                       set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
> +               if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
> +                       set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
> +       }
> +
>         if (is_vid || is_vbi || is_sdr || is_tch || is_meta) {
>                 /* ioctls valid for video, vbi, sdr, touch and metadata */
>                 SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
> -               SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
> -               SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
>                 SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
> -               SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
> -               SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
> -               SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
> +               if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
> +                       set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
> +               if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
> +                       set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
> +               if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
> +                       set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
> +               if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
> +                       set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
> +               if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
> +                       set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
>                 SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
>                 SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
>         }
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 14a0def50f8ea..7ecdd9cc1bf48 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -527,6 +527,26 @@ static void v4l_print_buffer(const void *arg, bool write_only)
>                         tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
>  }
>
> +static void v4l_print_ext_buffer(const void *arg, bool write_only)
> +{
> +       const struct v4l2_ext_buffer *e = arg;
> +       const struct v4l2_ext_plane *plane;
> +       unsigned int i;
> +
> +       pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d\n",
> +               e->timestamp, e->index, prt_names(e->type, v4l2_type_names),
> +               e->flags, prt_names(e->field, v4l2_field_names), e->sequence);
> +
> +       for (i = 0; i < VIDEO_MAX_PLANES &&
> +                   e->planes[i].buffer_length; i++) {
> +               plane = &e->planes[i];
> +               pr_debug("plane %d: buffer_length=%d, plane_length=%d offset=0x%08x, memory=%s\n",
> +                        i, plane->buffer_length, plane->plane_length,
> +                        plane->offset,
> +                        prt_names(plane->memory, v4l2_memory_names));
> +       }
> +}
> +
>  static void v4l_print_exportbuffer(const void *arg, bool write_only)
>  {
>         const struct v4l2_exportbuffer *p = arg;
> @@ -546,6 +566,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
>         v4l_print_format(&p->format, write_only);
>  }
>
> +static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
> +{
> +       const struct v4l2_ext_create_buffers *p = arg;
> +
> +       pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
> +               prt_names(p->memory, v4l2_memory_names));
> +       v4l_print_ext_pix_format(&p->format, write_only);
> +}
> +
>  static void v4l_print_streamparm(const void *arg, bool write_only)
>  {
>         const struct v4l2_streamparm *p = arg;
> @@ -1220,6 +1249,143 @@ int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>
> +/*
> + * If mplane_cap is true, b->m.planes should have a valid pointer of a
> + * struct v4l2_plane array, and b->length with its size
> + */
> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
> +                             struct v4l2_buffer *b, bool mplane_cap)
> +{
> +       unsigned int planes_array_size = b->length;
> +       struct v4l2_plane *planes = b->m.planes;
> +       u64 nsecs;
> +
> +       if (!mplane_cap && e->planes[1].buffer_length != 0)
> +               return -EINVAL;
> +
> +       memset(b, 0, sizeof(*b));
> +
> +       b->index = e->index;
> +       b->flags = e->flags;
> +       b->field = e->field;
> +       b->sequence = e->sequence;
> +       b->memory = e->planes[0].memory;
> +       b->request_fd = e->request_fd;
> +       b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
> +       b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
> +
> +       if (mplane_cap) {
> +               unsigned int i;
> +
> +               if (!planes || !planes_array_size)
> +                       return -EINVAL;
> +
> +               b->m.planes = planes;
> +
> +               if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +                       b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> +               else
> +                       b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> +
> +               for (i = 0; i < VIDEO_MAX_PLANES && i < planes_array_size &&
> +                           e->planes[i].buffer_length; i++) {
> +
> +                       if (e->planes[0].memory != e->planes[i].memory)
> +                               return -EINVAL;
> +
> +                       if (e->planes[i].offset)
> +                               return -EINVAL;
> +
> +                       memset(&b->m.planes[i], 0, sizeof(b->m.planes[i]));
> +
> +                       if (b->memory == V4L2_MEMORY_MMAP)
> +                               b->m.planes[i].m.mem_offset = e->planes[i].m.mem_offset;
> +                       else if (b->memory == V4L2_MEMORY_DMABUF)
> +                               b->m.planes[i].m.fd = e->planes[i].m.dmabuf_fd;
> +                       else
> +                               b->m.planes[i].m.userptr = e->planes[i].m.userptr;
> +
> +                       b->m.planes[i].bytesused = e->planes[i].plane_length;
> +                       b->m.planes[i].length = e->planes[i].buffer_length;
> +               }
> +               /* In multi-planar, length contain the number of planes */
> +               b->length = i;
> +       } else {
> +               b->type = e->type;
> +               b->bytesused = e->planes[0].plane_length;
> +               b->length = e->planes[0].buffer_length;
> +
> +               if (e->planes[0].offset)
> +                       return -EINVAL;
> +
> +               if (b->memory == V4L2_MEMORY_MMAP)
> +                       b->m.offset = e->planes[0].m.mem_offset;
> +               else if (b->memory == V4L2_MEMORY_DMABUF)
> +                       b->m.fd = e->planes[0].m.dmabuf_fd;
> +               else
> +                       b->m.userptr = e->planes[0].m.userptr;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
> +
> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
> +                             struct v4l2_ext_buffer *e)
> +{
> +       memset(e, 0, sizeof(*e));
> +
> +       e->index = b->index;
> +       e->flags = b->flags;
> +       e->field = b->field;
> +       e->sequence = b->sequence;
> +       e->request_fd = b->request_fd;
> +       e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
> +               b->timestamp.tv_usec * NSEC_PER_USEC;
> +       if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
> +               unsigned int i;
> +
> +               if (!b->m.planes)
> +                       return -EINVAL;
> +
> +               if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> +                       e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +               else
> +                       e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +
> +               /* In multi-planar, length contain the number of planes */
> +               for (i = 0; i < b->length; i++) {
> +                       if (b->memory == V4L2_MEMORY_MMAP)
> +                               e->planes[i].m.mem_offset = b->m.planes[i].m.mem_offset;
> +                       else if (b->memory == V4L2_MEMORY_DMABUF)
> +                               e->planes[i].m.dmabuf_fd = b->m.planes[i].m.fd;
> +                       else
> +                               e->planes[i].m.userptr = b->m.planes[i].m.userptr;
> +
> +                       e->planes[i].memory = b->memory;
> +                       e->planes[i].buffer_length = b->m.planes[i].length;
> +                       e->planes[i].plane_length = b->m.planes[i].bytesused;
> +                       if (b->m.planes[i].data_offset)
> +                               pr_warn("Ignoring data_offset value %d\n",
> +                                       b->m.planes[i].data_offset);
> +               }
> +       } else {
> +               e->type = b->type;
> +               e->planes[0].memory = b->memory;
> +               e->planes[0].plane_length = b->bytesused;
> +               e->planes[0].buffer_length = b->length;
> +               if (b->memory == V4L2_MEMORY_MMAP)
> +                       e->planes[0].m.mem_offset = b->m.offset;
> +               else if (b->memory == V4L2_MEMORY_DMABUF)
> +                       e->planes[0].m.dmabuf_fd = b->m.fd;
> +               else
> +                       e->planes[0].m.userptr = b->m.userptr;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
> +
>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -2473,31 +2639,112 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
>         return ops->vidioc_reqbufs(file, fh, p);
>  }
>
> +static int v4l_do_buf_op(int (*op)(struct file *, void *,
> +                                  struct v4l2_buffer *),
> +                        int (*ext_op)(struct file *, void *,
> +                                      struct v4l2_ext_buffer *),
> +                        struct file *file, void *fh, struct v4l2_buffer *b)
> +{
> +       struct v4l2_ext_buffer e;
> +       int ret;
> +
> +       ret = check_fmt(file, b->type);
> +       if (ret)
> +               return ret;
> +
> +       if (op)
> +               return op(file, fh, b);
> +
> +       ret = v4l2_buffer_to_ext_buffer(b, &e);
> +       if (ret)
> +               return ret;
> +
> +       ret = ext_op(file, fh, &e);
> +       if (ret)
> +               return ret;
> +
> +       v4l2_ext_buffer_to_buffer(&e, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
> +       return 0;
> +}
> +
> +static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
> +                                      struct v4l2_buffer *),
> +                            int (*ext_op)(struct file *, void *,
> +                                          struct v4l2_ext_buffer *),
> +                            struct file *file, void *fh,
> +                            struct v4l2_ext_buffer *e)
> +{
> +       struct video_device *vdev = video_devdata(file);
> +       struct v4l2_plane planes[VIDEO_MAX_PLANES];
> +       struct v4l2_buffer b;
> +       bool mplane_cap;
> +       int ret;
> +
> +       ret = check_fmt(file, e->type);
> +       if (ret)
> +               return ret;
> +
> +       if (ext_op)
> +               return ext_op(file, fh, e);
> +
> +       mplane_cap = !!(vdev->device_caps &
> +                       (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +                        V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +                        V4L2_CAP_VIDEO_M2M_MPLANE));
> +       b.m.planes = planes;
> +       b.length = VIDEO_MAX_PLANES;
> +       ret = v4l2_ext_buffer_to_buffer(e, &b, mplane_cap);
> +       if (ret)
> +               return ret;
> +
> +       ret = op(file, fh, &b);
> +       if (ret)
> +               return ret;
> +
> +       v4l2_buffer_to_ext_buffer(&b, e);
> +       return 0;
> +}
> +
>  static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
> -                               struct file *file, void *fh, void *arg)
> +                       struct file *file, void *fh, void *arg)
>  {
> -       struct v4l2_buffer *p = arg;
> -       int ret = check_fmt(file, p->type);
> +       return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
> +                            file, fh, arg);
> +}
>
> -       return ret ? ret : ops->vidioc_querybuf(file, fh, p);
> +static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
> +                           struct file *file, void *fh, void *arg)
> +{
> +       return v4l_do_ext_buf_op(ops->vidioc_querybuf,
> +                                ops->vidioc_ext_querybuf, file, fh, arg);
>  }
>
>  static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
> -                               struct file *file, void *fh, void *arg)
> +                   struct file *file, void *fh, void *arg)
>  {
> -       struct v4l2_buffer *p = arg;
> -       int ret = check_fmt(file, p->type);
> +       return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
> +                            file, fh, arg);
> +}
>
> -       return ret ? ret : ops->vidioc_qbuf(file, fh, p);
> +static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
> +                       struct file *file, void *fh, void *arg)
> +{
> +       return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
> +                                file, fh, arg);
>  }
>
>  static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
> -                               struct file *file, void *fh, void *arg)
> +                    struct file *file, void *fh, void *arg)
>  {
> -       struct v4l2_buffer *p = arg;
> -       int ret = check_fmt(file, p->type);
> +       return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
> +                            file, fh, arg);
> +}
>
> -       return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
> +static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
> +                        struct file *file, void *fh, void *arg)
> +{
> +       return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
> +                                file, fh, arg);
>  }
>
>  static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
> @@ -2513,7 +2760,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>
>         v4l_sanitize_format(&create->format);
>
> -       ret = ops->vidioc_create_bufs(file, fh, create);
> +       if (ops->vidioc_create_bufs) {
> +               ret = ops->vidioc_create_bufs(file, fh, create);
> +       } else {
> +               struct v4l2_ext_create_buffers ecreate = {
> +                       .count = create->count,
> +                       .memory = create->memory,
> +               };
> +
> +               ret = v4l2_format_to_ext_pix_format(&create->format,
> +                                                   &ecreate.format, true);
> +               if (ret)
> +                       return ret;
> +
> +               ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
> +               if (ret)
> +                       return ret;
> +
> +               create->index = ecreate.index;
> +               create->count = ecreate.count;
> +               create->capabilities = ecreate.capabilities;
> +       }
>
>         if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>             create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
> @@ -2522,13 +2789,60 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>         return ret;
>  }
>
> +static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
> +                              struct file *file, void *fh, void *arg)
> +{
> +       struct v4l2_ext_create_buffers *ecreate = arg;
> +       struct video_device *vdev = video_devdata(file);
> +       struct v4l2_create_buffers create = {
> +               .count = ecreate->count,
> +               .memory = ecreate->memory,
> +               .flags = ecreate->flags,
> +       };
> +       bool mplane_cap;
> +       int ret;
> +
> +       ret = check_fmt(file, ecreate->format.type);
> +       if (ret)
> +               return ret;
> +
> +       if (ops->vidioc_ext_create_bufs)
> +               return ops->vidioc_ext_create_bufs(file, fh, ecreate);
> +
> +       mplane_cap = !!(vdev->device_caps &
> +                       (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +                        V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +                        V4L2_CAP_VIDEO_M2M_MPLANE));
> +       ret = v4l2_ext_pix_format_to_format(&ecreate->format,
> +                                           &create.format, mplane_cap, true);
> +       if (ret)
> +               return ret;
> +
> +       ret = v4l_create_bufs(ops, file, fh, &create);
> +       if (ret)
> +               return ret;
> +
> +       ecreate->index = create.index;
> +       ecreate->count = create.count;
> +       ecreate->capabilities = create.capabilities;
> +
> +       return 0;
> +}
> +
>  static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
> -                               struct file *file, void *fh, void *arg)
> +                          struct file *file, void *fh, void *arg)
>  {
> -       struct v4l2_buffer *b = arg;
> -       int ret = check_fmt(file, b->type);
> +       return v4l_do_buf_op(ops->vidioc_prepare_buf,
> +                            ops->vidioc_ext_prepare_buf,
> +                            file, fh, arg);
> +}
>
> -       return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
> +static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
> +                              struct file *file, void *fh, void *arg)
> +{
> +       return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
> +                                ops->vidioc_ext_prepare_buf,
> +                                file, fh, arg);
>  }
>
>  static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
> @@ -3202,12 +3516,15 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>         IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>         IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>         IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
> +       IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, planes)),
>         IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>         IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
>         IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
>         IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
>         IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
> +       IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>         IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
> +       IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>         IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>         IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>         IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
> @@ -3272,7 +3589,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>         IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
>         IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
>         IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
> +       IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>         IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
> +       IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>         IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
>         IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
>         IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
> index 8bbcb74d8ee31..75996657ad1ba 100644
> --- a/include/media/v4l2-ioctl.h
> +++ b/include/media/v4l2-ioctl.h
> @@ -169,16 +169,26 @@ struct v4l2_fh;
>   *     :ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
>   * @vidioc_querybuf: pointer to the function that implements
>   *     :ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
> + * @vidioc_ext_querybuf: pointer to the function that implements
> + *     :ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
>   * @vidioc_qbuf: pointer to the function that implements
>   *     :ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
> + * @vidioc_ext_qbuf: pointer to the function that implements
> + *     :ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
>   * @vidioc_expbuf: pointer to the function that implements
>   *     :ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
>   * @vidioc_dqbuf: pointer to the function that implements
>   *     :ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
> + * @vidioc_ext_dqbuf: pointer to the function that implements
> + *     :ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
>   * @vidioc_create_bufs: pointer to the function that implements
>   *     :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
> + * @vidioc_ext_create_bufs: pointer to the function that implements
> + *     :ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
>   * @vidioc_prepare_buf: pointer to the function that implements
>   *     :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
> + * @vidioc_ext_prepare_buf: pointer to the function that implements
> + *     :ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
>   * @vidioc_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
>   * @vidioc_g_fbuf: pointer to the function that implements
> @@ -439,17 +449,27 @@ struct v4l2_ioctl_ops {
>                               struct v4l2_requestbuffers *b);
>         int (*vidioc_querybuf)(struct file *file, void *fh,
>                                struct v4l2_buffer *b);
> +       int (*vidioc_ext_querybuf)(struct file *file, void *fh,
> +                                  struct v4l2_ext_buffer *b);
>         int (*vidioc_qbuf)(struct file *file, void *fh,
>                            struct v4l2_buffer *b);
> +       int (*vidioc_ext_qbuf)(struct file *file, void *fh,
> +                              struct v4l2_ext_buffer *b);
>         int (*vidioc_expbuf)(struct file *file, void *fh,
>                              struct v4l2_exportbuffer *e);
>         int (*vidioc_dqbuf)(struct file *file, void *fh,
>                             struct v4l2_buffer *b);
> +       int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
> +                               struct v4l2_ext_buffer *b);
>
>         int (*vidioc_create_bufs)(struct file *file, void *fh,
>                                   struct v4l2_create_buffers *b);
> +       int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
> +                                     struct v4l2_ext_create_buffers *b);
>         int (*vidioc_prepare_buf)(struct file *file, void *fh,
>                                   struct v4l2_buffer *b);
> +       int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
> +                                     struct v4l2_ext_buffer *b);
>
>         int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
>         int (*vidioc_g_fbuf)(struct file *file, void *fh,
> @@ -758,6 +778,12 @@ int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>                                   struct v4l2_format *f,
>                                   bool mplane_cap, bool strict);
>
> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
> +                             struct v4l2_buffer *b,
> +                             bool mplane_cap);
> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
> +                             struct v4l2_ext_buffer *e);
> +
>  /*
>   * The user space interpretation of the 'v4l2_event' differs
>   * based on the 'time_t' definition on 32-bit architectures, so
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 7123c6a4d9569..334cafdd2be97 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -996,6 +996,41 @@ struct v4l2_plane {
>         __u32                   reserved[11];
>  };
>
> +/**
> + * struct v4l2_ext_plane - extended plane buffer info
> + * @buffer_length:     size of the entire buffer in bytes, should fit
> + *                     @offset + @plane_length
> + * @plane_length:      size of the plane in bytes.
> + * @mem_offset:                If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
> + *                     that should be passed to mmap() called on the video node.
> + * @userptr:           when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> + *                     to this plane.
> + * @dmabuf_fd:         when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> + *                     associated with this plane.
> + * @offset:            offset in the memory buffer where the plane starts.
> + * @memory:            enum v4l2_memory; the method, in which the actual video
> + *                     data is passed
> + * @reserved:          extra space reserved for future fields, must be set to 0.
> + *
> + *
> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> + * can have one plane for Y, and another for interleaved CbCr components.
> + * Each plane can reside in a separate memory buffer, or even in
> + * a completely separate memory node (e.g. in embedded devices).
> + */
> +struct v4l2_ext_plane {
> +       __u32 buffer_length;
> +       __u32 plane_length;
> +       union {
> +               __u32 mem_offset;
> +               __u64 userptr;
> +               __s32 dmabuf_fd;
> +       } m;
> +       __u32 offset;
> +       __u32 memory;
> +       __u32 reserved[4];
> +};
> +
>  /**
>   * struct v4l2_buffer - video buffer info
>   * @index:     id number of the buffer
> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>         };
>  };
>
> +/**
> + * struct v4l2_ext_buffer - extended video buffer info
> + * @index:     id number of the buffer
> + * @type:      V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> + * @flags:     buffer informational flags
> + * @field:     enum v4l2_field; field order of the image in the buffer
> + * @timestamp: frame timestamp
> + * @sequence:  sequence count of this frame
> + * @planes:    per-plane buffer information
> + * @request_fd:        fd of the request that this buffer should use
> + * @reserved:  extra space reserved for future fields, must be set to 0
> + *
> + * Contains data exchanged by application and driver using one of the Streaming
> + * I/O methods.
> + */
> +struct v4l2_ext_buffer {
> +       __u32 index;
> +       __u32 type;
> +       __u32 field;
> +       __u32 sequence;
> +       __u64 flags;
> +       __u64 timestamp;
> +       struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> +       __s32 request_fd;
> +       __u32 reserved[9];
> +};
> +
>  #ifndef __KERNEL__
>  /**
>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
>         __u32                   reserved[6];
>  };
>
> +/**
> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
> + * @index:     on return, index of the first created buffer
> + * @count:     entry: number of requested buffers,
> + *             return: number of created buffers
> + * @memory:    enum v4l2_memory; buffer memory type
> + * @capabilities: capabilities of this buffer type.
> + * @format:    frame format, for which buffers are requested
> + * @flags:     additional buffer management attributes (ignored unless the
> + *             queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
> + *             and configured for MMAP streaming I/O).
> + * @reserved:  extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_ext_create_buffers {
> +       __u32                           index;
> +       __u32                           count;
> +       __u32                           memory;
> +       __u32                           capabilities;
> +       struct v4l2_ext_pix_format      format;
> +       __u32                           flags;
> +       __u32 reserved[5];
> +};
> +
>  /*
>   *     I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>   *
> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
>  #define VIDIOC_G_EXT_PIX_FMT   _IOWR('V', 104, struct v4l2_ext_pix_format)
>  #define VIDIOC_S_EXT_PIX_FMT   _IOWR('V', 105, struct v4l2_ext_pix_format)
>  #define VIDIOC_TRY_EXT_PIX_FMT _IOWR('V', 106, struct v4l2_ext_pix_format)
> +#define VIDIOC_EXT_CREATE_BUFS _IOWR('V', 107, struct v4l2_ext_create_buffers)
> +#define VIDIOC_EXT_QUERYBUF    _IOWR('V', 108, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_QBUF                _IOWR('V', 109, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_DQBUF       _IOWR('V', 110, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_PREPARE_BUF _IOWR('V', 111, struct v4l2_ext_buffer)
>
>  /* Reminder: when adding new ioctls please add support for them to
>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
> --
> 2.28.0.rc2
>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 4/7] media: mediabus: Add helpers to convert a ext_pix format to/from a mbus_fmt
  2020-08-04 19:29 ` [PATCH v5 4/7] media: mediabus: Add helpers to convert a ext_pix format to/from a mbus_fmt Helen Koike
@ 2020-08-14  7:49   ` Alexandre Courbot
  0 siblings, 0 replies; 51+ messages in thread
From: Alexandre Courbot @ 2020-08-14  7:49 UTC (permalink / raw)
  To: Helen Koike
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Boris Brezillon,
	Tomasz Figa, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, narmstrong, LKML, frkoenig, Maxime Jourdan,
	Stanimir Varbanov

On Wed, Aug 5, 2020 at 4:32 AM Helen Koike <helen.koike@collabora.com> wrote:
>
> Just a new version of v4l2_fill_mbus_format() and v4l2_fill_ext_pix_format()
> to deal with the new v4l2_ext_pix_format struct.
> This is needed to convert the VIMC driver to the EXT_FMT/EXT_BUF iocts.
>
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v4:
> - Add helper v4l2_fill_ext_pix_format()
> - Rebased on top of media/master (post 5.8-rc1)
>
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
>
> Changes in v2:
> - New patch
> ---
>  include/media/v4l2-mediabus.h | 42 +++++++++++++++++++++++++++++++++++
>  1 file changed, 42 insertions(+)
>
> diff --git a/include/media/v4l2-mediabus.h b/include/media/v4l2-mediabus.h
> index 45f88f0248c4e..8133407377f7d 100644
> --- a/include/media/v4l2-mediabus.h
> +++ b/include/media/v4l2-mediabus.h
> @@ -119,6 +119,26 @@ v4l2_fill_pix_format(struct v4l2_pix_format *pix_fmt,
>         pix_fmt->xfer_func = mbus_fmt->xfer_func;
>  }
>
> +/**
> + * v4l2_fill_ext_pix_format - Ancillary routine that fills a &struct

I feel the names of these functions are maybe too generic - maybe they
should carry the fact that the source is a mbus or ext_pix format?


> + *     v4l2_ext_pix_format fields from a &struct v4l2_mbus_framefmt.
> + *
> + * @pix_fmt:   pointer to &struct v4l2_ext_pix_format to be filled
> + * @mbus_fmt:  pointer to &struct v4l2_mbus_framefmt to be used as model
> + */
> +static inline void
> +v4l2_fill_ext_pix_format(struct v4l2_ext_pix_format *pix_fmt,
> +                    const struct v4l2_mbus_framefmt *mbus_fmt)
> +{
> +       pix_fmt->width = mbus_fmt->width;
> +       pix_fmt->height = mbus_fmt->height;
> +       pix_fmt->field = mbus_fmt->field;
> +       pix_fmt->colorspace = mbus_fmt->colorspace;
> +       pix_fmt->ycbcr_enc = mbus_fmt->ycbcr_enc;
> +       pix_fmt->quantization = mbus_fmt->quantization;
> +       pix_fmt->xfer_func = mbus_fmt->xfer_func;
> +}
> +
>  /**
>   * v4l2_fill_pix_format - Ancillary routine that fills a &struct
>   *     v4l2_mbus_framefmt from a &struct v4l2_pix_format and a
> @@ -182,4 +202,26 @@ v4l2_fill_mbus_format_mplane(struct v4l2_mbus_framefmt *mbus_fmt,
>         mbus_fmt->xfer_func = pix_mp_fmt->xfer_func;
>  }
>
> +/**
> + * v4l2_fill_mbus_format_ext - Ancillary routine that fills a &struct
> + *     v4l2_mbus_framefmt from a &struct v4l2_ext_pix_format.
> + *
> + * @mbus_fmt:  pointer to &struct v4l2_mbus_framefmt to be filled
> + * @pix_fmt:   pointer to &struct v4l2_ext_pix_format to be used as model
> + * @code:      data format code (from &enum v4l2_mbus_pixelcode)
> + */
> +static inline void
> +v4l2_fill_mbus_format_ext(struct v4l2_mbus_framefmt *mbus_fmt,
> +                         const struct v4l2_ext_pix_format *pix_fmt, u32 code)
> +{
> +       mbus_fmt->width = pix_fmt->width;
> +       mbus_fmt->height = pix_fmt->height;
> +       mbus_fmt->field = pix_fmt->field;
> +       mbus_fmt->colorspace = pix_fmt->colorspace;
> +       mbus_fmt->ycbcr_enc = pix_fmt->ycbcr_enc;
> +       mbus_fmt->quantization = pix_fmt->quantization;
> +       mbus_fmt->xfer_func = pix_fmt->xfer_func;
> +       mbus_fmt->code = code;
> +}
> +
>  #endif
> --
> 2.28.0.rc2
>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 7/7] media: docs: add documentation for the Extended API
  2020-08-04 19:29 ` [PATCH v5 7/7] media: docs: add documentation for the Extended API Helen Koike
@ 2020-08-14  7:49   ` Alexandre Courbot
  2020-11-19 10:28   ` Helen Koike
  2020-11-20 11:06   ` Tomasz Figa
  2 siblings, 0 replies; 51+ messages in thread
From: Alexandre Courbot @ 2020-08-14  7:49 UTC (permalink / raw)
  To: Helen Koike
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Tomasz Figa,
	Hirokazu Honda, Nicolas Dufresne, Brian Starkey, kernel,
	Boris Brezillon, narmstrong, LKML, frkoenig, Maxime Jourdan,
	Stanimir Varbanov

On Wed, Aug 5, 2020 at 4:32 AM Helen Koike <helen.koike@collabora.com> wrote:
>
> Add documentation and update references in current documentation for the
> Extended API.
>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v5:
> - new patch
>
>  .../userspace-api/media/v4l/buffer.rst        |   5 +
>  .../userspace-api/media/v4l/common.rst        |   1 +
>  .../userspace-api/media/v4l/dev-capture.rst   |   5 +
>  .../userspace-api/media/v4l/dev-output.rst    |   5 +
>  .../userspace-api/media/v4l/ext-api.rst       | 107 +++++++++
>  .../userspace-api/media/v4l/format.rst        |  16 +-
>  .../userspace-api/media/v4l/user-func.rst     |   5 +
>  .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++++++++
>  .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++++++
>  .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++++++++++++++++
>  .../media/v4l/vidioc-ext-querybuf.rst         |  79 +++++++
>  .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 ++++++++++
>  12 files changed, 697 insertions(+), 4 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>
> diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> index 57e752aaf414a..c832bedd64e4c 100644
> --- a/Documentation/userspace-api/media/v4l/buffer.rst
> +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> @@ -27,6 +27,11 @@ such as pointers and sizes for each plane, are stored in
>  struct :c:type:`v4l2_plane` instead. In that case,
>  struct :c:type:`v4l2_buffer` contains an array of plane structures.
>
> +.. note::
> +
> +    The :ref:`ext_api` version can also be used, and it is
> +    preferable when applicable.
> +
>  Dequeued video buffers come with timestamps. The driver decides at which
>  part of the frame and with which clock the timestamp is taken. Please
>  see flags in the masks ``V4L2_BUF_FLAG_TIMESTAMP_MASK`` and
> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst
> index 7d81c58a13cd7..3430e0bdad667 100644
> --- a/Documentation/userspace-api/media/v4l/common.rst
> +++ b/Documentation/userspace-api/media/v4l/common.rst
> @@ -59,6 +59,7 @@ applicable to all devices.
>      ext-ctrls-detect
>      fourcc
>      format
> +    ext-api
>      planar-apis
>      selection-api
>      crop
> diff --git a/Documentation/userspace-api/media/v4l/dev-capture.rst b/Documentation/userspace-api/media/v4l/dev-capture.rst
> index 44d3094093ab6..5077639787d92 100644
> --- a/Documentation/userspace-api/media/v4l/dev-capture.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-capture.rst
> @@ -102,6 +102,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>
> +.. note::
> +
> +    The :ref:`ext_api` version can also be used, and it is
> +    preferable when applicable.
> +
>
>  Reading Images
>  ==============
> diff --git a/Documentation/userspace-api/media/v4l/dev-output.rst b/Documentation/userspace-api/media/v4l/dev-output.rst
> index e4f2a1d8b0fc7..f8f40c708e49f 100644
> --- a/Documentation/userspace-api/media/v4l/dev-output.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-output.rst
> @@ -99,6 +99,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>
> +.. note::
> +
> +    The :ref:`ext_api` version can also be used, and it is
> +    preferable when applicable.
> +
>
>  Writing Images
>  ==============
> diff --git a/Documentation/userspace-api/media/v4l/ext-api.rst b/Documentation/userspace-api/media/v4l/ext-api.rst
> new file mode 100644
> index 0000000000000..da2a82960d22f
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/ext-api.rst
> @@ -0,0 +1,107 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _ext_api:
> +
> +*************
> +Extendend API

s/Extendend/Extended

> +*************
> +
> +Introduction
> +============
> +
> +The Extended Format API was conceived to extend V4L2 capabilities and
> +to simplify certain mechanisms.
> +It unifies single- vs multi- planar handling,
> +brings the concept of pixelformat + modifiers, allows planes to be placed
> +in any offset inside a buffer and let userspace to change colorspace
> +attributes if supported by the driver.
> +
> +Data format negotiation and buffer handling works very similar to the classical
> +version, thus in this document we'll just mention the main differences.
> +
> +Data Format Negotiation
> +=======================
> +
> +The API replaces the classical ioctls:
> +
> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>`, :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`,
> +:ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>`
> +(with :c:type:`v4l2_format` as the main parameter).
> +
> +By the extended versions:
> +
> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
> +:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
> +(with :c:type:`v4l2_ext_pix_format` as the main parameter).
> +
> +For CAPTURE and OUTPUT queues only.
> +
> +The ``type`` field of struct :c:type:`v4l2_ext_pix_format` only accepts
> +``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or ``V4L2_BUF_TYPE_VIDEO_OUTPUT``, and it
> +supports multiplanar format through a combination of ``pixelfomat`` and
> +``modifier`` fields.
> +
> +Only the single-planar variants of the pixel formats are valid in the
> +``pixelformat`` field.
> +To support multi-planar, a modifier should be set in the ``modifier`` field to
> +provide more information regarding the memory layout of pixels and number of
> +planes.
> +
> +The ``plane_fmt`` field is an array which holds information by plane using
> +the :c:type:`v4l2_plane_ext_pix_format`. When this struct is filled, its
> +``sizeimage`` field should be non-zero for all valid planes for a given
> +``pixelformat`` + ``modifier`` combination, and zeroed for the invalid ones.
> +
> +Colospace attributes are not read-only as in the classical version, i.e, they

s/Colospace/Colorspace

> +can be set by application and drivers will adjust accordingly depending on what
> +is supported by the underlying hardware.
> +
> +Buffers
> +=======
> +
> +The API replaces the classical ioctls:
> +
> +:ref:`VIDIOC_CREATE_BUFS <VIDIOC_CREATE_BUFS>`
> +(with :c:type:`v4l2_create_buffers` as the main parameter),
> +:ref:`VIDIOC_QUERYBUF <VIDIOC_QUERYBUF>`, :ref:`VIDIOC_QBUF <VIDIOC_QBUF>`,
> +:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`, :ref:`VIDIOC_PREPARE_BUF <VIDIOC_PREPARE_BUF>`
> +(with :c:type:`v4l2_buffer` as the main parameter)
> +
> +By the extended versions:
> +
> +:ref:`VIDIOC_EXT_CREATE_BUFS <VIDIOC_EXT_CREATE_BUFS>`
> +(with :c:type:`v4l2_ext_create_buffers` as the main parameter),
> +:ref:`VIDIOC_EXT_QUERYBUF <VIDIOC_EXT_QUERYBUF>`,
> +:ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>`,
> +:ref:`VIDIOC_EXT_DQBUF <VIDIOC_EXT_QBUF>`,
> +:ref:`VIDIOC_EXT_PREPARE_BUF <VIDIOC_EXT_PREPARE_BUF>`
> +(with :c:type:`v4l2_ext_buffer` as the main parameter)
> +
> +The basic difference between :c:type:`v4l2_create_buffers` and
> +:c:type:`v4l2_ext_create_buffers` is that the later have a
> +:c:type:`v4l2_ext_pix_format` as the type of the ``format`` field.
> +
> +Comparing :c:type:`v4l2_ext_buffer` with :c:type:`v4l2_buffer`, in the
> +extended version there is a unification of the single-/multi- planar handling
> +through the ``planes`` field of type :c:type:`v4l2_ext_plane`.
> +
> +The :c:type:`v4l2_ext_plane also allows planes to be placed in a given offset

Missing ` after v4l2_ext_plane.


> +inside a buffer.
> +Planes can be placed in different locations inside the same buffer, or in
> +different buffers.
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_plane
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_buffer
> diff --git a/Documentation/userspace-api/media/v4l/format.rst b/Documentation/userspace-api/media/v4l/format.rst
> index e47fc0505727c..b96d26f26793c 100644
> --- a/Documentation/userspace-api/media/v4l/format.rst
> +++ b/Documentation/userspace-api/media/v4l/format.rst
> @@ -28,13 +28,19 @@ format and the driver selects and reports the best the hardware can do
>  to satisfy the request. Of course applications can also just query the
>  current selection.
>
> -A single mechanism exists to negotiate all data formats using the
> -aggregate struct :c:type:`v4l2_format` and the
> +There are two mechanism to negociate data formats:
> +
> +The first one is using the aggregate struct :c:type:`v4l2_format` and the
>  :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and
>  :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctls. Additionally the
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` ioctl can be used to examine
>  what the hardware *could* do, without actually selecting a new data
> -format. The data formats supported by the V4L2 API are covered in the
> +format.
> +
> +The second is through the :ref:`ext_api`, please refer to its documentation
> +for more information.
> +
> +The data formats supported by the V4L2 API are covered in the
>  respective device section in :ref:`devices`. For a closer look at
>  image formats see :ref:`pixfmt`.
>
> @@ -71,7 +77,9 @@ earlier versions of V4L2. Switching the logical stream or returning into
>  *may* support a switch using :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`.
>
>  All drivers exchanging data with applications must support the
> -:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl. Implementation of the
> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl
> +or the Extended respective versions (TODO: link).
> +Implementation of the
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is highly recommended but optional.
>
>
> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
> index bf77c842718e5..67b254812791c 100644
> --- a/Documentation/userspace-api/media/v4l/user-func.rst
> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
> @@ -20,6 +20,7 @@ Function Reference
>      func-close
>      func-ioctl
>      vidioc-create-bufs
> +    vidioc-ext-create-bufs
>      vidioc-cropcap
>      vidioc-dbg-g-chip-info
>      vidioc-dbg-g-register
> @@ -48,6 +49,7 @@ Function Reference
>      vidioc-g-ext-ctrls
>      vidioc-g-fbuf
>      vidioc-g-fmt
> +    vidioc-g-ext-pix-fmt
>      vidioc-g-frequency
>      vidioc-g-input
>      vidioc-g-jpegcomp
> @@ -62,8 +64,11 @@ Function Reference
>      vidioc-log-status
>      vidioc-overlay
>      vidioc-prepare-buf
> +    vidioc-ext-prepare-buf
>      vidioc-qbuf
> +    vidioc-ext-qbuf
>      vidioc-querybuf
> +    vidioc-ext-querybuf
>      vidioc-querycap
>      vidioc-queryctrl
>      vidioc-query-dv-timings
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
> new file mode 100644
> index 0000000000000..67f2c541e4d79
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
> @@ -0,0 +1,95 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_CREATE_BUFS:
> +
> +****************************
> +ioctl VIDIOC_EXT_CREATE_BUFS
> +****************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_CREATE_BUFS - Create buffers for Memory Mapped or User Pointer or DMA Buffer I/O
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp )
> +    :name: VIDIOC_EXT_CREATE_BUFS
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_create_buffers`.
> +
> +
> +Description
> +===========
> +
> +This ioctl is used to create buffers for :ref:`memory mapped <mmap>`
> +or :ref:`user pointer <userp>` or :ref:`DMA buffer <dmabuf>` I/O.
> +This ioctl can be called multiple times to
> +create buffers of different sizes.
> +
> +To allocate the device buffers applications must initialize the relevant
> +fields of the struct :c:type:`v4l2_ext_create_buffers` structure. The
> +``count`` field must be set to the number of requested buffers, the
> +``memory`` field specifies the requested I/O method and the ``reserved``
> +array must be zeroed.
> +
> +The ``format`` field specifies the image format that the buffers must be
> +able to handle. The application has to fill in this struct
> +:c:type:`v4l2_format`. Usually this will be done using the
> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` or
> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctls to ensure that the
> +requested format is supported by the driver.
> +The driver may return an error if the size(s) are not supported by the
> +hardware (usually because they are too small).
> +
> +The driver can create a memory buffer per plane, or a single memory buffer
> +containing all the planes, with a minimum size according to the size
> +defined by the ``format.plane_fmt[i].sizeimage`` field.
> +Usually if the ``format.plane_fmt[i].sizeimage``
> +field is less than the minimum required for the given format, then an
> +error will be returned since drivers will typically not allow this. If
> +it is larger, then the value will be used as-is. In other words, the
> +driver may reject the requested size, but if it is accepted the driver
> +will use it unchanged.
> +
> +When the ioctl is called with a pointer to this structure the driver
> +will attempt to allocate up to the requested number of buffers and store
> +the actual number allocated and the starting index in the ``count`` and
> +the ``index`` fields respectively. On return ``count`` can be smaller
> +than the number requested.
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_create_buffers
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +ENOMEM
> +    No memory to allocate buffers for :ref:`memory mapped <mmap>` I/O.
> +
> +EINVAL
> +    The buffer type (``format.type`` field), requested I/O method
> +    (``memory``) or format (``format`` field) is not valid.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
> new file mode 100644
> index 0000000000000..88e4b57121254
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
> @@ -0,0 +1,62 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_PREPARE_BUF:
> +
> +****************************
> +ioctl VIDIOC_EXT_PREPARE_BUF
> +****************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_PREPARE_BUF - Prepare a buffer for I/O
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_PREPARE_BUF
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> +
> +
> +Description
> +===========
> +
> +Applications can optionally call the :ref:`VIDIOC_EXT_PREPARE_BUF` ioctl to
> +pass ownership of the buffer to the driver before actually enqueuing it,
> +using the :ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>` ioctl, and to prepare it for future I/O. Such
> +preparations may include cache invalidation or cleaning. Performing them
> +in advance saves time during the actual I/O.
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EBUSY
> +    File I/O is in progress.
> +
> +EINVAL
> +    The buffer ``type`` is not supported, or the ``index`` is out of
> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
> +    ``length`` are invalid.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
> new file mode 100644
> index 0000000000000..083e106cf6bad
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
> @@ -0,0 +1,204 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_QBUF:
> +
> +***************************************
> +ioctl VIDIOC_EXT_QBUF, VIDIOC_EXT_DQBUF
> +***************************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_QBUF - VIDIOC_EXT_DQBUF - Exchange a buffer with the driver
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_QBUF
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_DQBUF
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> +
> +
> +Description
> +===========
> +
> +Applications call the ``VIDIOC_EXT_QBUF`` ioctl to enqueue an empty
> +(capturing) or filled (output) buffer in the driver's incoming queue.
> +The semantics depend on the selected I/O method.
> +
> +To enqueue a buffer applications set the ``type`` field of a struct
> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``.
> +Applications must also set the ``index`` field. Valid index numbers
> +range from zero to the number of buffers allocated with
> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
> +one. The contents of the struct :c:type:`v4l2_ext_buffer` returned
> +by a :ref:`VIDIOC_EXT_QUERYBUF` ioctl will do as well.
> +When the buffer is intended for output (``type`` is
> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``) applications must also initialize the
> +``timestamp`` fields and the ``planes[i].plane_length`` for each valid plane,
> +and invalid ones must be set as zero.
> +see :ref:`buffer` for details. Applications must also set ``flags`` to 0. The
> +``reserved`` field must be set to 0.
> +
> +To enqueue a :ref:`memory mapped <mmap>` buffer applications set the
> +``planes[i].memory`` field to ``V4L2_MEMORY_MMAP`` in all the valid planes,
> +and invalid ones must be set as zero.
> +When ``VIDIOC_EXT_QBUF`` is called
> +with a pointer to this structure the driver sets the
> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_QUEUED`` flags and clears
> +the ``V4L2_BUF_FLAG_DONE`` flag in the ``flags`` field, or it returns an
> +``EINVAL`` error code.
> +
> +To enqueue a :ref:`user pointer <userp>` buffer applications set the
> +``planes[i].memory`` field to ``V4L2_MEMORY_USERPTR`` in all the valid planes,
> +and invalid ones must be set as zero, the ``planes[i].m.userptr`` field to the
> +address of the buffer,``planes[i].buffer_length`` to the size of the memory

These backquotes appear as-is in the html document for some reason.

> +buffer, ``planes[i].plane_length`` to the size that should be used by the plane,
> +and ``planes[i].offset`` of the plane in the memory buffer.
> +
> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure
> +the driver sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
> +``flags`` field, or it returns an error code. This ioctl locks the
> +memory pages of the buffer in physical memory, they cannot be swapped
> +out to disk. Buffers remain locked until dequeued, until the
> +:ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the

Just for my education, why does VIDIOC_EXT_CREATE_BUFS unlock user buffer pages?

> +device is closed.
> +
> +To enqueue a :ref:`DMABUF <dmabuf>` buffer applications set the
> +``planes[i].memory`` field to ``V4L2_MEMORY_DMABUF`` in all the valid planes,
> +and invalid ones must be set as zero, the ``planes[i].m.fd`` field to a
> +file descriptor associated with a DMABUF buffer.,``planes[i].buffer_length`` to

Same issue with the backquotes appearing as-is in the html - maybe we
need a space before them?


> +the size of the memory buffer, ``planes[i].plane_length`` to the size that
> +should be used by the plane, and ``planes[i].offset`` of the plane in the memory buffer.
> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure the driver
> +sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
> +``flags`` field, or it returns an error code. This ioctl locks the
> +buffer. Locking a buffer means passing it to a driver for a hardware
> +access (usually DMA). If an application accesses (reads/writes) a locked
> +buffer then the result is undefined. Buffers remain locked until
> +dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
> +device is closed.
> +
> +The ``request_fd`` field can be used with the ``VIDIOC_EXT_QBUF`` ioctl to specify
> +the file descriptor of a :ref:`request <media-request-api>`, if requests are
> +in use. Setting it means that the buffer will not be passed to the driver
> +until the request itself is queued. Also, the driver will apply any
> +settings associated with the request for this buffer. This field will
> +be ignored unless the ``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
> +If the device does not support requests, then ``EBADR`` will be returned.
> +If requests are supported but an invalid request file descriptor is given,
> +then ``EINVAL`` will be returned.
> +
> +.. caution::
> +   It is not allowed to mix queuing requests with queuing buffers directly.
> +   ``EBUSY`` will be returned if the first buffer was queued directly and
> +   then the application tries to queue a request, or vice versa. After
> +   closing the file descriptor, calling
> +   :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or calling :ref:`VIDIOC_EXT_CREATE_BUFS`
> +   the check for this will be reset.
> +
> +   For :ref:`memory-to-memory devices <mem2mem>` you can specify the
> +   ``request_fd`` only for output buffers, not for capture buffers. Attempting
> +   to specify this for a capture buffer will result in an ``EBADR`` error.
> +
> +Applications call the ``VIDIOC_EXT_DQBUF`` ioctl to dequeue a filled
> +(capturing) or displayed (output) buffer from the driver's outgoing
> +queue. They just set the ``type``, ``planes[i].memory`` and ``reserved`` fields of
> +a struct :c:type:`v4l2_ext_buffer` as above, when
> +``VIDIOC_EXT_DQBUF`` is called with a pointer to this structure the driver
> +fills the remaining fields or returns an error code. The driver may also
> +set ``V4L2_BUF_FLAG_ERROR`` in the ``flags`` field. It indicates a
> +non-critical (recoverable) streaming error. In such case the application
> +may continue as normal, but should be aware that data in the dequeued
> +buffer might be corrupted. When using the multi-planar API, the planes
> +array must be passed in as well.
> +
> +If the application sets the ``memory`` field to ``V4L2_MEMORY_DMABUF`` to
> +dequeue a :ref:`DMABUF <dmabuf>` buffer, the driver fills the ``m.fd`` field
> +with a file descriptor numerically the same as the one given to ``VIDIOC_EXT_QBUF``
> +when the buffer was enqueued. No new file descriptor is created at dequeue time
> +and the value is only for the application convenience.
> +
> +By default ``VIDIOC_EXT_DQBUF`` blocks when no buffer is in the outgoing
> +queue. When the ``O_NONBLOCK`` flag was given to the
> +:ref:`open() <func-open>` function, ``VIDIOC_EXT_DQBUF`` returns
> +immediately with an ``EAGAIN`` error code when no buffer is available.
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_buffers
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EAGAIN
> +    Non-blocking I/O has been selected using ``O_NONBLOCK`` and no
> +    buffer was in the outgoing queue.
> +
> +EINVAL
> +    The buffer ``type`` is not supported, or the ``index`` is out of
> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
> +    ``length`` are invalid, or the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was
> +    set but the the given ``request_fd`` was invalid, or ``m.fd`` was
> +    an invalid DMABUF file descriptor.
> +
> +EIO
> +    ``VIDIOC_EXT_DQBUF`` failed due to an internal error. Can also indicate
> +    temporary problems like signal loss.
> +
> +    .. note::
> +
> +       The driver might dequeue an (empty) buffer despite returning
> +       an error, or even stop capturing. Reusing such buffer may be unsafe
> +       though and its details (e.g. ``index``) may not be returned either.
> +       It is recommended that drivers indicate recoverable errors by setting
> +       the ``V4L2_BUF_FLAG_ERROR`` and returning 0 instead. In that case the
> +       application should be able to safely reuse the buffer and continue
> +       streaming.
> +
> +EPIPE
> +    ``VIDIOC_EXT_DQBUF`` returns this on an empty capture queue for mem2mem
> +    codecs if a buffer with the ``V4L2_BUF_FLAG_LAST`` was already
> +    dequeued and no new buffers are expected to become available.
> +
> +EBADR
> +    The ``V4L2_BUF_FLAG_REQUEST_FD`` flag was set but the device does not
> +    support requests for the given buffer type, or
> +    the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was not set but the device requires
> +    that the buffer is part of a request.
> +
> +EBUSY
> +    The first buffer was queued via a request, but the application now tries
> +    to queue it directly, or vice versa (it is not permitted to mix the two
> +    APIs).
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
> new file mode 100644
> index 0000000000000..f2a12017253f6
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
> @@ -0,0 +1,79 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_QUERYBUF:
> +
> +*************************
> +ioctl VIDIOC_EXT_QUERYBUF
> +*************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_QUERYBUF - Query the status of a buffer
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_QUERYBUF
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> +
> +
> +Description
> +===========
> +
> +This ioctl is part of the :ref:`streaming <mmap>` I/O method. It can
> +be used to query the status of a buffer at any time after buffers have
> +been allocated with the :ref:`VIDIOC_EXT_CREATE_BUFS` ioctl.
> +
> +Applications set the ``type`` field of a struct
> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``,
> +and the ``index`` field. Valid index numbers range from zero to the
> +number of buffers allocated with
> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
> +one. The ``reserved`` field must be set to 0.
> +
> +In the ``flags`` field the ``V4L2_BUF_FLAG_MAPPED``,
> +``V4L2_BUF_FLAG_PREPARED``, ``V4L2_BUF_FLAG_QUEUED`` and
> +``V4L2_BUF_FLAG_DONE`` flags will be valid. The ``planes.memory`` fields will be
> +set to the current I/O method for each plane.
> +
> +For every valid plane, an entry in ``planes`` will be filled, and zeroed for
> +invalid ones. ``planes[i].buffer_length`` is the size of the memory buffer
> +which contains the plane, ``planes[i].plane_length`` is the length of the plane,
> +and ``planes[i].offset` is where the plane is placed in the memory buffer.
> +
> +The size of the ``planes`` array can be calculated by the number of sequential
> +planes with ``planes[i].buffer_length`` that differs from zero up to the max
> +size of the array.
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EINVAL
> +    The buffer ``type`` is not supported, or the ``index`` is out of
> +    bounds.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> new file mode 100644
> index 0000000000000..008e6c98a88a5
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> @@ -0,0 +1,117 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_G_EXT_PIX_FMT:
> +
> +************************************************************************
> +ioctl VIDIOC_G_EXT_PIX_FMT, VIDIOC_S_EXT_PIX_FMT, VIDIOC_TRY_EXT_PIX_FMT
> +************************************************************************
> +
> +Name
> +====
> +
> +VIDIOC_G_EXT_PIX_FMT - VIDIOC_S_EXT_PIX_FMT - VIDIOC_TRY_EXT_PIX_FMT - Get or set the data format, try a format
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> +    :name: VIDIOC_G_EXT_PIX_FMT
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> +    :name: VIDIOC_S_EXT_PIX_FMT
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> +    :name: VIDIOC_TRY_EXT_PIX_FMT
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_pix_format`.
> +
> +
> +Description
> +===========
> +
> +These ioctls are used to negotiate the format of data (typically image
> +format) exchanged between driver and application.
> +
> +To query the current parameters applications set the ``type`` field of a
> +struct :c:type:`v4l2_ext_pix_format` to  ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or
> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``, all the other types are invalid in this API,
> +and multiplanar is supported through modifiers.
> +
> +When the application calls the
> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with a pointer to this
> +structure the driver fills the other members.
> +When the requested buffer type is not supported drivers return
> +an ``EINVAL`` error code.
> +
> +To change the current format parameters applications initialize all
> +the fields in the struct.
> +For details see the documentation of the various devices types in
> +:ref:`devices`. Good practice is to query the current parameters
> +first, and to modify only those parameters not suitable for the
> +application. When the application calls the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with
> +a pointer to a struct :c:type:`v4l2_ext_pix_format` structure the driver
> +checks and adjusts the parameters against hardware abilities. Drivers
> +should not return an error code unless the ``type`` field is invalid,
> +this is a mechanism to fathom device capabilities and to approach
> +parameters acceptable for both the application and driver. On success
> +the driver may program the hardware, allocate resources and generally
> +prepare for data exchange. Finally the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl returns
> +the current format parameters as :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` does. Very simple,
> +inflexible devices may even ignore all input and always return the
> +default parameters. However all V4L2 devices exchanging data with the
> +application must implement the :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` and :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
> +ioctl. When the requested buffer type is not supported drivers return an
> +EINVAL error code on a :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` attempt. When I/O is already in
> +progress or the resource is not available for other reasons drivers
> +return the ``EBUSY`` error code.
> +
> +The :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl is equivalent to :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` with one
> +exception: it does not change driver state. It can also be called at any
> +time, never returning ``EBUSY``. This function is provided to negotiate
> +parameters, to learn about hardware limitations, without disabling I/O
> +or possibly time consuming hardware preparations. Although strongly
> +recommended drivers are not required to implement this ioctl.
> +
> +The format as returned by :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` must be identical to what
> +:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` returns for the same input or output.
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_plane_ext_pix_format
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_pix_format
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EINVAL
> +    The struct :c:type:`v4l2_ext_pix_format` ``type`` field is
> +    invalid or the requested buffer type not supported.
> +
> +EBUSY
> +    The device is busy and cannot change the format. This could be
> +    because or the device is streaming or buffers are allocated or
> +    queued to the driver. Relevant for :ref:`VIDIOC_S_EXT_PIX_FMT
> +    <VIDIOC_G_EXT_PIX_FMT>` only.
> --
> 2.28.0.rc2
>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-08-04 19:29 ` [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
  2020-08-14  7:49   ` Alexandre Courbot
@ 2020-09-09 11:41   ` Hans Verkuil
  2020-09-14  2:14     ` Helen Koike
  2020-10-02 19:49   ` Tomasz Figa
  2 siblings, 1 reply; 51+ messages in thread
From: Hans Verkuil @ 2020-09-09 11:41 UTC (permalink / raw)
  To: Helen Koike, mchehab, laurent.pinchart, sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

Hi Helen,

Some review comments, concentrating on the uAPI.

On 04/08/2020 21:29, Helen Koike wrote:
> This is part of the multiplanar and singleplanar unification process.
> v4l2_ext_pix_format is supposed to work for both cases.
> 
> We also add the concept of modifiers already employed in DRM to expose
> HW-specific formats (like tiled or compressed formats) and allow
> exchanging this information with the DRM subsystem in a consistent way.
> 
> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
> v4l2_ext_format, other types will be rejected if you use the
> {G,S,TRY}_EXT_PIX_FMT ioctls.
> 
> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
> in drivers, but, in the meantime, the core takes care of converting
> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
> still work if the userspace app/lib uses the new ioctls.
> The conversion is also done the other around to allow userspace
> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
> _ext_ hooks.
> 
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v5:
> - change sizes and reorder fields to avoid holes in the struct and make
>   it the same for 32 and 64 bits
> - removed __attribute__ ((packed)) from uapi structs
> - Fix doc warning from make htmldocs
> - Updated commit message with EXT_PIX prefix for the ioctls.
> 
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Add reserved fields
> - Removed num_planes from struct v4l2_ext_pix_format
> - Removed flag field from struct v4l2_ext_pix_format, since the only
>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>   where we can use modifiers, or add it back later through the reserved
>   bits.
> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>   != MOD_INVALID
> - Fix type assignment in v4l_g_fmt_ext_pix()
> - Rebased on top of media/master (post 5.8-rc1)
> 
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
> 
> Changes in v2:
> - Move the modifier in v4l2_ext_format (was formerly placed in
>   v4l2_ext_plane)
> - Fix a few bugs in the converters and add a strict parameter to
>   allow conversion of uninitialized/mis-initialized objects
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>  include/media/v4l2-ioctl.h           |  34 ++
>  include/uapi/linux/videodev2.h       |  56 +++
>  4 files changed, 615 insertions(+), 81 deletions(-)
> 

<snip>

> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index c7b70ff53bc1d..7123c6a4d9569 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>  	__u8				reserved[7];
>  } __attribute__ ((packed));
>  
> +/**
> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
> + * @sizeimage:		maximum size in bytes required for data, for which
> + *			this plane will be used.
> + *			Should be set to zero for unused planes.
> + * @bytesperline:	distance in bytes between the leftmost pixels in two
> + *			adjacent lines
> + * @reserved:		extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_plane_ext_pix_format {
> +	__u32 sizeimage;
> +	__u32 bytesperline;
> +	__u32 reserved;

I'd use reserved[4] here.

> +};
> +
> +/**
> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
> + * @type:		type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
> + *			V4L2_BUF_TYPE_VIDEO_OUTPUT
> + * @width:		image width in pixels
> + * @height:		image height in pixels
> + * @field:		enum v4l2_field; field order (for interlaced video)
> + * @pixelformat:	little endian four character code (fourcc)
> + * @modifier:		modifier applied to the format (used for tiled formats
> + *			and other kind of HW-specific formats, like compressed
> + *			formats)

This should refer to the drm.h header since we're reusing the DRM modifiers.

> + * @colorspace:		enum v4l2_colorspace; supplemental to pixelformat
> + * @plane_fmt:		per-plane information
> + * @ycbcr_enc:		enum v4l2_ycbcr_encoding, Y'CbCr encoding
> + * @hsv_enc:		enum v4l2_hsv_encoding, HSV encoding
> + * @quantization:	enum v4l2_quantization, colorspace quantization
> + * @xfer_func:		enum v4l2_xfer_func, colorspace transfer function
> + * @reserved:		extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_ext_pix_format {
> +	__u32 type;
> +	__u32 width;
> +	__u32 height;
> +	__u32 field;
> +	__u64 modifier;
> +	__u32 pixelformat;
> +	__u32 colorspace;
> +	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
> +	union {
> +		__u32 ycbcr_enc;
> +		__u32 hsv_enc;
> +	};
> +	__u32 quantization;
> +	__u32 xfer_func;

I'd reorder this:

	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
	__u32 pixelformat;
	__u32 colorspace;
	__u32 xfer_func;
	union {
		__u32 ycbcr_enc;
		__u32 hsv_enc;
	};
	__u32 quantization;

The reason for reordering is that I like to keep the colorimetry fields in
that order since that is how these fields are processed mathematically: you
apply the colorspace matrix first, then the transfer function, then optionally
convert to Y'CbCr or HSV and finally you quantize the result.

There is also a __u32 flags; field missing (needed for V4L2_PIX_FMT_FLAG_PREMUL_ALPHA
and for the upcoming CSC support).

> +	__u32 reserved[9];
> +};
> +
>  /**
>   * struct v4l2_sdr_format - SDR format definition
>   * @pixelformat:	little endian four character code (fourcc)
> @@ -2571,6 +2623,10 @@ struct v4l2_create_buffers {
>  
>  #define VIDIOC_QUERY_EXT_CTRL	_IOWR('V', 103, struct v4l2_query_ext_ctrl)
>  
> +#define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
> +#define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
> +#define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
> +
>  /* Reminder: when adding new ioctls please add support for them to
>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>  
> 

Regards,

	Hans

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-08-04 19:29 ` [PATCH v5 2/7] media: v4l2: Add extended buffer operations Helen Koike
  2020-08-14  7:49   ` Alexandre Courbot
@ 2020-09-09 12:27   ` Hans Verkuil
  2020-11-23 15:08     ` Helen Koike
  2020-11-20 11:14   ` Tomasz Figa
  2 siblings, 1 reply; 51+ messages in thread
From: Hans Verkuil @ 2020-09-09 12:27 UTC (permalink / raw)
  To: Helen Koike, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

Hi Helen,

Again I'm just reviewing the uAPI.

On 04/08/2020 21:29, Helen Koike wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Those extended buffer ops have several purpose:
> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>    the number of ns elapsed since 1970
> 2/ Unify single/multiplanar handling
> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>    to support the case where a single buffer object is storing all
>    planes data, each one being placed at a different offset
> 
> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> these new objects.
> 
> The core takes care of converting new ioctls requests to old ones
> if the driver does not support the new hooks, and vice versa.
> 
> Note that the timecode field is gone, since there doesn't seem to be
> in-kernel users. We can be added back in the reserved area if needed or
> use the Request API to collect more metadata information from the
> frame.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v5:
> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> - return mem_offset to struct v4l2_ext_plane
> - change sizes and reorder fields to avoid holes in the struct and make
>   it the same for 32 and 64 bits
> 
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> I think we can add this later, so I removed it from this RFC to simplify it.
> - Remove num_planes field from struct v4l2_ext_buffer
> - Add flags field to struct v4l2_ext_create_buffers
> - Reformulate struct v4l2_ext_plane
> - Fix some bugs caught by v4l2-compliance
> - Rebased on top of media/master (post 5.8-rc1)
> 
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
> 
> Changes in v2:
> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>   later on
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>  include/media/v4l2-ioctl.h           |  26 ++
>  include/uapi/linux/videodev2.h       |  90 +++++++
>  4 files changed, 476 insertions(+), 22 deletions(-)
> 

<snip>

> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 7123c6a4d9569..334cafdd2be97 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -996,6 +996,41 @@ struct v4l2_plane {
>  	__u32			reserved[11];
>  };
>  
> +/**
> + * struct v4l2_ext_plane - extended plane buffer info
> + * @buffer_length:	size of the entire buffer in bytes, should fit
> + *			@offset + @plane_length
> + * @plane_length:	size of the plane in bytes.
> + * @mem_offset:		If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
> + *			that should be passed to mmap() called on the video node.
> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> + *			to this plane.
> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> + *			associated with this plane.
> + * @offset:		offset in the memory buffer where the plane starts.
> + * @memory:		enum v4l2_memory; the method, in which the actual video
> + *			data is passed
> + * @reserved:		extra space reserved for future fields, must be set to 0.
> + *
> + *
> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> + * can have one plane for Y, and another for interleaved CbCr components.
> + * Each plane can reside in a separate memory buffer, or even in
> + * a completely separate memory node (e.g. in embedded devices).
> + */
> +struct v4l2_ext_plane {
> +	__u32 buffer_length;
> +	__u32 plane_length;
> +	union {
> +		__u32 mem_offset;
> +		__u64 userptr;
> +		__s32 dmabuf_fd;
> +	} m;
> +	__u32 offset;

I'd rename this plane_offset. I think some reordering would make this struct easier
to understand:

struct v4l2_ext_plane {
	__u32 buffer_length;
	__u32 plane_offset;
	__u32 plane_length;
	__u32 memory;
	union {
		__u32 mem_offset;
		__u64 userptr;
		__s32 dmabuf_fd;
	} m;
	__u32 reserved[4];
};

> +	__u32 memory;
> +	__u32 reserved[4];
> +};

What is not clear is how to tell the different between a single buffer containing
multiple planes, and using a separate buffer per plane. E.g. what would this look
like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
planes are also combined in a single buffer?

I would guess that the m union is set to 0 if the plane is part of the buffer
defined in the previous plane?

> +
>  /**
>   * struct v4l2_buffer - video buffer info
>   * @index:	id number of the buffer
> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>  	};
>  };
>  
> +/**
> + * struct v4l2_ext_buffer - extended video buffer info
> + * @index:	id number of the buffer
> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> + * @flags:	buffer informational flags
> + * @field:	enum v4l2_field; field order of the image in the buffer
> + * @timestamp:	frame timestamp
> + * @sequence:	sequence count of this frame
> + * @planes:	per-plane buffer information
> + * @request_fd:	fd of the request that this buffer should use
> + * @reserved:	extra space reserved for future fields, must be set to 0
> + *
> + * Contains data exchanged by application and driver using one of the Streaming
> + * I/O methods.
> + */
> +struct v4l2_ext_buffer {
> +	__u32 index;
> +	__u32 type;
> +	__u32 field;
> +	__u32 sequence;
> +	__u64 flags;
> +	__u64 timestamp;
> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> +	__s32 request_fd;
> +	__u32 reserved[9];
> +};

Brainstorming:

Some ideas I have to make it easier to support mid stream resolution/colorimetry
changes:

Adding width and height would support resolution changes (requires the use of
CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
information is provided here, then there are no race conditions.

Same for adding the colorimetry fields here, this too can change on the fly (esp.
with HDMI), so reporting this information here avoids race conditions as well.

And thirdly, I would like to have a __u64 boot_timestamp field containing the
CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
is that for m2m devices it is just copied and that for other devices it can have
different meanings depending on the timestamp buffer flags.

There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
to add support for this. That way you know exactly when the driver was finished with
the buffer and that helps in detecting missed frames or instrumentation.

> +
>  #ifndef __KERNEL__
>  /**
>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
>  	__u32			reserved[6];
>  };
>  
> +/**
> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
> + * @index:	on return, index of the first created buffer
> + * @count:	entry: number of requested buffers,
> + *		return: number of created buffers
> + * @memory:	enum v4l2_memory; buffer memory type
> + * @capabilities: capabilities of this buffer type.
> + * @format:	frame format, for which buffers are requested
> + * @flags:	additional buffer management attributes (ignored unless the
> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
> + *		and configured for MMAP streaming I/O).
> + * @reserved:	extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_ext_create_buffers {
> +	__u32				index;
> +	__u32				count;
> +	__u32				memory;
> +	__u32				capabilities;
> +	struct v4l2_ext_pix_format	format;

The reality is that the only field that is ever used in the original v4l2_format
struct is sizeimage. So this can be replaced with:

	__u32				plane_size[VIDEO_MAX_PLANES];

(the field name I picked is debatable, but you get the idea)

The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
is needed for the current format. The original idea of using struct v4l2_format
was that drivers would use the full format information to calculate the
memory size, but that was just much too complicated to implement and nobody
ever used that. Only the sizeimage field was ever used.

> +	__u32				flags;
> +	__u32 reserved[5];
> +};
> +
>  /*
>   *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>   *
> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
>  #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>  #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>  #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
> +#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
> +#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
> +#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
>  
>  /* Reminder: when adding new ioctls please add support for them to
>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
> 

Regards,

	Hans

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-09-09 11:41   ` Hans Verkuil
@ 2020-09-14  2:14     ` Helen Koike
  0 siblings, 0 replies; 51+ messages in thread
From: Helen Koike @ 2020-09-14  2:14 UTC (permalink / raw)
  To: Hans Verkuil, mchehab, laurent.pinchart, sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

Hi Hans,

On 9/9/20 8:41 AM, Hans Verkuil wrote:
> Hi Helen,
> 
> Some review comments, concentrating on the uAPI.
> 
> On 04/08/2020 21:29, Helen Koike wrote:
>> This is part of the multiplanar and singleplanar unification process.
>> v4l2_ext_pix_format is supposed to work for both cases.
>>
>> We also add the concept of modifiers already employed in DRM to expose
>> HW-specific formats (like tiled or compressed formats) and allow
>> exchanging this information with the DRM subsystem in a consistent way.
>>
>> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
>> v4l2_ext_format, other types will be rejected if you use the
>> {G,S,TRY}_EXT_PIX_FMT ioctls.
>>
>> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
>> in drivers, but, in the meantime, the core takes care of converting
>> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
>> still work if the userspace app/lib uses the new ioctls.
>> The conversion is also done the other around to allow userspace
>> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
>> _ext_ hooks.
>>
>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> ---
>> Changes in v5:
>> - change sizes and reorder fields to avoid holes in the struct and make
>>   it the same for 32 and 64 bits
>> - removed __attribute__ ((packed)) from uapi structs
>> - Fix doc warning from make htmldocs
>> - Updated commit message with EXT_PIX prefix for the ioctls.
>>
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Add reserved fields
>> - Removed num_planes from struct v4l2_ext_pix_format
>> - Removed flag field from struct v4l2_ext_pix_format, since the only
>>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>>   where we can use modifiers, or add it back later through the reserved
>>   bits.
>> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>>   != MOD_INVALID
>> - Fix type assignment in v4l_g_fmt_ext_pix()
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Move the modifier in v4l2_ext_format (was formerly placed in
>>   v4l2_ext_plane)
>> - Fix a few bugs in the converters and add a strict parameter to
>>   allow conversion of uninitialized/mis-initialized objects
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>>  include/media/v4l2-ioctl.h           |  34 ++
>>  include/uapi/linux/videodev2.h       |  56 +++
>>  4 files changed, 615 insertions(+), 81 deletions(-)
>>
> 
> <snip>
> 
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index c7b70ff53bc1d..7123c6a4d9569 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>>  	__u8				reserved[7];
>>  } __attribute__ ((packed));
>>  
>> +/**
>> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
>> + * @sizeimage:		maximum size in bytes required for data, for which
>> + *			this plane will be used.
>> + *			Should be set to zero for unused planes.
>> + * @bytesperline:	distance in bytes between the leftmost pixels in two
>> + *			adjacent lines
>> + * @reserved:		extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_plane_ext_pix_format {
>> +	__u32 sizeimage;
>> +	__u32 bytesperline;
>> +	__u32 reserved;
> 
> I'd use reserved[4] here.
> 
>> +};
>> +
>> +/**
>> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
>> + * @type:		type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
>> + *			V4L2_BUF_TYPE_VIDEO_OUTPUT
>> + * @width:		image width in pixels
>> + * @height:		image height in pixels
>> + * @field:		enum v4l2_field; field order (for interlaced video)
>> + * @pixelformat:	little endian four character code (fourcc)
>> + * @modifier:		modifier applied to the format (used for tiled formats
>> + *			and other kind of HW-specific formats, like compressed
>> + *			formats)
> 
> This should refer to the drm.h header since we're reusing the DRM modifiers.
> 
>> + * @colorspace:		enum v4l2_colorspace; supplemental to pixelformat
>> + * @plane_fmt:		per-plane information
>> + * @ycbcr_enc:		enum v4l2_ycbcr_encoding, Y'CbCr encoding
>> + * @hsv_enc:		enum v4l2_hsv_encoding, HSV encoding
>> + * @quantization:	enum v4l2_quantization, colorspace quantization
>> + * @xfer_func:		enum v4l2_xfer_func, colorspace transfer function
>> + * @reserved:		extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_ext_pix_format {
>> +	__u32 type;
>> +	__u32 width;
>> +	__u32 height;
>> +	__u32 field;
>> +	__u64 modifier;
>> +	__u32 pixelformat;
>> +	__u32 colorspace;
>> +	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
>> +	union {
>> +		__u32 ycbcr_enc;
>> +		__u32 hsv_enc;
>> +	};
>> +	__u32 quantization;
>> +	__u32 xfer_func;
> 
> I'd reorder this:
> 
> 	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
> 	__u32 pixelformat;
> 	__u32 colorspace;
> 	__u32 xfer_func;
> 	union {
> 		__u32 ycbcr_enc;
> 		__u32 hsv_enc;
> 	};
> 	__u32 quantization;
> 
> The reason for reordering is that I like to keep the colorimetry fields in
> that order since that is how these fields are processed mathematically: you
> apply the colorspace matrix first, then the transfer function, then optionally
> convert to Y'CbCr or HSV and finally you quantize the result.
> 
> There is also a __u32 flags; field missing (needed for V4L2_PIX_FMT_FLAG_PREMUL_ALPHA
> and for the upcoming CSC support).

We discussed this on v4 https://patchwork.linuxtv.org/project/linux-media/cover/20200717115435.2632623-1-helen.koike@collabora.com/

The idea is to replace V4L2_PIX_FMT_FLAG_PREMUL_ALPHA by a modifier, and this API
won't need CSC, since the colorspace fields are read-write.

Regards,
Helen

> 
>> +	__u32 reserved[9];
>> +};
>> +
>>  /**
>>   * struct v4l2_sdr_format - SDR format definition
>>   * @pixelformat:	little endian four character code (fourcc)
>> @@ -2571,6 +2623,10 @@ struct v4l2_create_buffers {
>>  
>>  #define VIDIOC_QUERY_EXT_CTRL	_IOWR('V', 103, struct v4l2_query_ext_ctrl)
>>  
>> +#define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>> +#define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>> +#define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>> +
>>  /* Reminder: when adding new ioctls please add support for them to
>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>  
>>
> 
> Regards,
> 
> 	Hans
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-08-04 19:29 ` [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
  2020-08-14  7:49   ` Alexandre Courbot
  2020-09-09 11:41   ` Hans Verkuil
@ 2020-10-02 19:49   ` Tomasz Figa
  2020-11-14 14:21     ` Helen Koike
  2 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-10-02 19:49 UTC (permalink / raw)
  To: Helen Koike
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, Boris Brezillon, hiroh, nicolas, Brian.Starkey,
	kernel, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Hi Helen,

On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
> This is part of the multiplanar and singleplanar unification process.
> v4l2_ext_pix_format is supposed to work for both cases.
> 
> We also add the concept of modifiers already employed in DRM to expose
> HW-specific formats (like tiled or compressed formats) and allow
> exchanging this information with the DRM subsystem in a consistent way.
> 
> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
> v4l2_ext_format, other types will be rejected if you use the
> {G,S,TRY}_EXT_PIX_FMT ioctls.
> 
> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
> in drivers, but, in the meantime, the core takes care of converting
> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
> still work if the userspace app/lib uses the new ioctls.
> The conversion is also done the other around to allow userspace
> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
> _ext_ hooks.
> 

Thank you for the patch. Please see my comments inline.

> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v5:
> - change sizes and reorder fields to avoid holes in the struct and make
>   it the same for 32 and 64 bits
> - removed __attribute__ ((packed)) from uapi structs
> - Fix doc warning from make htmldocs
> - Updated commit message with EXT_PIX prefix for the ioctls.
> 
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Add reserved fields
> - Removed num_planes from struct v4l2_ext_pix_format
> - Removed flag field from struct v4l2_ext_pix_format, since the only
>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>   where we can use modifiers, or add it back later through the reserved
>   bits.
> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>   != MOD_INVALID
> - Fix type assignment in v4l_g_fmt_ext_pix()
> - Rebased on top of media/master (post 5.8-rc1)
> 
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
> 
> Changes in v2:
> - Move the modifier in v4l2_ext_format (was formerly placed in
>   v4l2_ext_plane)
> - Fix a few bugs in the converters and add a strict parameter to
>   allow conversion of uninitialized/mis-initialized objects
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>  include/media/v4l2-ioctl.h           |  34 ++
>  include/uapi/linux/videodev2.h       |  56 +++
>  4 files changed, 615 insertions(+), 81 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index a593ea0598b55..e1829906bc086 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
>  			set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
>  		if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
>  			       ops->vidioc_g_fmt_vid_cap_mplane ||
> +			       ops->vidioc_g_ext_pix_fmt_vid_cap ||
>  			       ops->vidioc_g_fmt_vid_overlay)) ||
>  		    (is_tx && (ops->vidioc_g_fmt_vid_out ||
>  			       ops->vidioc_g_fmt_vid_out_mplane ||
> -			       ops->vidioc_g_fmt_vid_out_overlay)))
> +			       ops->vidioc_g_ext_pix_fmt_vid_out ||
> +			       ops->vidioc_g_fmt_vid_out_overlay))) {
>  			 set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
> +			 set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);

Is it expected to allow the new ioctls for drivers which implement the old
vid_out_overlay callbacks?

> +		}
>  		if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
>  			       ops->vidioc_s_fmt_vid_cap_mplane ||
> +			       ops->vidioc_s_ext_pix_fmt_vid_cap ||
>  			       ops->vidioc_s_fmt_vid_overlay)) ||
>  		    (is_tx && (ops->vidioc_s_fmt_vid_out ||
>  			       ops->vidioc_s_fmt_vid_out_mplane ||
> -			       ops->vidioc_s_fmt_vid_out_overlay)))
> +			       ops->vidioc_s_ext_pix_fmt_vid_out ||
> +			       ops->vidioc_s_fmt_vid_out_overlay))) {
>  			 set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
> +			 set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
> +		}
>  		if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
>  			       ops->vidioc_try_fmt_vid_cap_mplane ||
> +			       ops->vidioc_try_ext_pix_fmt_vid_cap ||
>  			       ops->vidioc_try_fmt_vid_overlay)) ||
>  		    (is_tx && (ops->vidioc_try_fmt_vid_out ||
>  			       ops->vidioc_try_fmt_vid_out_mplane ||
> -			       ops->vidioc_try_fmt_vid_out_overlay)))
> +			       ops->vidioc_try_ext_pix_fmt_vid_out ||
> +			       ops->vidioc_try_fmt_vid_out_overlay))) {
>  			 set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
> +			 set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
> +		}
>  		SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
>  		SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
>  		SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
> @@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
>  		/* touch specific ioctls */
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
> +		SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
> +		SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
> +		SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index a556880f225a5..14a0def50f8ea 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -17,6 +17,8 @@
>  
>  #include <linux/videodev2.h>
>  
> +#include <drm/drm_fourcc.h>
> +
>  #include <media/v4l2-common.h>
>  #include <media/v4l2-ioctl.h>
>  #include <media/v4l2-ctrls.h>
> @@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
>  	}
>  }
>  
> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
> +{
> +	const struct v4l2_ext_pix_format *pix = arg;
> +	unsigned int i;
> +
> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
> +		prt_names(pix->type, v4l2_type_names),
> +		pix->width, pix->height,
> +		(pix->pixelformat & 0xff),
> +		(pix->pixelformat >>  8) & 0xff,
> +		(pix->pixelformat >> 16) & 0xff,
> +		(pix->pixelformat >> 24) & 0xff,
> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
> +		pix->colorspace, pix->ycbcr_enc,
> +		pix->quantization, pix->xfer_func);
> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)

This is going to print 8 lines every time. Maybe we could skip 0-sized
planes or something?

> +		pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
> +			 i, pix->plane_fmt[i].bytesperline,
> +			 pix->plane_fmt[i].sizeimage);
> +}
> +
>  static void v4l_print_framebuffer(const void *arg, bool write_only)
>  {
>  	const struct v4l2_framebuffer *p = arg;
> @@ -964,11 +987,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>  	switch (type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>  		if ((is_vid || is_tch) && is_rx &&
> -		    (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
> +		    (ops->vidioc_g_fmt_vid_cap ||
> +		     ops->vidioc_g_ext_pix_fmt_vid_cap ||
> +		     ops->vidioc_g_fmt_vid_cap_mplane))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
> +		if ((is_vid || is_tch) && is_rx &&
> +		    (ops->vidioc_g_fmt_vid_cap_mplane ||
> +		     ops->vidioc_g_ext_pix_fmt_vid_cap))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
> @@ -977,11 +1004,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>  		if (is_vid && is_tx &&
> -		    (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
> +		    (ops->vidioc_g_fmt_vid_out ||
> +		     ops->vidioc_g_ext_pix_fmt_vid_out ||
> +		     ops->vidioc_g_fmt_vid_out_mplane))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
> +		if (is_vid && is_tx &&
> +		    (ops->vidioc_g_ext_pix_fmt_vid_out ||
> +		     ops->vidioc_g_fmt_vid_out_mplane))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
> @@ -1061,6 +1092,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>  	       sizeof(fmt->fmt.pix) - offset);
>  }
>  
> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> +				  struct v4l2_format *f, bool mplane_cap,
> +				  bool strict)
> +{
> +	const struct v4l2_plane_ext_pix_format *pe;
> +	struct v4l2_plane_pix_format *p;
> +	unsigned int i;
> +
> +	memset(f, 0, sizeof(*f));
> +
> +	/*
> +	 * Make sure no modifier is required before doing the
> +	 * conversion.
> +	 */
> +	if (e->modifier && strict &&

Do we need the explicit check for e->modifier != 0 if we have to check for
the 2 specific values below anyway?

> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
> +		return -EINVAL;
> +
> +	if (!e->plane_fmt[0].sizeimage && strict)
> +		return -EINVAL;

Why is this incorrect?

> +
> +	if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
> +		return 0;

Again this seems to be different from what we discussed before. In the ext
API, the planes would mean color planes and would be all filled in with the
right values. So for this conversion, if !mplane cap, then we should check
if bytesperline of planes >= 1 match the format definition and use the sum
of all planes sizeimage as the sizeimage of the legacy struct.

> +
> +	if (!mplane_cap) {
> +		f->fmt.pix.width = e->width;
> +		f->fmt.pix.height = e->height;
> +		f->fmt.pix.pixelformat = e->pixelformat;
> +		f->fmt.pix.field = e->field;
> +		f->fmt.pix.colorspace = e->colorspace;
> +		f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
> +		f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +		f->fmt.pix.quantization = e->quantization;
> +		pe = &e->plane_fmt[0];
> +		f->fmt.pix.bytesperline = pe->bytesperline;
> +		f->fmt.pix.sizeimage = pe->sizeimage;

See above for how these two should be filled in.

> +		f->type = e->type;
> +		return 0;
> +	}
> +
> +	f->fmt.pix_mp.width = e->width;
> +	f->fmt.pix_mp.height = e->height;
> +	f->fmt.pix_mp.pixelformat = e->pixelformat;
> +	f->fmt.pix_mp.field = e->field;
> +	f->fmt.pix_mp.colorspace = e->colorspace;
> +	f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
> +	f->fmt.pix_mp.quantization = e->quantization;
> +	if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> +	else
> +		f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> +
> +	for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> +		pe = &e->plane_fmt[i];
> +		p = &f->fmt.pix_mp.plane_fmt[i];
> +		p->bytesperline = pe->bytesperline;
> +		p->sizeimage = pe->sizeimage;

This is similar to the above, but the behavior depends on whether the
pixelformat is an M or non-M variant.

> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
> +
> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> +				  struct v4l2_ext_pix_format *e, bool strict)

In other functions "ef" is used for the extended format. Let's make it
consistent.

> +{
> +	const struct v4l2_plane_pix_format *p;
> +	struct v4l2_plane_ext_pix_format *pe;
> +	unsigned int i;
> +
> +	memset(e, 0, sizeof(*e));
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		e->width = f->fmt.pix.width;
> +		e->height = f->fmt.pix.height;
> +		e->pixelformat = f->fmt.pix.pixelformat;
> +		e->field = f->fmt.pix.field;
> +		e->colorspace = f->fmt.pix.colorspace;
> +		if (f->fmt.pix.flags)
> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
> +				f->fmt.pix.flags);

Would it make sense to print something like video node name and/or function
name to explain where this warning comes from?

> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
> +		e->quantization = f->fmt.pix.quantization;

Missing xfer_func?

> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;

This doesn't look right. In the ext API we expected the planes to describe
color planes, which means that bytesperline needs to be computed for planes
>= 1 and sizeimage replaced with per-plane sizes, according to the
>pixelformat.

> +		e->type = f->type;
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
> +		     !f->fmt.pix_mp.num_planes) && strict)
> +			return -EINVAL;
> +
> +		e->width = f->fmt.pix_mp.width;
> +		e->height = f->fmt.pix_mp.height;
> +		e->pixelformat = f->fmt.pix_mp.pixelformat;
> +		e->field = f->fmt.pix_mp.field;
> +		e->colorspace = f->fmt.pix_mp.colorspace;
> +		if (f->fmt.pix.flags)
> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
> +				f->fmt.pix.flags);
> +		e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
> +		e->quantization = f->fmt.pix_mp.quantization;

Missing xfer_func?

> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +		else
> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +
> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> +			pe = &e->plane_fmt[i];
> +			p = &f->fmt.pix_mp.plane_fmt[i];
> +			pe->bytesperline = p->bytesperline;
> +			pe->sizeimage = p->sizeimage;
> +		}

Same here. A blind copy is not enough. For non-M formats, the plane
parameters need to be filled according to the pixelformat.

> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
> +
>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1564,6 +1723,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
>  	p->xfer_func = 0;
>  }
>  
> +static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh,
> +			     struct v4l2_format *f)
> +{
> +	struct v4l2_ext_pix_format ef;
> +	int ret;
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +		ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +		ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +		ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_ext_pix_format_to_format(&ef, f,
> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +					     true);
> +}
> +
>  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1600,17 +1791,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  
>  	switch (p->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -		if (unlikely(!ops->vidioc_g_fmt_vid_cap))
> -			break;
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
> -			v4l_pix_format_touch(&p->fmt.pix);
> -		return ret;
> +		if (ops->vidioc_g_fmt_vid_cap) {
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;
> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
> +			ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> +		if (ops->vidioc_g_fmt_vid_cap_mplane)
> +			return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> +		else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>  		return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
>  	case V4L2_BUF_TYPE_VBI_CAPTURE:
> @@ -1618,15 +1819,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  	case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
>  		return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -		if (unlikely(!ops->vidioc_g_fmt_vid_out))
> -			break;
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		return ret;
> +		if (ops->vidioc_g_fmt_vid_out) {
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			return ret;
> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> +		if (ops->vidioc_g_fmt_vid_out_mplane)
> +			return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> +		else if (ops->vidioc_g_ext_pix_fmt_vid_out)
> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>  		return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
>  	case V4L2_BUF_TYPE_VBI_OUTPUT:
> @@ -1645,6 +1853,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  	return -EINVAL;
>  }
>  
> +static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh, void *arg)
> +{
> +	struct v4l2_ext_pix_format *ef = arg;
> +	struct v4l2_format f = {
> +		.type = ef->type,
> +	};
> +	int ret;
> +
> +	ret = check_fmt(file, ef->type);
> +	if (ret)
> +		return ret;
> +
> +	memset(ef, 0, sizeof(*ef));
> +	ef->type = f.type;
> +
> +	switch (f.type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +		if (ops->vidioc_g_ext_pix_fmt_vid_cap)
> +			return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
> +		break;
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		if (ops->vidioc_g_ext_pix_fmt_vid_out)
> +			return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	ret = v4l_g_fmt(ops, file, fh, &f);
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
> +static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh,
> +			     struct v4l2_format *f)
> +{
> +	struct v4l2_ext_pix_format ef;
> +	int ret;
> +
> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
> +	if (ret)
> +		return ret;
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +		ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_ext_pix_format_to_format(&ef, f,
> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +					     true);
> +}
> +
>  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1663,23 +1941,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  
>  	switch (p->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap))
> +		if (ops->vidioc_s_fmt_vid_cap) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
> +			ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		} else {
>  			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +		}
> +
>  		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>  			v4l_pix_format_touch(&p->fmt.pix);
>  		return ret;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> +		if (ops->vidioc_s_fmt_vid_cap_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>  		if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
>  			break;
> @@ -1696,21 +1982,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>  		return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_out))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		return ret;
> +		if (ops->vidioc_s_fmt_vid_out) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			return ret;
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
> +		if (ops->vidioc_s_fmt_vid_out_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>  		if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
>  			break;
> @@ -1750,6 +2042,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  	return -EINVAL;
>  }
>  
> +static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh, void *arg)
> +{
> +	struct video_device *vfd = video_devdata(file);
> +	struct v4l2_ext_pix_format *ef = arg;
> +	struct v4l2_format f;
> +	int ret;
> +
> +	ret = check_fmt(file, ef->type);
> +	if (ret)
> +		return ret;
> +

Should we zero the reserved fields as it's done with the current structs?

> +	switch (ef->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +		if (ops->vidioc_s_ext_pix_fmt_vid_cap)
> +			return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
> +		break;
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		if (ops->vidioc_s_ext_pix_fmt_vid_out)
> +			return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
> +					    vfd->device_caps &
> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
> +					    false);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l_s_fmt(ops, file, fh, &f);
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
> +static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +			       struct file *file, void *fh,
> +			       struct v4l2_format *f)
> +{
> +	struct v4l2_ext_pix_format ef;
> +	int ret;
> +
> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
> +	if (ret)
> +		return ret;
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +		ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_ext_pix_format_to_format(&ef, f,
> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +					     true);
> +}
> +
> +
>  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1765,23 +2133,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  
>  	switch (p->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
> -			v4l_pix_format_touch(&p->fmt.pix);
> -		return ret;
> +		if (ops->vidioc_try_fmt_vid_cap) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;

Should the 3 lines above be outside of the if/else block?

> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
> +		if (ops->vidioc_try_fmt_vid_cap_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>  		if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
>  			break;
> @@ -1798,21 +2175,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>  		return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_out))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		return ret;
> +		if (ops->vidioc_try_fmt_vid_out) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			return ret;
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
> +		if (ops->vidioc_try_fmt_vid_out_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>  		if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
>  			break;
> @@ -1852,6 +2235,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  	return -EINVAL;
>  }
>  
> +static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +			       struct file *file, void *fh, void *arg)
> +{
> +	struct video_device *vfd = video_devdata(file);
> +	struct v4l2_ext_pix_format *ef = arg;
> +	struct v4l2_format f;
> +	int ret;
> +
> +	ret = check_fmt(file, ef->type);
> +	if (ret)
> +		return ret;

Should we zero the reserved fields as it's done with the current structs?

> +
> +	switch (ef->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +		if (ops->vidioc_try_ext_pix_fmt_vid_cap)
> +			return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
> +								   ef);
> +		break;
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		if (ops->vidioc_try_ext_pix_fmt_vid_out)
> +			return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
> +								   ef);
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
> +					    vfd->device_caps &
> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
> +					    false);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l_try_fmt(ops, file, fh, &f);
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
>  static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -2771,7 +3197,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>  	IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
>  	IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
>  	IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
> +	IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>  	IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
> +	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>  	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
> @@ -2818,6 +3246,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>  	IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
>  	IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
> +	IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>  	IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
>  	IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
>  	IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
> index 86878fba332b0..8bbcb74d8ee31 100644
> --- a/include/media/v4l2-ioctl.h
> +++ b/include/media/v4l2-ioctl.h
> @@ -48,11 +48,17 @@ struct v4l2_fh;
>   * @vidioc_g_fmt_vid_cap: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *	in single plane mode
> + * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *	capture
>   * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_g_fmt_vid_out: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *	in single plane mode
> + * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *	out
>   * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>   * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
> @@ -82,11 +88,16 @@ struct v4l2_fh;
>   * @vidioc_s_fmt_vid_cap: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *	in single plane mode
> + * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *	capture
>   * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_s_fmt_vid_out: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *	in single plane mode
> + * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>   * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>   * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
> @@ -116,11 +127,16 @@ struct v4l2_fh;
>   * @vidioc_try_fmt_vid_cap: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *	in single plane mode
> + * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for
> +	video capture
>   * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_try_fmt_vid_out: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *	in single plane mode
> + * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>   * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   *	output
> @@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
>  	/* VIDIOC_G_FMT handlers */
>  	int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
>  					struct v4l2_format *f);
>  	int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
>  					    struct v4l2_format *f);
>  	int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
>  	/* VIDIOC_S_FMT handlers */
>  	int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
>  					struct v4l2_format *f);
>  	int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
>  					    struct v4l2_format *f);
>  	int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
>  	/* VIDIOC_TRY_FMT handlers */
>  	int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
>  				      struct v4l2_format *f);
> +	int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +					      struct v4l2_ext_pix_format *f);
>  	int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
>  					  struct v4l2_format *f);
>  	int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
>  				      struct v4l2_format *f);
> +	int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +					      struct v4l2_ext_pix_format *f);
>  	int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
>  					     struct v4l2_format *f);
>  	int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
>  long int video_ioctl2(struct file *file,
>  		      unsigned int cmd, unsigned long int arg);
>  
> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> +				  struct v4l2_ext_pix_format *e, bool strict);
> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> +				  struct v4l2_format *f,
> +				  bool mplane_cap, bool strict);
> +
>  /*
>   * The user space interpretation of the 'v4l2_event' differs
>   * based on the 'time_t' definition on 32-bit architectures, so
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index c7b70ff53bc1d..7123c6a4d9569 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>  	__u8				reserved[7];
>  } __attribute__ ((packed));
>  
> +/**
> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
> + * @sizeimage:		maximum size in bytes required for data, for which
> + *			this plane will be used.
> + *			Should be set to zero for unused planes.
> + * @bytesperline:	distance in bytes between the leftmost pixels in two
> + *			adjacent lines
> + * @reserved:		extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_plane_ext_pix_format {

nit: Maybe v4l2_ext_plane_pix_format, since the struct describes an
ext variant of a plane_pix_format?

> +	__u32 sizeimage;
> +	__u32 bytesperline;
> +	__u32 reserved;
> +};

Actually, this seems to be exactly the same as the existing
v4l2_plane_pix_format, except actually having less reserved space. Could it
make sense to just reuse the existing struct?

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-10-02 19:49   ` Tomasz Figa
@ 2020-11-14 14:21     ` Helen Koike
  2020-11-19  5:45       ` Tomasz Figa
  0 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-11-14 14:21 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, Boris Brezillon, hiroh, nicolas, Brian.Starkey,
	kernel, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Hi Tomasz,

On 10/2/20 4:49 PM, Tomasz Figa wrote:
> Hi Helen,
> 
> On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
>> This is part of the multiplanar and singleplanar unification process.
>> v4l2_ext_pix_format is supposed to work for both cases.
>>
>> We also add the concept of modifiers already employed in DRM to expose
>> HW-specific formats (like tiled or compressed formats) and allow
>> exchanging this information with the DRM subsystem in a consistent way.
>>
>> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
>> v4l2_ext_format, other types will be rejected if you use the
>> {G,S,TRY}_EXT_PIX_FMT ioctls.
>>
>> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
>> in drivers, but, in the meantime, the core takes care of converting
>> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
>> still work if the userspace app/lib uses the new ioctls.
>> The conversion is also done the other around to allow userspace
>> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
>> _ext_ hooks.
>>
> 
> Thank you for the patch. Please see my comments inline.

Thanks for reviewing.

> 
>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> ---
>> Changes in v5:
>> - change sizes and reorder fields to avoid holes in the struct and make
>>   it the same for 32 and 64 bits
>> - removed __attribute__ ((packed)) from uapi structs
>> - Fix doc warning from make htmldocs
>> - Updated commit message with EXT_PIX prefix for the ioctls.
>>
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Add reserved fields
>> - Removed num_planes from struct v4l2_ext_pix_format
>> - Removed flag field from struct v4l2_ext_pix_format, since the only
>>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>>   where we can use modifiers, or add it back later through the reserved
>>   bits.
>> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>>   != MOD_INVALID
>> - Fix type assignment in v4l_g_fmt_ext_pix()
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Move the modifier in v4l2_ext_format (was formerly placed in
>>   v4l2_ext_plane)
>> - Fix a few bugs in the converters and add a strict parameter to
>>   allow conversion of uninitialized/mis-initialized objects
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>>  include/media/v4l2-ioctl.h           |  34 ++
>>  include/uapi/linux/videodev2.h       |  56 +++
>>  4 files changed, 615 insertions(+), 81 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>> index a593ea0598b55..e1829906bc086 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>  			set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
>>  		if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
>>  			       ops->vidioc_g_fmt_vid_cap_mplane ||
>> +			       ops->vidioc_g_ext_pix_fmt_vid_cap ||
>>  			       ops->vidioc_g_fmt_vid_overlay)) ||
>>  		    (is_tx && (ops->vidioc_g_fmt_vid_out ||
>>  			       ops->vidioc_g_fmt_vid_out_mplane ||
>> -			       ops->vidioc_g_fmt_vid_out_overlay)))
>> +			       ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +			       ops->vidioc_g_fmt_vid_out_overlay))) {
>>  			 set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
>> +			 set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
> 
> Is it expected to allow the new ioctls for drivers which implement the old
> vid_out_overlay callbacks?

Thanks for noticing it, I don't think so, I'll update this in next version.

> 
>> +		}
>>  		if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
>>  			       ops->vidioc_s_fmt_vid_cap_mplane ||
>> +			       ops->vidioc_s_ext_pix_fmt_vid_cap ||
>>  			       ops->vidioc_s_fmt_vid_overlay)) ||
>>  		    (is_tx && (ops->vidioc_s_fmt_vid_out ||
>>  			       ops->vidioc_s_fmt_vid_out_mplane ||
>> -			       ops->vidioc_s_fmt_vid_out_overlay)))
>> +			       ops->vidioc_s_ext_pix_fmt_vid_out ||
>> +			       ops->vidioc_s_fmt_vid_out_overlay))) {
>>  			 set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
>> +			 set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
>> +		}
>>  		if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
>>  			       ops->vidioc_try_fmt_vid_cap_mplane ||
>> +			       ops->vidioc_try_ext_pix_fmt_vid_cap ||
>>  			       ops->vidioc_try_fmt_vid_overlay)) ||
>>  		    (is_tx && (ops->vidioc_try_fmt_vid_out ||
>>  			       ops->vidioc_try_fmt_vid_out_mplane ||
>> -			       ops->vidioc_try_fmt_vid_out_overlay)))
>> +			       ops->vidioc_try_ext_pix_fmt_vid_out ||
>> +			       ops->vidioc_try_fmt_vid_out_overlay))) {
>>  			 set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
>> +			 set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
>> +		}
>>  		SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
>>  		SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
>>  		SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
>> @@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>  		/* touch specific ioctls */
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
>> +		SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
>> +		SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
>> +		SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index a556880f225a5..14a0def50f8ea 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -17,6 +17,8 @@
>>  
>>  #include <linux/videodev2.h>
>>  
>> +#include <drm/drm_fourcc.h>
>> +
>>  #include <media/v4l2-common.h>
>>  #include <media/v4l2-ioctl.h>
>>  #include <media/v4l2-ctrls.h>
>> @@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
>>  	}
>>  }
>>  
>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
>> +{
>> +	const struct v4l2_ext_pix_format *pix = arg;
>> +	unsigned int i;
>> +
>> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
>> +		prt_names(pix->type, v4l2_type_names),
>> +		pix->width, pix->height,
>> +		(pix->pixelformat & 0xff),
>> +		(pix->pixelformat >>  8) & 0xff,
>> +		(pix->pixelformat >> 16) & 0xff,
>> +		(pix->pixelformat >> 24) & 0xff,
>> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
>> +		pix->colorspace, pix->ycbcr_enc,
>> +		pix->quantization, pix->xfer_func);
>> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
> 
> This is going to print 8 lines every time. Maybe we could skip 0-sized
> planes or something?

I'm already checking pix->plane_fmt[i].sizeimage in the loop, it shouldn't
print 8 lines every time.

> 
>> +		pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
>> +			 i, pix->plane_fmt[i].bytesperline,
>> +			 pix->plane_fmt[i].sizeimage);
>> +}
>> +
>>  static void v4l_print_framebuffer(const void *arg, bool write_only)
>>  {
>>  	const struct v4l2_framebuffer *p = arg;
>> @@ -964,11 +987,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>  	switch (type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>>  		if ((is_vid || is_tch) && is_rx &&
>> -		    (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
>> +		    (ops->vidioc_g_fmt_vid_cap ||
>> +		     ops->vidioc_g_ext_pix_fmt_vid_cap ||
>> +		     ops->vidioc_g_fmt_vid_cap_mplane))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
>> +		if ((is_vid || is_tch) && is_rx &&
>> +		    (ops->vidioc_g_fmt_vid_cap_mplane ||
>> +		     ops->vidioc_g_ext_pix_fmt_vid_cap))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>> @@ -977,11 +1004,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>>  		if (is_vid && is_tx &&
>> -		    (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
>> +		    (ops->vidioc_g_fmt_vid_out ||
>> +		     ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +		     ops->vidioc_g_fmt_vid_out_mplane))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
>> +		if (is_vid && is_tx &&
>> +		    (ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +		     ops->vidioc_g_fmt_vid_out_mplane))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>> @@ -1061,6 +1092,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>  	       sizeof(fmt->fmt.pix) - offset);
>>  }
>>  
>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>> +				  struct v4l2_format *f, bool mplane_cap,
>> +				  bool strict)
>> +{
>> +	const struct v4l2_plane_ext_pix_format *pe;
>> +	struct v4l2_plane_pix_format *p;
>> +	unsigned int i;
>> +
>> +	memset(f, 0, sizeof(*f));
>> +
>> +	/*
>> +	 * Make sure no modifier is required before doing the
>> +	 * conversion.
>> +	 */
>> +	if (e->modifier && strict &&
> 
> Do we need the explicit check for e->modifier != 0 if we have to check for
> the 2 specific values below anyway?

We don't, since DRM_FORMAT_MOD_LINEAR is zero.

But I wanted to make it explicit we don't support modifiers in this conversion.
But I can remove this check, no problem.

> 
>> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
>> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
>> +		return -EINVAL;
>> +
>> +	if (!e->plane_fmt[0].sizeimage && strict)
>> +		return -EINVAL;
> 
> Why is this incorrect?

!sizeimage for the first plane means that there are no planes in ef.
strict is true if the result for the conversion should be returned to userspace
and it is not some internal handling.

So if there are no planes, we would return an incomplete v4l2_format struct
to userspace.

But this is not very clear, I'll improve this for the next version.

> 
>> +
>> +	if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
>> +		return 0;
> 
> Again this seems to be different from what we discussed before. In the ext
> API, the planes would mean color planes and would be all filled in with the
> right values. So for this conversion, if !mplane cap, then we should check
> if bytesperline of planes >= 1 match the format definition and use the sum
> of all planes sizeimage as the sizeimage of the legacy struct.

Agreed, I'll update for next version.

> 
>> +
>> +	if (!mplane_cap) {
>> +		f->fmt.pix.width = e->width;
>> +		f->fmt.pix.height = e->height;
>> +		f->fmt.pix.pixelformat = e->pixelformat;
>> +		f->fmt.pix.field = e->field;
>> +		f->fmt.pix.colorspace = e->colorspace;
>> +		f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
>> +		f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +		f->fmt.pix.quantization = e->quantization;
>> +		pe = &e->plane_fmt[0];
>> +		f->fmt.pix.bytesperline = pe->bytesperline;
>> +		f->fmt.pix.sizeimage = pe->sizeimage;
> 
> See above for how these two should be filled in.

Ack.

> 
>> +		f->type = e->type;
>> +		return 0;
>> +	}
>> +
>> +	f->fmt.pix_mp.width = e->width;
>> +	f->fmt.pix_mp.height = e->height;
>> +	f->fmt.pix_mp.pixelformat = e->pixelformat;
>> +	f->fmt.pix_mp.field = e->field;
>> +	f->fmt.pix_mp.colorspace = e->colorspace;
>> +	f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
>> +	f->fmt.pix_mp.quantization = e->quantization;
>> +	if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> +		f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>> +	else
>> +		f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
>> +
>> +	for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>> +		pe = &e->plane_fmt[i];
>> +		p = &f->fmt.pix_mp.plane_fmt[i];
>> +		p->bytesperline = pe->bytesperline;
>> +		p->sizeimage = pe->sizeimage;
> 
> This is similar to the above, but the behavior depends on whether the
> pixelformat is an M or non-M variant.

Ack.

> 
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
>> +
>> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>> +				  struct v4l2_ext_pix_format *e, bool strict)
> 
> In other functions "ef" is used for the extended format. Let's make it
> consistent.

Ack.

> 
>> +{
>> +	const struct v4l2_plane_pix_format *p;
>> +	struct v4l2_plane_ext_pix_format *pe;
>> +	unsigned int i;
>> +
>> +	memset(e, 0, sizeof(*e));
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		e->width = f->fmt.pix.width;
>> +		e->height = f->fmt.pix.height;
>> +		e->pixelformat = f->fmt.pix.pixelformat;
>> +		e->field = f->fmt.pix.field;
>> +		e->colorspace = f->fmt.pix.colorspace;
>> +		if (f->fmt.pix.flags)
>> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
>> +				f->fmt.pix.flags);
> 
> Would it make sense to print something like video node name and/or function
> name to explain where this warning comes from?

I would need to update the function to receive this information, I can try but
I'm not sure if it is worthy.

> 
>> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
>> +		e->quantization = f->fmt.pix.quantization;
> 
> Missing xfer_func?

Yes, thanks for catching this.

> 
>> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
> 
> This doesn't look right. In the ext API we expected the planes to describe
> color planes, which means that bytesperline needs to be computed for planes
>> = 1 and sizeimage replaced with per-plane sizes, according to the
>> pixelformat.

Ack.

Just to be clear, even if we are using a planar format that isn't a V4L2_PIX_FMT_*M
variant, we should describe every plane separatly.

For instance, if V4L2_PIX_FMT_YVU420 is being used, then f->fmt.pix.bytesperline
will have data, and we need to calculate bytesperline for all 3 planes, so we'll fill
out:

f->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
f->plane_fmt[1].bytesperline = f->fmt.pix.bytesperline / hdiv;
f->plane_fmt[2].bytesperline = f->fmt.pix.bytesperline / hdiv;

I'll update this for the next version.

> 
>> +		e->type = f->type;
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
>> +		     !f->fmt.pix_mp.num_planes) && strict)
>> +			return -EINVAL;
>> +
>> +		e->width = f->fmt.pix_mp.width;
>> +		e->height = f->fmt.pix_mp.height;
>> +		e->pixelformat = f->fmt.pix_mp.pixelformat;
>> +		e->field = f->fmt.pix_mp.field;
>> +		e->colorspace = f->fmt.pix_mp.colorspace;
>> +		if (f->fmt.pix.flags)
>> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
>> +				f->fmt.pix.flags);
>> +		e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
>> +		e->quantization = f->fmt.pix_mp.quantization;
> 
> Missing xfer_func?

Yes, thanks for catching this.

> 
>> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +		else
>> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +
>> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>> +			pe = &e->plane_fmt[i];
>> +			p = &f->fmt.pix_mp.plane_fmt[i];
>> +			pe->bytesperline = p->bytesperline;
>> +			pe->sizeimage = p->sizeimage;
>> +		}
> 
> Same here. A blind copy is not enough. For non-M formats, the plane
> parameters need to be filled according to the pixelformat.


Right, following the idea above, we need a different handling if we
aren't using a M-variant of the pixelformat, and we also need to
convert the pixelformat from the M-variant to non-M-variant.

I'll also need to save that the original format was a
M-variant or not, so I can convert it back as expected.

I'll change this and submit for review.


> 
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>> +
>>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1564,6 +1723,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
>>  	p->xfer_func = 0;
>>  }
>>  
>> +static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh,
>> +			     struct v4l2_format *f)
>> +{
>> +	struct v4l2_ext_pix_format ef;
>> +	int ret;
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +		ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +		ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +		ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_ext_pix_format_to_format(&ef, f,
>> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +					     true);
>> +}
>> +
>>  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1600,17 +1791,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  
>>  	switch (p->type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -		if (unlikely(!ops->vidioc_g_fmt_vid_cap))
>> -			break;
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> -			v4l_pix_format_touch(&p->fmt.pix);
>> -		return ret;
>> +		if (ops->vidioc_g_fmt_vid_cap) {
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
>> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
>> +			ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
>> +		if (ops->vidioc_g_fmt_vid_cap_mplane)
>> +			return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
>> +		else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
>> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>  		return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VBI_CAPTURE:
>> @@ -1618,15 +1819,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  	case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
>>  		return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -		if (unlikely(!ops->vidioc_g_fmt_vid_out))
>> -			break;
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		return ret;
>> +		if (ops->vidioc_g_fmt_vid_out) {
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			return ret;
>> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
>> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
>> +		if (ops->vidioc_g_fmt_vid_out_mplane)
>> +			return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
>> +		else if (ops->vidioc_g_ext_pix_fmt_vid_out)
>> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>  		return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VBI_OUTPUT:
>> @@ -1645,6 +1853,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  	return -EINVAL;
>>  }
>>  
>> +static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh, void *arg)
>> +{
>> +	struct v4l2_ext_pix_format *ef = arg;
>> +	struct v4l2_format f = {
>> +		.type = ef->type,
>> +	};
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ef->type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	memset(ef, 0, sizeof(*ef));
>> +	ef->type = f.type;
>> +
>> +	switch (f.type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +		if (ops->vidioc_g_ext_pix_fmt_vid_cap)
>> +			return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
>> +		break;
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		if (ops->vidioc_g_ext_pix_fmt_vid_out)
>> +			return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l_g_fmt(ops, file, fh, &f);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>> +static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh,
>> +			     struct v4l2_format *f)
>> +{
>> +	struct v4l2_ext_pix_format ef;
>> +	int ret;
>> +
>> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +		ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_ext_pix_format_to_format(&ef, f,
>> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +					     true);
>> +}
>> +
>>  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1663,23 +1941,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  
>>  	switch (p->type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap))
>> +		if (ops->vidioc_s_fmt_vid_cap) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
>> +			ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		} else {
>>  			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +		}
>> +
>>  		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>>  			v4l_pix_format_touch(&p->fmt.pix);
>>  		return ret;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
>> +		if (ops->vidioc_s_fmt_vid_cap_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
>> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>  		if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
>>  			break;
>> @@ -1696,21 +1982,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>>  		return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_out))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		return ret;
>> +		if (ops->vidioc_s_fmt_vid_out) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			return ret;
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
>> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
>> +		if (ops->vidioc_s_fmt_vid_out_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
>> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>  		if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
>>  			break;
>> @@ -1750,6 +2042,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  	return -EINVAL;
>>  }
>>  
>> +static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh, void *arg)
>> +{
>> +	struct video_device *vfd = video_devdata(file);
>> +	struct v4l2_ext_pix_format *ef = arg;
>> +	struct v4l2_format f;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ef->type);
>> +	if (ret)
>> +		return ret;
>> +
> 
> Should we zero the reserved fields as it's done with the current structs?

Yes, I'll update this.

> 
>> +	switch (ef->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +		if (ops->vidioc_s_ext_pix_fmt_vid_cap)
>> +			return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
>> +		break;
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		if (ops->vidioc_s_ext_pix_fmt_vid_out)
>> +			return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
>> +					    vfd->device_caps &
>> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
>> +					    false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = v4l_s_fmt(ops, file, fh, &f);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>> +static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +			       struct file *file, void *fh,
>> +			       struct v4l2_format *f)
>> +{
>> +	struct v4l2_ext_pix_format ef;
>> +	int ret;
>> +
>> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +		ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_ext_pix_format_to_format(&ef, f,
>> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +					     true);
>> +}
>> +
>> +
>>  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1765,23 +2133,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  
>>  	switch (p->type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> -			v4l_pix_format_touch(&p->fmt.pix);
>> -		return ret;
>> +		if (ops->vidioc_try_fmt_vid_cap) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
> 
> Should the 3 lines above be outside of the if/else block?

I can move them out, I just need to add an else condition for the break below.
I'll update this in the next version.

> 
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
>> +		if (ops->vidioc_try_fmt_vid_cap_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>  		if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
>>  			break;
>> @@ -1798,21 +2175,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>>  		return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_out))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		return ret;
>> +		if (ops->vidioc_try_fmt_vid_out) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			return ret;
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
>> +		if (ops->vidioc_try_fmt_vid_out_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>  		if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
>>  			break;
>> @@ -1852,6 +2235,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  	return -EINVAL;
>>  }
>>  
>> +static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +			       struct file *file, void *fh, void *arg)
>> +{
>> +	struct video_device *vfd = video_devdata(file);
>> +	struct v4l2_ext_pix_format *ef = arg;
>> +	struct v4l2_format f;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ef->type);
>> +	if (ret)
>> +		return ret;
> 
> Should we zero the reserved fields as it's done with the current structs?

Yes, I'll update this in the next version.

> 
>> +
>> +	switch (ef->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +		if (ops->vidioc_try_ext_pix_fmt_vid_cap)
>> +			return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
>> +								   ef);
>> +		break;
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		if (ops->vidioc_try_ext_pix_fmt_vid_out)
>> +			return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
>> +								   ef);
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
>> +					    vfd->device_caps &
>> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
>> +					    false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = v4l_try_fmt(ops, file, fh, &f);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>>  static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -2771,7 +3197,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>  	IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
>>  	IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
>>  	IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
>> +	IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>>  	IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
>> +	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>>  	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>> @@ -2818,6 +3246,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>  	IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
>>  	IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
>> +	IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>>  	IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
>>  	IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
>>  	IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
>> index 86878fba332b0..8bbcb74d8ee31 100644
>> --- a/include/media/v4l2-ioctl.h
>> +++ b/include/media/v4l2-ioctl.h
>> @@ -48,11 +48,17 @@ struct v4l2_fh;
>>   * @vidioc_g_fmt_vid_cap: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *	in single plane mode
>> + * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *	capture
>>   * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_g_fmt_vid_out: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *	in single plane mode
>> + * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *	out
>>   * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>>   * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
>> @@ -82,11 +88,16 @@ struct v4l2_fh;
>>   * @vidioc_s_fmt_vid_cap: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *	in single plane mode
>> + * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *	capture
>>   * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_s_fmt_vid_out: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *	in single plane mode
>> + * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>>   * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
>> @@ -116,11 +127,16 @@ struct v4l2_fh;
>>   * @vidioc_try_fmt_vid_cap: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *	in single plane mode
>> + * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for
>> +	video capture
>>   * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_try_fmt_vid_out: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *	in single plane mode
>> + * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   *	output
>> @@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
>>  	/* VIDIOC_G_FMT handlers */
>>  	int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
>>  					struct v4l2_format *f);
>>  	int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
>>  					    struct v4l2_format *f);
>>  	int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
>>  	/* VIDIOC_S_FMT handlers */
>>  	int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
>>  					struct v4l2_format *f);
>>  	int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
>>  					    struct v4l2_format *f);
>>  	int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
>>  	/* VIDIOC_TRY_FMT handlers */
>>  	int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
>>  				      struct v4l2_format *f);
>> +	int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +					      struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
>>  					  struct v4l2_format *f);
>>  	int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
>>  				      struct v4l2_format *f);
>> +	int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +					      struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
>>  					     struct v4l2_format *f);
>>  	int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
>>  long int video_ioctl2(struct file *file,
>>  		      unsigned int cmd, unsigned long int arg);
>>  
>> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>> +				  struct v4l2_ext_pix_format *e, bool strict);
>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>> +				  struct v4l2_format *f,
>> +				  bool mplane_cap, bool strict);
>> +
>>  /*
>>   * The user space interpretation of the 'v4l2_event' differs
>>   * based on the 'time_t' definition on 32-bit architectures, so
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index c7b70ff53bc1d..7123c6a4d9569 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>>  	__u8				reserved[7];
>>  } __attribute__ ((packed));
>>  
>> +/**
>> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
>> + * @sizeimage:		maximum size in bytes required for data, for which
>> + *			this plane will be used.
>> + *			Should be set to zero for unused planes.
>> + * @bytesperline:	distance in bytes between the leftmost pixels in two
>> + *			adjacent lines
>> + * @reserved:		extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_plane_ext_pix_format {
> 
> nit: Maybe v4l2_ext_plane_pix_format, since the struct describes an
> ext variant of a plane_pix_format?

I'm removing this struct as mentioned below.

> 
>> +	__u32 sizeimage;
>> +	__u32 bytesperline;
>> +	__u32 reserved;
>> +};
> 
> Actually, this seems to be exactly the same as the existing
> v4l2_plane_pix_format, except actually having less reserved space. Could it
> make sense to just reuse the existing struct?

I think so, I'm removing this struct for the next version.

Regards,
Helen

> 
> Best regards,
> Tomasz
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-11-14 14:21     ` Helen Koike
@ 2020-11-19  5:45       ` Tomasz Figa
  2020-11-19 10:08         ` Helen Koike
  0 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-11-19  5:45 UTC (permalink / raw)
  To: Helen Koike
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, Boris Brezillon, hiroh, nicolas, Brian.Starkey,
	kernel, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

On Sat, Nov 14, 2020 at 11:21:26AM -0300, Helen Koike wrote:
> Hi Tomasz,
> 
> On 10/2/20 4:49 PM, Tomasz Figa wrote:
> > Hi Helen,
> > 
> > On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
[snip]
> >> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
> >> +{
> >> +	const struct v4l2_ext_pix_format *pix = arg;
> >> +	unsigned int i;
> >> +
> >> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
> >> +		prt_names(pix->type, v4l2_type_names),
> >> +		pix->width, pix->height,
> >> +		(pix->pixelformat & 0xff),
> >> +		(pix->pixelformat >>  8) & 0xff,
> >> +		(pix->pixelformat >> 16) & 0xff,
> >> +		(pix->pixelformat >> 24) & 0xff,
> >> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
> >> +		pix->colorspace, pix->ycbcr_enc,
> >> +		pix->quantization, pix->xfer_func);
> >> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
> > 
> > This is going to print 8 lines every time. Maybe we could skip 0-sized
> > planes or something?
> 
> I'm already checking pix->plane_fmt[i].sizeimage in the loop, it shouldn't
> print 8 lines every time.
> 

Oops, how could I not notice it. Sorry for the noise.

[snip]
> >> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> >> +				  struct v4l2_format *f, bool mplane_cap,
> >> +				  bool strict)
> >> +{
> >> +	const struct v4l2_plane_ext_pix_format *pe;
> >> +	struct v4l2_plane_pix_format *p;
> >> +	unsigned int i;
> >> +
> >> +	memset(f, 0, sizeof(*f));
> >> +
> >> +	/*
> >> +	 * Make sure no modifier is required before doing the
> >> +	 * conversion.
> >> +	 */
> >> +	if (e->modifier && strict &&
> > 
> > Do we need the explicit check for e->modifier != 0 if we have to check for
> > the 2 specific values below anyway?
> 
> We don't, since DRM_FORMAT_MOD_LINEAR is zero.
> 
> But I wanted to make it explicit we don't support modifiers in this conversion.
> But I can remove this check, no problem.
> 

Yes, please. I think the double checking is confusing for the reader.

> > 
> >> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
> >> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
> >> +		return -EINVAL;
> >> +
> >> +	if (!e->plane_fmt[0].sizeimage && strict)
> >> +		return -EINVAL;
> > 
> > Why is this incorrect?
> 
> !sizeimage for the first plane means that there are no planes in ef.
> strict is true if the result for the conversion should be returned to userspace
> and it is not some internal handling.
> 
> So if there are no planes, we would return an incomplete v4l2_format struct
> to userspace.
> 
> But this is not very clear, I'll improve this for the next version.
> 

So I can see 2 cases here:

1) Userspace gives ext struct and driver accepts legacy.

In this case, the kernel needs to adjust the structure to be correct.
-EINVAL is only valid if

"The struct v4l2_format type field is invalid or the requested buffer type not supported."

as per the current uAPI documentation.

2) Driver gives ext struct and userspace accepts legacy.

If at this point we get a struct with no planes, that sounds like a
driver bug, so rather than signaling -EINVAL to the userspace, we should
probably WARN()?

Or am I getting something wrong? :)

[snip]
> >> +{
> >> +	const struct v4l2_plane_pix_format *p;
> >> +	struct v4l2_plane_ext_pix_format *pe;
> >> +	unsigned int i;
> >> +
> >> +	memset(e, 0, sizeof(*e));
> >> +
> >> +	switch (f->type) {
> >> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> >> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> >> +		e->width = f->fmt.pix.width;
> >> +		e->height = f->fmt.pix.height;
> >> +		e->pixelformat = f->fmt.pix.pixelformat;
> >> +		e->field = f->fmt.pix.field;
> >> +		e->colorspace = f->fmt.pix.colorspace;
> >> +		if (f->fmt.pix.flags)
> >> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
> >> +				f->fmt.pix.flags);
> > 
> > Would it make sense to print something like video node name and/or function
> > name to explain where this warning comes from?
> 
> I would need to update the function to receive this information, I can try but
> I'm not sure if it is worthy.
> 

I don't have a strong opinion on this, so maybe let's see if others have
any comments.

> > 
> >> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
> >> +		e->quantization = f->fmt.pix.quantization;
> > 
> > Missing xfer_func?
> 
> Yes, thanks for catching this.
> 
> > 
> >> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> >> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
> > 
> > This doesn't look right. In the ext API we expected the planes to describe
> > color planes, which means that bytesperline needs to be computed for planes
> >> = 1 and sizeimage replaced with per-plane sizes, according to the
> >> pixelformat.
> 
> Ack.
> 
> Just to be clear, even if we are using a planar format that isn't a V4L2_PIX_FMT_*M
> variant, we should describe every plane separatly.
> 
> For instance, if V4L2_PIX_FMT_YVU420 is being used, then f->fmt.pix.bytesperline
> will have data, and we need to calculate bytesperline for all 3 planes, so we'll fill
> out:
> 
> f->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> f->plane_fmt[1].bytesperline = f->fmt.pix.bytesperline / hdiv;
> f->plane_fmt[2].bytesperline = f->fmt.pix.bytesperline / hdiv;
> 
> I'll update this for the next version.
> 

Yes. This basically gives us a unified representation across all
pixelformats and allows userspace to handle them in a uniform way, as
opposed to current uAPI.

[snip]
> >> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> >> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> >> +		else
> >> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> >> +
> >> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> >> +			pe = &e->plane_fmt[i];
> >> +			p = &f->fmt.pix_mp.plane_fmt[i];
> >> +			pe->bytesperline = p->bytesperline;
> >> +			pe->sizeimage = p->sizeimage;
> >> +		}
> > 
> > Same here. A blind copy is not enough. For non-M formats, the plane
> > parameters need to be filled according to the pixelformat.
> 
> 
> Right, following the idea above, we need a different handling if we
> aren't using a M-variant of the pixelformat, and we also need to
> convert the pixelformat from the M-variant to non-M-variant.
> 
> I'll also need to save that the original format was a
> M-variant or not, so I can convert it back as expected.

I'm still reading the rest of the series, so it might be answered
already, but did we decide to do anything about the pixelformat codes
themselves? If both M and non-M variants would be allowed with the new
API, then I guess there isn't anything to save, because the original
format would be preserved?

> 
> I'll change this and submit for review.
> 

Cool, thanks.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-11-19  5:45       ` Tomasz Figa
@ 2020-11-19 10:08         ` Helen Koike
  2020-11-19 13:43           ` Helen Koike
  0 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-11-19 10:08 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, Boris Brezillon, hiroh, nicolas, Brian.Starkey,
	kernel, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Hi Tomasz,

On 11/19/20 2:45 AM, Tomasz Figa wrote:
> On Sat, Nov 14, 2020 at 11:21:26AM -0300, Helen Koike wrote:
>> Hi Tomasz,
>>
>> On 10/2/20 4:49 PM, Tomasz Figa wrote:
>>> Hi Helen,
>>>
>>> On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
> [snip]
>>>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
>>>> +{
>>>> +	const struct v4l2_ext_pix_format *pix = arg;
>>>> +	unsigned int i;
>>>> +
>>>> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
>>>> +		prt_names(pix->type, v4l2_type_names),
>>>> +		pix->width, pix->height,
>>>> +		(pix->pixelformat & 0xff),
>>>> +		(pix->pixelformat >>  8) & 0xff,
>>>> +		(pix->pixelformat >> 16) & 0xff,
>>>> +		(pix->pixelformat >> 24) & 0xff,
>>>> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
>>>> +		pix->colorspace, pix->ycbcr_enc,
>>>> +		pix->quantization, pix->xfer_func);
>>>> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
>>>
>>> This is going to print 8 lines every time. Maybe we could skip 0-sized
>>> planes or something?
>>
>> I'm already checking pix->plane_fmt[i].sizeimage in the loop, it shouldn't
>> print 8 lines every time.
>>
> 
> Oops, how could I not notice it. Sorry for the noise.
> 
> [snip]
>>>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>>>> +				  struct v4l2_format *f, bool mplane_cap,
>>>> +				  bool strict)
>>>> +{
>>>> +	const struct v4l2_plane_ext_pix_format *pe;
>>>> +	struct v4l2_plane_pix_format *p;
>>>> +	unsigned int i;
>>>> +
>>>> +	memset(f, 0, sizeof(*f));
>>>> +
>>>> +	/*
>>>> +	 * Make sure no modifier is required before doing the
>>>> +	 * conversion.
>>>> +	 */
>>>> +	if (e->modifier && strict &&
>>>
>>> Do we need the explicit check for e->modifier != 0 if we have to check for
>>> the 2 specific values below anyway?
>>
>> We don't, since DRM_FORMAT_MOD_LINEAR is zero.
>>
>> But I wanted to make it explicit we don't support modifiers in this conversion.
>> But I can remove this check, no problem.
>>
> 
> Yes, please. I think the double checking is confusing for the reader.

ok.

> 
>>>
>>>> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
>>>> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
>>>> +		return -EINVAL;
>>>> +
>>>> +	if (!e->plane_fmt[0].sizeimage && strict)
>>>> +		return -EINVAL;
>>>
>>> Why is this incorrect?
>>
>> !sizeimage for the first plane means that there are no planes in ef.
>> strict is true if the result for the conversion should be returned to userspace
>> and it is not some internal handling.
>>
>> So if there are no planes, we would return an incomplete v4l2_format struct
>> to userspace.
>>
>> But this is not very clear, I'll improve this for the next version.
>>
> 
> So I can see 2 cases here:
> 
> 1) Userspace gives ext struct and driver accepts legacy.
> 
> In this case, the kernel needs to adjust the structure to be correct.
> -EINVAL is only valid if
> 
> "The struct v4l2_format type field is invalid or the requested buffer type not supported."
> 
> as per the current uAPI documentation.
> 
> 2) Driver gives ext struct and userspace accepts legacy.
> 
> If at this point we get a struct with no planes, that sounds like a
> driver bug, so rather than signaling -EINVAL to the userspace, we should
> probably WARN()?
> 
> Or am I getting something wrong? :)

Make sense, I'll restructure this for the next version.

> 
> [snip]
>>>> +{
>>>> +	const struct v4l2_plane_pix_format *p;
>>>> +	struct v4l2_plane_ext_pix_format *pe;
>>>> +	unsigned int i;
>>>> +
>>>> +	memset(e, 0, sizeof(*e));
>>>> +
>>>> +	switch (f->type) {
>>>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>>>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>>>> +		e->width = f->fmt.pix.width;
>>>> +		e->height = f->fmt.pix.height;
>>>> +		e->pixelformat = f->fmt.pix.pixelformat;
>>>> +		e->field = f->fmt.pix.field;
>>>> +		e->colorspace = f->fmt.pix.colorspace;
>>>> +		if (f->fmt.pix.flags)
>>>> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
>>>> +				f->fmt.pix.flags);
>>>
>>> Would it make sense to print something like video node name and/or function
>>> name to explain where this warning comes from?
>>
>> I would need to update the function to receive this information, I can try but
>> I'm not sure if it is worthy.
>>
> 
> I don't have a strong opinion on this, so maybe let's see if others have
> any comments.
> 
>>>
>>>> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
>>>> +		e->quantization = f->fmt.pix.quantization;
>>>
>>> Missing xfer_func?
>>
>> Yes, thanks for catching this.
>>
>>>
>>>> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>>>> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
>>>
>>> This doesn't look right. In the ext API we expected the planes to describe
>>> color planes, which means that bytesperline needs to be computed for planes
>>>> = 1 and sizeimage replaced with per-plane sizes, according to the
>>>> pixelformat.
>>
>> Ack.
>>
>> Just to be clear, even if we are using a planar format that isn't a V4L2_PIX_FMT_*M
>> variant, we should describe every plane separatly.
>>
>> For instance, if V4L2_PIX_FMT_YVU420 is being used, then f->fmt.pix.bytesperline
>> will have data, and we need to calculate bytesperline for all 3 planes, so we'll fill
>> out:
>>
>> f->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>> f->plane_fmt[1].bytesperline = f->fmt.pix.bytesperline / hdiv;
>> f->plane_fmt[2].bytesperline = f->fmt.pix.bytesperline / hdiv;
>>
>> I'll update this for the next version.
>>
> 
> Yes. This basically gives us a unified representation across all
> pixelformats and allows userspace to handle them in a uniform way, as
> opposed to current uAPI.

Right, I already updated this in my wip branch for next version.

> 
> [snip]
>>>> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>>>> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>>>> +		else
>>>> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>>>> +
>>>> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>>>> +			pe = &e->plane_fmt[i];
>>>> +			p = &f->fmt.pix_mp.plane_fmt[i];
>>>> +			pe->bytesperline = p->bytesperline;
>>>> +			pe->sizeimage = p->sizeimage;
>>>> +		}
>>>
>>> Same here. A blind copy is not enough. For non-M formats, the plane
>>> parameters need to be filled according to the pixelformat.
>>
>>
>> Right, following the idea above, we need a different handling if we
>> aren't using a M-variant of the pixelformat, and we also need to
>> convert the pixelformat from the M-variant to non-M-variant.
>>
>> I'll also need to save that the original format was a
>> M-variant or not, so I can convert it back as expected.
> 
> I'm still reading the rest of the series, so it might be answered
> already, but did we decide to do anything about the pixelformat codes
> themselves? If both M and non-M variants would be allowed with the new
> API, then I guess there isn't anything to save, because the original
> format would be preserved?

I was working with the idea that M-variants wouldn't be allowed.
But then, we have cases where non-M-variant don't exist, such as:

V4L2_PIX_FMT_YVU422M
V4L2_PIX_FMT_YVU444M

(at least, I couldn't find non-M-variant equivalent for those)

But actually, I don't think we formally decided this (and it seems
easier to implement if both are allowed).

Should we allow both variants in the Ext API ?

Thanks
Helen

> 
>>
>> I'll change this and submit for review.
>>
> 
> Cool, thanks.
> 
> Best regards,
> Tomasz
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 7/7] media: docs: add documentation for the Extended API
  2020-08-04 19:29 ` [PATCH v5 7/7] media: docs: add documentation for the Extended API Helen Koike
  2020-08-14  7:49   ` Alexandre Courbot
@ 2020-11-19 10:28   ` Helen Koike
  2020-11-20 11:06   ` Tomasz Figa
  2 siblings, 0 replies; 51+ messages in thread
From: Helen Koike @ 2020-11-19 10:28 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: tfiga, hiroh, nicolas, Brian.Starkey, kernel, boris.brezillon,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov



On 8/4/20 4:29 PM, Helen Koike wrote:
> Add documentation and update references in current documentation for the
> Extended API.
> 
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v5:
> - new patch
> 
>  .../userspace-api/media/v4l/buffer.rst        |   5 +
>  .../userspace-api/media/v4l/common.rst        |   1 +
>  .../userspace-api/media/v4l/dev-capture.rst   |   5 +
>  .../userspace-api/media/v4l/dev-output.rst    |   5 +
>  .../userspace-api/media/v4l/ext-api.rst       | 107 +++++++++
>  .../userspace-api/media/v4l/format.rst        |  16 +-
>  .../userspace-api/media/v4l/user-func.rst     |   5 +
>  .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++++++++
>  .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++++++
>  .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++++++++++++++++
>  .../media/v4l/vidioc-ext-querybuf.rst         |  79 +++++++
>  .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 ++++++++++
>  12 files changed, 697 insertions(+), 4 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> index 57e752aaf414a..c832bedd64e4c 100644
> --- a/Documentation/userspace-api/media/v4l/buffer.rst
> +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> @@ -27,6 +27,11 @@ such as pointers and sizes for each plane, are stored in
>  struct :c:type:`v4l2_plane` instead. In that case,
>  struct :c:type:`v4l2_buffer` contains an array of plane structures.
>  
> +.. note::
> +
> +    The :ref:`ext_api` version can also be used, and it is
> +    preferable when applicable.
> +
>  Dequeued video buffers come with timestamps. The driver decides at which
>  part of the frame and with which clock the timestamp is taken. Please
>  see flags in the masks ``V4L2_BUF_FLAG_TIMESTAMP_MASK`` and
> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst
> index 7d81c58a13cd7..3430e0bdad667 100644
> --- a/Documentation/userspace-api/media/v4l/common.rst
> +++ b/Documentation/userspace-api/media/v4l/common.rst
> @@ -59,6 +59,7 @@ applicable to all devices.
>      ext-ctrls-detect
>      fourcc
>      format
> +    ext-api
>      planar-apis
>      selection-api
>      crop
> diff --git a/Documentation/userspace-api/media/v4l/dev-capture.rst b/Documentation/userspace-api/media/v4l/dev-capture.rst
> index 44d3094093ab6..5077639787d92 100644
> --- a/Documentation/userspace-api/media/v4l/dev-capture.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-capture.rst
> @@ -102,6 +102,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>  
> +.. note::
> +
> +    The :ref:`ext_api` version can also be used, and it is
> +    preferable when applicable.
> +
>  
>  Reading Images
>  ==============
> diff --git a/Documentation/userspace-api/media/v4l/dev-output.rst b/Documentation/userspace-api/media/v4l/dev-output.rst
> index e4f2a1d8b0fc7..f8f40c708e49f 100644
> --- a/Documentation/userspace-api/media/v4l/dev-output.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-output.rst
> @@ -99,6 +99,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>  
> +.. note::
> +
> +    The :ref:`ext_api` version can also be used, and it is
> +    preferable when applicable.
> +
>  
>  Writing Images
>  ==============
> diff --git a/Documentation/userspace-api/media/v4l/ext-api.rst b/Documentation/userspace-api/media/v4l/ext-api.rst
> new file mode 100644
> index 0000000000000..da2a82960d22f
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/ext-api.rst
> @@ -0,0 +1,107 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _ext_api:
> +
> +*************
> +Extendend API
> +*************
> +
> +Introduction
> +============
> +
> +The Extended Format API was conceived to extend V4L2 capabilities and
> +to simplify certain mechanisms.
> +It unifies single- vs multi- planar handling,
> +brings the concept of pixelformat + modifiers, allows planes to be placed
> +in any offset inside a buffer and let userspace to change colorspace
> +attributes if supported by the driver.
> +
> +Data format negotiation and buffer handling works very similar to the classical
> +version, thus in this document we'll just mention the main differences.
> +
> +Data Format Negotiation
> +=======================
> +
> +The API replaces the classical ioctls:
> +
> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>`, :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`,
> +:ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>`
> +(with :c:type:`v4l2_format` as the main parameter).
> +
> +By the extended versions:
> +
> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
> +:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
> +(with :c:type:`v4l2_ext_pix_format` as the main parameter).
> +
> +For CAPTURE and OUTPUT queues only.
> +
> +The ``type`` field of struct :c:type:`v4l2_ext_pix_format` only accepts 
> +``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or ``V4L2_BUF_TYPE_VIDEO_OUTPUT``, and it
> +supports multiplanar format through a combination of ``pixelfomat`` and
> +``modifier`` fields.
> +
> +Only the single-planar variants of the pixel formats are valid in the
> +``pixelformat`` field.

Just a correction, here I meant s/single-planar/non-M-variant.
But this is under discussion, see patch 1/7.

> +To support multi-planar, a modifier should be set in the ``modifier`` field to
> +provide more information regarding the memory layout of pixels and number of
> +planes.

This is wrong too.
Or we don't use the M-variant, or we say both variants are valid and equivalent.
This is under discussion, see patch 1/7.

I'll update in the next version.

Regards,
Helen

> +
> +The ``plane_fmt`` field is an array which holds information by plane using
> +the :c:type:`v4l2_plane_ext_pix_format`. When this struct is filled, its
> +``sizeimage`` field should be non-zero for all valid planes for a given
> +``pixelformat`` + ``modifier`` combination, and zeroed for the invalid ones.
> +
> +Colospace attributes are not read-only as in the classical version, i.e, they
> +can be set by application and drivers will adjust accordingly depending on what
> +is supported by the underlying hardware.
> +
> +Buffers
> +=======
> +
> +The API replaces the classical ioctls:
> +
> +:ref:`VIDIOC_CREATE_BUFS <VIDIOC_CREATE_BUFS>`
> +(with :c:type:`v4l2_create_buffers` as the main parameter),
> +:ref:`VIDIOC_QUERYBUF <VIDIOC_QUERYBUF>`, :ref:`VIDIOC_QBUF <VIDIOC_QBUF>`,
> +:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`, :ref:`VIDIOC_PREPARE_BUF <VIDIOC_PREPARE_BUF>`
> +(with :c:type:`v4l2_buffer` as the main parameter)
> +
> +By the extended versions:
> +
> +:ref:`VIDIOC_EXT_CREATE_BUFS <VIDIOC_EXT_CREATE_BUFS>`
> +(with :c:type:`v4l2_ext_create_buffers` as the main parameter),
> +:ref:`VIDIOC_EXT_QUERYBUF <VIDIOC_EXT_QUERYBUF>`,
> +:ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>`,
> +:ref:`VIDIOC_EXT_DQBUF <VIDIOC_EXT_QBUF>`,
> +:ref:`VIDIOC_EXT_PREPARE_BUF <VIDIOC_EXT_PREPARE_BUF>`
> +(with :c:type:`v4l2_ext_buffer` as the main parameter)
> +
> +The basic difference between :c:type:`v4l2_create_buffers` and
> +:c:type:`v4l2_ext_create_buffers` is that the later have a
> +:c:type:`v4l2_ext_pix_format` as the type of the ``format`` field.
> +
> +Comparing :c:type:`v4l2_ext_buffer` with :c:type:`v4l2_buffer`, in the
> +extended version there is a unification of the single-/multi- planar handling
> +through the ``planes`` field of type :c:type:`v4l2_ext_plane`.
> +
> +The :c:type:`v4l2_ext_plane also allows planes to be placed in a given offset
> +inside a buffer.
> +Planes can be placed in different locations inside the same buffer, or in
> +different buffers.
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_plane
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_buffer
> diff --git a/Documentation/userspace-api/media/v4l/format.rst b/Documentation/userspace-api/media/v4l/format.rst
> index e47fc0505727c..b96d26f26793c 100644
> --- a/Documentation/userspace-api/media/v4l/format.rst
> +++ b/Documentation/userspace-api/media/v4l/format.rst
> @@ -28,13 +28,19 @@ format and the driver selects and reports the best the hardware can do
>  to satisfy the request. Of course applications can also just query the
>  current selection.
>  
> -A single mechanism exists to negotiate all data formats using the
> -aggregate struct :c:type:`v4l2_format` and the
> +There are two mechanism to negociate data formats:
> +
> +The first one is using the aggregate struct :c:type:`v4l2_format` and the
>  :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and
>  :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctls. Additionally the
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` ioctl can be used to examine
>  what the hardware *could* do, without actually selecting a new data
> -format. The data formats supported by the V4L2 API are covered in the
> +format.
> +
> +The second is through the :ref:`ext_api`, please refer to its documentation
> +for more information.
> +
> +The data formats supported by the V4L2 API are covered in the
>  respective device section in :ref:`devices`. For a closer look at
>  image formats see :ref:`pixfmt`.
>  
> @@ -71,7 +77,9 @@ earlier versions of V4L2. Switching the logical stream or returning into
>  *may* support a switch using :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`.
>  
>  All drivers exchanging data with applications must support the
> -:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl. Implementation of the
> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl
> +or the Extended respective versions (TODO: link).
> +Implementation of the
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is highly recommended but optional.
>  
>  
> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
> index bf77c842718e5..67b254812791c 100644
> --- a/Documentation/userspace-api/media/v4l/user-func.rst
> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
> @@ -20,6 +20,7 @@ Function Reference
>      func-close
>      func-ioctl
>      vidioc-create-bufs
> +    vidioc-ext-create-bufs
>      vidioc-cropcap
>      vidioc-dbg-g-chip-info
>      vidioc-dbg-g-register
> @@ -48,6 +49,7 @@ Function Reference
>      vidioc-g-ext-ctrls
>      vidioc-g-fbuf
>      vidioc-g-fmt
> +    vidioc-g-ext-pix-fmt
>      vidioc-g-frequency
>      vidioc-g-input
>      vidioc-g-jpegcomp
> @@ -62,8 +64,11 @@ Function Reference
>      vidioc-log-status
>      vidioc-overlay
>      vidioc-prepare-buf
> +    vidioc-ext-prepare-buf
>      vidioc-qbuf
> +    vidioc-ext-qbuf
>      vidioc-querybuf
> +    vidioc-ext-querybuf
>      vidioc-querycap
>      vidioc-queryctrl
>      vidioc-query-dv-timings
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
> new file mode 100644
> index 0000000000000..67f2c541e4d79
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
> @@ -0,0 +1,95 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_CREATE_BUFS:
> +
> +****************************
> +ioctl VIDIOC_EXT_CREATE_BUFS
> +****************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_CREATE_BUFS - Create buffers for Memory Mapped or User Pointer or DMA Buffer I/O
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp )
> +    :name: VIDIOC_EXT_CREATE_BUFS
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_create_buffers`.
> +
> +
> +Description
> +===========
> +
> +This ioctl is used to create buffers for :ref:`memory mapped <mmap>`
> +or :ref:`user pointer <userp>` or :ref:`DMA buffer <dmabuf>` I/O.
> +This ioctl can be called multiple times to
> +create buffers of different sizes.
> +
> +To allocate the device buffers applications must initialize the relevant
> +fields of the struct :c:type:`v4l2_ext_create_buffers` structure. The
> +``count`` field must be set to the number of requested buffers, the
> +``memory`` field specifies the requested I/O method and the ``reserved``
> +array must be zeroed.
> +
> +The ``format`` field specifies the image format that the buffers must be
> +able to handle. The application has to fill in this struct
> +:c:type:`v4l2_format`. Usually this will be done using the
> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` or
> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctls to ensure that the
> +requested format is supported by the driver.
> +The driver may return an error if the size(s) are not supported by the
> +hardware (usually because they are too small).
> +
> +The driver can create a memory buffer per plane, or a single memory buffer
> +containing all the planes, with a minimum size according to the size
> +defined by the ``format.plane_fmt[i].sizeimage`` field.
> +Usually if the ``format.plane_fmt[i].sizeimage``
> +field is less than the minimum required for the given format, then an
> +error will be returned since drivers will typically not allow this. If
> +it is larger, then the value will be used as-is. In other words, the
> +driver may reject the requested size, but if it is accepted the driver
> +will use it unchanged.
> +
> +When the ioctl is called with a pointer to this structure the driver
> +will attempt to allocate up to the requested number of buffers and store
> +the actual number allocated and the starting index in the ``count`` and
> +the ``index`` fields respectively. On return ``count`` can be smaller
> +than the number requested.
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_create_buffers
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +ENOMEM
> +    No memory to allocate buffers for :ref:`memory mapped <mmap>` I/O.
> +
> +EINVAL
> +    The buffer type (``format.type`` field), requested I/O method
> +    (``memory``) or format (``format`` field) is not valid.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
> new file mode 100644
> index 0000000000000..88e4b57121254
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
> @@ -0,0 +1,62 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_PREPARE_BUF:
> +
> +****************************
> +ioctl VIDIOC_EXT_PREPARE_BUF
> +****************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_PREPARE_BUF - Prepare a buffer for I/O
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_PREPARE_BUF
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> +
> +
> +Description
> +===========
> +
> +Applications can optionally call the :ref:`VIDIOC_EXT_PREPARE_BUF` ioctl to
> +pass ownership of the buffer to the driver before actually enqueuing it,
> +using the :ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>` ioctl, and to prepare it for future I/O. Such
> +preparations may include cache invalidation or cleaning. Performing them
> +in advance saves time during the actual I/O.
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EBUSY
> +    File I/O is in progress.
> +
> +EINVAL
> +    The buffer ``type`` is not supported, or the ``index`` is out of
> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
> +    ``length`` are invalid.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
> new file mode 100644
> index 0000000000000..083e106cf6bad
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
> @@ -0,0 +1,204 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_QBUF:
> +
> +***************************************
> +ioctl VIDIOC_EXT_QBUF, VIDIOC_EXT_DQBUF
> +***************************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_QBUF - VIDIOC_EXT_DQBUF - Exchange a buffer with the driver
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_QBUF
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_DQBUF
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> +
> +
> +Description
> +===========
> +
> +Applications call the ``VIDIOC_EXT_QBUF`` ioctl to enqueue an empty
> +(capturing) or filled (output) buffer in the driver's incoming queue.
> +The semantics depend on the selected I/O method.
> +
> +To enqueue a buffer applications set the ``type`` field of a struct
> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``.
> +Applications must also set the ``index`` field. Valid index numbers
> +range from zero to the number of buffers allocated with
> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
> +one. The contents of the struct :c:type:`v4l2_ext_buffer` returned
> +by a :ref:`VIDIOC_EXT_QUERYBUF` ioctl will do as well.
> +When the buffer is intended for output (``type`` is
> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``) applications must also initialize the
> +``timestamp`` fields and the ``planes[i].plane_length`` for each valid plane,
> +and invalid ones must be set as zero.
> +see :ref:`buffer` for details. Applications must also set ``flags`` to 0. The
> +``reserved`` field must be set to 0.
> +
> +To enqueue a :ref:`memory mapped <mmap>` buffer applications set the
> +``planes[i].memory`` field to ``V4L2_MEMORY_MMAP`` in all the valid planes,
> +and invalid ones must be set as zero.
> +When ``VIDIOC_EXT_QBUF`` is called
> +with a pointer to this structure the driver sets the
> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_QUEUED`` flags and clears
> +the ``V4L2_BUF_FLAG_DONE`` flag in the ``flags`` field, or it returns an
> +``EINVAL`` error code.
> +
> +To enqueue a :ref:`user pointer <userp>` buffer applications set the
> +``planes[i].memory`` field to ``V4L2_MEMORY_USERPTR`` in all the valid planes,
> +and invalid ones must be set as zero, the ``planes[i].m.userptr`` field to the
> +address of the buffer,``planes[i].buffer_length`` to the size of the memory
> +buffer, ``planes[i].plane_length`` to the size that should be used by the plane,
> +and ``planes[i].offset`` of the plane in the memory buffer.
> +
> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure
> +the driver sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
> +``flags`` field, or it returns an error code. This ioctl locks the
> +memory pages of the buffer in physical memory, they cannot be swapped
> +out to disk. Buffers remain locked until dequeued, until the
> +:ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
> +device is closed.
> +
> +To enqueue a :ref:`DMABUF <dmabuf>` buffer applications set the
> +``planes[i].memory`` field to ``V4L2_MEMORY_DMABUF`` in all the valid planes,
> +and invalid ones must be set as zero, the ``planes[i].m.fd`` field to a
> +file descriptor associated with a DMABUF buffer.,``planes[i].buffer_length`` to
> +the size of the memory buffer, ``planes[i].plane_length`` to the size that
> +should be used by the plane, and ``planes[i].offset`` of the plane in the memory buffer.
> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure the driver
> +sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
> +``flags`` field, or it returns an error code. This ioctl locks the
> +buffer. Locking a buffer means passing it to a driver for a hardware
> +access (usually DMA). If an application accesses (reads/writes) a locked
> +buffer then the result is undefined. Buffers remain locked until
> +dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
> +device is closed.
> +
> +The ``request_fd`` field can be used with the ``VIDIOC_EXT_QBUF`` ioctl to specify
> +the file descriptor of a :ref:`request <media-request-api>`, if requests are
> +in use. Setting it means that the buffer will not be passed to the driver
> +until the request itself is queued. Also, the driver will apply any
> +settings associated with the request for this buffer. This field will
> +be ignored unless the ``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
> +If the device does not support requests, then ``EBADR`` will be returned.
> +If requests are supported but an invalid request file descriptor is given,
> +then ``EINVAL`` will be returned.
> +
> +.. caution::
> +   It is not allowed to mix queuing requests with queuing buffers directly.
> +   ``EBUSY`` will be returned if the first buffer was queued directly and
> +   then the application tries to queue a request, or vice versa. After
> +   closing the file descriptor, calling
> +   :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or calling :ref:`VIDIOC_EXT_CREATE_BUFS`
> +   the check for this will be reset.
> +
> +   For :ref:`memory-to-memory devices <mem2mem>` you can specify the
> +   ``request_fd`` only for output buffers, not for capture buffers. Attempting
> +   to specify this for a capture buffer will result in an ``EBADR`` error.
> +
> +Applications call the ``VIDIOC_EXT_DQBUF`` ioctl to dequeue a filled
> +(capturing) or displayed (output) buffer from the driver's outgoing
> +queue. They just set the ``type``, ``planes[i].memory`` and ``reserved`` fields of
> +a struct :c:type:`v4l2_ext_buffer` as above, when
> +``VIDIOC_EXT_DQBUF`` is called with a pointer to this structure the driver
> +fills the remaining fields or returns an error code. The driver may also
> +set ``V4L2_BUF_FLAG_ERROR`` in the ``flags`` field. It indicates a
> +non-critical (recoverable) streaming error. In such case the application
> +may continue as normal, but should be aware that data in the dequeued
> +buffer might be corrupted. When using the multi-planar API, the planes
> +array must be passed in as well.
> +
> +If the application sets the ``memory`` field to ``V4L2_MEMORY_DMABUF`` to
> +dequeue a :ref:`DMABUF <dmabuf>` buffer, the driver fills the ``m.fd`` field
> +with a file descriptor numerically the same as the one given to ``VIDIOC_EXT_QBUF``
> +when the buffer was enqueued. No new file descriptor is created at dequeue time
> +and the value is only for the application convenience.
> +
> +By default ``VIDIOC_EXT_DQBUF`` blocks when no buffer is in the outgoing
> +queue. When the ``O_NONBLOCK`` flag was given to the
> +:ref:`open() <func-open>` function, ``VIDIOC_EXT_DQBUF`` returns
> +immediately with an ``EAGAIN`` error code when no buffer is available.
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_buffers
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EAGAIN
> +    Non-blocking I/O has been selected using ``O_NONBLOCK`` and no
> +    buffer was in the outgoing queue.
> +
> +EINVAL
> +    The buffer ``type`` is not supported, or the ``index`` is out of
> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
> +    ``length`` are invalid, or the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was
> +    set but the the given ``request_fd`` was invalid, or ``m.fd`` was
> +    an invalid DMABUF file descriptor.
> +
> +EIO
> +    ``VIDIOC_EXT_DQBUF`` failed due to an internal error. Can also indicate
> +    temporary problems like signal loss.
> +
> +    .. note::
> +
> +       The driver might dequeue an (empty) buffer despite returning
> +       an error, or even stop capturing. Reusing such buffer may be unsafe
> +       though and its details (e.g. ``index``) may not be returned either.
> +       It is recommended that drivers indicate recoverable errors by setting
> +       the ``V4L2_BUF_FLAG_ERROR`` and returning 0 instead. In that case the
> +       application should be able to safely reuse the buffer and continue
> +       streaming.
> +
> +EPIPE
> +    ``VIDIOC_EXT_DQBUF`` returns this on an empty capture queue for mem2mem
> +    codecs if a buffer with the ``V4L2_BUF_FLAG_LAST`` was already
> +    dequeued and no new buffers are expected to become available.
> +
> +EBADR
> +    The ``V4L2_BUF_FLAG_REQUEST_FD`` flag was set but the device does not
> +    support requests for the given buffer type, or
> +    the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was not set but the device requires
> +    that the buffer is part of a request.
> +
> +EBUSY
> +    The first buffer was queued via a request, but the application now tries
> +    to queue it directly, or vice versa (it is not permitted to mix the two
> +    APIs).
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
> new file mode 100644
> index 0000000000000..f2a12017253f6
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
> @@ -0,0 +1,79 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_QUERYBUF:
> +
> +*************************
> +ioctl VIDIOC_EXT_QUERYBUF
> +*************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_QUERYBUF - Query the status of a buffer
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_QUERYBUF
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> +
> +
> +Description
> +===========
> +
> +This ioctl is part of the :ref:`streaming <mmap>` I/O method. It can
> +be used to query the status of a buffer at any time after buffers have
> +been allocated with the :ref:`VIDIOC_EXT_CREATE_BUFS` ioctl.
> +
> +Applications set the ``type`` field of a struct
> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``,
> +and the ``index`` field. Valid index numbers range from zero to the
> +number of buffers allocated with
> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
> +one. The ``reserved`` field must be set to 0.
> +
> +In the ``flags`` field the ``V4L2_BUF_FLAG_MAPPED``,
> +``V4L2_BUF_FLAG_PREPARED``, ``V4L2_BUF_FLAG_QUEUED`` and
> +``V4L2_BUF_FLAG_DONE`` flags will be valid. The ``planes.memory`` fields will be
> +set to the current I/O method for each plane.
> +
> +For every valid plane, an entry in ``planes`` will be filled, and zeroed for
> +invalid ones. ``planes[i].buffer_length`` is the size of the memory buffer
> +which contains the plane, ``planes[i].plane_length`` is the length of the plane,
> +and ``planes[i].offset` is where the plane is placed in the memory buffer.
> +
> +The size of the ``planes`` array can be calculated by the number of sequential
> +planes with ``planes[i].buffer_length`` that differs from zero up to the max
> +size of the array.
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EINVAL
> +    The buffer ``type`` is not supported, or the ``index`` is out of
> +    bounds.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> new file mode 100644
> index 0000000000000..008e6c98a88a5
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> @@ -0,0 +1,117 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_G_EXT_PIX_FMT:
> +
> +************************************************************************
> +ioctl VIDIOC_G_EXT_PIX_FMT, VIDIOC_S_EXT_PIX_FMT, VIDIOC_TRY_EXT_PIX_FMT
> +************************************************************************
> +
> +Name
> +====
> +
> +VIDIOC_G_EXT_PIX_FMT - VIDIOC_S_EXT_PIX_FMT - VIDIOC_TRY_EXT_PIX_FMT - Get or set the data format, try a format
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> +    :name: VIDIOC_G_EXT_PIX_FMT
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> +    :name: VIDIOC_S_EXT_PIX_FMT
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> +    :name: VIDIOC_TRY_EXT_PIX_FMT
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_pix_format`.
> +
> +
> +Description
> +===========
> +
> +These ioctls are used to negotiate the format of data (typically image
> +format) exchanged between driver and application.
> +
> +To query the current parameters applications set the ``type`` field of a
> +struct :c:type:`v4l2_ext_pix_format` to  ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or
> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``, all the other types are invalid in this API,
> +and multiplanar is supported through modifiers.
> +
> +When the application calls the
> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with a pointer to this
> +structure the driver fills the other members.
> +When the requested buffer type is not supported drivers return
> +an ``EINVAL`` error code.
> +
> +To change the current format parameters applications initialize all
> +the fields in the struct.
> +For details see the documentation of the various devices types in
> +:ref:`devices`. Good practice is to query the current parameters
> +first, and to modify only those parameters not suitable for the
> +application. When the application calls the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with
> +a pointer to a struct :c:type:`v4l2_ext_pix_format` structure the driver
> +checks and adjusts the parameters against hardware abilities. Drivers
> +should not return an error code unless the ``type`` field is invalid,
> +this is a mechanism to fathom device capabilities and to approach
> +parameters acceptable for both the application and driver. On success
> +the driver may program the hardware, allocate resources and generally
> +prepare for data exchange. Finally the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl returns
> +the current format parameters as :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` does. Very simple,
> +inflexible devices may even ignore all input and always return the
> +default parameters. However all V4L2 devices exchanging data with the
> +application must implement the :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` and :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
> +ioctl. When the requested buffer type is not supported drivers return an
> +EINVAL error code on a :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` attempt. When I/O is already in
> +progress or the resource is not available for other reasons drivers
> +return the ``EBUSY`` error code.
> +
> +The :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl is equivalent to :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` with one
> +exception: it does not change driver state. It can also be called at any
> +time, never returning ``EBUSY``. This function is provided to negotiate
> +parameters, to learn about hardware limitations, without disabling I/O
> +or possibly time consuming hardware preparations. Although strongly
> +recommended drivers are not required to implement this ioctl.
> +
> +The format as returned by :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` must be identical to what
> +:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` returns for the same input or output.
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_plane_ext_pix_format
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_pix_format
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EINVAL
> +    The struct :c:type:`v4l2_ext_pix_format` ``type`` field is
> +    invalid or the requested buffer type not supported.
> +
> +EBUSY
> +    The device is busy and cannot change the format. This could be
> +    because or the device is streaming or buffers are allocated or
> +    queued to the driver. Relevant for :ref:`VIDIOC_S_EXT_PIX_FMT
> +    <VIDIOC_G_EXT_PIX_FMT>` only.
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-11-19 10:08         ` Helen Koike
@ 2020-11-19 13:43           ` Helen Koike
  2020-12-14 10:19             ` Tomasz Figa
  0 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-11-19 13:43 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, Boris Brezillon, hiroh, nicolas, Brian.Starkey,
	kernel, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov



On 11/19/20 7:08 AM, Helen Koike wrote:
> Hi Tomasz,
> 
> On 11/19/20 2:45 AM, Tomasz Figa wrote:
>> On Sat, Nov 14, 2020 at 11:21:26AM -0300, Helen Koike wrote:
>>> Hi Tomasz,
>>>
>>> On 10/2/20 4:49 PM, Tomasz Figa wrote:
>>>> Hi Helen,
>>>>
>>>> On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
>> [snip]
>>>>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
>>>>> +{
>>>>> +	const struct v4l2_ext_pix_format *pix = arg;
>>>>> +	unsigned int i;
>>>>> +
>>>>> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
>>>>> +		prt_names(pix->type, v4l2_type_names),
>>>>> +		pix->width, pix->height,
>>>>> +		(pix->pixelformat & 0xff),
>>>>> +		(pix->pixelformat >>  8) & 0xff,
>>>>> +		(pix->pixelformat >> 16) & 0xff,
>>>>> +		(pix->pixelformat >> 24) & 0xff,
>>>>> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
>>>>> +		pix->colorspace, pix->ycbcr_enc,
>>>>> +		pix->quantization, pix->xfer_func);
>>>>> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
>>>>
>>>> This is going to print 8 lines every time. Maybe we could skip 0-sized
>>>> planes or something?
>>>
>>> I'm already checking pix->plane_fmt[i].sizeimage in the loop, it shouldn't
>>> print 8 lines every time.
>>>
>>
>> Oops, how could I not notice it. Sorry for the noise.
>>
>> [snip]
>>>>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>>>>> +				  struct v4l2_format *f, bool mplane_cap,
>>>>> +				  bool strict)
>>>>> +{
>>>>> +	const struct v4l2_plane_ext_pix_format *pe;
>>>>> +	struct v4l2_plane_pix_format *p;
>>>>> +	unsigned int i;
>>>>> +
>>>>> +	memset(f, 0, sizeof(*f));
>>>>> +
>>>>> +	/*
>>>>> +	 * Make sure no modifier is required before doing the
>>>>> +	 * conversion.
>>>>> +	 */
>>>>> +	if (e->modifier && strict &&
>>>>
>>>> Do we need the explicit check for e->modifier != 0 if we have to check for
>>>> the 2 specific values below anyway?
>>>
>>> We don't, since DRM_FORMAT_MOD_LINEAR is zero.
>>>
>>> But I wanted to make it explicit we don't support modifiers in this conversion.
>>> But I can remove this check, no problem.
>>>
>>
>> Yes, please. I think the double checking is confusing for the reader.
> 
> ok.
> 
>>
>>>>
>>>>> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
>>>>> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
>>>>> +		return -EINVAL;
>>>>> +
>>>>> +	if (!e->plane_fmt[0].sizeimage && strict)
>>>>> +		return -EINVAL;
>>>>
>>>> Why is this incorrect?
>>>
>>> !sizeimage for the first plane means that there are no planes in ef.
>>> strict is true if the result for the conversion should be returned to userspace
>>> and it is not some internal handling.
>>>
>>> So if there are no planes, we would return an incomplete v4l2_format struct
>>> to userspace.
>>>
>>> But this is not very clear, I'll improve this for the next version.
>>>
>>
>> So I can see 2 cases here:
>>
>> 1) Userspace gives ext struct and driver accepts legacy.
>>
>> In this case, the kernel needs to adjust the structure to be correct.
>> -EINVAL is only valid if
>>
>> "The struct v4l2_format type field is invalid or the requested buffer type not supported."
>>
>> as per the current uAPI documentation.
>>
>> 2) Driver gives ext struct and userspace accepts legacy.
>>
>> If at this point we get a struct with no planes, that sounds like a
>> driver bug, so rather than signaling -EINVAL to the userspace, we should
>> probably WARN()?
>>
>> Or am I getting something wrong? :)
> 
> Make sense, I'll restructure this for the next version.
> 
>>
>> [snip]
>>>>> +{
>>>>> +	const struct v4l2_plane_pix_format *p;
>>>>> +	struct v4l2_plane_ext_pix_format *pe;
>>>>> +	unsigned int i;
>>>>> +
>>>>> +	memset(e, 0, sizeof(*e));
>>>>> +
>>>>> +	switch (f->type) {
>>>>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>>>>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>>>>> +		e->width = f->fmt.pix.width;
>>>>> +		e->height = f->fmt.pix.height;
>>>>> +		e->pixelformat = f->fmt.pix.pixelformat;
>>>>> +		e->field = f->fmt.pix.field;
>>>>> +		e->colorspace = f->fmt.pix.colorspace;
>>>>> +		if (f->fmt.pix.flags)
>>>>> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
>>>>> +				f->fmt.pix.flags);
>>>>
>>>> Would it make sense to print something like video node name and/or function
>>>> name to explain where this warning comes from?
>>>
>>> I would need to update the function to receive this information, I can try but
>>> I'm not sure if it is worthy.
>>>
>>
>> I don't have a strong opinion on this, so maybe let's see if others have
>> any comments.
>>
>>>>
>>>>> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
>>>>> +		e->quantization = f->fmt.pix.quantization;
>>>>
>>>> Missing xfer_func?
>>>
>>> Yes, thanks for catching this.
>>>
>>>>
>>>>> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>>>>> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
>>>>
>>>> This doesn't look right. In the ext API we expected the planes to describe
>>>> color planes, which means that bytesperline needs to be computed for planes
>>>>> = 1 and sizeimage replaced with per-plane sizes, according to the
>>>>> pixelformat.
>>>
>>> Ack.
>>>
>>> Just to be clear, even if we are using a planar format that isn't a V4L2_PIX_FMT_*M
>>> variant, we should describe every plane separatly.
>>>
>>> For instance, if V4L2_PIX_FMT_YVU420 is being used, then f->fmt.pix.bytesperline
>>> will have data, and we need to calculate bytesperline for all 3 planes, so we'll fill
>>> out:
>>>
>>> f->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>>> f->plane_fmt[1].bytesperline = f->fmt.pix.bytesperline / hdiv;
>>> f->plane_fmt[2].bytesperline = f->fmt.pix.bytesperline / hdiv;
>>>
>>> I'll update this for the next version.
>>>
>>
>> Yes. This basically gives us a unified representation across all
>> pixelformats and allows userspace to handle them in a uniform way, as
>> opposed to current uAPI.
> 
> Right, I already updated this in my wip branch for next version.
> 
>>
>> [snip]
>>>>> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>>>>> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>>>>> +		else
>>>>> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>>>>> +
>>>>> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>>>>> +			pe = &e->plane_fmt[i];
>>>>> +			p = &f->fmt.pix_mp.plane_fmt[i];
>>>>> +			pe->bytesperline = p->bytesperline;
>>>>> +			pe->sizeimage = p->sizeimage;
>>>>> +		}
>>>>
>>>> Same here. A blind copy is not enough. For non-M formats, the plane
>>>> parameters need to be filled according to the pixelformat.
>>>
>>>
>>> Right, following the idea above, we need a different handling if we
>>> aren't using a M-variant of the pixelformat, and we also need to
>>> convert the pixelformat from the M-variant to non-M-variant.
>>>
>>> I'll also need to save that the original format was a
>>> M-variant or not, so I can convert it back as expected.
>>
>> I'm still reading the rest of the series, so it might be answered
>> already, but did we decide to do anything about the pixelformat codes
>> themselves? If both M and non-M variants would be allowed with the new
>> API, then I guess there isn't anything to save, because the original
>> format would be preserved?
> 
> I was working with the idea that M-variants wouldn't be allowed.
> But then, we have cases where non-M-variant don't exist, such as:
> 
> V4L2_PIX_FMT_YVU422M
> V4L2_PIX_FMT_YVU444M
> 
> (at least, I couldn't find non-M-variant equivalent for those)
> 
> But actually, I don't think we formally decided this (and it seems
> easier to implement if both are allowed).
> 
> Should we allow both variants in the Ext API ?

I see 3 options:

1) Ext API doesn't accept M-variants and return -EINVAL.

    But this doesn't seem to be the v4l2 way, where we avoid returning
    errors and try to adjust to what we think it is better.

    At the same time, this could allow us, in a very remote hypothetical
    future situation, to remove the M-variants from the kernel when/if
    the old API gets obsolete.

    Future ENUM_EXT_FMT won't enumerate M-variants in this option.

2) Ext API accept M-variants without normalization.

    The driver can receive both variants, and need to handle both as
    equivalents, i.e. if (V4L2_PIX_FMT_YUV420M || V4L2_PIX_FMT_YUV420)

    Both can be returned to userspace.

    Future ENUM_EXT_FMT can enumerate M-variants in this option.

3) Ext API accept M-variants with normalization.

    If userspace uses V4L2_PIX_FMT_YUV420M, the framework converts
    to V4L2_PIX_FMT_YUV420 before passing it to the driver.

    Only V4L2_PIX_FMT_YUV420 is sent back to userspace (even if userspace
    used the M variant, the kernel normalizes it, which is a similar behavior
    when userspace try to use a non-supported resolution and the kernel
    adjusts it).

    (we could also leave this pixelformat normalization to the driver,
    but I don't see a reason to that)

    Future ENUM_EXT_FMT won't enumerate M-variants in this option.



I'm leaning towards option 3, please let me know your thoughts.

Thanks
Helen


> 
> Thanks
> Helen
> 
>>
>>>
>>> I'll change this and submit for review.
>>>
>>
>> Cool, thanks.
>>
>> Best regards,
>> Tomasz
>>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-08-14  7:49   ` Alexandre Courbot
@ 2020-11-19 16:23     ` Helen Koike
  0 siblings, 0 replies; 51+ messages in thread
From: Helen Koike @ 2020-11-19 16:23 UTC (permalink / raw)
  To: Alexandre Courbot
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Boris Brezillon,
	Tomasz Figa, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, narmstrong, LKML, frkoenig, Maxime Jourdan,
	Stanimir Varbanov

Hi Alexandre,

Thanks for your review.

On 8/14/20 4:49 AM, Alexandre Courbot wrote:
> On Wed, Aug 5, 2020 at 4:32 AM Helen Koike <helen.koike@collabora.com> wrote:
>>
>> This is part of the multiplanar and singleplanar unification process.
>> v4l2_ext_pix_format is supposed to work for both cases.
>>
>> We also add the concept of modifiers already employed in DRM to expose
>> HW-specific formats (like tiled or compressed formats) and allow
>> exchanging this information with the DRM subsystem in a consistent way.
>>
>> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
>> v4l2_ext_format, other types will be rejected if you use the
>> {G,S,TRY}_EXT_PIX_FMT ioctls.
>>
>> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
>> in drivers, but, in the meantime, the core takes care of converting
>> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
>> still work if the userspace app/lib uses the new ioctls.
>> The conversion is also done the other around to allow userspace
>> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
>> _ext_ hooks.
>>
>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> ---
>> Changes in v5:
>> - change sizes and reorder fields to avoid holes in the struct and make
>>   it the same for 32 and 64 bits
>> - removed __attribute__ ((packed)) from uapi structs
>> - Fix doc warning from make htmldocs
>> - Updated commit message with EXT_PIX prefix for the ioctls.
>>
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Add reserved fields
>> - Removed num_planes from struct v4l2_ext_pix_format
>> - Removed flag field from struct v4l2_ext_pix_format, since the only
>>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>>   where we can use modifiers, or add it back later through the reserved
>>   bits.
>> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>>   != MOD_INVALID
>> - Fix type assignment in v4l_g_fmt_ext_pix()
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Move the modifier in v4l2_ext_format (was formerly placed in
>>   v4l2_ext_plane)
>> - Fix a few bugs in the converters and add a strict parameter to
>>   allow conversion of uninitialized/mis-initialized objects
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>>  include/media/v4l2-ioctl.h           |  34 ++
>>  include/uapi/linux/videodev2.h       |  56 +++
>>  4 files changed, 615 insertions(+), 81 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>> index a593ea0598b55..e1829906bc086 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>                         set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
>>                 if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
>>                                ops->vidioc_g_fmt_vid_cap_mplane ||
>> +                              ops->vidioc_g_ext_pix_fmt_vid_cap ||
>>                                ops->vidioc_g_fmt_vid_overlay)) ||
>>                     (is_tx && (ops->vidioc_g_fmt_vid_out ||
>>                                ops->vidioc_g_fmt_vid_out_mplane ||
>> -                              ops->vidioc_g_fmt_vid_out_overlay)))
>> +                              ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +                              ops->vidioc_g_fmt_vid_out_overlay))) {
>>                          set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
>> +                        set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
>> +               }
>>                 if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
>>                                ops->vidioc_s_fmt_vid_cap_mplane ||
>> +                              ops->vidioc_s_ext_pix_fmt_vid_cap ||
>>                                ops->vidioc_s_fmt_vid_overlay)) ||
>>                     (is_tx && (ops->vidioc_s_fmt_vid_out ||
>>                                ops->vidioc_s_fmt_vid_out_mplane ||
>> -                              ops->vidioc_s_fmt_vid_out_overlay)))
>> +                              ops->vidioc_s_ext_pix_fmt_vid_out ||
>> +                              ops->vidioc_s_fmt_vid_out_overlay))) {
>>                          set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
>> +                        set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
>> +               }
>>                 if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
>>                                ops->vidioc_try_fmt_vid_cap_mplane ||
>> +                              ops->vidioc_try_ext_pix_fmt_vid_cap ||
>>                                ops->vidioc_try_fmt_vid_overlay)) ||
>>                     (is_tx && (ops->vidioc_try_fmt_vid_out ||
>>                                ops->vidioc_try_fmt_vid_out_mplane ||
>> -                              ops->vidioc_try_fmt_vid_out_overlay)))
>> +                              ops->vidioc_try_ext_pix_fmt_vid_out ||
>> +                              ops->vidioc_try_fmt_vid_out_overlay))) {
>>                          set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
>> +                        set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
>> +               }
>>                 SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
>>                 SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
>>                 SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
>> @@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>                 /* touch specific ioctls */
>>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
>>                 SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
>> +               SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
>>                 SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
>> +               SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
>>                 SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
>> +               SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
>>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
>>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
>>                 SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index a556880f225a5..14a0def50f8ea 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -17,6 +17,8 @@
>>
>>  #include <linux/videodev2.h>
>>
>> +#include <drm/drm_fourcc.h>
>> +
>>  #include <media/v4l2-common.h>
>>  #include <media/v4l2-ioctl.h>
>>  #include <media/v4l2-ctrls.h>
>> @@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
>>         }
>>  }
>>
>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
>> +{
>> +       const struct v4l2_ext_pix_format *pix = arg;
>> +       unsigned int i;
>> +
>> +       pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
>> +               prt_names(pix->type, v4l2_type_names),
>> +               pix->width, pix->height,
>> +               (pix->pixelformat & 0xff),
>> +               (pix->pixelformat >>  8) & 0xff,
>> +               (pix->pixelformat >> 16) & 0xff,
>> +               (pix->pixelformat >> 24) & 0xff,
>> +               pix->modifier, prt_names(pix->field, v4l2_field_names),
>> +               pix->colorspace, pix->ycbcr_enc,
>> +               pix->quantization, pix->xfer_func);
>> +       for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
>> +               pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
>> +                        i, pix->plane_fmt[i].bytesperline,
>> +                        pix->plane_fmt[i].sizeimage);
>> +}
>> +
>>  static void v4l_print_framebuffer(const void *arg, bool write_only)
>>  {
>>         const struct v4l2_framebuffer *p = arg;
>> @@ -964,11 +987,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>         switch (type) {
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>>                 if ((is_vid || is_tch) && is_rx &&
>> -                   (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
>> +                   (ops->vidioc_g_fmt_vid_cap ||
>> +                    ops->vidioc_g_ext_pix_fmt_vid_cap ||
>> +                    ops->vidioc_g_fmt_vid_cap_mplane))
>>                         return 0;
>>                 break;
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -               if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
>> +               if ((is_vid || is_tch) && is_rx &&
>> +                   (ops->vidioc_g_fmt_vid_cap_mplane ||
>> +                    ops->vidioc_g_ext_pix_fmt_vid_cap))
>>                         return 0;
>>                 break;
>>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>> @@ -977,11 +1004,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>                 break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>>                 if (is_vid && is_tx &&
>> -                   (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
>> +                   (ops->vidioc_g_fmt_vid_out ||
>> +                    ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +                    ops->vidioc_g_fmt_vid_out_mplane))
>>                         return 0;
>>                 break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -               if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
>> +               if (is_vid && is_tx &&
>> +                   (ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +                    ops->vidioc_g_fmt_vid_out_mplane))
>>                         return 0;
>>                 break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>> @@ -1061,6 +1092,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>                sizeof(fmt->fmt.pix) - offset);
>>  }
>>
>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>> +                                 struct v4l2_format *f, bool mplane_cap,
>> +                                 bool strict)
>> +{
>> +       const struct v4l2_plane_ext_pix_format *pe;
>> +       struct v4l2_plane_pix_format *p;
>> +       unsigned int i;
>> +
>> +       memset(f, 0, sizeof(*f));
>> +
>> +       /*
>> +        * Make sure no modifier is required before doing the
>> +        * conversion.
>> +        */
>> +       if (e->modifier && strict &&
>> +           e->modifier != DRM_FORMAT_MOD_LINEAR &&
>> +           e->modifier != DRM_FORMAT_MOD_INVALID)
>> +               return -EINVAL;
>> +
>> +       if (!e->plane_fmt[0].sizeimage && strict)
>> +               return -EINVAL;
>> +
>> +       if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
>> +               return 0;
>> +
>> +       if (!mplane_cap) {
>> +               f->fmt.pix.width = e->width;
>> +               f->fmt.pix.height = e->height;
>> +               f->fmt.pix.pixelformat = e->pixelformat;
>> +               f->fmt.pix.field = e->field;
>> +               f->fmt.pix.colorspace = e->colorspace;
>> +               f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
>> +               f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +               f->fmt.pix.quantization = e->quantization;
>> +               pe = &e->plane_fmt[0];
>> +               f->fmt.pix.bytesperline = pe->bytesperline;
>> +               f->fmt.pix.sizeimage = pe->sizeimage;
>> +               f->type = e->type;
>> +               return 0;
>> +       }
>> +
>> +       f->fmt.pix_mp.width = e->width;
>> +       f->fmt.pix_mp.height = e->height;
>> +       f->fmt.pix_mp.pixelformat = e->pixelformat;
>> +       f->fmt.pix_mp.field = e->field;
>> +       f->fmt.pix_mp.colorspace = e->colorspace;
>> +       f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
>> +       f->fmt.pix_mp.quantization = e->quantization;
>> +       if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> +               f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>> +       else
>> +               f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
>> +
>> +       for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>> +               pe = &e->plane_fmt[i];
>> +               p = &f->fmt.pix_mp.plane_fmt[i];
>> +               p->bytesperline = pe->bytesperline;
>> +               p->sizeimage = pe->sizeimage;
>> +       }
>> +
>> +       return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
>> +
>> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>> +                                 struct v4l2_ext_pix_format *e, bool strict)
>> +{
>> +       const struct v4l2_plane_pix_format *p;
>> +       struct v4l2_plane_ext_pix_format *pe;
>> +       unsigned int i;
>> +
>> +       memset(e, 0, sizeof(*e));
>> +
>> +       switch (f->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +               e->width = f->fmt.pix.width;
>> +               e->height = f->fmt.pix.height;
>> +               e->pixelformat = f->fmt.pix.pixelformat;
>> +               e->field = f->fmt.pix.field;
>> +               e->colorspace = f->fmt.pix.colorspace;
>> +               if (f->fmt.pix.flags)
>> +                       pr_warn("Ignoring pixelformat flags 0x%x\n",
>> +                               f->fmt.pix.flags);
>> +               e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
>> +               e->quantization = f->fmt.pix.quantization;
>> +               e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>> +               e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
>> +               e->type = f->type;
>> +               break;
>> +
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +               if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
>> +                    !f->fmt.pix_mp.num_planes) && strict)
>> +                       return -EINVAL;
>> +
>> +               e->width = f->fmt.pix_mp.width;
>> +               e->height = f->fmt.pix_mp.height;
>> +               e->pixelformat = f->fmt.pix_mp.pixelformat;
>> +               e->field = f->fmt.pix_mp.field;
>> +               e->colorspace = f->fmt.pix_mp.colorspace;
>> +               if (f->fmt.pix.flags)
>> +                       pr_warn("Ignoring pixelformat flags 0x%x\n",
>> +                               f->fmt.pix.flags);
>> +               e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
>> +               e->quantization = f->fmt.pix_mp.quantization;
>> +               if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>> +                       e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +               else
>> +                       e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +
>> +               for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>> +                       pe = &e->plane_fmt[i];
>> +                       p = &f->fmt.pix_mp.plane_fmt[i];
>> +                       pe->bytesperline = p->bytesperline;
>> +                       pe->sizeimage = p->sizeimage;
>> +               }
>> +               break;
>> +
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>> +
>>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>>                                 struct file *file, void *fh, void *arg)
>>  {
>> @@ -1564,6 +1723,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
>>         p->xfer_func = 0;
>>  }
>>
>> +static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +                            struct file *file, void *fh,
>> +                            struct v4l2_format *f)
>> +{
>> +       struct v4l2_ext_pix_format ef;
>> +       int ret;
>> +
>> +       switch (f->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +               ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +               ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +               break;
>> +
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +               ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +               ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
>> +               break;
>> +
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_ext_pix_format_to_format(&ef, f,
>> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +                                            true);
>> +}
>> +
>>  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>                                 struct file *file, void *fh, void *arg)
>>  {
>> @@ -1600,17 +1791,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>
>>         switch (p->type) {
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -               if (unlikely(!ops->vidioc_g_fmt_vid_cap))
>> -                       break;
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> -                       v4l_pix_format_touch(&p->fmt.pix);
>> -               return ret;
>> +               if (ops->vidioc_g_fmt_vid_cap) {
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +                               v4l_pix_format_touch(&p->fmt.pix);
>> +                       return ret;
>> +               } else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
>> +                       ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +                               v4l_pix_format_touch(&p->fmt.pix);
>> +                       return ret;
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -               return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
>> +               if (ops->vidioc_g_fmt_vid_cap_mplane)
>> +                       return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
>> +               else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
>> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>                 return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
>>         case V4L2_BUF_TYPE_VBI_CAPTURE:
>> @@ -1618,15 +1819,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>         case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
>>                 return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -               if (unlikely(!ops->vidioc_g_fmt_vid_out))
>> -                       break;
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               return ret;
>> +               if (ops->vidioc_g_fmt_vid_out) {
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       return ret;
>> +               } else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
>> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -               return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
>> +               if (ops->vidioc_g_fmt_vid_out_mplane)
>> +                       return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
>> +               else if (ops->vidioc_g_ext_pix_fmt_vid_out)
>> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>                 return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
>>         case V4L2_BUF_TYPE_VBI_OUTPUT:
>> @@ -1645,6 +1853,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>         return -EINVAL;
>>  }
>>
>> +static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +                            struct file *file, void *fh, void *arg)
>> +{
>> +       struct v4l2_ext_pix_format *ef = arg;
>> +       struct v4l2_format f = {
>> +               .type = ef->type,
>> +       };
>> +       int ret;
>> +
>> +       ret = check_fmt(file, ef->type);
>> +       if (ret)
>> +               return ret;
>> +
>> +       memset(ef, 0, sizeof(*ef));
>> +       ef->type = f.type;
>> +
>> +       switch (f.type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +               if (ops->vidioc_g_ext_pix_fmt_vid_cap)
>> +                       return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
>> +               break;
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +               if (ops->vidioc_g_ext_pix_fmt_vid_out)
>> +                       return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
>> +               break;
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       ret = v4l_g_fmt(ops, file, fh, &f);
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>> +static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +                            struct file *file, void *fh,
>> +                            struct v4l2_format *f)
>> +{
>> +       struct v4l2_ext_pix_format ef;
>> +       int ret;
>> +
>> +       ret = v4l2_format_to_ext_pix_format(f, &ef, false);
>> +       if (ret)
>> +               return ret;
>> +
>> +       switch (f->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +               ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +               break;
>> +
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +               ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
>> +               break;
>> +
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_ext_pix_format_to_format(&ef, f,
>> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +                                            true);
>> +}
>> +
>>  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>                                 struct file *file, void *fh, void *arg)
>>  {
>> @@ -1663,23 +1941,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>
>>         switch (p->type) {
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -               if (unlikely(!ops->vidioc_s_fmt_vid_cap))
>> +               if (ops->vidioc_s_fmt_vid_cap) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
>> +                       ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
>> +                       ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +               } else {
>>                         break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix);
>> -               ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +               }
>> +
>>                 if (vfd->vfl_type == VFL_TYPE_TOUCH)
>>                         v4l_pix_format_touch(&p->fmt.pix);
>>                 return ret;
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -               if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -                                         bytesperline);
>> -               return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
>> +               if (ops->vidioc_s_fmt_vid_cap_mplane) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +                                                 bytesperline);
>> +                       return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
>> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
>> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>                 if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
>>                         break;
>> @@ -1696,21 +1982,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>                 CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>>                 return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -               if (unlikely(!ops->vidioc_s_fmt_vid_out))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix);
>> -               ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               return ret;
>> +               if (ops->vidioc_s_fmt_vid_out) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
>> +                       ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       return ret;
>> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
>> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -               if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -                                         bytesperline);
>> -               return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
>> +               if (ops->vidioc_s_fmt_vid_out_mplane) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +                                                 bytesperline);
>> +                       return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
>> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
>> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>                 if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
>>                         break;
>> @@ -1750,6 +2042,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>         return -EINVAL;
>>  }
>>
>> +static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +                            struct file *file, void *fh, void *arg)
>> +{
>> +       struct video_device *vfd = video_devdata(file);
>> +       struct v4l2_ext_pix_format *ef = arg;
>> +       struct v4l2_format f;
>> +       int ret;
>> +
>> +       ret = check_fmt(file, ef->type);
>> +       if (ret)
>> +               return ret;
>> +
>> +       switch (ef->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +               if (ops->vidioc_s_ext_pix_fmt_vid_cap)
>> +                       return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
>> +               break;
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +               if (ops->vidioc_s_ext_pix_fmt_vid_out)
>> +                       return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
>> +               break;
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       ret = v4l2_ext_pix_format_to_format(ef, &f,
>> +                                           vfd->device_caps &
>> +                                           (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +                                            V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +                                            V4L2_CAP_VIDEO_M2M_MPLANE),
>> +                                           false);
>> +       if (ret)
>> +               return ret;
>> +
>> +       ret = v4l_s_fmt(ops, file, fh, &f);
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>> +static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +                              struct file *file, void *fh,
>> +                              struct v4l2_format *f)
>> +{
>> +       struct v4l2_ext_pix_format ef;
>> +       int ret;
>> +
>> +       ret = v4l2_format_to_ext_pix_format(f, &ef, false);
>> +       if (ret)
>> +               return ret;
>> +
>> +       switch (f->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +               ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +               break;
>> +
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +               ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
>> +               break;
>> +
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_ext_pix_format_to_format(&ef, f,
>> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +                                            true);
>> +}
>> +
>> +
>>  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>                                 struct file *file, void *fh, void *arg)
>>  {
>> @@ -1765,23 +2133,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>
>>         switch (p->type) {
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -               if (unlikely(!ops->vidioc_try_fmt_vid_cap))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix);
>> -               ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> -                       v4l_pix_format_touch(&p->fmt.pix);
>> -               return ret;
>> +               if (ops->vidioc_try_fmt_vid_cap) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
>> +                       ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +                               v4l_pix_format_touch(&p->fmt.pix);
>> +                       return ret;
>> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +                       ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +                               v4l_pix_format_touch(&p->fmt.pix);
>> +                       return ret;
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -               if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -                                         bytesperline);
>> -               return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
>> +               if (ops->vidioc_try_fmt_vid_cap_mplane) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +                                                 bytesperline);
>> +                       return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
>> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>                 if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
>>                         break;
>> @@ -1798,21 +2175,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>                 CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>>                 return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -               if (unlikely(!ops->vidioc_try_fmt_vid_out))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix);
>> -               ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               return ret;
>> +               if (ops->vidioc_try_fmt_vid_out) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
>> +                       ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       return ret;
>> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -               if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -                                         bytesperline);
>> -               return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
>> +               if (ops->vidioc_try_fmt_vid_out_mplane) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +                                                 bytesperline);
>> +                       return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
>> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>                 if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
>>                         break;
>> @@ -1852,6 +2235,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>         return -EINVAL;
>>  }
>>
>> +static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +                              struct file *file, void *fh, void *arg)
>> +{
>> +       struct video_device *vfd = video_devdata(file);
>> +       struct v4l2_ext_pix_format *ef = arg;
>> +       struct v4l2_format f;
>> +       int ret;
>> +
>> +       ret = check_fmt(file, ef->type);
>> +       if (ret)
>> +               return ret;
>> +
>> +       switch (ef->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +               if (ops->vidioc_try_ext_pix_fmt_vid_cap)
>> +                       return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
>> +                                                                  ef);
>> +               break;
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +               if (ops->vidioc_try_ext_pix_fmt_vid_out)
>> +                       return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
>> +                                                                  ef);
>> +               break;
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       ret = v4l2_ext_pix_format_to_format(ef, &f,
>> +                                           vfd->device_caps &
>> +                                           (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +                                            V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +                                            V4L2_CAP_VIDEO_M2M_MPLANE),
>> +                                           false);
>> +       if (ret)
>> +               return ret;
>> +
>> +       ret = v4l_try_fmt(ops, file, fh, &f);
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>>  static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
>>                                 struct file *file, void *fh, void *arg)
>>  {
>> @@ -2771,7 +3197,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>         IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
>>         IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
>>         IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
>> +       IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>>         IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
>> +       IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>>         IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>         IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>>         IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>> @@ -2818,6 +3246,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>         IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
>>         IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
>>         IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
>> +       IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>>         IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
>>         IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
>>         IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
>> index 86878fba332b0..8bbcb74d8ee31 100644
>> --- a/include/media/v4l2-ioctl.h
>> +++ b/include/media/v4l2-ioctl.h
>> @@ -48,11 +48,17 @@ struct v4l2_fh;
>>   * @vidioc_g_fmt_vid_cap: pointer to the function that implements
>>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *     in single plane mode
>> + * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *     :ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *     capture
>>   * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_g_fmt_vid_out: pointer to the function that implements
>>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *     in single plane mode
>> + * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *     :ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *     out
>>   * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>>   * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
>> @@ -82,11 +88,16 @@ struct v4l2_fh;
>>   * @vidioc_s_fmt_vid_cap: pointer to the function that implements
>>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *     in single plane mode
>> + * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *     :ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *     capture
>>   * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_s_fmt_vid_out: pointer to the function that implements
>>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *     in single plane mode
>> + * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *     :ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>>   * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
>> @@ -116,11 +127,16 @@ struct v4l2_fh;
>>   * @vidioc_try_fmt_vid_cap: pointer to the function that implements
>>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *     in single plane mode
>> + * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *     :ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for
>> +       video capture
>>   * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_try_fmt_vid_out: pointer to the function that implements
>>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *     in single plane mode
>> + * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *     :ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   *     output
>> @@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
>>         /* VIDIOC_G_FMT handlers */
>>         int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
>>                                     struct v4l2_format *f);
>> +       int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +                                           struct v4l2_ext_pix_format *f);
>>         int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
>>                                         struct v4l2_format *f);
>>         int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
>>                                     struct v4l2_format *f);
>> +       int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +                                           struct v4l2_ext_pix_format *f);
>>         int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
>>                                             struct v4l2_format *f);
>>         int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
>>         /* VIDIOC_S_FMT handlers */
>>         int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
>>                                     struct v4l2_format *f);
>> +       int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +                                           struct v4l2_ext_pix_format *f);
>>         int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
>>                                         struct v4l2_format *f);
>>         int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
>>                                     struct v4l2_format *f);
>> +       int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +                                           struct v4l2_ext_pix_format *f);
>>         int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
>>                                             struct v4l2_format *f);
>>         int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
>>         /* VIDIOC_TRY_FMT handlers */
>>         int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
>>                                       struct v4l2_format *f);
>> +       int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +                                             struct v4l2_ext_pix_format *f);
>>         int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
>>                                           struct v4l2_format *f);
>>         int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
>>                                       struct v4l2_format *f);
>> +       int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +                                             struct v4l2_ext_pix_format *f);
>>         int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
>>                                              struct v4l2_format *f);
>>         int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
>>  long int video_ioctl2(struct file *file,
>>                       unsigned int cmd, unsigned long int arg);
>>
>> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>> +                                 struct v4l2_ext_pix_format *e, bool strict);
>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>> +                                 struct v4l2_format *f,
>> +                                 bool mplane_cap, bool strict);
>> +
>>  /*
>>   * The user space interpretation of the 'v4l2_event' differs
>>   * based on the 'time_t' definition on 32-bit architectures, so
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index c7b70ff53bc1d..7123c6a4d9569 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>>         __u8                            reserved[7];
>>  } __attribute__ ((packed));
>>
>> +/**
>> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
>> + * @sizeimage:         maximum size in bytes required for data, for which
>> + *                     this plane will be used.
>> + *                     Should be set to zero for unused planes.
>> + * @bytesperline:      distance in bytes between the leftmost pixels in two
>> + *                     adjacent lines
>> + * @reserved:          extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_plane_ext_pix_format {
>> +       __u32 sizeimage;
>> +       __u32 bytesperline;
>> +       __u32 reserved;
> 
> Do we want to make this field a __u64 so the size of the struct is a
> multiple of 64 bits?

For next version, I'm reusing struct v4l2_plane_pix_format as suggested by Tomasz,
so this adjustment won't be required.

Thanks,
Helen

> 
> 
> 
>> +};
>> +
>> +/**
>> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
>> + * @type:              type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
>> + *                     V4L2_BUF_TYPE_VIDEO_OUTPUT
>> + * @width:             image width in pixels
>> + * @height:            image height in pixels
>> + * @field:             enum v4l2_field; field order (for interlaced video)
>> + * @pixelformat:       little endian four character code (fourcc)
>> + * @modifier:          modifier applied to the format (used for tiled formats
>> + *                     and other kind of HW-specific formats, like compressed
>> + *                     formats)
>> + * @colorspace:                enum v4l2_colorspace; supplemental to pixelformat
>> + * @plane_fmt:         per-plane information
>> + * @ycbcr_enc:         enum v4l2_ycbcr_encoding, Y'CbCr encoding
>> + * @hsv_enc:           enum v4l2_hsv_encoding, HSV encoding
>> + * @quantization:      enum v4l2_quantization, colorspace quantization
>> + * @xfer_func:         enum v4l2_xfer_func, colorspace transfer function
>> + * @reserved:          extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_ext_pix_format {
>> +       __u32 type;
>> +       __u32 width;
>> +       __u32 height;
>> +       __u32 field;
>> +       __u64 modifier;
>> +       __u32 pixelformat;
>> +       __u32 colorspace;
>> +       struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
>> +       union {
>> +               __u32 ycbcr_enc;
>> +               __u32 hsv_enc;
>> +       };
>> +       __u32 quantization;
>> +       __u32 xfer_func;
>> +       __u32 reserved[9];
>> +};
>> +
>>  /**
>>   * struct v4l2_sdr_format - SDR format definition
>>   * @pixelformat:       little endian four character code (fourcc)
>> @@ -2571,6 +2623,10 @@ struct v4l2_create_buffers {
>>
>>  #define VIDIOC_QUERY_EXT_CTRL  _IOWR('V', 103, struct v4l2_query_ext_ctrl)
>>
>> +#define VIDIOC_G_EXT_PIX_FMT   _IOWR('V', 104, struct v4l2_ext_pix_format)
>> +#define VIDIOC_S_EXT_PIX_FMT   _IOWR('V', 105, struct v4l2_ext_pix_format)
>> +#define VIDIOC_TRY_EXT_PIX_FMT _IOWR('V', 106, struct v4l2_ext_pix_format)
>> +
>>  /* Reminder: when adding new ioctls please add support for them to
>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>
>> --
>> 2.28.0.rc2
>>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 7/7] media: docs: add documentation for the Extended API
  2020-08-04 19:29 ` [PATCH v5 7/7] media: docs: add documentation for the Extended API Helen Koike
  2020-08-14  7:49   ` Alexandre Courbot
  2020-11-19 10:28   ` Helen Koike
@ 2020-11-20 11:06   ` Tomasz Figa
  2020-11-20 12:24     ` Hans Verkuil
  2 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-11-20 11:06 UTC (permalink / raw)
  To: Helen Koike
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, hiroh, nicolas, Brian.Starkey, kernel,
	boris.brezillon, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Hi Helen,

On Tue, Aug 04, 2020 at 04:29:39PM -0300, Helen Koike wrote:
> Add documentation and update references in current documentation for the
> Extended API.
> 

Thank you for the patch. Please see my comments inline.

> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v5:
> - new patch
> 
>  .../userspace-api/media/v4l/buffer.rst        |   5 +
>  .../userspace-api/media/v4l/common.rst        |   1 +
>  .../userspace-api/media/v4l/dev-capture.rst   |   5 +
>  .../userspace-api/media/v4l/dev-output.rst    |   5 +
>  .../userspace-api/media/v4l/ext-api.rst       | 107 +++++++++
>  .../userspace-api/media/v4l/format.rst        |  16 +-
>  .../userspace-api/media/v4l/user-func.rst     |   5 +
>  .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++++++++
>  .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++++++
>  .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++++++++++++++++
>  .../media/v4l/vidioc-ext-querybuf.rst         |  79 +++++++
>  .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 ++++++++++
>  12 files changed, 697 insertions(+), 4 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> index 57e752aaf414a..c832bedd64e4c 100644
> --- a/Documentation/userspace-api/media/v4l/buffer.rst
> +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> @@ -27,6 +27,11 @@ such as pointers and sizes for each plane, are stored in
>  struct :c:type:`v4l2_plane` instead. In that case,
>  struct :c:type:`v4l2_buffer` contains an array of plane structures.
>  
> +.. note::
> +
> +    The :ref:`ext_api` version can also be used, and it is
> +    preferable when applicable.

Would rephrasing this as below making a bit more definitive?

	For modern applications, these operations are replaced by their
	:ref:`ext_api` counterparts, which should be used instead.

> +
>  Dequeued video buffers come with timestamps. The driver decides at which
>  part of the frame and with which clock the timestamp is taken. Please
>  see flags in the masks ``V4L2_BUF_FLAG_TIMESTAMP_MASK`` and
> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst
> index 7d81c58a13cd7..3430e0bdad667 100644
> --- a/Documentation/userspace-api/media/v4l/common.rst
> +++ b/Documentation/userspace-api/media/v4l/common.rst
> @@ -59,6 +59,7 @@ applicable to all devices.
>      ext-ctrls-detect
>      fourcc
>      format
> +    ext-api
>      planar-apis
>      selection-api
>      crop
> diff --git a/Documentation/userspace-api/media/v4l/dev-capture.rst b/Documentation/userspace-api/media/v4l/dev-capture.rst
> index 44d3094093ab6..5077639787d92 100644
> --- a/Documentation/userspace-api/media/v4l/dev-capture.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-capture.rst
> @@ -102,6 +102,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>  
> +.. note::
> +
> +    The :ref:`ext_api` version can also be used, and it is
> +    preferable when applicable.
> +
>  
>  Reading Images
>  ==============
> diff --git a/Documentation/userspace-api/media/v4l/dev-output.rst b/Documentation/userspace-api/media/v4l/dev-output.rst
> index e4f2a1d8b0fc7..f8f40c708e49f 100644
> --- a/Documentation/userspace-api/media/v4l/dev-output.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-output.rst
> @@ -99,6 +99,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>  
> +.. note::
> +
> +    The :ref:`ext_api` version can also be used, and it is
> +    preferable when applicable.
> +
>  
>  Writing Images
>  ==============
> diff --git a/Documentation/userspace-api/media/v4l/ext-api.rst b/Documentation/userspace-api/media/v4l/ext-api.rst
> new file mode 100644
> index 0000000000000..da2a82960d22f
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/ext-api.rst
> @@ -0,0 +1,107 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _ext_api:
> +
> +*************
> +Extendend API
> +*************
> +
> +Introduction
> +============
> +
> +The Extended Format API was conceived to extend V4L2 capabilities and
> +to simplify certain mechanisms.
> +It unifies single- vs multi- planar handling,
> +brings the concept of pixelformat + modifiers, allows planes to be placed
> +in any offset inside a buffer and let userspace to change colorspace
> +attributes if supported by the driver.

How about simplifying this paragraph a bit:

The Extended Format API unifies single- and multi- planar format handling
and provides a base for further functional extensions, such as format
modifiers, arbitrary plane offsets, colorspace configuration.

> +
> +Data format negotiation and buffer handling works very similar to the classical
> +version, thus in this document we'll just mention the main differences.

nit: "we will" or maybe even just "we mention only the main differences".

> +
> +Data Format Negotiation
> +=======================
> +
> +The API replaces the classical ioctls:
> +
> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>`, :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`,
> +:ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>`
> +(with :c:type:`v4l2_format` as the main parameter).
> +
> +By the extended versions:
> +
> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
> +:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
> +(with :c:type:`v4l2_ext_pix_format` as the main parameter).
> +
> +For CAPTURE and OUTPUT queues only.
> +
> +The ``type`` field of struct :c:type:`v4l2_ext_pix_format` only accepts 
> +``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or ``V4L2_BUF_TYPE_VIDEO_OUTPUT``, and it
> +supports multiplanar format through a combination of ``pixelfomat`` and
> +``modifier`` fields.
> +
> +Only the single-planar variants of the pixel formats are valid in the
> +``pixelformat`` field.

Some formats have only multi-planar variants, like NV12MT. What should we
do about them? Since we have to keep those defines to support the legacy
API anyway, maybe we can just accept both variants and treat them equally?

> +To support multi-planar, a modifier should be set in the ``modifier`` field to
> +provide more information regarding the memory layout of pixels and number of
> +planes.
> +
> +The ``plane_fmt`` field is an array which holds information by plane using
> +the :c:type:`v4l2_plane_ext_pix_format`. When this struct is filled, its
> +``sizeimage`` field should be non-zero for all valid planes for a given
> +``pixelformat`` + ``modifier`` combination, and zeroed for the invalid ones.
> +
> +Colospace attributes are not read-only as in the classical version, i.e, they
> +can be set by application and drivers will adjust accordingly depending on what
> +is supported by the underlying hardware.
> +
> +Buffers
> +=======
> +
> +The API replaces the classical ioctls:
> +
> +:ref:`VIDIOC_CREATE_BUFS <VIDIOC_CREATE_BUFS>`
> +(with :c:type:`v4l2_create_buffers` as the main parameter),
> +:ref:`VIDIOC_QUERYBUF <VIDIOC_QUERYBUF>`, :ref:`VIDIOC_QBUF <VIDIOC_QBUF>`,
> +:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`, :ref:`VIDIOC_PREPARE_BUF <VIDIOC_PREPARE_BUF>`
> +(with :c:type:`v4l2_buffer` as the main parameter)
> +
> +By the extended versions:
> +
> +:ref:`VIDIOC_EXT_CREATE_BUFS <VIDIOC_EXT_CREATE_BUFS>`
> +(with :c:type:`v4l2_ext_create_buffers` as the main parameter),
> +:ref:`VIDIOC_EXT_QUERYBUF <VIDIOC_EXT_QUERYBUF>`,
> +:ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>`,
> +:ref:`VIDIOC_EXT_DQBUF <VIDIOC_EXT_QBUF>`,
> +:ref:`VIDIOC_EXT_PREPARE_BUF <VIDIOC_EXT_PREPARE_BUF>`
> +(with :c:type:`v4l2_ext_buffer` as the main parameter)
> +
> +The basic difference between :c:type:`v4l2_create_buffers` and
> +:c:type:`v4l2_ext_create_buffers` is that the later have a
> +:c:type:`v4l2_ext_pix_format` as the type of the ``format`` field.
> +
> +Comparing :c:type:`v4l2_ext_buffer` with :c:type:`v4l2_buffer`, in the
> +extended version there is a unification of the single-/multi- planar handling
> +through the ``planes`` field of type :c:type:`v4l2_ext_plane`.
> +
> +The :c:type:`v4l2_ext_plane also allows planes to be placed in a given offset
> +inside a buffer.
> +Planes can be placed in different locations inside the same buffer, or in
> +different buffers.
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_plane
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_buffer
> diff --git a/Documentation/userspace-api/media/v4l/format.rst b/Documentation/userspace-api/media/v4l/format.rst
> index e47fc0505727c..b96d26f26793c 100644
> --- a/Documentation/userspace-api/media/v4l/format.rst
> +++ b/Documentation/userspace-api/media/v4l/format.rst
> @@ -28,13 +28,19 @@ format and the driver selects and reports the best the hardware can do
>  to satisfy the request. Of course applications can also just query the
>  current selection.
>  
> -A single mechanism exists to negotiate all data formats using the
> -aggregate struct :c:type:`v4l2_format` and the
> +There are two mechanism to negociate data formats:

negotiate

> +
> +The first one is using the aggregate struct :c:type:`v4l2_format` and the
>  :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and
>  :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctls. Additionally the
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` ioctl can be used to examine
>  what the hardware *could* do, without actually selecting a new data
> -format. The data formats supported by the V4L2 API are covered in the
> +format.
> +
> +The second is through the :ref:`ext_api`, please refer to its documentation
> +for more information.
> +

Should we reverse the order to prefer the new API?

> +The data formats supported by the V4L2 API are covered in the
>  respective device section in :ref:`devices`. For a closer look at
>  image formats see :ref:`pixfmt`.
>  
> @@ -71,7 +77,9 @@ earlier versions of V4L2. Switching the logical stream or returning into
>  *may* support a switch using :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`.
>  
>  All drivers exchanging data with applications must support the
> -:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl. Implementation of the
> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl
> +or the Extended respective versions (TODO: link).

TODO? :)

> +Implementation of the
>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is highly recommended but optional.
>  
>  
> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
> index bf77c842718e5..67b254812791c 100644
> --- a/Documentation/userspace-api/media/v4l/user-func.rst
> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
> @@ -20,6 +20,7 @@ Function Reference
>      func-close
>      func-ioctl
>      vidioc-create-bufs
> +    vidioc-ext-create-bufs
>      vidioc-cropcap
>      vidioc-dbg-g-chip-info
>      vidioc-dbg-g-register
> @@ -48,6 +49,7 @@ Function Reference
>      vidioc-g-ext-ctrls
>      vidioc-g-fbuf
>      vidioc-g-fmt
> +    vidioc-g-ext-pix-fmt
>      vidioc-g-frequency
>      vidioc-g-input
>      vidioc-g-jpegcomp
> @@ -62,8 +64,11 @@ Function Reference
>      vidioc-log-status
>      vidioc-overlay
>      vidioc-prepare-buf
> +    vidioc-ext-prepare-buf
>      vidioc-qbuf
> +    vidioc-ext-qbuf
>      vidioc-querybuf
> +    vidioc-ext-querybuf
>      vidioc-querycap
>      vidioc-queryctrl
>      vidioc-query-dv-timings
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
> new file mode 100644
> index 0000000000000..67f2c541e4d79
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
> @@ -0,0 +1,95 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +

TODO? :)

> +.. _VIDIOC_EXT_CREATE_BUFS:
> +
> +****************************
> +ioctl VIDIOC_EXT_CREATE_BUFS
> +****************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_CREATE_BUFS - Create buffers for Memory Mapped or User Pointer or DMA Buffer I/O
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp )
> +    :name: VIDIOC_EXT_CREATE_BUFS
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_create_buffers`.
> +
> +
> +Description
> +===========
> +
> +This ioctl is used to create buffers for :ref:`memory mapped <mmap>`
> +or :ref:`user pointer <userp>` or :ref:`DMA buffer <dmabuf>` I/O.
> +This ioctl can be called multiple times to
> +create buffers of different sizes.
> +
> +To allocate the device buffers applications must initialize the relevant
> +fields of the struct :c:type:`v4l2_ext_create_buffers` structure. The
> +``count`` field must be set to the number of requested buffers, the
> +``memory`` field specifies the requested I/O method and the ``reserved``
> +array must be zeroed.
> +
> +The ``format`` field specifies the image format that the buffers must be
> +able to handle. The application has to fill in this struct
> +:c:type:`v4l2_format`. Usually this will be done using the
> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` or
> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctls to ensure that the
> +requested format is supported by the driver.
> +The driver may return an error if the size(s) are not supported by the
> +hardware (usually because they are too small).
> +
> +The driver can create a memory buffer per plane, or a single memory buffer
> +containing all the planes, with a minimum size according to the size
> +defined by the ``format.plane_fmt[i].sizeimage`` field.
> +Usually if the ``format.plane_fmt[i].sizeimage``
> +field is less than the minimum required for the given format, then an
> +error will be returned since drivers will typically not allow this. If
> +it is larger, then the value will be used as-is. In other words, the
> +driver may reject the requested size, but if it is accepted the driver
> +will use it unchanged.
> +
> +When the ioctl is called with a pointer to this structure the driver
> +will attempt to allocate up to the requested number of buffers and store
> +the actual number allocated and the starting index in the ``count`` and
> +the ``index`` fields respectively. On return ``count`` can be smaller
> +than the number requested.

The 'capabilities' and 'flags' fields seem to go undocumented.

> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_create_buffers
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +ENOMEM
> +    No memory to allocate buffers for :ref:`memory mapped <mmap>` I/O.
> +
> +EINVAL
> +    The buffer type (``format.type`` field), requested I/O method
> +    (``memory``) or format (``format`` field) is not valid.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
> new file mode 100644
> index 0000000000000..88e4b57121254
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
> @@ -0,0 +1,62 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_PREPARE_BUF:
> +
> +****************************
> +ioctl VIDIOC_EXT_PREPARE_BUF
> +****************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_PREPARE_BUF - Prepare a buffer for I/O
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_PREPARE_BUF
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> +
> +
> +Description
> +===========
> +
> +Applications can optionally call the :ref:`VIDIOC_EXT_PREPARE_BUF` ioctl to
> +pass ownership of the buffer to the driver before actually enqueuing it,
> +using the :ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>` ioctl, and to prepare it for future I/O. Such
> +preparations may include cache invalidation or cleaning. Performing them
> +in advance saves time during the actual I/O.
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EBUSY
> +    File I/O is in progress.
> +
> +EINVAL
> +    The buffer ``type`` is not supported, or the ``index`` is out of
> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
> +    ``length`` are invalid.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
> new file mode 100644
> index 0000000000000..083e106cf6bad
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
> @@ -0,0 +1,204 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_QBUF:
> +
> +***************************************
> +ioctl VIDIOC_EXT_QBUF, VIDIOC_EXT_DQBUF
> +***************************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_QBUF - VIDIOC_EXT_DQBUF - Exchange a buffer with the driver
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_QBUF
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_DQBUF
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> +
> +
> +Description
> +===========
> +
> +Applications call the ``VIDIOC_EXT_QBUF`` ioctl to enqueue an empty
> +(capturing) or filled (output) buffer in the driver's incoming queue.
> +The semantics depend on the selected I/O method.
> +
> +To enqueue a buffer applications set the ``type`` field of a struct
> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``.
> +Applications must also set the ``index`` field. Valid index numbers
> +range from zero to the number of buffers allocated with
> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
> +one. The contents of the struct :c:type:`v4l2_ext_buffer` returned

I think this might not be entirely accurate. One can call
VIDIOC_EXT_CREATE_BUFS multiple times. Moreover, I don't think we've
prohibited VIDIOC_REQBUFS for the ext API, or have we?

> +by a :ref:`VIDIOC_EXT_QUERYBUF` ioctl will do as well.
> +When the buffer is intended for output (``type`` is
> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``) applications must also initialize the
> +``timestamp`` fields and the ``planes[i].plane_length`` for each valid plane,
> +and invalid ones must be set as zero.
> +see :ref:`buffer` for details. Applications must also set ``flags`` to 0. The
> +``reserved`` field must be set to 0.
> +
> +To enqueue a :ref:`memory mapped <mmap>` buffer applications set the
> +``planes[i].memory`` field to ``V4L2_MEMORY_MMAP`` in all the valid planes,
> +and invalid ones must be set as zero.
> +When ``VIDIOC_EXT_QBUF`` is called
> +with a pointer to this structure the driver sets the
> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_QUEUED`` flags and clears
> +the ``V4L2_BUF_FLAG_DONE`` flag in the ``flags`` field, or it returns an
> +``EINVAL`` error code.
> +
> +To enqueue a :ref:`user pointer <userp>` buffer applications set the
> +``planes[i].memory`` field to ``V4L2_MEMORY_USERPTR`` in all the valid planes,
> +and invalid ones must be set as zero, the ``planes[i].m.userptr`` field to the
> +address of the buffer,``planes[i].buffer_length`` to the size of the memory
> +buffer, ``planes[i].plane_length`` to the size that should be used by the plane,
> +and ``planes[i].offset`` of the plane in the memory buffer.

Does offset make sense here if an arbitrary pointer can be given for the
start of the plane?

> +
> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure
> +the driver sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
> +``flags`` field, or it returns an error code. This ioctl locks the

This seems to be a copy of the text few lines above.

> +memory pages of the buffer in physical memory, they cannot be swapped
> +out to disk. Buffers remain locked until dequeued, until the
> +:ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
> +device is closed.

Why would CREATE_BUFS affect existing buffers?

> +
> +To enqueue a :ref:`DMABUF <dmabuf>` buffer applications set the
> +``planes[i].memory`` field to ``V4L2_MEMORY_DMABUF`` in all the valid planes,
> +and invalid ones must be set as zero, the ``planes[i].m.fd`` field to a
> +file descriptor associated with a DMABUF buffer.,``planes[i].buffer_length`` to
> +the size of the memory buffer, ``planes[i].plane_length`` to the size that

What is the buffer_length field needed for? DMA-bufs by definition are
kernel objects and the kernel should already know their sizes.

> +should be used by the plane, and ``planes[i].offset`` of the plane in the memory buffer.
> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure the driver
> +sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
> +``flags`` field, or it returns an error code. This ioctl locks the

Ditto.

> +buffer. Locking a buffer means passing it to a driver for a hardware
> +access (usually DMA). If an application accesses (reads/writes) a locked
> +buffer then the result is undefined. Buffers remain locked until
> +dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
> +device is closed.

Ditto.

> +
> +The ``request_fd`` field can be used with the ``VIDIOC_EXT_QBUF`` ioctl to specify
> +the file descriptor of a :ref:`request <media-request-api>`, if requests are
> +in use. Setting it means that the buffer will not be passed to the driver
> +until the request itself is queued. Also, the driver will apply any
> +settings associated with the request for this buffer. This field will
> +be ignored unless the ``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
> +If the device does not support requests, then ``EBADR`` will be returned.
> +If requests are supported but an invalid request file descriptor is given,
> +then ``EINVAL`` will be returned.
> +
> +.. caution::
> +   It is not allowed to mix queuing requests with queuing buffers directly.
> +   ``EBUSY`` will be returned if the first buffer was queued directly and
> +   then the application tries to queue a request, or vice versa. After
> +   closing the file descriptor, calling
> +   :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or calling :ref:`VIDIOC_EXT_CREATE_BUFS`
> +   the check for this will be reset.
> +
> +   For :ref:`memory-to-memory devices <mem2mem>` you can specify the
> +   ``request_fd`` only for output buffers, not for capture buffers. Attempting
> +   to specify this for a capture buffer will result in an ``EBADR`` error.
> +
> +Applications call the ``VIDIOC_EXT_DQBUF`` ioctl to dequeue a filled
> +(capturing) or displayed (output) buffer from the driver's outgoing

"displayed" is quite inaccurate given that the majority of use cases for
OUTPUT in V4L2 now are m2m devices. Maybe "consumed"?

> +queue. They just set the ``type``, ``planes[i].memory`` and ``reserved`` fields of

Why does the application have to set the memory field? It should be already
known by the kernel which owns the buffer at the given time.

> +a struct :c:type:`v4l2_ext_buffer` as above, when
> +``VIDIOC_EXT_DQBUF`` is called with a pointer to this structure the driver
> +fills the remaining fields or returns an error code. The driver may also
> +set ``V4L2_BUF_FLAG_ERROR`` in the ``flags`` field. It indicates a
> +non-critical (recoverable) streaming error. In such case the application
> +may continue as normal, but should be aware that data in the dequeued
> +buffer might be corrupted. When using the multi-planar API, the planes
> +array must be passed in as well.
> +
> +If the application sets the ``memory`` field to ``V4L2_MEMORY_DMABUF`` to
> +dequeue a :ref:`DMABUF <dmabuf>` buffer, the driver fills the ``m.fd`` field
> +with a file descriptor numerically the same as the one given to ``VIDIOC_EXT_QBUF``
> +when the buffer was enqueued. No new file descriptor is created at dequeue time
> +and the value is only for the application convenience.

Honestly, I don't like this aspect of the current API. It kind of implies
that the value returned is valid, but the file descriptor might have been
already closed. Could we just remove this and explicitly state that for
DMA-buf, it is the user space responsibility to track the mapping between
V4L2 indexes and its own buffer objects?

> +
> +By default ``VIDIOC_EXT_DQBUF`` blocks when no buffer is in the outgoing
> +queue. When the ``O_NONBLOCK`` flag was given to the
> +:ref:`open() <func-open>` function, ``VIDIOC_EXT_DQBUF`` returns
> +immediately with an ``EAGAIN`` error code when no buffer is available.
> +
> +
> +.. kernel-doc:: include/uapi/linux/videodev2.h
> +   :functions: v4l2_ext_buffers
> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EAGAIN
> +    Non-blocking I/O has been selected using ``O_NONBLOCK`` and no
> +    buffer was in the outgoing queue.
> +
> +EINVAL
> +    The buffer ``type`` is not supported, or the ``index`` is out of
> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
> +    ``length`` are invalid, or the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was

There is no 'length' now.

> +    set but the the given ``request_fd`` was invalid, or ``m.fd`` was
> +    an invalid DMABUF file descriptor.
> +
> +EIO
> +    ``VIDIOC_EXT_DQBUF`` failed due to an internal error. Can also indicate
> +    temporary problems like signal loss.
> +
> +    .. note::
> +
> +       The driver might dequeue an (empty) buffer despite returning
> +       an error, or even stop capturing. Reusing such buffer may be unsafe
> +       though and its details (e.g. ``index``) may not be returned either.
> +       It is recommended that drivers indicate recoverable errors by setting
> +       the ``V4L2_BUF_FLAG_ERROR`` and returning 0 instead. In that case the
> +       application should be able to safely reuse the buffer and continue
> +       streaming.

What is this for? It sounds like some legacy behavior, which maybe we could
drop with this new API?

> +
> +EPIPE
> +    ``VIDIOC_EXT_DQBUF`` returns this on an empty capture queue for mem2mem
> +    codecs if a buffer with the ``V4L2_BUF_FLAG_LAST`` was already
> +    dequeued and no new buffers are expected to become available.
> +
> +EBADR
> +    The ``V4L2_BUF_FLAG_REQUEST_FD`` flag was set but the device does not
> +    support requests for the given buffer type, or
> +    the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was not set but the device requires
> +    that the buffer is part of a request.
> +
> +EBUSY
> +    The first buffer was queued via a request, but the application now tries
> +    to queue it directly, or vice versa (it is not permitted to mix the two
> +    APIs).
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
> new file mode 100644
> index 0000000000000..f2a12017253f6
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
> @@ -0,0 +1,79 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_EXT_QUERYBUF:
> +
> +*************************
> +ioctl VIDIOC_EXT_QUERYBUF
> +*************************
> +
> +Name
> +====
> +
> +VIDIOC_EXT_QUERYBUF - Query the status of a buffer
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp )
> +    :name: VIDIOC_EXT_QUERYBUF
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> +
> +
> +Description
> +===========
> +
> +This ioctl is part of the :ref:`streaming <mmap>` I/O method. It can
> +be used to query the status of a buffer at any time after buffers have
> +been allocated with the :ref:`VIDIOC_EXT_CREATE_BUFS` ioctl.
> +
> +Applications set the ``type`` field of a struct
> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``,
> +and the ``index`` field. Valid index numbers range from zero to the
> +number of buffers allocated with
> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
> +one. The ``reserved`` field must be set to 0.
> +
> +In the ``flags`` field the ``V4L2_BUF_FLAG_MAPPED``,
> +``V4L2_BUF_FLAG_PREPARED``, ``V4L2_BUF_FLAG_QUEUED`` and
> +``V4L2_BUF_FLAG_DONE`` flags will be valid. The ``planes.memory`` fields will be
> +set to the current I/O method for each plane.
> +
> +For every valid plane, an entry in ``planes`` will be filled, and zeroed for
> +invalid ones. ``planes[i].buffer_length`` is the size of the memory buffer
> +which contains the plane, ``planes[i].plane_length`` is the length of the plane,

What is the length of the plane here? It seems like depending on the
context, the field can mean size of the payload (for filled buffer) or
total space for the payload (for empty buffer), which is a bit confusing.

> +and ``planes[i].offset` is where the plane is placed in the memory buffer.
> +
> +The size of the ``planes`` array can be calculated by the number of sequential
> +planes with ``planes[i].buffer_length`` that differs from zero up to the max
> +size of the array.

Not sure if it's just me, but "size of array" sounds to me like the
total memory size of the array, i.e. the max number of elements, which
is fixed for this struct. How about

  "The entries of the planes array are valid up to the last entry with
  non-zero buffer_length or the last entry of the array, whichever is
  first."

?

> +
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EINVAL
> +    The buffer ``type`` is not supported, or the ``index`` is out of
> +    bounds.
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> new file mode 100644
> index 0000000000000..008e6c98a88a5
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> @@ -0,0 +1,117 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> +
> +.. _VIDIOC_G_EXT_PIX_FMT:
> +
> +************************************************************************
> +ioctl VIDIOC_G_EXT_PIX_FMT, VIDIOC_S_EXT_PIX_FMT, VIDIOC_TRY_EXT_PIX_FMT
> +************************************************************************
> +
> +Name
> +====
> +
> +VIDIOC_G_EXT_PIX_FMT - VIDIOC_S_EXT_PIX_FMT - VIDIOC_TRY_EXT_PIX_FMT - Get or set the data format, try a format
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> +    :name: VIDIOC_G_EXT_PIX_FMT
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> +    :name: VIDIOC_S_EXT_PIX_FMT
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> +    :name: VIDIOC_TRY_EXT_PIX_FMT
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_ext_pix_format`.
> +
> +
> +Description
> +===========
> +
> +These ioctls are used to negotiate the format of data (typically image
> +format) exchanged between driver and application.
> +
> +To query the current parameters applications set the ``type`` field of a
> +struct :c:type:`v4l2_ext_pix_format` to  ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or
> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``, all the other types are invalid in this API,
> +and multiplanar is supported through modifiers.
> +
> +When the application calls the
> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with a pointer to this
> +structure the driver fills the other members.
> +When the requested buffer type is not supported drivers return
> +an ``EINVAL`` error code.
> +
> +To change the current format parameters applications initialize all
> +the fields in the struct.
> +For details see the documentation of the various devices types in
> +:ref:`devices`. Good practice is to query the current parameters
> +first, and to modify only those parameters not suitable for the
> +application. When the application calls the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with
> +a pointer to a struct :c:type:`v4l2_ext_pix_format` structure the driver
> +checks and adjusts the parameters against hardware abilities. Drivers
> +should not return an error code unless the ``type`` field is invalid,
> +this is a mechanism to fathom device capabilities and to approach
> +parameters acceptable for both the application and driver. On success
> +the driver may program the hardware, allocate resources and generally
> +prepare for data exchange. Finally the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl returns
> +the current format parameters as :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` does. Very simple,
> +inflexible devices may even ignore all input and always return the
> +default parameters. However all V4L2 devices exchanging data with the
> +application must implement the :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` and :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
> +ioctl. When the requested buffer type is not supported drivers return an
> +EINVAL error code on a :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` attempt. When I/O is already in
> +progress or the resource is not available for other reasons drivers
> +return the ``EBUSY`` error code.
> +
> +The :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl is equivalent to :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` with one
> +exception: it does not change driver state. It can also be called at any
> +time, never returning ``EBUSY``. This function is provided to negotiate
> +parameters, to learn about hardware limitations, without disabling I/O
> +or possibly time consuming hardware preparations. Although strongly
> +recommended drivers are not required to implement this ioctl.
> +

Could we make this mandatory to avoid ambiguities in the API? In
general, most drivers implemented it anyway, without much added
complexity, because one can then simplify S_FMT to TRY_FMT + state
update.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-08-04 19:29 ` [PATCH v5 2/7] media: v4l2: Add extended buffer operations Helen Koike
  2020-08-14  7:49   ` Alexandre Courbot
  2020-09-09 12:27   ` Hans Verkuil
@ 2020-11-20 11:14   ` Tomasz Figa
  2020-11-23 20:33     ` Helen Koike
  2 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-11-20 11:14 UTC (permalink / raw)
  To: Helen Koike
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, Boris Brezillon, hiroh, nicolas, Brian.Starkey,
	kernel, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Hi Helen,

On Tue, Aug 04, 2020 at 04:29:34PM -0300, Helen Koike wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Those extended buffer ops have several purpose:
> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>    the number of ns elapsed since 1970
> 2/ Unify single/multiplanar handling
> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>    to support the case where a single buffer object is storing all
>    planes data, each one being placed at a different offset
> 
> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> these new objects.
> 
> The core takes care of converting new ioctls requests to old ones
> if the driver does not support the new hooks, and vice versa.
> 
> Note that the timecode field is gone, since there doesn't seem to be
> in-kernel users. We can be added back in the reserved area if needed or
> use the Request API to collect more metadata information from the
> frame.
> 

Thanks for the patch. Please see my comments inline.

> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v5:
> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> - return mem_offset to struct v4l2_ext_plane
> - change sizes and reorder fields to avoid holes in the struct and make
>   it the same for 32 and 64 bits
> 
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> I think we can add this later, so I removed it from this RFC to simplify it.
> - Remove num_planes field from struct v4l2_ext_buffer
> - Add flags field to struct v4l2_ext_create_buffers
> - Reformulate struct v4l2_ext_plane
> - Fix some bugs caught by v4l2-compliance
> - Rebased on top of media/master (post 5.8-rc1)
> 
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
> 
> Changes in v2:
> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>   later on
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>  include/media/v4l2-ioctl.h           |  26 ++
>  include/uapi/linux/videodev2.h       |  90 +++++++
>  4 files changed, 476 insertions(+), 22 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index e1829906bc086..cb21ee8eb075c 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -720,15 +720,34 @@ static void determine_valid_ioctls(struct video_device *vdev)
>  		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_sdr_out);
>  	}
>  
> +	if (is_vid || is_tch) {
> +		/* ioctls valid for video and touch */
> +		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
> +			set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
> +		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
> +			set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
> +		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
> +			set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
> +		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
> +			set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
> +		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
> +			set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);

nit: Could we stick to the SET_VALID_IOCTL() macro and just call it twice,
once for the new and once for the legacy callback?

> +	}
> +
>  	if (is_vid || is_vbi || is_sdr || is_tch || is_meta) {
>  		/* ioctls valid for video, vbi, sdr, touch and metadata */
>  		SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
> -		SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
> -		SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
>  		SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
> -		SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
> -		SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
> -		SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
> +		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
> +			set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
> +		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
> +			set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
> +		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
> +			set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
> +		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
> +			set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
> +		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
> +			set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);

Is it valid to check the new callbacks for devices that the new API is not
valid for (e.g. vbi)? Perhaps we could call SET_VALID_IOCTL(ops, <ioctl>,
vidioc_ext_*) in the upper if added in this patch and keep the code above
as is?

>  		SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
>  		SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
>  	}
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 14a0def50f8ea..7ecdd9cc1bf48 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -527,6 +527,26 @@ static void v4l_print_buffer(const void *arg, bool write_only)
>  			tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
>  }
>  
> +static void v4l_print_ext_buffer(const void *arg, bool write_only)
> +{
> +	const struct v4l2_ext_buffer *e = arg;
> +	const struct v4l2_ext_plane *plane;
> +	unsigned int i;
> +
> +	pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d\n",
> +		e->timestamp, e->index, prt_names(e->type, v4l2_type_names),
> +		e->flags, prt_names(e->field, v4l2_field_names), e->sequence);

Should we also print the request FD?

> +
> +	for (i = 0; i < VIDEO_MAX_PLANES &&
> +		    e->planes[i].buffer_length; i++) {
> +		plane = &e->planes[i];
> +		pr_debug("plane %d: buffer_length=%d, plane_length=%d offset=0x%08x, memory=%s\n",
> +			 i, plane->buffer_length, plane->plane_length,
> +			 plane->offset,
> +			 prt_names(plane->memory, v4l2_memory_names));

Should we also print mem_offset/userptr/dmabuf_fd?

> +	}
> +}
> +
>  static void v4l_print_exportbuffer(const void *arg, bool write_only)
>  {
>  	const struct v4l2_exportbuffer *p = arg;
> @@ -546,6 +566,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
>  	v4l_print_format(&p->format, write_only);
>  }
>  
> +static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
> +{
> +	const struct v4l2_ext_create_buffers *p = arg;
> +
> +	pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
> +		prt_names(p->memory, v4l2_memory_names));
> +	v4l_print_ext_pix_format(&p->format, write_only);

Should we also print capabilities and flags?

> +}
> +
>  static void v4l_print_streamparm(const void *arg, bool write_only)
>  {
>  	const struct v4l2_streamparm *p = arg;
> @@ -1220,6 +1249,143 @@ int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>  
> +/*
> + * If mplane_cap is true, b->m.planes should have a valid pointer of a
> + * struct v4l2_plane array, and b->length with its size
> + */
> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
> +			      struct v4l2_buffer *b, bool mplane_cap)
> +{
> +	unsigned int planes_array_size = b->length;
> +	struct v4l2_plane *planes = b->m.planes;
> +	u64 nsecs;
> +
> +	if (!mplane_cap && e->planes[1].buffer_length != 0)
> +		return -EINVAL;
> +
> +	memset(b, 0, sizeof(*b));
> +
> +	b->index = e->index;
> +	b->flags = e->flags;
> +	b->field = e->field;
> +	b->sequence = e->sequence;
> +	b->memory = e->planes[0].memory;
> +	b->request_fd = e->request_fd;
> +	b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
> +	b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
> +
> +	if (mplane_cap) {
> +		unsigned int i;
> +
> +		if (!planes || !planes_array_size)
> +			return -EINVAL;
> +
> +		b->m.planes = planes;

planes was initialized to b->m.planes at declaration time. Should we
perhaps move its declaration to within this block to make it more clear and
remove this assignment?

> +
> +		if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +			b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> +		else
> +			b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> +
> +		for (i = 0; i < VIDEO_MAX_PLANES && i < planes_array_size &&
> +			    e->planes[i].buffer_length; i++) {
> +
> +			if (e->planes[0].memory != e->planes[i].memory)
> +				return -EINVAL;
> +
> +			if (e->planes[i].offset)
> +				return -EINVAL;

Is it really invalid to have a non-zero offset? Shouldn't the data_offset
field of the legacy struct be populated instead, in the cases where it was
defined to be valid?

> +
> +			memset(&b->m.planes[i], 0, sizeof(b->m.planes[i]));
> +
> +			if (b->memory == V4L2_MEMORY_MMAP)
> +				b->m.planes[i].m.mem_offset = e->planes[i].m.mem_offset;
> +			else if (b->memory == V4L2_MEMORY_DMABUF)
> +				b->m.planes[i].m.fd = e->planes[i].m.dmabuf_fd;
> +			else
> +				b->m.planes[i].m.userptr = e->planes[i].m.userptr;
> +
> +			b->m.planes[i].bytesused = e->planes[i].plane_length;

I might be getting the meaning of plane_length wrong, but doesn't this
depend on the direction? If the userspace gives a CAPTURE buffer, it would
have bytesused = 0, but if the kernel returns it, it would have bytesused =
<size of the payload>.

> +			b->m.planes[i].length = e->planes[i].buffer_length;
> +		}
> +		/* In multi-planar, length contain the number of planes */
> +		b->length = i;
> +	} else {
> +		b->type = e->type;
> +		b->bytesused = e->planes[0].plane_length;
> +		b->length = e->planes[0].buffer_length;
> +
> +		if (e->planes[0].offset)
> +			return -EINVAL;

Ditto.

> +
> +		if (b->memory == V4L2_MEMORY_MMAP)
> +			b->m.offset = e->planes[0].m.mem_offset;
> +		else if (b->memory == V4L2_MEMORY_DMABUF)
> +			b->m.fd = e->planes[0].m.dmabuf_fd;
> +		else
> +			b->m.userptr = e->planes[0].m.userptr;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
> +
> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
> +			      struct v4l2_ext_buffer *e)
> +{
> +	memset(e, 0, sizeof(*e));
> +
> +	e->index = b->index;
> +	e->flags = b->flags;
> +	e->field = b->field;
> +	e->sequence = b->sequence;
> +	e->request_fd = b->request_fd;
> +	e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
> +		b->timestamp.tv_usec * NSEC_PER_USEC;
> +	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
> +		unsigned int i;
> +
> +		if (!b->m.planes)
> +			return -EINVAL;
> +
> +		if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +		else
> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +
> +		/* In multi-planar, length contain the number of planes */
> +		for (i = 0; i < b->length; i++) {

The design of the new struct implies that the planes describe color planes
and not memory planes, so this code is incorrect, because for non-M format
variants it would fill in only the first plane of the new struct.

> +			if (b->memory == V4L2_MEMORY_MMAP)
> +				e->planes[i].m.mem_offset = b->m.planes[i].m.mem_offset;
> +			else if (b->memory == V4L2_MEMORY_DMABUF)
> +				e->planes[i].m.dmabuf_fd = b->m.planes[i].m.fd;
> +			else
> +				e->planes[i].m.userptr = b->m.planes[i].m.userptr;
> +
> +			e->planes[i].memory = b->memory;
> +			e->planes[i].buffer_length = b->m.planes[i].length;
> +			e->planes[i].plane_length = b->m.planes[i].bytesused;
> +			if (b->m.planes[i].data_offset)
> +				pr_warn("Ignoring data_offset value %d\n",
> +					b->m.planes[i].data_offset);

Why? As per my comment above, there are valid use cases defined in the spec.

> +		}
> +	} else {
> +		e->type = b->type;
> +		e->planes[0].memory = b->memory;
> +		e->planes[0].plane_length = b->bytesused;
> +		e->planes[0].buffer_length = b->length;
> +		if (b->memory == V4L2_MEMORY_MMAP)
> +			e->planes[0].m.mem_offset = b->m.offset;
> +		else if (b->memory == V4L2_MEMORY_DMABUF)
> +			e->planes[0].m.dmabuf_fd = b->m.fd;
> +		else
> +			e->planes[0].m.userptr = b->m.userptr;

Similar to the MULTIPLANAR case, we should fill in the planes[] entries
corresponding to the number of color planes of the format, e.g. 2 for NV12.

> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
> +
>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -2473,31 +2639,112 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
>  	return ops->vidioc_reqbufs(file, fh, p);
>  }
>  
> +static int v4l_do_buf_op(int (*op)(struct file *, void *,
> +				   struct v4l2_buffer *),
> +			 int (*ext_op)(struct file *, void *,
> +				       struct v4l2_ext_buffer *),
> +			 struct file *file, void *fh, struct v4l2_buffer *b)
> +{
> +	struct v4l2_ext_buffer e;
> +	int ret;
> +
> +	ret = check_fmt(file, b->type);
> +	if (ret)
> +		return ret;
> +
> +	if (op)
> +		return op(file, fh, b);
> +
> +	ret = v4l2_buffer_to_ext_buffer(b, &e);
> +	if (ret)
> +		return ret;
> +
> +	ret = ext_op(file, fh, &e);
> +	if (ret)
> +		return ret;
> +
> +	v4l2_ext_buffer_to_buffer(&e, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
> +	return 0;
> +}
> +
> +static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
> +				       struct v4l2_buffer *),
> +			     int (*ext_op)(struct file *, void *,
> +					   struct v4l2_ext_buffer *),
> +			     struct file *file, void *fh,
> +			     struct v4l2_ext_buffer *e)
> +{
> +	struct video_device *vdev = video_devdata(file);
> +	struct v4l2_plane planes[VIDEO_MAX_PLANES];
> +	struct v4l2_buffer b;
> +	bool mplane_cap;
> +	int ret;
> +
> +	ret = check_fmt(file, e->type);
> +	if (ret)
> +		return ret;
> +
> +	if (ext_op)
> +		return ext_op(file, fh, e);
> +
> +	mplane_cap = !!(vdev->device_caps &
> +			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +			 V4L2_CAP_VIDEO_M2M_MPLANE));
> +	b.m.planes = planes;
> +	b.length = VIDEO_MAX_PLANES;
> +	ret = v4l2_ext_buffer_to_buffer(e, &b, mplane_cap);
> +	if (ret)
> +		return ret;
> +
> +	ret = op(file, fh, &b);
> +	if (ret)
> +		return ret;
> +
> +	v4l2_buffer_to_ext_buffer(&b, e);
> +	return 0;
> +}
> +
>  static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
> -				struct file *file, void *fh, void *arg)
> +			struct file *file, void *fh, void *arg)
>  {
> -	struct v4l2_buffer *p = arg;
> -	int ret = check_fmt(file, p->type);
> +	return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
> +			     file, fh, arg);
> +}
>  
> -	return ret ? ret : ops->vidioc_querybuf(file, fh, p);
> +static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
> +			    struct file *file, void *fh, void *arg)
> +{
> +	return v4l_do_ext_buf_op(ops->vidioc_querybuf,
> +				 ops->vidioc_ext_querybuf, file, fh, arg);
>  }
>  
>  static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
> -				struct file *file, void *fh, void *arg)
> +		    struct file *file, void *fh, void *arg)
>  {
> -	struct v4l2_buffer *p = arg;
> -	int ret = check_fmt(file, p->type);
> +	return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
> +			     file, fh, arg);
> +}
>  
> -	return ret ? ret : ops->vidioc_qbuf(file, fh, p);
> +static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
> +			struct file *file, void *fh, void *arg)
> +{
> +	return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
> +				 file, fh, arg);
>  }
>  
>  static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
> -				struct file *file, void *fh, void *arg)
> +		     struct file *file, void *fh, void *arg)
>  {
> -	struct v4l2_buffer *p = arg;
> -	int ret = check_fmt(file, p->type);
> +	return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
> +			     file, fh, arg);
> +}
>  
> -	return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
> +static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
> +			 struct file *file, void *fh, void *arg)
> +{
> +	return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
> +				 file, fh, arg);
>  }
>  
>  static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
> @@ -2513,7 +2760,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>  
>  	v4l_sanitize_format(&create->format);
>  
> -	ret = ops->vidioc_create_bufs(file, fh, create);
> +	if (ops->vidioc_create_bufs) {
> +		ret = ops->vidioc_create_bufs(file, fh, create);
> +	} else {
> +		struct v4l2_ext_create_buffers ecreate = {
> +			.count = create->count,
> +			.memory = create->memory,
> +		};
> +
> +		ret = v4l2_format_to_ext_pix_format(&create->format,
> +						    &ecreate.format, true);
> +		if (ret)
> +			return ret;
> +
> +		ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
> +		if (ret)
> +			return ret;
> +
> +		create->index = ecreate.index;
> +		create->count = ecreate.count;
> +		create->capabilities = ecreate.capabilities;
> +	}
>  
>  	if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>  	    create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
> @@ -2522,13 +2789,60 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>  	return ret;
>  }
>  
> +static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
> +			       struct file *file, void *fh, void *arg)
> +{
> +	struct v4l2_ext_create_buffers *ecreate = arg;
> +	struct video_device *vdev = video_devdata(file);
> +	struct v4l2_create_buffers create = {
> +		.count = ecreate->count,
> +		.memory = ecreate->memory,
> +		.flags = ecreate->flags,
> +	};
> +	bool mplane_cap;
> +	int ret;
> +
> +	ret = check_fmt(file, ecreate->format.type);
> +	if (ret)
> +		return ret;
> +
> +	if (ops->vidioc_ext_create_bufs)
> +		return ops->vidioc_ext_create_bufs(file, fh, ecreate);
> +
> +	mplane_cap = !!(vdev->device_caps &
> +			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +			 V4L2_CAP_VIDEO_M2M_MPLANE));
> +	ret = v4l2_ext_pix_format_to_format(&ecreate->format,
> +					    &create.format, mplane_cap, true);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l_create_bufs(ops, file, fh, &create);
> +	if (ret)
> +		return ret;
> +
> +	ecreate->index = create.index;
> +	ecreate->count = create.count;
> +	ecreate->capabilities = create.capabilities;
> +
> +	return 0;
> +}
> +
>  static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
> -				struct file *file, void *fh, void *arg)
> +			   struct file *file, void *fh, void *arg)
>  {
> -	struct v4l2_buffer *b = arg;
> -	int ret = check_fmt(file, b->type);
> +	return v4l_do_buf_op(ops->vidioc_prepare_buf,
> +			     ops->vidioc_ext_prepare_buf,
> +			     file, fh, arg);
> +}
>  
> -	return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
> +static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
> +			       struct file *file, void *fh, void *arg)
> +{
> +	return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
> +				 ops->vidioc_ext_prepare_buf,
> +				 file, fh, arg);
>  }
>  
>  static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
> @@ -3202,12 +3516,15 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>  	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
> +	IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, planes)),
>  	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>  	IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
> +	IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),

Looking at the other entries, shouldn't this one be 1 line higher?

That said, I wonder if it wouldn't look cleaner if we just put all the
EXT ioctls together at the bottom.

>  	IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
> +	IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
> @@ -3272,7 +3589,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>  	IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
>  	IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
>  	IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
> +	IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
> +	IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
>  	IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
>  	IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
> index 8bbcb74d8ee31..75996657ad1ba 100644
> --- a/include/media/v4l2-ioctl.h
> +++ b/include/media/v4l2-ioctl.h
> @@ -169,16 +169,26 @@ struct v4l2_fh;
>   *	:ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
>   * @vidioc_querybuf: pointer to the function that implements
>   *	:ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
> + * @vidioc_ext_querybuf: pointer to the function that implements
> + *	:ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
>   * @vidioc_qbuf: pointer to the function that implements
>   *	:ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
> + * @vidioc_ext_qbuf: pointer to the function that implements
> + *	:ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
>   * @vidioc_expbuf: pointer to the function that implements
>   *	:ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
>   * @vidioc_dqbuf: pointer to the function that implements
>   *	:ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
> + * @vidioc_ext_dqbuf: pointer to the function that implements
> + *	:ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
>   * @vidioc_create_bufs: pointer to the function that implements
>   *	:ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
> + * @vidioc_ext_create_bufs: pointer to the function that implements
> + *	:ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
>   * @vidioc_prepare_buf: pointer to the function that implements
>   *	:ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
> + * @vidioc_ext_prepare_buf: pointer to the function that implements
> + *	:ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
>   * @vidioc_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
>   * @vidioc_g_fbuf: pointer to the function that implements
> @@ -439,17 +449,27 @@ struct v4l2_ioctl_ops {
>  			      struct v4l2_requestbuffers *b);
>  	int (*vidioc_querybuf)(struct file *file, void *fh,
>  			       struct v4l2_buffer *b);
> +	int (*vidioc_ext_querybuf)(struct file *file, void *fh,
> +				   struct v4l2_ext_buffer *b);
>  	int (*vidioc_qbuf)(struct file *file, void *fh,
>  			   struct v4l2_buffer *b);
> +	int (*vidioc_ext_qbuf)(struct file *file, void *fh,
> +			       struct v4l2_ext_buffer *b);
>  	int (*vidioc_expbuf)(struct file *file, void *fh,
>  			     struct v4l2_exportbuffer *e);
>  	int (*vidioc_dqbuf)(struct file *file, void *fh,
>  			    struct v4l2_buffer *b);
> +	int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
> +				struct v4l2_ext_buffer *b);
>  
>  	int (*vidioc_create_bufs)(struct file *file, void *fh,
>  				  struct v4l2_create_buffers *b);
> +	int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
> +				      struct v4l2_ext_create_buffers *b);
>  	int (*vidioc_prepare_buf)(struct file *file, void *fh,
>  				  struct v4l2_buffer *b);
> +	int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
> +				      struct v4l2_ext_buffer *b);
>  
>  	int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
>  	int (*vidioc_g_fbuf)(struct file *file, void *fh,
> @@ -758,6 +778,12 @@ int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>  				  struct v4l2_format *f,
>  				  bool mplane_cap, bool strict);
>  
> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
> +			      struct v4l2_buffer *b,
> +			      bool mplane_cap);
> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
> +			      struct v4l2_ext_buffer *e);
> +
>  /*
>   * The user space interpretation of the 'v4l2_event' differs
>   * based on the 'time_t' definition on 32-bit architectures, so
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 7123c6a4d9569..334cafdd2be97 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -996,6 +996,41 @@ struct v4l2_plane {
>  	__u32			reserved[11];
>  };
>  
> +/**
> + * struct v4l2_ext_plane - extended plane buffer info
> + * @buffer_length:	size of the entire buffer in bytes, should fit
> + *			@offset + @plane_length

Do we actually need this buffer_length at all? We have 3 memory types:

1) MMAP - here vb2 already knows the buffer size, because it created it.

2) DMABUF - the DMA-buf kAPI provides the information about buffer size.

3) USERPTR - this might actually benefit from buffer_length, because there
   are additional alignmnent requirements for the user memory, e.g. the
   offset and size must be cacheline aligned.

Arguably, 1) and 2) are the main usage scenarios, while the user space that
uses them would have to suffer from added complexity, because of the
legacy/niche case 3).

Could we make this field valid only for USERPTR?

> + * @plane_length:	size of the plane in bytes.
> + * @mem_offset:		If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
> + *			that should be passed to mmap() called on the video node.
> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> + *			to this plane.
> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> + *			associated with this plane.
> + * @offset:		offset in the memory buffer where the plane starts.
> + * @memory:		enum v4l2_memory; the method, in which the actual video
> + *			data is passed
> + * @reserved:		extra space reserved for future fields, must be set to 0.
> + *
> + *
> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> + * can have one plane for Y, and another for interleaved CbCr components.
> + * Each plane can reside in a separate memory buffer, or even in
> + * a completely separate memory node (e.g. in embedded devices).
> + */
> +struct v4l2_ext_plane {
> +	__u32 buffer_length;
> +	__u32 plane_length;
> +	union {
> +		__u32 mem_offset;
> +		__u64 userptr;
> +		__s32 dmabuf_fd;
> +	} m;
> +	__u32 offset;
> +	__u32 memory;
> +	__u32 reserved[4];
> +};

Don't we also need bytesused? Or would plane_length essentially mean the
amount of space or payload, depending on the usage context?

Similarly, the original data_offset was useful as a return field, which
some drivers use to indicate that the beginning of the plane is occupied by
some header or otherwise irrelevant data, which must be skipped. Would the
offset field be used for this purpose now?

> +
>  /**
>   * struct v4l2_buffer - video buffer info
>   * @index:	id number of the buffer
> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>  	};
>  };
>  
> +/**
> + * struct v4l2_ext_buffer - extended video buffer info
> + * @index:	id number of the buffer
> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> + * @flags:	buffer informational flags

nit: The order of comments doesn't match the order of fields in the struct.

> + * @field:	enum v4l2_field; field order of the image in the buffer
> + * @timestamp:	frame timestamp
> + * @sequence:	sequence count of this frame
> + * @planes:	per-plane buffer information
> + * @request_fd:	fd of the request that this buffer should use
> + * @reserved:	extra space reserved for future fields, must be set to 0
> + *
> + * Contains data exchanged by application and driver using one of the Streaming
> + * I/O methods.
> + */
> +struct v4l2_ext_buffer {
> +	__u32 index;
> +	__u32 type;
> +	__u32 field;
> +	__u32 sequence;
> +	__u64 flags;
> +	__u64 timestamp;

What's the unit? How does this play with the other UAPI that the user space
may use, e.g. clock_gettime() which returns struct timespec?

> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> +	__s32 request_fd;
> +	__u32 reserved[9];
> +};
> +
>  #ifndef __KERNEL__
>  /**
>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
>  	__u32			reserved[6];
>  };
>  
> +/**
> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
> + * @index:	on return, index of the first created buffer
> + * @count:	entry: number of requested buffers,
> + *		return: number of created buffers
> + * @memory:	enum v4l2_memory; buffer memory type
> + * @capabilities: capabilities of this buffer type.
> + * @format:	frame format, for which buffers are requested
> + * @flags:	additional buffer management attributes (ignored unless the
> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
> + *		and configured for MMAP streaming I/O).

Do we need to say that it is ignored here? I'd consider this to be a
per-flag decision.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 7/7] media: docs: add documentation for the Extended API
  2020-11-20 11:06   ` Tomasz Figa
@ 2020-11-20 12:24     ` Hans Verkuil
  2020-11-20 12:40       ` Tomasz Figa
  0 siblings, 1 reply; 51+ messages in thread
From: Hans Verkuil @ 2020-11-20 12:24 UTC (permalink / raw)
  To: Tomasz Figa, Helen Koike
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, hiroh, nicolas, Brian.Starkey, kernel,
	boris.brezillon, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

On 20/11/2020 12:06, Tomasz Figa wrote:z
> Hi Helen,
> 
> On Tue, Aug 04, 2020 at 04:29:39PM -0300, Helen Koike wrote:
>> Add documentation and update references in current documentation for the
>> Extended API.
>>
> 
> Thank you for the patch. Please see my comments inline.
> 
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> ---
>> Changes in v5:
>> - new patch
>>
>>  .../userspace-api/media/v4l/buffer.rst        |   5 +
>>  .../userspace-api/media/v4l/common.rst        |   1 +
>>  .../userspace-api/media/v4l/dev-capture.rst   |   5 +
>>  .../userspace-api/media/v4l/dev-output.rst    |   5 +
>>  .../userspace-api/media/v4l/ext-api.rst       | 107 +++++++++
>>  .../userspace-api/media/v4l/format.rst        |  16 +-
>>  .../userspace-api/media/v4l/user-func.rst     |   5 +
>>  .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++++++++
>>  .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++++++
>>  .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++++++++++++++++
>>  .../media/v4l/vidioc-ext-querybuf.rst         |  79 +++++++
>>  .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 ++++++++++
>>  12 files changed, 697 insertions(+), 4 deletions(-)
>>  create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>>
>> diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
>> index 57e752aaf414a..c832bedd64e4c 100644
>> --- a/Documentation/userspace-api/media/v4l/buffer.rst
>> +++ b/Documentation/userspace-api/media/v4l/buffer.rst
>> @@ -27,6 +27,11 @@ such as pointers and sizes for each plane, are stored in
>>  struct :c:type:`v4l2_plane` instead. In that case,
>>  struct :c:type:`v4l2_buffer` contains an array of plane structures.
>>  
>> +.. note::
>> +
>> +    The :ref:`ext_api` version can also be used, and it is
>> +    preferable when applicable.
> 
> Would rephrasing this as below making a bit more definitive?
> 
> 	For modern applications, these operations are replaced by their
> 	:ref:`ext_api` counterparts, which should be used instead.

You can't say that, since especially in the beginning userspace will be running
on older kernels that do not support this.

This will work: "should be used instead, if supported by the driver."

> 
>> +
>>  Dequeued video buffers come with timestamps. The driver decides at which
>>  part of the frame and with which clock the timestamp is taken. Please
>>  see flags in the masks ``V4L2_BUF_FLAG_TIMESTAMP_MASK`` and
>> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst
>> index 7d81c58a13cd7..3430e0bdad667 100644
>> --- a/Documentation/userspace-api/media/v4l/common.rst
>> +++ b/Documentation/userspace-api/media/v4l/common.rst
>> @@ -59,6 +59,7 @@ applicable to all devices.
>>      ext-ctrls-detect
>>      fourcc
>>      format
>> +    ext-api
>>      planar-apis
>>      selection-api
>>      crop
>> diff --git a/Documentation/userspace-api/media/v4l/dev-capture.rst b/Documentation/userspace-api/media/v4l/dev-capture.rst
>> index 44d3094093ab6..5077639787d92 100644
>> --- a/Documentation/userspace-api/media/v4l/dev-capture.rst
>> +++ b/Documentation/userspace-api/media/v4l/dev-capture.rst
>> @@ -102,6 +102,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>>  
>> +.. note::
>> +
>> +    The :ref:`ext_api` version can also be used, and it is
>> +    preferable when applicable.
>> +
>>  
>>  Reading Images
>>  ==============
>> diff --git a/Documentation/userspace-api/media/v4l/dev-output.rst b/Documentation/userspace-api/media/v4l/dev-output.rst
>> index e4f2a1d8b0fc7..f8f40c708e49f 100644
>> --- a/Documentation/userspace-api/media/v4l/dev-output.rst
>> +++ b/Documentation/userspace-api/media/v4l/dev-output.rst
>> @@ -99,6 +99,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>>  
>> +.. note::
>> +
>> +    The :ref:`ext_api` version can also be used, and it is
>> +    preferable when applicable.
>> +
>>  
>>  Writing Images
>>  ==============
>> diff --git a/Documentation/userspace-api/media/v4l/ext-api.rst b/Documentation/userspace-api/media/v4l/ext-api.rst
>> new file mode 100644
>> index 0000000000000..da2a82960d22f
>> --- /dev/null
>> +++ b/Documentation/userspace-api/media/v4l/ext-api.rst
>> @@ -0,0 +1,107 @@
>> +.. Permission is granted to copy, distribute and/or modify this
>> +.. document under the terms of the GNU Free Documentation License,
>> +.. Version 1.1 or any later version published by the Free Software
>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>> +.. and no Back-Cover Texts. A copy of the license is included at
>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>> +..
>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>> +
>> +.. _ext_api:
>> +
>> +*************
>> +Extendend API
>> +*************
>> +
>> +Introduction
>> +============
>> +
>> +The Extended Format API was conceived to extend V4L2 capabilities and
>> +to simplify certain mechanisms.
>> +It unifies single- vs multi- planar handling,
>> +brings the concept of pixelformat + modifiers, allows planes to be placed
>> +in any offset inside a buffer and let userspace to change colorspace
>> +attributes if supported by the driver.
> 
> How about simplifying this paragraph a bit:
> 
> The Extended Format API unifies single- and multi- planar format handling
> and provides a base for further functional extensions, such as format
> modifiers, arbitrary plane offsets, colorspace configuration.

Actually, I prefer Helen's version.

> 
>> +
>> +Data format negotiation and buffer handling works very similar to the classical
>> +version, thus in this document we'll just mention the main differences.
> 
> nit: "we will" or maybe even just "we mention only the main differences".
> 
>> +
>> +Data Format Negotiation
>> +=======================
>> +
>> +The API replaces the classical ioctls:
>> +
>> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>`, :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`,
>> +:ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>`
>> +(with :c:type:`v4l2_format` as the main parameter).
>> +
>> +By the extended versions:
>> +
>> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
>> +:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
>> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
>> +(with :c:type:`v4l2_ext_pix_format` as the main parameter).
>> +
>> +For CAPTURE and OUTPUT queues only.
>> +
>> +The ``type`` field of struct :c:type:`v4l2_ext_pix_format` only accepts 
>> +``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or ``V4L2_BUF_TYPE_VIDEO_OUTPUT``, and it
>> +supports multiplanar format through a combination of ``pixelfomat`` and
>> +``modifier`` fields.
>> +
>> +Only the single-planar variants of the pixel formats are valid in the
>> +``pixelformat`` field.
> 
> Some formats have only multi-planar variants, like NV12MT. What should we
> do about them? Since we have to keep those defines to support the legacy
> API anyway, maybe we can just accept both variants and treat them equally?

Helen mentioned that as well in a reply to patch 1/7. I'll have a think about
that myself and reply to that post because this is a good question.

> 
>> +To support multi-planar, a modifier should be set in the ``modifier`` field to
>> +provide more information regarding the memory layout of pixels and number of
>> +planes.
>> +
>> +The ``plane_fmt`` field is an array which holds information by plane using
>> +the :c:type:`v4l2_plane_ext_pix_format`. When this struct is filled, its
>> +``sizeimage`` field should be non-zero for all valid planes for a given
>> +``pixelformat`` + ``modifier`` combination, and zeroed for the invalid ones.
>> +
>> +Colospace attributes are not read-only as in the classical version, i.e, they
>> +can be set by application and drivers will adjust accordingly depending on what
>> +is supported by the underlying hardware.
>> +
>> +Buffers
>> +=======
>> +
>> +The API replaces the classical ioctls:
>> +
>> +:ref:`VIDIOC_CREATE_BUFS <VIDIOC_CREATE_BUFS>`
>> +(with :c:type:`v4l2_create_buffers` as the main parameter),
>> +:ref:`VIDIOC_QUERYBUF <VIDIOC_QUERYBUF>`, :ref:`VIDIOC_QBUF <VIDIOC_QBUF>`,
>> +:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`, :ref:`VIDIOC_PREPARE_BUF <VIDIOC_PREPARE_BUF>`
>> +(with :c:type:`v4l2_buffer` as the main parameter)
>> +
>> +By the extended versions:
>> +
>> +:ref:`VIDIOC_EXT_CREATE_BUFS <VIDIOC_EXT_CREATE_BUFS>`
>> +(with :c:type:`v4l2_ext_create_buffers` as the main parameter),
>> +:ref:`VIDIOC_EXT_QUERYBUF <VIDIOC_EXT_QUERYBUF>`,
>> +:ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>`,
>> +:ref:`VIDIOC_EXT_DQBUF <VIDIOC_EXT_QBUF>`,
>> +:ref:`VIDIOC_EXT_PREPARE_BUF <VIDIOC_EXT_PREPARE_BUF>`
>> +(with :c:type:`v4l2_ext_buffer` as the main parameter)
>> +
>> +The basic difference between :c:type:`v4l2_create_buffers` and
>> +:c:type:`v4l2_ext_create_buffers` is that the later have a
>> +:c:type:`v4l2_ext_pix_format` as the type of the ``format`` field.
>> +
>> +Comparing :c:type:`v4l2_ext_buffer` with :c:type:`v4l2_buffer`, in the
>> +extended version there is a unification of the single-/multi- planar handling
>> +through the ``planes`` field of type :c:type:`v4l2_ext_plane`.
>> +
>> +The :c:type:`v4l2_ext_plane also allows planes to be placed in a given offset
>> +inside a buffer.
>> +Planes can be placed in different locations inside the same buffer, or in
>> +different buffers.
>> +
>> +
>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>> +   :functions: v4l2_ext_plane
>> +
>> +
>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>> +   :functions: v4l2_ext_buffer
>> diff --git a/Documentation/userspace-api/media/v4l/format.rst b/Documentation/userspace-api/media/v4l/format.rst
>> index e47fc0505727c..b96d26f26793c 100644
>> --- a/Documentation/userspace-api/media/v4l/format.rst
>> +++ b/Documentation/userspace-api/media/v4l/format.rst
>> @@ -28,13 +28,19 @@ format and the driver selects and reports the best the hardware can do
>>  to satisfy the request. Of course applications can also just query the
>>  current selection.
>>  
>> -A single mechanism exists to negotiate all data formats using the
>> -aggregate struct :c:type:`v4l2_format` and the
>> +There are two mechanism to negociate data formats:
> 
> negotiate
> 
>> +
>> +The first one is using the aggregate struct :c:type:`v4l2_format` and the
>>  :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and
>>  :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctls. Additionally the
>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` ioctl can be used to examine
>>  what the hardware *could* do, without actually selecting a new data
>> -format. The data formats supported by the V4L2 API are covered in the
>> +format.
>> +
>> +The second is through the :ref:`ext_api`, please refer to its documentation
>> +for more information.
>> +
> 
> Should we reverse the order to prefer the new API?
> 
>> +The data formats supported by the V4L2 API are covered in the
>>  respective device section in :ref:`devices`. For a closer look at
>>  image formats see :ref:`pixfmt`.
>>  
>> @@ -71,7 +77,9 @@ earlier versions of V4L2. Switching the logical stream or returning into
>>  *may* support a switch using :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`.
>>  
>>  All drivers exchanging data with applications must support the
>> -:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl. Implementation of the
>> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl
>> +or the Extended respective versions (TODO: link).
> 
> TODO? :)
> 
>> +Implementation of the
>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is highly recommended but optional.
>>  
>>  
>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
>> index bf77c842718e5..67b254812791c 100644
>> --- a/Documentation/userspace-api/media/v4l/user-func.rst
>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
>> @@ -20,6 +20,7 @@ Function Reference
>>      func-close
>>      func-ioctl
>>      vidioc-create-bufs
>> +    vidioc-ext-create-bufs
>>      vidioc-cropcap
>>      vidioc-dbg-g-chip-info
>>      vidioc-dbg-g-register
>> @@ -48,6 +49,7 @@ Function Reference
>>      vidioc-g-ext-ctrls
>>      vidioc-g-fbuf
>>      vidioc-g-fmt
>> +    vidioc-g-ext-pix-fmt
>>      vidioc-g-frequency
>>      vidioc-g-input
>>      vidioc-g-jpegcomp
>> @@ -62,8 +64,11 @@ Function Reference
>>      vidioc-log-status
>>      vidioc-overlay
>>      vidioc-prepare-buf
>> +    vidioc-ext-prepare-buf
>>      vidioc-qbuf
>> +    vidioc-ext-qbuf
>>      vidioc-querybuf
>> +    vidioc-ext-querybuf
>>      vidioc-querycap
>>      vidioc-queryctrl
>>      vidioc-query-dv-timings
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>> new file mode 100644
>> index 0000000000000..67f2c541e4d79
>> --- /dev/null
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>> @@ -0,0 +1,95 @@
>> +.. Permission is granted to copy, distribute and/or modify this
>> +.. document under the terms of the GNU Free Documentation License,
>> +.. Version 1.1 or any later version published by the Free Software
>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>> +.. and no Back-Cover Texts. A copy of the license is included at
>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>> +..
>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>> +
> 
> TODO? :)
> 
>> +.. _VIDIOC_EXT_CREATE_BUFS:
>> +
>> +****************************
>> +ioctl VIDIOC_EXT_CREATE_BUFS
>> +****************************
>> +
>> +Name
>> +====
>> +
>> +VIDIOC_EXT_CREATE_BUFS - Create buffers for Memory Mapped or User Pointer or DMA Buffer I/O
>> +
>> +
>> +Synopsis
>> +========
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp )
>> +    :name: VIDIOC_EXT_CREATE_BUFS
>> +
>> +
>> +Arguments
>> +=========
>> +
>> +``fd``
>> +    File descriptor returned by :ref:`open() <func-open>`.
>> +
>> +``argp``
>> +    Pointer to struct :c:type:`v4l2_ext_create_buffers`.
>> +
>> +
>> +Description
>> +===========
>> +
>> +This ioctl is used to create buffers for :ref:`memory mapped <mmap>`
>> +or :ref:`user pointer <userp>` or :ref:`DMA buffer <dmabuf>` I/O.
>> +This ioctl can be called multiple times to
>> +create buffers of different sizes.
>> +
>> +To allocate the device buffers applications must initialize the relevant
>> +fields of the struct :c:type:`v4l2_ext_create_buffers` structure. The
>> +``count`` field must be set to the number of requested buffers, the
>> +``memory`` field specifies the requested I/O method and the ``reserved``
>> +array must be zeroed.
>> +
>> +The ``format`` field specifies the image format that the buffers must be
>> +able to handle. The application has to fill in this struct
>> +:c:type:`v4l2_format`. Usually this will be done using the
>> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` or
>> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctls to ensure that the
>> +requested format is supported by the driver.
>> +The driver may return an error if the size(s) are not supported by the
>> +hardware (usually because they are too small).
>> +
>> +The driver can create a memory buffer per plane, or a single memory buffer
>> +containing all the planes, with a minimum size according to the size
>> +defined by the ``format.plane_fmt[i].sizeimage`` field.
>> +Usually if the ``format.plane_fmt[i].sizeimage``
>> +field is less than the minimum required for the given format, then an
>> +error will be returned since drivers will typically not allow this. If
>> +it is larger, then the value will be used as-is. In other words, the
>> +driver may reject the requested size, but if it is accepted the driver
>> +will use it unchanged.
>> +
>> +When the ioctl is called with a pointer to this structure the driver
>> +will attempt to allocate up to the requested number of buffers and store
>> +the actual number allocated and the starting index in the ``count`` and
>> +the ``index`` fields respectively. On return ``count`` can be smaller
>> +than the number requested.
> 
> The 'capabilities' and 'flags' fields seem to go undocumented.
> 
>> +
>> +
>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>> +   :functions: v4l2_ext_create_buffers
>> +
>> +
>> +Return Value
>> +============
>> +
>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>> +appropriately. The generic error codes are described at the
>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>> +
>> +ENOMEM
>> +    No memory to allocate buffers for :ref:`memory mapped <mmap>` I/O.
>> +
>> +EINVAL
>> +    The buffer type (``format.type`` field), requested I/O method
>> +    (``memory``) or format (``format`` field) is not valid.
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>> new file mode 100644
>> index 0000000000000..88e4b57121254
>> --- /dev/null
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>> @@ -0,0 +1,62 @@
>> +.. Permission is granted to copy, distribute and/or modify this
>> +.. document under the terms of the GNU Free Documentation License,
>> +.. Version 1.1 or any later version published by the Free Software
>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>> +.. and no Back-Cover Texts. A copy of the license is included at
>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>> +..
>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>> +
>> +.. _VIDIOC_EXT_PREPARE_BUF:
>> +
>> +****************************
>> +ioctl VIDIOC_EXT_PREPARE_BUF
>> +****************************
>> +
>> +Name
>> +====
>> +
>> +VIDIOC_EXT_PREPARE_BUF - Prepare a buffer for I/O
>> +
>> +
>> +Synopsis
>> +========
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp )
>> +    :name: VIDIOC_EXT_PREPARE_BUF
>> +
>> +
>> +Arguments
>> +=========
>> +
>> +``fd``
>> +    File descriptor returned by :ref:`open() <func-open>`.
>> +
>> +``argp``
>> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
>> +
>> +
>> +Description
>> +===========
>> +
>> +Applications can optionally call the :ref:`VIDIOC_EXT_PREPARE_BUF` ioctl to
>> +pass ownership of the buffer to the driver before actually enqueuing it,
>> +using the :ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>` ioctl, and to prepare it for future I/O. Such
>> +preparations may include cache invalidation or cleaning. Performing them
>> +in advance saves time during the actual I/O.
>> +
>> +
>> +Return Value
>> +============
>> +
>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>> +appropriately. The generic error codes are described at the
>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>> +
>> +EBUSY
>> +    File I/O is in progress.
>> +
>> +EINVAL
>> +    The buffer ``type`` is not supported, or the ``index`` is out of
>> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
>> +    ``length`` are invalid.
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>> new file mode 100644
>> index 0000000000000..083e106cf6bad
>> --- /dev/null
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>> @@ -0,0 +1,204 @@
>> +.. Permission is granted to copy, distribute and/or modify this
>> +.. document under the terms of the GNU Free Documentation License,
>> +.. Version 1.1 or any later version published by the Free Software
>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>> +.. and no Back-Cover Texts. A copy of the license is included at
>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>> +..
>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>> +
>> +.. _VIDIOC_EXT_QBUF:
>> +
>> +***************************************
>> +ioctl VIDIOC_EXT_QBUF, VIDIOC_EXT_DQBUF
>> +***************************************
>> +
>> +Name
>> +====
>> +
>> +VIDIOC_EXT_QBUF - VIDIOC_EXT_DQBUF - Exchange a buffer with the driver
>> +
>> +
>> +Synopsis
>> +========
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
>> +    :name: VIDIOC_EXT_QBUF
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp )
>> +    :name: VIDIOC_EXT_DQBUF
>> +
>> +
>> +Arguments
>> +=========
>> +
>> +``fd``
>> +    File descriptor returned by :ref:`open() <func-open>`.
>> +
>> +``argp``
>> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
>> +
>> +
>> +Description
>> +===========
>> +
>> +Applications call the ``VIDIOC_EXT_QBUF`` ioctl to enqueue an empty
>> +(capturing) or filled (output) buffer in the driver's incoming queue.
>> +The semantics depend on the selected I/O method.
>> +
>> +To enqueue a buffer applications set the ``type`` field of a struct
>> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
>> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``.
>> +Applications must also set the ``index`` field. Valid index numbers
>> +range from zero to the number of buffers allocated with
>> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
>> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
>> +one. The contents of the struct :c:type:`v4l2_ext_buffer` returned
> 
> I think this might not be entirely accurate. One can call
> VIDIOC_EXT_CREATE_BUFS multiple times. Moreover, I don't think we've
> prohibited VIDIOC_REQBUFS for the ext API, or have we?
> 
>> +by a :ref:`VIDIOC_EXT_QUERYBUF` ioctl will do as well.
>> +When the buffer is intended for output (``type`` is
>> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``) applications must also initialize the
>> +``timestamp`` fields and the ``planes[i].plane_length`` for each valid plane,
>> +and invalid ones must be set as zero.
>> +see :ref:`buffer` for details. Applications must also set ``flags`` to 0. The
>> +``reserved`` field must be set to 0.
>> +
>> +To enqueue a :ref:`memory mapped <mmap>` buffer applications set the
>> +``planes[i].memory`` field to ``V4L2_MEMORY_MMAP`` in all the valid planes,
>> +and invalid ones must be set as zero.
>> +When ``VIDIOC_EXT_QBUF`` is called
>> +with a pointer to this structure the driver sets the
>> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_QUEUED`` flags and clears
>> +the ``V4L2_BUF_FLAG_DONE`` flag in the ``flags`` field, or it returns an
>> +``EINVAL`` error code.
>> +
>> +To enqueue a :ref:`user pointer <userp>` buffer applications set the
>> +``planes[i].memory`` field to ``V4L2_MEMORY_USERPTR`` in all the valid planes,
>> +and invalid ones must be set as zero, the ``planes[i].m.userptr`` field to the
>> +address of the buffer,``planes[i].buffer_length`` to the size of the memory
>> +buffer, ``planes[i].plane_length`` to the size that should be used by the plane,
>> +and ``planes[i].offset`` of the plane in the memory buffer.
> 
> Does offset make sense here if an arbitrary pointer can be given for the
> start of the plane?
> 
>> +
>> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure
>> +the driver sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
>> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
>> +``flags`` field, or it returns an error code. This ioctl locks the
> 
> This seems to be a copy of the text few lines above.
> 
>> +memory pages of the buffer in physical memory, they cannot be swapped
>> +out to disk. Buffers remain locked until dequeued, until the
>> +:ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
>> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
>> +device is closed.
> 
> Why would CREATE_BUFS affect existing buffers?
> 
>> +
>> +To enqueue a :ref:`DMABUF <dmabuf>` buffer applications set the
>> +``planes[i].memory`` field to ``V4L2_MEMORY_DMABUF`` in all the valid planes,
>> +and invalid ones must be set as zero, the ``planes[i].m.fd`` field to a
>> +file descriptor associated with a DMABUF buffer.,``planes[i].buffer_length`` to
>> +the size of the memory buffer, ``planes[i].plane_length`` to the size that
> 
> What is the buffer_length field needed for? DMA-bufs by definition are
> kernel objects and the kernel should already know their sizes.

See this discussion:

https://patchwork.linuxtv.org/project/linux-media/patch/gh5kef5bkeel3o6b2dkgc2dfagu9klj4c0@4ax.com/

> 
>> +should be used by the plane, and ``planes[i].offset`` of the plane in the memory buffer.
>> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure the driver
>> +sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
>> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
>> +``flags`` field, or it returns an error code. This ioctl locks the
> 
> Ditto.
> 
>> +buffer. Locking a buffer means passing it to a driver for a hardware
>> +access (usually DMA). If an application accesses (reads/writes) a locked
>> +buffer then the result is undefined. Buffers remain locked until
>> +dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
>> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
>> +device is closed.
> 
> Ditto.
> 
>> +
>> +The ``request_fd`` field can be used with the ``VIDIOC_EXT_QBUF`` ioctl to specify
>> +the file descriptor of a :ref:`request <media-request-api>`, if requests are
>> +in use. Setting it means that the buffer will not be passed to the driver
>> +until the request itself is queued. Also, the driver will apply any
>> +settings associated with the request for this buffer. This field will
>> +be ignored unless the ``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
>> +If the device does not support requests, then ``EBADR`` will be returned.
>> +If requests are supported but an invalid request file descriptor is given,
>> +then ``EINVAL`` will be returned.
>> +
>> +.. caution::
>> +   It is not allowed to mix queuing requests with queuing buffers directly.
>> +   ``EBUSY`` will be returned if the first buffer was queued directly and
>> +   then the application tries to queue a request, or vice versa. After
>> +   closing the file descriptor, calling
>> +   :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or calling :ref:`VIDIOC_EXT_CREATE_BUFS`
>> +   the check for this will be reset.
>> +
>> +   For :ref:`memory-to-memory devices <mem2mem>` you can specify the
>> +   ``request_fd`` only for output buffers, not for capture buffers. Attempting
>> +   to specify this for a capture buffer will result in an ``EBADR`` error.
>> +
>> +Applications call the ``VIDIOC_EXT_DQBUF`` ioctl to dequeue a filled
>> +(capturing) or displayed (output) buffer from the driver's outgoing
> 
> "displayed" is quite inaccurate given that the majority of use cases for
> OUTPUT in V4L2 now are m2m devices. Maybe "consumed"?
> 
>> +queue. They just set the ``type``, ``planes[i].memory`` and ``reserved`` fields of
> 
> Why does the application have to set the memory field? It should be already
> known by the kernel which owns the buffer at the given time.
> 
>> +a struct :c:type:`v4l2_ext_buffer` as above, when
>> +``VIDIOC_EXT_DQBUF`` is called with a pointer to this structure the driver
>> +fills the remaining fields or returns an error code. The driver may also
>> +set ``V4L2_BUF_FLAG_ERROR`` in the ``flags`` field. It indicates a
>> +non-critical (recoverable) streaming error. In such case the application
>> +may continue as normal, but should be aware that data in the dequeued
>> +buffer might be corrupted. When using the multi-planar API, the planes
>> +array must be passed in as well.
>> +
>> +If the application sets the ``memory`` field to ``V4L2_MEMORY_DMABUF`` to
>> +dequeue a :ref:`DMABUF <dmabuf>` buffer, the driver fills the ``m.fd`` field
>> +with a file descriptor numerically the same as the one given to ``VIDIOC_EXT_QBUF``
>> +when the buffer was enqueued. No new file descriptor is created at dequeue time
>> +and the value is only for the application convenience.
> 
> Honestly, I don't like this aspect of the current API. It kind of implies
> that the value returned is valid, but the file descriptor might have been
> already closed. Could we just remove this and explicitly state that for
> DMA-buf, it is the user space responsibility to track the mapping between
> V4L2 indexes and its own buffer objects?
> 
>> +
>> +By default ``VIDIOC_EXT_DQBUF`` blocks when no buffer is in the outgoing
>> +queue. When the ``O_NONBLOCK`` flag was given to the
>> +:ref:`open() <func-open>` function, ``VIDIOC_EXT_DQBUF`` returns
>> +immediately with an ``EAGAIN`` error code when no buffer is available.
>> +
>> +
>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>> +   :functions: v4l2_ext_buffers
>> +
>> +
>> +Return Value
>> +============
>> +
>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>> +appropriately. The generic error codes are described at the
>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>> +
>> +EAGAIN
>> +    Non-blocking I/O has been selected using ``O_NONBLOCK`` and no
>> +    buffer was in the outgoing queue.
>> +
>> +EINVAL
>> +    The buffer ``type`` is not supported, or the ``index`` is out of
>> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
>> +    ``length`` are invalid, or the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was
> 
> There is no 'length' now.
> 
>> +    set but the the given ``request_fd`` was invalid, or ``m.fd`` was
>> +    an invalid DMABUF file descriptor.
>> +
>> +EIO
>> +    ``VIDIOC_EXT_DQBUF`` failed due to an internal error. Can also indicate
>> +    temporary problems like signal loss.

Drop the last sentence, -EIO should never be used for temporary problems like
that. EIO really means that the DMA engine crashed and userspace will have to call
STREAMOFF and try to restart the DMA with STREAMON.

>> +
>> +    .. note::
>> +
>> +       The driver might dequeue an (empty) buffer despite returning
>> +       an error, or even stop capturing. Reusing such buffer may be unsafe
>> +       though and its details (e.g. ``index``) may not be returned either.

The first part of this note is rather vague. I'm not sure what it meant with this.

>> +       It is recommended that drivers indicate recoverable errors by setting
>> +       the ``V4L2_BUF_FLAG_ERROR`` and returning 0 instead. In that case the
>> +       application should be able to safely reuse the buffer and continue
>> +       streaming.
> 
> What is this for? It sounds like some legacy behavior, which maybe we could
> drop with this new API?

No, V4L2_BUF_FLAG_ERROR is really needed. Some hardware can really have
intermittent failures causing video frames to be corrupt. And yes, that's
typically due to hardware bugs. It usually either happens when you start
streaming or just intermittent, but it is not severe enough to stop streaming
completely (that's the -EIO error due to a call to vb2_queue_error).

It's also used when you stop streaming and pending buffers are returned to
userspace. They are marked as ERROR since they don't contain any video, or
only partial video (if the DMA was stopped midway a frame).

> 
>> +
>> +EPIPE
>> +    ``VIDIOC_EXT_DQBUF`` returns this on an empty capture queue for mem2mem
>> +    codecs if a buffer with the ``V4L2_BUF_FLAG_LAST`` was already
>> +    dequeued and no new buffers are expected to become available.
>> +
>> +EBADR
>> +    The ``V4L2_BUF_FLAG_REQUEST_FD`` flag was set but the device does not
>> +    support requests for the given buffer type, or
>> +    the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was not set but the device requires
>> +    that the buffer is part of a request.
>> +
>> +EBUSY
>> +    The first buffer was queued via a request, but the application now tries
>> +    to queue it directly, or vice versa (it is not permitted to mix the two
>> +    APIs).
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>> new file mode 100644
>> index 0000000000000..f2a12017253f6
>> --- /dev/null
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>> @@ -0,0 +1,79 @@
>> +.. Permission is granted to copy, distribute and/or modify this
>> +.. document under the terms of the GNU Free Documentation License,
>> +.. Version 1.1 or any later version published by the Free Software
>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>> +.. and no Back-Cover Texts. A copy of the license is included at
>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>> +..
>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>> +
>> +.. _VIDIOC_EXT_QUERYBUF:
>> +
>> +*************************
>> +ioctl VIDIOC_EXT_QUERYBUF
>> +*************************
>> +
>> +Name
>> +====
>> +
>> +VIDIOC_EXT_QUERYBUF - Query the status of a buffer
>> +
>> +
>> +Synopsis
>> +========
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp )
>> +    :name: VIDIOC_EXT_QUERYBUF
>> +
>> +
>> +Arguments
>> +=========
>> +
>> +``fd``
>> +    File descriptor returned by :ref:`open() <func-open>`.
>> +
>> +``argp``
>> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
>> +
>> +
>> +Description
>> +===========
>> +
>> +This ioctl is part of the :ref:`streaming <mmap>` I/O method. It can
>> +be used to query the status of a buffer at any time after buffers have
>> +been allocated with the :ref:`VIDIOC_EXT_CREATE_BUFS` ioctl.
>> +
>> +Applications set the ``type`` field of a struct
>> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
>> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``,
>> +and the ``index`` field. Valid index numbers range from zero to the
>> +number of buffers allocated with
>> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
>> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
>> +one. The ``reserved`` field must be set to 0.
>> +
>> +In the ``flags`` field the ``V4L2_BUF_FLAG_MAPPED``,
>> +``V4L2_BUF_FLAG_PREPARED``, ``V4L2_BUF_FLAG_QUEUED`` and
>> +``V4L2_BUF_FLAG_DONE`` flags will be valid. The ``planes.memory`` fields will be
>> +set to the current I/O method for each plane.
>> +
>> +For every valid plane, an entry in ``planes`` will be filled, and zeroed for
>> +invalid ones. ``planes[i].buffer_length`` is the size of the memory buffer
>> +which contains the plane, ``planes[i].plane_length`` is the length of the plane,
> 
> What is the length of the plane here? It seems like depending on the
> context, the field can mean size of the payload (for filled buffer) or
> total space for the payload (for empty buffer), which is a bit confusing.
> 
>> +and ``planes[i].offset` is where the plane is placed in the memory buffer.
>> +
>> +The size of the ``planes`` array can be calculated by the number of sequential
>> +planes with ``planes[i].buffer_length`` that differs from zero up to the max
>> +size of the array.
> 
> Not sure if it's just me, but "size of array" sounds to me like the
> total memory size of the array, i.e. the max number of elements, which
> is fixed for this struct. How about
> 
>   "The entries of the planes array are valid up to the last entry with
>   non-zero buffer_length or the last entry of the array, whichever is
>   first."
> 
> ?
> 
>> +
>> +
>> +Return Value
>> +============
>> +
>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>> +appropriately. The generic error codes are described at the
>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>> +
>> +EINVAL
>> +    The buffer ``type`` is not supported, or the ``index`` is out of
>> +    bounds.
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>> new file mode 100644
>> index 0000000000000..008e6c98a88a5
>> --- /dev/null
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>> @@ -0,0 +1,117 @@
>> +.. Permission is granted to copy, distribute and/or modify this
>> +.. document under the terms of the GNU Free Documentation License,
>> +.. Version 1.1 or any later version published by the Free Software
>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>> +.. and no Back-Cover Texts. A copy of the license is included at
>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>> +..
>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>> +
>> +.. _VIDIOC_G_EXT_PIX_FMT:
>> +
>> +************************************************************************
>> +ioctl VIDIOC_G_EXT_PIX_FMT, VIDIOC_S_EXT_PIX_FMT, VIDIOC_TRY_EXT_PIX_FMT
>> +************************************************************************
>> +
>> +Name
>> +====
>> +
>> +VIDIOC_G_EXT_PIX_FMT - VIDIOC_S_EXT_PIX_FMT - VIDIOC_TRY_EXT_PIX_FMT - Get or set the data format, try a format
>> +
>> +
>> +Synopsis
>> +========
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
>> +    :name: VIDIOC_G_EXT_PIX_FMT
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
>> +    :name: VIDIOC_S_EXT_PIX_FMT
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
>> +    :name: VIDIOC_TRY_EXT_PIX_FMT
>> +
>> +Arguments
>> +=========
>> +
>> +``fd``
>> +    File descriptor returned by :ref:`open() <func-open>`.
>> +
>> +``argp``
>> +    Pointer to struct :c:type:`v4l2_ext_pix_format`.
>> +
>> +
>> +Description
>> +===========
>> +
>> +These ioctls are used to negotiate the format of data (typically image
>> +format) exchanged between driver and application.
>> +
>> +To query the current parameters applications set the ``type`` field of a
>> +struct :c:type:`v4l2_ext_pix_format` to  ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or
>> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``, all the other types are invalid in this API,
>> +and multiplanar is supported through modifiers.
>> +
>> +When the application calls the
>> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with a pointer to this
>> +structure the driver fills the other members.
>> +When the requested buffer type is not supported drivers return
>> +an ``EINVAL`` error code.
>> +
>> +To change the current format parameters applications initialize all
>> +the fields in the struct.
>> +For details see the documentation of the various devices types in
>> +:ref:`devices`. Good practice is to query the current parameters
>> +first, and to modify only those parameters not suitable for the
>> +application. When the application calls the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with
>> +a pointer to a struct :c:type:`v4l2_ext_pix_format` structure the driver
>> +checks and adjusts the parameters against hardware abilities. Drivers
>> +should not return an error code unless the ``type`` field is invalid,
>> +this is a mechanism to fathom device capabilities and to approach
>> +parameters acceptable for both the application and driver. On success
>> +the driver may program the hardware, allocate resources and generally
>> +prepare for data exchange. Finally the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl returns
>> +the current format parameters as :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` does. Very simple,
>> +inflexible devices may even ignore all input and always return the
>> +default parameters. However all V4L2 devices exchanging data with the
>> +application must implement the :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` and :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
>> +ioctl. When the requested buffer type is not supported drivers return an
>> +EINVAL error code on a :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` attempt. When I/O is already in
>> +progress or the resource is not available for other reasons drivers
>> +return the ``EBUSY`` error code.
>> +
>> +The :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl is equivalent to :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` with one
>> +exception: it does not change driver state. It can also be called at any
>> +time, never returning ``EBUSY``. This function is provided to negotiate
>> +parameters, to learn about hardware limitations, without disabling I/O
>> +or possibly time consuming hardware preparations. Although strongly
>> +recommended drivers are not required to implement this ioctl.
>> +
> 
> Could we make this mandatory to avoid ambiguities in the API? In

Yes, please!

> general, most drivers implemented it anyway, without much added
> complexity, because one can then simplify S_FMT to TRY_FMT + state
> update.

Ideally we would also fix the ambiguous situation regarding TRY/S_FMT
and unsupported pixelformats: some drivers will replace unsupported
pixelformats with a valid pixelformat, others will return an error.

v4l2-compliance warns about drivers that return an error and point to
this very old thread:

http://www.mail-archive.com/linux-media@vger.kernel.org/msg56550.html

However, I'm not really sure if we can resolve this ambiguity. I believe
that it is only older webcam drivers that return an error.

Regards,

	Hans

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 7/7] media: docs: add documentation for the Extended API
  2020-11-20 12:24     ` Hans Verkuil
@ 2020-11-20 12:40       ` Tomasz Figa
  2020-11-20 13:20         ` Hans Verkuil
  0 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-11-20 12:40 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Helen Koike, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Hirokazu Honda, Nicolas Dufresne, Brian Starkey, kernel,
	Boris Brezillon, Neil Armstrong, Linux Kernel Mailing List,
	Fritz Koenig, Maxime Jourdan, Stanimir Varbanov

On Fri, Nov 20, 2020 at 9:24 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> On 20/11/2020 12:06, Tomasz Figa wrote:z
> > Hi Helen,
> >
> > On Tue, Aug 04, 2020 at 04:29:39PM -0300, Helen Koike wrote:
> >> Add documentation and update references in current documentation for the
> >> Extended API.
> >>
> >
> > Thank you for the patch. Please see my comments inline.
> >
> >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >> ---
> >> Changes in v5:
> >> - new patch
> >>
> >>  .../userspace-api/media/v4l/buffer.rst        |   5 +
> >>  .../userspace-api/media/v4l/common.rst        |   1 +
> >>  .../userspace-api/media/v4l/dev-capture.rst   |   5 +
> >>  .../userspace-api/media/v4l/dev-output.rst    |   5 +
> >>  .../userspace-api/media/v4l/ext-api.rst       | 107 +++++++++
> >>  .../userspace-api/media/v4l/format.rst        |  16 +-
> >>  .../userspace-api/media/v4l/user-func.rst     |   5 +
> >>  .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++++++++
> >>  .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++++++
> >>  .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++++++++++++++++
> >>  .../media/v4l/vidioc-ext-querybuf.rst         |  79 +++++++
> >>  .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 ++++++++++
> >>  12 files changed, 697 insertions(+), 4 deletions(-)
> >>  create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
> >>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
> >>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
> >>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
> >>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
> >>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> >>
> >> diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
> >> index 57e752aaf414a..c832bedd64e4c 100644
> >> --- a/Documentation/userspace-api/media/v4l/buffer.rst
> >> +++ b/Documentation/userspace-api/media/v4l/buffer.rst
> >> @@ -27,6 +27,11 @@ such as pointers and sizes for each plane, are stored in
> >>  struct :c:type:`v4l2_plane` instead. In that case,
> >>  struct :c:type:`v4l2_buffer` contains an array of plane structures.
> >>
> >> +.. note::
> >> +
> >> +    The :ref:`ext_api` version can also be used, and it is
> >> +    preferable when applicable.
> >
> > Would rephrasing this as below making a bit more definitive?
> >
> >       For modern applications, these operations are replaced by their
> >       :ref:`ext_api` counterparts, which should be used instead.
>
> You can't say that, since especially in the beginning userspace will be running
> on older kernels that do not support this.
>
> This will work: "should be used instead, if supported by the driver."
>

With the wrappers that the patches provide, all drivers would support
the new API, so this boils down to the kernel version only, not
specific drivers.

Agreed, though, that application developers must be made aware that
the new API is only available in new kernels and old API must be used
for compatibility with old kernels.

> >
> >> +
> >>  Dequeued video buffers come with timestamps. The driver decides at which
> >>  part of the frame and with which clock the timestamp is taken. Please
> >>  see flags in the masks ``V4L2_BUF_FLAG_TIMESTAMP_MASK`` and
> >> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst
> >> index 7d81c58a13cd7..3430e0bdad667 100644
> >> --- a/Documentation/userspace-api/media/v4l/common.rst
> >> +++ b/Documentation/userspace-api/media/v4l/common.rst
> >> @@ -59,6 +59,7 @@ applicable to all devices.
> >>      ext-ctrls-detect
> >>      fourcc
> >>      format
> >> +    ext-api
> >>      planar-apis
> >>      selection-api
> >>      crop
> >> diff --git a/Documentation/userspace-api/media/v4l/dev-capture.rst b/Documentation/userspace-api/media/v4l/dev-capture.rst
> >> index 44d3094093ab6..5077639787d92 100644
> >> --- a/Documentation/userspace-api/media/v4l/dev-capture.rst
> >> +++ b/Documentation/userspace-api/media/v4l/dev-capture.rst
> >> @@ -102,6 +102,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
> >>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
> >>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
> >>
> >> +.. note::
> >> +
> >> +    The :ref:`ext_api` version can also be used, and it is
> >> +    preferable when applicable.
> >> +
> >>
> >>  Reading Images
> >>  ==============
> >> diff --git a/Documentation/userspace-api/media/v4l/dev-output.rst b/Documentation/userspace-api/media/v4l/dev-output.rst
> >> index e4f2a1d8b0fc7..f8f40c708e49f 100644
> >> --- a/Documentation/userspace-api/media/v4l/dev-output.rst
> >> +++ b/Documentation/userspace-api/media/v4l/dev-output.rst
> >> @@ -99,6 +99,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
> >>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
> >>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
> >>
> >> +.. note::
> >> +
> >> +    The :ref:`ext_api` version can also be used, and it is
> >> +    preferable when applicable.
> >> +
> >>
> >>  Writing Images
> >>  ==============
> >> diff --git a/Documentation/userspace-api/media/v4l/ext-api.rst b/Documentation/userspace-api/media/v4l/ext-api.rst
> >> new file mode 100644
> >> index 0000000000000..da2a82960d22f
> >> --- /dev/null
> >> +++ b/Documentation/userspace-api/media/v4l/ext-api.rst
> >> @@ -0,0 +1,107 @@
> >> +.. Permission is granted to copy, distribute and/or modify this
> >> +.. document under the terms of the GNU Free Documentation License,
> >> +.. Version 1.1 or any later version published by the Free Software
> >> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> >> +.. and no Back-Cover Texts. A copy of the license is included at
> >> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> >> +..
> >> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> >> +
> >> +.. _ext_api:
> >> +
> >> +*************
> >> +Extendend API
> >> +*************
> >> +
> >> +Introduction
> >> +============
> >> +
> >> +The Extended Format API was conceived to extend V4L2 capabilities and
> >> +to simplify certain mechanisms.
> >> +It unifies single- vs multi- planar handling,
> >> +brings the concept of pixelformat + modifiers, allows planes to be placed
> >> +in any offset inside a buffer and let userspace to change colorspace
> >> +attributes if supported by the driver.
> >
> > How about simplifying this paragraph a bit:
> >
> > The Extended Format API unifies single- and multi- planar format handling
> > and provides a base for further functional extensions, such as format
> > modifiers, arbitrary plane offsets, colorspace configuration.
>
> Actually, I prefer Helen's version.
>

To me it's a lot of text to read to get the same information, but I
won't insist.

> >
> >> +
> >> +Data format negotiation and buffer handling works very similar to the classical
> >> +version, thus in this document we'll just mention the main differences.
> >
> > nit: "we will" or maybe even just "we mention only the main differences".
> >
> >> +
> >> +Data Format Negotiation
> >> +=======================
> >> +
> >> +The API replaces the classical ioctls:
> >> +
> >> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>`, :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`,
> >> +:ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>`
> >> +(with :c:type:`v4l2_format` as the main parameter).
> >> +
> >> +By the extended versions:
> >> +
> >> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
> >> +:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
> >> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
> >> +(with :c:type:`v4l2_ext_pix_format` as the main parameter).
> >> +
> >> +For CAPTURE and OUTPUT queues only.
> >> +
> >> +The ``type`` field of struct :c:type:`v4l2_ext_pix_format` only accepts
> >> +``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or ``V4L2_BUF_TYPE_VIDEO_OUTPUT``, and it
> >> +supports multiplanar format through a combination of ``pixelfomat`` and
> >> +``modifier`` fields.
> >> +
> >> +Only the single-planar variants of the pixel formats are valid in the
> >> +``pixelformat`` field.
> >
> > Some formats have only multi-planar variants, like NV12MT. What should we
> > do about them? Since we have to keep those defines to support the legacy
> > API anyway, maybe we can just accept both variants and treat them equally?
>
> Helen mentioned that as well in a reply to patch 1/7. I'll have a think about
> that myself and reply to that post because this is a good question.
>

I'm thinking about it as well. :)

> >
> >> +To support multi-planar, a modifier should be set in the ``modifier`` field to
> >> +provide more information regarding the memory layout of pixels and number of
> >> +planes.
> >> +
> >> +The ``plane_fmt`` field is an array which holds information by plane using
> >> +the :c:type:`v4l2_plane_ext_pix_format`. When this struct is filled, its
> >> +``sizeimage`` field should be non-zero for all valid planes for a given
> >> +``pixelformat`` + ``modifier`` combination, and zeroed for the invalid ones.
> >> +
> >> +Colospace attributes are not read-only as in the classical version, i.e, they
> >> +can be set by application and drivers will adjust accordingly depending on what
> >> +is supported by the underlying hardware.
> >> +
> >> +Buffers
> >> +=======
> >> +
> >> +The API replaces the classical ioctls:
> >> +
> >> +:ref:`VIDIOC_CREATE_BUFS <VIDIOC_CREATE_BUFS>`
> >> +(with :c:type:`v4l2_create_buffers` as the main parameter),
> >> +:ref:`VIDIOC_QUERYBUF <VIDIOC_QUERYBUF>`, :ref:`VIDIOC_QBUF <VIDIOC_QBUF>`,
> >> +:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`, :ref:`VIDIOC_PREPARE_BUF <VIDIOC_PREPARE_BUF>`
> >> +(with :c:type:`v4l2_buffer` as the main parameter)
> >> +
> >> +By the extended versions:
> >> +
> >> +:ref:`VIDIOC_EXT_CREATE_BUFS <VIDIOC_EXT_CREATE_BUFS>`
> >> +(with :c:type:`v4l2_ext_create_buffers` as the main parameter),
> >> +:ref:`VIDIOC_EXT_QUERYBUF <VIDIOC_EXT_QUERYBUF>`,
> >> +:ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>`,
> >> +:ref:`VIDIOC_EXT_DQBUF <VIDIOC_EXT_QBUF>`,
> >> +:ref:`VIDIOC_EXT_PREPARE_BUF <VIDIOC_EXT_PREPARE_BUF>`
> >> +(with :c:type:`v4l2_ext_buffer` as the main parameter)
> >> +
> >> +The basic difference between :c:type:`v4l2_create_buffers` and
> >> +:c:type:`v4l2_ext_create_buffers` is that the later have a
> >> +:c:type:`v4l2_ext_pix_format` as the type of the ``format`` field.
> >> +
> >> +Comparing :c:type:`v4l2_ext_buffer` with :c:type:`v4l2_buffer`, in the
> >> +extended version there is a unification of the single-/multi- planar handling
> >> +through the ``planes`` field of type :c:type:`v4l2_ext_plane`.
> >> +
> >> +The :c:type:`v4l2_ext_plane also allows planes to be placed in a given offset
> >> +inside a buffer.
> >> +Planes can be placed in different locations inside the same buffer, or in
> >> +different buffers.
> >> +
> >> +
> >> +.. kernel-doc:: include/uapi/linux/videodev2.h
> >> +   :functions: v4l2_ext_plane
> >> +
> >> +
> >> +.. kernel-doc:: include/uapi/linux/videodev2.h
> >> +   :functions: v4l2_ext_buffer
> >> diff --git a/Documentation/userspace-api/media/v4l/format.rst b/Documentation/userspace-api/media/v4l/format.rst
> >> index e47fc0505727c..b96d26f26793c 100644
> >> --- a/Documentation/userspace-api/media/v4l/format.rst
> >> +++ b/Documentation/userspace-api/media/v4l/format.rst
> >> @@ -28,13 +28,19 @@ format and the driver selects and reports the best the hardware can do
> >>  to satisfy the request. Of course applications can also just query the
> >>  current selection.
> >>
> >> -A single mechanism exists to negotiate all data formats using the
> >> -aggregate struct :c:type:`v4l2_format` and the
> >> +There are two mechanism to negociate data formats:
> >
> > negotiate
> >
> >> +
> >> +The first one is using the aggregate struct :c:type:`v4l2_format` and the
> >>  :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and
> >>  :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctls. Additionally the
> >>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` ioctl can be used to examine
> >>  what the hardware *could* do, without actually selecting a new data
> >> -format. The data formats supported by the V4L2 API are covered in the
> >> +format.
> >> +
> >> +The second is through the :ref:`ext_api`, please refer to its documentation
> >> +for more information.
> >> +
> >
> > Should we reverse the order to prefer the new API?
> >
> >> +The data formats supported by the V4L2 API are covered in the
> >>  respective device section in :ref:`devices`. For a closer look at
> >>  image formats see :ref:`pixfmt`.
> >>
> >> @@ -71,7 +77,9 @@ earlier versions of V4L2. Switching the logical stream or returning into
> >>  *may* support a switch using :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`.
> >>
> >>  All drivers exchanging data with applications must support the
> >> -:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl. Implementation of the
> >> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl
> >> +or the Extended respective versions (TODO: link).
> >
> > TODO? :)
> >
> >> +Implementation of the
> >>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is highly recommended but optional.
> >>
> >>
> >> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
> >> index bf77c842718e5..67b254812791c 100644
> >> --- a/Documentation/userspace-api/media/v4l/user-func.rst
> >> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
> >> @@ -20,6 +20,7 @@ Function Reference
> >>      func-close
> >>      func-ioctl
> >>      vidioc-create-bufs
> >> +    vidioc-ext-create-bufs
> >>      vidioc-cropcap
> >>      vidioc-dbg-g-chip-info
> >>      vidioc-dbg-g-register
> >> @@ -48,6 +49,7 @@ Function Reference
> >>      vidioc-g-ext-ctrls
> >>      vidioc-g-fbuf
> >>      vidioc-g-fmt
> >> +    vidioc-g-ext-pix-fmt
> >>      vidioc-g-frequency
> >>      vidioc-g-input
> >>      vidioc-g-jpegcomp
> >> @@ -62,8 +64,11 @@ Function Reference
> >>      vidioc-log-status
> >>      vidioc-overlay
> >>      vidioc-prepare-buf
> >> +    vidioc-ext-prepare-buf
> >>      vidioc-qbuf
> >> +    vidioc-ext-qbuf
> >>      vidioc-querybuf
> >> +    vidioc-ext-querybuf
> >>      vidioc-querycap
> >>      vidioc-queryctrl
> >>      vidioc-query-dv-timings
> >> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
> >> new file mode 100644
> >> index 0000000000000..67f2c541e4d79
> >> --- /dev/null
> >> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
> >> @@ -0,0 +1,95 @@
> >> +.. Permission is granted to copy, distribute and/or modify this
> >> +.. document under the terms of the GNU Free Documentation License,
> >> +.. Version 1.1 or any later version published by the Free Software
> >> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> >> +.. and no Back-Cover Texts. A copy of the license is included at
> >> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> >> +..
> >> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> >> +
> >
> > TODO? :)
> >
> >> +.. _VIDIOC_EXT_CREATE_BUFS:
> >> +
> >> +****************************
> >> +ioctl VIDIOC_EXT_CREATE_BUFS
> >> +****************************
> >> +
> >> +Name
> >> +====
> >> +
> >> +VIDIOC_EXT_CREATE_BUFS - Create buffers for Memory Mapped or User Pointer or DMA Buffer I/O
> >> +
> >> +
> >> +Synopsis
> >> +========
> >> +
> >> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp )
> >> +    :name: VIDIOC_EXT_CREATE_BUFS
> >> +
> >> +
> >> +Arguments
> >> +=========
> >> +
> >> +``fd``
> >> +    File descriptor returned by :ref:`open() <func-open>`.
> >> +
> >> +``argp``
> >> +    Pointer to struct :c:type:`v4l2_ext_create_buffers`.
> >> +
> >> +
> >> +Description
> >> +===========
> >> +
> >> +This ioctl is used to create buffers for :ref:`memory mapped <mmap>`
> >> +or :ref:`user pointer <userp>` or :ref:`DMA buffer <dmabuf>` I/O.
> >> +This ioctl can be called multiple times to
> >> +create buffers of different sizes.
> >> +
> >> +To allocate the device buffers applications must initialize the relevant
> >> +fields of the struct :c:type:`v4l2_ext_create_buffers` structure. The
> >> +``count`` field must be set to the number of requested buffers, the
> >> +``memory`` field specifies the requested I/O method and the ``reserved``
> >> +array must be zeroed.
> >> +
> >> +The ``format`` field specifies the image format that the buffers must be
> >> +able to handle. The application has to fill in this struct
> >> +:c:type:`v4l2_format`. Usually this will be done using the
> >> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` or
> >> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctls to ensure that the
> >> +requested format is supported by the driver.
> >> +The driver may return an error if the size(s) are not supported by the
> >> +hardware (usually because they are too small).
> >> +
> >> +The driver can create a memory buffer per plane, or a single memory buffer
> >> +containing all the planes, with a minimum size according to the size
> >> +defined by the ``format.plane_fmt[i].sizeimage`` field.
> >> +Usually if the ``format.plane_fmt[i].sizeimage``
> >> +field is less than the minimum required for the given format, then an
> >> +error will be returned since drivers will typically not allow this. If
> >> +it is larger, then the value will be used as-is. In other words, the
> >> +driver may reject the requested size, but if it is accepted the driver
> >> +will use it unchanged.
> >> +
> >> +When the ioctl is called with a pointer to this structure the driver
> >> +will attempt to allocate up to the requested number of buffers and store
> >> +the actual number allocated and the starting index in the ``count`` and
> >> +the ``index`` fields respectively. On return ``count`` can be smaller
> >> +than the number requested.
> >
> > The 'capabilities' and 'flags' fields seem to go undocumented.
> >
> >> +
> >> +
> >> +.. kernel-doc:: include/uapi/linux/videodev2.h
> >> +   :functions: v4l2_ext_create_buffers
> >> +
> >> +
> >> +Return Value
> >> +============
> >> +
> >> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> >> +appropriately. The generic error codes are described at the
> >> +:ref:`Generic Error Codes <gen-errors>` chapter.
> >> +
> >> +ENOMEM
> >> +    No memory to allocate buffers for :ref:`memory mapped <mmap>` I/O.
> >> +
> >> +EINVAL
> >> +    The buffer type (``format.type`` field), requested I/O method
> >> +    (``memory``) or format (``format`` field) is not valid.
> >> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
> >> new file mode 100644
> >> index 0000000000000..88e4b57121254
> >> --- /dev/null
> >> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
> >> @@ -0,0 +1,62 @@
> >> +.. Permission is granted to copy, distribute and/or modify this
> >> +.. document under the terms of the GNU Free Documentation License,
> >> +.. Version 1.1 or any later version published by the Free Software
> >> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> >> +.. and no Back-Cover Texts. A copy of the license is included at
> >> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> >> +..
> >> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> >> +
> >> +.. _VIDIOC_EXT_PREPARE_BUF:
> >> +
> >> +****************************
> >> +ioctl VIDIOC_EXT_PREPARE_BUF
> >> +****************************
> >> +
> >> +Name
> >> +====
> >> +
> >> +VIDIOC_EXT_PREPARE_BUF - Prepare a buffer for I/O
> >> +
> >> +
> >> +Synopsis
> >> +========
> >> +
> >> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp )
> >> +    :name: VIDIOC_EXT_PREPARE_BUF
> >> +
> >> +
> >> +Arguments
> >> +=========
> >> +
> >> +``fd``
> >> +    File descriptor returned by :ref:`open() <func-open>`.
> >> +
> >> +``argp``
> >> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> >> +
> >> +
> >> +Description
> >> +===========
> >> +
> >> +Applications can optionally call the :ref:`VIDIOC_EXT_PREPARE_BUF` ioctl to
> >> +pass ownership of the buffer to the driver before actually enqueuing it,
> >> +using the :ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>` ioctl, and to prepare it for future I/O. Such
> >> +preparations may include cache invalidation or cleaning. Performing them
> >> +in advance saves time during the actual I/O.
> >> +
> >> +
> >> +Return Value
> >> +============
> >> +
> >> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> >> +appropriately. The generic error codes are described at the
> >> +:ref:`Generic Error Codes <gen-errors>` chapter.
> >> +
> >> +EBUSY
> >> +    File I/O is in progress.
> >> +
> >> +EINVAL
> >> +    The buffer ``type`` is not supported, or the ``index`` is out of
> >> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
> >> +    ``length`` are invalid.
> >> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
> >> new file mode 100644
> >> index 0000000000000..083e106cf6bad
> >> --- /dev/null
> >> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
> >> @@ -0,0 +1,204 @@
> >> +.. Permission is granted to copy, distribute and/or modify this
> >> +.. document under the terms of the GNU Free Documentation License,
> >> +.. Version 1.1 or any later version published by the Free Software
> >> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> >> +.. and no Back-Cover Texts. A copy of the license is included at
> >> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> >> +..
> >> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> >> +
> >> +.. _VIDIOC_EXT_QBUF:
> >> +
> >> +***************************************
> >> +ioctl VIDIOC_EXT_QBUF, VIDIOC_EXT_DQBUF
> >> +***************************************
> >> +
> >> +Name
> >> +====
> >> +
> >> +VIDIOC_EXT_QBUF - VIDIOC_EXT_DQBUF - Exchange a buffer with the driver
> >> +
> >> +
> >> +Synopsis
> >> +========
> >> +
> >> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
> >> +    :name: VIDIOC_EXT_QBUF
> >> +
> >> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp )
> >> +    :name: VIDIOC_EXT_DQBUF
> >> +
> >> +
> >> +Arguments
> >> +=========
> >> +
> >> +``fd``
> >> +    File descriptor returned by :ref:`open() <func-open>`.
> >> +
> >> +``argp``
> >> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> >> +
> >> +
> >> +Description
> >> +===========
> >> +
> >> +Applications call the ``VIDIOC_EXT_QBUF`` ioctl to enqueue an empty
> >> +(capturing) or filled (output) buffer in the driver's incoming queue.
> >> +The semantics depend on the selected I/O method.
> >> +
> >> +To enqueue a buffer applications set the ``type`` field of a struct
> >> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
> >> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``.
> >> +Applications must also set the ``index`` field. Valid index numbers
> >> +range from zero to the number of buffers allocated with
> >> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
> >> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
> >> +one. The contents of the struct :c:type:`v4l2_ext_buffer` returned
> >
> > I think this might not be entirely accurate. One can call
> > VIDIOC_EXT_CREATE_BUFS multiple times. Moreover, I don't think we've
> > prohibited VIDIOC_REQBUFS for the ext API, or have we?
> >
> >> +by a :ref:`VIDIOC_EXT_QUERYBUF` ioctl will do as well.
> >> +When the buffer is intended for output (``type`` is
> >> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``) applications must also initialize the
> >> +``timestamp`` fields and the ``planes[i].plane_length`` for each valid plane,
> >> +and invalid ones must be set as zero.
> >> +see :ref:`buffer` for details. Applications must also set ``flags`` to 0. The
> >> +``reserved`` field must be set to 0.
> >> +
> >> +To enqueue a :ref:`memory mapped <mmap>` buffer applications set the
> >> +``planes[i].memory`` field to ``V4L2_MEMORY_MMAP`` in all the valid planes,
> >> +and invalid ones must be set as zero.
> >> +When ``VIDIOC_EXT_QBUF`` is called
> >> +with a pointer to this structure the driver sets the
> >> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_QUEUED`` flags and clears
> >> +the ``V4L2_BUF_FLAG_DONE`` flag in the ``flags`` field, or it returns an
> >> +``EINVAL`` error code.
> >> +
> >> +To enqueue a :ref:`user pointer <userp>` buffer applications set the
> >> +``planes[i].memory`` field to ``V4L2_MEMORY_USERPTR`` in all the valid planes,
> >> +and invalid ones must be set as zero, the ``planes[i].m.userptr`` field to the
> >> +address of the buffer,``planes[i].buffer_length`` to the size of the memory
> >> +buffer, ``planes[i].plane_length`` to the size that should be used by the plane,
> >> +and ``planes[i].offset`` of the plane in the memory buffer.
> >
> > Does offset make sense here if an arbitrary pointer can be given for the
> > start of the plane?
> >
> >> +
> >> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure
> >> +the driver sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
> >> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
> >> +``flags`` field, or it returns an error code. This ioctl locks the
> >
> > This seems to be a copy of the text few lines above.
> >
> >> +memory pages of the buffer in physical memory, they cannot be swapped
> >> +out to disk. Buffers remain locked until dequeued, until the
> >> +:ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
> >> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
> >> +device is closed.
> >
> > Why would CREATE_BUFS affect existing buffers?
> >
> >> +
> >> +To enqueue a :ref:`DMABUF <dmabuf>` buffer applications set the
> >> +``planes[i].memory`` field to ``V4L2_MEMORY_DMABUF`` in all the valid planes,
> >> +and invalid ones must be set as zero, the ``planes[i].m.fd`` field to a
> >> +file descriptor associated with a DMABUF buffer.,``planes[i].buffer_length`` to
> >> +the size of the memory buffer, ``planes[i].plane_length`` to the size that
> >
> > What is the buffer_length field needed for? DMA-bufs by definition are
> > kernel objects and the kernel should already know their sizes.
>
> See this discussion:
>
> https://patchwork.linuxtv.org/project/linux-media/patch/gh5kef5bkeel3o6b2dkgc2dfagu9klj4c0@4ax.com/
>

I briefly read through it and I don't see what the conclusion was.
Generally I'd still prefer if the need to manually specify the buffer
size was eliminated, because that imposes unnecessary burden on the
userspace for the majority of cases. It could still remain valid for
userspace that wants to restrict the buffer size as it was mentioned
in that thread, though.

> >
> >> +should be used by the plane, and ``planes[i].offset`` of the plane in the memory buffer.
> >> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure the driver
> >> +sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
> >> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
> >> +``flags`` field, or it returns an error code. This ioctl locks the
> >
> > Ditto.
> >
> >> +buffer. Locking a buffer means passing it to a driver for a hardware
> >> +access (usually DMA). If an application accesses (reads/writes) a locked
> >> +buffer then the result is undefined. Buffers remain locked until
> >> +dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
> >> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
> >> +device is closed.
> >
> > Ditto.
> >
> >> +
> >> +The ``request_fd`` field can be used with the ``VIDIOC_EXT_QBUF`` ioctl to specify
> >> +the file descriptor of a :ref:`request <media-request-api>`, if requests are
> >> +in use. Setting it means that the buffer will not be passed to the driver
> >> +until the request itself is queued. Also, the driver will apply any
> >> +settings associated with the request for this buffer. This field will
> >> +be ignored unless the ``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
> >> +If the device does not support requests, then ``EBADR`` will be returned.
> >> +If requests are supported but an invalid request file descriptor is given,
> >> +then ``EINVAL`` will be returned.
> >> +
> >> +.. caution::
> >> +   It is not allowed to mix queuing requests with queuing buffers directly.
> >> +   ``EBUSY`` will be returned if the first buffer was queued directly and
> >> +   then the application tries to queue a request, or vice versa. After
> >> +   closing the file descriptor, calling
> >> +   :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or calling :ref:`VIDIOC_EXT_CREATE_BUFS`
> >> +   the check for this will be reset.
> >> +
> >> +   For :ref:`memory-to-memory devices <mem2mem>` you can specify the
> >> +   ``request_fd`` only for output buffers, not for capture buffers. Attempting
> >> +   to specify this for a capture buffer will result in an ``EBADR`` error.
> >> +
> >> +Applications call the ``VIDIOC_EXT_DQBUF`` ioctl to dequeue a filled
> >> +(capturing) or displayed (output) buffer from the driver's outgoing
> >
> > "displayed" is quite inaccurate given that the majority of use cases for
> > OUTPUT in V4L2 now are m2m devices. Maybe "consumed"?
> >
> >> +queue. They just set the ``type``, ``planes[i].memory`` and ``reserved`` fields of
> >
> > Why does the application have to set the memory field? It should be already
> > known by the kernel which owns the buffer at the given time.
> >
> >> +a struct :c:type:`v4l2_ext_buffer` as above, when
> >> +``VIDIOC_EXT_DQBUF`` is called with a pointer to this structure the driver
> >> +fills the remaining fields or returns an error code. The driver may also
> >> +set ``V4L2_BUF_FLAG_ERROR`` in the ``flags`` field. It indicates a
> >> +non-critical (recoverable) streaming error. In such case the application
> >> +may continue as normal, but should be aware that data in the dequeued
> >> +buffer might be corrupted. When using the multi-planar API, the planes
> >> +array must be passed in as well.
> >> +
> >> +If the application sets the ``memory`` field to ``V4L2_MEMORY_DMABUF`` to
> >> +dequeue a :ref:`DMABUF <dmabuf>` buffer, the driver fills the ``m.fd`` field
> >> +with a file descriptor numerically the same as the one given to ``VIDIOC_EXT_QBUF``
> >> +when the buffer was enqueued. No new file descriptor is created at dequeue time
> >> +and the value is only for the application convenience.
> >
> > Honestly, I don't like this aspect of the current API. It kind of implies
> > that the value returned is valid, but the file descriptor might have been
> > already closed. Could we just remove this and explicitly state that for
> > DMA-buf, it is the user space responsibility to track the mapping between
> > V4L2 indexes and its own buffer objects?
> >
> >> +
> >> +By default ``VIDIOC_EXT_DQBUF`` blocks when no buffer is in the outgoing
> >> +queue. When the ``O_NONBLOCK`` flag was given to the
> >> +:ref:`open() <func-open>` function, ``VIDIOC_EXT_DQBUF`` returns
> >> +immediately with an ``EAGAIN`` error code when no buffer is available.
> >> +
> >> +
> >> +.. kernel-doc:: include/uapi/linux/videodev2.h
> >> +   :functions: v4l2_ext_buffers
> >> +
> >> +
> >> +Return Value
> >> +============
> >> +
> >> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> >> +appropriately. The generic error codes are described at the
> >> +:ref:`Generic Error Codes <gen-errors>` chapter.
> >> +
> >> +EAGAIN
> >> +    Non-blocking I/O has been selected using ``O_NONBLOCK`` and no
> >> +    buffer was in the outgoing queue.
> >> +
> >> +EINVAL
> >> +    The buffer ``type`` is not supported, or the ``index`` is out of
> >> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
> >> +    ``length`` are invalid, or the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was
> >
> > There is no 'length' now.
> >
> >> +    set but the the given ``request_fd`` was invalid, or ``m.fd`` was
> >> +    an invalid DMABUF file descriptor.
> >> +
> >> +EIO
> >> +    ``VIDIOC_EXT_DQBUF`` failed due to an internal error. Can also indicate
> >> +    temporary problems like signal loss.
>
> Drop the last sentence, -EIO should never be used for temporary problems like
> that. EIO really means that the DMA engine crashed and userspace will have to call
> STREAMOFF and try to restart the DMA with STREAMON.
>

I suspect the users of the new API wouldn't actually have to deal with
a signal loss, but if not, what should be the behavior then?

> >> +
> >> +    .. note::
> >> +
> >> +       The driver might dequeue an (empty) buffer despite returning
> >> +       an error, or even stop capturing. Reusing such buffer may be unsafe
> >> +       though and its details (e.g. ``index``) may not be returned either.
>
> The first part of this note is rather vague. I'm not sure what it meant with this.
>
> >> +       It is recommended that drivers indicate recoverable errors by setting
> >> +       the ``V4L2_BUF_FLAG_ERROR`` and returning 0 instead. In that case the
> >> +       application should be able to safely reuse the buffer and continue
> >> +       streaming.
> >
> > What is this for? It sounds like some legacy behavior, which maybe we could
> > drop with this new API?
>
> No, V4L2_BUF_FLAG_ERROR is really needed. Some hardware can really have
> intermittent failures causing video frames to be corrupt. And yes, that's
> typically due to hardware bugs. It usually either happens when you start
> streaming or just intermittent, but it is not severe enough to stop streaming
> completely (that's the -EIO error due to a call to vb2_queue_error).
>
> It's also used when you stop streaming and pending buffers are returned to
> userspace. They are marked as ERROR since they don't contain any video, or
> only partial video (if the DMA was stopped midway a frame).
>

Sorry, I meant to insert that comment below the first half. The second
half is fine.

> >
> >> +
> >> +EPIPE
> >> +    ``VIDIOC_EXT_DQBUF`` returns this on an empty capture queue for mem2mem
> >> +    codecs if a buffer with the ``V4L2_BUF_FLAG_LAST`` was already
> >> +    dequeued and no new buffers are expected to become available.
> >> +
> >> +EBADR
> >> +    The ``V4L2_BUF_FLAG_REQUEST_FD`` flag was set but the device does not
> >> +    support requests for the given buffer type, or
> >> +    the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was not set but the device requires
> >> +    that the buffer is part of a request.
> >> +
> >> +EBUSY
> >> +    The first buffer was queued via a request, but the application now tries
> >> +    to queue it directly, or vice versa (it is not permitted to mix the two
> >> +    APIs).
> >> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
> >> new file mode 100644
> >> index 0000000000000..f2a12017253f6
> >> --- /dev/null
> >> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
> >> @@ -0,0 +1,79 @@
> >> +.. Permission is granted to copy, distribute and/or modify this
> >> +.. document under the terms of the GNU Free Documentation License,
> >> +.. Version 1.1 or any later version published by the Free Software
> >> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> >> +.. and no Back-Cover Texts. A copy of the license is included at
> >> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> >> +..
> >> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> >> +
> >> +.. _VIDIOC_EXT_QUERYBUF:
> >> +
> >> +*************************
> >> +ioctl VIDIOC_EXT_QUERYBUF
> >> +*************************
> >> +
> >> +Name
> >> +====
> >> +
> >> +VIDIOC_EXT_QUERYBUF - Query the status of a buffer
> >> +
> >> +
> >> +Synopsis
> >> +========
> >> +
> >> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp )
> >> +    :name: VIDIOC_EXT_QUERYBUF
> >> +
> >> +
> >> +Arguments
> >> +=========
> >> +
> >> +``fd``
> >> +    File descriptor returned by :ref:`open() <func-open>`.
> >> +
> >> +``argp``
> >> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
> >> +
> >> +
> >> +Description
> >> +===========
> >> +
> >> +This ioctl is part of the :ref:`streaming <mmap>` I/O method. It can
> >> +be used to query the status of a buffer at any time after buffers have
> >> +been allocated with the :ref:`VIDIOC_EXT_CREATE_BUFS` ioctl.
> >> +
> >> +Applications set the ``type`` field of a struct
> >> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
> >> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``,
> >> +and the ``index`` field. Valid index numbers range from zero to the
> >> +number of buffers allocated with
> >> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
> >> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
> >> +one. The ``reserved`` field must be set to 0.
> >> +
> >> +In the ``flags`` field the ``V4L2_BUF_FLAG_MAPPED``,
> >> +``V4L2_BUF_FLAG_PREPARED``, ``V4L2_BUF_FLAG_QUEUED`` and
> >> +``V4L2_BUF_FLAG_DONE`` flags will be valid. The ``planes.memory`` fields will be
> >> +set to the current I/O method for each plane.
> >> +
> >> +For every valid plane, an entry in ``planes`` will be filled, and zeroed for
> >> +invalid ones. ``planes[i].buffer_length`` is the size of the memory buffer
> >> +which contains the plane, ``planes[i].plane_length`` is the length of the plane,
> >
> > What is the length of the plane here? It seems like depending on the
> > context, the field can mean size of the payload (for filled buffer) or
> > total space for the payload (for empty buffer), which is a bit confusing.
> >
> >> +and ``planes[i].offset` is where the plane is placed in the memory buffer.
> >> +
> >> +The size of the ``planes`` array can be calculated by the number of sequential
> >> +planes with ``planes[i].buffer_length`` that differs from zero up to the max
> >> +size of the array.
> >
> > Not sure if it's just me, but "size of array" sounds to me like the
> > total memory size of the array, i.e. the max number of elements, which
> > is fixed for this struct. How about
> >
> >   "The entries of the planes array are valid up to the last entry with
> >   non-zero buffer_length or the last entry of the array, whichever is
> >   first."
> >
> > ?
> >
> >> +
> >> +
> >> +Return Value
> >> +============
> >> +
> >> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> >> +appropriately. The generic error codes are described at the
> >> +:ref:`Generic Error Codes <gen-errors>` chapter.
> >> +
> >> +EINVAL
> >> +    The buffer ``type`` is not supported, or the ``index`` is out of
> >> +    bounds.
> >> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> >> new file mode 100644
> >> index 0000000000000..008e6c98a88a5
> >> --- /dev/null
> >> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> >> @@ -0,0 +1,117 @@
> >> +.. Permission is granted to copy, distribute and/or modify this
> >> +.. document under the terms of the GNU Free Documentation License,
> >> +.. Version 1.1 or any later version published by the Free Software
> >> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> >> +.. and no Back-Cover Texts. A copy of the license is included at
> >> +.. Documentation/userspace-api/media/fdl-appendix.rst.
> >> +..
> >> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
> >> +
> >> +.. _VIDIOC_G_EXT_PIX_FMT:
> >> +
> >> +************************************************************************
> >> +ioctl VIDIOC_G_EXT_PIX_FMT, VIDIOC_S_EXT_PIX_FMT, VIDIOC_TRY_EXT_PIX_FMT
> >> +************************************************************************
> >> +
> >> +Name
> >> +====
> >> +
> >> +VIDIOC_G_EXT_PIX_FMT - VIDIOC_S_EXT_PIX_FMT - VIDIOC_TRY_EXT_PIX_FMT - Get or set the data format, try a format
> >> +
> >> +
> >> +Synopsis
> >> +========
> >> +
> >> +.. c:function:: int ioctl( int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> >> +    :name: VIDIOC_G_EXT_PIX_FMT
> >> +
> >> +.. c:function:: int ioctl( int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> >> +    :name: VIDIOC_S_EXT_PIX_FMT
> >> +
> >> +.. c:function:: int ioctl( int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
> >> +    :name: VIDIOC_TRY_EXT_PIX_FMT
> >> +
> >> +Arguments
> >> +=========
> >> +
> >> +``fd``
> >> +    File descriptor returned by :ref:`open() <func-open>`.
> >> +
> >> +``argp``
> >> +    Pointer to struct :c:type:`v4l2_ext_pix_format`.
> >> +
> >> +
> >> +Description
> >> +===========
> >> +
> >> +These ioctls are used to negotiate the format of data (typically image
> >> +format) exchanged between driver and application.
> >> +
> >> +To query the current parameters applications set the ``type`` field of a
> >> +struct :c:type:`v4l2_ext_pix_format` to  ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or
> >> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``, all the other types are invalid in this API,
> >> +and multiplanar is supported through modifiers.
> >> +
> >> +When the application calls the
> >> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with a pointer to this
> >> +structure the driver fills the other members.
> >> +When the requested buffer type is not supported drivers return
> >> +an ``EINVAL`` error code.
> >> +
> >> +To change the current format parameters applications initialize all
> >> +the fields in the struct.
> >> +For details see the documentation of the various devices types in
> >> +:ref:`devices`. Good practice is to query the current parameters
> >> +first, and to modify only those parameters not suitable for the
> >> +application. When the application calls the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with
> >> +a pointer to a struct :c:type:`v4l2_ext_pix_format` structure the driver
> >> +checks and adjusts the parameters against hardware abilities. Drivers
> >> +should not return an error code unless the ``type`` field is invalid,
> >> +this is a mechanism to fathom device capabilities and to approach
> >> +parameters acceptable for both the application and driver. On success
> >> +the driver may program the hardware, allocate resources and generally
> >> +prepare for data exchange. Finally the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl returns
> >> +the current format parameters as :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` does. Very simple,
> >> +inflexible devices may even ignore all input and always return the
> >> +default parameters. However all V4L2 devices exchanging data with the
> >> +application must implement the :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` and :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
> >> +ioctl. When the requested buffer type is not supported drivers return an
> >> +EINVAL error code on a :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` attempt. When I/O is already in
> >> +progress or the resource is not available for other reasons drivers
> >> +return the ``EBUSY`` error code.
> >> +
> >> +The :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl is equivalent to :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` with one
> >> +exception: it does not change driver state. It can also be called at any
> >> +time, never returning ``EBUSY``. This function is provided to negotiate
> >> +parameters, to learn about hardware limitations, without disabling I/O
> >> +or possibly time consuming hardware preparations. Although strongly
> >> +recommended drivers are not required to implement this ioctl.
> >> +
> >
> > Could we make this mandatory to avoid ambiguities in the API? In
>
> Yes, please!
>
> > general, most drivers implemented it anyway, without much added
> > complexity, because one can then simplify S_FMT to TRY_FMT + state
> > update.
>
> Ideally we would also fix the ambiguous situation regarding TRY/S_FMT
> and unsupported pixelformats: some drivers will replace unsupported
> pixelformats with a valid pixelformat, others will return an error.
>
> v4l2-compliance warns about drivers that return an error and point to
> this very old thread:
>
> http://www.mail-archive.com/linux-media@vger.kernel.org/msg56550.html
>
> However, I'm not really sure if we can resolve this ambiguity. I believe
> that it is only older webcam drivers that return an error.

I guess we can at least enforce a specific behavior in the new API,
before we get compatibility obligations. :)

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 7/7] media: docs: add documentation for the Extended API
  2020-11-20 12:40       ` Tomasz Figa
@ 2020-11-20 13:20         ` Hans Verkuil
  2021-01-14 18:04           ` Helen Koike
  0 siblings, 1 reply; 51+ messages in thread
From: Hans Verkuil @ 2020-11-20 13:20 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Helen Koike, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Hirokazu Honda, Nicolas Dufresne, Brian Starkey, kernel,
	Boris Brezillon, Neil Armstrong, Linux Kernel Mailing List,
	Fritz Koenig, Maxime Jourdan, Stanimir Varbanov

On 20/11/2020 13:40, Tomasz Figa wrote:
> On Fri, Nov 20, 2020 at 9:24 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>
>> On 20/11/2020 12:06, Tomasz Figa wrote:z
>>> Hi Helen,
>>>
>>> On Tue, Aug 04, 2020 at 04:29:39PM -0300, Helen Koike wrote:
>>>> Add documentation and update references in current documentation for the
>>>> Extended API.
>>>>
>>>
>>> Thank you for the patch. Please see my comments inline.
>>>
>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>> ---
>>>> Changes in v5:
>>>> - new patch
>>>>
>>>>  .../userspace-api/media/v4l/buffer.rst        |   5 +
>>>>  .../userspace-api/media/v4l/common.rst        |   1 +
>>>>  .../userspace-api/media/v4l/dev-capture.rst   |   5 +
>>>>  .../userspace-api/media/v4l/dev-output.rst    |   5 +
>>>>  .../userspace-api/media/v4l/ext-api.rst       | 107 +++++++++
>>>>  .../userspace-api/media/v4l/format.rst        |  16 +-
>>>>  .../userspace-api/media/v4l/user-func.rst     |   5 +
>>>>  .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++++++++
>>>>  .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++++++
>>>>  .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++++++++++++++++
>>>>  .../media/v4l/vidioc-ext-querybuf.rst         |  79 +++++++
>>>>  .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 ++++++++++
>>>>  12 files changed, 697 insertions(+), 4 deletions(-)
>>>>  create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
>>>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>>>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>>>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>>>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>>>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>>>>
>>>> diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
>>>> index 57e752aaf414a..c832bedd64e4c 100644
>>>> --- a/Documentation/userspace-api/media/v4l/buffer.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/buffer.rst
>>>> @@ -27,6 +27,11 @@ such as pointers and sizes for each plane, are stored in
>>>>  struct :c:type:`v4l2_plane` instead. In that case,
>>>>  struct :c:type:`v4l2_buffer` contains an array of plane structures.
>>>>
>>>> +.. note::
>>>> +
>>>> +    The :ref:`ext_api` version can also be used, and it is
>>>> +    preferable when applicable.
>>>
>>> Would rephrasing this as below making a bit more definitive?
>>>
>>>       For modern applications, these operations are replaced by their
>>>       :ref:`ext_api` counterparts, which should be used instead.
>>
>> You can't say that, since especially in the beginning userspace will be running
>> on older kernels that do not support this.
>>
>> This will work: "should be used instead, if supported by the driver."
>>
> 
> With the wrappers that the patches provide, all drivers would support
> the new API, so this boils down to the kernel version only, not
> specific drivers.
> 
> Agreed, though, that application developers must be made aware that
> the new API is only available in new kernels and old API must be used
> for compatibility with old kernels.

Is the conversion layer fully independent of vb2? We still have older
drivers that do not use vb2, and I'm not sure if the extended buffer
operations work with those drivers as well. (Sorry, it's been a while
since I last looked at this series, so I can't remember).

> 
>>>
>>>> +
>>>>  Dequeued video buffers come with timestamps. The driver decides at which
>>>>  part of the frame and with which clock the timestamp is taken. Please
>>>>  see flags in the masks ``V4L2_BUF_FLAG_TIMESTAMP_MASK`` and
>>>> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst
>>>> index 7d81c58a13cd7..3430e0bdad667 100644
>>>> --- a/Documentation/userspace-api/media/v4l/common.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/common.rst
>>>> @@ -59,6 +59,7 @@ applicable to all devices.
>>>>      ext-ctrls-detect
>>>>      fourcc
>>>>      format
>>>> +    ext-api
>>>>      planar-apis
>>>>      selection-api
>>>>      crop
>>>> diff --git a/Documentation/userspace-api/media/v4l/dev-capture.rst b/Documentation/userspace-api/media/v4l/dev-capture.rst
>>>> index 44d3094093ab6..5077639787d92 100644
>>>> --- a/Documentation/userspace-api/media/v4l/dev-capture.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/dev-capture.rst
>>>> @@ -102,6 +102,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>>>>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>>>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>>>>
>>>> +.. note::
>>>> +
>>>> +    The :ref:`ext_api` version can also be used, and it is
>>>> +    preferable when applicable.
>>>> +
>>>>
>>>>  Reading Images
>>>>  ==============
>>>> diff --git a/Documentation/userspace-api/media/v4l/dev-output.rst b/Documentation/userspace-api/media/v4l/dev-output.rst
>>>> index e4f2a1d8b0fc7..f8f40c708e49f 100644
>>>> --- a/Documentation/userspace-api/media/v4l/dev-output.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/dev-output.rst
>>>> @@ -99,6 +99,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>>>>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>>>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>>>>
>>>> +.. note::
>>>> +
>>>> +    The :ref:`ext_api` version can also be used, and it is
>>>> +    preferable when applicable.
>>>> +
>>>>
>>>>  Writing Images
>>>>  ==============
>>>> diff --git a/Documentation/userspace-api/media/v4l/ext-api.rst b/Documentation/userspace-api/media/v4l/ext-api.rst
>>>> new file mode 100644
>>>> index 0000000000000..da2a82960d22f
>>>> --- /dev/null
>>>> +++ b/Documentation/userspace-api/media/v4l/ext-api.rst
>>>> @@ -0,0 +1,107 @@
>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>> +.. document under the terms of the GNU Free Documentation License,
>>>> +.. Version 1.1 or any later version published by the Free Software
>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>> +..
>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>> +
>>>> +.. _ext_api:
>>>> +
>>>> +*************
>>>> +Extendend API
>>>> +*************
>>>> +
>>>> +Introduction
>>>> +============
>>>> +
>>>> +The Extended Format API was conceived to extend V4L2 capabilities and
>>>> +to simplify certain mechanisms.
>>>> +It unifies single- vs multi- planar handling,
>>>> +brings the concept of pixelformat + modifiers, allows planes to be placed
>>>> +in any offset inside a buffer and let userspace to change colorspace
>>>> +attributes if supported by the driver.
>>>
>>> How about simplifying this paragraph a bit:
>>>
>>> The Extended Format API unifies single- and multi- planar format handling
>>> and provides a base for further functional extensions, such as format
>>> modifiers, arbitrary plane offsets, colorspace configuration.
>>
>> Actually, I prefer Helen's version.
>>
> 
> To me it's a lot of text to read to get the same information, but I
> won't insist.
> 
>>>
>>>> +
>>>> +Data format negotiation and buffer handling works very similar to the classical
>>>> +version, thus in this document we'll just mention the main differences.
>>>
>>> nit: "we will" or maybe even just "we mention only the main differences".
>>>
>>>> +
>>>> +Data Format Negotiation
>>>> +=======================
>>>> +
>>>> +The API replaces the classical ioctls:
>>>> +
>>>> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>`, :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`,
>>>> +:ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>`
>>>> +(with :c:type:`v4l2_format` as the main parameter).
>>>> +
>>>> +By the extended versions:
>>>> +
>>>> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
>>>> +:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
>>>> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
>>>> +(with :c:type:`v4l2_ext_pix_format` as the main parameter).
>>>> +
>>>> +For CAPTURE and OUTPUT queues only.
>>>> +
>>>> +The ``type`` field of struct :c:type:`v4l2_ext_pix_format` only accepts
>>>> +``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or ``V4L2_BUF_TYPE_VIDEO_OUTPUT``, and it
>>>> +supports multiplanar format through a combination of ``pixelfomat`` and
>>>> +``modifier`` fields.
>>>> +
>>>> +Only the single-planar variants of the pixel formats are valid in the
>>>> +``pixelformat`` field.
>>>
>>> Some formats have only multi-planar variants, like NV12MT. What should we
>>> do about them? Since we have to keep those defines to support the legacy
>>> API anyway, maybe we can just accept both variants and treat them equally?
>>
>> Helen mentioned that as well in a reply to patch 1/7. I'll have a think about
>> that myself and reply to that post because this is a good question.
>>
> 
> I'm thinking about it as well. :)
> 
>>>
>>>> +To support multi-planar, a modifier should be set in the ``modifier`` field to
>>>> +provide more information regarding the memory layout of pixels and number of
>>>> +planes.
>>>> +
>>>> +The ``plane_fmt`` field is an array which holds information by plane using
>>>> +the :c:type:`v4l2_plane_ext_pix_format`. When this struct is filled, its
>>>> +``sizeimage`` field should be non-zero for all valid planes for a given
>>>> +``pixelformat`` + ``modifier`` combination, and zeroed for the invalid ones.
>>>> +
>>>> +Colospace attributes are not read-only as in the classical version, i.e, they
>>>> +can be set by application and drivers will adjust accordingly depending on what
>>>> +is supported by the underlying hardware.
>>>> +
>>>> +Buffers
>>>> +=======
>>>> +
>>>> +The API replaces the classical ioctls:
>>>> +
>>>> +:ref:`VIDIOC_CREATE_BUFS <VIDIOC_CREATE_BUFS>`
>>>> +(with :c:type:`v4l2_create_buffers` as the main parameter),
>>>> +:ref:`VIDIOC_QUERYBUF <VIDIOC_QUERYBUF>`, :ref:`VIDIOC_QBUF <VIDIOC_QBUF>`,
>>>> +:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`, :ref:`VIDIOC_PREPARE_BUF <VIDIOC_PREPARE_BUF>`
>>>> +(with :c:type:`v4l2_buffer` as the main parameter)
>>>> +
>>>> +By the extended versions:
>>>> +
>>>> +:ref:`VIDIOC_EXT_CREATE_BUFS <VIDIOC_EXT_CREATE_BUFS>`
>>>> +(with :c:type:`v4l2_ext_create_buffers` as the main parameter),
>>>> +:ref:`VIDIOC_EXT_QUERYBUF <VIDIOC_EXT_QUERYBUF>`,
>>>> +:ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>`,
>>>> +:ref:`VIDIOC_EXT_DQBUF <VIDIOC_EXT_QBUF>`,
>>>> +:ref:`VIDIOC_EXT_PREPARE_BUF <VIDIOC_EXT_PREPARE_BUF>`
>>>> +(with :c:type:`v4l2_ext_buffer` as the main parameter)
>>>> +
>>>> +The basic difference between :c:type:`v4l2_create_buffers` and
>>>> +:c:type:`v4l2_ext_create_buffers` is that the later have a
>>>> +:c:type:`v4l2_ext_pix_format` as the type of the ``format`` field.
>>>> +
>>>> +Comparing :c:type:`v4l2_ext_buffer` with :c:type:`v4l2_buffer`, in the
>>>> +extended version there is a unification of the single-/multi- planar handling
>>>> +through the ``planes`` field of type :c:type:`v4l2_ext_plane`.
>>>> +
>>>> +The :c:type:`v4l2_ext_plane also allows planes to be placed in a given offset
>>>> +inside a buffer.
>>>> +Planes can be placed in different locations inside the same buffer, or in
>>>> +different buffers.
>>>> +
>>>> +
>>>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>>>> +   :functions: v4l2_ext_plane
>>>> +
>>>> +
>>>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>>>> +   :functions: v4l2_ext_buffer
>>>> diff --git a/Documentation/userspace-api/media/v4l/format.rst b/Documentation/userspace-api/media/v4l/format.rst
>>>> index e47fc0505727c..b96d26f26793c 100644
>>>> --- a/Documentation/userspace-api/media/v4l/format.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/format.rst
>>>> @@ -28,13 +28,19 @@ format and the driver selects and reports the best the hardware can do
>>>>  to satisfy the request. Of course applications can also just query the
>>>>  current selection.
>>>>
>>>> -A single mechanism exists to negotiate all data formats using the
>>>> -aggregate struct :c:type:`v4l2_format` and the
>>>> +There are two mechanism to negociate data formats:
>>>
>>> negotiate
>>>
>>>> +
>>>> +The first one is using the aggregate struct :c:type:`v4l2_format` and the
>>>>  :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and
>>>>  :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctls. Additionally the
>>>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` ioctl can be used to examine
>>>>  what the hardware *could* do, without actually selecting a new data
>>>> -format. The data formats supported by the V4L2 API are covered in the
>>>> +format.
>>>> +
>>>> +The second is through the :ref:`ext_api`, please refer to its documentation
>>>> +for more information.
>>>> +
>>>
>>> Should we reverse the order to prefer the new API?
>>>
>>>> +The data formats supported by the V4L2 API are covered in the
>>>>  respective device section in :ref:`devices`. For a closer look at
>>>>  image formats see :ref:`pixfmt`.
>>>>
>>>> @@ -71,7 +77,9 @@ earlier versions of V4L2. Switching the logical stream or returning into
>>>>  *may* support a switch using :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`.
>>>>
>>>>  All drivers exchanging data with applications must support the
>>>> -:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl. Implementation of the
>>>> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl
>>>> +or the Extended respective versions (TODO: link).
>>>
>>> TODO? :)
>>>
>>>> +Implementation of the
>>>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is highly recommended but optional.
>>>>
>>>>
>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
>>>> index bf77c842718e5..67b254812791c 100644
>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst
>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
>>>> @@ -20,6 +20,7 @@ Function Reference
>>>>      func-close
>>>>      func-ioctl
>>>>      vidioc-create-bufs
>>>> +    vidioc-ext-create-bufs
>>>>      vidioc-cropcap
>>>>      vidioc-dbg-g-chip-info
>>>>      vidioc-dbg-g-register
>>>> @@ -48,6 +49,7 @@ Function Reference
>>>>      vidioc-g-ext-ctrls
>>>>      vidioc-g-fbuf
>>>>      vidioc-g-fmt
>>>> +    vidioc-g-ext-pix-fmt
>>>>      vidioc-g-frequency
>>>>      vidioc-g-input
>>>>      vidioc-g-jpegcomp
>>>> @@ -62,8 +64,11 @@ Function Reference
>>>>      vidioc-log-status
>>>>      vidioc-overlay
>>>>      vidioc-prepare-buf
>>>> +    vidioc-ext-prepare-buf
>>>>      vidioc-qbuf
>>>> +    vidioc-ext-qbuf
>>>>      vidioc-querybuf
>>>> +    vidioc-ext-querybuf
>>>>      vidioc-querycap
>>>>      vidioc-queryctrl
>>>>      vidioc-query-dv-timings
>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>>>> new file mode 100644
>>>> index 0000000000000..67f2c541e4d79
>>>> --- /dev/null
>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>>>> @@ -0,0 +1,95 @@
>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>> +.. document under the terms of the GNU Free Documentation License,
>>>> +.. Version 1.1 or any later version published by the Free Software
>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>> +..
>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>> +
>>>
>>> TODO? :)
>>>
>>>> +.. _VIDIOC_EXT_CREATE_BUFS:
>>>> +
>>>> +****************************
>>>> +ioctl VIDIOC_EXT_CREATE_BUFS
>>>> +****************************
>>>> +
>>>> +Name
>>>> +====
>>>> +
>>>> +VIDIOC_EXT_CREATE_BUFS - Create buffers for Memory Mapped or User Pointer or DMA Buffer I/O
>>>> +
>>>> +
>>>> +Synopsis
>>>> +========
>>>> +
>>>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp )
>>>> +    :name: VIDIOC_EXT_CREATE_BUFS
>>>> +
>>>> +
>>>> +Arguments
>>>> +=========
>>>> +
>>>> +``fd``
>>>> +    File descriptor returned by :ref:`open() <func-open>`.
>>>> +
>>>> +``argp``
>>>> +    Pointer to struct :c:type:`v4l2_ext_create_buffers`.
>>>> +
>>>> +
>>>> +Description
>>>> +===========
>>>> +
>>>> +This ioctl is used to create buffers for :ref:`memory mapped <mmap>`
>>>> +or :ref:`user pointer <userp>` or :ref:`DMA buffer <dmabuf>` I/O.
>>>> +This ioctl can be called multiple times to
>>>> +create buffers of different sizes.
>>>> +
>>>> +To allocate the device buffers applications must initialize the relevant
>>>> +fields of the struct :c:type:`v4l2_ext_create_buffers` structure. The
>>>> +``count`` field must be set to the number of requested buffers, the
>>>> +``memory`` field specifies the requested I/O method and the ``reserved``
>>>> +array must be zeroed.
>>>> +
>>>> +The ``format`` field specifies the image format that the buffers must be
>>>> +able to handle. The application has to fill in this struct
>>>> +:c:type:`v4l2_format`. Usually this will be done using the
>>>> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` or
>>>> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctls to ensure that the
>>>> +requested format is supported by the driver.
>>>> +The driver may return an error if the size(s) are not supported by the
>>>> +hardware (usually because they are too small).
>>>> +
>>>> +The driver can create a memory buffer per plane, or a single memory buffer
>>>> +containing all the planes, with a minimum size according to the size
>>>> +defined by the ``format.plane_fmt[i].sizeimage`` field.
>>>> +Usually if the ``format.plane_fmt[i].sizeimage``
>>>> +field is less than the minimum required for the given format, then an
>>>> +error will be returned since drivers will typically not allow this. If
>>>> +it is larger, then the value will be used as-is. In other words, the
>>>> +driver may reject the requested size, but if it is accepted the driver
>>>> +will use it unchanged.
>>>> +
>>>> +When the ioctl is called with a pointer to this structure the driver
>>>> +will attempt to allocate up to the requested number of buffers and store
>>>> +the actual number allocated and the starting index in the ``count`` and
>>>> +the ``index`` fields respectively. On return ``count`` can be smaller
>>>> +than the number requested.
>>>
>>> The 'capabilities' and 'flags' fields seem to go undocumented.
>>>
>>>> +
>>>> +
>>>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>>>> +   :functions: v4l2_ext_create_buffers
>>>> +
>>>> +
>>>> +Return Value
>>>> +============
>>>> +
>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>> +appropriately. The generic error codes are described at the
>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>> +
>>>> +ENOMEM
>>>> +    No memory to allocate buffers for :ref:`memory mapped <mmap>` I/O.
>>>> +
>>>> +EINVAL
>>>> +    The buffer type (``format.type`` field), requested I/O method
>>>> +    (``memory``) or format (``format`` field) is not valid.
>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>>>> new file mode 100644
>>>> index 0000000000000..88e4b57121254
>>>> --- /dev/null
>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>>>> @@ -0,0 +1,62 @@
>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>> +.. document under the terms of the GNU Free Documentation License,
>>>> +.. Version 1.1 or any later version published by the Free Software
>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>> +..
>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>> +
>>>> +.. _VIDIOC_EXT_PREPARE_BUF:
>>>> +
>>>> +****************************
>>>> +ioctl VIDIOC_EXT_PREPARE_BUF
>>>> +****************************
>>>> +
>>>> +Name
>>>> +====
>>>> +
>>>> +VIDIOC_EXT_PREPARE_BUF - Prepare a buffer for I/O
>>>> +
>>>> +
>>>> +Synopsis
>>>> +========
>>>> +
>>>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp )
>>>> +    :name: VIDIOC_EXT_PREPARE_BUF
>>>> +
>>>> +
>>>> +Arguments
>>>> +=========
>>>> +
>>>> +``fd``
>>>> +    File descriptor returned by :ref:`open() <func-open>`.
>>>> +
>>>> +``argp``
>>>> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
>>>> +
>>>> +
>>>> +Description
>>>> +===========
>>>> +
>>>> +Applications can optionally call the :ref:`VIDIOC_EXT_PREPARE_BUF` ioctl to
>>>> +pass ownership of the buffer to the driver before actually enqueuing it,
>>>> +using the :ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>` ioctl, and to prepare it for future I/O. Such
>>>> +preparations may include cache invalidation or cleaning. Performing them
>>>> +in advance saves time during the actual I/O.
>>>> +
>>>> +
>>>> +Return Value
>>>> +============
>>>> +
>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>> +appropriately. The generic error codes are described at the
>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>> +
>>>> +EBUSY
>>>> +    File I/O is in progress.
>>>> +
>>>> +EINVAL
>>>> +    The buffer ``type`` is not supported, or the ``index`` is out of
>>>> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
>>>> +    ``length`` are invalid.
>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>>>> new file mode 100644
>>>> index 0000000000000..083e106cf6bad
>>>> --- /dev/null
>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>>>> @@ -0,0 +1,204 @@
>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>> +.. document under the terms of the GNU Free Documentation License,
>>>> +.. Version 1.1 or any later version published by the Free Software
>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>> +..
>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>> +
>>>> +.. _VIDIOC_EXT_QBUF:
>>>> +
>>>> +***************************************
>>>> +ioctl VIDIOC_EXT_QBUF, VIDIOC_EXT_DQBUF
>>>> +***************************************
>>>> +
>>>> +Name
>>>> +====
>>>> +
>>>> +VIDIOC_EXT_QBUF - VIDIOC_EXT_DQBUF - Exchange a buffer with the driver
>>>> +
>>>> +
>>>> +Synopsis
>>>> +========
>>>> +
>>>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
>>>> +    :name: VIDIOC_EXT_QBUF
>>>> +
>>>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp )
>>>> +    :name: VIDIOC_EXT_DQBUF
>>>> +
>>>> +
>>>> +Arguments
>>>> +=========
>>>> +
>>>> +``fd``
>>>> +    File descriptor returned by :ref:`open() <func-open>`.
>>>> +
>>>> +``argp``
>>>> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
>>>> +
>>>> +
>>>> +Description
>>>> +===========
>>>> +
>>>> +Applications call the ``VIDIOC_EXT_QBUF`` ioctl to enqueue an empty
>>>> +(capturing) or filled (output) buffer in the driver's incoming queue.
>>>> +The semantics depend on the selected I/O method.
>>>> +
>>>> +To enqueue a buffer applications set the ``type`` field of a struct
>>>> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
>>>> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``.
>>>> +Applications must also set the ``index`` field. Valid index numbers
>>>> +range from zero to the number of buffers allocated with
>>>> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
>>>> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
>>>> +one. The contents of the struct :c:type:`v4l2_ext_buffer` returned
>>>
>>> I think this might not be entirely accurate. One can call
>>> VIDIOC_EXT_CREATE_BUFS multiple times. Moreover, I don't think we've
>>> prohibited VIDIOC_REQBUFS for the ext API, or have we?
>>>
>>>> +by a :ref:`VIDIOC_EXT_QUERYBUF` ioctl will do as well.
>>>> +When the buffer is intended for output (``type`` is
>>>> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``) applications must also initialize the
>>>> +``timestamp`` fields and the ``planes[i].plane_length`` for each valid plane,
>>>> +and invalid ones must be set as zero.
>>>> +see :ref:`buffer` for details. Applications must also set ``flags`` to 0. The
>>>> +``reserved`` field must be set to 0.
>>>> +
>>>> +To enqueue a :ref:`memory mapped <mmap>` buffer applications set the
>>>> +``planes[i].memory`` field to ``V4L2_MEMORY_MMAP`` in all the valid planes,
>>>> +and invalid ones must be set as zero.
>>>> +When ``VIDIOC_EXT_QBUF`` is called
>>>> +with a pointer to this structure the driver sets the
>>>> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_QUEUED`` flags and clears
>>>> +the ``V4L2_BUF_FLAG_DONE`` flag in the ``flags`` field, or it returns an
>>>> +``EINVAL`` error code.
>>>> +
>>>> +To enqueue a :ref:`user pointer <userp>` buffer applications set the
>>>> +``planes[i].memory`` field to ``V4L2_MEMORY_USERPTR`` in all the valid planes,
>>>> +and invalid ones must be set as zero, the ``planes[i].m.userptr`` field to the
>>>> +address of the buffer,``planes[i].buffer_length`` to the size of the memory
>>>> +buffer, ``planes[i].plane_length`` to the size that should be used by the plane,
>>>> +and ``planes[i].offset`` of the plane in the memory buffer.
>>>
>>> Does offset make sense here if an arbitrary pointer can be given for the
>>> start of the plane?
>>>
>>>> +
>>>> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure
>>>> +the driver sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
>>>> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
>>>> +``flags`` field, or it returns an error code. This ioctl locks the
>>>
>>> This seems to be a copy of the text few lines above.
>>>
>>>> +memory pages of the buffer in physical memory, they cannot be swapped
>>>> +out to disk. Buffers remain locked until dequeued, until the
>>>> +:ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
>>>> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
>>>> +device is closed.
>>>
>>> Why would CREATE_BUFS affect existing buffers?
>>>
>>>> +
>>>> +To enqueue a :ref:`DMABUF <dmabuf>` buffer applications set the
>>>> +``planes[i].memory`` field to ``V4L2_MEMORY_DMABUF`` in all the valid planes,
>>>> +and invalid ones must be set as zero, the ``planes[i].m.fd`` field to a
>>>> +file descriptor associated with a DMABUF buffer.,``planes[i].buffer_length`` to
>>>> +the size of the memory buffer, ``planes[i].plane_length`` to the size that
>>>
>>> What is the buffer_length field needed for? DMA-bufs by definition are
>>> kernel objects and the kernel should already know their sizes.
>>
>> See this discussion:
>>
>> https://patchwork.linuxtv.org/project/linux-media/patch/gh5kef5bkeel3o6b2dkgc2dfagu9klj4c0@4ax.com/
>>
> 
> I briefly read through it and I don't see what the conclusion was.
> Generally I'd still prefer if the need to manually specify the buffer
> size was eliminated, because that imposes unnecessary burden on the
> userspace for the majority of cases. It could still remain valid for
> userspace that wants to restrict the buffer size as it was mentioned
> in that thread, though.

That's really the open issue: if userspace sets length to 0, then it should
always be replaced by the dmabuf size, that's a no-brainer. But if it is non-zero,
what do you do then? Your input would be appreciated.

> 
>>>
>>>> +should be used by the plane, and ``planes[i].offset`` of the plane in the memory buffer.
>>>> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure the driver
>>>> +sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
>>>> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
>>>> +``flags`` field, or it returns an error code. This ioctl locks the
>>>
>>> Ditto.
>>>
>>>> +buffer. Locking a buffer means passing it to a driver for a hardware
>>>> +access (usually DMA). If an application accesses (reads/writes) a locked
>>>> +buffer then the result is undefined. Buffers remain locked until
>>>> +dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
>>>> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
>>>> +device is closed.
>>>
>>> Ditto.
>>>
>>>> +
>>>> +The ``request_fd`` field can be used with the ``VIDIOC_EXT_QBUF`` ioctl to specify
>>>> +the file descriptor of a :ref:`request <media-request-api>`, if requests are
>>>> +in use. Setting it means that the buffer will not be passed to the driver
>>>> +until the request itself is queued. Also, the driver will apply any
>>>> +settings associated with the request for this buffer. This field will
>>>> +be ignored unless the ``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
>>>> +If the device does not support requests, then ``EBADR`` will be returned.
>>>> +If requests are supported but an invalid request file descriptor is given,
>>>> +then ``EINVAL`` will be returned.
>>>> +
>>>> +.. caution::
>>>> +   It is not allowed to mix queuing requests with queuing buffers directly.
>>>> +   ``EBUSY`` will be returned if the first buffer was queued directly and
>>>> +   then the application tries to queue a request, or vice versa. After
>>>> +   closing the file descriptor, calling
>>>> +   :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or calling :ref:`VIDIOC_EXT_CREATE_BUFS`
>>>> +   the check for this will be reset.
>>>> +
>>>> +   For :ref:`memory-to-memory devices <mem2mem>` you can specify the
>>>> +   ``request_fd`` only for output buffers, not for capture buffers. Attempting
>>>> +   to specify this for a capture buffer will result in an ``EBADR`` error.
>>>> +
>>>> +Applications call the ``VIDIOC_EXT_DQBUF`` ioctl to dequeue a filled
>>>> +(capturing) or displayed (output) buffer from the driver's outgoing
>>>
>>> "displayed" is quite inaccurate given that the majority of use cases for
>>> OUTPUT in V4L2 now are m2m devices. Maybe "consumed"?
>>>
>>>> +queue. They just set the ``type``, ``planes[i].memory`` and ``reserved`` fields of
>>>
>>> Why does the application have to set the memory field? It should be already
>>> known by the kernel which owns the buffer at the given time.
>>>
>>>> +a struct :c:type:`v4l2_ext_buffer` as above, when
>>>> +``VIDIOC_EXT_DQBUF`` is called with a pointer to this structure the driver
>>>> +fills the remaining fields or returns an error code. The driver may also
>>>> +set ``V4L2_BUF_FLAG_ERROR`` in the ``flags`` field. It indicates a
>>>> +non-critical (recoverable) streaming error. In such case the application
>>>> +may continue as normal, but should be aware that data in the dequeued
>>>> +buffer might be corrupted. When using the multi-planar API, the planes
>>>> +array must be passed in as well.
>>>> +
>>>> +If the application sets the ``memory`` field to ``V4L2_MEMORY_DMABUF`` to
>>>> +dequeue a :ref:`DMABUF <dmabuf>` buffer, the driver fills the ``m.fd`` field
>>>> +with a file descriptor numerically the same as the one given to ``VIDIOC_EXT_QBUF``
>>>> +when the buffer was enqueued. No new file descriptor is created at dequeue time
>>>> +and the value is only for the application convenience.
>>>
>>> Honestly, I don't like this aspect of the current API. It kind of implies
>>> that the value returned is valid, but the file descriptor might have been
>>> already closed. Could we just remove this and explicitly state that for
>>> DMA-buf, it is the user space responsibility to track the mapping between
>>> V4L2 indexes and its own buffer objects?
>>>
>>>> +
>>>> +By default ``VIDIOC_EXT_DQBUF`` blocks when no buffer is in the outgoing
>>>> +queue. When the ``O_NONBLOCK`` flag was given to the
>>>> +:ref:`open() <func-open>` function, ``VIDIOC_EXT_DQBUF`` returns
>>>> +immediately with an ``EAGAIN`` error code when no buffer is available.
>>>> +
>>>> +
>>>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>>>> +   :functions: v4l2_ext_buffers
>>>> +
>>>> +
>>>> +Return Value
>>>> +============
>>>> +
>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>> +appropriately. The generic error codes are described at the
>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>> +
>>>> +EAGAIN
>>>> +    Non-blocking I/O has been selected using ``O_NONBLOCK`` and no
>>>> +    buffer was in the outgoing queue.
>>>> +
>>>> +EINVAL
>>>> +    The buffer ``type`` is not supported, or the ``index`` is out of
>>>> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
>>>> +    ``length`` are invalid, or the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was
>>>
>>> There is no 'length' now.
>>>
>>>> +    set but the the given ``request_fd`` was invalid, or ``m.fd`` was
>>>> +    an invalid DMABUF file descriptor.
>>>> +
>>>> +EIO
>>>> +    ``VIDIOC_EXT_DQBUF`` failed due to an internal error. Can also indicate
>>>> +    temporary problems like signal loss.
>>
>> Drop the last sentence, -EIO should never be used for temporary problems like
>> that. EIO really means that the DMA engine crashed and userspace will have to call
>> STREAMOFF and try to restart the DMA with STREAMON.
>>
> 
> I suspect the users of the new API wouldn't actually have to deal with
> a signal loss, but if not, what should be the behavior then?

The typical behavior is to send out the SOURCE_CHANGE event and stall streaming,
waiting for a valid signal.

That said, this is something I will dig into a bit deeper for the tegra HDMI
support patch series next week, since this might be hard to do for that hardware.

But regardless of what is chosen for signal loss, EIO is wrong. Return values
as are documented for VIDIOC_QUERY_DV_TIMINGS would be much more appropriate for this.

> 
>>>> +
>>>> +    .. note::
>>>> +
>>>> +       The driver might dequeue an (empty) buffer despite returning
>>>> +       an error, or even stop capturing. Reusing such buffer may be unsafe
>>>> +       though and its details (e.g. ``index``) may not be returned either.
>>
>> The first part of this note is rather vague. I'm not sure what it meant with this.
>>
>>>> +       It is recommended that drivers indicate recoverable errors by setting
>>>> +       the ``V4L2_BUF_FLAG_ERROR`` and returning 0 instead. In that case the
>>>> +       application should be able to safely reuse the buffer and continue
>>>> +       streaming.
>>>
>>> What is this for? It sounds like some legacy behavior, which maybe we could
>>> drop with this new API?
>>
>> No, V4L2_BUF_FLAG_ERROR is really needed. Some hardware can really have
>> intermittent failures causing video frames to be corrupt. And yes, that's
>> typically due to hardware bugs. It usually either happens when you start
>> streaming or just intermittent, but it is not severe enough to stop streaming
>> completely (that's the -EIO error due to a call to vb2_queue_error).
>>
>> It's also used when you stop streaming and pending buffers are returned to
>> userspace. They are marked as ERROR since they don't contain any video, or
>> only partial video (if the DMA was stopped midway a frame).
>>
> 
> Sorry, I meant to insert that comment below the first half. The second
> half is fine.

Ah, that makes much more sense :-)

> 
>>>
>>>> +
>>>> +EPIPE
>>>> +    ``VIDIOC_EXT_DQBUF`` returns this on an empty capture queue for mem2mem
>>>> +    codecs if a buffer with the ``V4L2_BUF_FLAG_LAST`` was already
>>>> +    dequeued and no new buffers are expected to become available.
>>>> +
>>>> +EBADR
>>>> +    The ``V4L2_BUF_FLAG_REQUEST_FD`` flag was set but the device does not
>>>> +    support requests for the given buffer type, or
>>>> +    the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was not set but the device requires
>>>> +    that the buffer is part of a request.
>>>> +
>>>> +EBUSY
>>>> +    The first buffer was queued via a request, but the application now tries
>>>> +    to queue it directly, or vice versa (it is not permitted to mix the two
>>>> +    APIs).
>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>>>> new file mode 100644
>>>> index 0000000000000..f2a12017253f6
>>>> --- /dev/null
>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>>>> @@ -0,0 +1,79 @@
>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>> +.. document under the terms of the GNU Free Documentation License,
>>>> +.. Version 1.1 or any later version published by the Free Software
>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>> +..
>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>> +
>>>> +.. _VIDIOC_EXT_QUERYBUF:
>>>> +
>>>> +*************************
>>>> +ioctl VIDIOC_EXT_QUERYBUF
>>>> +*************************
>>>> +
>>>> +Name
>>>> +====
>>>> +
>>>> +VIDIOC_EXT_QUERYBUF - Query the status of a buffer
>>>> +
>>>> +
>>>> +Synopsis
>>>> +========
>>>> +
>>>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp )
>>>> +    :name: VIDIOC_EXT_QUERYBUF
>>>> +
>>>> +
>>>> +Arguments
>>>> +=========
>>>> +
>>>> +``fd``
>>>> +    File descriptor returned by :ref:`open() <func-open>`.
>>>> +
>>>> +``argp``
>>>> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
>>>> +
>>>> +
>>>> +Description
>>>> +===========
>>>> +
>>>> +This ioctl is part of the :ref:`streaming <mmap>` I/O method. It can
>>>> +be used to query the status of a buffer at any time after buffers have
>>>> +been allocated with the :ref:`VIDIOC_EXT_CREATE_BUFS` ioctl.
>>>> +
>>>> +Applications set the ``type`` field of a struct
>>>> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
>>>> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``,
>>>> +and the ``index`` field. Valid index numbers range from zero to the
>>>> +number of buffers allocated with
>>>> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
>>>> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
>>>> +one. The ``reserved`` field must be set to 0.
>>>> +
>>>> +In the ``flags`` field the ``V4L2_BUF_FLAG_MAPPED``,
>>>> +``V4L2_BUF_FLAG_PREPARED``, ``V4L2_BUF_FLAG_QUEUED`` and
>>>> +``V4L2_BUF_FLAG_DONE`` flags will be valid. The ``planes.memory`` fields will be
>>>> +set to the current I/O method for each plane.
>>>> +
>>>> +For every valid plane, an entry in ``planes`` will be filled, and zeroed for
>>>> +invalid ones. ``planes[i].buffer_length`` is the size of the memory buffer
>>>> +which contains the plane, ``planes[i].plane_length`` is the length of the plane,
>>>
>>> What is the length of the plane here? It seems like depending on the
>>> context, the field can mean size of the payload (for filled buffer) or
>>> total space for the payload (for empty buffer), which is a bit confusing.
>>>
>>>> +and ``planes[i].offset` is where the plane is placed in the memory buffer.
>>>> +
>>>> +The size of the ``planes`` array can be calculated by the number of sequential
>>>> +planes with ``planes[i].buffer_length`` that differs from zero up to the max
>>>> +size of the array.
>>>
>>> Not sure if it's just me, but "size of array" sounds to me like the
>>> total memory size of the array, i.e. the max number of elements, which
>>> is fixed for this struct. How about
>>>
>>>   "The entries of the planes array are valid up to the last entry with
>>>   non-zero buffer_length or the last entry of the array, whichever is
>>>   first."
>>>
>>> ?
>>>
>>>> +
>>>> +
>>>> +Return Value
>>>> +============
>>>> +
>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>> +appropriately. The generic error codes are described at the
>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>> +
>>>> +EINVAL
>>>> +    The buffer ``type`` is not supported, or the ``index`` is out of
>>>> +    bounds.
>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>>>> new file mode 100644
>>>> index 0000000000000..008e6c98a88a5
>>>> --- /dev/null
>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>>>> @@ -0,0 +1,117 @@
>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>> +.. document under the terms of the GNU Free Documentation License,
>>>> +.. Version 1.1 or any later version published by the Free Software
>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>> +..
>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>> +
>>>> +.. _VIDIOC_G_EXT_PIX_FMT:
>>>> +
>>>> +************************************************************************
>>>> +ioctl VIDIOC_G_EXT_PIX_FMT, VIDIOC_S_EXT_PIX_FMT, VIDIOC_TRY_EXT_PIX_FMT
>>>> +************************************************************************
>>>> +
>>>> +Name
>>>> +====
>>>> +
>>>> +VIDIOC_G_EXT_PIX_FMT - VIDIOC_S_EXT_PIX_FMT - VIDIOC_TRY_EXT_PIX_FMT - Get or set the data format, try a format
>>>> +
>>>> +
>>>> +Synopsis
>>>> +========
>>>> +
>>>> +.. c:function:: int ioctl( int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
>>>> +    :name: VIDIOC_G_EXT_PIX_FMT
>>>> +
>>>> +.. c:function:: int ioctl( int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
>>>> +    :name: VIDIOC_S_EXT_PIX_FMT
>>>> +
>>>> +.. c:function:: int ioctl( int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
>>>> +    :name: VIDIOC_TRY_EXT_PIX_FMT
>>>> +
>>>> +Arguments
>>>> +=========
>>>> +
>>>> +``fd``
>>>> +    File descriptor returned by :ref:`open() <func-open>`.
>>>> +
>>>> +``argp``
>>>> +    Pointer to struct :c:type:`v4l2_ext_pix_format`.
>>>> +
>>>> +
>>>> +Description
>>>> +===========
>>>> +
>>>> +These ioctls are used to negotiate the format of data (typically image
>>>> +format) exchanged between driver and application.
>>>> +
>>>> +To query the current parameters applications set the ``type`` field of a
>>>> +struct :c:type:`v4l2_ext_pix_format` to  ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or
>>>> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``, all the other types are invalid in this API,
>>>> +and multiplanar is supported through modifiers.
>>>> +
>>>> +When the application calls the
>>>> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with a pointer to this
>>>> +structure the driver fills the other members.
>>>> +When the requested buffer type is not supported drivers return
>>>> +an ``EINVAL`` error code.
>>>> +
>>>> +To change the current format parameters applications initialize all
>>>> +the fields in the struct.
>>>> +For details see the documentation of the various devices types in
>>>> +:ref:`devices`. Good practice is to query the current parameters
>>>> +first, and to modify only those parameters not suitable for the
>>>> +application. When the application calls the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with
>>>> +a pointer to a struct :c:type:`v4l2_ext_pix_format` structure the driver
>>>> +checks and adjusts the parameters against hardware abilities. Drivers
>>>> +should not return an error code unless the ``type`` field is invalid,
>>>> +this is a mechanism to fathom device capabilities and to approach
>>>> +parameters acceptable for both the application and driver. On success
>>>> +the driver may program the hardware, allocate resources and generally
>>>> +prepare for data exchange. Finally the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl returns
>>>> +the current format parameters as :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` does. Very simple,
>>>> +inflexible devices may even ignore all input and always return the
>>>> +default parameters. However all V4L2 devices exchanging data with the
>>>> +application must implement the :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` and :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
>>>> +ioctl. When the requested buffer type is not supported drivers return an
>>>> +EINVAL error code on a :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` attempt. When I/O is already in
>>>> +progress or the resource is not available for other reasons drivers
>>>> +return the ``EBUSY`` error code.
>>>> +
>>>> +The :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl is equivalent to :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` with one
>>>> +exception: it does not change driver state. It can also be called at any
>>>> +time, never returning ``EBUSY``. This function is provided to negotiate
>>>> +parameters, to learn about hardware limitations, without disabling I/O
>>>> +or possibly time consuming hardware preparations. Although strongly
>>>> +recommended drivers are not required to implement this ioctl.
>>>> +
>>>
>>> Could we make this mandatory to avoid ambiguities in the API? In
>>
>> Yes, please!
>>
>>> general, most drivers implemented it anyway, without much added
>>> complexity, because one can then simplify S_FMT to TRY_FMT + state
>>> update.
>>
>> Ideally we would also fix the ambiguous situation regarding TRY/S_FMT
>> and unsupported pixelformats: some drivers will replace unsupported
>> pixelformats with a valid pixelformat, others will return an error.
>>
>> v4l2-compliance warns about drivers that return an error and point to
>> this very old thread:
>>
>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg56550.html
>>
>> However, I'm not really sure if we can resolve this ambiguity. I believe
>> that it is only older webcam drivers that return an error.
> 
> I guess we can at least enforce a specific behavior in the new API,
> before we get compatibility obligations. :)
> 
> Best regards,
> Tomasz
> 

Regards,

	Hans

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-09-09 12:27   ` Hans Verkuil
@ 2020-11-23 15:08     ` Helen Koike
  2020-11-23 15:46       ` Tomasz Figa
  0 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-11-23 15:08 UTC (permalink / raw)
  To: Hans Verkuil, mchehab, hans.verkuil, laurent.pinchart,
	sakari.ailus, linux-media
  Cc: Boris Brezillon, tfiga, hiroh, nicolas, Brian.Starkey, kernel,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

Hi Hans,

Thank you for your review.

On 9/9/20 9:27 AM, Hans Verkuil wrote:
> Hi Helen,
> 
> Again I'm just reviewing the uAPI.
> 
> On 04/08/2020 21:29, Helen Koike wrote:
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> Those extended buffer ops have several purpose:
>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>    the number of ns elapsed since 1970
>> 2/ Unify single/multiplanar handling
>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>    to support the case where a single buffer object is storing all
>>    planes data, each one being placed at a different offset
>>
>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>> these new objects.
>>
>> The core takes care of converting new ioctls requests to old ones
>> if the driver does not support the new hooks, and vice versa.
>>
>> Note that the timecode field is gone, since there doesn't seem to be
>> in-kernel users. We can be added back in the reserved area if needed or
>> use the Request API to collect more metadata information from the
>> frame.
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> ---
>> Changes in v5:
>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
>> - return mem_offset to struct v4l2_ext_plane
>> - change sizes and reorder fields to avoid holes in the struct and make
>>   it the same for 32 and 64 bits
>>
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>> I think we can add this later, so I removed it from this RFC to simplify it.
>> - Remove num_planes field from struct v4l2_ext_buffer
>> - Add flags field to struct v4l2_ext_create_buffers
>> - Reformulate struct v4l2_ext_plane
>> - Fix some bugs caught by v4l2-compliance
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>   later on
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>>  include/media/v4l2-ioctl.h           |  26 ++
>>  include/uapi/linux/videodev2.h       |  90 +++++++
>>  4 files changed, 476 insertions(+), 22 deletions(-)
>>
> 
> <snip>
> 
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index 7123c6a4d9569..334cafdd2be97 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -996,6 +996,41 @@ struct v4l2_plane {
>>  	__u32			reserved[11];
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_plane - extended plane buffer info
>> + * @buffer_length:	size of the entire buffer in bytes, should fit
>> + *			@offset + @plane_length
>> + * @plane_length:	size of the plane in bytes.
>> + * @mem_offset:		If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
>> + *			that should be passed to mmap() called on the video node.
>> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>> + *			to this plane.
>> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>> + *			associated with this plane.
>> + * @offset:		offset in the memory buffer where the plane starts.
>> + * @memory:		enum v4l2_memory; the method, in which the actual video
>> + *			data is passed
>> + * @reserved:		extra space reserved for future fields, must be set to 0.
>> + *
>> + *
>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>> + * can have one plane for Y, and another for interleaved CbCr components.
>> + * Each plane can reside in a separate memory buffer, or even in
>> + * a completely separate memory node (e.g. in embedded devices).
>> + */
>> +struct v4l2_ext_plane {
>> +	__u32 buffer_length;
>> +	__u32 plane_length;
>> +	union {
>> +		__u32 mem_offset;
>> +		__u64 userptr;
>> +		__s32 dmabuf_fd;
>> +	} m;
>> +	__u32 offset;
> 
> I'd rename this plane_offset. I think some reordering would make this struct easier
> to understand:
> 
> struct v4l2_ext_plane {
> 	__u32 buffer_length;
> 	__u32 plane_offset;
> 	__u32 plane_length;
> 	__u32 memory;
> 	union {
> 		__u32 mem_offset;
> 		__u64 userptr;
> 		__s32 dmabuf_fd;
> 	} m;
> 	__u32 reserved[4];
> };
> 
>> +	__u32 memory;
>> +	__u32 reserved[4];
>> +};

Ok, I'll apply this to the next version.

> 
> What is not clear is how to tell the different between a single buffer containing
> multiple planes, and using a separate buffer per plane. E.g. what would this look
> like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
> V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
> planes are also combined in a single buffer?
> 
> I would guess that the m union is set to 0 if the plane is part of the buffer
> defined in the previous plane?

The difference would be if m are equal or differ between planes, example:

For V4L2_PIX_FMT_YVU420:

    Y:
        plane_offset = 0
        m.dmabuf_fd = 3
    Cb:
        plane_offset = 300
        m.dmabuf_fd = 3
    Cr:
        plane_offset = 375
        m.dmabuf_fd = 3

For V4L2_PIX_FMT_YVU420M:

    Y:
        plane_offset = 0
        m.dmabuf_fd = 4
    Cb:
        plane_offset = 0
        m.dmabuf_fd = 5
    Cr:
        plane_offset = 0
        m.dmabuf_fd = 6


Does it make sense?

> 
>> +
>>  /**
>>   * struct v4l2_buffer - video buffer info
>>   * @index:	id number of the buffer
>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>>  	};
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_buffer - extended video buffer info
>> + * @index:	id number of the buffer
>> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>> + * @flags:	buffer informational flags
>> + * @field:	enum v4l2_field; field order of the image in the buffer
>> + * @timestamp:	frame timestamp
>> + * @sequence:	sequence count of this frame
>> + * @planes:	per-plane buffer information
>> + * @request_fd:	fd of the request that this buffer should use
>> + * @reserved:	extra space reserved for future fields, must be set to 0
>> + *
>> + * Contains data exchanged by application and driver using one of the Streaming
>> + * I/O methods.
>> + */
>> +struct v4l2_ext_buffer {
>> +	__u32 index;
>> +	__u32 type;
>> +	__u32 field;
>> +	__u32 sequence;
>> +	__u64 flags;
>> +	__u64 timestamp;
>> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>> +	__s32 request_fd;
>> +	__u32 reserved[9];
>> +};
> 
> Brainstorming:
> 
> Some ideas I have to make it easier to support mid stream resolution/colorimetry
> changes:
> 
> Adding width and height would support resolution changes (requires the use of
> CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
> information is provided here, then there are no race conditions.
> 
> Same for adding the colorimetry fields here, this too can change on the fly (esp.
> with HDMI), so reporting this information here avoids race conditions as well.

Right, do you think this is something we can discuss later in a different RFC?
So we can have a better view on how dynamic resolution change would be used?

We can add more reserved fields or maybe try to do something to what has been
discussed in about extensible system calls [1]

[1] https://lwn.net/Articles/830666/

> 
> And thirdly, I would like to have a __u64 boot_timestamp field containing the
> CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
> is that for m2m devices it is just copied and that for other devices it can have
> different meanings depending on the timestamp buffer flags.
> 
> There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
> to add support for this. That way you know exactly when the driver was finished with
> the buffer and that helps in detecting missed frames or instrumentation.

I don't mind adding it. Does it make sense to have both timestamp and boot_timestamp?

> 
>> +
>>  #ifndef __KERNEL__
>>  /**
>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
>>  	__u32			reserved[6];
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>> + * @index:	on return, index of the first created buffer
>> + * @count:	entry: number of requested buffers,
>> + *		return: number of created buffers
>> + * @memory:	enum v4l2_memory; buffer memory type
>> + * @capabilities: capabilities of this buffer type.
>> + * @format:	frame format, for which buffers are requested
>> + * @flags:	additional buffer management attributes (ignored unless the
>> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>> + *		and configured for MMAP streaming I/O).
>> + * @reserved:	extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_ext_create_buffers {
>> +	__u32				index;
>> +	__u32				count;
>> +	__u32				memory;
>> +	__u32				capabilities;
>> +	struct v4l2_ext_pix_format	format;
> 
> The reality is that the only field that is ever used in the original v4l2_format
> struct is sizeimage. So this can be replaced with:
> 
> 	__u32				plane_size[VIDEO_MAX_PLANES];
> 
> (the field name I picked is debatable, but you get the idea)
> 
> The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
> is needed for the current format. The original idea of using struct v4l2_format
> was that drivers would use the full format information to calculate the
> memory size, but that was just much too complicated to implement and nobody
> ever used that. Only the sizeimage field was ever used.

Right, I'll update this in next version, This should simplify things.


Thanks,
Helen

> 
>> +	__u32				flags;
>> +	__u32 reserved[5];
>> +};
>> +
>>  /*
>>   *	I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>   *
>> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
>>  #define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>>  #define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>>  #define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>> +#define VIDIOC_EXT_CREATE_BUFS	_IOWR('V', 107, struct v4l2_ext_create_buffers)
>> +#define VIDIOC_EXT_QUERYBUF	_IOWR('V', 108, struct v4l2_ext_buffer)
>> +#define VIDIOC_EXT_QBUF		_IOWR('V', 109, struct v4l2_ext_buffer)
>> +#define VIDIOC_EXT_DQBUF	_IOWR('V', 110, struct v4l2_ext_buffer)
>> +#define VIDIOC_EXT_PREPARE_BUF	_IOWR('V', 111, struct v4l2_ext_buffer)
>>  
>>  /* Reminder: when adding new ioctls please add support for them to
>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>
> 
> Regards,
> 
> 	Hans
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-11-23 15:08     ` Helen Koike
@ 2020-11-23 15:46       ` Tomasz Figa
  2020-11-23 17:40         ` Helen Koike
  0 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-11-23 15:46 UTC (permalink / raw)
  To: Helen Koike
  Cc: Hans Verkuil, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi Hans,
>
> Thank you for your review.
>
> On 9/9/20 9:27 AM, Hans Verkuil wrote:
> > Hi Helen,
> >
> > Again I'm just reviewing the uAPI.
> >
> > On 04/08/2020 21:29, Helen Koike wrote:
> >> From: Hans Verkuil <hans.verkuil@cisco.com>
> >>
> >> Those extended buffer ops have several purpose:
> >> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
> >>    the number of ns elapsed since 1970
> >> 2/ Unify single/multiplanar handling
> >> 3/ Add a new start offset field to each v4l2 plane buffer info struct
> >>    to support the case where a single buffer object is storing all
> >>    planes data, each one being placed at a different offset
> >>
> >> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> >> these new objects.
> >>
> >> The core takes care of converting new ioctls requests to old ones
> >> if the driver does not support the new hooks, and vice versa.
> >>
> >> Note that the timecode field is gone, since there doesn't seem to be
> >> in-kernel users. We can be added back in the reserved area if needed or
> >> use the Request API to collect more metadata information from the
> >> frame.
> >>
> >> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> >> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >> ---
> >> Changes in v5:
> >> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> >> - return mem_offset to struct v4l2_ext_plane
> >> - change sizes and reorder fields to avoid holes in the struct and make
> >>   it the same for 32 and 64 bits
> >>
> >> Changes in v4:
> >> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> >> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> >> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> >> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> >> I think we can add this later, so I removed it from this RFC to simplify it.
> >> - Remove num_planes field from struct v4l2_ext_buffer
> >> - Add flags field to struct v4l2_ext_create_buffers
> >> - Reformulate struct v4l2_ext_plane
> >> - Fix some bugs caught by v4l2-compliance
> >> - Rebased on top of media/master (post 5.8-rc1)
> >>
> >> Changes in v3:
> >> - Rebased on top of media/master (post 5.4-rc1)
> >>
> >> Changes in v2:
> >> - Add reserved space to v4l2_ext_buffer so that new fields can be added
> >>   later on
> >> ---
> >>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
> >>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
> >>  include/media/v4l2-ioctl.h           |  26 ++
> >>  include/uapi/linux/videodev2.h       |  90 +++++++
> >>  4 files changed, 476 insertions(+), 22 deletions(-)
> >>
> >
> > <snip>
> >
> >> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> >> index 7123c6a4d9569..334cafdd2be97 100644
> >> --- a/include/uapi/linux/videodev2.h
> >> +++ b/include/uapi/linux/videodev2.h
> >> @@ -996,6 +996,41 @@ struct v4l2_plane {
> >>      __u32                   reserved[11];
> >>  };
> >>
> >> +/**
> >> + * struct v4l2_ext_plane - extended plane buffer info
> >> + * @buffer_length:  size of the entire buffer in bytes, should fit
> >> + *                  @offset + @plane_length
> >> + * @plane_length:   size of the plane in bytes.
> >> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
> >> + *                  that should be passed to mmap() called on the video node.
> >> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> >> + *                  to this plane.
> >> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> >> + *                  associated with this plane.
> >> + * @offset:         offset in the memory buffer where the plane starts.
> >> + * @memory:         enum v4l2_memory; the method, in which the actual video
> >> + *                  data is passed
> >> + * @reserved:               extra space reserved for future fields, must be set to 0.
> >> + *
> >> + *
> >> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> >> + * can have one plane for Y, and another for interleaved CbCr components.
> >> + * Each plane can reside in a separate memory buffer, or even in
> >> + * a completely separate memory node (e.g. in embedded devices).
> >> + */
> >> +struct v4l2_ext_plane {
> >> +    __u32 buffer_length;
> >> +    __u32 plane_length;
> >> +    union {
> >> +            __u32 mem_offset;
> >> +            __u64 userptr;
> >> +            __s32 dmabuf_fd;
> >> +    } m;
> >> +    __u32 offset;
> >
> > I'd rename this plane_offset. I think some reordering would make this struct easier
> > to understand:
> >
> > struct v4l2_ext_plane {
> >       __u32 buffer_length;
> >       __u32 plane_offset;
> >       __u32 plane_length;
> >       __u32 memory;
> >       union {
> >               __u32 mem_offset;
> >               __u64 userptr;
> >               __s32 dmabuf_fd;
> >       } m;
> >       __u32 reserved[4];
> > };
> >
> >> +    __u32 memory;
> >> +    __u32 reserved[4];
> >> +};
>
> Ok, I'll apply this to the next version.
>
> >
> > What is not clear is how to tell the different between a single buffer containing
> > multiple planes, and using a separate buffer per plane. E.g. what would this look
> > like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
> > V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
> > planes are also combined in a single buffer?
> >
> > I would guess that the m union is set to 0 if the plane is part of the buffer
> > defined in the previous plane?
>
> The difference would be if m are equal or differ between planes, example:
>
> For V4L2_PIX_FMT_YVU420:
>
>     Y:
>         plane_offset = 0
>         m.dmabuf_fd = 3
>     Cb:
>         plane_offset = 300
>         m.dmabuf_fd = 3
>     Cr:
>         plane_offset = 375
>         m.dmabuf_fd = 3
>
> For V4L2_PIX_FMT_YVU420M:
>
>     Y:
>         plane_offset = 0
>         m.dmabuf_fd = 4
>     Cb:
>         plane_offset = 0
>         m.dmabuf_fd = 5
>     Cr:
>         plane_offset = 0
>         m.dmabuf_fd = 6
>
>
> Does it make sense?
>

Actually all the 3 file descriptors can still point to the same
buffer, because they might have been dup()ed. The kernel needs to
resolve the file descriptors into struct dma_buf and then check
whether it's one or more buffers.

In fact, dup()ed FD for each plane is quite a common case in other
APIs, e.g. EGL, but current V4L2 API can't handle it. In Chromium we
basically work around it by assuming that if we receive a buffer for a
V4L2 device that only supports non-M formats, then we can safely
ignore all but first FD. The new API gives the ability to handle the
case properly, with full validation by the kernel.

> >
> >> +
> >>  /**
> >>   * struct v4l2_buffer - video buffer info
> >>   * @index:  id number of the buffer
> >> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
> >>      };
> >>  };
> >>
> >> +/**
> >> + * struct v4l2_ext_buffer - extended video buffer info
> >> + * @index:  id number of the buffer
> >> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> >> + * @flags:  buffer informational flags
> >> + * @field:  enum v4l2_field; field order of the image in the buffer
> >> + * @timestamp:      frame timestamp
> >> + * @sequence:       sequence count of this frame
> >> + * @planes: per-plane buffer information
> >> + * @request_fd:     fd of the request that this buffer should use
> >> + * @reserved:       extra space reserved for future fields, must be set to 0
> >> + *
> >> + * Contains data exchanged by application and driver using one of the Streaming
> >> + * I/O methods.
> >> + */
> >> +struct v4l2_ext_buffer {
> >> +    __u32 index;
> >> +    __u32 type;
> >> +    __u32 field;
> >> +    __u32 sequence;
> >> +    __u64 flags;
> >> +    __u64 timestamp;
> >> +    struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> >> +    __s32 request_fd;
> >> +    __u32 reserved[9];
> >> +};
> >
> > Brainstorming:
> >
> > Some ideas I have to make it easier to support mid stream resolution/colorimetry
> > changes:
> >
> > Adding width and height would support resolution changes (requires the use of
> > CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
> > information is provided here, then there are no race conditions.
> >
> > Same for adding the colorimetry fields here, this too can change on the fly (esp.
> > with HDMI), so reporting this information here avoids race conditions as well.
>
> Right, do you think this is something we can discuss later in a different RFC?
> So we can have a better view on how dynamic resolution change would be used?
>
> We can add more reserved fields or maybe try to do something to what has been
> discussed in about extensible system calls [1]
>
> [1] https://lwn.net/Articles/830666/
>
> >
> > And thirdly, I would like to have a __u64 boot_timestamp field containing the
> > CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
> > is that for m2m devices it is just copied and that for other devices it can have
> > different meanings depending on the timestamp buffer flags.
> >
> > There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
> > to add support for this. That way you know exactly when the driver was finished with
> > the buffer and that helps in detecting missed frames or instrumentation.
>
> I don't mind adding it. Does it make sense to have both timestamp and boot_timestamp?
>

I think this is quite independent from the ext API work. AFAIR there
was an RFC to request the timestamp source from the userspace by the
flags field in QBUF, which would work with the existing API as well,
or it wasn't posted in the end?

> >
> >> +
> >>  #ifndef __KERNEL__
> >>  /**
> >>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
> >> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
> >>      __u32                   reserved[6];
> >>  };
> >>
> >> +/**
> >> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
> >> + * @index:  on return, index of the first created buffer
> >> + * @count:  entry: number of requested buffers,
> >> + *          return: number of created buffers
> >> + * @memory: enum v4l2_memory; buffer memory type
> >> + * @capabilities: capabilities of this buffer type.
> >> + * @format: frame format, for which buffers are requested
> >> + * @flags:  additional buffer management attributes (ignored unless the
> >> + *          queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
> >> + *          and configured for MMAP streaming I/O).
> >> + * @reserved:       extra space reserved for future fields, must be set to 0
> >> + */
> >> +struct v4l2_ext_create_buffers {
> >> +    __u32                           index;
> >> +    __u32                           count;
> >> +    __u32                           memory;
> >> +    __u32                           capabilities;
> >> +    struct v4l2_ext_pix_format      format;
> >
> > The reality is that the only field that is ever used in the original v4l2_format
> > struct is sizeimage. So this can be replaced with:
> >
> >       __u32                           plane_size[VIDEO_MAX_PLANES];
> >
> > (the field name I picked is debatable, but you get the idea)
> >
> > The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
> > is needed for the current format. The original idea of using struct v4l2_format
> > was that drivers would use the full format information to calculate the
> > memory size, but that was just much too complicated to implement and nobody
> > ever used that. Only the sizeimage field was ever used.
>
> Right, I'll update this in next version, This should simplify things.
>

I think this might need a bit more discussion. How would the userspace
know what size is enough for the desired resolution? The hardware
and/or drivers often have various alignment/padding restrictions,
which might not be easy to guess for the userspace.

Also I don't quite understand what's so complicated in handling the
full format, or at least the most important parts of it. The
implementation of TRY_FMT/S_FMT, which exists in every driver, should
already be able to calculate the right plane sizes.

Best regards,
Tomasz

>
> Thanks,
> Helen
>
> >
> >> +    __u32                           flags;
> >> +    __u32 reserved[5];
> >> +};
> >> +
> >>  /*
> >>   *  I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
> >>   *
> >> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
> >>  #define VIDIOC_G_EXT_PIX_FMT        _IOWR('V', 104, struct v4l2_ext_pix_format)
> >>  #define VIDIOC_S_EXT_PIX_FMT        _IOWR('V', 105, struct v4l2_ext_pix_format)
> >>  #define VIDIOC_TRY_EXT_PIX_FMT      _IOWR('V', 106, struct v4l2_ext_pix_format)
> >> +#define VIDIOC_EXT_CREATE_BUFS      _IOWR('V', 107, struct v4l2_ext_create_buffers)
> >> +#define VIDIOC_EXT_QUERYBUF _IOWR('V', 108, struct v4l2_ext_buffer)
> >> +#define VIDIOC_EXT_QBUF             _IOWR('V', 109, struct v4l2_ext_buffer)
> >> +#define VIDIOC_EXT_DQBUF    _IOWR('V', 110, struct v4l2_ext_buffer)
> >> +#define VIDIOC_EXT_PREPARE_BUF      _IOWR('V', 111, struct v4l2_ext_buffer)
> >>
> >>  /* Reminder: when adding new ioctls please add support for them to
> >>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
> >>
> >
> > Regards,
> >
> >       Hans
> >

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-11-23 15:46       ` Tomasz Figa
@ 2020-11-23 17:40         ` Helen Koike
  2020-12-03 15:11           ` Hans Verkuil
  0 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-11-23 17:40 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Hans Verkuil, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov



On 11/23/20 12:46 PM, Tomasz Figa wrote:
> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
>>
>> Hi Hans,
>>
>> Thank you for your review.
>>
>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
>>> Hi Helen,
>>>
>>> Again I'm just reviewing the uAPI.
>>>
>>> On 04/08/2020 21:29, Helen Koike wrote:
>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>
>>>> Those extended buffer ops have several purpose:
>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>    the number of ns elapsed since 1970
>>>> 2/ Unify single/multiplanar handling
>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>    to support the case where a single buffer object is storing all
>>>>    planes data, each one being placed at a different offset
>>>>
>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>> these new objects.
>>>>
>>>> The core takes care of converting new ioctls requests to old ones
>>>> if the driver does not support the new hooks, and vice versa.
>>>>
>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>> use the Request API to collect more metadata information from the
>>>> frame.
>>>>
>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>> ---
>>>> Changes in v5:
>>>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
>>>> - return mem_offset to struct v4l2_ext_plane
>>>> - change sizes and reorder fields to avoid holes in the struct and make
>>>>   it the same for 32 and 64 bits
>>>>
>>>> Changes in v4:
>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>> - Reformulate struct v4l2_ext_plane
>>>> - Fix some bugs caught by v4l2-compliance
>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>
>>>> Changes in v3:
>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>
>>>> Changes in v2:
>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>   later on
>>>> ---
>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>  include/uapi/linux/videodev2.h       |  90 +++++++
>>>>  4 files changed, 476 insertions(+), 22 deletions(-)
>>>>
>>>
>>> <snip>
>>>
>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>>> index 7123c6a4d9569..334cafdd2be97 100644
>>>> --- a/include/uapi/linux/videodev2.h
>>>> +++ b/include/uapi/linux/videodev2.h
>>>> @@ -996,6 +996,41 @@ struct v4l2_plane {
>>>>      __u32                   reserved[11];
>>>>  };
>>>>
>>>> +/**
>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>> + * @buffer_length:  size of the entire buffer in bytes, should fit
>>>> + *                  @offset + @plane_length
>>>> + * @plane_length:   size of the plane in bytes.
>>>> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
>>>> + *                  that should be passed to mmap() called on the video node.
>>>> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>> + *                  to this plane.
>>>> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>> + *                  associated with this plane.
>>>> + * @offset:         offset in the memory buffer where the plane starts.
>>>> + * @memory:         enum v4l2_memory; the method, in which the actual video
>>>> + *                  data is passed
>>>> + * @reserved:               extra space reserved for future fields, must be set to 0.
>>>> + *
>>>> + *
>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>> + */
>>>> +struct v4l2_ext_plane {
>>>> +    __u32 buffer_length;
>>>> +    __u32 plane_length;
>>>> +    union {
>>>> +            __u32 mem_offset;
>>>> +            __u64 userptr;
>>>> +            __s32 dmabuf_fd;
>>>> +    } m;
>>>> +    __u32 offset;
>>>
>>> I'd rename this plane_offset. I think some reordering would make this struct easier
>>> to understand:
>>>
>>> struct v4l2_ext_plane {
>>>       __u32 buffer_length;
>>>       __u32 plane_offset;
>>>       __u32 plane_length;
>>>       __u32 memory;
>>>       union {
>>>               __u32 mem_offset;
>>>               __u64 userptr;
>>>               __s32 dmabuf_fd;
>>>       } m;
>>>       __u32 reserved[4];
>>> };
>>>
>>>> +    __u32 memory;
>>>> +    __u32 reserved[4];
>>>> +};
>>
>> Ok, I'll apply this to the next version.
>>
>>>
>>> What is not clear is how to tell the different between a single buffer containing
>>> multiple planes, and using a separate buffer per plane. E.g. what would this look
>>> like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
>>> V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
>>> planes are also combined in a single buffer?
>>>
>>> I would guess that the m union is set to 0 if the plane is part of the buffer
>>> defined in the previous plane?
>>
>> The difference would be if m are equal or differ between planes, example:
>>
>> For V4L2_PIX_FMT_YVU420:
>>
>>     Y:
>>         plane_offset = 0
>>         m.dmabuf_fd = 3
>>     Cb:
>>         plane_offset = 300
>>         m.dmabuf_fd = 3
>>     Cr:
>>         plane_offset = 375
>>         m.dmabuf_fd = 3
>>
>> For V4L2_PIX_FMT_YVU420M:
>>
>>     Y:
>>         plane_offset = 0
>>         m.dmabuf_fd = 4
>>     Cb:
>>         plane_offset = 0
>>         m.dmabuf_fd = 5
>>     Cr:
>>         plane_offset = 0
>>         m.dmabuf_fd = 6
>>
>>
>> Does it make sense?
>>
> 
> Actually all the 3 file descriptors can still point to the same
> buffer, because they might have been dup()ed. The kernel needs to
> resolve the file descriptors into struct dma_buf and then check
> whether it's one or more buffers.

Right, thanks for this.

> 
> In fact, dup()ed FD for each plane is quite a common case in other
> APIs, e.g. EGL, but current V4L2 API can't handle it. In Chromium we
> basically work around it by assuming that if we receive a buffer for a
> V4L2 device that only supports non-M formats, then we can safely
> ignore all but first FD. The new API gives the ability to handle the
> case properly, with full validation by the kernel.
> 
>>>
>>>> +
>>>>  /**
>>>>   * struct v4l2_buffer - video buffer info
>>>>   * @index:  id number of the buffer
>>>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>>>>      };
>>>>  };
>>>>
>>>> +/**
>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>> + * @index:  id number of the buffer
>>>> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>> + * @flags:  buffer informational flags
>>>> + * @field:  enum v4l2_field; field order of the image in the buffer
>>>> + * @timestamp:      frame timestamp
>>>> + * @sequence:       sequence count of this frame
>>>> + * @planes: per-plane buffer information
>>>> + * @request_fd:     fd of the request that this buffer should use
>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>> + *
>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>> + * I/O methods.
>>>> + */
>>>> +struct v4l2_ext_buffer {
>>>> +    __u32 index;
>>>> +    __u32 type;
>>>> +    __u32 field;
>>>> +    __u32 sequence;
>>>> +    __u64 flags;
>>>> +    __u64 timestamp;
>>>> +    struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>>> +    __s32 request_fd;
>>>> +    __u32 reserved[9];
>>>> +};
>>>
>>> Brainstorming:
>>>
>>> Some ideas I have to make it easier to support mid stream resolution/colorimetry
>>> changes:
>>>
>>> Adding width and height would support resolution changes (requires the use of
>>> CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
>>> information is provided here, then there are no race conditions.
>>>
>>> Same for adding the colorimetry fields here, this too can change on the fly (esp.
>>> with HDMI), so reporting this information here avoids race conditions as well.
>>
>> Right, do you think this is something we can discuss later in a different RFC?
>> So we can have a better view on how dynamic resolution change would be used?
>>
>> We can add more reserved fields or maybe try to do something to what has been
>> discussed in about extensible system calls [1]
>>
>> [1] https://lwn.net/Articles/830666/
>>
>>>
>>> And thirdly, I would like to have a __u64 boot_timestamp field containing the
>>> CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
>>> is that for m2m devices it is just copied and that for other devices it can have
>>> different meanings depending on the timestamp buffer flags.
>>>
>>> There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
>>> to add support for this. That way you know exactly when the driver was finished with
>>> the buffer and that helps in detecting missed frames or instrumentation.
>>
>> I don't mind adding it. Does it make sense to have both timestamp and boot_timestamp?
>>
> 
> I think this is quite independent from the ext API work. AFAIR there
> was an RFC to request the timestamp source from the userspace by the
> flags field in QBUF, which would work with the existing API as well,
> or it wasn't posted in the end?

I was recalling the discussions we had regarding this:

1.
    This first attempt in the uvc driver is to use a specific kernel parameter for that case:
    https://patchwork.kernel.org/patch/10644887/
    The conclusion that the support should be in the core API and not driver specific.

2.
    Then an attempt to add global v4l2 support was sent with the Mediatek patch series:
    https://patchwork.linuxtv.org/patch/60878/
    The major problem is that clock type should be something selectable by userspace, and
    not pre-defined by the driver.

3.
    Another idea was to use the 'flags' field in the structs v4l2_requestbuffers and
    v4l2_create_buffers.
    But this field was removed in
    129134e5415d ("media: media/v4l2: remove V4L2_FLAG_MEMORY_NON_CONSISTENT flag")
    The major concern with this approach was with the uAPI, since it doesn't make much
    sense to select a clock when creating buffers.

4.
    Another suggestion by Nicolas Dufresne was to add this as a menu control so that userspace
    can choose the clock for the timestamps from a given list, the enum in the list can also match
    the clocks ids.
    We would need to add a new buf flag in struct v4l2_buffer, like V4L2_BUF_FLAG_TIMESTAMP_OTHER,
    which would be "as specified through controls ...."


So my current question is, should we have both __u32 timestamp and __u32 boottimestamp?
Or should we have a mechanism that allows switching from one to the other and use
a single field? And if this mechanism should be implemented in both APIs? Can this be
defined later?


Please, let me know your thoughts.

Thanks,
Helen

> 
>>>
>>>> +
>>>>  #ifndef __KERNEL__
>>>>  /**
>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>>>> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
>>>>      __u32                   reserved[6];
>>>>  };
>>>>
>>>> +/**
>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>>>> + * @index:  on return, index of the first created buffer
>>>> + * @count:  entry: number of requested buffers,
>>>> + *          return: number of created buffers
>>>> + * @memory: enum v4l2_memory; buffer memory type
>>>> + * @capabilities: capabilities of this buffer type.
>>>> + * @format: frame format, for which buffers are requested
>>>> + * @flags:  additional buffer management attributes (ignored unless the
>>>> + *          queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>>>> + *          and configured for MMAP streaming I/O).
>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>> + */
>>>> +struct v4l2_ext_create_buffers {
>>>> +    __u32                           index;
>>>> +    __u32                           count;
>>>> +    __u32                           memory;
>>>> +    __u32                           capabilities;
>>>> +    struct v4l2_ext_pix_format      format;
>>>
>>> The reality is that the only field that is ever used in the original v4l2_format
>>> struct is sizeimage. So this can be replaced with:
>>>
>>>       __u32                           plane_size[VIDEO_MAX_PLANES];
>>>
>>> (the field name I picked is debatable, but you get the idea)
>>>
>>> The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
>>> is needed for the current format. The original idea of using struct v4l2_format
>>> was that drivers would use the full format information to calculate the
>>> memory size, but that was just much too complicated to implement and nobody
>>> ever used that. Only the sizeimage field was ever used.
>>
>> Right, I'll update this in next version, This should simplify things.
>>
> 
> I think this might need a bit more discussion. How would the userspace
> know what size is enough for the desired resolution? The hardware
> and/or drivers often have various alignment/padding restrictions,
> which might not be easy to guess for the userspace.
> 
> Also I don't quite understand what's so complicated in handling the
> full format, or at least the most important parts of it. The
> implementation of TRY_FMT/S_FMT, which exists in every driver, should
> already be able to calculate the right plane sizes.
> 
> Best regards,
> Tomasz
> 
>>
>> Thanks,
>> Helen
>>
>>>
>>>> +    __u32                           flags;
>>>> +    __u32 reserved[5];
>>>> +};
>>>> +
>>>>  /*
>>>>   *  I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>>>   *
>>>> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
>>>>  #define VIDIOC_G_EXT_PIX_FMT        _IOWR('V', 104, struct v4l2_ext_pix_format)
>>>>  #define VIDIOC_S_EXT_PIX_FMT        _IOWR('V', 105, struct v4l2_ext_pix_format)
>>>>  #define VIDIOC_TRY_EXT_PIX_FMT      _IOWR('V', 106, struct v4l2_ext_pix_format)
>>>> +#define VIDIOC_EXT_CREATE_BUFS      _IOWR('V', 107, struct v4l2_ext_create_buffers)
>>>> +#define VIDIOC_EXT_QUERYBUF _IOWR('V', 108, struct v4l2_ext_buffer)
>>>> +#define VIDIOC_EXT_QBUF             _IOWR('V', 109, struct v4l2_ext_buffer)
>>>> +#define VIDIOC_EXT_DQBUF    _IOWR('V', 110, struct v4l2_ext_buffer)
>>>> +#define VIDIOC_EXT_PREPARE_BUF      _IOWR('V', 111, struct v4l2_ext_buffer)
>>>>
>>>>  /* Reminder: when adding new ioctls please add support for them to
>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>>>
>>>
>>> Regards,
>>>
>>>       Hans
>>>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-11-20 11:14   ` Tomasz Figa
@ 2020-11-23 20:33     ` Helen Koike
  2020-12-14 10:36       ` Tomasz Figa
  0 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-11-23 20:33 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, Boris Brezillon, hiroh, nicolas, Brian.Starkey,
	kernel, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Hi Tomasz,


On 11/20/20 8:14 AM, Tomasz Figa wrote:
> Hi Helen,
> 
> On Tue, Aug 04, 2020 at 04:29:34PM -0300, Helen Koike wrote:
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> Those extended buffer ops have several purpose:
>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>    the number of ns elapsed since 1970
>> 2/ Unify single/multiplanar handling
>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>    to support the case where a single buffer object is storing all
>>    planes data, each one being placed at a different offset
>>
>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>> these new objects.
>>
>> The core takes care of converting new ioctls requests to old ones
>> if the driver does not support the new hooks, and vice versa.
>>
>> Note that the timecode field is gone, since there doesn't seem to be
>> in-kernel users. We can be added back in the reserved area if needed or
>> use the Request API to collect more metadata information from the
>> frame.
>>
> 
> Thanks for the patch. Please see my comments inline.

Thank you for your detailed review, please see my comments below.

> 
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> ---
>> Changes in v5:
>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
>> - return mem_offset to struct v4l2_ext_plane
>> - change sizes and reorder fields to avoid holes in the struct and make
>>   it the same for 32 and 64 bits
>>
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>> I think we can add this later, so I removed it from this RFC to simplify it.
>> - Remove num_planes field from struct v4l2_ext_buffer
>> - Add flags field to struct v4l2_ext_create_buffers
>> - Reformulate struct v4l2_ext_plane
>> - Fix some bugs caught by v4l2-compliance
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>   later on
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>>  include/media/v4l2-ioctl.h           |  26 ++
>>  include/uapi/linux/videodev2.h       |  90 +++++++
>>  4 files changed, 476 insertions(+), 22 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>> index e1829906bc086..cb21ee8eb075c 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -720,15 +720,34 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>  		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_sdr_out);
>>  	}
>>  
>> +	if (is_vid || is_tch) {
>> +		/* ioctls valid for video and touch */
>> +		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
>> +			set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
>> +		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
>> +			set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
>> +		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
>> +			set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
>> +		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
>> +			set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
>> +		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
>> +			set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
> 
> nit: Could we stick to the SET_VALID_IOCTL() macro and just call it twice,
> once for the new and once for the legacy callback?

Ack.

> 
>> +	}
>> +
>>  	if (is_vid || is_vbi || is_sdr || is_tch || is_meta) {
>>  		/* ioctls valid for video, vbi, sdr, touch and metadata */
>>  		SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
>> -		SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
>> -		SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
>>  		SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
>> -		SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
>> -		SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
>> -		SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
>> +		if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
>> +			set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
>> +		if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
>> +			set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
>> +		if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
>> +			set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
>> +		if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
>> +			set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
>> +		if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
>> +			set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
> 
> Is it valid to check the new callbacks for devices that the new API is not
> valid for (e.g. vbi)? Perhaps we could call SET_VALID_IOCTL(ops, <ioctl>,
> vidioc_ext_*) in the upper if added in this patch and keep the code above
> as is?

Just to be clear, the only valid type should be VFL_TYPE_VIDEO right?
Ext but API won't support touch devices for instance, right?


> 
>>  		SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
>>  		SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
>>  	}
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index 14a0def50f8ea..7ecdd9cc1bf48 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -527,6 +527,26 @@ static void v4l_print_buffer(const void *arg, bool write_only)
>>  			tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
>>  }
>>  
>> +static void v4l_print_ext_buffer(const void *arg, bool write_only)
>> +{
>> +	const struct v4l2_ext_buffer *e = arg;
>> +	const struct v4l2_ext_plane *plane;
>> +	unsigned int i;
>> +
>> +	pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d\n",
>> +		e->timestamp, e->index, prt_names(e->type, v4l2_type_names),
>> +		e->flags, prt_names(e->field, v4l2_field_names), e->sequence);
> 
> Should we also print the request FD?

Yes, I'll update this for next version.

> 
>> +
>> +	for (i = 0; i < VIDEO_MAX_PLANES &&
>> +		    e->planes[i].buffer_length; i++) {
>> +		plane = &e->planes[i];
>> +		pr_debug("plane %d: buffer_length=%d, plane_length=%d offset=0x%08x, memory=%s\n",
>> +			 i, plane->buffer_length, plane->plane_length,
>> +			 plane->offset,
>> +			 prt_names(plane->memory, v4l2_memory_names));
> 
> Should we also print mem_offset/userptr/dmabuf_fd?

I see they are not printed by v4l_print_buffer(), since these fields are in an
union, the value of two of them will be invalid (I wonder if this can bring
confusion).
I also wondered if printing them can't cause a security issue.

But I can add those prints if you think it make sense.

> 
>> +	}
>> +}
>> +
>>  static void v4l_print_exportbuffer(const void *arg, bool write_only)
>>  {
>>  	const struct v4l2_exportbuffer *p = arg;
>> @@ -546,6 +566,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
>>  	v4l_print_format(&p->format, write_only);
>>  }
>>  
>> +static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
>> +{
>> +	const struct v4l2_ext_create_buffers *p = arg;
>> +
>> +	pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
>> +		prt_names(p->memory, v4l2_memory_names));
>> +	v4l_print_ext_pix_format(&p->format, write_only);
> 
> Should we also print capabilities and flags?

I just saw these prints are called after the ioctl handler, and not before,
to I guess it make sense.

It is not printed by v4l_print_create_buffers(), I think we can add in both then.

> 
>> +}
>> +
>>  static void v4l_print_streamparm(const void *arg, bool write_only)
>>  {
>>  	const struct v4l2_streamparm *p = arg;
>> @@ -1220,6 +1249,143 @@ int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>>  }
>>  EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>>  
>> +/*
>> + * If mplane_cap is true, b->m.planes should have a valid pointer of a
>> + * struct v4l2_plane array, and b->length with its size
>> + */
>> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
>> +			      struct v4l2_buffer *b, bool mplane_cap)
>> +{
>> +	unsigned int planes_array_size = b->length;
>> +	struct v4l2_plane *planes = b->m.planes;
>> +	u64 nsecs;
>> +
>> +	if (!mplane_cap && e->planes[1].buffer_length != 0)
>> +		return -EINVAL;
>> +
>> +	memset(b, 0, sizeof(*b));
>> +
>> +	b->index = e->index;
>> +	b->flags = e->flags;
>> +	b->field = e->field;
>> +	b->sequence = e->sequence;
>> +	b->memory = e->planes[0].memory;
>> +	b->request_fd = e->request_fd;
>> +	b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
>> +	b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
>> +
>> +	if (mplane_cap) {
>> +		unsigned int i;
>> +
>> +		if (!planes || !planes_array_size)
>> +			return -EINVAL;
>> +
>> +		b->m.planes = planes;
> 
> planes was initialized to b->m.planes at declaration time. Should we
> perhaps move its declaration to within this block to make it more clear and
> remove this assignment?

The variable "planes" is saving the pointer of b->m.planes before we do
memset(b, 0, sizeof(*b)), so I can reassing it back if it make sense.

I can add a comment to make this more clear.

> 
>> +
>> +		if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> +			b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>> +		else
>> +			b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
>> +
>> +		for (i = 0; i < VIDEO_MAX_PLANES && i < planes_array_size &&
>> +			    e->planes[i].buffer_length; i++) {
>> +
>> +			if (e->planes[0].memory != e->planes[i].memory)
>> +				return -EINVAL;
>> +
>> +			if (e->planes[i].offset)
>> +				return -EINVAL;
> 
> Is it really invalid to have a non-zero offset? Shouldn't the data_offset
> field of the legacy struct be populated instead, in the cases where it was
> defined to be valid?

My understanding of data_offset, is that it is used when the hardware can
write/read a header to/from the buffer.

But this doesn't seem to be used by any driver, so there is an attempt to
repourpose it here:

    https://patchwork.linuxtv.org/project/linux-media/patch/1429040689-23808-2-git-send-email-laurent.pinchart@ideasonboard.com/

But this wans't merged.

So in the current API, there is no way to specify an offset in the buffer.

I guess we can repurpose data_offset first, what do you think?

> 
>> +
>> +			memset(&b->m.planes[i], 0, sizeof(b->m.planes[i]));
>> +
>> +			if (b->memory == V4L2_MEMORY_MMAP)
>> +				b->m.planes[i].m.mem_offset = e->planes[i].m.mem_offset;
>> +			else if (b->memory == V4L2_MEMORY_DMABUF)
>> +				b->m.planes[i].m.fd = e->planes[i].m.dmabuf_fd;
>> +			else
>> +				b->m.planes[i].m.userptr = e->planes[i].m.userptr;
>> +
>> +			b->m.planes[i].bytesused = e->planes[i].plane_length;
> 
> I might be getting the meaning of plane_length wrong, but doesn't this
> depend on the direction? If the userspace gives a CAPTURE buffer, it would
> have bytesused = 0, but if the kernel returns it, it would have bytesused =
> <size of the payload>.

You are right, it depends on the direction, thanks for catching this.

Also, in the Ext api, we don't have the <size of the payload>, so I'm using the
plane_length instead, I'm not sure if there is a better way.

> 
>> +			b->m.planes[i].length = e->planes[i].buffer_length;
>> +		}
>> +		/* In multi-planar, length contain the number of planes */
>> +		b->length = i;
>> +	} else {
>> +		b->type = e->type;
>> +		b->bytesused = e->planes[0].plane_length;
>> +		b->length = e->planes[0].buffer_length;
>> +
>> +		if (e->planes[0].offset)
>> +			return -EINVAL;
> 
> Ditto.

Ack.

> 
>> +
>> +		if (b->memory == V4L2_MEMORY_MMAP)
>> +			b->m.offset = e->planes[0].m.mem_offset;
>> +		else if (b->memory == V4L2_MEMORY_DMABUF)
>> +			b->m.fd = e->planes[0].m.dmabuf_fd;
>> +		else
>> +			b->m.userptr = e->planes[0].m.userptr;
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
>> +
>> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
>> +			      struct v4l2_ext_buffer *e)
>> +{
>> +	memset(e, 0, sizeof(*e));
>> +
>> +	e->index = b->index;
>> +	e->flags = b->flags;
>> +	e->field = b->field;
>> +	e->sequence = b->sequence;
>> +	e->request_fd = b->request_fd;
>> +	e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
>> +		b->timestamp.tv_usec * NSEC_PER_USEC;
>> +	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
>> +		unsigned int i;
>> +
>> +		if (!b->m.planes)
>> +			return -EINVAL;
>> +
>> +		if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +		else
>> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +
>> +		/* In multi-planar, length contain the number of planes */
>> +		for (i = 0; i < b->length; i++) {
> 
> The design of the new struct implies that the planes describe color planes
> and not memory planes, so this code is incorrect, because for non-M format
> variants it would fill in only the first plane of the new struct.

Yes, as discussed in 1/7, the handling of planes is wrong, I'll correct
this for next version.

> 
>> +			if (b->memory == V4L2_MEMORY_MMAP)
>> +				e->planes[i].m.mem_offset = b->m.planes[i].m.mem_offset;
>> +			else if (b->memory == V4L2_MEMORY_DMABUF)
>> +				e->planes[i].m.dmabuf_fd = b->m.planes[i].m.fd;
>> +			else
>> +				e->planes[i].m.userptr = b->m.planes[i].m.userptr;
>> +
>> +			e->planes[i].memory = b->memory;
>> +			e->planes[i].buffer_length = b->m.planes[i].length;
>> +			e->planes[i].plane_length = b->m.planes[i].bytesused;
>> +			if (b->m.planes[i].data_offset)
>> +				pr_warn("Ignoring data_offset value %d\n",
>> +					b->m.planes[i].data_offset);
> 
> Why? As per my comment above, there are valid use cases defined in the spec.

Please see my comment about about data_offset.

> 
>> +		}
>> +	} else {
>> +		e->type = b->type;
>> +		e->planes[0].memory = b->memory;
>> +		e->planes[0].plane_length = b->bytesused;
>> +		e->planes[0].buffer_length = b->length;
>> +		if (b->memory == V4L2_MEMORY_MMAP)
>> +			e->planes[0].m.mem_offset = b->m.offset;
>> +		else if (b->memory == V4L2_MEMORY_DMABUF)
>> +			e->planes[0].m.dmabuf_fd = b->m.fd;
>> +		else
>> +			e->planes[0].m.userptr = b->m.userptr;
> 
> Similar to the MULTIPLANAR case, we should fill in the planes[] entries
> corresponding to the number of color planes of the format, e.g. 2 for NV12.

Ack.

> 
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
>> +
>>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -2473,31 +2639,112 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
>>  	return ops->vidioc_reqbufs(file, fh, p);
>>  }
>>  
>> +static int v4l_do_buf_op(int (*op)(struct file *, void *,
>> +				   struct v4l2_buffer *),
>> +			 int (*ext_op)(struct file *, void *,
>> +				       struct v4l2_ext_buffer *),
>> +			 struct file *file, void *fh, struct v4l2_buffer *b)
>> +{
>> +	struct v4l2_ext_buffer e;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, b->type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	if (op)
>> +		return op(file, fh, b);
>> +
>> +	ret = v4l2_buffer_to_ext_buffer(b, &e);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = ext_op(file, fh, &e);
>> +	if (ret)
>> +		return ret;
>> +
>> +	v4l2_ext_buffer_to_buffer(&e, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
>> +	return 0;
>> +}
>> +
>> +static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
>> +				       struct v4l2_buffer *),
>> +			     int (*ext_op)(struct file *, void *,
>> +					   struct v4l2_ext_buffer *),
>> +			     struct file *file, void *fh,
>> +			     struct v4l2_ext_buffer *e)
>> +{
>> +	struct video_device *vdev = video_devdata(file);
>> +	struct v4l2_plane planes[VIDEO_MAX_PLANES];
>> +	struct v4l2_buffer b;
>> +	bool mplane_cap;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, e->type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	if (ext_op)
>> +		return ext_op(file, fh, e);
>> +
>> +	mplane_cap = !!(vdev->device_caps &
>> +			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +			 V4L2_CAP_VIDEO_M2M_MPLANE));
>> +	b.m.planes = planes;
>> +	b.length = VIDEO_MAX_PLANES;
>> +	ret = v4l2_ext_buffer_to_buffer(e, &b, mplane_cap);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = op(file, fh, &b);
>> +	if (ret)
>> +		return ret;
>> +
>> +	v4l2_buffer_to_ext_buffer(&b, e);
>> +	return 0;
>> +}
>> +
>>  static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
>> -				struct file *file, void *fh, void *arg)
>> +			struct file *file, void *fh, void *arg)
>>  {
>> -	struct v4l2_buffer *p = arg;
>> -	int ret = check_fmt(file, p->type);
>> +	return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
>> +			     file, fh, arg);
>> +}
>>  
>> -	return ret ? ret : ops->vidioc_querybuf(file, fh, p);
>> +static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
>> +			    struct file *file, void *fh, void *arg)
>> +{
>> +	return v4l_do_ext_buf_op(ops->vidioc_querybuf,
>> +				 ops->vidioc_ext_querybuf, file, fh, arg);
>>  }
>>  
>>  static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
>> -				struct file *file, void *fh, void *arg)
>> +		    struct file *file, void *fh, void *arg)
>>  {
>> -	struct v4l2_buffer *p = arg;
>> -	int ret = check_fmt(file, p->type);
>> +	return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
>> +			     file, fh, arg);
>> +}
>>  
>> -	return ret ? ret : ops->vidioc_qbuf(file, fh, p);
>> +static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
>> +			struct file *file, void *fh, void *arg)
>> +{
>> +	return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
>> +				 file, fh, arg);
>>  }
>>  
>>  static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
>> -				struct file *file, void *fh, void *arg)
>> +		     struct file *file, void *fh, void *arg)
>>  {
>> -	struct v4l2_buffer *p = arg;
>> -	int ret = check_fmt(file, p->type);
>> +	return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
>> +			     file, fh, arg);
>> +}
>>  
>> -	return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
>> +static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
>> +			 struct file *file, void *fh, void *arg)
>> +{
>> +	return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
>> +				 file, fh, arg);
>>  }
>>  
>>  static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>> @@ -2513,7 +2760,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>>  
>>  	v4l_sanitize_format(&create->format);
>>  
>> -	ret = ops->vidioc_create_bufs(file, fh, create);
>> +	if (ops->vidioc_create_bufs) {
>> +		ret = ops->vidioc_create_bufs(file, fh, create);
>> +	} else {
>> +		struct v4l2_ext_create_buffers ecreate = {
>> +			.count = create->count,
>> +			.memory = create->memory,
>> +		};
>> +
>> +		ret = v4l2_format_to_ext_pix_format(&create->format,
>> +						    &ecreate.format, true);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
>> +		if (ret)
>> +			return ret;
>> +
>> +		create->index = ecreate.index;
>> +		create->count = ecreate.count;
>> +		create->capabilities = ecreate.capabilities;
>> +	}
>>  
>>  	if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>>  	    create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
>> @@ -2522,13 +2789,60 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>>  	return ret;
>>  }
>>  
>> +static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
>> +			       struct file *file, void *fh, void *arg)
>> +{
>> +	struct v4l2_ext_create_buffers *ecreate = arg;
>> +	struct video_device *vdev = video_devdata(file);
>> +	struct v4l2_create_buffers create = {
>> +		.count = ecreate->count,
>> +		.memory = ecreate->memory,
>> +		.flags = ecreate->flags,
>> +	};
>> +	bool mplane_cap;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ecreate->format.type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	if (ops->vidioc_ext_create_bufs)
>> +		return ops->vidioc_ext_create_bufs(file, fh, ecreate);
>> +
>> +	mplane_cap = !!(vdev->device_caps &
>> +			(V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +			 V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +			 V4L2_CAP_VIDEO_M2M_MPLANE));
>> +	ret = v4l2_ext_pix_format_to_format(&ecreate->format,
>> +					    &create.format, mplane_cap, true);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = v4l_create_bufs(ops, file, fh, &create);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ecreate->index = create.index;
>> +	ecreate->count = create.count;
>> +	ecreate->capabilities = create.capabilities;
>> +
>> +	return 0;
>> +}
>> +
>>  static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
>> -				struct file *file, void *fh, void *arg)
>> +			   struct file *file, void *fh, void *arg)
>>  {
>> -	struct v4l2_buffer *b = arg;
>> -	int ret = check_fmt(file, b->type);
>> +	return v4l_do_buf_op(ops->vidioc_prepare_buf,
>> +			     ops->vidioc_ext_prepare_buf,
>> +			     file, fh, arg);
>> +}
>>  
>> -	return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
>> +static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
>> +			       struct file *file, void *fh, void *arg)
>> +{
>> +	return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
>> +				 ops->vidioc_ext_prepare_buf,
>> +				 file, fh, arg);
>>  }
>>  
>>  static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
>> @@ -3202,12 +3516,15 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>  	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>> +	IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, planes)),
>>  	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>>  	IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
>> +	IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
> 
> Looking at the other entries, shouldn't this one be 1 line higher?

Yes.

> 
> That said, I wonder if it wouldn't look cleaner if we just put all the
> EXT ioctls together at the bottom.

I can move them and we can see if it is better or not.

> 
>>  	IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
>> +	IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
>> @@ -3272,7 +3589,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>  	IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
>>  	IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
>>  	IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>> +	IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
>> +	IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
>>  	IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
>>  	IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
>> index 8bbcb74d8ee31..75996657ad1ba 100644
>> --- a/include/media/v4l2-ioctl.h
>> +++ b/include/media/v4l2-ioctl.h
>> @@ -169,16 +169,26 @@ struct v4l2_fh;
>>   *	:ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
>>   * @vidioc_querybuf: pointer to the function that implements
>>   *	:ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
>> + * @vidioc_ext_querybuf: pointer to the function that implements
>> + *	:ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
>>   * @vidioc_qbuf: pointer to the function that implements
>>   *	:ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
>> + * @vidioc_ext_qbuf: pointer to the function that implements
>> + *	:ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
>>   * @vidioc_expbuf: pointer to the function that implements
>>   *	:ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
>>   * @vidioc_dqbuf: pointer to the function that implements
>>   *	:ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
>> + * @vidioc_ext_dqbuf: pointer to the function that implements
>> + *	:ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
>>   * @vidioc_create_bufs: pointer to the function that implements
>>   *	:ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
>> + * @vidioc_ext_create_bufs: pointer to the function that implements
>> + *	:ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
>>   * @vidioc_prepare_buf: pointer to the function that implements
>>   *	:ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
>> + * @vidioc_ext_prepare_buf: pointer to the function that implements
>> + *	:ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
>>   * @vidioc_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
>>   * @vidioc_g_fbuf: pointer to the function that implements
>> @@ -439,17 +449,27 @@ struct v4l2_ioctl_ops {
>>  			      struct v4l2_requestbuffers *b);
>>  	int (*vidioc_querybuf)(struct file *file, void *fh,
>>  			       struct v4l2_buffer *b);
>> +	int (*vidioc_ext_querybuf)(struct file *file, void *fh,
>> +				   struct v4l2_ext_buffer *b);
>>  	int (*vidioc_qbuf)(struct file *file, void *fh,
>>  			   struct v4l2_buffer *b);
>> +	int (*vidioc_ext_qbuf)(struct file *file, void *fh,
>> +			       struct v4l2_ext_buffer *b);
>>  	int (*vidioc_expbuf)(struct file *file, void *fh,
>>  			     struct v4l2_exportbuffer *e);
>>  	int (*vidioc_dqbuf)(struct file *file, void *fh,
>>  			    struct v4l2_buffer *b);
>> +	int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
>> +				struct v4l2_ext_buffer *b);
>>  
>>  	int (*vidioc_create_bufs)(struct file *file, void *fh,
>>  				  struct v4l2_create_buffers *b);
>> +	int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
>> +				      struct v4l2_ext_create_buffers *b);
>>  	int (*vidioc_prepare_buf)(struct file *file, void *fh,
>>  				  struct v4l2_buffer *b);
>> +	int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
>> +				      struct v4l2_ext_buffer *b);
>>  
>>  	int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
>>  	int (*vidioc_g_fbuf)(struct file *file, void *fh,
>> @@ -758,6 +778,12 @@ int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>>  				  struct v4l2_format *f,
>>  				  bool mplane_cap, bool strict);
>>  
>> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
>> +			      struct v4l2_buffer *b,
>> +			      bool mplane_cap);
>> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
>> +			      struct v4l2_ext_buffer *e);
>> +
>>  /*
>>   * The user space interpretation of the 'v4l2_event' differs
>>   * based on the 'time_t' definition on 32-bit architectures, so
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index 7123c6a4d9569..334cafdd2be97 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -996,6 +996,41 @@ struct v4l2_plane {
>>  	__u32			reserved[11];
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_plane - extended plane buffer info
>> + * @buffer_length:	size of the entire buffer in bytes, should fit
>> + *			@offset + @plane_length
> 
> Do we actually need this buffer_length at all? We have 3 memory types:
> 
> 1) MMAP - here vb2 already knows the buffer size, because it created it.
> 
> 2) DMABUF - the DMA-buf kAPI provides the information about buffer size.
> 
> 3) USERPTR - this might actually benefit from buffer_length, because there
>    are additional alignmnent requirements for the user memory, e.g. the
>    offset and size must be cacheline aligned.
> 
> Arguably, 1) and 2) are the main usage scenarios, while the user space that
> uses them would have to suffer from added complexity, because of the
> legacy/niche case 3).
> 
> Could we make this field valid only for USERPTR?

I think so, make sense, I'll implement this for next version.

> 
>> + * @plane_length:	size of the plane in bytes.
>> + * @mem_offset:		If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
>> + *			that should be passed to mmap() called on the video node.
>> + * @userptr:		when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>> + *			to this plane.
>> + * @dmabuf_fd:		when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>> + *			associated with this plane.
>> + * @offset:		offset in the memory buffer where the plane starts.
>> + * @memory:		enum v4l2_memory; the method, in which the actual video
>> + *			data is passed
>> + * @reserved:		extra space reserved for future fields, must be set to 0.
>> + *
>> + *
>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>> + * can have one plane for Y, and another for interleaved CbCr components.
>> + * Each plane can reside in a separate memory buffer, or even in
>> + * a completely separate memory node (e.g. in embedded devices).
>> + */
>> +struct v4l2_ext_plane {
>> +	__u32 buffer_length;
>> +	__u32 plane_length;
>> +	union {
>> +		__u32 mem_offset;
>> +		__u64 userptr;
>> +		__s32 dmabuf_fd;
>> +	} m;
>> +	__u32 offset;
>> +	__u32 memory;
>> +	__u32 reserved[4];
>> +};
> 
> Don't we also need bytesused? Or would plane_length essentially mean the
> amount of space or payload, depending on the usage context?

In my understanding, plane_length is the max amount of data the plane can
occupy, othersize it can overflow the buffer or mess with another plane
that is in the same buffer on a different offset.

I'm probably wrong, but I don't really see why the payload size is usefull,
unless if we set a plane_legth that is much bigger then the data it can carry,
that can impact performance.
Payload can also be calculated from the format.

I can add it back if it is usefull. Please let me know your thoughts.

> 
> Similarly, the original data_offset was useful as a return field, which
> some drivers use to indicate that the beginning of the plane is occupied by
> some header or otherwise irrelevant data, which must be skipped. Would the
> offset field be used for this purpose now?

I didn't add an equivalent of the data_offset, since it seemed to be
unused (please see my comments about this above).

> 
>> +
>>  /**
>>   * struct v4l2_buffer - video buffer info
>>   * @index:	id number of the buffer
>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>>  	};
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_buffer - extended video buffer info
>> + * @index:	id number of the buffer
>> + * @type:	V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>> + * @flags:	buffer informational flags
> 
> nit: The order of comments doesn't match the order of fields in the struct.

Ack.

> 
>> + * @field:	enum v4l2_field; field order of the image in the buffer
>> + * @timestamp:	frame timestamp
>> + * @sequence:	sequence count of this frame
>> + * @planes:	per-plane buffer information
>> + * @request_fd:	fd of the request that this buffer should use
>> + * @reserved:	extra space reserved for future fields, must be set to 0
>> + *
>> + * Contains data exchanged by application and driver using one of the Streaming
>> + * I/O methods.
>> + */
>> +struct v4l2_ext_buffer {
>> +	__u32 index;
>> +	__u32 type;
>> +	__u32 field;
>> +	__u32 sequence;
>> +	__u64 flags;
>> +	__u64 timestamp;
> 
> What's the unit? How does this play with the other UAPI that the user space
> may use, e.g. clock_gettime() which returns struct timespec?

The unity is nsec:

	e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
		b->timestamp.tv_usec * NSEC_PER_USEC;

I can clarify in the docs, is this ok?

> 
>> +	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>> +	__s32 request_fd;
>> +	__u32 reserved[9];
>> +};
>> +
>>  #ifndef __KERNEL__
>>  /**
>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
>>  	__u32			reserved[6];
>>  };
>>  
>> +/**
>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>> + * @index:	on return, index of the first created buffer
>> + * @count:	entry: number of requested buffers,
>> + *		return: number of created buffers
>> + * @memory:	enum v4l2_memory; buffer memory type
>> + * @capabilities: capabilities of this buffer type.
>> + * @format:	frame format, for which buffers are requested
>> + * @flags:	additional buffer management attributes (ignored unless the
>> + *		queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>> + *		and configured for MMAP streaming I/O).
> 
> Do we need to say that it is ignored here? I'd consider this to be a
> per-flag decision.

flags field was removed from v4l2_create_buffers, so I'm removing it from
v4l2_ext_create_buffers for the same reason.


Thanks,
Helen

> 
> Best regards,
> Tomasz
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls
  2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
                   ` (8 preceding siblings ...)
  2020-08-14  7:49 ` Alexandre Courbot
@ 2020-11-27 15:06 ` Helen Koike
  9 siblings, 0 replies; 51+ messages in thread
From: Helen Koike @ 2020-11-27 15:06 UTC (permalink / raw)
  To: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus, linux-media
  Cc: tfiga, hiroh, nicolas, Brian.Starkey, kernel, boris.brezillon,
	narmstrong, linux-kernel, frkoenig, mjourdan, stanimir.varbanov

Hello,

On 8/4/20 4:29 PM, Helen Koike wrote:
> Hello,
> 
> This is v5 of the Extended API for formats and buffers, which introduces
> the following new ioctls:
> 
> int ioctl(int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
> int ioctl(int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
> int ioctl(int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp)
> 
> int ioctl(int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp)
> int ioctl(int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp)
> int ioctl(int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp)
> int ioctl(int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp)
> int ioctl(int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp)
> 
> Please check v4 cover letter specific topic past discussions
> https://patchwork.linuxtv.org/project/linux-media/cover/20200717115435.2632623-1-helen.koike@collabora.com/
> 
> Documentation
> =============
> I added a first version of the documentation.
> I would appreciate some tips in how to organize this better, since there are
> several parts of the docs which reference the "old" API, and for now
> I just added a note pointing to the Extended API.
> 
> Instead of duplicating documentation from the code to the Docs (as used by
> most part of v4l2 documentation), I just added a reference to let Sphinx get
> the structs documentation from the code, so we can avoid duplicating.
> 
> For reviewing convenience, I uploaded the generated html docs at
> https://people.collabora.com/~koike/ext-doc-v5/userspace-api/media/v4l/ext-api.html
> 
> There is room for improvements, it would be great to get your suggestions.
> 
> uAPI
> ====
> This version have some minor changes in the uAPI structs, highlight to the
> mem_offset that was returned to struct v4l2_ext_plane, memory field that now
> is per plane, and some adjustments in field sizes and re-ordering to make
> structs the same for 32 and 64 bits (which I verified with pahole tool).
> 
> I also updated v4l2-compliance:
> https://gitlab.collabora.com/koike/v4l-utils/-/tree/ext-api/wip
> 
> We need to decide the size of the reserved fields left, how much reserved
> fields do you think we should leave for each struct?
> 
> What is next
> ============
> I would like to improve implementation (for the kernel and v4l2-compliane) and
> drop the RFC tag for next version, so please review the uAPI.
> 
> 
> List of changes in v5:
> ======================
> * first version of Documentation
> * migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> * return mem_offset to struct v4l2_ext_plane
> * change sizes and reorder fields to avoid holes in the struct and make it
>   the same for 32 and 64 bits
> * removed __attribute__ ((packed)) from uapi structs
> * set request_fd to signed
> * add documentation
> * updated some commit messages
> 
> Hans Verkuil (1):
>   media: v4l2: Add extended buffer operations
> 
> Helen Koike (6):
>   media: v4l2: Extend pixel formats to unify single/multi-planar
>     handling (and more)
>   media: videobuf2: Expose helpers to implement the _ext_fmt and
>     _ext_buf hooks
>   media: mediabus: Add helpers to convert a ext_pix format to/from a
>     mbus_fmt
>   media: vivid: Convert the capture and output drivers to
>     EXT_FMT/EXT_BUF
>   media: vimc: Implement the ext_fmt and ext_buf hooks
>   media: docs: add documentation for the Extended API
> 
>  .../userspace-api/media/v4l/buffer.rst        |   5 +
>  .../userspace-api/media/v4l/common.rst        |   1 +
>  .../userspace-api/media/v4l/dev-capture.rst   |   5 +
>  .../userspace-api/media/v4l/dev-output.rst    |   5 +
>  .../userspace-api/media/v4l/ext-api.rst       | 107 ++
>  .../userspace-api/media/v4l/format.rst        |  16 +-
>  .../userspace-api/media/v4l/user-func.rst     |   5 +
>  .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++
>  .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++
>  .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++
>  .../media/v4l/vidioc-ext-querybuf.rst         |  79 ++
>  .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 +++
>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>  .../media/common/videobuf2/videobuf2-v4l2.c   | 560 ++++++-----
>  .../media/test-drivers/vimc/vimc-capture.c    |  61 +-
>  drivers/media/test-drivers/vimc/vimc-common.c |   6 +-
>  drivers/media/test-drivers/vimc/vimc-common.h |   2 +-
>  drivers/media/test-drivers/vivid/vivid-core.c |  70 +-
>  .../test-drivers/vivid/vivid-touch-cap.c      |  26 +-
>  .../test-drivers/vivid/vivid-touch-cap.h      |   3 +-
>  .../media/test-drivers/vivid/vivid-vid-cap.c  | 169 +---
>  .../media/test-drivers/vivid/vivid-vid-cap.h  |  15 +-
>  .../media/test-drivers/vivid/vivid-vid-out.c  | 193 ++--
>  .../media/test-drivers/vivid/vivid-vid-out.h  |  15 +-
>  drivers/media/v4l2-core/v4l2-dev.c            |  50 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 938 ++++++++++++++++--
>  include/media/v4l2-ioctl.h                    |  60 ++
>  include/media/v4l2-mediabus.h                 |  42 +
>  include/media/videobuf2-core.h                |   6 +-
>  include/media/videobuf2-v4l2.h                |  21 +-
>  include/uapi/linux/videodev2.h                | 146 +++
>  31 files changed, 2363 insertions(+), 723 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
> 



I'm fixing the format and buffer conversions to separate planes per color
component as pointed by Tomasz.
It turns out this it a bit more complicated and will require changes in vb2
driver ops interface.

I want to share my approach in case you see that I'm missing something while
I'm implementing it, or if I'm making wrong assumptions. So I wrote different
scenarios below. Please check, specially the NOTEs in the scenarios.

Thanks!

----------------------------------------

Considerations:
* Multiplanar API vs Singleplanar API.
* M-variant vs non-M-variant,
* data_offset is only available in Multiplanar API, not in single planar.
* Single buffer: contiguous planes or not.
* Memory buffers vs color components (planes)
* Planes with offsets or not.

NOTE: I'm assuming that multiplanar and singleplanar capabilities are mutually exclusive,
      i.e. if a driver reports V4L2_CAP_VIDEO_CAPTURE_MPLANE then it won't report
      V4L2_CAP_VIDEO_CAPTURE, which means that only multiplanar API is valid,
      and if V4L2_CAP_VIDEO_CAPTURE is reported, then _MPLANE isn't and only singleplanar
      API is valid and supported by the driver.

NOTE: I'm assuming both non-M-variants and M-variants are acceptable and
      equivalent for the Ext API.
      If we have two pixelformats that are equivalent, then we chose one to be the
      normalized format.

NOTE: Just for the sake of the examples below, I'm assuming that all M-variants have
      non-M-variants (which I'm considering that will be the normalized form), but
      I'm not doing this assumption in the code and it doesn't seem to be a problem.

===========================================
A: Scenarios where:
	userspace -> implements clasic API
	driver -> implements ext API
-------------------------------------------

QUESTION: if we want a ext_reqbuf, then queue_setup() (or a ext_queue_setup)
	  should return a buffer per plane? Or a buffer for all planes? Which
	  method should be favored? Also, see NOTE A-1.3.
	  Another option is: ext_reqbufs won't be supported, and leave to userspace
	  to decide this in ext_create_bufs.
	  For the examples below, I'm assuming reqbuf isn't supported

Case A-1: Single planar API
- I'm assuming that M-variants pixelformats are invalid
- I'm assuming the driver report in its capabilities only V4L2_CAP_VIDEO_CAPTURE/OUTPUT/M2M,
  and not the MPLANE equivalent.

	FORMATS:
	1. framework receives v4l2_format
	2. framework converts to v4l2_ext_pix_format, separating color components
	   per plane, in the best effort way (without returning errors).
	3. v4l2_ext_pix_format gets processed
	4. framework converts back to v4l2_format and sent to userspace.

	CREATE BUFFER:
	1. vb2 receives v4l2_buffer from userspace
	2. vb2 converts to v4l2_ext_buffer, dividing the buffer per color component.
	   Planes will be part of the same buffer, just in a different offset.

		NOTE A-1.1: vb2 needs to know the pixelformat and image size to
			    calculate the number of color components and plane's
			    offset for the conversion, my current solution is to
			    add another callback in vb2_ops to get the format.
			    See NOTE A-2.3.

	3. memory sizes is validated by queue_setup()
	4. vb2 converts it back to v4l2_buffer and sends to userspace

	REQ BUFFER:
	1. vb2 calls queue_setup to ask the driver about the number of memory
	   buffers in a frame buffer
	2. queue_setup() returns with a single memory buffer with contiguous planes.

		NOTE A-1.2: If the driver decides to add offset between planes, we
			    won't be able to convert back to v4l2_buffer and we
			    fail. Even if we repurpose data_offset, there is
			    no data_offset for single plane API

		NOTE A-1.3: I need to send a hint to queue_setup() to tell the
			    driver if we need a single mem-buffer for all planes
			    or a different mem-buffer per plane.
			    Or maybe implement a ext_queue_setup().

	3. v4l2_ext_buffer is created
	3. v4l2_ext_buffer is converted to v4l2_buffer and returned to userspace.

	QUEUE BUFFER:
	1. vb2 receives v4l2_buffer from userspace (only index and type that matters)
	2. vb2 converts to v4l2_ext_buffer
	3. vb2_buffer is validated by .buf_prepare() and gets processed.
	4. v4l2_ext_buffer is converted to v4l2_buffer and sent to userspace.


Case A-2: Multiplanar API with non-M-variant pixelformat
- I'm assuming the driver only reports V4L2_CAP_VIDEO_*_MPLANE

	FORMATS: Same as Case A-1, just changing fields from v4l2_format to use
		 mplane API.

	CREATE BUFFER: If data-offset is zero, same as Case A-1.

		NOTE A-2.1: If data_offset isn't zero, then I return an error,
			    since we don't support it. Unless if we re-pourpose
			    the data_offset field as previously proposed.
			    But this looks non consistent with single planar,
			    see NOTE A-1.2.

	REQ BUFFER:
	1. vb2 calls queue_setup to ask the driver about the number of memory
	   buffers in a frame buffer
	2. If queue_setup() return a single mem-buffer then it is the same as
	   Case A-1.

		NOTE A-2.2: If driver returns multiple buffers, then we can't
			    convert back to clasic non-M-variant and we fail.
			    My proposed solution is adding a hint to queue_setup(),
			    see NOTE A-1.3.
			    This hint would depend if the API was multiplanar
			    or not, and if it was M-variant or not.

	QUEUE BUFFER:
	1. vb2 receives v4l2_buffer from userspace (index, type and planes array are filled)
	2. vb2 converts v4l2_buffer to v4l2_ext_buffer according to the pixelformat.

		NOTE: A-2.3: to convert this properly, I thought we could use
			     buf_prepare() to get the updated information of the
			     pixelformat ans sizeimage, but .buf_prepare() is called
			     after this conversion, so we need to call v4l2_g_ext_pix_format()
			     from vb2 somehow, my current solution is to add another
			     callback in vb2_ops.

	2. vb2 gets vb2_buffer from index and fills vb2_v4l2_buffer.planes from
	   the converted v4l2_ext_buffer.
	3. vb2_buffer is validated by .buf_prepare() and gets processed.
	4. v4l2_buffer is converted back to v4l2_ext_buffer and sent to userspace.

Case A-3: Multiplanar API with M-variant pixelformat
- I'm assuming the driver only reports V4L2_CAP_VIDEO_*_MPLANE

	FORMATS:
	1. framework receives v4l2_format
	2. framework converts to v4l2_ext_pix_format, which is easier, we just
	   need to normalize the pixelformat.
	3. v4l2_ext_pix_format gets processed
	4. framework converts back to v4l2_format keeping the original and sends
	   pixelformat to userspace.

		NOTE A-3.1: I need to save the original pixelformat, so I can
			    know that I need to convert back to M-variant.

	CREATE BUFFERS:
	1. vb2 receives v4l2_buffer from userspace
	2. vb2 converts to v4l2_ext_buffer, which is easier, besides complication
	   from NOTE A-1.1.
	3. vb2 validates the sizes with queue_setup()
	4. vb2 converts it back to v4l2_buffer and sends to userspace

	REQ BUFFER:
	1. vb2 calls queue_setup to ask the driver about the number of memory
	   buffers in a frame buffer
	2. If queue_setup() returns a mem buffer per component then we can continue.

		NOTE A-3.2: If driver doesn't return a different mem_buffer per
			    color component, then we can't convert back to clasic
			    M-variant and we fail, another reason to a hint in
			    queue_setup(). See NOTE A-1.3 and A-2-2.

	QUEUE BUFFER: same as Case A-2.

===========================================
B: Scenarios where:
	userspace -> implements ext API
	driver -> implements classic API
-------------------------------------------
- I'm assuming that userspace won't differentiate V4L2_CAP_VIDEO_* from V4L2_CAP_VIDEO_*_MPLANE
- I'm not handling the case where we could have the two first planes in a memory buffer
  and a third plane in another memory buffer

Case B-1: Single memory buffer with contiguous planes / driver single planar API

	FORMATS:
	1. framework receives v4l2_ext_pix_format
	2. framework converts to v4l2_format non non-M-variant pixelformat (we
	   know we need to convert o non-M-variants because of the driver's
	   capabilities)
	3. v4l2_format gets processed
	4. framework converts back to v4l2_ext_pix_format and sends to userspace.

	CREATE BUFFERS:
	1. vb2 receives v4l2_ext_buffer from userspace with single mem buffer
	   and contiguous planes.
	2. vb2 converts to v4l2_buffer with a single plane
	3. v4l2_buffer is validated by queue_setup()
	4. vb2 converts back to v4l2_ext_format and sends to userspace.

Case B-2: Single memory buffer with contiguous planes / driver multi planar API

	FORMATS:
	1. framework receives v4l2_ext_pix_format
	2. framework converts to v4l2_format non non-M-variant pixelformat

		QUESTION: should we convert to M-variant instead?

	3. v4l2_format gets processed

		QUESTION: if it fails, should we convert to M-variant and re-try?

	4. framework converts back to v4l2_ext_pix_format and sent to userspace.

Case B-3: Single memory buffer with non contiguous planes

	FORMATS: same as B-1 or B-2 (if mplane API or not)

	CREATE BUFFERS:
	1. vb2 receives v4l2_ext_buffer from userspace with single mem buffer
	   and non contiguous planes.
	2. we fail, since we can't convert to v4l2_buffer

Case B-3: Multi memory buffers / driver is single planar API

	FORMATS: Same as Case B-2

	CREATE BUFFERS:
	1. vb2 receives v4l2_ext_buffer from userspace requesting a memory buffer
	   per plane.
	2. we fail, since driver doesn't support it

Case B-4: Multi memory buffers / driver is multi planar API

	FORMATS: Same as Case B-2

	CREATE BUFFERS:
	1. vb2 receives v4l2_ext_buffer from userspace requesting a memory buffer
	   per plane.
	2. vb2 convert v4l2_ext_buffer to v4l2_buffer using multiplanar API
	3. v4l2_buffer is validated by queue_setup()
	4. vb2 converts back to v4l2_ext_format and sends to userspace.


Regards,
Helen

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-11-23 17:40         ` Helen Koike
@ 2020-12-03 15:11           ` Hans Verkuil
  2020-12-03 19:52             ` Helen Koike
  2020-12-14 10:38             ` Tomasz Figa
  0 siblings, 2 replies; 51+ messages in thread
From: Hans Verkuil @ 2020-12-03 15:11 UTC (permalink / raw)
  To: Helen Koike, Tomasz Figa
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Boris Brezillon,
	Hirokazu Honda, Nicolas Dufresne, Brian Starkey, kernel,
	Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

On 23/11/2020 18:40, Helen Koike wrote:
> 
> 
> On 11/23/20 12:46 PM, Tomasz Figa wrote:
>> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>
>>> Hi Hans,
>>>
>>> Thank you for your review.
>>>
>>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
>>>> Hi Helen,
>>>>
>>>> Again I'm just reviewing the uAPI.
>>>>
>>>> On 04/08/2020 21:29, Helen Koike wrote:
>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>
>>>>> Those extended buffer ops have several purpose:
>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>>    the number of ns elapsed since 1970
>>>>> 2/ Unify single/multiplanar handling
>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>>    to support the case where a single buffer object is storing all
>>>>>    planes data, each one being placed at a different offset
>>>>>
>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>>> these new objects.
>>>>>
>>>>> The core takes care of converting new ioctls requests to old ones
>>>>> if the driver does not support the new hooks, and vice versa.
>>>>>
>>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>>> use the Request API to collect more metadata information from the
>>>>> frame.
>>>>>
>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>> ---
>>>>> Changes in v5:
>>>>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
>>>>> - return mem_offset to struct v4l2_ext_plane
>>>>> - change sizes and reorder fields to avoid holes in the struct and make
>>>>>   it the same for 32 and 64 bits
>>>>>
>>>>> Changes in v4:
>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>>> - Reformulate struct v4l2_ext_plane
>>>>> - Fix some bugs caught by v4l2-compliance
>>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>>
>>>>> Changes in v3:
>>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>>
>>>>> Changes in v2:
>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>>   later on
>>>>> ---
>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>>  include/uapi/linux/videodev2.h       |  90 +++++++
>>>>>  4 files changed, 476 insertions(+), 22 deletions(-)
>>>>>
>>>>
>>>> <snip>
>>>>
>>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>>>> index 7123c6a4d9569..334cafdd2be97 100644
>>>>> --- a/include/uapi/linux/videodev2.h
>>>>> +++ b/include/uapi/linux/videodev2.h
>>>>> @@ -996,6 +996,41 @@ struct v4l2_plane {
>>>>>      __u32                   reserved[11];
>>>>>  };
>>>>>
>>>>> +/**
>>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>>> + * @buffer_length:  size of the entire buffer in bytes, should fit
>>>>> + *                  @offset + @plane_length
>>>>> + * @plane_length:   size of the plane in bytes.
>>>>> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
>>>>> + *                  that should be passed to mmap() called on the video node.
>>>>> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>>> + *                  to this plane.
>>>>> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>>> + *                  associated with this plane.
>>>>> + * @offset:         offset in the memory buffer where the plane starts.
>>>>> + * @memory:         enum v4l2_memory; the method, in which the actual video
>>>>> + *                  data is passed
>>>>> + * @reserved:               extra space reserved for future fields, must be set to 0.
>>>>> + *
>>>>> + *
>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>>> + */
>>>>> +struct v4l2_ext_plane {
>>>>> +    __u32 buffer_length;
>>>>> +    __u32 plane_length;
>>>>> +    union {
>>>>> +            __u32 mem_offset;
>>>>> +            __u64 userptr;
>>>>> +            __s32 dmabuf_fd;
>>>>> +    } m;
>>>>> +    __u32 offset;
>>>>
>>>> I'd rename this plane_offset. I think some reordering would make this struct easier
>>>> to understand:
>>>>
>>>> struct v4l2_ext_plane {
>>>>       __u32 buffer_length;
>>>>       __u32 plane_offset;
>>>>       __u32 plane_length;
>>>>       __u32 memory;
>>>>       union {
>>>>               __u32 mem_offset;
>>>>               __u64 userptr;
>>>>               __s32 dmabuf_fd;
>>>>       } m;
>>>>       __u32 reserved[4];
>>>> };
>>>>
>>>>> +    __u32 memory;
>>>>> +    __u32 reserved[4];
>>>>> +};
>>>
>>> Ok, I'll apply this to the next version.
>>>
>>>>
>>>> What is not clear is how to tell the different between a single buffer containing
>>>> multiple planes, and using a separate buffer per plane. E.g. what would this look
>>>> like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
>>>> V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
>>>> planes are also combined in a single buffer?
>>>>
>>>> I would guess that the m union is set to 0 if the plane is part of the buffer
>>>> defined in the previous plane?
>>>
>>> The difference would be if m are equal or differ between planes, example:
>>>
>>> For V4L2_PIX_FMT_YVU420:
>>>
>>>     Y:
>>>         plane_offset = 0
>>>         m.dmabuf_fd = 3
>>>     Cb:
>>>         plane_offset = 300
>>>         m.dmabuf_fd = 3
>>>     Cr:
>>>         plane_offset = 375
>>>         m.dmabuf_fd = 3
>>>
>>> For V4L2_PIX_FMT_YVU420M:
>>>
>>>     Y:
>>>         plane_offset = 0
>>>         m.dmabuf_fd = 4
>>>     Cb:
>>>         plane_offset = 0
>>>         m.dmabuf_fd = 5
>>>     Cr:
>>>         plane_offset = 0
>>>         m.dmabuf_fd = 6
>>>
>>>
>>> Does it make sense?
>>>
>>
>> Actually all the 3 file descriptors can still point to the same
>> buffer, because they might have been dup()ed. The kernel needs to
>> resolve the file descriptors into struct dma_buf and then check
>> whether it's one or more buffers.
> 
> Right, thanks for this.
> 
>>
>> In fact, dup()ed FD for each plane is quite a common case in other
>> APIs, e.g. EGL, but current V4L2 API can't handle it. In Chromium we
>> basically work around it by assuming that if we receive a buffer for a
>> V4L2 device that only supports non-M formats, then we can safely
>> ignore all but first FD. The new API gives the ability to handle the
>> case properly, with full validation by the kernel.
>>
>>>>
>>>>> +
>>>>>  /**
>>>>>   * struct v4l2_buffer - video buffer info
>>>>>   * @index:  id number of the buffer
>>>>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>>>>>      };
>>>>>  };
>>>>>
>>>>> +/**
>>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>>> + * @index:  id number of the buffer
>>>>> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>>> + * @flags:  buffer informational flags
>>>>> + * @field:  enum v4l2_field; field order of the image in the buffer
>>>>> + * @timestamp:      frame timestamp
>>>>> + * @sequence:       sequence count of this frame
>>>>> + * @planes: per-plane buffer information
>>>>> + * @request_fd:     fd of the request that this buffer should use
>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>>> + *
>>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>>> + * I/O methods.
>>>>> + */
>>>>> +struct v4l2_ext_buffer {
>>>>> +    __u32 index;
>>>>> +    __u32 type;
>>>>> +    __u32 field;
>>>>> +    __u32 sequence;
>>>>> +    __u64 flags;
>>>>> +    __u64 timestamp;
>>>>> +    struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>>>> +    __s32 request_fd;
>>>>> +    __u32 reserved[9];
>>>>> +};
>>>>
>>>> Brainstorming:
>>>>
>>>> Some ideas I have to make it easier to support mid stream resolution/colorimetry
>>>> changes:
>>>>
>>>> Adding width and height would support resolution changes (requires the use of
>>>> CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
>>>> information is provided here, then there are no race conditions.
>>>>
>>>> Same for adding the colorimetry fields here, this too can change on the fly (esp.
>>>> with HDMI), so reporting this information here avoids race conditions as well.
>>>
>>> Right, do you think this is something we can discuss later in a different RFC?
>>> So we can have a better view on how dynamic resolution change would be used?
>>>
>>> We can add more reserved fields or maybe try to do something to what has been
>>> discussed in about extensible system calls [1]
>>>
>>> [1] https://lwn.net/Articles/830666/
>>>
>>>>
>>>> And thirdly, I would like to have a __u64 boot_timestamp field containing the
>>>> CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
>>>> is that for m2m devices it is just copied and that for other devices it can have
>>>> different meanings depending on the timestamp buffer flags.
>>>>
>>>> There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
>>>> to add support for this. That way you know exactly when the driver was finished with
>>>> the buffer and that helps in detecting missed frames or instrumentation.
>>>
>>> I don't mind adding it. Does it make sense to have both timestamp and boot_timestamp?
>>>
>>
>> I think this is quite independent from the ext API work. AFAIR there
>> was an RFC to request the timestamp source from the userspace by the
>> flags field in QBUF, which would work with the existing API as well,
>> or it wasn't posted in the end?

It's not about selecting a specific clock source. I think that option 4 as described
below would work for that.

This problem I'm describing here is specific to m2m devices where the timestamp is
either just passed through untouched, or it is used as an identifier for a buffer
for use with stateless decoders.

In both cases you cannot use the timestamp as a proper timestamp that tells you when
the buffer was marked done by the driver. So this is about adding a second timestamp
field (timestamp_done or something like that). Whether this would be hardcoded as using
CLOCK_BOOTTIME or uses the same clock source as selected through a control is something
that can be discussed, but since it does require a new field I believe this is part of
this proposal.

Regards,

	Hans

> 
> I was recalling the discussions we had regarding this:
> 
> 1.
>     This first attempt in the uvc driver is to use a specific kernel parameter for that case:
>     https://patchwork.kernel.org/patch/10644887/
>     The conclusion that the support should be in the core API and not driver specific.
> 
> 2.
>     Then an attempt to add global v4l2 support was sent with the Mediatek patch series:
>     https://patchwork.linuxtv.org/patch/60878/
>     The major problem is that clock type should be something selectable by userspace, and
>     not pre-defined by the driver.
> 
> 3.
>     Another idea was to use the 'flags' field in the structs v4l2_requestbuffers and
>     v4l2_create_buffers.
>     But this field was removed in
>     129134e5415d ("media: media/v4l2: remove V4L2_FLAG_MEMORY_NON_CONSISTENT flag")
>     The major concern with this approach was with the uAPI, since it doesn't make much
>     sense to select a clock when creating buffers.
> 
> 4.
>     Another suggestion by Nicolas Dufresne was to add this as a menu control so that userspace
>     can choose the clock for the timestamps from a given list, the enum in the list can also match
>     the clocks ids.
>     We would need to add a new buf flag in struct v4l2_buffer, like V4L2_BUF_FLAG_TIMESTAMP_OTHER,
>     which would be "as specified through controls ...."
> 
> 
> So my current question is, should we have both __u32 timestamp and __u32 boottimestamp?
> Or should we have a mechanism that allows switching from one to the other and use
> a single field? And if this mechanism should be implemented in both APIs? Can this be
> defined later?
> 
> 
> Please, let me know your thoughts.
> 
> Thanks,
> Helen
> 
>>
>>>>
>>>>> +
>>>>>  #ifndef __KERNEL__
>>>>>  /**
>>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>>>>> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
>>>>>      __u32                   reserved[6];
>>>>>  };
>>>>>
>>>>> +/**
>>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>>>>> + * @index:  on return, index of the first created buffer
>>>>> + * @count:  entry: number of requested buffers,
>>>>> + *          return: number of created buffers
>>>>> + * @memory: enum v4l2_memory; buffer memory type
>>>>> + * @capabilities: capabilities of this buffer type.
>>>>> + * @format: frame format, for which buffers are requested
>>>>> + * @flags:  additional buffer management attributes (ignored unless the
>>>>> + *          queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>>>>> + *          and configured for MMAP streaming I/O).
>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>>> + */
>>>>> +struct v4l2_ext_create_buffers {
>>>>> +    __u32                           index;
>>>>> +    __u32                           count;
>>>>> +    __u32                           memory;
>>>>> +    __u32                           capabilities;
>>>>> +    struct v4l2_ext_pix_format      format;
>>>>
>>>> The reality is that the only field that is ever used in the original v4l2_format
>>>> struct is sizeimage. So this can be replaced with:
>>>>
>>>>       __u32                           plane_size[VIDEO_MAX_PLANES];
>>>>
>>>> (the field name I picked is debatable, but you get the idea)
>>>>
>>>> The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
>>>> is needed for the current format. The original idea of using struct v4l2_format
>>>> was that drivers would use the full format information to calculate the
>>>> memory size, but that was just much too complicated to implement and nobody
>>>> ever used that. Only the sizeimage field was ever used.
>>>
>>> Right, I'll update this in next version, This should simplify things.
>>>
>>
>> I think this might need a bit more discussion. How would the userspace
>> know what size is enough for the desired resolution? The hardware
>> and/or drivers often have various alignment/padding restrictions,
>> which might not be easy to guess for the userspace.
>>
>> Also I don't quite understand what's so complicated in handling the
>> full format, or at least the most important parts of it. The
>> implementation of TRY_FMT/S_FMT, which exists in every driver, should
>> already be able to calculate the right plane sizes.
>>
>> Best regards,
>> Tomasz
>>
>>>
>>> Thanks,
>>> Helen
>>>
>>>>
>>>>> +    __u32                           flags;
>>>>> +    __u32 reserved[5];
>>>>> +};
>>>>> +
>>>>>  /*
>>>>>   *  I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>>>>   *
>>>>> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
>>>>>  #define VIDIOC_G_EXT_PIX_FMT        _IOWR('V', 104, struct v4l2_ext_pix_format)
>>>>>  #define VIDIOC_S_EXT_PIX_FMT        _IOWR('V', 105, struct v4l2_ext_pix_format)
>>>>>  #define VIDIOC_TRY_EXT_PIX_FMT      _IOWR('V', 106, struct v4l2_ext_pix_format)
>>>>> +#define VIDIOC_EXT_CREATE_BUFS      _IOWR('V', 107, struct v4l2_ext_create_buffers)
>>>>> +#define VIDIOC_EXT_QUERYBUF _IOWR('V', 108, struct v4l2_ext_buffer)
>>>>> +#define VIDIOC_EXT_QBUF             _IOWR('V', 109, struct v4l2_ext_buffer)
>>>>> +#define VIDIOC_EXT_DQBUF    _IOWR('V', 110, struct v4l2_ext_buffer)
>>>>> +#define VIDIOC_EXT_PREPARE_BUF      _IOWR('V', 111, struct v4l2_ext_buffer)
>>>>>
>>>>>  /* Reminder: when adding new ioctls please add support for them to
>>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>>>>
>>>>
>>>> Regards,
>>>>
>>>>       Hans
>>>>


^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-03 15:11           ` Hans Verkuil
@ 2020-12-03 19:52             ` Helen Koike
  2020-12-14 10:46               ` Tomasz Figa
  2020-12-14 10:38             ` Tomasz Figa
  1 sibling, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-12-03 19:52 UTC (permalink / raw)
  To: Hans Verkuil, Tomasz Figa
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Boris Brezillon,
	Hirokazu Honda, Nicolas Dufresne, Brian Starkey, kernel,
	Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

Hi,

Please see my 2 points below (about v4l2_ext_buffer and another about timestamp).

On 12/3/20 12:11 PM, Hans Verkuil wrote:
> On 23/11/2020 18:40, Helen Koike wrote:
>>
>>
>> On 11/23/20 12:46 PM, Tomasz Figa wrote:
>>> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>
>>>> Hi Hans,
>>>>
>>>> Thank you for your review.
>>>>
>>>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
>>>>> Hi Helen,
>>>>>
>>>>> Again I'm just reviewing the uAPI.
>>>>>
>>>>> On 04/08/2020 21:29, Helen Koike wrote:
>>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>>
>>>>>> Those extended buffer ops have several purpose:
>>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>>>    the number of ns elapsed since 1970
>>>>>> 2/ Unify single/multiplanar handling
>>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>>>    to support the case where a single buffer object is storing all
>>>>>>    planes data, each one being placed at a different offset
>>>>>>
>>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>>>> these new objects.
>>>>>>
>>>>>> The core takes care of converting new ioctls requests to old ones
>>>>>> if the driver does not support the new hooks, and vice versa.
>>>>>>
>>>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>>>> use the Request API to collect more metadata information from the
>>>>>> frame.
>>>>>>
>>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>>> ---
>>>>>> Changes in v5:
>>>>>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
>>>>>> - return mem_offset to struct v4l2_ext_plane
>>>>>> - change sizes and reorder fields to avoid holes in the struct and make
>>>>>>   it the same for 32 and 64 bits
>>>>>>
>>>>>> Changes in v4:
>>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>>>> - Reformulate struct v4l2_ext_plane
>>>>>> - Fix some bugs caught by v4l2-compliance
>>>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>>>
>>>>>> Changes in v3:
>>>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>>>
>>>>>> Changes in v2:
>>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>>>   later on
>>>>>> ---
>>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>>>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>>>  include/uapi/linux/videodev2.h       |  90 +++++++
>>>>>>  4 files changed, 476 insertions(+), 22 deletions(-)
>>>>>>
>>>>>
>>>>> <snip>
>>>>>
>>>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>>>>> index 7123c6a4d9569..334cafdd2be97 100644
>>>>>> --- a/include/uapi/linux/videodev2.h
>>>>>> +++ b/include/uapi/linux/videodev2.h
>>>>>> @@ -996,6 +996,41 @@ struct v4l2_plane {
>>>>>>      __u32                   reserved[11];
>>>>>>  };
>>>>>>
>>>>>> +/**
>>>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>>>> + * @buffer_length:  size of the entire buffer in bytes, should fit
>>>>>> + *                  @offset + @plane_length
>>>>>> + * @plane_length:   size of the plane in bytes.
>>>>>> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
>>>>>> + *                  that should be passed to mmap() called on the video node.
>>>>>> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>>>> + *                  to this plane.
>>>>>> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>>>> + *                  associated with this plane.
>>>>>> + * @offset:         offset in the memory buffer where the plane starts.
>>>>>> + * @memory:         enum v4l2_memory; the method, in which the actual video
>>>>>> + *                  data is passed
>>>>>> + * @reserved:               extra space reserved for future fields, must be set to 0.
>>>>>> + *
>>>>>> + *
>>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>>>> + */
>>>>>> +struct v4l2_ext_plane {
>>>>>> +    __u32 buffer_length;
>>>>>> +    __u32 plane_length;
>>>>>> +    union {
>>>>>> +            __u32 mem_offset;
>>>>>> +            __u64 userptr;
>>>>>> +            __s32 dmabuf_fd;
>>>>>> +    } m;
>>>>>> +    __u32 offset;
>>>>>
>>>>> I'd rename this plane_offset. I think some reordering would make this struct easier
>>>>> to understand:
>>>>>
>>>>> struct v4l2_ext_plane {
>>>>>       __u32 buffer_length;
>>>>>       __u32 plane_offset;
>>>>>       __u32 plane_length;
>>>>>       __u32 memory;
>>>>>       union {
>>>>>               __u32 mem_offset;
>>>>>               __u64 userptr;
>>>>>               __s32 dmabuf_fd;
>>>>>       } m;
>>>>>       __u32 reserved[4];
>>>>> };
>>>>>
>>>>>> +    __u32 memory;
>>>>>> +    __u32 reserved[4];
>>>>>> +};
>>>>
>>>> Ok, I'll apply this to the next version.
>>>>
>>>>>
>>>>> What is not clear is how to tell the different between a single buffer containing
>>>>> multiple planes, and using a separate buffer per plane. E.g. what would this look
>>>>> like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
>>>>> V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
>>>>> planes are also combined in a single buffer?
>>>>>
>>>>> I would guess that the m union is set to 0 if the plane is part of the buffer
>>>>> defined in the previous plane?
>>>>
>>>> The difference would be if m are equal or differ between planes, example:
>>>>
>>>> For V4L2_PIX_FMT_YVU420:
>>>>
>>>>     Y:
>>>>         plane_offset = 0
>>>>         m.dmabuf_fd = 3
>>>>     Cb:
>>>>         plane_offset = 300
>>>>         m.dmabuf_fd = 3
>>>>     Cr:
>>>>         plane_offset = 375
>>>>         m.dmabuf_fd = 3
>>>>
>>>> For V4L2_PIX_FMT_YVU420M:
>>>>
>>>>     Y:
>>>>         plane_offset = 0
>>>>         m.dmabuf_fd = 4
>>>>     Cb:
>>>>         plane_offset = 0
>>>>         m.dmabuf_fd = 5
>>>>     Cr:
>>>>         plane_offset = 0
>>>>         m.dmabuf_fd = 6
>>>>
>>>>
>>>> Does it make sense?
>>>>
>>>
>>> Actually all the 3 file descriptors can still point to the same
>>> buffer, because they might have been dup()ed. The kernel needs to
>>> resolve the file descriptors into struct dma_buf and then check
>>> whether it's one or more buffers.
>>
>> Right, thanks for this.
>>
>>>
>>> In fact, dup()ed FD for each plane is quite a common case in other
>>> APIs, e.g. EGL, but current V4L2 API can't handle it. In Chromium we
>>> basically work around it by assuming that if we receive a buffer for a
>>> V4L2 device that only supports non-M formats, then we can safely
>>> ignore all but first FD. The new API gives the ability to handle the
>>> case properly, with full validation by the kernel.
>>>
>>>>>
>>>>>> +
>>>>>>  /**
>>>>>>   * struct v4l2_buffer - video buffer info
>>>>>>   * @index:  id number of the buffer
>>>>>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>>>>>>      };
>>>>>>  };
>>>>>>
>>>>>> +/**
>>>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>>>> + * @index:  id number of the buffer
>>>>>> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>>>> + * @flags:  buffer informational flags
>>>>>> + * @field:  enum v4l2_field; field order of the image in the buffer
>>>>>> + * @timestamp:      frame timestamp
>>>>>> + * @sequence:       sequence count of this frame
>>>>>> + * @planes: per-plane buffer information
>>>>>> + * @request_fd:     fd of the request that this buffer should use
>>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>>>> + *
>>>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>>>> + * I/O methods.
>>>>>> + */
>>>>>> +struct v4l2_ext_buffer {
>>>>>> +    __u32 index;
>>>>>> +    __u32 type;
>>>>>> +    __u32 field;
>>>>>> +    __u32 sequence;
>>>>>> +    __u64 flags;
>>>>>> +    __u64 timestamp;
>>>>>> +    struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];

I would like your opinion on the following:

We have two concepts here
* memory buffers (that belongs to a frame buffer object aka v4l2_ext_buffer)
* color components/planes (that we need to indicate to userspace where the planes
  are located, which buffer and which offset inside the buffer).

A v4l2_ext_buffer can be reused to a different format if it fits the image
(which is checked in QBUF time, by .buf_prepare() callback).

Which means that, the information regarding where each color component is placed
just make sense after the buffer is queued.

So if userspace calls EXT_QUERYBUF, and the buffer isn't queued, the color component
information doesn't make sense.

One option is to fill in the plane information according to the current
configured format, but only the information returned from EXT_QBUF is guaranteed
to be the correct one.

Another options is to split struct v4l2_ext_plane in two:

struct v4l2_ext_membuffer {
	__u32 memory;
	union {
		__u32 mmap_offset;
		__u64 userptr;
		__s32 dmabuf_fd;
	} m;
	_u32 buffer_length;
}

struct v4l2_ext_plane {
	/*
	 * memory buffer where this plane belongs, index is the position in of
	 * membuffers[] in struct v4l2_ext_buffer below
	 */
	unsigned int membuf_index;
	_u32 plane_length;
	_u32 plane_offset;
}

Then we would have

struct v4l2_ext_buffer {
	... <snip>
	struct v4l2_ext_membuffer membuffers[VIDEO_MAX_PLANES];
	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
	... <snip>
}

Where planes would only be filled by the core only if the buffer is
queued (i.e. "locked" to a given format).

This also avoids having several planes with different dmabuf_fd that are dup()ed,
since we'll have an entry per memory buffer.
Which also avoids the following:
If we are working with a single membuf for all planes for instance, vb2 would need
to know how many planes (let's say there are 3) and repeat the mem buffer information
3 times, and if userspace changes to a pixelformat with 2 color components, we would
repeat 2 times with the same information. And we wouldn't have this issue
if we split both information.

I was also assuming that once the buffer is queued, userspace can't modify
the configured format (I need to check this, but make sense to me due to how
.buf_prepare() works).

What do you think? Does it make sense?


>>>>>> +    __s32 request_fd;
>>>>>> +    __u32 reserved[9];
>>>>>> +};
>>>>>
>>>>> Brainstorming:
>>>>>
>>>>> Some ideas I have to make it easier to support mid stream resolution/colorimetry
>>>>> changes:
>>>>>
>>>>> Adding width and height would support resolution changes (requires the use of
>>>>> CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
>>>>> information is provided here, then there are no race conditions.
>>>>>
>>>>> Same for adding the colorimetry fields here, this too can change on the fly (esp.
>>>>> with HDMI), so reporting this information here avoids race conditions as well.
>>>>
>>>> Right, do you think this is something we can discuss later in a different RFC?
>>>> So we can have a better view on how dynamic resolution change would be used?
>>>>
>>>> We can add more reserved fields or maybe try to do something to what has been
>>>> discussed in about extensible system calls [1]
>>>>
>>>> [1] https://lwn.net/Articles/830666/
>>>>
>>>>>
>>>>> And thirdly, I would like to have a __u64 boot_timestamp field containing the
>>>>> CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
>>>>> is that for m2m devices it is just copied and that for other devices it can have
>>>>> different meanings depending on the timestamp buffer flags.
>>>>>
>>>>> There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
>>>>> to add support for this. That way you know exactly when the driver was finished with
>>>>> the buffer and that helps in detecting missed frames or instrumentation.
>>>>
>>>> I don't mind adding it. Does it make sense to have both timestamp and boot_timestamp?
>>>>
>>>
>>> I think this is quite independent from the ext API work. AFAIR there
>>> was an RFC to request the timestamp source from the userspace by the
>>> flags field in QBUF, which would work with the existing API as well,
>>> or it wasn't posted in the end?
> 
> It's not about selecting a specific clock source. I think that option 4 as described
> below would work for that.
> 
> This problem I'm describing here is specific to m2m devices where the timestamp is
> either just passed through untouched, or it is used as an identifier for a buffer
> for use with stateless decoders.
> 
> In both cases you cannot use the timestamp as a proper timestamp that tells you when
> the buffer was marked done by the driver. So this is about adding a second timestamp
> field (timestamp_done or something like that). Whether this would be hardcoded as using
> CLOCK_BOOTTIME or uses the same clock source as selected through a control is something
> that can be discussed, but since it does require a new field I believe this is part of
> this proposal.

I'm probably lacking m2m knowledge here.
timestamp_done would be when the driver set the buffer as done, what about the other timestamp?
I would like to rename it to make it clear what it means, maybe image_timestamp?
Where, in a capture device, image_timestamp would be the timestamp from the hardware when
it captured that specific frame, and in a decoder, it would be the timestamp of that frame
when it got encoded?
Is this correct?

Thanks
Helen

> 
> Regards,
> 
> 	Hans
> 
>>
>> I was recalling the discussions we had regarding this:
>>
>> 1.
>>     This first attempt in the uvc driver is to use a specific kernel parameter for that case:
>>     https://patchwork.kernel.org/patch/10644887/
>>     The conclusion that the support should be in the core API and not driver specific.
>>
>> 2.
>>     Then an attempt to add global v4l2 support was sent with the Mediatek patch series:
>>     https://patchwork.linuxtv.org/patch/60878/
>>     The major problem is that clock type should be something selectable by userspace, and
>>     not pre-defined by the driver.
>>
>> 3.
>>     Another idea was to use the 'flags' field in the structs v4l2_requestbuffers and
>>     v4l2_create_buffers.
>>     But this field was removed in
>>     129134e5415d ("media: media/v4l2: remove V4L2_FLAG_MEMORY_NON_CONSISTENT flag")
>>     The major concern with this approach was with the uAPI, since it doesn't make much
>>     sense to select a clock when creating buffers.
>>
>> 4.
>>     Another suggestion by Nicolas Dufresne was to add this as a menu control so that userspace
>>     can choose the clock for the timestamps from a given list, the enum in the list can also match
>>     the clocks ids.
>>     We would need to add a new buf flag in struct v4l2_buffer, like V4L2_BUF_FLAG_TIMESTAMP_OTHER,
>>     which would be "as specified through controls ...."
>>
>>
>> So my current question is, should we have both __u32 timestamp and __u32 boottimestamp?
>> Or should we have a mechanism that allows switching from one to the other and use
>> a single field? And if this mechanism should be implemented in both APIs? Can this be
>> defined later?
>>
>>
>> Please, let me know your thoughts.
>>
>> Thanks,
>> Helen
>>
>>>
>>>>>
>>>>>> +
>>>>>>  #ifndef __KERNEL__
>>>>>>  /**
>>>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>>>>>> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
>>>>>>      __u32                   reserved[6];
>>>>>>  };
>>>>>>
>>>>>> +/**
>>>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>>>>>> + * @index:  on return, index of the first created buffer
>>>>>> + * @count:  entry: number of requested buffers,
>>>>>> + *          return: number of created buffers
>>>>>> + * @memory: enum v4l2_memory; buffer memory type
>>>>>> + * @capabilities: capabilities of this buffer type.
>>>>>> + * @format: frame format, for which buffers are requested
>>>>>> + * @flags:  additional buffer management attributes (ignored unless the
>>>>>> + *          queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>>>>>> + *          and configured for MMAP streaming I/O).
>>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>>>> + */
>>>>>> +struct v4l2_ext_create_buffers {
>>>>>> +    __u32                           index;
>>>>>> +    __u32                           count;
>>>>>> +    __u32                           memory;
>>>>>> +    __u32                           capabilities;
>>>>>> +    struct v4l2_ext_pix_format      format;
>>>>>
>>>>> The reality is that the only field that is ever used in the original v4l2_format
>>>>> struct is sizeimage. So this can be replaced with:
>>>>>
>>>>>       __u32                           plane_size[VIDEO_MAX_PLANES];
>>>>>
>>>>> (the field name I picked is debatable, but you get the idea)
>>>>>
>>>>> The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
>>>>> is needed for the current format. The original idea of using struct v4l2_format
>>>>> was that drivers would use the full format information to calculate the
>>>>> memory size, but that was just much too complicated to implement and nobody
>>>>> ever used that. Only the sizeimage field was ever used.
>>>>
>>>> Right, I'll update this in next version, This should simplify things.
>>>>
>>>
>>> I think this might need a bit more discussion. How would the userspace
>>> know what size is enough for the desired resolution? The hardware
>>> and/or drivers often have various alignment/padding restrictions,
>>> which might not be easy to guess for the userspace.
>>>
>>> Also I don't quite understand what's so complicated in handling the
>>> full format, or at least the most important parts of it. The
>>> implementation of TRY_FMT/S_FMT, which exists in every driver, should
>>> already be able to calculate the right plane sizes.
>>>
>>> Best regards,
>>> Tomasz
>>>
>>>>
>>>> Thanks,
>>>> Helen
>>>>
>>>>>
>>>>>> +    __u32                           flags;
>>>>>> +    __u32 reserved[5];
>>>>>> +};
>>>>>> +
>>>>>>  /*
>>>>>>   *  I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>>>>>   *
>>>>>> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
>>>>>>  #define VIDIOC_G_EXT_PIX_FMT        _IOWR('V', 104, struct v4l2_ext_pix_format)
>>>>>>  #define VIDIOC_S_EXT_PIX_FMT        _IOWR('V', 105, struct v4l2_ext_pix_format)
>>>>>>  #define VIDIOC_TRY_EXT_PIX_FMT      _IOWR('V', 106, struct v4l2_ext_pix_format)
>>>>>> +#define VIDIOC_EXT_CREATE_BUFS      _IOWR('V', 107, struct v4l2_ext_create_buffers)
>>>>>> +#define VIDIOC_EXT_QUERYBUF _IOWR('V', 108, struct v4l2_ext_buffer)
>>>>>> +#define VIDIOC_EXT_QBUF             _IOWR('V', 109, struct v4l2_ext_buffer)
>>>>>> +#define VIDIOC_EXT_DQBUF    _IOWR('V', 110, struct v4l2_ext_buffer)
>>>>>> +#define VIDIOC_EXT_PREPARE_BUF      _IOWR('V', 111, struct v4l2_ext_buffer)
>>>>>>
>>>>>>  /* Reminder: when adding new ioctls please add support for them to
>>>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>       Hans
>>>>>
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 3/7] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks
  2020-08-04 19:29 ` [PATCH v5 3/7] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks Helen Koike
@ 2020-12-14  8:52   ` Tomasz Figa
  2020-12-14 12:29     ` Helen Koike
  0 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-12-14  8:52 UTC (permalink / raw)
  To: Helen Koike
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, Boris Brezillon, hiroh, nicolas, Brian.Starkey,
	kernel, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Hi Helen,

On Tue, Aug 04, 2020 at 04:29:35PM -0300, Helen Koike wrote:
> The VB2 layer is used by a lot of drivers. Patch it to support the
> _EXT_PIX_FMT and _EXT_BUF ioctls in order to ease conversion of existing
> drivers to these new APIs.
> 
> Note that internally, the VB2 core is now only using ext structs and old
> APIs are supported through conversion wrappers.

We decided to only support V4L2_BUF_TYPE_VIDEO* for the ext structs. Still,
existing drivers may use vb2 with the other, legacy, buf types. How would
they be handled with this change?

> 
> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> ---
> Changes in v5:
> - Update with new format and buffer structs
> - Updated commit message with the uAPI prefix
> 
> Changes in v4:
> - Update with new format and buffer structs
> - Fix some bugs caught by v4l2-compliance
> - Rebased on top of media/master (post 5.8-rc1)
> 
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
> 
> Changes in v2:
> - New patch
> ---
>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>  .../media/common/videobuf2/videobuf2-v4l2.c   | 560 ++++++++++--------
>  include/media/videobuf2-core.h                |   6 +-
>  include/media/videobuf2-v4l2.h                |  21 +-
>  4 files changed, 345 insertions(+), 244 deletions(-)
> 
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index f544d3393e9d6..d719b1e9c148b 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -1270,6 +1270,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb)
>  		vb->planes[plane].length = 0;
>  		vb->planes[plane].m.fd = 0;
>  		vb->planes[plane].data_offset = 0;
> +		vb->planes[plane].dbuf_offset = 0;
>  
>  		/* Acquire each plane's memory */
>  		mem_priv = call_ptr_memop(vb, attach_dmabuf,
> @@ -1313,6 +1314,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb)
>  		vb->planes[plane].length = planes[plane].length;
>  		vb->planes[plane].m.fd = planes[plane].m.fd;
>  		vb->planes[plane].data_offset = planes[plane].data_offset;
> +		vb->planes[plane].dbuf_offset = planes[plane].dbuf_offset;
>  	}
>  
>  	if (reacquired) {
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index 30caad27281e1..911681d24b3ae 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -29,6 +29,7 @@
>  #include <media/v4l2-fh.h>
>  #include <media/v4l2-event.h>
>  #include <media/v4l2-common.h>
> +#include <media/v4l2-ioctl.h>
>  
>  #include <media/videobuf2-v4l2.h>
>  
> @@ -56,72 +57,39 @@ module_param(debug, int, 0644);
>  				 V4L2_BUF_FLAG_TIMECODE | \
>  				 V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF)
>  
> -/*
> - * __verify_planes_array() - verify that the planes array passed in struct
> - * v4l2_buffer from userspace can be safely used
> - */
> -static int __verify_planes_array(struct vb2_buffer *vb, const struct v4l2_buffer *b)
> -{
> -	if (!V4L2_TYPE_IS_MULTIPLANAR(b->type))
> -		return 0;
> -
> -	/* Is memory for copying plane information present? */
> -	if (b->m.planes == NULL) {
> -		dprintk(vb->vb2_queue, 1,
> -			"multi-planar buffer passed but planes array not provided\n");
> -		return -EINVAL;
> -	}
> -
> -	if (b->length < vb->num_planes || b->length > VB2_MAX_PLANES) {
> -		dprintk(vb->vb2_queue, 1,
> -			"incorrect planes array length, expected %d, got %d\n",
> -			vb->num_planes, b->length);
> -		return -EINVAL;
> -	}
> -
> -	return 0;
> -}
> -
>  static int __verify_planes_array_core(struct vb2_buffer *vb, const void *pb)
>  {
> -	return __verify_planes_array(vb, pb);
> +	return 0;
>  }
>  
>  /*
>   * __verify_length() - Verify that the bytesused value for each plane fits in
>   * the plane length and that the data offset doesn't exceed the bytesused value.
>   */
> -static int __verify_length(struct vb2_buffer *vb, const struct v4l2_buffer *b)
> +static int __verify_length(struct vb2_buffer *vb,
> +			   const struct v4l2_ext_buffer *b)
>  {
>  	unsigned int length;
>  	unsigned int bytesused;
> -	unsigned int plane;
> +	unsigned int i;
>  
>  	if (V4L2_TYPE_IS_CAPTURE(b->type))
>  		return 0;
>  
> -	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
> -		for (plane = 0; plane < vb->num_planes; ++plane) {
> -			length = (b->memory == VB2_MEMORY_USERPTR ||
> -				  b->memory == VB2_MEMORY_DMABUF)
> -			       ? b->m.planes[plane].length
> -				: vb->planes[plane].length;
> -			bytesused = b->m.planes[plane].bytesused
> -				  ? b->m.planes[plane].bytesused : length;
> -
> -			if (b->m.planes[plane].bytesused > length)
> -				return -EINVAL;
> -
> -			if (b->m.planes[plane].data_offset > 0 &&
> -			    b->m.planes[plane].data_offset >= bytesused)
> -				return -EINVAL;
> -		}
> -	} else {
> -		length = (b->memory == VB2_MEMORY_USERPTR ||
> -			  b->memory == VB2_MEMORY_DMABUF)
> -			? b->length : vb->planes[0].length;
> +	for (i = 0; i < vb->num_planes; ++i) {
> +		const struct v4l2_ext_plane *plane = &b->planes[i];
>  
> -		if (b->bytesused > length)
> +		length = (plane->memory == VB2_MEMORY_USERPTR ||
> +			  plane->memory == VB2_MEMORY_DMABUF) ?
> +			 plane->buffer_length :
> +			 vb->planes[i].length;
> +		bytesused = plane->plane_length ?
> +			    plane->plane_length : length;
> +
> +		if (bytesused > length)
> +			return -EINVAL;
> +
> +		if (plane->offset + bytesused > length)
>  			return -EINVAL;
>  	}
>  
> @@ -140,21 +108,12 @@ static void __init_vb2_v4l2_buffer(struct vb2_buffer *vb)
>  
>  static void __copy_timestamp(struct vb2_buffer *vb, const void *pb)
>  {
> -	const struct v4l2_buffer *b = pb;
> -	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> +	const struct v4l2_ext_buffer *b = pb;
>  	struct vb2_queue *q = vb->vb2_queue;
>  
> -	if (q->is_output) {
> -		/*
> -		 * For output buffers copy the timestamp if needed,
> -		 * and the timecode field and flag if needed.
> -		 */
> -		if (q->copy_timestamp)
> -			vb->timestamp = v4l2_buffer_get_timestamp(b);
> -		vbuf->flags |= b->flags & V4L2_BUF_FLAG_TIMECODE;
> -		if (b->flags & V4L2_BUF_FLAG_TIMECODE)
> -			vbuf->timecode = b->timecode;

Is it okay to remove the timecode copying functionality?

> -	}
> +	/* For output buffers copy the timestamp if needed. */
> +	if (q->is_output && q->copy_timestamp)
> +		vb->timestamp = b->timestamp;
>  };
>  
>  static void vb2_warn_zero_bytesused(struct vb2_buffer *vb)
> @@ -173,7 +132,8 @@ static void vb2_warn_zero_bytesused(struct vb2_buffer *vb)
>  		pr_warn("use the actual size instead.\n");
>  }
>  
> -static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b)
> +static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb,
> +				    struct v4l2_ext_buffer *b)
>  {
>  	struct vb2_queue *q = vb->vb2_queue;
>  	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
> @@ -203,110 +163,67 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
>  	vbuf->request_fd = -1;
>  	vbuf->is_held = false;
>  
> -	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
> -		switch (b->memory) {
> -		case VB2_MEMORY_USERPTR:
> -			for (plane = 0; plane < vb->num_planes; ++plane) {
> -				planes[plane].m.userptr =
> -					b->m.planes[plane].m.userptr;
> -				planes[plane].length =
> -					b->m.planes[plane].length;
> -			}
> -			break;
> -		case VB2_MEMORY_DMABUF:
> -			for (plane = 0; plane < vb->num_planes; ++plane) {
> -				planes[plane].m.fd =
> -					b->m.planes[plane].m.fd;
> -				planes[plane].length =
> -					b->m.planes[plane].length;
> -			}
> -			break;
> -		default:
> -			for (plane = 0; plane < vb->num_planes; ++plane) {
> -				planes[plane].m.offset =
> -					vb->planes[plane].m.offset;
> -				planes[plane].length =
> -					vb->planes[plane].length;
> -			}
> -			break;
> +	for (plane = 0; plane < vb->num_planes; ++plane) {
> +		if (b->planes[0].memory != b->planes[plane].memory) {
> +			dprintk(q, 1, "planes should have the same memory type\n");
> +			return -EINVAL;
>  		}
> +	}
>  
> -		/* Fill in driver-provided information for OUTPUT types */
> -		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
> -			/*
> -			 * Will have to go up to b->length when API starts
> -			 * accepting variable number of planes.
> -			 *
> -			 * If bytesused == 0 for the output buffer, then fall
> -			 * back to the full buffer size. In that case
> -			 * userspace clearly never bothered to set it and
> -			 * it's a safe assumption that they really meant to
> -			 * use the full plane sizes.
> -			 *
> -			 * Some drivers, e.g. old codec drivers, use bytesused == 0
> -			 * as a way to indicate that streaming is finished.
> -			 * In that case, the driver should use the
> -			 * allow_zero_bytesused flag to keep old userspace
> -			 * applications working.
> -			 */
> -			for (plane = 0; plane < vb->num_planes; ++plane) {
> -				struct vb2_plane *pdst = &planes[plane];
> -				struct v4l2_plane *psrc = &b->m.planes[plane];
> -
> -				if (psrc->bytesused == 0)
> -					vb2_warn_zero_bytesused(vb);
> -
> -				if (vb->vb2_queue->allow_zero_bytesused)
> -					pdst->bytesused = psrc->bytesused;
> -				else
> -					pdst->bytesused = psrc->bytesused ?
> -						psrc->bytesused : pdst->length;
> -				pdst->data_offset = psrc->data_offset;
> -			}
> +	switch (b->planes[0].memory) {
> +	case VB2_MEMORY_USERPTR:
> +		for (plane = 0; plane < vb->num_planes; ++plane) {
> +			planes[plane].m.userptr = b->planes[plane].m.userptr;
> +			planes[plane].length = b->planes[plane].buffer_length;
>  		}
> -	} else {
> +		break;
> +	case VB2_MEMORY_DMABUF:
> +		for (plane = 0; plane < vb->num_planes; ++plane) {
> +			planes[plane].m.fd = b->planes[plane].m.dmabuf_fd;
> +			planes[plane].dbuf_offset = b->planes[plane].offset;
> +			planes[plane].length = b->planes[plane].buffer_length;
> +		}
> +		break;
> +	default:
> +		for (plane = 0; plane < vb->num_planes; ++plane) {
> +			planes[plane].m.offset = vb->planes[plane].m.offset;
> +			planes[plane].length = vb->planes[plane].length;
> +		}
> +		break;
> +	}
> +
> +	/* Fill in driver-provided information for OUTPUT types */
> +	if (V4L2_TYPE_IS_OUTPUT(b->type)) {
>  		/*
> -		 * Single-planar buffers do not use planes array,
> -		 * so fill in relevant v4l2_buffer struct fields instead.
> -		 * In videobuf we use our internal V4l2_planes struct for
> -		 * single-planar buffers as well, for simplicity.
> +		 * Will have to go up to b->length when API starts
> +		 * accepting variable number of planes.
>  		 *
> -		 * If bytesused == 0 for the output buffer, then fall back
> -		 * to the full buffer size as that's a sensible default.
> +		 * If bytesused == 0 for the output buffer, then fall

Should this be updated with the new field name?

> +		 * back to the full buffer size. In that case
> +		 * userspace clearly never bothered to set it and
> +		 * it's a safe assumption that they really meant to
> +		 * use the full plane sizes.
>  		 *
> -		 * Some drivers, e.g. old codec drivers, use bytesused == 0 as
> -		 * a way to indicate that streaming is finished. In that case,
> -		 * the driver should use the allow_zero_bytesused flag to keep
> -		 * old userspace applications working.
> +		 * Some drivers, e.g. old codec drivers, use bytesused == 0
> +		 * as a way to indicate that streaming is finished.
> +		 * In that case, the driver should use the
> +		 * allow_zero_bytesused flag to keep old userspace
> +		 * applications working.

Do we retain this behavior in the new API? If no, I guess we need to check
if this was old or new API and react appropriately.

>  		 */
> -		switch (b->memory) {
> -		case VB2_MEMORY_USERPTR:
> -			planes[0].m.userptr = b->m.userptr;
> -			planes[0].length = b->length;
> -			break;
> -		case VB2_MEMORY_DMABUF:
> -			planes[0].m.fd = b->m.fd;
> -			planes[0].length = b->length;
> -			break;
> -		default:
> -			planes[0].m.offset = vb->planes[0].m.offset;
> -			planes[0].length = vb->planes[0].length;
> -			break;
> -		}
> +		for (plane = 0; plane < vb->num_planes; ++plane) {
> +			struct vb2_plane *pdst = &planes[plane];
> +			struct v4l2_ext_plane *psrc = &b->planes[plane];
>  
> -		planes[0].data_offset = 0;
> -		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
> -			if (b->bytesused == 0)
> +			if (psrc->plane_length == 0)
>  				vb2_warn_zero_bytesused(vb);
>  
>  			if (vb->vb2_queue->allow_zero_bytesused)
> -				planes[0].bytesused = b->bytesused;
> +				pdst->bytesused = psrc->plane_length;
>  			else
> -				planes[0].bytesused = b->bytesused ?
> -					b->bytesused : planes[0].length;
> -		} else
> -			planes[0].bytesused = 0;
> -
> +				pdst->bytesused = psrc->plane_length ?
> +						  psrc->plane_length :
> +						  pdst->length;
> +		}
>  	}
>  
>  	/* Zero flags that we handle */
> @@ -343,7 +260,7 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
>  
>  static void set_buffer_cache_hints(struct vb2_queue *q,
>  				   struct vb2_buffer *vb,
> -				   struct v4l2_buffer *b)
> +				   struct v4l2_ext_buffer *b)
>  {
>  	/*
>  	 * DMA exporter should take care of cache syncs, so we can avoid
> @@ -388,17 +305,44 @@ static void set_buffer_cache_hints(struct vb2_queue *q,
>  		vb->need_cache_sync_on_prepare = 0;
>  }
>  
> +static enum v4l2_buf_type vb2_ext_qtype(struct vb2_queue *q)
> +{
> +	if (!q->is_multiplanar)
> +		return q->type;

Do we need this explicit check? Wouldn't the code below would return
the right thing anyway?

> +
> +	if (q->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> +		return V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	else if (q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
> +		return V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +
> +	return q->type;
> +}
> +
> +static enum v4l2_buf_type vb2_ext_type(unsigned int type, bool is_multiplanar)
> +{
> +	if (!is_multiplanar)
> +		return type;

Ditto.

> +
> +	if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		return V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> +	else if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
> +		return V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> +
> +	return type;
> +}
> +
>  static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
> -				    struct v4l2_buffer *b, bool is_prepare,
> +				    struct v4l2_ext_buffer *b, bool is_prepare,
>  				    struct media_request **p_req)
>  {
>  	const char *opname = is_prepare ? "prepare_buf" : "qbuf";
>  	struct media_request *req;
>  	struct vb2_v4l2_buffer *vbuf;
>  	struct vb2_buffer *vb;
> +	unsigned int i;
>  	int ret;
>  
> -	if (b->type != q->type) {
> +	if (b->type != vb2_ext_qtype(q)) {
>  		dprintk(q, 1, "%s: invalid buffer type\n", opname);
>  		return -EINVAL;
>  	}
> @@ -414,16 +358,14 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
>  		return -EINVAL;
>  	}
>  
> -	if (b->memory != q->memory) {
> +	for (i = 0; i < VIDEO_MAX_PLANES && b->planes[i].buffer_length; i++)
> +		if (b->planes[i].memory != q->memory) {
>  		dprintk(q, 1, "%s: invalid memory type\n", opname);
>  		return -EINVAL;
>  	}
>  
>  	vb = q->bufs[b->index];
>  	vbuf = to_vb2_v4l2_buffer(vb);
> -	ret = __verify_planes_array(vb, b);
> -	if (ret)
> -		return ret;
>  
>  	if (!is_prepare && (b->flags & V4L2_BUF_FLAG_REQUEST_FD) &&
>  	    vb->state != VB2_BUF_STATE_DEQUEUED) {
> @@ -487,11 +429,6 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
>  		    !q->ops->buf_out_validate))
>  		return -EINVAL;
>  
> -	if (b->request_fd < 0) {
> -		dprintk(q, 1, "%s: request_fd < 0\n", opname);
> -		return -EINVAL;
> -	}
> -

I guess this code was useless already, because the call below would
validate the FD, but should it be perhaps removed in a separate patch?

>  	req = media_request_get_by_fd(mdev, b->request_fd);
>  	if (IS_ERR(req)) {
>  		dprintk(q, 1, "%s: invalid request_fd\n", opname);
> @@ -516,64 +453,46 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
>  }
>  
>  /*
> - * __fill_v4l2_buffer() - fill in a struct v4l2_buffer with information to be
> - * returned to userspace
> + * __fill_v4l2_buffer() - fill in a struct v4l2_ext_buffer with information to
> + * be returned to userspace
>   */
>  static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>  {
> -	struct v4l2_buffer *b = pb;
> +	struct v4l2_ext_buffer *b = pb;
>  	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>  	struct vb2_queue *q = vb->vb2_queue;
>  	unsigned int plane;
>  
>  	/* Copy back data such as timestamp, flags, etc. */
>  	b->index = vb->index;
> -	b->type = vb->type;
> -	b->memory = vb->memory;
> -	b->bytesused = 0;
> +	b->type = vb2_ext_qtype(q);
>  
>  	b->flags = vbuf->flags;
>  	b->field = vbuf->field;
> -	v4l2_buffer_set_timestamp(b, vb->timestamp);
> -	b->timecode = vbuf->timecode;
> +	b->timestamp = vb->timestamp;
>  	b->sequence = vbuf->sequence;
> -	b->reserved2 = 0;
>  	b->request_fd = 0;
> +	memset(b->reserved, 0, sizeof(b->reserved));
>  
> -	if (q->is_multiplanar) {
> -		/*
> -		 * Fill in plane-related data if userspace provided an array
> -		 * for it. The caller has already verified memory and size.
> -		 */
> -		b->length = vb->num_planes;
> -		for (plane = 0; plane < vb->num_planes; ++plane) {
> -			struct v4l2_plane *pdst = &b->m.planes[plane];
> -			struct vb2_plane *psrc = &vb->planes[plane];
> -
> -			pdst->bytesused = psrc->bytesused;
> -			pdst->length = psrc->length;
> -			if (q->memory == VB2_MEMORY_MMAP)
> -				pdst->m.mem_offset = psrc->m.offset;
> -			else if (q->memory == VB2_MEMORY_USERPTR)
> -				pdst->m.userptr = psrc->m.userptr;
> -			else if (q->memory == VB2_MEMORY_DMABUF)
> -				pdst->m.fd = psrc->m.fd;
> -			pdst->data_offset = psrc->data_offset;
> -			memset(pdst->reserved, 0, sizeof(pdst->reserved));
> -		}
> -	} else {
> -		/*
> -		 * We use length and offset in v4l2_planes array even for
> -		 * single-planar buffers, but userspace does not.
> -		 */
> -		b->length = vb->planes[0].length;
> -		b->bytesused = vb->planes[0].bytesused;
> +	/*
> +	 * Fill in plane-related data if userspace provided an array
> +	 * for it. The caller has already verified memory and size.
> +	 */
> +	for (plane = 0; plane < vb->num_planes; ++plane) {
> +		struct v4l2_ext_plane *pdst = &b->planes[plane];
> +		struct vb2_plane *psrc = &vb->planes[plane];
> +
> +		pdst->memory = vb->memory;
> +		pdst->plane_length = psrc->bytesused;
> +		pdst->buffer_length = psrc->length;
> +		pdst->offset = psrc->dbuf_offset;
>  		if (q->memory == VB2_MEMORY_MMAP)
> -			b->m.offset = vb->planes[0].m.offset;
> +			pdst->m.mem_offset = psrc->m.offset;
>  		else if (q->memory == VB2_MEMORY_USERPTR)
> -			b->m.userptr = vb->planes[0].m.userptr;
> +			pdst->m.userptr = psrc->m.userptr;
>  		else if (q->memory == VB2_MEMORY_DMABUF)
> -			b->m.fd = vb->planes[0].m.fd;
> +			pdst->m.dmabuf_fd = psrc->m.fd;
> +		memset(pdst->reserved, 0, sizeof(pdst->reserved));
>  	}
>  
>  	/*
> @@ -668,6 +587,35 @@ int vb2_find_timestamp(const struct vb2_queue *q, u64 timestamp,
>  }
>  EXPORT_SYMBOL_GPL(vb2_find_timestamp);
>  
> +#define vb2_buf_to_ext_buf_op(_name, ...)				\
> +({									\
> +	int ret;							\
> +									\
> +	ret = v4l2_buffer_to_ext_buffer(b, &eb);			\
> +	if (!ret)							\
> +		ret = _name(__VA_ARGS__);				\
> +	if (!ret)							\
> +		ret = v4l2_ext_buffer_to_buffer(&eb, b,			\
> +						q->is_multiplanar);	\

I'm not very happy with this macro implicitly relying on existence of some
specific variables with predefined names in the scope of the caller.

That said, I don't have a good idea on how this could be fixed, other than
just repeating this code manually in each function.

If we don't find anything better, I think at least a comment here
explaining the assumptions would make it a bit better.

> +	ret;								\
> +})
> +
> +int vb2_ext_querybuf(struct vb2_queue *q, struct v4l2_ext_buffer *b)
> +{
> +	if (b->type != vb2_ext_qtype(q)) {
> +		dprintk(q, 1, "wrong buffer type\n");
> +		return -EINVAL;
> +	}
> +
> +	if (b->index >= q->num_buffers) {
> +		dprintk(q, 1, "buffer index out of range\n");
> +		return -EINVAL;
> +	}
> +	vb2_core_querybuf(q, b->index, b);
> +	return 0;
> +}
> +EXPORT_SYMBOL(vb2_ext_querybuf);
> +
>  /*
>   * vb2_querybuf() - query video buffer information
>   * @q:		videobuf queue
> @@ -683,23 +631,9 @@ EXPORT_SYMBOL_GPL(vb2_find_timestamp);
>   */
>  int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  {
> -	struct vb2_buffer *vb;
> -	int ret;
> -
> -	if (b->type != q->type) {
> -		dprintk(q, 1, "wrong buffer type\n");
> -		return -EINVAL;
> -	}
> +	struct v4l2_ext_buffer eb;
>  
> -	if (b->index >= q->num_buffers) {
> -		dprintk(q, 1, "buffer index out of range\n");
> -		return -EINVAL;
> -	}
> -	vb = q->bufs[b->index];
> -	ret = __verify_planes_array(vb, b);
> -	if (!ret)
> -		vb2_core_querybuf(q, b->index, b);
> -	return ret;
> +	return vb2_buf_to_ext_buf_op(vb2_ext_querybuf, q, &eb);
>  }
>  EXPORT_SYMBOL(vb2_querybuf);
>  
> @@ -741,8 +675,8 @@ int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req)
>  }
>  EXPORT_SYMBOL_GPL(vb2_reqbufs);
>  
> -int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
> -		    struct v4l2_buffer *b)
> +int vb2_ext_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
> +			struct v4l2_ext_buffer *b)
>  {
>  	int ret;
>  
> @@ -758,16 +692,59 @@ int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
>  
>  	return ret ? ret : vb2_core_prepare_buf(q, b->index, b);
>  }
> +EXPORT_SYMBOL_GPL(vb2_ext_prepare_buf);
> +
> +int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
> +		    struct v4l2_buffer *b)
> +{
> +	struct v4l2_ext_buffer eb;
> +
> +	return vb2_buf_to_ext_buf_op(vb2_ext_prepare_buf, q, mdev, &eb);
> +}
>  EXPORT_SYMBOL_GPL(vb2_prepare_buf);
>  
> +int vb2_ext_create_bufs(struct vb2_queue *q,
> +			struct v4l2_ext_create_buffers *create)
> +{
> +	unsigned int requested_sizes[VIDEO_MAX_PLANES];
> +	struct v4l2_ext_pix_format *f = &create->format;
> +	int ret = vb2_verify_memory_type(q, create->memory,
> +				vb2_ext_type(f->type, q->is_multiplanar));
> +	unsigned i;
> +
> +	if (create->format.type != V4L2_BUF_TYPE_VIDEO_CAPTURE &&
> +	    create->format.type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
> +		return -EINVAL;
> +
> +	fill_buf_caps(q, &create->capabilities);
> +	clear_consistency_attr(q, create->memory, &create->flags);
> +	create->index = q->num_buffers;
> +	if (create->count == 0)
> +		return ret != -EBUSY ? ret : 0;
> +
> +	if (!f->plane_fmt[0].sizeimage)
> +		return -EINVAL;
> +	for (i = 0; i < VIDEO_MAX_PLANES &&
> +		    f->plane_fmt[i].sizeimage; i++)
> +		requested_sizes[i] = f->plane_fmt[i].sizeimage;
> +
> +	return ret ? ret : vb2_core_create_bufs(q, create->memory,
> +						create->flags,
> +						&create->count,
> +						i,
> +						requested_sizes);
> +}
> +EXPORT_SYMBOL_GPL(vb2_ext_create_bufs);
> +
>  int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
>  {
> +	struct v4l2_format *f = &create->format;
>  	unsigned requested_planes = 1;
>  	unsigned requested_sizes[VIDEO_MAX_PLANES];
> -	struct v4l2_format *f = &create->format;
>  	int ret = vb2_verify_memory_type(q, create->memory, f->type);
>  	unsigned i;
>  
> +
>  	fill_buf_caps(q, &create->capabilities);
>  	clear_consistency_attr(q, create->memory, &create->flags);
>  	create->index = q->num_buffers;
> @@ -777,18 +754,29 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
>  	switch (f->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		requested_planes = f->fmt.pix_mp.num_planes;
> -		if (requested_planes == 0 ||
> -		    requested_planes > VIDEO_MAX_PLANES)
> -			return -EINVAL;
> -		for (i = 0; i < requested_planes; i++)
> -			requested_sizes[i] =
> -				f->fmt.pix_mp.plane_fmt[i].sizeimage;
> -		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -		requested_sizes[0] = f->fmt.pix.sizeimage;
> -		break;
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT: {
> +		struct v4l2_ext_create_buffers ecreate = {
> +			.count = create->count,
> +			.memory = create->memory,
> +			.flags = create->flags,
> +		};
> +
> +		ret = v4l2_format_to_ext_pix_format(&create->format,
> +						    &ecreate.format, true);
> +		if (ret)
> +			return ret;
> +
> +		ret = vb2_ext_create_bufs(q, &ecreate);
> +		if (ret)
> +			return ret;
> +
> +		create->index = ecreate.index;
> +		create->count = ecreate.count;
> +		create->flags = ecreate.flags;
> +		create->capabilities = ecreate.capabilities;
> +		return 0;
> +	}
>  	case V4L2_BUF_TYPE_VBI_CAPTURE:
>  	case V4L2_BUF_TYPE_VBI_OUTPUT:
>  		requested_sizes[0] = f->fmt.vbi.samples_per_line *
> @@ -820,8 +808,8 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
>  }
>  EXPORT_SYMBOL_GPL(vb2_create_bufs);
>  
> -int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
> -	     struct v4l2_buffer *b)
> +int vb2_ext_qbuf(struct vb2_queue *q, struct media_device *mdev,
> +		 struct v4l2_ext_buffer *b)
>  {
>  	struct media_request *req = NULL;
>  	int ret;
> @@ -839,9 +827,19 @@ int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
>  		media_request_put(req);
>  	return ret;
>  }
> +EXPORT_SYMBOL_GPL(vb2_ext_qbuf);
> +
> +int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
> +	     struct v4l2_buffer *b)
> +{
> +	struct v4l2_ext_buffer eb;
> +
> +	return vb2_buf_to_ext_buf_op(vb2_ext_qbuf, q, mdev, &eb);
> +}
>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>  
> -int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
> +int vb2_ext_dqbuf(struct vb2_queue *q, struct v4l2_ext_buffer *b,
> +		  bool nonblocking)
>  {
>  	int ret;
>  
> @@ -850,7 +848,7 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
>  		return -EBUSY;
>  	}
>  
> -	if (b->type != q->type) {
> +	if (b->type != vb2_ext_qtype(q)) {
>  		dprintk(q, 1, "invalid buffer type\n");
>  		return -EINVAL;
>  	}
> @@ -870,6 +868,14 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
>  
>  	return ret;
>  }
> +EXPORT_SYMBOL_GPL(vb2_ext_dqbuf);
> +
> +int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
> +{
> +	struct v4l2_ext_buffer eb;
> +
> +	return vb2_buf_to_ext_buf_op(vb2_ext_dqbuf, q, &eb, nonblocking);
> +}
>  EXPORT_SYMBOL_GPL(vb2_dqbuf);
>  
>  int vb2_streamon(struct vb2_queue *q, enum v4l2_buf_type type)
> @@ -1040,6 +1046,33 @@ int vb2_ioctl_create_bufs(struct file *file, void *priv,
>  }
>  EXPORT_SYMBOL_GPL(vb2_ioctl_create_bufs);
>  
> +int vb2_ioctl_ext_create_bufs(struct file *file, void *priv,
> +			      struct v4l2_ext_create_buffers *p)
> +{
> +	struct video_device *vdev = video_devdata(file);
> +	int res = vb2_verify_memory_type(vdev->queue, p->memory,
> +		vb2_ext_type(p->format.type, vdev->queue->is_multiplanar));
> +
> +	p->index = vdev->queue->num_buffers;
> +	fill_buf_caps(vdev->queue, &p->capabilities);
> +	/*
> +	 * If count == 0, then just check if memory and type are valid.
> +	 * Any -EBUSY result from vb2_verify_memory_type can be mapped to 0.
> +	 */
> +	if (p->count == 0)
> +		return res != -EBUSY ? res : 0;
> +	if (res)
> +		return res;
> +	if (vb2_queue_is_busy(vdev, file))
> +		return -EBUSY;
> +
> +	res = vb2_ext_create_bufs(vdev->queue, p);
> +	if (res == 0)
> +		vdev->queue->owner = file->private_data;
> +	return res;
> +}
> +EXPORT_SYMBOL_GPL(vb2_ioctl_ext_create_bufs);
> +
>  int vb2_ioctl_prepare_buf(struct file *file, void *priv,
>  			  struct v4l2_buffer *p)
>  {
> @@ -1051,6 +1084,17 @@ int vb2_ioctl_prepare_buf(struct file *file, void *priv,
>  }
>  EXPORT_SYMBOL_GPL(vb2_ioctl_prepare_buf);
>  
> +int vb2_ioctl_ext_prepare_buf(struct file *file, void *priv,
> +			      struct v4l2_ext_buffer *p)
> +{
> +	struct video_device *vdev = video_devdata(file);
> +
> +	if (vb2_queue_is_busy(vdev, file))
> +		return -EBUSY;
> +	return vb2_ext_prepare_buf(vdev->queue, vdev->v4l2_dev->mdev, p);
> +}
> +EXPORT_SYMBOL_GPL(vb2_ioctl_ext_prepare_buf);
> +
>  int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p)
>  {
>  	struct video_device *vdev = video_devdata(file);
> @@ -1060,6 +1104,16 @@ int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p)
>  }
>  EXPORT_SYMBOL_GPL(vb2_ioctl_querybuf);
>  
> +int vb2_ioctl_ext_querybuf(struct file *file, void *priv,
> +			   struct v4l2_ext_buffer *p)
> +{
> +	struct video_device *vdev = video_devdata(file);
> +
> +	/* No need to call vb2_queue_is_busy(), anyone can query buffers. */
> +	return vb2_ext_querybuf(vdev->queue, p);
> +}
> +EXPORT_SYMBOL_GPL(vb2_ioctl_ext_querybuf);
> +
>  int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
>  {
>  	struct video_device *vdev = video_devdata(file);
> @@ -1070,6 +1124,17 @@ int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
>  }
>  EXPORT_SYMBOL_GPL(vb2_ioctl_qbuf);
>  
> +int vb2_ioctl_ext_qbuf(struct file *file, void *priv,
> +		       struct v4l2_ext_buffer *p)
> +{
> +	struct video_device *vdev = video_devdata(file);
> +
> +	if (vb2_queue_is_busy(vdev, file))
> +		return -EBUSY;
> +	return vb2_ext_qbuf(vdev->queue, vdev->v4l2_dev->mdev, p);
> +}
> +EXPORT_SYMBOL_GPL(vb2_ioctl_ext_qbuf);
> +
>  int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
>  {
>  	struct video_device *vdev = video_devdata(file);
> @@ -1080,6 +1145,17 @@ int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
>  }
>  EXPORT_SYMBOL_GPL(vb2_ioctl_dqbuf);
>  
> +int vb2_ioctl_ext_dqbuf(struct file *file, void *priv,
> +			struct v4l2_ext_buffer *p)
> +{
> +	struct video_device *vdev = video_devdata(file);
> +
> +	if (vb2_queue_is_busy(vdev, file))
> +		return -EBUSY;
> +	return vb2_ext_dqbuf(vdev->queue, p, file->f_flags & O_NONBLOCK);
> +}
> +EXPORT_SYMBOL_GPL(vb2_ioctl_ext_dqbuf);
> +
>  int vb2_ioctl_streamon(struct file *file, void *priv, enum v4l2_buf_type i)
>  {
>  	struct video_device *vdev = video_devdata(file);
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 52ef92049073e..f334767785bd9 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -152,6 +152,8 @@ struct vb2_mem_ops {
>   * @mem_priv:	private data with this plane.
>   * @dbuf:	dma_buf - shared buffer object.
>   * @dbuf_mapped:	flag to show whether dbuf is mapped or not
> + * @dbuf_offset: offset where the plane starts. Usually 0, unless the buffer
> + *		 is shared by all planes of a multi-planar format.

The comment kind of implies that non-zero is a special case, but I wouldn't
want driver authors to come to incorrect conclusions based on that. Do we
actually need the last sentence at all?

>   * @bytesused:	number of bytes occupied by data in the plane (payload).
>   * @length:	size of this plane (NOT the payload) in bytes.
>   * @min_length:	minimum required size of this plane (NOT the payload) in bytes.
> @@ -175,6 +177,7 @@ struct vb2_plane {
>  	void			*mem_priv;
>  	struct dma_buf		*dbuf;
>  	unsigned int		dbuf_mapped;
> +	unsigned int		dbuf_offset;
>  	unsigned int		bytesused;
>  	unsigned int		length;
>  	unsigned int		min_length;
> @@ -446,7 +449,8 @@ struct vb2_ops {
>   *			struct vb2_buffer.
>   *			For V4L2 this is a &struct vb2_v4l2_buffer.
>   * @fill_user_buffer:	given a &vb2_buffer fill in the userspace structure.
> - *			For V4L2 this is a &struct v4l2_buffer.
> + *			For V4L2 this is a &struct v4l2_buffer or
> + *			&struct v4l2_ext_buffer.

Isn't it always v4l2_ext_buffer with current code?

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)
  2020-11-19 13:43           ` Helen Koike
@ 2020-12-14 10:19             ` Tomasz Figa
  0 siblings, 0 replies; 51+ messages in thread
From: Tomasz Figa @ 2020-12-14 10:19 UTC (permalink / raw)
  To: Helen Koike
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Boris Brezillon,
	Hirokazu Honda, Nicolas Dufresne, Brian Starkey, kernel,
	Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

On Thu, Nov 19, 2020 at 10:43 PM Helen Koike <helen.koike@collabora.com> wrote:
>
>
>
> On 11/19/20 7:08 AM, Helen Koike wrote:
> > Hi Tomasz,
> >
> > On 11/19/20 2:45 AM, Tomasz Figa wrote:
> >> On Sat, Nov 14, 2020 at 11:21:26AM -0300, Helen Koike wrote:
> >>> Hi Tomasz,
> >>>
> >>> On 10/2/20 4:49 PM, Tomasz Figa wrote:
> >>>> Hi Helen,
> >>>>
> >>>> On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
> >> [snip]
> >>>>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
> >>>>> +{
> >>>>> + const struct v4l2_ext_pix_format *pix = arg;
> >>>>> + unsigned int i;
> >>>>> +
> >>>>> + pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
> >>>>> +         prt_names(pix->type, v4l2_type_names),
> >>>>> +         pix->width, pix->height,
> >>>>> +         (pix->pixelformat & 0xff),
> >>>>> +         (pix->pixelformat >>  8) & 0xff,
> >>>>> +         (pix->pixelformat >> 16) & 0xff,
> >>>>> +         (pix->pixelformat >> 24) & 0xff,
> >>>>> +         pix->modifier, prt_names(pix->field, v4l2_field_names),
> >>>>> +         pix->colorspace, pix->ycbcr_enc,
> >>>>> +         pix->quantization, pix->xfer_func);
> >>>>> + for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
> >>>>
> >>>> This is going to print 8 lines every time. Maybe we could skip 0-sized
> >>>> planes or something?
> >>>
> >>> I'm already checking pix->plane_fmt[i].sizeimage in the loop, it shouldn't
> >>> print 8 lines every time.
> >>>
> >>
> >> Oops, how could I not notice it. Sorry for the noise.
> >>
> >> [snip]
> >>>>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> >>>>> +                           struct v4l2_format *f, bool mplane_cap,
> >>>>> +                           bool strict)
> >>>>> +{
> >>>>> + const struct v4l2_plane_ext_pix_format *pe;
> >>>>> + struct v4l2_plane_pix_format *p;
> >>>>> + unsigned int i;
> >>>>> +
> >>>>> + memset(f, 0, sizeof(*f));
> >>>>> +
> >>>>> + /*
> >>>>> +  * Make sure no modifier is required before doing the
> >>>>> +  * conversion.
> >>>>> +  */
> >>>>> + if (e->modifier && strict &&
> >>>>
> >>>> Do we need the explicit check for e->modifier != 0 if we have to check for
> >>>> the 2 specific values below anyway?
> >>>
> >>> We don't, since DRM_FORMAT_MOD_LINEAR is zero.
> >>>
> >>> But I wanted to make it explicit we don't support modifiers in this conversion.
> >>> But I can remove this check, no problem.
> >>>
> >>
> >> Yes, please. I think the double checking is confusing for the reader.
> >
> > ok.
> >
> >>
> >>>>
> >>>>> +     e->modifier != DRM_FORMAT_MOD_LINEAR &&
> >>>>> +     e->modifier != DRM_FORMAT_MOD_INVALID)
> >>>>> +         return -EINVAL;
> >>>>> +
> >>>>> + if (!e->plane_fmt[0].sizeimage && strict)
> >>>>> +         return -EINVAL;
> >>>>
> >>>> Why is this incorrect?
> >>>
> >>> !sizeimage for the first plane means that there are no planes in ef.
> >>> strict is true if the result for the conversion should be returned to userspace
> >>> and it is not some internal handling.
> >>>
> >>> So if there are no planes, we would return an incomplete v4l2_format struct
> >>> to userspace.
> >>>
> >>> But this is not very clear, I'll improve this for the next version.
> >>>
> >>
> >> So I can see 2 cases here:
> >>
> >> 1) Userspace gives ext struct and driver accepts legacy.
> >>
> >> In this case, the kernel needs to adjust the structure to be correct.
> >> -EINVAL is only valid if
> >>
> >> "The struct v4l2_format type field is invalid or the requested buffer type not supported."
> >>
> >> as per the current uAPI documentation.
> >>
> >> 2) Driver gives ext struct and userspace accepts legacy.
> >>
> >> If at this point we get a struct with no planes, that sounds like a
> >> driver bug, so rather than signaling -EINVAL to the userspace, we should
> >> probably WARN()?
> >>
> >> Or am I getting something wrong? :)
> >
> > Make sense, I'll restructure this for the next version.
> >
> >>
> >> [snip]
> >>>>> +{
> >>>>> + const struct v4l2_plane_pix_format *p;
> >>>>> + struct v4l2_plane_ext_pix_format *pe;
> >>>>> + unsigned int i;
> >>>>> +
> >>>>> + memset(e, 0, sizeof(*e));
> >>>>> +
> >>>>> + switch (f->type) {
> >>>>> + case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> >>>>> + case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> >>>>> +         e->width = f->fmt.pix.width;
> >>>>> +         e->height = f->fmt.pix.height;
> >>>>> +         e->pixelformat = f->fmt.pix.pixelformat;
> >>>>> +         e->field = f->fmt.pix.field;
> >>>>> +         e->colorspace = f->fmt.pix.colorspace;
> >>>>> +         if (f->fmt.pix.flags)
> >>>>> +                 pr_warn("Ignoring pixelformat flags 0x%x\n",
> >>>>> +                         f->fmt.pix.flags);
> >>>>
> >>>> Would it make sense to print something like video node name and/or function
> >>>> name to explain where this warning comes from?
> >>>
> >>> I would need to update the function to receive this information, I can try but
> >>> I'm not sure if it is worthy.
> >>>
> >>
> >> I don't have a strong opinion on this, so maybe let's see if others have
> >> any comments.
> >>
> >>>>
> >>>>> +         e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
> >>>>> +         e->quantization = f->fmt.pix.quantization;
> >>>>
> >>>> Missing xfer_func?
> >>>
> >>> Yes, thanks for catching this.
> >>>
> >>>>
> >>>>> +         e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> >>>>> +         e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
> >>>>
> >>>> This doesn't look right. In the ext API we expected the planes to describe
> >>>> color planes, which means that bytesperline needs to be computed for planes
> >>>>> = 1 and sizeimage replaced with per-plane sizes, according to the
> >>>>> pixelformat.
> >>>
> >>> Ack.
> >>>
> >>> Just to be clear, even if we are using a planar format that isn't a V4L2_PIX_FMT_*M
> >>> variant, we should describe every plane separatly.
> >>>
> >>> For instance, if V4L2_PIX_FMT_YVU420 is being used, then f->fmt.pix.bytesperline
> >>> will have data, and we need to calculate bytesperline for all 3 planes, so we'll fill
> >>> out:
> >>>
> >>> f->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> >>> f->plane_fmt[1].bytesperline = f->fmt.pix.bytesperline / hdiv;
> >>> f->plane_fmt[2].bytesperline = f->fmt.pix.bytesperline / hdiv;
> >>>
> >>> I'll update this for the next version.
> >>>
> >>
> >> Yes. This basically gives us a unified representation across all
> >> pixelformats and allows userspace to handle them in a uniform way, as
> >> opposed to current uAPI.
> >
> > Right, I already updated this in my wip branch for next version.
> >
> >>
> >> [snip]
> >>>>> +         if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> >>>>> +                 e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> >>>>> +         else
> >>>>> +                 e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> >>>>> +
> >>>>> +         for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> >>>>> +                 pe = &e->plane_fmt[i];
> >>>>> +                 p = &f->fmt.pix_mp.plane_fmt[i];
> >>>>> +                 pe->bytesperline = p->bytesperline;
> >>>>> +                 pe->sizeimage = p->sizeimage;
> >>>>> +         }
> >>>>
> >>>> Same here. A blind copy is not enough. For non-M formats, the plane
> >>>> parameters need to be filled according to the pixelformat.
> >>>
> >>>
> >>> Right, following the idea above, we need a different handling if we
> >>> aren't using a M-variant of the pixelformat, and we also need to
> >>> convert the pixelformat from the M-variant to non-M-variant.
> >>>
> >>> I'll also need to save that the original format was a
> >>> M-variant or not, so I can convert it back as expected.
> >>
> >> I'm still reading the rest of the series, so it might be answered
> >> already, but did we decide to do anything about the pixelformat codes
> >> themselves? If both M and non-M variants would be allowed with the new
> >> API, then I guess there isn't anything to save, because the original
> >> format would be preserved?
> >
> > I was working with the idea that M-variants wouldn't be allowed.
> > But then, we have cases where non-M-variant don't exist, such as:
> >
> > V4L2_PIX_FMT_YVU422M
> > V4L2_PIX_FMT_YVU444M
> >
> > (at least, I couldn't find non-M-variant equivalent for those)
> >
> > But actually, I don't think we formally decided this (and it seems
> > easier to implement if both are allowed).
> >
> > Should we allow both variants in the Ext API ?
>
> I see 3 options:
>
> 1) Ext API doesn't accept M-variants and return -EINVAL.
>
>     But this doesn't seem to be the v4l2 way, where we avoid returning
>     errors and try to adjust to what we think it is better.
>
>     At the same time, this could allow us, in a very remote hypothetical
>     future situation, to remove the M-variants from the kernel when/if
>     the old API gets obsolete.
>
>     Future ENUM_EXT_FMT won't enumerate M-variants in this option.
>
> 2) Ext API accept M-variants without normalization.
>
>     The driver can receive both variants, and need to handle both as
>     equivalents, i.e. if (V4L2_PIX_FMT_YUV420M || V4L2_PIX_FMT_YUV420)
>
>     Both can be returned to userspace.
>
>     Future ENUM_EXT_FMT can enumerate M-variants in this option.
>
> 3) Ext API accept M-variants with normalization.
>
>     If userspace uses V4L2_PIX_FMT_YUV420M, the framework converts
>     to V4L2_PIX_FMT_YUV420 before passing it to the driver.
>
>     Only V4L2_PIX_FMT_YUV420 is sent back to userspace (even if userspace
>     used the M variant, the kernel normalizes it, which is a similar behavior
>     when userspace try to use a non-supported resolution and the kernel
>     adjusts it).
>
>     (we could also leave this pixelformat normalization to the driver,
>     but I don't see a reason to that)
>
>     Future ENUM_EXT_FMT won't enumerate M-variants in this option.
>
>
>
> I'm leaning towards option 3, please let me know your thoughts.

Same here. I think it would make it easier to support both APIs at the
same time and also allows having M-only formats as is.

The drawback of option 3 is that it would basically carve the non-M
and M fourcc duality in stone and we couldn't clean them up even if we
eventually deprecate the old API after many years. On the other hand,
in practice we might not be able to deprecate the old API anyway, like
what happened with the current single-planar API, so maybe that's
fine? I'd like others' opinions on this.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-11-23 20:33     ` Helen Koike
@ 2020-12-14 10:36       ` Tomasz Figa
  2020-12-14 13:23         ` Helen Koike
  0 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-12-14 10:36 UTC (permalink / raw)
  To: Helen Koike
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Boris Brezillon,
	Hirokazu Honda, Nicolas Dufresne, Brian Starkey, kernel,
	Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

On Tue, Nov 24, 2020 at 5:33 AM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi Tomasz,
>
>
> On 11/20/20 8:14 AM, Tomasz Figa wrote:
> > Hi Helen,
> >
> > On Tue, Aug 04, 2020 at 04:29:34PM -0300, Helen Koike wrote:
> >> From: Hans Verkuil <hans.verkuil@cisco.com>
> >>
> >> Those extended buffer ops have several purpose:
> >> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
> >>    the number of ns elapsed since 1970
> >> 2/ Unify single/multiplanar handling
> >> 3/ Add a new start offset field to each v4l2 plane buffer info struct
> >>    to support the case where a single buffer object is storing all
> >>    planes data, each one being placed at a different offset
> >>
> >> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> >> these new objects.
> >>
> >> The core takes care of converting new ioctls requests to old ones
> >> if the driver does not support the new hooks, and vice versa.
> >>
> >> Note that the timecode field is gone, since there doesn't seem to be
> >> in-kernel users. We can be added back in the reserved area if needed or
> >> use the Request API to collect more metadata information from the
> >> frame.
> >>
> >
> > Thanks for the patch. Please see my comments inline.
>
> Thank you for your detailed review, please see my comments below.
>
> >
> >> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> >> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> >> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >> ---
> >> Changes in v5:
> >> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> >> - return mem_offset to struct v4l2_ext_plane
> >> - change sizes and reorder fields to avoid holes in the struct and make
> >>   it the same for 32 and 64 bits
> >>
> >> Changes in v4:
> >> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> >> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> >> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> >> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> >> I think we can add this later, so I removed it from this RFC to simplify it.
> >> - Remove num_planes field from struct v4l2_ext_buffer
> >> - Add flags field to struct v4l2_ext_create_buffers
> >> - Reformulate struct v4l2_ext_plane
> >> - Fix some bugs caught by v4l2-compliance
> >> - Rebased on top of media/master (post 5.8-rc1)
> >>
> >> Changes in v3:
> >> - Rebased on top of media/master (post 5.4-rc1)
> >>
> >> Changes in v2:
> >> - Add reserved space to v4l2_ext_buffer so that new fields can be added
> >>   later on
> >> ---
> >>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
> >>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
> >>  include/media/v4l2-ioctl.h           |  26 ++
> >>  include/uapi/linux/videodev2.h       |  90 +++++++
> >>  4 files changed, 476 insertions(+), 22 deletions(-)
> >>
> >> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> >> index e1829906bc086..cb21ee8eb075c 100644
> >> --- a/drivers/media/v4l2-core/v4l2-dev.c
> >> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> >> @@ -720,15 +720,34 @@ static void determine_valid_ioctls(struct video_device *vdev)
> >>              SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_sdr_out);
> >>      }
> >>
> >> +    if (is_vid || is_tch) {
> >> +            /* ioctls valid for video and touch */
> >> +            if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
> >> +                    set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
> >> +            if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
> >> +                    set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
> >> +            if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
> >> +                    set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
> >> +            if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
> >> +                    set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
> >> +            if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
> >> +                    set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
> >
> > nit: Could we stick to the SET_VALID_IOCTL() macro and just call it twice,
> > once for the new and once for the legacy callback?
>
> Ack.
>
> >
> >> +    }
> >> +
> >>      if (is_vid || is_vbi || is_sdr || is_tch || is_meta) {
> >>              /* ioctls valid for video, vbi, sdr, touch and metadata */
> >>              SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
> >> -            SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
> >> -            SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
> >>              SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
> >> -            SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
> >> -            SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
> >> -            SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
> >> +            if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
> >> +                    set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
> >> +            if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
> >> +                    set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
> >> +            if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
> >> +                    set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
> >> +            if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
> >> +                    set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
> >> +            if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
> >> +                    set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
> >
> > Is it valid to check the new callbacks for devices that the new API is not
> > valid for (e.g. vbi)? Perhaps we could call SET_VALID_IOCTL(ops, <ioctl>,
> > vidioc_ext_*) in the upper if added in this patch and keep the code above
> > as is?
>
> Just to be clear, the only valid type should be VFL_TYPE_VIDEO right?
> Ext but API won't support touch devices for instance, right?
>

Yes, at least at this point. If one needs, it could be added in the
future, but honestly I don't see much use of the other types these
days.

>
> >
> >>              SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
> >>              SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
> >>      }
> >> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> >> index 14a0def50f8ea..7ecdd9cc1bf48 100644
> >> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> >> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> >> @@ -527,6 +527,26 @@ static void v4l_print_buffer(const void *arg, bool write_only)
> >>                      tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
> >>  }
> >>
> >> +static void v4l_print_ext_buffer(const void *arg, bool write_only)
> >> +{
> >> +    const struct v4l2_ext_buffer *e = arg;
> >> +    const struct v4l2_ext_plane *plane;
> >> +    unsigned int i;
> >> +
> >> +    pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d\n",
> >> +            e->timestamp, e->index, prt_names(e->type, v4l2_type_names),
> >> +            e->flags, prt_names(e->field, v4l2_field_names), e->sequence);
> >
> > Should we also print the request FD?
>
> Yes, I'll update this for next version.
>
> >
> >> +
> >> +    for (i = 0; i < VIDEO_MAX_PLANES &&
> >> +                e->planes[i].buffer_length; i++) {
> >> +            plane = &e->planes[i];
> >> +            pr_debug("plane %d: buffer_length=%d, plane_length=%d offset=0x%08x, memory=%s\n",
> >> +                     i, plane->buffer_length, plane->plane_length,
> >> +                     plane->offset,
> >> +                     prt_names(plane->memory, v4l2_memory_names));
> >
> > Should we also print mem_offset/userptr/dmabuf_fd?
>
> I see they are not printed by v4l_print_buffer(),

offset/userptr are printed in v4l2_print_buffer:
https://elixir.bootlin.com/linux/latest/source/drivers/media/v4l2-core/v4l2-ioctl.c#L494

> since these fields are in an
> union, the value of two of them will be invalid (I wonder if this can bring
> confusion).
> I also wondered if printing them can't cause a security issue.
>
> But I can add those prints if you think it make sense.
>

We know the memory type, so we can interpret the union appropriately
and adjust the message printed.

I don't think this poses any security issue, as it prints things that
belong to the userspace already and are only meaningful in the context
of the given userspace process.

> >
> >> +    }
> >> +}
> >> +
> >>  static void v4l_print_exportbuffer(const void *arg, bool write_only)
> >>  {
> >>      const struct v4l2_exportbuffer *p = arg;
> >> @@ -546,6 +566,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
> >>      v4l_print_format(&p->format, write_only);
> >>  }
> >>
> >> +static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
> >> +{
> >> +    const struct v4l2_ext_create_buffers *p = arg;
> >> +
> >> +    pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
> >> +            prt_names(p->memory, v4l2_memory_names));
> >> +    v4l_print_ext_pix_format(&p->format, write_only);
> >
> > Should we also print capabilities and flags?
>
> I just saw these prints are called after the ioctl handler, and not before,
> to I guess it make sense.
>
> It is not printed by v4l_print_create_buffers(), I think we can add in both then.
>
> >
> >> +}
> >> +
> >>  static void v4l_print_streamparm(const void *arg, bool write_only)
> >>  {
> >>      const struct v4l2_streamparm *p = arg;
> >> @@ -1220,6 +1249,143 @@ int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> >>  }
> >>  EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
> >>
> >> +/*
> >> + * If mplane_cap is true, b->m.planes should have a valid pointer of a
> >> + * struct v4l2_plane array, and b->length with its size
> >> + */
> >> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
> >> +                          struct v4l2_buffer *b, bool mplane_cap)
> >> +{
> >> +    unsigned int planes_array_size = b->length;
> >> +    struct v4l2_plane *planes = b->m.planes;
> >> +    u64 nsecs;
> >> +
> >> +    if (!mplane_cap && e->planes[1].buffer_length != 0)
> >> +            return -EINVAL;
> >> +
> >> +    memset(b, 0, sizeof(*b));
> >> +
> >> +    b->index = e->index;
> >> +    b->flags = e->flags;
> >> +    b->field = e->field;
> >> +    b->sequence = e->sequence;
> >> +    b->memory = e->planes[0].memory;
> >> +    b->request_fd = e->request_fd;
> >> +    b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
> >> +    b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
> >> +
> >> +    if (mplane_cap) {
> >> +            unsigned int i;
> >> +
> >> +            if (!planes || !planes_array_size)
> >> +                    return -EINVAL;
> >> +
> >> +            b->m.planes = planes;
> >
> > planes was initialized to b->m.planes at declaration time. Should we
> > perhaps move its declaration to within this block to make it more clear and
> > remove this assignment?
>
> The variable "planes" is saving the pointer of b->m.planes before we do
> memset(b, 0, sizeof(*b)), so I can reassing it back if it make sense.
>
> I can add a comment to make this more clear.
>

Right, I think the code is a bit confusing. Should we just pass the
planes array pointer as a separate argument to the function?

> >
> >> +
> >> +            if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> >> +                    b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> >> +            else
> >> +                    b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> >> +
> >> +            for (i = 0; i < VIDEO_MAX_PLANES && i < planes_array_size &&
> >> +                        e->planes[i].buffer_length; i++) {
> >> +
> >> +                    if (e->planes[0].memory != e->planes[i].memory)
> >> +                            return -EINVAL;
> >> +
> >> +                    if (e->planes[i].offset)
> >> +                            return -EINVAL;
> >
> > Is it really invalid to have a non-zero offset? Shouldn't the data_offset
> > field of the legacy struct be populated instead, in the cases where it was
> > defined to be valid?
>
> My understanding of data_offset, is that it is used when the hardware can
> write/read a header to/from the buffer.
>
> But this doesn't seem to be used by any driver

This is not true:
https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/qcom/venus/venc.c#L963

>, so there is an attempt to
> repourpose it here:
>
>     https://patchwork.linuxtv.org/project/linux-media/patch/1429040689-23808-2-git-send-email-laurent.pinchart@ideasonboard.com/
>
> But this wans't merged.
>
> So in the current API, there is no way to specify an offset in the buffer.
>
> I guess we can repurpose data_offset first, what do you think?
>

We need to stick to the original behavior for data_offset, so that an
encoder has a way to tell the userspace how many bytes of the header
it needs to skip.

> >
> >> +
> >> +                    memset(&b->m.planes[i], 0, sizeof(b->m.planes[i]));
> >> +
> >> +                    if (b->memory == V4L2_MEMORY_MMAP)
> >> +                            b->m.planes[i].m.mem_offset = e->planes[i].m.mem_offset;
> >> +                    else if (b->memory == V4L2_MEMORY_DMABUF)
> >> +                            b->m.planes[i].m.fd = e->planes[i].m.dmabuf_fd;
> >> +                    else
> >> +                            b->m.planes[i].m.userptr = e->planes[i].m.userptr;
> >> +
> >> +                    b->m.planes[i].bytesused = e->planes[i].plane_length;
> >
> > I might be getting the meaning of plane_length wrong, but doesn't this
> > depend on the direction? If the userspace gives a CAPTURE buffer, it would
> > have bytesused = 0, but if the kernel returns it, it would have bytesused =
> > <size of the payload>.
>
> You are right, it depends on the direction, thanks for catching this.
>
> Also, in the Ext api, we don't have the <size of the payload>, so I'm using the
> plane_length instead, I'm not sure if there is a better way.

I thought about this for a while and it sounds like in the code added
by this series, plane_length is basically used as the old bytesused
and buffer_length as the old length. Would it make sense to just
preserve the old naming?

>
> >
> >> +                    b->m.planes[i].length = e->planes[i].buffer_length;
> >> +            }
> >> +            /* In multi-planar, length contain the number of planes */
> >> +            b->length = i;
> >> +    } else {
> >> +            b->type = e->type;
> >> +            b->bytesused = e->planes[0].plane_length;
> >> +            b->length = e->planes[0].buffer_length;
> >> +
> >> +            if (e->planes[0].offset)
> >> +                    return -EINVAL;
> >
> > Ditto.
>
> Ack.
>
> >
> >> +
> >> +            if (b->memory == V4L2_MEMORY_MMAP)
> >> +                    b->m.offset = e->planes[0].m.mem_offset;
> >> +            else if (b->memory == V4L2_MEMORY_DMABUF)
> >> +                    b->m.fd = e->planes[0].m.dmabuf_fd;
> >> +            else
> >> +                    b->m.userptr = e->planes[0].m.userptr;
> >> +    }
> >> +
> >> +    return 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
> >> +
> >> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
> >> +                          struct v4l2_ext_buffer *e)
> >> +{
> >> +    memset(e, 0, sizeof(*e));
> >> +
> >> +    e->index = b->index;
> >> +    e->flags = b->flags;
> >> +    e->field = b->field;
> >> +    e->sequence = b->sequence;
> >> +    e->request_fd = b->request_fd;
> >> +    e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
> >> +            b->timestamp.tv_usec * NSEC_PER_USEC;
> >> +    if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
> >> +            unsigned int i;
> >> +
> >> +            if (!b->m.planes)
> >> +                    return -EINVAL;
> >> +
> >> +            if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> >> +                    e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> >> +            else
> >> +                    e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> >> +
> >> +            /* In multi-planar, length contain the number of planes */
> >> +            for (i = 0; i < b->length; i++) {
> >
> > The design of the new struct implies that the planes describe color planes
> > and not memory planes, so this code is incorrect, because for non-M format
> > variants it would fill in only the first plane of the new struct.
>
> Yes, as discussed in 1/7, the handling of planes is wrong, I'll correct
> this for next version.
>
> >
> >> +                    if (b->memory == V4L2_MEMORY_MMAP)
> >> +                            e->planes[i].m.mem_offset = b->m.planes[i].m.mem_offset;
> >> +                    else if (b->memory == V4L2_MEMORY_DMABUF)
> >> +                            e->planes[i].m.dmabuf_fd = b->m.planes[i].m.fd;
> >> +                    else
> >> +                            e->planes[i].m.userptr = b->m.planes[i].m.userptr;
> >> +
> >> +                    e->planes[i].memory = b->memory;
> >> +                    e->planes[i].buffer_length = b->m.planes[i].length;
> >> +                    e->planes[i].plane_length = b->m.planes[i].bytesused;
> >> +                    if (b->m.planes[i].data_offset)
> >> +                            pr_warn("Ignoring data_offset value %d\n",
> >> +                                    b->m.planes[i].data_offset);
> >
> > Why? As per my comment above, there are valid use cases defined in the spec.
>
> Please see my comment about about data_offset.
>
> >
> >> +            }
> >> +    } else {
> >> +            e->type = b->type;
> >> +            e->planes[0].memory = b->memory;
> >> +            e->planes[0].plane_length = b->bytesused;
> >> +            e->planes[0].buffer_length = b->length;
> >> +            if (b->memory == V4L2_MEMORY_MMAP)
> >> +                    e->planes[0].m.mem_offset = b->m.offset;
> >> +            else if (b->memory == V4L2_MEMORY_DMABUF)
> >> +                    e->planes[0].m.dmabuf_fd = b->m.fd;
> >> +            else
> >> +                    e->planes[0].m.userptr = b->m.userptr;
> >
> > Similar to the MULTIPLANAR case, we should fill in the planes[] entries
> > corresponding to the number of color planes of the format, e.g. 2 for NV12.
>
> Ack.
>
> >
> >> +    }
> >> +
> >> +    return 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
> >> +
> >>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
> >>                              struct file *file, void *fh, void *arg)
> >>  {
> >> @@ -2473,31 +2639,112 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
> >>      return ops->vidioc_reqbufs(file, fh, p);
> >>  }
> >>
> >> +static int v4l_do_buf_op(int (*op)(struct file *, void *,
> >> +                               struct v4l2_buffer *),
> >> +                     int (*ext_op)(struct file *, void *,
> >> +                                   struct v4l2_ext_buffer *),
> >> +                     struct file *file, void *fh, struct v4l2_buffer *b)
> >> +{
> >> +    struct v4l2_ext_buffer e;
> >> +    int ret;
> >> +
> >> +    ret = check_fmt(file, b->type);
> >> +    if (ret)
> >> +            return ret;
> >> +
> >> +    if (op)
> >> +            return op(file, fh, b);
> >> +
> >> +    ret = v4l2_buffer_to_ext_buffer(b, &e);
> >> +    if (ret)
> >> +            return ret;
> >> +
> >> +    ret = ext_op(file, fh, &e);
> >> +    if (ret)
> >> +            return ret;
> >> +
> >> +    v4l2_ext_buffer_to_buffer(&e, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
> >> +    return 0;
> >> +}
> >> +
> >> +static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
> >> +                                   struct v4l2_buffer *),
> >> +                         int (*ext_op)(struct file *, void *,
> >> +                                       struct v4l2_ext_buffer *),
> >> +                         struct file *file, void *fh,
> >> +                         struct v4l2_ext_buffer *e)
> >> +{
> >> +    struct video_device *vdev = video_devdata(file);
> >> +    struct v4l2_plane planes[VIDEO_MAX_PLANES];
> >> +    struct v4l2_buffer b;
> >> +    bool mplane_cap;
> >> +    int ret;
> >> +
> >> +    ret = check_fmt(file, e->type);
> >> +    if (ret)
> >> +            return ret;
> >> +
> >> +    if (ext_op)
> >> +            return ext_op(file, fh, e);
> >> +
> >> +    mplane_cap = !!(vdev->device_caps &
> >> +                    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> >> +                     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> >> +                     V4L2_CAP_VIDEO_M2M_MPLANE));
> >> +    b.m.planes = planes;
> >> +    b.length = VIDEO_MAX_PLANES;
> >> +    ret = v4l2_ext_buffer_to_buffer(e, &b, mplane_cap);
> >> +    if (ret)
> >> +            return ret;
> >> +
> >> +    ret = op(file, fh, &b);
> >> +    if (ret)
> >> +            return ret;
> >> +
> >> +    v4l2_buffer_to_ext_buffer(&b, e);
> >> +    return 0;
> >> +}
> >> +
> >>  static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
> >> -                            struct file *file, void *fh, void *arg)
> >> +                    struct file *file, void *fh, void *arg)
> >>  {
> >> -    struct v4l2_buffer *p = arg;
> >> -    int ret = check_fmt(file, p->type);
> >> +    return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
> >> +                         file, fh, arg);
> >> +}
> >>
> >> -    return ret ? ret : ops->vidioc_querybuf(file, fh, p);
> >> +static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
> >> +                        struct file *file, void *fh, void *arg)
> >> +{
> >> +    return v4l_do_ext_buf_op(ops->vidioc_querybuf,
> >> +                             ops->vidioc_ext_querybuf, file, fh, arg);
> >>  }
> >>
> >>  static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
> >> -                            struct file *file, void *fh, void *arg)
> >> +                struct file *file, void *fh, void *arg)
> >>  {
> >> -    struct v4l2_buffer *p = arg;
> >> -    int ret = check_fmt(file, p->type);
> >> +    return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
> >> +                         file, fh, arg);
> >> +}
> >>
> >> -    return ret ? ret : ops->vidioc_qbuf(file, fh, p);
> >> +static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
> >> +                    struct file *file, void *fh, void *arg)
> >> +{
> >> +    return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
> >> +                             file, fh, arg);
> >>  }
> >>
> >>  static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
> >> -                            struct file *file, void *fh, void *arg)
> >> +                 struct file *file, void *fh, void *arg)
> >>  {
> >> -    struct v4l2_buffer *p = arg;
> >> -    int ret = check_fmt(file, p->type);
> >> +    return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
> >> +                         file, fh, arg);
> >> +}
> >>
> >> -    return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
> >> +static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
> >> +                     struct file *file, void *fh, void *arg)
> >> +{
> >> +    return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
> >> +                             file, fh, arg);
> >>  }
> >>
> >>  static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
> >> @@ -2513,7 +2760,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
> >>
> >>      v4l_sanitize_format(&create->format);
> >>
> >> -    ret = ops->vidioc_create_bufs(file, fh, create);
> >> +    if (ops->vidioc_create_bufs) {
> >> +            ret = ops->vidioc_create_bufs(file, fh, create);
> >> +    } else {
> >> +            struct v4l2_ext_create_buffers ecreate = {
> >> +                    .count = create->count,
> >> +                    .memory = create->memory,
> >> +            };
> >> +
> >> +            ret = v4l2_format_to_ext_pix_format(&create->format,
> >> +                                                &ecreate.format, true);
> >> +            if (ret)
> >> +                    return ret;
> >> +
> >> +            ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
> >> +            if (ret)
> >> +                    return ret;
> >> +
> >> +            create->index = ecreate.index;
> >> +            create->count = ecreate.count;
> >> +            create->capabilities = ecreate.capabilities;
> >> +    }
> >>
> >>      if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
> >>          create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
> >> @@ -2522,13 +2789,60 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
> >>      return ret;
> >>  }
> >>
> >> +static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
> >> +                           struct file *file, void *fh, void *arg)
> >> +{
> >> +    struct v4l2_ext_create_buffers *ecreate = arg;
> >> +    struct video_device *vdev = video_devdata(file);
> >> +    struct v4l2_create_buffers create = {
> >> +            .count = ecreate->count,
> >> +            .memory = ecreate->memory,
> >> +            .flags = ecreate->flags,
> >> +    };
> >> +    bool mplane_cap;
> >> +    int ret;
> >> +
> >> +    ret = check_fmt(file, ecreate->format.type);
> >> +    if (ret)
> >> +            return ret;
> >> +
> >> +    if (ops->vidioc_ext_create_bufs)
> >> +            return ops->vidioc_ext_create_bufs(file, fh, ecreate);
> >> +
> >> +    mplane_cap = !!(vdev->device_caps &
> >> +                    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> >> +                     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> >> +                     V4L2_CAP_VIDEO_M2M_MPLANE));
> >> +    ret = v4l2_ext_pix_format_to_format(&ecreate->format,
> >> +                                        &create.format, mplane_cap, true);
> >> +    if (ret)
> >> +            return ret;
> >> +
> >> +    ret = v4l_create_bufs(ops, file, fh, &create);
> >> +    if (ret)
> >> +            return ret;
> >> +
> >> +    ecreate->index = create.index;
> >> +    ecreate->count = create.count;
> >> +    ecreate->capabilities = create.capabilities;
> >> +
> >> +    return 0;
> >> +}
> >> +
> >>  static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
> >> -                            struct file *file, void *fh, void *arg)
> >> +                       struct file *file, void *fh, void *arg)
> >>  {
> >> -    struct v4l2_buffer *b = arg;
> >> -    int ret = check_fmt(file, b->type);
> >> +    return v4l_do_buf_op(ops->vidioc_prepare_buf,
> >> +                         ops->vidioc_ext_prepare_buf,
> >> +                         file, fh, arg);
> >> +}
> >>
> >> -    return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
> >> +static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
> >> +                           struct file *file, void *fh, void *arg)
> >> +{
> >> +    return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
> >> +                             ops->vidioc_ext_prepare_buf,
> >> +                             file, fh, arg);
> >>  }
> >>
> >>  static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
> >> @@ -3202,12 +3516,15 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
> >>      IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
> >>      IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
> >>      IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
> >> +    IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, planes)),
> >>      IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
> >>      IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
> >>      IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
> >>      IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
> >>      IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
> >> +    IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
> >
> > Looking at the other entries, shouldn't this one be 1 line higher?
>
> Yes.
>
> >
> > That said, I wonder if it wouldn't look cleaner if we just put all the
> > EXT ioctls together at the bottom.
>
> I can move them and we can see if it is better or not.
>
> >
> >>      IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
> >> +    IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
> >>      IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
> >>      IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
> >>      IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
> >> @@ -3272,7 +3589,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
> >>      IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
> >>      IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
> >>      IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
> >> +    IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
> >>      IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
> >> +    IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
> >>      IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
> >>      IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
> >>      IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
> >> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
> >> index 8bbcb74d8ee31..75996657ad1ba 100644
> >> --- a/include/media/v4l2-ioctl.h
> >> +++ b/include/media/v4l2-ioctl.h
> >> @@ -169,16 +169,26 @@ struct v4l2_fh;
> >>   *  :ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
> >>   * @vidioc_querybuf: pointer to the function that implements
> >>   *  :ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
> >> + * @vidioc_ext_querybuf: pointer to the function that implements
> >> + *  :ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
> >>   * @vidioc_qbuf: pointer to the function that implements
> >>   *  :ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
> >> + * @vidioc_ext_qbuf: pointer to the function that implements
> >> + *  :ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
> >>   * @vidioc_expbuf: pointer to the function that implements
> >>   *  :ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
> >>   * @vidioc_dqbuf: pointer to the function that implements
> >>   *  :ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
> >> + * @vidioc_ext_dqbuf: pointer to the function that implements
> >> + *  :ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
> >>   * @vidioc_create_bufs: pointer to the function that implements
> >>   *  :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
> >> + * @vidioc_ext_create_bufs: pointer to the function that implements
> >> + *  :ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
> >>   * @vidioc_prepare_buf: pointer to the function that implements
> >>   *  :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
> >> + * @vidioc_ext_prepare_buf: pointer to the function that implements
> >> + *  :ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
> >>   * @vidioc_overlay: pointer to the function that implements
> >>   *  :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
> >>   * @vidioc_g_fbuf: pointer to the function that implements
> >> @@ -439,17 +449,27 @@ struct v4l2_ioctl_ops {
> >>                            struct v4l2_requestbuffers *b);
> >>      int (*vidioc_querybuf)(struct file *file, void *fh,
> >>                             struct v4l2_buffer *b);
> >> +    int (*vidioc_ext_querybuf)(struct file *file, void *fh,
> >> +                               struct v4l2_ext_buffer *b);
> >>      int (*vidioc_qbuf)(struct file *file, void *fh,
> >>                         struct v4l2_buffer *b);
> >> +    int (*vidioc_ext_qbuf)(struct file *file, void *fh,
> >> +                           struct v4l2_ext_buffer *b);
> >>      int (*vidioc_expbuf)(struct file *file, void *fh,
> >>                           struct v4l2_exportbuffer *e);
> >>      int (*vidioc_dqbuf)(struct file *file, void *fh,
> >>                          struct v4l2_buffer *b);
> >> +    int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
> >> +                            struct v4l2_ext_buffer *b);
> >>
> >>      int (*vidioc_create_bufs)(struct file *file, void *fh,
> >>                                struct v4l2_create_buffers *b);
> >> +    int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
> >> +                                  struct v4l2_ext_create_buffers *b);
> >>      int (*vidioc_prepare_buf)(struct file *file, void *fh,
> >>                                struct v4l2_buffer *b);
> >> +    int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
> >> +                                  struct v4l2_ext_buffer *b);
> >>
> >>      int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
> >>      int (*vidioc_g_fbuf)(struct file *file, void *fh,
> >> @@ -758,6 +778,12 @@ int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> >>                                struct v4l2_format *f,
> >>                                bool mplane_cap, bool strict);
> >>
> >> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
> >> +                          struct v4l2_buffer *b,
> >> +                          bool mplane_cap);
> >> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
> >> +                          struct v4l2_ext_buffer *e);
> >> +
> >>  /*
> >>   * The user space interpretation of the 'v4l2_event' differs
> >>   * based on the 'time_t' definition on 32-bit architectures, so
> >> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> >> index 7123c6a4d9569..334cafdd2be97 100644
> >> --- a/include/uapi/linux/videodev2.h
> >> +++ b/include/uapi/linux/videodev2.h
> >> @@ -996,6 +996,41 @@ struct v4l2_plane {
> >>      __u32                   reserved[11];
> >>  };
> >>
> >> +/**
> >> + * struct v4l2_ext_plane - extended plane buffer info
> >> + * @buffer_length:  size of the entire buffer in bytes, should fit
> >> + *                  @offset + @plane_length
> >
> > Do we actually need this buffer_length at all? We have 3 memory types:
> >
> > 1) MMAP - here vb2 already knows the buffer size, because it created it.
> >
> > 2) DMABUF - the DMA-buf kAPI provides the information about buffer size.
> >
> > 3) USERPTR - this might actually benefit from buffer_length, because there
> >    are additional alignmnent requirements for the user memory, e.g. the
> >    offset and size must be cacheline aligned.
> >
> > Arguably, 1) and 2) are the main usage scenarios, while the user space that
> > uses them would have to suffer from added complexity, because of the
> > legacy/niche case 3).
> >
> > Could we make this field valid only for USERPTR?
>
> I think so, make sense, I'll implement this for next version.
>
> >
> >> + * @plane_length:   size of the plane in bytes.
> >> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
> >> + *                  that should be passed to mmap() called on the video node.
> >> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> >> + *                  to this plane.
> >> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> >> + *                  associated with this plane.
> >> + * @offset:         offset in the memory buffer where the plane starts.
> >> + * @memory:         enum v4l2_memory; the method, in which the actual video
> >> + *                  data is passed
> >> + * @reserved:               extra space reserved for future fields, must be set to 0.
> >> + *
> >> + *
> >> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> >> + * can have one plane for Y, and another for interleaved CbCr components.
> >> + * Each plane can reside in a separate memory buffer, or even in
> >> + * a completely separate memory node (e.g. in embedded devices).
> >> + */
> >> +struct v4l2_ext_plane {
> >> +    __u32 buffer_length;
> >> +    __u32 plane_length;
> >> +    union {
> >> +            __u32 mem_offset;
> >> +            __u64 userptr;
> >> +            __s32 dmabuf_fd;
> >> +    } m;
> >> +    __u32 offset;
> >> +    __u32 memory;
> >> +    __u32 reserved[4];
> >> +};
> >
> > Don't we also need bytesused? Or would plane_length essentially mean the
> > amount of space or payload, depending on the usage context?
>
> In my understanding, plane_length is the max amount of data the plane can
> occupy, othersize it can overflow the buffer or mess with another plane
> that is in the same buffer on a different offset.
>
> I'm probably wrong, but I don't really see why the payload size is usefull,
> unless if we set a plane_legth that is much bigger then the data it can carry,
> that can impact performance.
> Payload can also be calculated from the format.

It is required for non-2D-image formats, such as compressed bitstream.
In that case, the size of the image varies between frames and is
usually less than the size of the buffer.

>
> I can add it back if it is usefull. Please let me know your thoughts.
>

As I mentioned above, it looks like plane_length is used almost
exactly the same way bytesused was in the original code, so maybe it
could just stay this way?

> >
> > Similarly, the original data_offset was useful as a return field, which
> > some drivers use to indicate that the beginning of the plane is occupied by
> > some header or otherwise irrelevant data, which must be skipped. Would the
> > offset field be used for this purpose now?
>
> I didn't add an equivalent of the data_offset, since it seemed to be
> unused (please see my comments about this above).
>
> >
> >> +
> >>  /**
> >>   * struct v4l2_buffer - video buffer info
> >>   * @index:  id number of the buffer
> >> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
> >>      };
> >>  };
> >>
> >> +/**
> >> + * struct v4l2_ext_buffer - extended video buffer info
> >> + * @index:  id number of the buffer
> >> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> >> + * @flags:  buffer informational flags
> >
> > nit: The order of comments doesn't match the order of fields in the struct.
>
> Ack.
>
> >
> >> + * @field:  enum v4l2_field; field order of the image in the buffer
> >> + * @timestamp:      frame timestamp
> >> + * @sequence:       sequence count of this frame
> >> + * @planes: per-plane buffer information
> >> + * @request_fd:     fd of the request that this buffer should use
> >> + * @reserved:       extra space reserved for future fields, must be set to 0
> >> + *
> >> + * Contains data exchanged by application and driver using one of the Streaming
> >> + * I/O methods.
> >> + */
> >> +struct v4l2_ext_buffer {
> >> +    __u32 index;
> >> +    __u32 type;
> >> +    __u32 field;
> >> +    __u32 sequence;
> >> +    __u64 flags;
> >> +    __u64 timestamp;
> >
> > What's the unit? How does this play with the other UAPI that the user space
> > may use, e.g. clock_gettime() which returns struct timespec?
>
> The unity is nsec:
>
>         e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
>                 b->timestamp.tv_usec * NSEC_PER_USEC;
>
> I can clarify in the docs, is this ok?
>

Yes, it definitely needs to be documented. That said, what's the
rationale for switching from the timeval representation to the flat
nsec-based one?

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-03 15:11           ` Hans Verkuil
  2020-12-03 19:52             ` Helen Koike
@ 2020-12-14 10:38             ` Tomasz Figa
  1 sibling, 0 replies; 51+ messages in thread
From: Tomasz Figa @ 2020-12-14 10:38 UTC (permalink / raw)
  To: Hans Verkuil
  Cc: Helen Koike, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

On Fri, Dec 4, 2020 at 12:11 AM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> On 23/11/2020 18:40, Helen Koike wrote:
> >
> >
> > On 11/23/20 12:46 PM, Tomasz Figa wrote:
> >> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>
> >>> Hi Hans,
> >>>
> >>> Thank you for your review.
> >>>
> >>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
> >>>> Hi Helen,
> >>>>
> >>>> Again I'm just reviewing the uAPI.
> >>>>
> >>>> On 04/08/2020 21:29, Helen Koike wrote:
> >>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>>
> >>>>> Those extended buffer ops have several purpose:
> >>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
> >>>>>    the number of ns elapsed since 1970
> >>>>> 2/ Unify single/multiplanar handling
> >>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
> >>>>>    to support the case where a single buffer object is storing all
> >>>>>    planes data, each one being placed at a different offset
> >>>>>
> >>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> >>>>> these new objects.
> >>>>>
> >>>>> The core takes care of converting new ioctls requests to old ones
> >>>>> if the driver does not support the new hooks, and vice versa.
> >>>>>
> >>>>> Note that the timecode field is gone, since there doesn't seem to be
> >>>>> in-kernel users. We can be added back in the reserved area if needed or
> >>>>> use the Request API to collect more metadata information from the
> >>>>> frame.
> >>>>>
> >>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> >>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>>>> ---
> >>>>> Changes in v5:
> >>>>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> >>>>> - return mem_offset to struct v4l2_ext_plane
> >>>>> - change sizes and reorder fields to avoid holes in the struct and make
> >>>>>   it the same for 32 and 64 bits
> >>>>>
> >>>>> Changes in v4:
> >>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> >>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> >>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> >>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> >>>>> I think we can add this later, so I removed it from this RFC to simplify it.
> >>>>> - Remove num_planes field from struct v4l2_ext_buffer
> >>>>> - Add flags field to struct v4l2_ext_create_buffers
> >>>>> - Reformulate struct v4l2_ext_plane
> >>>>> - Fix some bugs caught by v4l2-compliance
> >>>>> - Rebased on top of media/master (post 5.8-rc1)
> >>>>>
> >>>>> Changes in v3:
> >>>>> - Rebased on top of media/master (post 5.4-rc1)
> >>>>>
> >>>>> Changes in v2:
> >>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
> >>>>>   later on
> >>>>> ---
> >>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
> >>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
> >>>>>  include/media/v4l2-ioctl.h           |  26 ++
> >>>>>  include/uapi/linux/videodev2.h       |  90 +++++++
> >>>>>  4 files changed, 476 insertions(+), 22 deletions(-)
> >>>>>
> >>>>
> >>>> <snip>
> >>>>
> >>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> >>>>> index 7123c6a4d9569..334cafdd2be97 100644
> >>>>> --- a/include/uapi/linux/videodev2.h
> >>>>> +++ b/include/uapi/linux/videodev2.h
> >>>>> @@ -996,6 +996,41 @@ struct v4l2_plane {
> >>>>>      __u32                   reserved[11];
> >>>>>  };
> >>>>>
> >>>>> +/**
> >>>>> + * struct v4l2_ext_plane - extended plane buffer info
> >>>>> + * @buffer_length:  size of the entire buffer in bytes, should fit
> >>>>> + *                  @offset + @plane_length
> >>>>> + * @plane_length:   size of the plane in bytes.
> >>>>> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
> >>>>> + *                  that should be passed to mmap() called on the video node.
> >>>>> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> >>>>> + *                  to this plane.
> >>>>> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> >>>>> + *                  associated with this plane.
> >>>>> + * @offset:         offset in the memory buffer where the plane starts.
> >>>>> + * @memory:         enum v4l2_memory; the method, in which the actual video
> >>>>> + *                  data is passed
> >>>>> + * @reserved:               extra space reserved for future fields, must be set to 0.
> >>>>> + *
> >>>>> + *
> >>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> >>>>> + * can have one plane for Y, and another for interleaved CbCr components.
> >>>>> + * Each plane can reside in a separate memory buffer, or even in
> >>>>> + * a completely separate memory node (e.g. in embedded devices).
> >>>>> + */
> >>>>> +struct v4l2_ext_plane {
> >>>>> +    __u32 buffer_length;
> >>>>> +    __u32 plane_length;
> >>>>> +    union {
> >>>>> +            __u32 mem_offset;
> >>>>> +            __u64 userptr;
> >>>>> +            __s32 dmabuf_fd;
> >>>>> +    } m;
> >>>>> +    __u32 offset;
> >>>>
> >>>> I'd rename this plane_offset. I think some reordering would make this struct easier
> >>>> to understand:
> >>>>
> >>>> struct v4l2_ext_plane {
> >>>>       __u32 buffer_length;
> >>>>       __u32 plane_offset;
> >>>>       __u32 plane_length;
> >>>>       __u32 memory;
> >>>>       union {
> >>>>               __u32 mem_offset;
> >>>>               __u64 userptr;
> >>>>               __s32 dmabuf_fd;
> >>>>       } m;
> >>>>       __u32 reserved[4];
> >>>> };
> >>>>
> >>>>> +    __u32 memory;
> >>>>> +    __u32 reserved[4];
> >>>>> +};
> >>>
> >>> Ok, I'll apply this to the next version.
> >>>
> >>>>
> >>>> What is not clear is how to tell the different between a single buffer containing
> >>>> multiple planes, and using a separate buffer per plane. E.g. what would this look
> >>>> like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
> >>>> V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
> >>>> planes are also combined in a single buffer?
> >>>>
> >>>> I would guess that the m union is set to 0 if the plane is part of the buffer
> >>>> defined in the previous plane?
> >>>
> >>> The difference would be if m are equal or differ between planes, example:
> >>>
> >>> For V4L2_PIX_FMT_YVU420:
> >>>
> >>>     Y:
> >>>         plane_offset = 0
> >>>         m.dmabuf_fd = 3
> >>>     Cb:
> >>>         plane_offset = 300
> >>>         m.dmabuf_fd = 3
> >>>     Cr:
> >>>         plane_offset = 375
> >>>         m.dmabuf_fd = 3
> >>>
> >>> For V4L2_PIX_FMT_YVU420M:
> >>>
> >>>     Y:
> >>>         plane_offset = 0
> >>>         m.dmabuf_fd = 4
> >>>     Cb:
> >>>         plane_offset = 0
> >>>         m.dmabuf_fd = 5
> >>>     Cr:
> >>>         plane_offset = 0
> >>>         m.dmabuf_fd = 6
> >>>
> >>>
> >>> Does it make sense?
> >>>
> >>
> >> Actually all the 3 file descriptors can still point to the same
> >> buffer, because they might have been dup()ed. The kernel needs to
> >> resolve the file descriptors into struct dma_buf and then check
> >> whether it's one or more buffers.
> >
> > Right, thanks for this.
> >
> >>
> >> In fact, dup()ed FD for each plane is quite a common case in other
> >> APIs, e.g. EGL, but current V4L2 API can't handle it. In Chromium we
> >> basically work around it by assuming that if we receive a buffer for a
> >> V4L2 device that only supports non-M formats, then we can safely
> >> ignore all but first FD. The new API gives the ability to handle the
> >> case properly, with full validation by the kernel.
> >>
> >>>>
> >>>>> +
> >>>>>  /**
> >>>>>   * struct v4l2_buffer - video buffer info
> >>>>>   * @index:  id number of the buffer
> >>>>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
> >>>>>      };
> >>>>>  };
> >>>>>
> >>>>> +/**
> >>>>> + * struct v4l2_ext_buffer - extended video buffer info
> >>>>> + * @index:  id number of the buffer
> >>>>> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> >>>>> + * @flags:  buffer informational flags
> >>>>> + * @field:  enum v4l2_field; field order of the image in the buffer
> >>>>> + * @timestamp:      frame timestamp
> >>>>> + * @sequence:       sequence count of this frame
> >>>>> + * @planes: per-plane buffer information
> >>>>> + * @request_fd:     fd of the request that this buffer should use
> >>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
> >>>>> + *
> >>>>> + * Contains data exchanged by application and driver using one of the Streaming
> >>>>> + * I/O methods.
> >>>>> + */
> >>>>> +struct v4l2_ext_buffer {
> >>>>> +    __u32 index;
> >>>>> +    __u32 type;
> >>>>> +    __u32 field;
> >>>>> +    __u32 sequence;
> >>>>> +    __u64 flags;
> >>>>> +    __u64 timestamp;
> >>>>> +    struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> >>>>> +    __s32 request_fd;
> >>>>> +    __u32 reserved[9];
> >>>>> +};
> >>>>
> >>>> Brainstorming:
> >>>>
> >>>> Some ideas I have to make it easier to support mid stream resolution/colorimetry
> >>>> changes:
> >>>>
> >>>> Adding width and height would support resolution changes (requires the use of
> >>>> CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
> >>>> information is provided here, then there are no race conditions.
> >>>>
> >>>> Same for adding the colorimetry fields here, this too can change on the fly (esp.
> >>>> with HDMI), so reporting this information here avoids race conditions as well.
> >>>
> >>> Right, do you think this is something we can discuss later in a different RFC?
> >>> So we can have a better view on how dynamic resolution change would be used?
> >>>
> >>> We can add more reserved fields or maybe try to do something to what has been
> >>> discussed in about extensible system calls [1]
> >>>
> >>> [1] https://lwn.net/Articles/830666/
> >>>
> >>>>
> >>>> And thirdly, I would like to have a __u64 boot_timestamp field containing the
> >>>> CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
> >>>> is that for m2m devices it is just copied and that for other devices it can have
> >>>> different meanings depending on the timestamp buffer flags.
> >>>>
> >>>> There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
> >>>> to add support for this. That way you know exactly when the driver was finished with
> >>>> the buffer and that helps in detecting missed frames or instrumentation.
> >>>
> >>> I don't mind adding it. Does it make sense to have both timestamp and boot_timestamp?
> >>>
> >>
> >> I think this is quite independent from the ext API work. AFAIR there
> >> was an RFC to request the timestamp source from the userspace by the
> >> flags field in QBUF, which would work with the existing API as well,
> >> or it wasn't posted in the end?
>
> It's not about selecting a specific clock source. I think that option 4 as described
> below would work for that.
>
> This problem I'm describing here is specific to m2m devices where the timestamp is
> either just passed through untouched, or it is used as an identifier for a buffer
> for use with stateless decoders.
>
> In both cases you cannot use the timestamp as a proper timestamp that tells you when
> the buffer was marked done by the driver. So this is about adding a second timestamp
> field (timestamp_done or something like that). Whether this would be hardcoded as using
> CLOCK_BOOTTIME or uses the same clock source as selected through a control is something
> that can be discussed, but since it does require a new field I believe this is part of
> this proposal.

What would be the use case for that value?

Best regards,
Tomasz

>
> Regards,
>
>         Hans
>
> >
> > I was recalling the discussions we had regarding this:
> >
> > 1.
> >     This first attempt in the uvc driver is to use a specific kernel parameter for that case:
> >     https://patchwork.kernel.org/patch/10644887/
> >     The conclusion that the support should be in the core API and not driver specific.
> >
> > 2.
> >     Then an attempt to add global v4l2 support was sent with the Mediatek patch series:
> >     https://patchwork.linuxtv.org/patch/60878/
> >     The major problem is that clock type should be something selectable by userspace, and
> >     not pre-defined by the driver.
> >
> > 3.
> >     Another idea was to use the 'flags' field in the structs v4l2_requestbuffers and
> >     v4l2_create_buffers.
> >     But this field was removed in
> >     129134e5415d ("media: media/v4l2: remove V4L2_FLAG_MEMORY_NON_CONSISTENT flag")
> >     The major concern with this approach was with the uAPI, since it doesn't make much
> >     sense to select a clock when creating buffers.
> >
> > 4.
> >     Another suggestion by Nicolas Dufresne was to add this as a menu control so that userspace
> >     can choose the clock for the timestamps from a given list, the enum in the list can also match
> >     the clocks ids.
> >     We would need to add a new buf flag in struct v4l2_buffer, like V4L2_BUF_FLAG_TIMESTAMP_OTHER,
> >     which would be "as specified through controls ...."
> >
> >
> > So my current question is, should we have both __u32 timestamp and __u32 boottimestamp?
> > Or should we have a mechanism that allows switching from one to the other and use
> > a single field? And if this mechanism should be implemented in both APIs? Can this be
> > defined later?
> >
> >
> > Please, let me know your thoughts.
> >
> > Thanks,
> > Helen
> >
> >>
> >>>>
> >>>>> +
> >>>>>  #ifndef __KERNEL__
> >>>>>  /**
> >>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
> >>>>> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
> >>>>>      __u32                   reserved[6];
> >>>>>  };
> >>>>>
> >>>>> +/**
> >>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
> >>>>> + * @index:  on return, index of the first created buffer
> >>>>> + * @count:  entry: number of requested buffers,
> >>>>> + *          return: number of created buffers
> >>>>> + * @memory: enum v4l2_memory; buffer memory type
> >>>>> + * @capabilities: capabilities of this buffer type.
> >>>>> + * @format: frame format, for which buffers are requested
> >>>>> + * @flags:  additional buffer management attributes (ignored unless the
> >>>>> + *          queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
> >>>>> + *          and configured for MMAP streaming I/O).
> >>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
> >>>>> + */
> >>>>> +struct v4l2_ext_create_buffers {
> >>>>> +    __u32                           index;
> >>>>> +    __u32                           count;
> >>>>> +    __u32                           memory;
> >>>>> +    __u32                           capabilities;
> >>>>> +    struct v4l2_ext_pix_format      format;
> >>>>
> >>>> The reality is that the only field that is ever used in the original v4l2_format
> >>>> struct is sizeimage. So this can be replaced with:
> >>>>
> >>>>       __u32                           plane_size[VIDEO_MAX_PLANES];
> >>>>
> >>>> (the field name I picked is debatable, but you get the idea)
> >>>>
> >>>> The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
> >>>> is needed for the current format. The original idea of using struct v4l2_format
> >>>> was that drivers would use the full format information to calculate the
> >>>> memory size, but that was just much too complicated to implement and nobody
> >>>> ever used that. Only the sizeimage field was ever used.
> >>>
> >>> Right, I'll update this in next version, This should simplify things.
> >>>
> >>
> >> I think this might need a bit more discussion. How would the userspace
> >> know what size is enough for the desired resolution? The hardware
> >> and/or drivers often have various alignment/padding restrictions,
> >> which might not be easy to guess for the userspace.
> >>
> >> Also I don't quite understand what's so complicated in handling the
> >> full format, or at least the most important parts of it. The
> >> implementation of TRY_FMT/S_FMT, which exists in every driver, should
> >> already be able to calculate the right plane sizes.
> >>
> >> Best regards,
> >> Tomasz
> >>
> >>>
> >>> Thanks,
> >>> Helen
> >>>
> >>>>
> >>>>> +    __u32                           flags;
> >>>>> +    __u32 reserved[5];
> >>>>> +};
> >>>>> +
> >>>>>  /*
> >>>>>   *  I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
> >>>>>   *
> >>>>> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
> >>>>>  #define VIDIOC_G_EXT_PIX_FMT        _IOWR('V', 104, struct v4l2_ext_pix_format)
> >>>>>  #define VIDIOC_S_EXT_PIX_FMT        _IOWR('V', 105, struct v4l2_ext_pix_format)
> >>>>>  #define VIDIOC_TRY_EXT_PIX_FMT      _IOWR('V', 106, struct v4l2_ext_pix_format)
> >>>>> +#define VIDIOC_EXT_CREATE_BUFS      _IOWR('V', 107, struct v4l2_ext_create_buffers)
> >>>>> +#define VIDIOC_EXT_QUERYBUF _IOWR('V', 108, struct v4l2_ext_buffer)
> >>>>> +#define VIDIOC_EXT_QBUF             _IOWR('V', 109, struct v4l2_ext_buffer)
> >>>>> +#define VIDIOC_EXT_DQBUF    _IOWR('V', 110, struct v4l2_ext_buffer)
> >>>>> +#define VIDIOC_EXT_PREPARE_BUF      _IOWR('V', 111, struct v4l2_ext_buffer)
> >>>>>
> >>>>>  /* Reminder: when adding new ioctls please add support for them to
> >>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
> >>>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>>       Hans
> >>>>
>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-03 19:52             ` Helen Koike
@ 2020-12-14 10:46               ` Tomasz Figa
  2020-12-15 14:36                 ` Helen Koike
  0 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-12-14 10:46 UTC (permalink / raw)
  To: Helen Koike
  Cc: Hans Verkuil, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

On Fri, Dec 4, 2020 at 4:52 AM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi,
>
> Please see my 2 points below (about v4l2_ext_buffer and another about timestamp).
>
> On 12/3/20 12:11 PM, Hans Verkuil wrote:
> > On 23/11/2020 18:40, Helen Koike wrote:
> >>
> >>
> >> On 11/23/20 12:46 PM, Tomasz Figa wrote:
> >>> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>
> >>>> Hi Hans,
> >>>>
> >>>> Thank you for your review.
> >>>>
> >>>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
> >>>>> Hi Helen,
> >>>>>
> >>>>> Again I'm just reviewing the uAPI.
> >>>>>
> >>>>> On 04/08/2020 21:29, Helen Koike wrote:
> >>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>>>
> >>>>>> Those extended buffer ops have several purpose:
> >>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
> >>>>>>    the number of ns elapsed since 1970
> >>>>>> 2/ Unify single/multiplanar handling
> >>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
> >>>>>>    to support the case where a single buffer object is storing all
> >>>>>>    planes data, each one being placed at a different offset
> >>>>>>
> >>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> >>>>>> these new objects.
> >>>>>>
> >>>>>> The core takes care of converting new ioctls requests to old ones
> >>>>>> if the driver does not support the new hooks, and vice versa.
> >>>>>>
> >>>>>> Note that the timecode field is gone, since there doesn't seem to be
> >>>>>> in-kernel users. We can be added back in the reserved area if needed or
> >>>>>> use the Request API to collect more metadata information from the
> >>>>>> frame.
> >>>>>>
> >>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> >>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>>>>> ---
> >>>>>> Changes in v5:
> >>>>>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> >>>>>> - return mem_offset to struct v4l2_ext_plane
> >>>>>> - change sizes and reorder fields to avoid holes in the struct and make
> >>>>>>   it the same for 32 and 64 bits
> >>>>>>
> >>>>>> Changes in v4:
> >>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> >>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> >>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> >>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> >>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
> >>>>>> - Remove num_planes field from struct v4l2_ext_buffer
> >>>>>> - Add flags field to struct v4l2_ext_create_buffers
> >>>>>> - Reformulate struct v4l2_ext_plane
> >>>>>> - Fix some bugs caught by v4l2-compliance
> >>>>>> - Rebased on top of media/master (post 5.8-rc1)
> >>>>>>
> >>>>>> Changes in v3:
> >>>>>> - Rebased on top of media/master (post 5.4-rc1)
> >>>>>>
> >>>>>> Changes in v2:
> >>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
> >>>>>>   later on
> >>>>>> ---
> >>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
> >>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
> >>>>>>  include/media/v4l2-ioctl.h           |  26 ++
> >>>>>>  include/uapi/linux/videodev2.h       |  90 +++++++
> >>>>>>  4 files changed, 476 insertions(+), 22 deletions(-)
> >>>>>>
> >>>>>
> >>>>> <snip>
> >>>>>
> >>>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> >>>>>> index 7123c6a4d9569..334cafdd2be97 100644
> >>>>>> --- a/include/uapi/linux/videodev2.h
> >>>>>> +++ b/include/uapi/linux/videodev2.h
> >>>>>> @@ -996,6 +996,41 @@ struct v4l2_plane {
> >>>>>>      __u32                   reserved[11];
> >>>>>>  };
> >>>>>>
> >>>>>> +/**
> >>>>>> + * struct v4l2_ext_plane - extended plane buffer info
> >>>>>> + * @buffer_length:  size of the entire buffer in bytes, should fit
> >>>>>> + *                  @offset + @plane_length
> >>>>>> + * @plane_length:   size of the plane in bytes.
> >>>>>> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
> >>>>>> + *                  that should be passed to mmap() called on the video node.
> >>>>>> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> >>>>>> + *                  to this plane.
> >>>>>> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> >>>>>> + *                  associated with this plane.
> >>>>>> + * @offset:         offset in the memory buffer where the plane starts.
> >>>>>> + * @memory:         enum v4l2_memory; the method, in which the actual video
> >>>>>> + *                  data is passed
> >>>>>> + * @reserved:               extra space reserved for future fields, must be set to 0.
> >>>>>> + *
> >>>>>> + *
> >>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> >>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
> >>>>>> + * Each plane can reside in a separate memory buffer, or even in
> >>>>>> + * a completely separate memory node (e.g. in embedded devices).
> >>>>>> + */
> >>>>>> +struct v4l2_ext_plane {
> >>>>>> +    __u32 buffer_length;
> >>>>>> +    __u32 plane_length;
> >>>>>> +    union {
> >>>>>> +            __u32 mem_offset;
> >>>>>> +            __u64 userptr;
> >>>>>> +            __s32 dmabuf_fd;
> >>>>>> +    } m;
> >>>>>> +    __u32 offset;
> >>>>>
> >>>>> I'd rename this plane_offset. I think some reordering would make this struct easier
> >>>>> to understand:
> >>>>>
> >>>>> struct v4l2_ext_plane {
> >>>>>       __u32 buffer_length;
> >>>>>       __u32 plane_offset;
> >>>>>       __u32 plane_length;
> >>>>>       __u32 memory;
> >>>>>       union {
> >>>>>               __u32 mem_offset;
> >>>>>               __u64 userptr;
> >>>>>               __s32 dmabuf_fd;
> >>>>>       } m;
> >>>>>       __u32 reserved[4];
> >>>>> };
> >>>>>
> >>>>>> +    __u32 memory;
> >>>>>> +    __u32 reserved[4];
> >>>>>> +};
> >>>>
> >>>> Ok, I'll apply this to the next version.
> >>>>
> >>>>>
> >>>>> What is not clear is how to tell the different between a single buffer containing
> >>>>> multiple planes, and using a separate buffer per plane. E.g. what would this look
> >>>>> like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
> >>>>> V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
> >>>>> planes are also combined in a single buffer?
> >>>>>
> >>>>> I would guess that the m union is set to 0 if the plane is part of the buffer
> >>>>> defined in the previous plane?
> >>>>
> >>>> The difference would be if m are equal or differ between planes, example:
> >>>>
> >>>> For V4L2_PIX_FMT_YVU420:
> >>>>
> >>>>     Y:
> >>>>         plane_offset = 0
> >>>>         m.dmabuf_fd = 3
> >>>>     Cb:
> >>>>         plane_offset = 300
> >>>>         m.dmabuf_fd = 3
> >>>>     Cr:
> >>>>         plane_offset = 375
> >>>>         m.dmabuf_fd = 3
> >>>>
> >>>> For V4L2_PIX_FMT_YVU420M:
> >>>>
> >>>>     Y:
> >>>>         plane_offset = 0
> >>>>         m.dmabuf_fd = 4
> >>>>     Cb:
> >>>>         plane_offset = 0
> >>>>         m.dmabuf_fd = 5
> >>>>     Cr:
> >>>>         plane_offset = 0
> >>>>         m.dmabuf_fd = 6
> >>>>
> >>>>
> >>>> Does it make sense?
> >>>>
> >>>
> >>> Actually all the 3 file descriptors can still point to the same
> >>> buffer, because they might have been dup()ed. The kernel needs to
> >>> resolve the file descriptors into struct dma_buf and then check
> >>> whether it's one or more buffers.
> >>
> >> Right, thanks for this.
> >>
> >>>
> >>> In fact, dup()ed FD for each plane is quite a common case in other
> >>> APIs, e.g. EGL, but current V4L2 API can't handle it. In Chromium we
> >>> basically work around it by assuming that if we receive a buffer for a
> >>> V4L2 device that only supports non-M formats, then we can safely
> >>> ignore all but first FD. The new API gives the ability to handle the
> >>> case properly, with full validation by the kernel.
> >>>
> >>>>>
> >>>>>> +
> >>>>>>  /**
> >>>>>>   * struct v4l2_buffer - video buffer info
> >>>>>>   * @index:  id number of the buffer
> >>>>>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
> >>>>>>      };
> >>>>>>  };
> >>>>>>
> >>>>>> +/**
> >>>>>> + * struct v4l2_ext_buffer - extended video buffer info
> >>>>>> + * @index:  id number of the buffer
> >>>>>> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> >>>>>> + * @flags:  buffer informational flags
> >>>>>> + * @field:  enum v4l2_field; field order of the image in the buffer
> >>>>>> + * @timestamp:      frame timestamp
> >>>>>> + * @sequence:       sequence count of this frame
> >>>>>> + * @planes: per-plane buffer information
> >>>>>> + * @request_fd:     fd of the request that this buffer should use
> >>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
> >>>>>> + *
> >>>>>> + * Contains data exchanged by application and driver using one of the Streaming
> >>>>>> + * I/O methods.
> >>>>>> + */
> >>>>>> +struct v4l2_ext_buffer {
> >>>>>> +    __u32 index;
> >>>>>> +    __u32 type;
> >>>>>> +    __u32 field;
> >>>>>> +    __u32 sequence;
> >>>>>> +    __u64 flags;
> >>>>>> +    __u64 timestamp;
> >>>>>> +    struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>
> I would like your opinion on the following:
>
> We have two concepts here
> * memory buffers (that belongs to a frame buffer object aka v4l2_ext_buffer)
> * color components/planes (that we need to indicate to userspace where the planes
>   are located, which buffer and which offset inside the buffer).
>
> A v4l2_ext_buffer can be reused to a different format if it fits the image
> (which is checked in QBUF time, by .buf_prepare() callback).
>
> Which means that, the information regarding where each color component is placed
> just make sense after the buffer is queued.
>
> So if userspace calls EXT_QUERYBUF, and the buffer isn't queued, the color component
> information doesn't make sense.
>
> One option is to fill in the plane information according to the current
> configured format, but only the information returned from EXT_QBUF is guaranteed
> to be the correct one.
>
> Another options is to split struct v4l2_ext_plane in two:
>
> struct v4l2_ext_membuffer {
>         __u32 memory;
>         union {
>                 __u32 mmap_offset;
>                 __u64 userptr;
>                 __s32 dmabuf_fd;
>         } m;
>         _u32 buffer_length;
> }
>
> struct v4l2_ext_plane {
>         /*
>          * memory buffer where this plane belongs, index is the position in of
>          * membuffers[] in struct v4l2_ext_buffer below
>          */
>         unsigned int membuf_index;
>         _u32 plane_length;
>         _u32 plane_offset;
> }
>
> Then we would have
>
> struct v4l2_ext_buffer {
>         ... <snip>
>         struct v4l2_ext_membuffer membuffers[VIDEO_MAX_PLANES];
>         struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>         ... <snip>
> }
>
> Where planes would only be filled by the core only if the buffer is
> queued (i.e. "locked" to a given format).
>
> This also avoids having several planes with different dmabuf_fd that are dup()ed,
> since we'll have an entry per memory buffer.
> Which also avoids the following:
> If we are working with a single membuf for all planes for instance, vb2 would need
> to know how many planes (let's say there are 3) and repeat the mem buffer information
> 3 times, and if userspace changes to a pixelformat with 2 color components, we would
> repeat 2 times with the same information. And we wouldn't have this issue
> if we split both information.
>
> I was also assuming that once the buffer is queued, userspace can't modify
> the configured format (I need to check this, but make sense to me due to how
> .buf_prepare() works).
>
> What do you think? Does it make sense?
>

How about making querybuf have its own structure that describes only
the buffer, as it is allocated? I.e.

struct v4l2_ext_mmap_plane {
        __u32 mmap_offset;
        __u32 length;
}

struct v4l2_ext_querybuf {
        /* ... */
        struct v4l2_ext_mmap_plane planes[VIDEO_MAX_PLANES];
};

Moreover, the ioctl would be only valid if the queue is operating in
MMAP mode, because otherwise it doesn't provide any useful information
- the userspace should already know what userptr or DMA-buf was
associated with the buffer. In fact, returning the DMA-buf FD from
QBUF time is confusing, because the userspace code might have already
closed that FD (it can rely on other way of referencing the buffer).

WDYT?

Best regards.
Tomasz

>
> >>>>>> +    __s32 request_fd;
> >>>>>> +    __u32 reserved[9];
> >>>>>> +};
> >>>>>
> >>>>> Brainstorming:
> >>>>>
> >>>>> Some ideas I have to make it easier to support mid stream resolution/colorimetry
> >>>>> changes:
> >>>>>
> >>>>> Adding width and height would support resolution changes (requires the use of
> >>>>> CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
> >>>>> information is provided here, then there are no race conditions.
> >>>>>
> >>>>> Same for adding the colorimetry fields here, this too can change on the fly (esp.
> >>>>> with HDMI), so reporting this information here avoids race conditions as well.
> >>>>
> >>>> Right, do you think this is something we can discuss later in a different RFC?
> >>>> So we can have a better view on how dynamic resolution change would be used?
> >>>>
> >>>> We can add more reserved fields or maybe try to do something to what has been
> >>>> discussed in about extensible system calls [1]
> >>>>
> >>>> [1] https://lwn.net/Articles/830666/
> >>>>
> >>>>>
> >>>>> And thirdly, I would like to have a __u64 boot_timestamp field containing the
> >>>>> CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
> >>>>> is that for m2m devices it is just copied and that for other devices it can have
> >>>>> different meanings depending on the timestamp buffer flags.
> >>>>>
> >>>>> There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
> >>>>> to add support for this. That way you know exactly when the driver was finished with
> >>>>> the buffer and that helps in detecting missed frames or instrumentation.
> >>>>
> >>>> I don't mind adding it. Does it make sense to have both timestamp and boot_timestamp?
> >>>>
> >>>
> >>> I think this is quite independent from the ext API work. AFAIR there
> >>> was an RFC to request the timestamp source from the userspace by the
> >>> flags field in QBUF, which would work with the existing API as well,
> >>> or it wasn't posted in the end?
> >
> > It's not about selecting a specific clock source. I think that option 4 as described
> > below would work for that.
> >
> > This problem I'm describing here is specific to m2m devices where the timestamp is
> > either just passed through untouched, or it is used as an identifier for a buffer
> > for use with stateless decoders.
> >
> > In both cases you cannot use the timestamp as a proper timestamp that tells you when
> > the buffer was marked done by the driver. So this is about adding a second timestamp
> > field (timestamp_done or something like that). Whether this would be hardcoded as using
> > CLOCK_BOOTTIME or uses the same clock source as selected through a control is something
> > that can be discussed, but since it does require a new field I believe this is part of
> > this proposal.
>
> I'm probably lacking m2m knowledge here.
> timestamp_done would be when the driver set the buffer as done, what about the other timestamp?
> I would like to rename it to make it clear what it means, maybe image_timestamp?
> Where, in a capture device, image_timestamp would be the timestamp from the hardware when
> it captured that specific frame, and in a decoder, it would be the timestamp of that frame
> when it got encoded?
> Is this correct?
>
> Thanks
> Helen
>
> >
> > Regards,
> >
> >       Hans
> >
> >>
> >> I was recalling the discussions we had regarding this:
> >>
> >> 1.
> >>     This first attempt in the uvc driver is to use a specific kernel parameter for that case:
> >>     https://patchwork.kernel.org/patch/10644887/
> >>     The conclusion that the support should be in the core API and not driver specific.
> >>
> >> 2.
> >>     Then an attempt to add global v4l2 support was sent with the Mediatek patch series:
> >>     https://patchwork.linuxtv.org/patch/60878/
> >>     The major problem is that clock type should be something selectable by userspace, and
> >>     not pre-defined by the driver.
> >>
> >> 3.
> >>     Another idea was to use the 'flags' field in the structs v4l2_requestbuffers and
> >>     v4l2_create_buffers.
> >>     But this field was removed in
> >>     129134e5415d ("media: media/v4l2: remove V4L2_FLAG_MEMORY_NON_CONSISTENT flag")
> >>     The major concern with this approach was with the uAPI, since it doesn't make much
> >>     sense to select a clock when creating buffers.
> >>
> >> 4.
> >>     Another suggestion by Nicolas Dufresne was to add this as a menu control so that userspace
> >>     can choose the clock for the timestamps from a given list, the enum in the list can also match
> >>     the clocks ids.
> >>     We would need to add a new buf flag in struct v4l2_buffer, like V4L2_BUF_FLAG_TIMESTAMP_OTHER,
> >>     which would be "as specified through controls ...."
> >>
> >>
> >> So my current question is, should we have both __u32 timestamp and __u32 boottimestamp?
> >> Or should we have a mechanism that allows switching from one to the other and use
> >> a single field? And if this mechanism should be implemented in both APIs? Can this be
> >> defined later?
> >>
> >>
> >> Please, let me know your thoughts.
> >>
> >> Thanks,
> >> Helen
> >>
> >>>
> >>>>>
> >>>>>> +
> >>>>>>  #ifndef __KERNEL__
> >>>>>>  /**
> >>>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
> >>>>>> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
> >>>>>>      __u32                   reserved[6];
> >>>>>>  };
> >>>>>>
> >>>>>> +/**
> >>>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
> >>>>>> + * @index:  on return, index of the first created buffer
> >>>>>> + * @count:  entry: number of requested buffers,
> >>>>>> + *          return: number of created buffers
> >>>>>> + * @memory: enum v4l2_memory; buffer memory type
> >>>>>> + * @capabilities: capabilities of this buffer type.
> >>>>>> + * @format: frame format, for which buffers are requested
> >>>>>> + * @flags:  additional buffer management attributes (ignored unless the
> >>>>>> + *          queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
> >>>>>> + *          and configured for MMAP streaming I/O).
> >>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
> >>>>>> + */
> >>>>>> +struct v4l2_ext_create_buffers {
> >>>>>> +    __u32                           index;
> >>>>>> +    __u32                           count;
> >>>>>> +    __u32                           memory;
> >>>>>> +    __u32                           capabilities;
> >>>>>> +    struct v4l2_ext_pix_format      format;
> >>>>>
> >>>>> The reality is that the only field that is ever used in the original v4l2_format
> >>>>> struct is sizeimage. So this can be replaced with:
> >>>>>
> >>>>>       __u32                           plane_size[VIDEO_MAX_PLANES];
> >>>>>
> >>>>> (the field name I picked is debatable, but you get the idea)
> >>>>>
> >>>>> The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
> >>>>> is needed for the current format. The original idea of using struct v4l2_format
> >>>>> was that drivers would use the full format information to calculate the
> >>>>> memory size, but that was just much too complicated to implement and nobody
> >>>>> ever used that. Only the sizeimage field was ever used.
> >>>>
> >>>> Right, I'll update this in next version, This should simplify things.
> >>>>
> >>>
> >>> I think this might need a bit more discussion. How would the userspace
> >>> know what size is enough for the desired resolution? The hardware
> >>> and/or drivers often have various alignment/padding restrictions,
> >>> which might not be easy to guess for the userspace.
> >>>
> >>> Also I don't quite understand what's so complicated in handling the
> >>> full format, or at least the most important parts of it. The
> >>> implementation of TRY_FMT/S_FMT, which exists in every driver, should
> >>> already be able to calculate the right plane sizes.
> >>>
> >>> Best regards,
> >>> Tomasz
> >>>
> >>>>
> >>>> Thanks,
> >>>> Helen
> >>>>
> >>>>>
> >>>>>> +    __u32                           flags;
> >>>>>> +    __u32 reserved[5];
> >>>>>> +};
> >>>>>> +
> >>>>>>  /*
> >>>>>>   *  I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
> >>>>>>   *
> >>>>>> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
> >>>>>>  #define VIDIOC_G_EXT_PIX_FMT        _IOWR('V', 104, struct v4l2_ext_pix_format)
> >>>>>>  #define VIDIOC_S_EXT_PIX_FMT        _IOWR('V', 105, struct v4l2_ext_pix_format)
> >>>>>>  #define VIDIOC_TRY_EXT_PIX_FMT      _IOWR('V', 106, struct v4l2_ext_pix_format)
> >>>>>> +#define VIDIOC_EXT_CREATE_BUFS      _IOWR('V', 107, struct v4l2_ext_create_buffers)
> >>>>>> +#define VIDIOC_EXT_QUERYBUF _IOWR('V', 108, struct v4l2_ext_buffer)
> >>>>>> +#define VIDIOC_EXT_QBUF             _IOWR('V', 109, struct v4l2_ext_buffer)
> >>>>>> +#define VIDIOC_EXT_DQBUF    _IOWR('V', 110, struct v4l2_ext_buffer)
> >>>>>> +#define VIDIOC_EXT_PREPARE_BUF      _IOWR('V', 111, struct v4l2_ext_buffer)
> >>>>>>
> >>>>>>  /* Reminder: when adding new ioctls please add support for them to
> >>>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
> >>>>>>
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>>       Hans
> >>>>>
> >

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 3/7] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks
  2020-12-14  8:52   ` Tomasz Figa
@ 2020-12-14 12:29     ` Helen Koike
  0 siblings, 0 replies; 51+ messages in thread
From: Helen Koike @ 2020-12-14 12:29 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: mchehab, hans.verkuil, laurent.pinchart, sakari.ailus,
	linux-media, Boris Brezillon, hiroh, nicolas, Brian.Starkey,
	kernel, narmstrong, linux-kernel, frkoenig, mjourdan,
	stanimir.varbanov

Hi Tomasz,

Thank you for your comments,

On 12/14/20 5:52 AM, Tomasz Figa wrote:
> Hi Helen,
> 
> On Tue, Aug 04, 2020 at 04:29:35PM -0300, Helen Koike wrote:
>> The VB2 layer is used by a lot of drivers. Patch it to support the
>> _EXT_PIX_FMT and _EXT_BUF ioctls in order to ease conversion of existing
>> drivers to these new APIs.
>>
>> Note that internally, the VB2 core is now only using ext structs and old
>> APIs are supported through conversion wrappers.
> 
> We decided to only support V4L2_BUF_TYPE_VIDEO* for the ext structs. Still,
> existing drivers may use vb2 with the other, legacy, buf types. How would
> they be handled with this change?

I completly refactored this patch in my wip branch, I'll submit for comments
soon after finishing addressing the other comments.

The way I'm approaching this is to support both structures v4l2_buffer and
v4l2_ext_buffer, but only in the vb2 entry points, since all the
information we need is in vb2_buffer struct.

To implement this I had to use new hooks in the framework. I think it is easier
if you take a look in next version when I submited it.

> 
>>
>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> ---
>> Changes in v5:
>> - Update with new format and buffer structs
>> - Updated commit message with the uAPI prefix
>>
>> Changes in v4:
>> - Update with new format and buffer structs
>> - Fix some bugs caught by v4l2-compliance
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - New patch
>> ---
>>  .../media/common/videobuf2/videobuf2-core.c   |   2 +
>>  .../media/common/videobuf2/videobuf2-v4l2.c   | 560 ++++++++++--------
>>  include/media/videobuf2-core.h                |   6 +-
>>  include/media/videobuf2-v4l2.h                |  21 +-
>>  4 files changed, 345 insertions(+), 244 deletions(-)
>>
>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
>> index f544d3393e9d6..d719b1e9c148b 100644
>> --- a/drivers/media/common/videobuf2/videobuf2-core.c
>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
>> @@ -1270,6 +1270,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb)
>>  		vb->planes[plane].length = 0;
>>  		vb->planes[plane].m.fd = 0;
>>  		vb->planes[plane].data_offset = 0;
>> +		vb->planes[plane].dbuf_offset = 0;
>>  
>>  		/* Acquire each plane's memory */
>>  		mem_priv = call_ptr_memop(vb, attach_dmabuf,
>> @@ -1313,6 +1314,7 @@ static int __prepare_dmabuf(struct vb2_buffer *vb)
>>  		vb->planes[plane].length = planes[plane].length;
>>  		vb->planes[plane].m.fd = planes[plane].m.fd;
>>  		vb->planes[plane].data_offset = planes[plane].data_offset;
>> +		vb->planes[plane].dbuf_offset = planes[plane].dbuf_offset;
>>  	}
>>  
>>  	if (reacquired) {
>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> index 30caad27281e1..911681d24b3ae 100644
>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> @@ -29,6 +29,7 @@
>>  #include <media/v4l2-fh.h>
>>  #include <media/v4l2-event.h>
>>  #include <media/v4l2-common.h>
>> +#include <media/v4l2-ioctl.h>
>>  
>>  #include <media/videobuf2-v4l2.h>
>>  
>> @@ -56,72 +57,39 @@ module_param(debug, int, 0644);
>>  				 V4L2_BUF_FLAG_TIMECODE | \
>>  				 V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF)
>>  
>> -/*
>> - * __verify_planes_array() - verify that the planes array passed in struct
>> - * v4l2_buffer from userspace can be safely used
>> - */
>> -static int __verify_planes_array(struct vb2_buffer *vb, const struct v4l2_buffer *b)
>> -{
>> -	if (!V4L2_TYPE_IS_MULTIPLANAR(b->type))
>> -		return 0;
>> -
>> -	/* Is memory for copying plane information present? */
>> -	if (b->m.planes == NULL) {
>> -		dprintk(vb->vb2_queue, 1,
>> -			"multi-planar buffer passed but planes array not provided\n");
>> -		return -EINVAL;
>> -	}
>> -
>> -	if (b->length < vb->num_planes || b->length > VB2_MAX_PLANES) {
>> -		dprintk(vb->vb2_queue, 1,
>> -			"incorrect planes array length, expected %d, got %d\n",
>> -			vb->num_planes, b->length);
>> -		return -EINVAL;
>> -	}
>> -
>> -	return 0;
>> -}
>> -
>>  static int __verify_planes_array_core(struct vb2_buffer *vb, const void *pb)
>>  {
>> -	return __verify_planes_array(vb, pb);
>> +	return 0;
>>  }
>>  
>>  /*
>>   * __verify_length() - Verify that the bytesused value for each plane fits in
>>   * the plane length and that the data offset doesn't exceed the bytesused value.
>>   */
>> -static int __verify_length(struct vb2_buffer *vb, const struct v4l2_buffer *b)
>> +static int __verify_length(struct vb2_buffer *vb,
>> +			   const struct v4l2_ext_buffer *b)
>>  {
>>  	unsigned int length;
>>  	unsigned int bytesused;
>> -	unsigned int plane;
>> +	unsigned int i;
>>  
>>  	if (V4L2_TYPE_IS_CAPTURE(b->type))
>>  		return 0;
>>  
>> -	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
>> -		for (plane = 0; plane < vb->num_planes; ++plane) {
>> -			length = (b->memory == VB2_MEMORY_USERPTR ||
>> -				  b->memory == VB2_MEMORY_DMABUF)
>> -			       ? b->m.planes[plane].length
>> -				: vb->planes[plane].length;
>> -			bytesused = b->m.planes[plane].bytesused
>> -				  ? b->m.planes[plane].bytesused : length;
>> -
>> -			if (b->m.planes[plane].bytesused > length)
>> -				return -EINVAL;
>> -
>> -			if (b->m.planes[plane].data_offset > 0 &&
>> -			    b->m.planes[plane].data_offset >= bytesused)
>> -				return -EINVAL;
>> -		}
>> -	} else {
>> -		length = (b->memory == VB2_MEMORY_USERPTR ||
>> -			  b->memory == VB2_MEMORY_DMABUF)
>> -			? b->length : vb->planes[0].length;
>> +	for (i = 0; i < vb->num_planes; ++i) {
>> +		const struct v4l2_ext_plane *plane = &b->planes[i];
>>  
>> -		if (b->bytesused > length)
>> +		length = (plane->memory == VB2_MEMORY_USERPTR ||
>> +			  plane->memory == VB2_MEMORY_DMABUF) ?
>> +			 plane->buffer_length :
>> +			 vb->planes[i].length;
>> +		bytesused = plane->plane_length ?
>> +			    plane->plane_length : length;
>> +
>> +		if (bytesused > length)
>> +			return -EINVAL;
>> +
>> +		if (plane->offset + bytesused > length)
>>  			return -EINVAL;
>>  	}
>>  
>> @@ -140,21 +108,12 @@ static void __init_vb2_v4l2_buffer(struct vb2_buffer *vb)
>>  
>>  static void __copy_timestamp(struct vb2_buffer *vb, const void *pb)
>>  {
>> -	const struct v4l2_buffer *b = pb;
>> -	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>> +	const struct v4l2_ext_buffer *b = pb;
>>  	struct vb2_queue *q = vb->vb2_queue;
>>  
>> -	if (q->is_output) {
>> -		/*
>> -		 * For output buffers copy the timestamp if needed,
>> -		 * and the timecode field and flag if needed.
>> -		 */
>> -		if (q->copy_timestamp)
>> -			vb->timestamp = v4l2_buffer_get_timestamp(b);
>> -		vbuf->flags |= b->flags & V4L2_BUF_FLAG_TIMECODE;
>> -		if (b->flags & V4L2_BUF_FLAG_TIMECODE)
>> -			vbuf->timecode = b->timecode;
> 
> Is it okay to remove the timecode copying functionality?
> 
>> -	}
>> +	/* For output buffers copy the timestamp if needed. */
>> +	if (q->is_output && q->copy_timestamp)
>> +		vb->timestamp = b->timestamp;
>>  };
>>  
>>  static void vb2_warn_zero_bytesused(struct vb2_buffer *vb)
>> @@ -173,7 +132,8 @@ static void vb2_warn_zero_bytesused(struct vb2_buffer *vb)
>>  		pr_warn("use the actual size instead.\n");
>>  }
>>  
>> -static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b)
>> +static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb,
>> +				    struct v4l2_ext_buffer *b)
>>  {
>>  	struct vb2_queue *q = vb->vb2_queue;
>>  	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>> @@ -203,110 +163,67 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
>>  	vbuf->request_fd = -1;
>>  	vbuf->is_held = false;
>>  
>> -	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
>> -		switch (b->memory) {
>> -		case VB2_MEMORY_USERPTR:
>> -			for (plane = 0; plane < vb->num_planes; ++plane) {
>> -				planes[plane].m.userptr =
>> -					b->m.planes[plane].m.userptr;
>> -				planes[plane].length =
>> -					b->m.planes[plane].length;
>> -			}
>> -			break;
>> -		case VB2_MEMORY_DMABUF:
>> -			for (plane = 0; plane < vb->num_planes; ++plane) {
>> -				planes[plane].m.fd =
>> -					b->m.planes[plane].m.fd;
>> -				planes[plane].length =
>> -					b->m.planes[plane].length;
>> -			}
>> -			break;
>> -		default:
>> -			for (plane = 0; plane < vb->num_planes; ++plane) {
>> -				planes[plane].m.offset =
>> -					vb->planes[plane].m.offset;
>> -				planes[plane].length =
>> -					vb->planes[plane].length;
>> -			}
>> -			break;
>> +	for (plane = 0; plane < vb->num_planes; ++plane) {
>> +		if (b->planes[0].memory != b->planes[plane].memory) {
>> +			dprintk(q, 1, "planes should have the same memory type\n");
>> +			return -EINVAL;
>>  		}
>> +	}
>>  
>> -		/* Fill in driver-provided information for OUTPUT types */
>> -		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
>> -			/*
>> -			 * Will have to go up to b->length when API starts
>> -			 * accepting variable number of planes.
>> -			 *
>> -			 * If bytesused == 0 for the output buffer, then fall
>> -			 * back to the full buffer size. In that case
>> -			 * userspace clearly never bothered to set it and
>> -			 * it's a safe assumption that they really meant to
>> -			 * use the full plane sizes.
>> -			 *
>> -			 * Some drivers, e.g. old codec drivers, use bytesused == 0
>> -			 * as a way to indicate that streaming is finished.
>> -			 * In that case, the driver should use the
>> -			 * allow_zero_bytesused flag to keep old userspace
>> -			 * applications working.
>> -			 */
>> -			for (plane = 0; plane < vb->num_planes; ++plane) {
>> -				struct vb2_plane *pdst = &planes[plane];
>> -				struct v4l2_plane *psrc = &b->m.planes[plane];
>> -
>> -				if (psrc->bytesused == 0)
>> -					vb2_warn_zero_bytesused(vb);
>> -
>> -				if (vb->vb2_queue->allow_zero_bytesused)
>> -					pdst->bytesused = psrc->bytesused;
>> -				else
>> -					pdst->bytesused = psrc->bytesused ?
>> -						psrc->bytesused : pdst->length;
>> -				pdst->data_offset = psrc->data_offset;
>> -			}
>> +	switch (b->planes[0].memory) {
>> +	case VB2_MEMORY_USERPTR:
>> +		for (plane = 0; plane < vb->num_planes; ++plane) {
>> +			planes[plane].m.userptr = b->planes[plane].m.userptr;
>> +			planes[plane].length = b->planes[plane].buffer_length;
>>  		}
>> -	} else {
>> +		break;
>> +	case VB2_MEMORY_DMABUF:
>> +		for (plane = 0; plane < vb->num_planes; ++plane) {
>> +			planes[plane].m.fd = b->planes[plane].m.dmabuf_fd;
>> +			planes[plane].dbuf_offset = b->planes[plane].offset;
>> +			planes[plane].length = b->planes[plane].buffer_length;
>> +		}
>> +		break;
>> +	default:
>> +		for (plane = 0; plane < vb->num_planes; ++plane) {
>> +			planes[plane].m.offset = vb->planes[plane].m.offset;
>> +			planes[plane].length = vb->planes[plane].length;
>> +		}
>> +		break;
>> +	}
>> +
>> +	/* Fill in driver-provided information for OUTPUT types */
>> +	if (V4L2_TYPE_IS_OUTPUT(b->type)) {
>>  		/*
>> -		 * Single-planar buffers do not use planes array,
>> -		 * so fill in relevant v4l2_buffer struct fields instead.
>> -		 * In videobuf we use our internal V4l2_planes struct for
>> -		 * single-planar buffers as well, for simplicity.
>> +		 * Will have to go up to b->length when API starts
>> +		 * accepting variable number of planes.
>>  		 *
>> -		 * If bytesused == 0 for the output buffer, then fall back
>> -		 * to the full buffer size as that's a sensible default.
>> +		 * If bytesused == 0 for the output buffer, then fall
> 
> Should this be updated with the new field name?

For next version I implemented a vb2_fill_vb2_v4l2_buffer_ext() and I didn't
touch this code.

> 
>> +		 * back to the full buffer size. In that case
>> +		 * userspace clearly never bothered to set it and
>> +		 * it's a safe assumption that they really meant to
>> +		 * use the full plane sizes.
>>  		 *
>> -		 * Some drivers, e.g. old codec drivers, use bytesused == 0 as
>> -		 * a way to indicate that streaming is finished. In that case,
>> -		 * the driver should use the allow_zero_bytesused flag to keep
>> -		 * old userspace applications working.
>> +		 * Some drivers, e.g. old codec drivers, use bytesused == 0
>> +		 * as a way to indicate that streaming is finished.
>> +		 * In that case, the driver should use the
>> +		 * allow_zero_bytesused flag to keep old userspace
>> +		 * applications working.
> 
> Do we retain this behavior in the new API? If no, I guess we need to check
> if this was old or new API and react appropriately.

For next version I implemented a vb2_fill_vb2_v4l2_buffer_ext() and I didn't
touch this code.

> 
>>  		 */
>> -		switch (b->memory) {
>> -		case VB2_MEMORY_USERPTR:
>> -			planes[0].m.userptr = b->m.userptr;
>> -			planes[0].length = b->length;
>> -			break;
>> -		case VB2_MEMORY_DMABUF:
>> -			planes[0].m.fd = b->m.fd;
>> -			planes[0].length = b->length;
>> -			break;
>> -		default:
>> -			planes[0].m.offset = vb->planes[0].m.offset;
>> -			planes[0].length = vb->planes[0].length;
>> -			break;
>> -		}
>> +		for (plane = 0; plane < vb->num_planes; ++plane) {
>> +			struct vb2_plane *pdst = &planes[plane];
>> +			struct v4l2_ext_plane *psrc = &b->planes[plane];
>>  
>> -		planes[0].data_offset = 0;
>> -		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
>> -			if (b->bytesused == 0)
>> +			if (psrc->plane_length == 0)
>>  				vb2_warn_zero_bytesused(vb);
>>  
>>  			if (vb->vb2_queue->allow_zero_bytesused)
>> -				planes[0].bytesused = b->bytesused;
>> +				pdst->bytesused = psrc->plane_length;
>>  			else
>> -				planes[0].bytesused = b->bytesused ?
>> -					b->bytesused : planes[0].length;
>> -		} else
>> -			planes[0].bytesused = 0;
>> -
>> +				pdst->bytesused = psrc->plane_length ?
>> +						  psrc->plane_length :
>> +						  pdst->length;
>> +		}
>>  	}
>>  
>>  	/* Zero flags that we handle */
>> @@ -343,7 +260,7 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
>>  
>>  static void set_buffer_cache_hints(struct vb2_queue *q,
>>  				   struct vb2_buffer *vb,
>> -				   struct v4l2_buffer *b)
>> +				   struct v4l2_ext_buffer *b)
>>  {
>>  	/*
>>  	 * DMA exporter should take care of cache syncs, so we can avoid
>> @@ -388,17 +305,44 @@ static void set_buffer_cache_hints(struct vb2_queue *q,
>>  		vb->need_cache_sync_on_prepare = 0;
>>  }
>>  
>> +static enum v4l2_buf_type vb2_ext_qtype(struct vb2_queue *q)
>> +{
>> +	if (!q->is_multiplanar)
>> +		return q->type;
> 
> Do we need this explicit check? Wouldn't the code below would return
> the right thing anyway?

Ack, I had already updated this in my wip branch.

> 
>> +
>> +	if (q->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>> +		return V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +	else if (q->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
>> +		return V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +
>> +	return q->type;
>> +}
>> +
>> +static enum v4l2_buf_type vb2_ext_type(unsigned int type, bool is_multiplanar)
>> +{
>> +	if (!is_multiplanar)
>> +		return type;
> 
> Ditto.

Not here, since ext type always use non MPLANE regardless if the driver
supports multiplanar or not, so we need to know to which type to
convert back.

> 
>> +
>> +	if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> +		return V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>> +	else if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
>> +		return V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
>> +
>> +	return type;
>> +}
>> +
>>  static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
>> -				    struct v4l2_buffer *b, bool is_prepare,
>> +				    struct v4l2_ext_buffer *b, bool is_prepare,
>>  				    struct media_request **p_req)
>>  {
>>  	const char *opname = is_prepare ? "prepare_buf" : "qbuf";
>>  	struct media_request *req;
>>  	struct vb2_v4l2_buffer *vbuf;
>>  	struct vb2_buffer *vb;
>> +	unsigned int i;
>>  	int ret;
>>  
>> -	if (b->type != q->type) {
>> +	if (b->type != vb2_ext_qtype(q)) {
>>  		dprintk(q, 1, "%s: invalid buffer type\n", opname);
>>  		return -EINVAL;
>>  	}
>> @@ -414,16 +358,14 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
>>  		return -EINVAL;
>>  	}
>>  
>> -	if (b->memory != q->memory) {
>> +	for (i = 0; i < VIDEO_MAX_PLANES && b->planes[i].buffer_length; i++)
>> +		if (b->planes[i].memory != q->memory) {
>>  		dprintk(q, 1, "%s: invalid memory type\n", opname);
>>  		return -EINVAL;
>>  	}
>>  
>>  	vb = q->bufs[b->index];
>>  	vbuf = to_vb2_v4l2_buffer(vb);
>> -	ret = __verify_planes_array(vb, b);
>> -	if (ret)
>> -		return ret;
>>  
>>  	if (!is_prepare && (b->flags & V4L2_BUF_FLAG_REQUEST_FD) &&
>>  	    vb->state != VB2_BUF_STATE_DEQUEUED) {
>> @@ -487,11 +429,6 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
>>  		    !q->ops->buf_out_validate))
>>  		return -EINVAL;
>>  
>> -	if (b->request_fd < 0) {
>> -		dprintk(q, 1, "%s: request_fd < 0\n", opname);
>> -		return -EINVAL;
>> -	}
>> -
> 
> I guess this code was useless already, because the call below would
> validate the FD, but should it be perhaps removed in a separate patch?

Ack, I'll submit a separate patch.

> 
>>  	req = media_request_get_by_fd(mdev, b->request_fd);
>>  	if (IS_ERR(req)) {
>>  		dprintk(q, 1, "%s: invalid request_fd\n", opname);
>> @@ -516,64 +453,46 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *md
>>  }
>>  
>>  /*
>> - * __fill_v4l2_buffer() - fill in a struct v4l2_buffer with information to be
>> - * returned to userspace
>> + * __fill_v4l2_buffer() - fill in a struct v4l2_ext_buffer with information to
>> + * be returned to userspace
>>   */
>>  static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>>  {
>> -	struct v4l2_buffer *b = pb;
>> +	struct v4l2_ext_buffer *b = pb;
>>  	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
>>  	struct vb2_queue *q = vb->vb2_queue;
>>  	unsigned int plane;
>>  
>>  	/* Copy back data such as timestamp, flags, etc. */
>>  	b->index = vb->index;
>> -	b->type = vb->type;
>> -	b->memory = vb->memory;
>> -	b->bytesused = 0;
>> +	b->type = vb2_ext_qtype(q);
>>  
>>  	b->flags = vbuf->flags;
>>  	b->field = vbuf->field;
>> -	v4l2_buffer_set_timestamp(b, vb->timestamp);
>> -	b->timecode = vbuf->timecode;
>> +	b->timestamp = vb->timestamp;
>>  	b->sequence = vbuf->sequence;
>> -	b->reserved2 = 0;
>>  	b->request_fd = 0;
>> +	memset(b->reserved, 0, sizeof(b->reserved));
>>  
>> -	if (q->is_multiplanar) {
>> -		/*
>> -		 * Fill in plane-related data if userspace provided an array
>> -		 * for it. The caller has already verified memory and size.
>> -		 */
>> -		b->length = vb->num_planes;
>> -		for (plane = 0; plane < vb->num_planes; ++plane) {
>> -			struct v4l2_plane *pdst = &b->m.planes[plane];
>> -			struct vb2_plane *psrc = &vb->planes[plane];
>> -
>> -			pdst->bytesused = psrc->bytesused;
>> -			pdst->length = psrc->length;
>> -			if (q->memory == VB2_MEMORY_MMAP)
>> -				pdst->m.mem_offset = psrc->m.offset;
>> -			else if (q->memory == VB2_MEMORY_USERPTR)
>> -				pdst->m.userptr = psrc->m.userptr;
>> -			else if (q->memory == VB2_MEMORY_DMABUF)
>> -				pdst->m.fd = psrc->m.fd;
>> -			pdst->data_offset = psrc->data_offset;
>> -			memset(pdst->reserved, 0, sizeof(pdst->reserved));
>> -		}
>> -	} else {
>> -		/*
>> -		 * We use length and offset in v4l2_planes array even for
>> -		 * single-planar buffers, but userspace does not.
>> -		 */
>> -		b->length = vb->planes[0].length;
>> -		b->bytesused = vb->planes[0].bytesused;
>> +	/*
>> +	 * Fill in plane-related data if userspace provided an array
>> +	 * for it. The caller has already verified memory and size.
>> +	 */
>> +	for (plane = 0; plane < vb->num_planes; ++plane) {
>> +		struct v4l2_ext_plane *pdst = &b->planes[plane];
>> +		struct vb2_plane *psrc = &vb->planes[plane];
>> +
>> +		pdst->memory = vb->memory;
>> +		pdst->plane_length = psrc->bytesused;
>> +		pdst->buffer_length = psrc->length;
>> +		pdst->offset = psrc->dbuf_offset;
>>  		if (q->memory == VB2_MEMORY_MMAP)
>> -			b->m.offset = vb->planes[0].m.offset;
>> +			pdst->m.mem_offset = psrc->m.offset;
>>  		else if (q->memory == VB2_MEMORY_USERPTR)
>> -			b->m.userptr = vb->planes[0].m.userptr;
>> +			pdst->m.userptr = psrc->m.userptr;
>>  		else if (q->memory == VB2_MEMORY_DMABUF)
>> -			b->m.fd = vb->planes[0].m.fd;
>> +			pdst->m.dmabuf_fd = psrc->m.fd;
>> +		memset(pdst->reserved, 0, sizeof(pdst->reserved));
>>  	}
>>  
>>  	/*
>> @@ -668,6 +587,35 @@ int vb2_find_timestamp(const struct vb2_queue *q, u64 timestamp,
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_find_timestamp);
>>  
>> +#define vb2_buf_to_ext_buf_op(_name, ...)				\
>> +({									\
>> +	int ret;							\
>> +									\
>> +	ret = v4l2_buffer_to_ext_buffer(b, &eb);			\
>> +	if (!ret)							\
>> +		ret = _name(__VA_ARGS__);				\
>> +	if (!ret)							\
>> +		ret = v4l2_ext_buffer_to_buffer(&eb, b,			\
>> +						q->is_multiplanar);	\
> 
> I'm not very happy with this macro implicitly relying on existence of some
> specific variables with predefined names in the scope of the caller.
> 
> That said, I don't have a good idea on how this could be fixed, other than
> just repeating this code manually in each function.
> 
> If we don't find anything better, I think at least a comment here
> explaining the assumptions would make it a bit better.

I completly removed this macro for the newer version.

> 
>> +	ret;								\
>> +})
>> +
>> +int vb2_ext_querybuf(struct vb2_queue *q, struct v4l2_ext_buffer *b)
>> +{
>> +	if (b->type != vb2_ext_qtype(q)) {
>> +		dprintk(q, 1, "wrong buffer type\n");
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (b->index >= q->num_buffers) {
>> +		dprintk(q, 1, "buffer index out of range\n");
>> +		return -EINVAL;
>> +	}
>> +	vb2_core_querybuf(q, b->index, b);
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL(vb2_ext_querybuf);
>> +
>>  /*
>>   * vb2_querybuf() - query video buffer information
>>   * @q:		videobuf queue
>> @@ -683,23 +631,9 @@ EXPORT_SYMBOL_GPL(vb2_find_timestamp);
>>   */
>>  int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b)
>>  {
>> -	struct vb2_buffer *vb;
>> -	int ret;
>> -
>> -	if (b->type != q->type) {
>> -		dprintk(q, 1, "wrong buffer type\n");
>> -		return -EINVAL;
>> -	}
>> +	struct v4l2_ext_buffer eb;
>>  
>> -	if (b->index >= q->num_buffers) {
>> -		dprintk(q, 1, "buffer index out of range\n");
>> -		return -EINVAL;
>> -	}
>> -	vb = q->bufs[b->index];
>> -	ret = __verify_planes_array(vb, b);
>> -	if (!ret)
>> -		vb2_core_querybuf(q, b->index, b);
>> -	return ret;
>> +	return vb2_buf_to_ext_buf_op(vb2_ext_querybuf, q, &eb);
>>  }
>>  EXPORT_SYMBOL(vb2_querybuf);
>>  
>> @@ -741,8 +675,8 @@ int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req)
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_reqbufs);
>>  
>> -int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
>> -		    struct v4l2_buffer *b)
>> +int vb2_ext_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
>> +			struct v4l2_ext_buffer *b)
>>  {
>>  	int ret;
>>  
>> @@ -758,16 +692,59 @@ int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
>>  
>>  	return ret ? ret : vb2_core_prepare_buf(q, b->index, b);
>>  }
>> +EXPORT_SYMBOL_GPL(vb2_ext_prepare_buf);
>> +
>> +int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
>> +		    struct v4l2_buffer *b)
>> +{
>> +	struct v4l2_ext_buffer eb;
>> +
>> +	return vb2_buf_to_ext_buf_op(vb2_ext_prepare_buf, q, mdev, &eb);
>> +}
>>  EXPORT_SYMBOL_GPL(vb2_prepare_buf);
>>  
>> +int vb2_ext_create_bufs(struct vb2_queue *q,
>> +			struct v4l2_ext_create_buffers *create)
>> +{
>> +	unsigned int requested_sizes[VIDEO_MAX_PLANES];
>> +	struct v4l2_ext_pix_format *f = &create->format;
>> +	int ret = vb2_verify_memory_type(q, create->memory,
>> +				vb2_ext_type(f->type, q->is_multiplanar));
>> +	unsigned i;
>> +
>> +	if (create->format.type != V4L2_BUF_TYPE_VIDEO_CAPTURE &&
>> +	    create->format.type != V4L2_BUF_TYPE_VIDEO_OUTPUT)
>> +		return -EINVAL;
>> +
>> +	fill_buf_caps(q, &create->capabilities);
>> +	clear_consistency_attr(q, create->memory, &create->flags);
>> +	create->index = q->num_buffers;
>> +	if (create->count == 0)
>> +		return ret != -EBUSY ? ret : 0;
>> +
>> +	if (!f->plane_fmt[0].sizeimage)
>> +		return -EINVAL;
>> +	for (i = 0; i < VIDEO_MAX_PLANES &&
>> +		    f->plane_fmt[i].sizeimage; i++)
>> +		requested_sizes[i] = f->plane_fmt[i].sizeimage;
>> +
>> +	return ret ? ret : vb2_core_create_bufs(q, create->memory,
>> +						create->flags,
>> +						&create->count,
>> +						i,
>> +						requested_sizes);
>> +}
>> +EXPORT_SYMBOL_GPL(vb2_ext_create_bufs);
>> +
>>  int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
>>  {
>> +	struct v4l2_format *f = &create->format;
>>  	unsigned requested_planes = 1;
>>  	unsigned requested_sizes[VIDEO_MAX_PLANES];
>> -	struct v4l2_format *f = &create->format;
>>  	int ret = vb2_verify_memory_type(q, create->memory, f->type);
>>  	unsigned i;
>>  
>> +
>>  	fill_buf_caps(q, &create->capabilities);
>>  	clear_consistency_attr(q, create->memory, &create->flags);
>>  	create->index = q->num_buffers;
>> @@ -777,18 +754,29 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
>>  	switch (f->type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		requested_planes = f->fmt.pix_mp.num_planes;
>> -		if (requested_planes == 0 ||
>> -		    requested_planes > VIDEO_MAX_PLANES)
>> -			return -EINVAL;
>> -		for (i = 0; i < requested_planes; i++)
>> -			requested_sizes[i] =
>> -				f->fmt.pix_mp.plane_fmt[i].sizeimage;
>> -		break;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -		requested_sizes[0] = f->fmt.pix.sizeimage;
>> -		break;
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT: {
>> +		struct v4l2_ext_create_buffers ecreate = {
>> +			.count = create->count,
>> +			.memory = create->memory,
>> +			.flags = create->flags,
>> +		};
>> +
>> +		ret = v4l2_format_to_ext_pix_format(&create->format,
>> +						    &ecreate.format, true);
>> +		if (ret)
>> +			return ret;
>> +
>> +		ret = vb2_ext_create_bufs(q, &ecreate);
>> +		if (ret)
>> +			return ret;
>> +
>> +		create->index = ecreate.index;
>> +		create->count = ecreate.count;
>> +		create->flags = ecreate.flags;
>> +		create->capabilities = ecreate.capabilities;
>> +		return 0;
>> +	}
>>  	case V4L2_BUF_TYPE_VBI_CAPTURE:
>>  	case V4L2_BUF_TYPE_VBI_OUTPUT:
>>  		requested_sizes[0] = f->fmt.vbi.samples_per_line *
>> @@ -820,8 +808,8 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_create_bufs);
>>  
>> -int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
>> -	     struct v4l2_buffer *b)
>> +int vb2_ext_qbuf(struct vb2_queue *q, struct media_device *mdev,
>> +		 struct v4l2_ext_buffer *b)
>>  {
>>  	struct media_request *req = NULL;
>>  	int ret;
>> @@ -839,9 +827,19 @@ int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
>>  		media_request_put(req);
>>  	return ret;
>>  }
>> +EXPORT_SYMBOL_GPL(vb2_ext_qbuf);
>> +
>> +int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
>> +	     struct v4l2_buffer *b)
>> +{
>> +	struct v4l2_ext_buffer eb;
>> +
>> +	return vb2_buf_to_ext_buf_op(vb2_ext_qbuf, q, mdev, &eb);
>> +}
>>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>>  
>> -int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
>> +int vb2_ext_dqbuf(struct vb2_queue *q, struct v4l2_ext_buffer *b,
>> +		  bool nonblocking)
>>  {
>>  	int ret;
>>  
>> @@ -850,7 +848,7 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
>>  		return -EBUSY;
>>  	}
>>  
>> -	if (b->type != q->type) {
>> +	if (b->type != vb2_ext_qtype(q)) {
>>  		dprintk(q, 1, "invalid buffer type\n");
>>  		return -EINVAL;
>>  	}
>> @@ -870,6 +868,14 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
>>  
>>  	return ret;
>>  }
>> +EXPORT_SYMBOL_GPL(vb2_ext_dqbuf);
>> +
>> +int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
>> +{
>> +	struct v4l2_ext_buffer eb;
>> +
>> +	return vb2_buf_to_ext_buf_op(vb2_ext_dqbuf, q, &eb, nonblocking);
>> +}
>>  EXPORT_SYMBOL_GPL(vb2_dqbuf);
>>  
>>  int vb2_streamon(struct vb2_queue *q, enum v4l2_buf_type type)
>> @@ -1040,6 +1046,33 @@ int vb2_ioctl_create_bufs(struct file *file, void *priv,
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_ioctl_create_bufs);
>>  
>> +int vb2_ioctl_ext_create_bufs(struct file *file, void *priv,
>> +			      struct v4l2_ext_create_buffers *p)
>> +{
>> +	struct video_device *vdev = video_devdata(file);
>> +	int res = vb2_verify_memory_type(vdev->queue, p->memory,
>> +		vb2_ext_type(p->format.type, vdev->queue->is_multiplanar));
>> +
>> +	p->index = vdev->queue->num_buffers;
>> +	fill_buf_caps(vdev->queue, &p->capabilities);
>> +	/*
>> +	 * If count == 0, then just check if memory and type are valid.
>> +	 * Any -EBUSY result from vb2_verify_memory_type can be mapped to 0.
>> +	 */
>> +	if (p->count == 0)
>> +		return res != -EBUSY ? res : 0;
>> +	if (res)
>> +		return res;
>> +	if (vb2_queue_is_busy(vdev, file))
>> +		return -EBUSY;
>> +
>> +	res = vb2_ext_create_bufs(vdev->queue, p);
>> +	if (res == 0)
>> +		vdev->queue->owner = file->private_data;
>> +	return res;
>> +}
>> +EXPORT_SYMBOL_GPL(vb2_ioctl_ext_create_bufs);
>> +
>>  int vb2_ioctl_prepare_buf(struct file *file, void *priv,
>>  			  struct v4l2_buffer *p)
>>  {
>> @@ -1051,6 +1084,17 @@ int vb2_ioctl_prepare_buf(struct file *file, void *priv,
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_ioctl_prepare_buf);
>>  
>> +int vb2_ioctl_ext_prepare_buf(struct file *file, void *priv,
>> +			      struct v4l2_ext_buffer *p)
>> +{
>> +	struct video_device *vdev = video_devdata(file);
>> +
>> +	if (vb2_queue_is_busy(vdev, file))
>> +		return -EBUSY;
>> +	return vb2_ext_prepare_buf(vdev->queue, vdev->v4l2_dev->mdev, p);
>> +}
>> +EXPORT_SYMBOL_GPL(vb2_ioctl_ext_prepare_buf);
>> +
>>  int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p)
>>  {
>>  	struct video_device *vdev = video_devdata(file);
>> @@ -1060,6 +1104,16 @@ int vb2_ioctl_querybuf(struct file *file, void *priv, struct v4l2_buffer *p)
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_ioctl_querybuf);
>>  
>> +int vb2_ioctl_ext_querybuf(struct file *file, void *priv,
>> +			   struct v4l2_ext_buffer *p)
>> +{
>> +	struct video_device *vdev = video_devdata(file);
>> +
>> +	/* No need to call vb2_queue_is_busy(), anyone can query buffers. */
>> +	return vb2_ext_querybuf(vdev->queue, p);
>> +}
>> +EXPORT_SYMBOL_GPL(vb2_ioctl_ext_querybuf);
>> +
>>  int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
>>  {
>>  	struct video_device *vdev = video_devdata(file);
>> @@ -1070,6 +1124,17 @@ int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_ioctl_qbuf);
>>  
>> +int vb2_ioctl_ext_qbuf(struct file *file, void *priv,
>> +		       struct v4l2_ext_buffer *p)
>> +{
>> +	struct video_device *vdev = video_devdata(file);
>> +
>> +	if (vb2_queue_is_busy(vdev, file))
>> +		return -EBUSY;
>> +	return vb2_ext_qbuf(vdev->queue, vdev->v4l2_dev->mdev, p);
>> +}
>> +EXPORT_SYMBOL_GPL(vb2_ioctl_ext_qbuf);
>> +
>>  int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
>>  {
>>  	struct video_device *vdev = video_devdata(file);
>> @@ -1080,6 +1145,17 @@ int vb2_ioctl_dqbuf(struct file *file, void *priv, struct v4l2_buffer *p)
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_ioctl_dqbuf);
>>  
>> +int vb2_ioctl_ext_dqbuf(struct file *file, void *priv,
>> +			struct v4l2_ext_buffer *p)
>> +{
>> +	struct video_device *vdev = video_devdata(file);
>> +
>> +	if (vb2_queue_is_busy(vdev, file))
>> +		return -EBUSY;
>> +	return vb2_ext_dqbuf(vdev->queue, p, file->f_flags & O_NONBLOCK);
>> +}
>> +EXPORT_SYMBOL_GPL(vb2_ioctl_ext_dqbuf);
>> +
>>  int vb2_ioctl_streamon(struct file *file, void *priv, enum v4l2_buf_type i)
>>  {
>>  	struct video_device *vdev = video_devdata(file);
>> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
>> index 52ef92049073e..f334767785bd9 100644
>> --- a/include/media/videobuf2-core.h
>> +++ b/include/media/videobuf2-core.h
>> @@ -152,6 +152,8 @@ struct vb2_mem_ops {
>>   * @mem_priv:	private data with this plane.
>>   * @dbuf:	dma_buf - shared buffer object.
>>   * @dbuf_mapped:	flag to show whether dbuf is mapped or not
>> + * @dbuf_offset: offset where the plane starts. Usually 0, unless the buffer
>> + *		 is shared by all planes of a multi-planar format.
> 
> The comment kind of implies that non-zero is a special case, but I wouldn't
> want driver authors to come to incorrect conclusions based on that. Do we
> actually need the last sentence at all?

I removed this field in next version to let drivers decide where to place the planes
in the memory buffers.

> 
>>   * @bytesused:	number of bytes occupied by data in the plane (payload).
>>   * @length:	size of this plane (NOT the payload) in bytes.
>>   * @min_length:	minimum required size of this plane (NOT the payload) in bytes.
>> @@ -175,6 +177,7 @@ struct vb2_plane {
>>  	void			*mem_priv;
>>  	struct dma_buf		*dbuf;
>>  	unsigned int		dbuf_mapped;
>> +	unsigned int		dbuf_offset;
>>  	unsigned int		bytesused;
>>  	unsigned int		length;
>>  	unsigned int		min_length;
>> @@ -446,7 +449,8 @@ struct vb2_ops {
>>   *			struct vb2_buffer.
>>   *			For V4L2 this is a &struct vb2_v4l2_buffer.
>>   * @fill_user_buffer:	given a &vb2_buffer fill in the userspace structure.
>> - *			For V4L2 this is a &struct v4l2_buffer.
>> + *			For V4L2 this is a &struct v4l2_buffer or
>> + *			&struct v4l2_ext_buffer.
> 
> Isn't it always v4l2_ext_buffer with current code?

I added a fill_user_ext_buffer for next version, since we need to keep both
v4l2_buffer and v4l2_ext_buffer to userspace.

> 
> Best regards,
> Tomasz
> 

Thanks for your comments,
Helen

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-14 10:36       ` Tomasz Figa
@ 2020-12-14 13:23         ` Helen Koike
  2020-12-15  9:03           ` Tomasz Figa
  0 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-12-14 13:23 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Boris Brezillon,
	Hirokazu Honda, Nicolas Dufresne, Brian Starkey, kernel,
	Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

Hi Tomasz,

Thank you for your comments,

On 12/14/20 7:36 AM, Tomasz Figa wrote:
> On Tue, Nov 24, 2020 at 5:33 AM Helen Koike <helen.koike@collabora.com> wrote:
>>
>> Hi Tomasz,
>>
>>
>> On 11/20/20 8:14 AM, Tomasz Figa wrote:
>>> Hi Helen,
>>>
>>> On Tue, Aug 04, 2020 at 04:29:34PM -0300, Helen Koike wrote:
>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>
>>>> Those extended buffer ops have several purpose:
>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>    the number of ns elapsed since 1970
>>>> 2/ Unify single/multiplanar handling
>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>    to support the case where a single buffer object is storing all
>>>>    planes data, each one being placed at a different offset
>>>>
>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>> these new objects.
>>>>
>>>> The core takes care of converting new ioctls requests to old ones
>>>> if the driver does not support the new hooks, and vice versa.
>>>>
>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>> use the Request API to collect more metadata information from the
>>>> frame.
>>>>
>>>
>>> Thanks for the patch. Please see my comments inline.
>>
>> Thank you for your detailed review, please see my comments below.
>>
>>>
>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>> ---
>>>> Changes in v5:
>>>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
>>>> - return mem_offset to struct v4l2_ext_plane
>>>> - change sizes and reorder fields to avoid holes in the struct and make
>>>>   it the same for 32 and 64 bits
>>>>
>>>> Changes in v4:
>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>> - Reformulate struct v4l2_ext_plane
>>>> - Fix some bugs caught by v4l2-compliance
>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>
>>>> Changes in v3:
>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>
>>>> Changes in v2:
>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>   later on
>>>> ---
>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>  include/uapi/linux/videodev2.h       |  90 +++++++
>>>>  4 files changed, 476 insertions(+), 22 deletions(-)
>>>>
>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>>>> index e1829906bc086..cb21ee8eb075c 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>>>> @@ -720,15 +720,34 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>>>              SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_sdr_out);
>>>>      }
>>>>
>>>> +    if (is_vid || is_tch) {
>>>> +            /* ioctls valid for video and touch */
>>>> +            if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
>>>> +                    set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
>>>> +            if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
>>>> +                    set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
>>>> +            if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
>>>> +                    set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
>>>> +            if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
>>>> +                    set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
>>>> +            if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
>>>> +                    set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
>>>
>>> nit: Could we stick to the SET_VALID_IOCTL() macro and just call it twice,
>>> once for the new and once for the legacy callback?
>>
>> Ack.
>>
>>>
>>>> +    }
>>>> +
>>>>      if (is_vid || is_vbi || is_sdr || is_tch || is_meta) {
>>>>              /* ioctls valid for video, vbi, sdr, touch and metadata */
>>>>              SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
>>>> -            SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
>>>> -            SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
>>>>              SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
>>>> -            SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
>>>> -            SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
>>>> -            SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
>>>> +            if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
>>>> +                    set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
>>>> +            if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
>>>> +                    set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
>>>> +            if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
>>>> +                    set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
>>>> +            if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
>>>> +                    set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
>>>> +            if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
>>>> +                    set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
>>>
>>> Is it valid to check the new callbacks for devices that the new API is not
>>> valid for (e.g. vbi)? Perhaps we could call SET_VALID_IOCTL(ops, <ioctl>,
>>> vidioc_ext_*) in the upper if added in this patch and keep the code above
>>> as is?
>>
>> Just to be clear, the only valid type should be VFL_TYPE_VIDEO right?
>> Ext but API won't support touch devices for instance, right?
>>
> 
> Yes, at least at this point. If one needs, it could be added in the
> future, but honestly I don't see much use of the other types these
> days.

Ok, I already updated this in my wip branch for next version.

> 
>>
>>>
>>>>              SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
>>>>              SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
>>>>      }
>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> index 14a0def50f8ea..7ecdd9cc1bf48 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> @@ -527,6 +527,26 @@ static void v4l_print_buffer(const void *arg, bool write_only)
>>>>                      tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
>>>>  }
>>>>
>>>> +static void v4l_print_ext_buffer(const void *arg, bool write_only)
>>>> +{
>>>> +    const struct v4l2_ext_buffer *e = arg;
>>>> +    const struct v4l2_ext_plane *plane;
>>>> +    unsigned int i;
>>>> +
>>>> +    pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d\n",
>>>> +            e->timestamp, e->index, prt_names(e->type, v4l2_type_names),
>>>> +            e->flags, prt_names(e->field, v4l2_field_names), e->sequence);
>>>
>>> Should we also print the request FD?
>>
>> Yes, I'll update this for next version.
>>
>>>
>>>> +
>>>> +    for (i = 0; i < VIDEO_MAX_PLANES &&
>>>> +                e->planes[i].buffer_length; i++) {
>>>> +            plane = &e->planes[i];
>>>> +            pr_debug("plane %d: buffer_length=%d, plane_length=%d offset=0x%08x, memory=%s\n",
>>>> +                     i, plane->buffer_length, plane->plane_length,
>>>> +                     plane->offset,
>>>> +                     prt_names(plane->memory, v4l2_memory_names));
>>>
>>> Should we also print mem_offset/userptr/dmabuf_fd?
>>
>> I see they are not printed by v4l_print_buffer(),
> 
> offset/userptr are printed in v4l2_print_buffer:
> https://elixir.bootlin.com/linux/latest/source/drivers/media/v4l2-core/v4l2-ioctl.c#L494

yes, sorry about that, I saw this later and I already updated it for next version.

> 
>> since these fields are in an
>> union, the value of two of them will be invalid (I wonder if this can bring
>> confusion).
>> I also wondered if printing them can't cause a security issue.
>>
>> But I can add those prints if you think it make sense.
>>
> 
> We know the memory type, so we can interpret the union appropriately
> and adjust the message printed.
> 
> I don't think this poses any security issue, as it prints things that
> belong to the userspace already and are only meaningful in the context
> of the given userspace process.

ok.

> 
>>>
>>>> +    }
>>>> +}
>>>> +
>>>>  static void v4l_print_exportbuffer(const void *arg, bool write_only)
>>>>  {
>>>>      const struct v4l2_exportbuffer *p = arg;
>>>> @@ -546,6 +566,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
>>>>      v4l_print_format(&p->format, write_only);
>>>>  }
>>>>
>>>> +static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
>>>> +{
>>>> +    const struct v4l2_ext_create_buffers *p = arg;
>>>> +
>>>> +    pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
>>>> +            prt_names(p->memory, v4l2_memory_names));
>>>> +    v4l_print_ext_pix_format(&p->format, write_only);
>>>
>>> Should we also print capabilities and flags?
>>
>> I just saw these prints are called after the ioctl handler, and not before,
>> to I guess it make sense.
>>
>> It is not printed by v4l_print_create_buffers(), I think we can add in both then.
>>
>>>
>>>> +}
>>>> +
>>>>  static void v4l_print_streamparm(const void *arg, bool write_only)
>>>>  {
>>>>      const struct v4l2_streamparm *p = arg;
>>>> @@ -1220,6 +1249,143 @@ int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>>>>  }
>>>>  EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>>>>
>>>> +/*
>>>> + * If mplane_cap is true, b->m.planes should have a valid pointer of a
>>>> + * struct v4l2_plane array, and b->length with its size
>>>> + */
>>>> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
>>>> +                          struct v4l2_buffer *b, bool mplane_cap)
>>>> +{
>>>> +    unsigned int planes_array_size = b->length;
>>>> +    struct v4l2_plane *planes = b->m.planes;
>>>> +    u64 nsecs;
>>>> +
>>>> +    if (!mplane_cap && e->planes[1].buffer_length != 0)
>>>> +            return -EINVAL;
>>>> +
>>>> +    memset(b, 0, sizeof(*b));
>>>> +
>>>> +    b->index = e->index;
>>>> +    b->flags = e->flags;
>>>> +    b->field = e->field;
>>>> +    b->sequence = e->sequence;
>>>> +    b->memory = e->planes[0].memory;
>>>> +    b->request_fd = e->request_fd;
>>>> +    b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
>>>> +    b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
>>>> +
>>>> +    if (mplane_cap) {
>>>> +            unsigned int i;
>>>> +
>>>> +            if (!planes || !planes_array_size)
>>>> +                    return -EINVAL;
>>>> +
>>>> +            b->m.planes = planes;
>>>
>>> planes was initialized to b->m.planes at declaration time. Should we
>>> perhaps move its declaration to within this block to make it more clear and
>>> remove this assignment?
>>
>> The variable "planes" is saving the pointer of b->m.planes before we do
>> memset(b, 0, sizeof(*b)), so I can reassing it back if it make sense.
>>
>> I can add a comment to make this more clear.
>>
> 
> Right, I think the code is a bit confusing. Should we just pass the
> planes array pointer as a separate argument to the function?

Make sense, this would make sense more clear. I'll update for next version.

> 
>>>
>>>> +
>>>> +            if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
>>>> +                    b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>>>> +            else
>>>> +                    b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
>>>> +
>>>> +            for (i = 0; i < VIDEO_MAX_PLANES && i < planes_array_size &&
>>>> +                        e->planes[i].buffer_length; i++) {
>>>> +
>>>> +                    if (e->planes[0].memory != e->planes[i].memory)
>>>> +                            return -EINVAL;
>>>> +
>>>> +                    if (e->planes[i].offset)
>>>> +                            return -EINVAL;
>>>
>>> Is it really invalid to have a non-zero offset? Shouldn't the data_offset
>>> field of the legacy struct be populated instead, in the cases where it was
>>> defined to be valid?
>>
>> My understanding of data_offset, is that it is used when the hardware can
>> write/read a header to/from the buffer.
>>
>> But this doesn't seem to be used by any driver
> 
> This is not true:
> https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/qcom/venus/venc.c#L963

Thanks for this pointer.

> 
>> , so there is an attempt to
>> repourpose it here:
>>
>>     https://patchwork.linuxtv.org/project/linux-media/patch/1429040689-23808-2-git-send-email-laurent.pinchart@ideasonboard.com/
>>
>> But this wans't merged.
>>
>> So in the current API, there is no way to specify an offset in the buffer.
>>
>> I guess we can repurpose data_offset first, what do you think?
>>
> 
> We need to stick to the original behavior for data_offset, so that an
> encoder has a way to tell the userspace how many bytes of the header
> it needs to skip.

Right, I'll update this for next version.

> 
>>>
>>>> +
>>>> +                    memset(&b->m.planes[i], 0, sizeof(b->m.planes[i]));
>>>> +
>>>> +                    if (b->memory == V4L2_MEMORY_MMAP)
>>>> +                            b->m.planes[i].m.mem_offset = e->planes[i].m.mem_offset;
>>>> +                    else if (b->memory == V4L2_MEMORY_DMABUF)
>>>> +                            b->m.planes[i].m.fd = e->planes[i].m.dmabuf_fd;
>>>> +                    else
>>>> +                            b->m.planes[i].m.userptr = e->planes[i].m.userptr;
>>>> +
>>>> +                    b->m.planes[i].bytesused = e->planes[i].plane_length;
>>>
>>> I might be getting the meaning of plane_length wrong, but doesn't this
>>> depend on the direction? If the userspace gives a CAPTURE buffer, it would
>>> have bytesused = 0, but if the kernel returns it, it would have bytesused =
>>> <size of the payload>.
>>
>> You are right, it depends on the direction, thanks for catching this.
>>
>> Also, in the Ext api, we don't have the <size of the payload>, so I'm using the
>> plane_length instead, I'm not sure if there is a better way.
> 
> I thought about this for a while and it sounds like in the code added
> by this series, plane_length is basically used as the old bytesused
> and buffer_length as the old length. Would it make sense to just
> preserve the old naming?

Make sense, I thought we could use the length of the plane to other validations,
but we don't need it. I'll preserve bytesused instead.

> 
>>
>>>
>>>> +                    b->m.planes[i].length = e->planes[i].buffer_length;
>>>> +            }
>>>> +            /* In multi-planar, length contain the number of planes */
>>>> +            b->length = i;
>>>> +    } else {
>>>> +            b->type = e->type;
>>>> +            b->bytesused = e->planes[0].plane_length;
>>>> +            b->length = e->planes[0].buffer_length;
>>>> +
>>>> +            if (e->planes[0].offset)
>>>> +                    return -EINVAL;
>>>
>>> Ditto.
>>
>> Ack.
>>
>>>
>>>> +
>>>> +            if (b->memory == V4L2_MEMORY_MMAP)
>>>> +                    b->m.offset = e->planes[0].m.mem_offset;
>>>> +            else if (b->memory == V4L2_MEMORY_DMABUF)
>>>> +                    b->m.fd = e->planes[0].m.dmabuf_fd;
>>>> +            else
>>>> +                    b->m.userptr = e->planes[0].m.userptr;
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
>>>> +
>>>> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
>>>> +                          struct v4l2_ext_buffer *e)
>>>> +{
>>>> +    memset(e, 0, sizeof(*e));
>>>> +
>>>> +    e->index = b->index;
>>>> +    e->flags = b->flags;
>>>> +    e->field = b->field;
>>>> +    e->sequence = b->sequence;
>>>> +    e->request_fd = b->request_fd;
>>>> +    e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
>>>> +            b->timestamp.tv_usec * NSEC_PER_USEC;
>>>> +    if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
>>>> +            unsigned int i;
>>>> +
>>>> +            if (!b->m.planes)
>>>> +                    return -EINVAL;
>>>> +
>>>> +            if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>>>> +                    e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>>>> +            else
>>>> +                    e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>>>> +
>>>> +            /* In multi-planar, length contain the number of planes */
>>>> +            for (i = 0; i < b->length; i++) {
>>>
>>> The design of the new struct implies that the planes describe color planes
>>> and not memory planes, so this code is incorrect, because for non-M format
>>> variants it would fill in only the first plane of the new struct.
>>
>> Yes, as discussed in 1/7, the handling of planes is wrong, I'll correct
>> this for next version.
>>
>>>
>>>> +                    if (b->memory == V4L2_MEMORY_MMAP)
>>>> +                            e->planes[i].m.mem_offset = b->m.planes[i].m.mem_offset;
>>>> +                    else if (b->memory == V4L2_MEMORY_DMABUF)
>>>> +                            e->planes[i].m.dmabuf_fd = b->m.planes[i].m.fd;
>>>> +                    else
>>>> +                            e->planes[i].m.userptr = b->m.planes[i].m.userptr;
>>>> +
>>>> +                    e->planes[i].memory = b->memory;
>>>> +                    e->planes[i].buffer_length = b->m.planes[i].length;
>>>> +                    e->planes[i].plane_length = b->m.planes[i].bytesused;
>>>> +                    if (b->m.planes[i].data_offset)
>>>> +                            pr_warn("Ignoring data_offset value %d\n",
>>>> +                                    b->m.planes[i].data_offset);
>>>
>>> Why? As per my comment above, there are valid use cases defined in the spec.
>>
>> Please see my comment about about data_offset.
>>
>>>
>>>> +            }
>>>> +    } else {
>>>> +            e->type = b->type;
>>>> +            e->planes[0].memory = b->memory;
>>>> +            e->planes[0].plane_length = b->bytesused;
>>>> +            e->planes[0].buffer_length = b->length;
>>>> +            if (b->memory == V4L2_MEMORY_MMAP)
>>>> +                    e->planes[0].m.mem_offset = b->m.offset;
>>>> +            else if (b->memory == V4L2_MEMORY_DMABUF)
>>>> +                    e->planes[0].m.dmabuf_fd = b->m.fd;
>>>> +            else
>>>> +                    e->planes[0].m.userptr = b->m.userptr;
>>>
>>> Similar to the MULTIPLANAR case, we should fill in the planes[] entries
>>> corresponding to the number of color planes of the format, e.g. 2 for NV12.
>>
>> Ack.
>>
>>>
>>>> +    }
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
>>>> +
>>>>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>>>>                              struct file *file, void *fh, void *arg)
>>>>  {
>>>> @@ -2473,31 +2639,112 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
>>>>      return ops->vidioc_reqbufs(file, fh, p);
>>>>  }
>>>>
>>>> +static int v4l_do_buf_op(int (*op)(struct file *, void *,
>>>> +                               struct v4l2_buffer *),
>>>> +                     int (*ext_op)(struct file *, void *,
>>>> +                                   struct v4l2_ext_buffer *),
>>>> +                     struct file *file, void *fh, struct v4l2_buffer *b)
>>>> +{
>>>> +    struct v4l2_ext_buffer e;
>>>> +    int ret;
>>>> +
>>>> +    ret = check_fmt(file, b->type);
>>>> +    if (ret)
>>>> +            return ret;
>>>> +
>>>> +    if (op)
>>>> +            return op(file, fh, b);
>>>> +
>>>> +    ret = v4l2_buffer_to_ext_buffer(b, &e);
>>>> +    if (ret)
>>>> +            return ret;
>>>> +
>>>> +    ret = ext_op(file, fh, &e);
>>>> +    if (ret)
>>>> +            return ret;
>>>> +
>>>> +    v4l2_ext_buffer_to_buffer(&e, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
>>>> +    return 0;
>>>> +}
>>>> +
>>>> +static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
>>>> +                                   struct v4l2_buffer *),
>>>> +                         int (*ext_op)(struct file *, void *,
>>>> +                                       struct v4l2_ext_buffer *),
>>>> +                         struct file *file, void *fh,
>>>> +                         struct v4l2_ext_buffer *e)
>>>> +{
>>>> +    struct video_device *vdev = video_devdata(file);
>>>> +    struct v4l2_plane planes[VIDEO_MAX_PLANES];
>>>> +    struct v4l2_buffer b;
>>>> +    bool mplane_cap;
>>>> +    int ret;
>>>> +
>>>> +    ret = check_fmt(file, e->type);
>>>> +    if (ret)
>>>> +            return ret;
>>>> +
>>>> +    if (ext_op)
>>>> +            return ext_op(file, fh, e);
>>>> +
>>>> +    mplane_cap = !!(vdev->device_caps &
>>>> +                    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>>>> +                     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>>>> +                     V4L2_CAP_VIDEO_M2M_MPLANE));
>>>> +    b.m.planes = planes;
>>>> +    b.length = VIDEO_MAX_PLANES;
>>>> +    ret = v4l2_ext_buffer_to_buffer(e, &b, mplane_cap);
>>>> +    if (ret)
>>>> +            return ret;
>>>> +
>>>> +    ret = op(file, fh, &b);
>>>> +    if (ret)
>>>> +            return ret;
>>>> +
>>>> +    v4l2_buffer_to_ext_buffer(&b, e);
>>>> +    return 0;
>>>> +}
>>>> +
>>>>  static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
>>>> -                            struct file *file, void *fh, void *arg)
>>>> +                    struct file *file, void *fh, void *arg)
>>>>  {
>>>> -    struct v4l2_buffer *p = arg;
>>>> -    int ret = check_fmt(file, p->type);
>>>> +    return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
>>>> +                         file, fh, arg);
>>>> +}
>>>>
>>>> -    return ret ? ret : ops->vidioc_querybuf(file, fh, p);
>>>> +static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
>>>> +                        struct file *file, void *fh, void *arg)
>>>> +{
>>>> +    return v4l_do_ext_buf_op(ops->vidioc_querybuf,
>>>> +                             ops->vidioc_ext_querybuf, file, fh, arg);
>>>>  }
>>>>
>>>>  static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
>>>> -                            struct file *file, void *fh, void *arg)
>>>> +                struct file *file, void *fh, void *arg)
>>>>  {
>>>> -    struct v4l2_buffer *p = arg;
>>>> -    int ret = check_fmt(file, p->type);
>>>> +    return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
>>>> +                         file, fh, arg);
>>>> +}
>>>>
>>>> -    return ret ? ret : ops->vidioc_qbuf(file, fh, p);
>>>> +static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
>>>> +                    struct file *file, void *fh, void *arg)
>>>> +{
>>>> +    return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
>>>> +                             file, fh, arg);
>>>>  }
>>>>
>>>>  static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
>>>> -                            struct file *file, void *fh, void *arg)
>>>> +                 struct file *file, void *fh, void *arg)
>>>>  {
>>>> -    struct v4l2_buffer *p = arg;
>>>> -    int ret = check_fmt(file, p->type);
>>>> +    return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
>>>> +                         file, fh, arg);
>>>> +}
>>>>
>>>> -    return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
>>>> +static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
>>>> +                     struct file *file, void *fh, void *arg)
>>>> +{
>>>> +    return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
>>>> +                             file, fh, arg);
>>>>  }
>>>>
>>>>  static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>>>> @@ -2513,7 +2760,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>>>>
>>>>      v4l_sanitize_format(&create->format);
>>>>
>>>> -    ret = ops->vidioc_create_bufs(file, fh, create);
>>>> +    if (ops->vidioc_create_bufs) {
>>>> +            ret = ops->vidioc_create_bufs(file, fh, create);
>>>> +    } else {
>>>> +            struct v4l2_ext_create_buffers ecreate = {
>>>> +                    .count = create->count,
>>>> +                    .memory = create->memory,
>>>> +            };
>>>> +
>>>> +            ret = v4l2_format_to_ext_pix_format(&create->format,
>>>> +                                                &ecreate.format, true);
>>>> +            if (ret)
>>>> +                    return ret;
>>>> +
>>>> +            ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
>>>> +            if (ret)
>>>> +                    return ret;
>>>> +
>>>> +            create->index = ecreate.index;
>>>> +            create->count = ecreate.count;
>>>> +            create->capabilities = ecreate.capabilities;
>>>> +    }
>>>>
>>>>      if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
>>>>          create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
>>>> @@ -2522,13 +2789,60 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
>>>>      return ret;
>>>>  }
>>>>
>>>> +static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
>>>> +                           struct file *file, void *fh, void *arg)
>>>> +{
>>>> +    struct v4l2_ext_create_buffers *ecreate = arg;
>>>> +    struct video_device *vdev = video_devdata(file);
>>>> +    struct v4l2_create_buffers create = {
>>>> +            .count = ecreate->count,
>>>> +            .memory = ecreate->memory,
>>>> +            .flags = ecreate->flags,
>>>> +    };
>>>> +    bool mplane_cap;
>>>> +    int ret;
>>>> +
>>>> +    ret = check_fmt(file, ecreate->format.type);
>>>> +    if (ret)
>>>> +            return ret;
>>>> +
>>>> +    if (ops->vidioc_ext_create_bufs)
>>>> +            return ops->vidioc_ext_create_bufs(file, fh, ecreate);
>>>> +
>>>> +    mplane_cap = !!(vdev->device_caps &
>>>> +                    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>>>> +                     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>>>> +                     V4L2_CAP_VIDEO_M2M_MPLANE));
>>>> +    ret = v4l2_ext_pix_format_to_format(&ecreate->format,
>>>> +                                        &create.format, mplane_cap, true);
>>>> +    if (ret)
>>>> +            return ret;
>>>> +
>>>> +    ret = v4l_create_bufs(ops, file, fh, &create);
>>>> +    if (ret)
>>>> +            return ret;
>>>> +
>>>> +    ecreate->index = create.index;
>>>> +    ecreate->count = create.count;
>>>> +    ecreate->capabilities = create.capabilities;
>>>> +
>>>> +    return 0;
>>>> +}
>>>> +
>>>>  static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
>>>> -                            struct file *file, void *fh, void *arg)
>>>> +                       struct file *file, void *fh, void *arg)
>>>>  {
>>>> -    struct v4l2_buffer *b = arg;
>>>> -    int ret = check_fmt(file, b->type);
>>>> +    return v4l_do_buf_op(ops->vidioc_prepare_buf,
>>>> +                         ops->vidioc_ext_prepare_buf,
>>>> +                         file, fh, arg);
>>>> +}
>>>>
>>>> -    return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
>>>> +static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
>>>> +                           struct file *file, void *fh, void *arg)
>>>> +{
>>>> +    return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
>>>> +                             ops->vidioc_ext_prepare_buf,
>>>> +                             file, fh, arg);
>>>>  }
>>>>
>>>>  static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
>>>> @@ -3202,12 +3516,15 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>>>      IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>>>>      IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>>>      IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>>>> +    IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, planes)),
>>>>      IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>>>>      IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
>>>>      IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
>>>>      IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
>>>>      IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
>>>> +    IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>>>
>>> Looking at the other entries, shouldn't this one be 1 line higher?
>>
>> Yes.
>>
>>>
>>> That said, I wonder if it wouldn't look cleaner if we just put all the
>>> EXT ioctls together at the bottom.
>>
>> I can move them and we can see if it is better or not.
>>
>>>
>>>>      IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
>>>> +    IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>>>>      IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>>>>      IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
>>>>      IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
>>>> @@ -3272,7 +3589,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>>>      IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
>>>>      IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
>>>>      IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>>> +    IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>>>      IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
>>>> +    IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
>>>>      IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
>>>>      IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
>>>>      IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
>>>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
>>>> index 8bbcb74d8ee31..75996657ad1ba 100644
>>>> --- a/include/media/v4l2-ioctl.h
>>>> +++ b/include/media/v4l2-ioctl.h
>>>> @@ -169,16 +169,26 @@ struct v4l2_fh;
>>>>   *  :ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
>>>>   * @vidioc_querybuf: pointer to the function that implements
>>>>   *  :ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
>>>> + * @vidioc_ext_querybuf: pointer to the function that implements
>>>> + *  :ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
>>>>   * @vidioc_qbuf: pointer to the function that implements
>>>>   *  :ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
>>>> + * @vidioc_ext_qbuf: pointer to the function that implements
>>>> + *  :ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
>>>>   * @vidioc_expbuf: pointer to the function that implements
>>>>   *  :ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
>>>>   * @vidioc_dqbuf: pointer to the function that implements
>>>>   *  :ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
>>>> + * @vidioc_ext_dqbuf: pointer to the function that implements
>>>> + *  :ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
>>>>   * @vidioc_create_bufs: pointer to the function that implements
>>>>   *  :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
>>>> + * @vidioc_ext_create_bufs: pointer to the function that implements
>>>> + *  :ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
>>>>   * @vidioc_prepare_buf: pointer to the function that implements
>>>>   *  :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
>>>> + * @vidioc_ext_prepare_buf: pointer to the function that implements
>>>> + *  :ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
>>>>   * @vidioc_overlay: pointer to the function that implements
>>>>   *  :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
>>>>   * @vidioc_g_fbuf: pointer to the function that implements
>>>> @@ -439,17 +449,27 @@ struct v4l2_ioctl_ops {
>>>>                            struct v4l2_requestbuffers *b);
>>>>      int (*vidioc_querybuf)(struct file *file, void *fh,
>>>>                             struct v4l2_buffer *b);
>>>> +    int (*vidioc_ext_querybuf)(struct file *file, void *fh,
>>>> +                               struct v4l2_ext_buffer *b);
>>>>      int (*vidioc_qbuf)(struct file *file, void *fh,
>>>>                         struct v4l2_buffer *b);
>>>> +    int (*vidioc_ext_qbuf)(struct file *file, void *fh,
>>>> +                           struct v4l2_ext_buffer *b);
>>>>      int (*vidioc_expbuf)(struct file *file, void *fh,
>>>>                           struct v4l2_exportbuffer *e);
>>>>      int (*vidioc_dqbuf)(struct file *file, void *fh,
>>>>                          struct v4l2_buffer *b);
>>>> +    int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
>>>> +                            struct v4l2_ext_buffer *b);
>>>>
>>>>      int (*vidioc_create_bufs)(struct file *file, void *fh,
>>>>                                struct v4l2_create_buffers *b);
>>>> +    int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
>>>> +                                  struct v4l2_ext_create_buffers *b);
>>>>      int (*vidioc_prepare_buf)(struct file *file, void *fh,
>>>>                                struct v4l2_buffer *b);
>>>> +    int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
>>>> +                                  struct v4l2_ext_buffer *b);
>>>>
>>>>      int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
>>>>      int (*vidioc_g_fbuf)(struct file *file, void *fh,
>>>> @@ -758,6 +778,12 @@ int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>>>>                                struct v4l2_format *f,
>>>>                                bool mplane_cap, bool strict);
>>>>
>>>> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
>>>> +                          struct v4l2_buffer *b,
>>>> +                          bool mplane_cap);
>>>> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
>>>> +                          struct v4l2_ext_buffer *e);
>>>> +
>>>>  /*
>>>>   * The user space interpretation of the 'v4l2_event' differs
>>>>   * based on the 'time_t' definition on 32-bit architectures, so
>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>>> index 7123c6a4d9569..334cafdd2be97 100644
>>>> --- a/include/uapi/linux/videodev2.h
>>>> +++ b/include/uapi/linux/videodev2.h
>>>> @@ -996,6 +996,41 @@ struct v4l2_plane {
>>>>      __u32                   reserved[11];
>>>>  };
>>>>
>>>> +/**
>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>> + * @buffer_length:  size of the entire buffer in bytes, should fit
>>>> + *                  @offset + @plane_length
>>>
>>> Do we actually need this buffer_length at all? We have 3 memory types:
>>>
>>> 1) MMAP - here vb2 already knows the buffer size, because it created it.
>>>
>>> 2) DMABUF - the DMA-buf kAPI provides the information about buffer size.
>>>
>>> 3) USERPTR - this might actually benefit from buffer_length, because there
>>>    are additional alignmnent requirements for the user memory, e.g. the
>>>    offset and size must be cacheline aligned.
>>>
>>> Arguably, 1) and 2) are the main usage scenarios, while the user space that
>>> uses them would have to suffer from added complexity, because of the
>>> legacy/niche case 3).
>>>
>>> Could we make this field valid only for USERPTR?
>>
>> I think so, make sense, I'll implement this for next version.
>>
>>>
>>>> + * @plane_length:   size of the plane in bytes.
>>>> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
>>>> + *                  that should be passed to mmap() called on the video node.
>>>> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>> + *                  to this plane.
>>>> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>> + *                  associated with this plane.
>>>> + * @offset:         offset in the memory buffer where the plane starts.
>>>> + * @memory:         enum v4l2_memory; the method, in which the actual video
>>>> + *                  data is passed
>>>> + * @reserved:               extra space reserved for future fields, must be set to 0.
>>>> + *
>>>> + *
>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>> + */
>>>> +struct v4l2_ext_plane {
>>>> +    __u32 buffer_length;
>>>> +    __u32 plane_length;
>>>> +    union {
>>>> +            __u32 mem_offset;
>>>> +            __u64 userptr;
>>>> +            __s32 dmabuf_fd;
>>>> +    } m;
>>>> +    __u32 offset;
>>>> +    __u32 memory;
>>>> +    __u32 reserved[4];
>>>> +};
>>>
>>> Don't we also need bytesused? Or would plane_length essentially mean the
>>> amount of space or payload, depending on the usage context?
>>
>> In my understanding, plane_length is the max amount of data the plane can
>> occupy, othersize it can overflow the buffer or mess with another plane
>> that is in the same buffer on a different offset.
>>
>> I'm probably wrong, but I don't really see why the payload size is usefull,
>> unless if we set a plane_legth that is much bigger then the data it can carry,
>> that can impact performance.
>> Payload can also be calculated from the format.
> 
> It is required for non-2D-image formats, such as compressed bitstream.
> In that case, the size of the image varies between frames and is
> usually less than the size of the buffer.

right, ok.

> 
>>
>> I can add it back if it is usefull. Please let me know your thoughts.
>>
> 
> As I mentioned above, it looks like plane_length is used almost
> exactly the same way bytesused was in the original code, so maybe it
> could just stay this way?

Ack.

> 
>>>
>>> Similarly, the original data_offset was useful as a return field, which
>>> some drivers use to indicate that the beginning of the plane is occupied by
>>> some header or otherwise irrelevant data, which must be skipped. Would the
>>> offset field be used for this purpose now?
>>
>> I didn't add an equivalent of the data_offset, since it seemed to be
>> unused (please see my comments about this above).
>>
>>>
>>>> +
>>>>  /**
>>>>   * struct v4l2_buffer - video buffer info
>>>>   * @index:  id number of the buffer
>>>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>>>>      };
>>>>  };
>>>>
>>>> +/**
>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>> + * @index:  id number of the buffer
>>>> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>> + * @flags:  buffer informational flags
>>>
>>> nit: The order of comments doesn't match the order of fields in the struct.
>>
>> Ack.
>>
>>>
>>>> + * @field:  enum v4l2_field; field order of the image in the buffer
>>>> + * @timestamp:      frame timestamp
>>>> + * @sequence:       sequence count of this frame
>>>> + * @planes: per-plane buffer information
>>>> + * @request_fd:     fd of the request that this buffer should use
>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>> + *
>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>> + * I/O methods.
>>>> + */
>>>> +struct v4l2_ext_buffer {
>>>> +    __u32 index;
>>>> +    __u32 type;
>>>> +    __u32 field;
>>>> +    __u32 sequence;
>>>> +    __u64 flags;
>>>> +    __u64 timestamp;
>>>
>>> What's the unit? How does this play with the other UAPI that the user space
>>> may use, e.g. clock_gettime() which returns struct timespec?
>>
>> The unity is nsec:
>>
>>         e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
>>                 b->timestamp.tv_usec * NSEC_PER_USEC;
>>
>> I can clarify in the docs, is this ok?
>>
> 
> Yes, it definitely needs to be documented. That said, what's the
> rationale for switching from the timeval representation to the flat
> nsec-based one?

According to https://patchwork.kernel.org/project/linux-media/patch/20190319145243.25047-4-boris.brezillon@collabora.com/
This avoids y2038 issue.

Regards,
Helen

> 
> Best regards,
> Tomasz
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-14 13:23         ` Helen Koike
@ 2020-12-15  9:03           ` Tomasz Figa
  0 siblings, 0 replies; 51+ messages in thread
From: Tomasz Figa @ 2020-12-15  9:03 UTC (permalink / raw)
  To: Helen Koike
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Boris Brezillon,
	Hirokazu Honda, Nicolas Dufresne, Brian Starkey, kernel,
	Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

On Mon, Dec 14, 2020 at 10:24 PM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi Tomasz,
>
> Thank you for your comments,
>
> On 12/14/20 7:36 AM, Tomasz Figa wrote:
> > On Tue, Nov 24, 2020 at 5:33 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>
> >> Hi Tomasz,
> >>
> >>
> >> On 11/20/20 8:14 AM, Tomasz Figa wrote:
> >>> Hi Helen,
> >>>
> >>> On Tue, Aug 04, 2020 at 04:29:34PM -0300, Helen Koike wrote:
> >>>> From: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>
> >>>> Those extended buffer ops have several purpose:
> >>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
> >>>>    the number of ns elapsed since 1970
> >>>> 2/ Unify single/multiplanar handling
> >>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
> >>>>    to support the case where a single buffer object is storing all
> >>>>    planes data, each one being placed at a different offset
> >>>>
> >>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> >>>> these new objects.
> >>>>
> >>>> The core takes care of converting new ioctls requests to old ones
> >>>> if the driver does not support the new hooks, and vice versa.
> >>>>
> >>>> Note that the timecode field is gone, since there doesn't seem to be
> >>>> in-kernel users. We can be added back in the reserved area if needed or
> >>>> use the Request API to collect more metadata information from the
> >>>> frame.
> >>>>
> >>>
> >>> Thanks for the patch. Please see my comments inline.
> >>
> >> Thank you for your detailed review, please see my comments below.
> >>
> >>>
> >>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> >>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> >>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>>> ---
> >>>> Changes in v5:
> >>>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> >>>> - return mem_offset to struct v4l2_ext_plane
> >>>> - change sizes and reorder fields to avoid holes in the struct and make
> >>>>   it the same for 32 and 64 bits
> >>>>
> >>>> Changes in v4:
> >>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> >>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> >>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> >>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> >>>> I think we can add this later, so I removed it from this RFC to simplify it.
> >>>> - Remove num_planes field from struct v4l2_ext_buffer
> >>>> - Add flags field to struct v4l2_ext_create_buffers
> >>>> - Reformulate struct v4l2_ext_plane
> >>>> - Fix some bugs caught by v4l2-compliance
> >>>> - Rebased on top of media/master (post 5.8-rc1)
> >>>>
> >>>> Changes in v3:
> >>>> - Rebased on top of media/master (post 5.4-rc1)
> >>>>
> >>>> Changes in v2:
> >>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
> >>>>   later on
> >>>> ---
> >>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
> >>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
> >>>>  include/media/v4l2-ioctl.h           |  26 ++
> >>>>  include/uapi/linux/videodev2.h       |  90 +++++++
> >>>>  4 files changed, 476 insertions(+), 22 deletions(-)
> >>>>
> >>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> >>>> index e1829906bc086..cb21ee8eb075c 100644
> >>>> --- a/drivers/media/v4l2-core/v4l2-dev.c
> >>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> >>>> @@ -720,15 +720,34 @@ static void determine_valid_ioctls(struct video_device *vdev)
> >>>>              SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_sdr_out);
> >>>>      }
> >>>>
> >>>> +    if (is_vid || is_tch) {
> >>>> +            /* ioctls valid for video and touch */
> >>>> +            if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
> >>>> +                    set_bit(_IOC_NR(VIDIOC_EXT_QUERYBUF), valid_ioctls);
> >>>> +            if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
> >>>> +                    set_bit(_IOC_NR(VIDIOC_EXT_QBUF), valid_ioctls);
> >>>> +            if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
> >>>> +                    set_bit(_IOC_NR(VIDIOC_EXT_DQBUF), valid_ioctls);
> >>>> +            if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
> >>>> +                    set_bit(_IOC_NR(VIDIOC_EXT_CREATE_BUFS), valid_ioctls);
> >>>> +            if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
> >>>> +                    set_bit(_IOC_NR(VIDIOC_EXT_PREPARE_BUF), valid_ioctls);
> >>>
> >>> nit: Could we stick to the SET_VALID_IOCTL() macro and just call it twice,
> >>> once for the new and once for the legacy callback?
> >>
> >> Ack.
> >>
> >>>
> >>>> +    }
> >>>> +
> >>>>      if (is_vid || is_vbi || is_sdr || is_tch || is_meta) {
> >>>>              /* ioctls valid for video, vbi, sdr, touch and metadata */
> >>>>              SET_VALID_IOCTL(ops, VIDIOC_REQBUFS, vidioc_reqbufs);
> >>>> -            SET_VALID_IOCTL(ops, VIDIOC_QUERYBUF, vidioc_querybuf);
> >>>> -            SET_VALID_IOCTL(ops, VIDIOC_QBUF, vidioc_qbuf);
> >>>>              SET_VALID_IOCTL(ops, VIDIOC_EXPBUF, vidioc_expbuf);
> >>>> -            SET_VALID_IOCTL(ops, VIDIOC_DQBUF, vidioc_dqbuf);
> >>>> -            SET_VALID_IOCTL(ops, VIDIOC_CREATE_BUFS, vidioc_create_bufs);
> >>>> -            SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, vidioc_prepare_buf);
> >>>> +            if (ops->vidioc_querybuf || ops->vidioc_ext_querybuf)
> >>>> +                    set_bit(_IOC_NR(VIDIOC_QUERYBUF), valid_ioctls);
> >>>> +            if (ops->vidioc_qbuf || ops->vidioc_ext_qbuf)
> >>>> +                    set_bit(_IOC_NR(VIDIOC_QBUF), valid_ioctls);
> >>>> +            if (ops->vidioc_dqbuf || ops->vidioc_ext_dqbuf)
> >>>> +                    set_bit(_IOC_NR(VIDIOC_DQBUF), valid_ioctls);
> >>>> +            if (ops->vidioc_create_bufs || ops->vidioc_ext_create_bufs)
> >>>> +                    set_bit(_IOC_NR(VIDIOC_CREATE_BUFS), valid_ioctls);
> >>>> +            if (ops->vidioc_prepare_buf || ops->vidioc_ext_prepare_buf)
> >>>> +                    set_bit(_IOC_NR(VIDIOC_PREPARE_BUF), valid_ioctls);
> >>>
> >>> Is it valid to check the new callbacks for devices that the new API is not
> >>> valid for (e.g. vbi)? Perhaps we could call SET_VALID_IOCTL(ops, <ioctl>,
> >>> vidioc_ext_*) in the upper if added in this patch and keep the code above
> >>> as is?
> >>
> >> Just to be clear, the only valid type should be VFL_TYPE_VIDEO right?
> >> Ext but API won't support touch devices for instance, right?
> >>
> >
> > Yes, at least at this point. If one needs, it could be added in the
> > future, but honestly I don't see much use of the other types these
> > days.
>
> Ok, I already updated this in my wip branch for next version.
>
> >
> >>
> >>>
> >>>>              SET_VALID_IOCTL(ops, VIDIOC_STREAMON, vidioc_streamon);
> >>>>              SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, vidioc_streamoff);
> >>>>      }
> >>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> >>>> index 14a0def50f8ea..7ecdd9cc1bf48 100644
> >>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> >>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> >>>> @@ -527,6 +527,26 @@ static void v4l_print_buffer(const void *arg, bool write_only)
> >>>>                      tc->type, tc->flags, tc->frames, *(__u32 *)tc->userbits);
> >>>>  }
> >>>>
> >>>> +static void v4l_print_ext_buffer(const void *arg, bool write_only)
> >>>> +{
> >>>> +    const struct v4l2_ext_buffer *e = arg;
> >>>> +    const struct v4l2_ext_plane *plane;
> >>>> +    unsigned int i;
> >>>> +
> >>>> +    pr_cont("%lld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d\n",
> >>>> +            e->timestamp, e->index, prt_names(e->type, v4l2_type_names),
> >>>> +            e->flags, prt_names(e->field, v4l2_field_names), e->sequence);
> >>>
> >>> Should we also print the request FD?
> >>
> >> Yes, I'll update this for next version.
> >>
> >>>
> >>>> +
> >>>> +    for (i = 0; i < VIDEO_MAX_PLANES &&
> >>>> +                e->planes[i].buffer_length; i++) {
> >>>> +            plane = &e->planes[i];
> >>>> +            pr_debug("plane %d: buffer_length=%d, plane_length=%d offset=0x%08x, memory=%s\n",
> >>>> +                     i, plane->buffer_length, plane->plane_length,
> >>>> +                     plane->offset,
> >>>> +                     prt_names(plane->memory, v4l2_memory_names));
> >>>
> >>> Should we also print mem_offset/userptr/dmabuf_fd?
> >>
> >> I see they are not printed by v4l_print_buffer(),
> >
> > offset/userptr are printed in v4l2_print_buffer:
> > https://elixir.bootlin.com/linux/latest/source/drivers/media/v4l2-core/v4l2-ioctl.c#L494
>
> yes, sorry about that, I saw this later and I already updated it for next version.
>
> >
> >> since these fields are in an
> >> union, the value of two of them will be invalid (I wonder if this can bring
> >> confusion).
> >> I also wondered if printing them can't cause a security issue.
> >>
> >> But I can add those prints if you think it make sense.
> >>
> >
> > We know the memory type, so we can interpret the union appropriately
> > and adjust the message printed.
> >
> > I don't think this poses any security issue, as it prints things that
> > belong to the userspace already and are only meaningful in the context
> > of the given userspace process.
>
> ok.
>
> >
> >>>
> >>>> +    }
> >>>> +}
> >>>> +
> >>>>  static void v4l_print_exportbuffer(const void *arg, bool write_only)
> >>>>  {
> >>>>      const struct v4l2_exportbuffer *p = arg;
> >>>> @@ -546,6 +566,15 @@ static void v4l_print_create_buffers(const void *arg, bool write_only)
> >>>>      v4l_print_format(&p->format, write_only);
> >>>>  }
> >>>>
> >>>> +static void v4l_print_ext_create_buffers(const void *arg, bool write_only)
> >>>> +{
> >>>> +    const struct v4l2_ext_create_buffers *p = arg;
> >>>> +
> >>>> +    pr_cont("index=%d, count=%d, memory=%s, ", p->index, p->count,
> >>>> +            prt_names(p->memory, v4l2_memory_names));
> >>>> +    v4l_print_ext_pix_format(&p->format, write_only);
> >>>
> >>> Should we also print capabilities and flags?
> >>
> >> I just saw these prints are called after the ioctl handler, and not before,
> >> to I guess it make sense.
> >>
> >> It is not printed by v4l_print_create_buffers(), I think we can add in both then.
> >>
> >>>
> >>>> +}
> >>>> +
> >>>>  static void v4l_print_streamparm(const void *arg, bool write_only)
> >>>>  {
> >>>>      const struct v4l2_streamparm *p = arg;
> >>>> @@ -1220,6 +1249,143 @@ int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> >>>>  }
> >>>>  EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
> >>>>
> >>>> +/*
> >>>> + * If mplane_cap is true, b->m.planes should have a valid pointer of a
> >>>> + * struct v4l2_plane array, and b->length with its size
> >>>> + */
> >>>> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
> >>>> +                          struct v4l2_buffer *b, bool mplane_cap)
> >>>> +{
> >>>> +    unsigned int planes_array_size = b->length;
> >>>> +    struct v4l2_plane *planes = b->m.planes;
> >>>> +    u64 nsecs;
> >>>> +
> >>>> +    if (!mplane_cap && e->planes[1].buffer_length != 0)
> >>>> +            return -EINVAL;
> >>>> +
> >>>> +    memset(b, 0, sizeof(*b));
> >>>> +
> >>>> +    b->index = e->index;
> >>>> +    b->flags = e->flags;
> >>>> +    b->field = e->field;
> >>>> +    b->sequence = e->sequence;
> >>>> +    b->memory = e->planes[0].memory;
> >>>> +    b->request_fd = e->request_fd;
> >>>> +    b->timestamp.tv_sec = div64_u64_rem(e->timestamp, NSEC_PER_SEC, &nsecs);
> >>>> +    b->timestamp.tv_usec = (u32)nsecs / NSEC_PER_USEC;
> >>>> +
> >>>> +    if (mplane_cap) {
> >>>> +            unsigned int i;
> >>>> +
> >>>> +            if (!planes || !planes_array_size)
> >>>> +                    return -EINVAL;
> >>>> +
> >>>> +            b->m.planes = planes;
> >>>
> >>> planes was initialized to b->m.planes at declaration time. Should we
> >>> perhaps move its declaration to within this block to make it more clear and
> >>> remove this assignment?
> >>
> >> The variable "planes" is saving the pointer of b->m.planes before we do
> >> memset(b, 0, sizeof(*b)), so I can reassing it back if it make sense.
> >>
> >> I can add a comment to make this more clear.
> >>
> >
> > Right, I think the code is a bit confusing. Should we just pass the
> > planes array pointer as a separate argument to the function?
>
> Make sense, this would make sense more clear. I'll update for next version.
>
> >
> >>>
> >>>> +
> >>>> +            if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> >>>> +                    b->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> >>>> +            else
> >>>> +                    b->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> >>>> +
> >>>> +            for (i = 0; i < VIDEO_MAX_PLANES && i < planes_array_size &&
> >>>> +                        e->planes[i].buffer_length; i++) {
> >>>> +
> >>>> +                    if (e->planes[0].memory != e->planes[i].memory)
> >>>> +                            return -EINVAL;
> >>>> +
> >>>> +                    if (e->planes[i].offset)
> >>>> +                            return -EINVAL;
> >>>
> >>> Is it really invalid to have a non-zero offset? Shouldn't the data_offset
> >>> field of the legacy struct be populated instead, in the cases where it was
> >>> defined to be valid?
> >>
> >> My understanding of data_offset, is that it is used when the hardware can
> >> write/read a header to/from the buffer.
> >>
> >> But this doesn't seem to be used by any driver
> >
> > This is not true:
> > https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/qcom/venus/venc.c#L963
>
> Thanks for this pointer.
>
> >
> >> , so there is an attempt to
> >> repourpose it here:
> >>
> >>     https://patchwork.linuxtv.org/project/linux-media/patch/1429040689-23808-2-git-send-email-laurent.pinchart@ideasonboard.com/
> >>
> >> But this wans't merged.
> >>
> >> So in the current API, there is no way to specify an offset in the buffer.
> >>
> >> I guess we can repurpose data_offset first, what do you think?
> >>
> >
> > We need to stick to the original behavior for data_offset, so that an
> > encoder has a way to tell the userspace how many bytes of the header
> > it needs to skip.
>
> Right, I'll update this for next version.
>
> >
> >>>
> >>>> +
> >>>> +                    memset(&b->m.planes[i], 0, sizeof(b->m.planes[i]));
> >>>> +
> >>>> +                    if (b->memory == V4L2_MEMORY_MMAP)
> >>>> +                            b->m.planes[i].m.mem_offset = e->planes[i].m.mem_offset;
> >>>> +                    else if (b->memory == V4L2_MEMORY_DMABUF)
> >>>> +                            b->m.planes[i].m.fd = e->planes[i].m.dmabuf_fd;
> >>>> +                    else
> >>>> +                            b->m.planes[i].m.userptr = e->planes[i].m.userptr;
> >>>> +
> >>>> +                    b->m.planes[i].bytesused = e->planes[i].plane_length;
> >>>
> >>> I might be getting the meaning of plane_length wrong, but doesn't this
> >>> depend on the direction? If the userspace gives a CAPTURE buffer, it would
> >>> have bytesused = 0, but if the kernel returns it, it would have bytesused =
> >>> <size of the payload>.
> >>
> >> You are right, it depends on the direction, thanks for catching this.
> >>
> >> Also, in the Ext api, we don't have the <size of the payload>, so I'm using the
> >> plane_length instead, I'm not sure if there is a better way.
> >
> > I thought about this for a while and it sounds like in the code added
> > by this series, plane_length is basically used as the old bytesused
> > and buffer_length as the old length. Would it make sense to just
> > preserve the old naming?
>
> Make sense, I thought we could use the length of the plane to other validations,
> but we don't need it. I'll preserve bytesused instead.
>
> >
> >>
> >>>
> >>>> +                    b->m.planes[i].length = e->planes[i].buffer_length;
> >>>> +            }
> >>>> +            /* In multi-planar, length contain the number of planes */
> >>>> +            b->length = i;
> >>>> +    } else {
> >>>> +            b->type = e->type;
> >>>> +            b->bytesused = e->planes[0].plane_length;
> >>>> +            b->length = e->planes[0].buffer_length;
> >>>> +
> >>>> +            if (e->planes[0].offset)
> >>>> +                    return -EINVAL;
> >>>
> >>> Ditto.
> >>
> >> Ack.
> >>
> >>>
> >>>> +
> >>>> +            if (b->memory == V4L2_MEMORY_MMAP)
> >>>> +                    b->m.offset = e->planes[0].m.mem_offset;
> >>>> +            else if (b->memory == V4L2_MEMORY_DMABUF)
> >>>> +                    b->m.fd = e->planes[0].m.dmabuf_fd;
> >>>> +            else
> >>>> +                    b->m.userptr = e->planes[0].m.userptr;
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +EXPORT_SYMBOL_GPL(v4l2_ext_buffer_to_buffer);
> >>>> +
> >>>> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
> >>>> +                          struct v4l2_ext_buffer *e)
> >>>> +{
> >>>> +    memset(e, 0, sizeof(*e));
> >>>> +
> >>>> +    e->index = b->index;
> >>>> +    e->flags = b->flags;
> >>>> +    e->field = b->field;
> >>>> +    e->sequence = b->sequence;
> >>>> +    e->request_fd = b->request_fd;
> >>>> +    e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
> >>>> +            b->timestamp.tv_usec * NSEC_PER_USEC;
> >>>> +    if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
> >>>> +            unsigned int i;
> >>>> +
> >>>> +            if (!b->m.planes)
> >>>> +                    return -EINVAL;
> >>>> +
> >>>> +            if (b->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> >>>> +                    e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> >>>> +            else
> >>>> +                    e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> >>>> +
> >>>> +            /* In multi-planar, length contain the number of planes */
> >>>> +            for (i = 0; i < b->length; i++) {
> >>>
> >>> The design of the new struct implies that the planes describe color planes
> >>> and not memory planes, so this code is incorrect, because for non-M format
> >>> variants it would fill in only the first plane of the new struct.
> >>
> >> Yes, as discussed in 1/7, the handling of planes is wrong, I'll correct
> >> this for next version.
> >>
> >>>
> >>>> +                    if (b->memory == V4L2_MEMORY_MMAP)
> >>>> +                            e->planes[i].m.mem_offset = b->m.planes[i].m.mem_offset;
> >>>> +                    else if (b->memory == V4L2_MEMORY_DMABUF)
> >>>> +                            e->planes[i].m.dmabuf_fd = b->m.planes[i].m.fd;
> >>>> +                    else
> >>>> +                            e->planes[i].m.userptr = b->m.planes[i].m.userptr;
> >>>> +
> >>>> +                    e->planes[i].memory = b->memory;
> >>>> +                    e->planes[i].buffer_length = b->m.planes[i].length;
> >>>> +                    e->planes[i].plane_length = b->m.planes[i].bytesused;
> >>>> +                    if (b->m.planes[i].data_offset)
> >>>> +                            pr_warn("Ignoring data_offset value %d\n",
> >>>> +                                    b->m.planes[i].data_offset);
> >>>
> >>> Why? As per my comment above, there are valid use cases defined in the spec.
> >>
> >> Please see my comment about about data_offset.
> >>
> >>>
> >>>> +            }
> >>>> +    } else {
> >>>> +            e->type = b->type;
> >>>> +            e->planes[0].memory = b->memory;
> >>>> +            e->planes[0].plane_length = b->bytesused;
> >>>> +            e->planes[0].buffer_length = b->length;
> >>>> +            if (b->memory == V4L2_MEMORY_MMAP)
> >>>> +                    e->planes[0].m.mem_offset = b->m.offset;
> >>>> +            else if (b->memory == V4L2_MEMORY_DMABUF)
> >>>> +                    e->planes[0].m.dmabuf_fd = b->m.fd;
> >>>> +            else
> >>>> +                    e->planes[0].m.userptr = b->m.userptr;
> >>>
> >>> Similar to the MULTIPLANAR case, we should fill in the planes[] entries
> >>> corresponding to the number of color planes of the format, e.g. 2 for NV12.
> >>
> >> Ack.
> >>
> >>>
> >>>> +    }
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +EXPORT_SYMBOL_GPL(v4l2_buffer_to_ext_buffer);
> >>>> +
> >>>>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
> >>>>                              struct file *file, void *fh, void *arg)
> >>>>  {
> >>>> @@ -2473,31 +2639,112 @@ static int v4l_reqbufs(const struct v4l2_ioctl_ops *ops,
> >>>>      return ops->vidioc_reqbufs(file, fh, p);
> >>>>  }
> >>>>
> >>>> +static int v4l_do_buf_op(int (*op)(struct file *, void *,
> >>>> +                               struct v4l2_buffer *),
> >>>> +                     int (*ext_op)(struct file *, void *,
> >>>> +                                   struct v4l2_ext_buffer *),
> >>>> +                     struct file *file, void *fh, struct v4l2_buffer *b)
> >>>> +{
> >>>> +    struct v4l2_ext_buffer e;
> >>>> +    int ret;
> >>>> +
> >>>> +    ret = check_fmt(file, b->type);
> >>>> +    if (ret)
> >>>> +            return ret;
> >>>> +
> >>>> +    if (op)
> >>>> +            return op(file, fh, b);
> >>>> +
> >>>> +    ret = v4l2_buffer_to_ext_buffer(b, &e);
> >>>> +    if (ret)
> >>>> +            return ret;
> >>>> +
> >>>> +    ret = ext_op(file, fh, &e);
> >>>> +    if (ret)
> >>>> +            return ret;
> >>>> +
> >>>> +    v4l2_ext_buffer_to_buffer(&e, b, V4L2_TYPE_IS_MULTIPLANAR(b->type));
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>> +static int v4l_do_ext_buf_op(int (*op)(struct file *, void *,
> >>>> +                                   struct v4l2_buffer *),
> >>>> +                         int (*ext_op)(struct file *, void *,
> >>>> +                                       struct v4l2_ext_buffer *),
> >>>> +                         struct file *file, void *fh,
> >>>> +                         struct v4l2_ext_buffer *e)
> >>>> +{
> >>>> +    struct video_device *vdev = video_devdata(file);
> >>>> +    struct v4l2_plane planes[VIDEO_MAX_PLANES];
> >>>> +    struct v4l2_buffer b;
> >>>> +    bool mplane_cap;
> >>>> +    int ret;
> >>>> +
> >>>> +    ret = check_fmt(file, e->type);
> >>>> +    if (ret)
> >>>> +            return ret;
> >>>> +
> >>>> +    if (ext_op)
> >>>> +            return ext_op(file, fh, e);
> >>>> +
> >>>> +    mplane_cap = !!(vdev->device_caps &
> >>>> +                    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> >>>> +                     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> >>>> +                     V4L2_CAP_VIDEO_M2M_MPLANE));
> >>>> +    b.m.planes = planes;
> >>>> +    b.length = VIDEO_MAX_PLANES;
> >>>> +    ret = v4l2_ext_buffer_to_buffer(e, &b, mplane_cap);
> >>>> +    if (ret)
> >>>> +            return ret;
> >>>> +
> >>>> +    ret = op(file, fh, &b);
> >>>> +    if (ret)
> >>>> +            return ret;
> >>>> +
> >>>> +    v4l2_buffer_to_ext_buffer(&b, e);
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>>  static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
> >>>> -                            struct file *file, void *fh, void *arg)
> >>>> +                    struct file *file, void *fh, void *arg)
> >>>>  {
> >>>> -    struct v4l2_buffer *p = arg;
> >>>> -    int ret = check_fmt(file, p->type);
> >>>> +    return v4l_do_buf_op(ops->vidioc_querybuf, ops->vidioc_ext_querybuf,
> >>>> +                         file, fh, arg);
> >>>> +}
> >>>>
> >>>> -    return ret ? ret : ops->vidioc_querybuf(file, fh, p);
> >>>> +static int v4l_ext_querybuf(const struct v4l2_ioctl_ops *ops,
> >>>> +                        struct file *file, void *fh, void *arg)
> >>>> +{
> >>>> +    return v4l_do_ext_buf_op(ops->vidioc_querybuf,
> >>>> +                             ops->vidioc_ext_querybuf, file, fh, arg);
> >>>>  }
> >>>>
> >>>>  static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
> >>>> -                            struct file *file, void *fh, void *arg)
> >>>> +                struct file *file, void *fh, void *arg)
> >>>>  {
> >>>> -    struct v4l2_buffer *p = arg;
> >>>> -    int ret = check_fmt(file, p->type);
> >>>> +    return v4l_do_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
> >>>> +                         file, fh, arg);
> >>>> +}
> >>>>
> >>>> -    return ret ? ret : ops->vidioc_qbuf(file, fh, p);
> >>>> +static int v4l_ext_qbuf(const struct v4l2_ioctl_ops *ops,
> >>>> +                    struct file *file, void *fh, void *arg)
> >>>> +{
> >>>> +    return v4l_do_ext_buf_op(ops->vidioc_qbuf, ops->vidioc_ext_qbuf,
> >>>> +                             file, fh, arg);
> >>>>  }
> >>>>
> >>>>  static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
> >>>> -                            struct file *file, void *fh, void *arg)
> >>>> +                 struct file *file, void *fh, void *arg)
> >>>>  {
> >>>> -    struct v4l2_buffer *p = arg;
> >>>> -    int ret = check_fmt(file, p->type);
> >>>> +    return v4l_do_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
> >>>> +                         file, fh, arg);
> >>>> +}
> >>>>
> >>>> -    return ret ? ret : ops->vidioc_dqbuf(file, fh, p);
> >>>> +static int v4l_ext_dqbuf(const struct v4l2_ioctl_ops *ops,
> >>>> +                     struct file *file, void *fh, void *arg)
> >>>> +{
> >>>> +    return v4l_do_ext_buf_op(ops->vidioc_dqbuf, ops->vidioc_ext_dqbuf,
> >>>> +                             file, fh, arg);
> >>>>  }
> >>>>
> >>>>  static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
> >>>> @@ -2513,7 +2760,27 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
> >>>>
> >>>>      v4l_sanitize_format(&create->format);
> >>>>
> >>>> -    ret = ops->vidioc_create_bufs(file, fh, create);
> >>>> +    if (ops->vidioc_create_bufs) {
> >>>> +            ret = ops->vidioc_create_bufs(file, fh, create);
> >>>> +    } else {
> >>>> +            struct v4l2_ext_create_buffers ecreate = {
> >>>> +                    .count = create->count,
> >>>> +                    .memory = create->memory,
> >>>> +            };
> >>>> +
> >>>> +            ret = v4l2_format_to_ext_pix_format(&create->format,
> >>>> +                                                &ecreate.format, true);
> >>>> +            if (ret)
> >>>> +                    return ret;
> >>>> +
> >>>> +            ret = ops->vidioc_ext_create_bufs(file, fh, &ecreate);
> >>>> +            if (ret)
> >>>> +                    return ret;
> >>>> +
> >>>> +            create->index = ecreate.index;
> >>>> +            create->count = ecreate.count;
> >>>> +            create->capabilities = ecreate.capabilities;
> >>>> +    }
> >>>>
> >>>>      if (create->format.type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
> >>>>          create->format.type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
> >>>> @@ -2522,13 +2789,60 @@ static int v4l_create_bufs(const struct v4l2_ioctl_ops *ops,
> >>>>      return ret;
> >>>>  }
> >>>>
> >>>> +static int v4l_ext_create_bufs(const struct v4l2_ioctl_ops *ops,
> >>>> +                           struct file *file, void *fh, void *arg)
> >>>> +{
> >>>> +    struct v4l2_ext_create_buffers *ecreate = arg;
> >>>> +    struct video_device *vdev = video_devdata(file);
> >>>> +    struct v4l2_create_buffers create = {
> >>>> +            .count = ecreate->count,
> >>>> +            .memory = ecreate->memory,
> >>>> +            .flags = ecreate->flags,
> >>>> +    };
> >>>> +    bool mplane_cap;
> >>>> +    int ret;
> >>>> +
> >>>> +    ret = check_fmt(file, ecreate->format.type);
> >>>> +    if (ret)
> >>>> +            return ret;
> >>>> +
> >>>> +    if (ops->vidioc_ext_create_bufs)
> >>>> +            return ops->vidioc_ext_create_bufs(file, fh, ecreate);
> >>>> +
> >>>> +    mplane_cap = !!(vdev->device_caps &
> >>>> +                    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> >>>> +                     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> >>>> +                     V4L2_CAP_VIDEO_M2M_MPLANE));
> >>>> +    ret = v4l2_ext_pix_format_to_format(&ecreate->format,
> >>>> +                                        &create.format, mplane_cap, true);
> >>>> +    if (ret)
> >>>> +            return ret;
> >>>> +
> >>>> +    ret = v4l_create_bufs(ops, file, fh, &create);
> >>>> +    if (ret)
> >>>> +            return ret;
> >>>> +
> >>>> +    ecreate->index = create.index;
> >>>> +    ecreate->count = create.count;
> >>>> +    ecreate->capabilities = create.capabilities;
> >>>> +
> >>>> +    return 0;
> >>>> +}
> >>>> +
> >>>>  static int v4l_prepare_buf(const struct v4l2_ioctl_ops *ops,
> >>>> -                            struct file *file, void *fh, void *arg)
> >>>> +                       struct file *file, void *fh, void *arg)
> >>>>  {
> >>>> -    struct v4l2_buffer *b = arg;
> >>>> -    int ret = check_fmt(file, b->type);
> >>>> +    return v4l_do_buf_op(ops->vidioc_prepare_buf,
> >>>> +                         ops->vidioc_ext_prepare_buf,
> >>>> +                         file, fh, arg);
> >>>> +}
> >>>>
> >>>> -    return ret ? ret : ops->vidioc_prepare_buf(file, fh, b);
> >>>> +static int v4l_ext_prepare_buf(const struct v4l2_ioctl_ops *ops,
> >>>> +                           struct file *file, void *fh, void *arg)
> >>>> +{
> >>>> +    return v4l_do_ext_buf_op(ops->vidioc_prepare_buf,
> >>>> +                             ops->vidioc_ext_prepare_buf,
> >>>> +                             file, fh, arg);
> >>>>  }
> >>>>
> >>>>  static int v4l_g_parm(const struct v4l2_ioctl_ops *ops,
> >>>> @@ -3202,12 +3516,15 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
> >>>>      IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
> >>>>      IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
> >>>>      IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
> >>>> +    IOCTL_INFO(VIDIOC_EXT_QUERYBUF, v4l_ext_querybuf, v4l_print_ext_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_ext_buffer, planes)),
> >>>>      IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
> >>>>      IOCTL_INFO(VIDIOC_S_FBUF, v4l_stub_s_fbuf, v4l_print_framebuffer, INFO_FL_PRIO),
> >>>>      IOCTL_INFO(VIDIOC_OVERLAY, v4l_overlay, v4l_print_u32, INFO_FL_PRIO),
> >>>>      IOCTL_INFO(VIDIOC_QBUF, v4l_qbuf, v4l_print_buffer, INFO_FL_QUEUE),
> >>>>      IOCTL_INFO(VIDIOC_EXPBUF, v4l_stub_expbuf, v4l_print_exportbuffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_exportbuffer, flags)),
> >>>> +    IOCTL_INFO(VIDIOC_EXT_QBUF, v4l_ext_qbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
> >>>
> >>> Looking at the other entries, shouldn't this one be 1 line higher?
> >>
> >> Yes.
> >>
> >>>
> >>> That said, I wonder if it wouldn't look cleaner if we just put all the
> >>> EXT ioctls together at the bottom.
> >>
> >> I can move them and we can see if it is better or not.
> >>
> >>>
> >>>>      IOCTL_INFO(VIDIOC_DQBUF, v4l_dqbuf, v4l_print_buffer, INFO_FL_QUEUE),
> >>>> +    IOCTL_INFO(VIDIOC_EXT_DQBUF, v4l_ext_dqbuf, v4l_print_ext_buffer, INFO_FL_QUEUE),
> >>>>      IOCTL_INFO(VIDIOC_STREAMON, v4l_streamon, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
> >>>>      IOCTL_INFO(VIDIOC_STREAMOFF, v4l_streamoff, v4l_print_buftype, INFO_FL_PRIO | INFO_FL_QUEUE),
> >>>>      IOCTL_INFO(VIDIOC_G_PARM, v4l_g_parm, v4l_print_streamparm, INFO_FL_CLEAR(v4l2_streamparm, type)),
> >>>> @@ -3272,7 +3589,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
> >>>>      IOCTL_INFO(VIDIOC_SUBSCRIBE_EVENT, v4l_subscribe_event, v4l_print_event_subscription, 0),
> >>>>      IOCTL_INFO(VIDIOC_UNSUBSCRIBE_EVENT, v4l_unsubscribe_event, v4l_print_event_subscription, 0),
> >>>>      IOCTL_INFO(VIDIOC_CREATE_BUFS, v4l_create_bufs, v4l_print_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
> >>>> +    IOCTL_INFO(VIDIOC_EXT_CREATE_BUFS, v4l_ext_create_bufs, v4l_print_ext_create_buffers, INFO_FL_PRIO | INFO_FL_QUEUE),
> >>>>      IOCTL_INFO(VIDIOC_PREPARE_BUF, v4l_prepare_buf, v4l_print_buffer, INFO_FL_QUEUE),
> >>>> +    IOCTL_INFO(VIDIOC_EXT_PREPARE_BUF, v4l_ext_prepare_buf, v4l_print_ext_buffer, INFO_FL_QUEUE),
> >>>>      IOCTL_INFO(VIDIOC_ENUM_DV_TIMINGS, v4l_stub_enum_dv_timings, v4l_print_enum_dv_timings, INFO_FL_CLEAR(v4l2_enum_dv_timings, pad)),
> >>>>      IOCTL_INFO(VIDIOC_QUERY_DV_TIMINGS, v4l_stub_query_dv_timings, v4l_print_dv_timings, INFO_FL_ALWAYS_COPY),
> >>>>      IOCTL_INFO(VIDIOC_DV_TIMINGS_CAP, v4l_stub_dv_timings_cap, v4l_print_dv_timings_cap, INFO_FL_CLEAR(v4l2_dv_timings_cap, pad)),
> >>>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
> >>>> index 8bbcb74d8ee31..75996657ad1ba 100644
> >>>> --- a/include/media/v4l2-ioctl.h
> >>>> +++ b/include/media/v4l2-ioctl.h
> >>>> @@ -169,16 +169,26 @@ struct v4l2_fh;
> >>>>   *  :ref:`VIDIOC_REQBUFS <vidioc_reqbufs>` ioctl
> >>>>   * @vidioc_querybuf: pointer to the function that implements
> >>>>   *  :ref:`VIDIOC_QUERYBUF <vidioc_querybuf>` ioctl
> >>>> + * @vidioc_ext_querybuf: pointer to the function that implements
> >>>> + *  :ref:`VIDIOC_EXT_QUERYBUF <vidioc_ext_querybuf>` ioctl
> >>>>   * @vidioc_qbuf: pointer to the function that implements
> >>>>   *  :ref:`VIDIOC_QBUF <vidioc_qbuf>` ioctl
> >>>> + * @vidioc_ext_qbuf: pointer to the function that implements
> >>>> + *  :ref:`VIDIOC_EXT_QBUF <vidioc_ext_qbuf>` ioctl
> >>>>   * @vidioc_expbuf: pointer to the function that implements
> >>>>   *  :ref:`VIDIOC_EXPBUF <vidioc_expbuf>` ioctl
> >>>>   * @vidioc_dqbuf: pointer to the function that implements
> >>>>   *  :ref:`VIDIOC_DQBUF <vidioc_qbuf>` ioctl
> >>>> + * @vidioc_ext_dqbuf: pointer to the function that implements
> >>>> + *  :ref:`VIDIOC_EXT_DQBUF <vidioc_ext_qbuf>` ioctl
> >>>>   * @vidioc_create_bufs: pointer to the function that implements
> >>>>   *  :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl
> >>>> + * @vidioc_ext_create_bufs: pointer to the function that implements
> >>>> + *  :ref:`VIDIOC_EXT_CREATE_BUFS <vidioc_ext_create_bufs>` ioctl
> >>>>   * @vidioc_prepare_buf: pointer to the function that implements
> >>>>   *  :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl
> >>>> + * @vidioc_ext_prepare_buf: pointer to the function that implements
> >>>> + *  :ref:`VIDIOC_EXT_PREPARE_BUF <vidioc_ext_prepare_buf>` ioctl
> >>>>   * @vidioc_overlay: pointer to the function that implements
> >>>>   *  :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl
> >>>>   * @vidioc_g_fbuf: pointer to the function that implements
> >>>> @@ -439,17 +449,27 @@ struct v4l2_ioctl_ops {
> >>>>                            struct v4l2_requestbuffers *b);
> >>>>      int (*vidioc_querybuf)(struct file *file, void *fh,
> >>>>                             struct v4l2_buffer *b);
> >>>> +    int (*vidioc_ext_querybuf)(struct file *file, void *fh,
> >>>> +                               struct v4l2_ext_buffer *b);
> >>>>      int (*vidioc_qbuf)(struct file *file, void *fh,
> >>>>                         struct v4l2_buffer *b);
> >>>> +    int (*vidioc_ext_qbuf)(struct file *file, void *fh,
> >>>> +                           struct v4l2_ext_buffer *b);
> >>>>      int (*vidioc_expbuf)(struct file *file, void *fh,
> >>>>                           struct v4l2_exportbuffer *e);
> >>>>      int (*vidioc_dqbuf)(struct file *file, void *fh,
> >>>>                          struct v4l2_buffer *b);
> >>>> +    int (*vidioc_ext_dqbuf)(struct file *file, void *fh,
> >>>> +                            struct v4l2_ext_buffer *b);
> >>>>
> >>>>      int (*vidioc_create_bufs)(struct file *file, void *fh,
> >>>>                                struct v4l2_create_buffers *b);
> >>>> +    int (*vidioc_ext_create_bufs)(struct file *file, void *fh,
> >>>> +                                  struct v4l2_ext_create_buffers *b);
> >>>>      int (*vidioc_prepare_buf)(struct file *file, void *fh,
> >>>>                                struct v4l2_buffer *b);
> >>>> +    int (*vidioc_ext_prepare_buf)(struct file *file, void *fh,
> >>>> +                                  struct v4l2_ext_buffer *b);
> >>>>
> >>>>      int (*vidioc_overlay)(struct file *file, void *fh, unsigned int i);
> >>>>      int (*vidioc_g_fbuf)(struct file *file, void *fh,
> >>>> @@ -758,6 +778,12 @@ int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> >>>>                                struct v4l2_format *f,
> >>>>                                bool mplane_cap, bool strict);
> >>>>
> >>>> +int v4l2_ext_buffer_to_buffer(const struct v4l2_ext_buffer *e,
> >>>> +                          struct v4l2_buffer *b,
> >>>> +                          bool mplane_cap);
> >>>> +int v4l2_buffer_to_ext_buffer(const struct v4l2_buffer *b,
> >>>> +                          struct v4l2_ext_buffer *e);
> >>>> +
> >>>>  /*
> >>>>   * The user space interpretation of the 'v4l2_event' differs
> >>>>   * based on the 'time_t' definition on 32-bit architectures, so
> >>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> >>>> index 7123c6a4d9569..334cafdd2be97 100644
> >>>> --- a/include/uapi/linux/videodev2.h
> >>>> +++ b/include/uapi/linux/videodev2.h
> >>>> @@ -996,6 +996,41 @@ struct v4l2_plane {
> >>>>      __u32                   reserved[11];
> >>>>  };
> >>>>
> >>>> +/**
> >>>> + * struct v4l2_ext_plane - extended plane buffer info
> >>>> + * @buffer_length:  size of the entire buffer in bytes, should fit
> >>>> + *                  @offset + @plane_length
> >>>
> >>> Do we actually need this buffer_length at all? We have 3 memory types:
> >>>
> >>> 1) MMAP - here vb2 already knows the buffer size, because it created it.
> >>>
> >>> 2) DMABUF - the DMA-buf kAPI provides the information about buffer size.
> >>>
> >>> 3) USERPTR - this might actually benefit from buffer_length, because there
> >>>    are additional alignmnent requirements for the user memory, e.g. the
> >>>    offset and size must be cacheline aligned.
> >>>
> >>> Arguably, 1) and 2) are the main usage scenarios, while the user space that
> >>> uses them would have to suffer from added complexity, because of the
> >>> legacy/niche case 3).
> >>>
> >>> Could we make this field valid only for USERPTR?
> >>
> >> I think so, make sense, I'll implement this for next version.
> >>
> >>>
> >>>> + * @plane_length:   size of the plane in bytes.
> >>>> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
> >>>> + *                  that should be passed to mmap() called on the video node.
> >>>> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> >>>> + *                  to this plane.
> >>>> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> >>>> + *                  associated with this plane.
> >>>> + * @offset:         offset in the memory buffer where the plane starts.
> >>>> + * @memory:         enum v4l2_memory; the method, in which the actual video
> >>>> + *                  data is passed
> >>>> + * @reserved:               extra space reserved for future fields, must be set to 0.
> >>>> + *
> >>>> + *
> >>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> >>>> + * can have one plane for Y, and another for interleaved CbCr components.
> >>>> + * Each plane can reside in a separate memory buffer, or even in
> >>>> + * a completely separate memory node (e.g. in embedded devices).
> >>>> + */
> >>>> +struct v4l2_ext_plane {
> >>>> +    __u32 buffer_length;
> >>>> +    __u32 plane_length;
> >>>> +    union {
> >>>> +            __u32 mem_offset;
> >>>> +            __u64 userptr;
> >>>> +            __s32 dmabuf_fd;
> >>>> +    } m;
> >>>> +    __u32 offset;
> >>>> +    __u32 memory;
> >>>> +    __u32 reserved[4];
> >>>> +};
> >>>
> >>> Don't we also need bytesused? Or would plane_length essentially mean the
> >>> amount of space or payload, depending on the usage context?
> >>
> >> In my understanding, plane_length is the max amount of data the plane can
> >> occupy, othersize it can overflow the buffer or mess with another plane
> >> that is in the same buffer on a different offset.
> >>
> >> I'm probably wrong, but I don't really see why the payload size is usefull,
> >> unless if we set a plane_legth that is much bigger then the data it can carry,
> >> that can impact performance.
> >> Payload can also be calculated from the format.
> >
> > It is required for non-2D-image formats, such as compressed bitstream.
> > In that case, the size of the image varies between frames and is
> > usually less than the size of the buffer.
>
> right, ok.
>
> >
> >>
> >> I can add it back if it is usefull. Please let me know your thoughts.
> >>
> >
> > As I mentioned above, it looks like plane_length is used almost
> > exactly the same way bytesused was in the original code, so maybe it
> > could just stay this way?
>
> Ack.
>
> >
> >>>
> >>> Similarly, the original data_offset was useful as a return field, which
> >>> some drivers use to indicate that the beginning of the plane is occupied by
> >>> some header or otherwise irrelevant data, which must be skipped. Would the
> >>> offset field be used for this purpose now?
> >>
> >> I didn't add an equivalent of the data_offset, since it seemed to be
> >> unused (please see my comments about this above).
> >>
> >>>
> >>>> +
> >>>>  /**
> >>>>   * struct v4l2_buffer - video buffer info
> >>>>   * @index:  id number of the buffer
> >>>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
> >>>>      };
> >>>>  };
> >>>>
> >>>> +/**
> >>>> + * struct v4l2_ext_buffer - extended video buffer info
> >>>> + * @index:  id number of the buffer
> >>>> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> >>>> + * @flags:  buffer informational flags
> >>>
> >>> nit: The order of comments doesn't match the order of fields in the struct.
> >>
> >> Ack.
> >>
> >>>
> >>>> + * @field:  enum v4l2_field; field order of the image in the buffer
> >>>> + * @timestamp:      frame timestamp
> >>>> + * @sequence:       sequence count of this frame
> >>>> + * @planes: per-plane buffer information
> >>>> + * @request_fd:     fd of the request that this buffer should use
> >>>> + * @reserved:       extra space reserved for future fields, must be set to 0
> >>>> + *
> >>>> + * Contains data exchanged by application and driver using one of the Streaming
> >>>> + * I/O methods.
> >>>> + */
> >>>> +struct v4l2_ext_buffer {
> >>>> +    __u32 index;
> >>>> +    __u32 type;
> >>>> +    __u32 field;
> >>>> +    __u32 sequence;
> >>>> +    __u64 flags;
> >>>> +    __u64 timestamp;
> >>>
> >>> What's the unit? How does this play with the other UAPI that the user space
> >>> may use, e.g. clock_gettime() which returns struct timespec?
> >>
> >> The unity is nsec:
> >>
> >>         e->timestamp = b->timestamp.tv_sec * NSEC_PER_SEC +
> >>                 b->timestamp.tv_usec * NSEC_PER_USEC;
> >>
> >> I can clarify in the docs, is this ok?
> >>
> >
> > Yes, it definitely needs to be documented. That said, what's the
> > rationale for switching from the timeval representation to the flat
> > nsec-based one?
>
> According to https://patchwork.kernel.org/project/linux-media/patch/20190319145243.25047-4-boris.brezillon@collabora.com/
> This avoids y2038 issue.

Fair enough, thanks.

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-14 10:46               ` Tomasz Figa
@ 2020-12-15 14:36                 ` Helen Koike
  2020-12-16  3:13                   ` Tomasz Figa
  0 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-12-15 14:36 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Hans Verkuil, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

Hi Tomasz,

On 12/14/20 7:46 AM, Tomasz Figa wrote:
> On Fri, Dec 4, 2020 at 4:52 AM Helen Koike <helen.koike@collabora.com> wrote:
>>
>> Hi,
>>
>> Please see my 2 points below (about v4l2_ext_buffer and another about timestamp).
>>
>> On 12/3/20 12:11 PM, Hans Verkuil wrote:
>>> On 23/11/2020 18:40, Helen Koike wrote:
>>>>
>>>>
>>>> On 11/23/20 12:46 PM, Tomasz Figa wrote:
>>>>> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>>
>>>>>> Hi Hans,
>>>>>>
>>>>>> Thank you for your review.
>>>>>>
>>>>>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
>>>>>>> Hi Helen,
>>>>>>>
>>>>>>> Again I'm just reviewing the uAPI.
>>>>>>>
>>>>>>> On 04/08/2020 21:29, Helen Koike wrote:
>>>>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>>>>
>>>>>>>> Those extended buffer ops have several purpose:
>>>>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>>>>>    the number of ns elapsed since 1970
>>>>>>>> 2/ Unify single/multiplanar handling
>>>>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>>>>>    to support the case where a single buffer object is storing all
>>>>>>>>    planes data, each one being placed at a different offset
>>>>>>>>
>>>>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>>>>>> these new objects.
>>>>>>>>
>>>>>>>> The core takes care of converting new ioctls requests to old ones
>>>>>>>> if the driver does not support the new hooks, and vice versa.
>>>>>>>>
>>>>>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>>>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>>>>>> use the Request API to collect more metadata information from the
>>>>>>>> frame.
>>>>>>>>
>>>>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>>>>> ---
>>>>>>>> Changes in v5:
>>>>>>>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
>>>>>>>> - return mem_offset to struct v4l2_ext_plane
>>>>>>>> - change sizes and reorder fields to avoid holes in the struct and make
>>>>>>>>   it the same for 32 and 64 bits
>>>>>>>>
>>>>>>>> Changes in v4:
>>>>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>>>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>>>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>>>>>> - Reformulate struct v4l2_ext_plane
>>>>>>>> - Fix some bugs caught by v4l2-compliance
>>>>>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>>>>>
>>>>>>>> Changes in v3:
>>>>>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>>>>>
>>>>>>>> Changes in v2:
>>>>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>>>>>   later on
>>>>>>>> ---
>>>>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>>>>>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>>>>>  include/uapi/linux/videodev2.h       |  90 +++++++
>>>>>>>>  4 files changed, 476 insertions(+), 22 deletions(-)
>>>>>>>>
>>>>>>>
>>>>>>> <snip>
>>>>>>>
>>>>>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>>>>>>> index 7123c6a4d9569..334cafdd2be97 100644
>>>>>>>> --- a/include/uapi/linux/videodev2.h
>>>>>>>> +++ b/include/uapi/linux/videodev2.h
>>>>>>>> @@ -996,6 +996,41 @@ struct v4l2_plane {
>>>>>>>>      __u32                   reserved[11];
>>>>>>>>  };
>>>>>>>>
>>>>>>>> +/**
>>>>>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>>>>>> + * @buffer_length:  size of the entire buffer in bytes, should fit
>>>>>>>> + *                  @offset + @plane_length
>>>>>>>> + * @plane_length:   size of the plane in bytes.
>>>>>>>> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
>>>>>>>> + *                  that should be passed to mmap() called on the video node.
>>>>>>>> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>>>>>> + *                  to this plane.
>>>>>>>> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>>>>>> + *                  associated with this plane.
>>>>>>>> + * @offset:         offset in the memory buffer where the plane starts.
>>>>>>>> + * @memory:         enum v4l2_memory; the method, in which the actual video
>>>>>>>> + *                  data is passed
>>>>>>>> + * @reserved:               extra space reserved for future fields, must be set to 0.
>>>>>>>> + *
>>>>>>>> + *
>>>>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>>>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>>>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>>>>>> + */
>>>>>>>> +struct v4l2_ext_plane {
>>>>>>>> +    __u32 buffer_length;
>>>>>>>> +    __u32 plane_length;
>>>>>>>> +    union {
>>>>>>>> +            __u32 mem_offset;
>>>>>>>> +            __u64 userptr;
>>>>>>>> +            __s32 dmabuf_fd;
>>>>>>>> +    } m;
>>>>>>>> +    __u32 offset;
>>>>>>>
>>>>>>> I'd rename this plane_offset. I think some reordering would make this struct easier
>>>>>>> to understand:
>>>>>>>
>>>>>>> struct v4l2_ext_plane {
>>>>>>>       __u32 buffer_length;
>>>>>>>       __u32 plane_offset;
>>>>>>>       __u32 plane_length;
>>>>>>>       __u32 memory;
>>>>>>>       union {
>>>>>>>               __u32 mem_offset;
>>>>>>>               __u64 userptr;
>>>>>>>               __s32 dmabuf_fd;
>>>>>>>       } m;
>>>>>>>       __u32 reserved[4];
>>>>>>> };
>>>>>>>
>>>>>>>> +    __u32 memory;
>>>>>>>> +    __u32 reserved[4];
>>>>>>>> +};
>>>>>>
>>>>>> Ok, I'll apply this to the next version.
>>>>>>
>>>>>>>
>>>>>>> What is not clear is how to tell the different between a single buffer containing
>>>>>>> multiple planes, and using a separate buffer per plane. E.g. what would this look
>>>>>>> like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
>>>>>>> V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
>>>>>>> planes are also combined in a single buffer?
>>>>>>>
>>>>>>> I would guess that the m union is set to 0 if the plane is part of the buffer
>>>>>>> defined in the previous plane?
>>>>>>
>>>>>> The difference would be if m are equal or differ between planes, example:
>>>>>>
>>>>>> For V4L2_PIX_FMT_YVU420:
>>>>>>
>>>>>>     Y:
>>>>>>         plane_offset = 0
>>>>>>         m.dmabuf_fd = 3
>>>>>>     Cb:
>>>>>>         plane_offset = 300
>>>>>>         m.dmabuf_fd = 3
>>>>>>     Cr:
>>>>>>         plane_offset = 375
>>>>>>         m.dmabuf_fd = 3
>>>>>>
>>>>>> For V4L2_PIX_FMT_YVU420M:
>>>>>>
>>>>>>     Y:
>>>>>>         plane_offset = 0
>>>>>>         m.dmabuf_fd = 4
>>>>>>     Cb:
>>>>>>         plane_offset = 0
>>>>>>         m.dmabuf_fd = 5
>>>>>>     Cr:
>>>>>>         plane_offset = 0
>>>>>>         m.dmabuf_fd = 6
>>>>>>
>>>>>>
>>>>>> Does it make sense?
>>>>>>
>>>>>
>>>>> Actually all the 3 file descriptors can still point to the same
>>>>> buffer, because they might have been dup()ed. The kernel needs to
>>>>> resolve the file descriptors into struct dma_buf and then check
>>>>> whether it's one or more buffers.
>>>>
>>>> Right, thanks for this.
>>>>
>>>>>
>>>>> In fact, dup()ed FD for each plane is quite a common case in other
>>>>> APIs, e.g. EGL, but current V4L2 API can't handle it. In Chromium we
>>>>> basically work around it by assuming that if we receive a buffer for a
>>>>> V4L2 device that only supports non-M formats, then we can safely
>>>>> ignore all but first FD. The new API gives the ability to handle the
>>>>> case properly, with full validation by the kernel.
>>>>>
>>>>>>>
>>>>>>>> +
>>>>>>>>  /**
>>>>>>>>   * struct v4l2_buffer - video buffer info
>>>>>>>>   * @index:  id number of the buffer
>>>>>>>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>>>>>>>>      };
>>>>>>>>  };
>>>>>>>>
>>>>>>>> +/**
>>>>>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>>>>>> + * @index:  id number of the buffer
>>>>>>>> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>>>>>> + * @flags:  buffer informational flags
>>>>>>>> + * @field:  enum v4l2_field; field order of the image in the buffer
>>>>>>>> + * @timestamp:      frame timestamp
>>>>>>>> + * @sequence:       sequence count of this frame
>>>>>>>> + * @planes: per-plane buffer information
>>>>>>>> + * @request_fd:     fd of the request that this buffer should use
>>>>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>>>>>> + *
>>>>>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>>>>>> + * I/O methods.
>>>>>>>> + */
>>>>>>>> +struct v4l2_ext_buffer {
>>>>>>>> +    __u32 index;
>>>>>>>> +    __u32 type;
>>>>>>>> +    __u32 field;
>>>>>>>> +    __u32 sequence;
>>>>>>>> +    __u64 flags;
>>>>>>>> +    __u64 timestamp;
>>>>>>>> +    struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>
>> I would like your opinion on the following:
>>
>> We have two concepts here
>> * memory buffers (that belongs to a frame buffer object aka v4l2_ext_buffer)
>> * color components/planes (that we need to indicate to userspace where the planes
>>   are located, which buffer and which offset inside the buffer).
>>
>> A v4l2_ext_buffer can be reused to a different format if it fits the image
>> (which is checked in QBUF time, by .buf_prepare() callback).
>>
>> Which means that, the information regarding where each color component is placed
>> just make sense after the buffer is queued.
>>
>> So if userspace calls EXT_QUERYBUF, and the buffer isn't queued, the color component
>> information doesn't make sense.
>>
>> One option is to fill in the plane information according to the current
>> configured format, but only the information returned from EXT_QBUF is guaranteed
>> to be the correct one.
>>
>> Another options is to split struct v4l2_ext_plane in two:
>>
>> struct v4l2_ext_membuffer {
>>         __u32 memory;
>>         union {
>>                 __u32 mmap_offset;
>>                 __u64 userptr;
>>                 __s32 dmabuf_fd;
>>         } m;
>>         _u32 buffer_length;
>> }
>>
>> struct v4l2_ext_plane {
>>         /*
>>          * memory buffer where this plane belongs, index is the position in of
>>          * membuffers[] in struct v4l2_ext_buffer below
>>          */
>>         unsigned int membuf_index;
>>         _u32 plane_length;
>>         _u32 plane_offset;
>> }
>>
>> Then we would have
>>
>> struct v4l2_ext_buffer {
>>         ... <snip>
>>         struct v4l2_ext_membuffer membuffers[VIDEO_MAX_PLANES];
>>         struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>         ... <snip>
>> }
>>
>> Where planes would only be filled by the core only if the buffer is
>> queued (i.e. "locked" to a given format).
>>
>> This also avoids having several planes with different dmabuf_fd that are dup()ed,
>> since we'll have an entry per memory buffer.
>> Which also avoids the following:
>> If we are working with a single membuf for all planes for instance, vb2 would need
>> to know how many planes (let's say there are 3) and repeat the mem buffer information
>> 3 times, and if userspace changes to a pixelformat with 2 color components, we would
>> repeat 2 times with the same information. And we wouldn't have this issue
>> if we split both information.
>>
>> I was also assuming that once the buffer is queued, userspace can't modify
>> the configured format (I need to check this, but make sense to me due to how
>> .buf_prepare() works).
>>
>> What do you think? Does it make sense?
>>
> 
> How about making querybuf have its own structure that describes only
> the buffer, as it is allocated? I.e.

Make sense to me to add a struct v4l2_ext_mmap_querybuf, so we don't need
to inform color information in this ioctl.

We still need a way to tell which memory buffer a color plane was placed.

I believe this must be in v4l2_ext_buffer, since userspace needs to know
on DQBUF time where to find each plane/color component.

So we need a way to enumerate memory buffers, and have planes pointing
to them in someway.

Just to be clear, the way I'm seeing this is:
A buffer object can be composed by one or more memory buffers (membufs).
Color planes can be placed randomly in any memory buffer, drivers (capture)
and userspace (output) will decide where these planes will be placed.

Or this doesn't make sense and we can assume that or all planes will be placed
in a single memory buffer, or each will be placed in a different memory buffer?
To avoid cases where we have 2 membufs and 3 planes?

Note: we probably should standardize terminology through the docs, structs
      and functions to differentiate color planes, memory buffer and buffer
      objects.

> 
> struct v4l2_ext_mmap_plane {

I would rename this to v4l2_ext_mmap_membuf to not confuse with color planes.

>         __u32 mmap_offset;
>         __u32 length;
> }
> 
> struct v4l2_ext_querybuf {

I would rename this to v4l2_ext_mmap_querybuf, since it only make
sense to mmap, and the ioctl can be called VIDIOC_EXT_QUERY_MMAP_BUF,
where it would return -EINVAL if the index is not a mmaped buffers.

>         /* ... */
>         struct v4l2_ext_mmap_plane planes[VIDEO_MAX_PLANES];

Then:
         struct v4l2_ext_mmap_membuf membufs[VIDEO_MAX_PLANES];

> };
> 
> Moreover, the ioctl would be only valid if the queue is operating in
> MMAP mode, because otherwise it doesn't provide any useful information
> - the userspace should already know what userptr or DMA-buf was
> associated with the buffer. In fact, returning the DMA-buf FD from
> QBUF time is confusing, because the userspace code might have already
> closed that FD (it can rely on other way of referencing the buffer).

so I guess we should zero those fields in QBUF.

> 
> WDYT?

Please check my proposal below and see what you think.


For DMA-buf and Userptr
-----------------------
userspace performs EXT_CREATE_BUF ioctl to reserve a buffer "index" range in
that mode, to be used in EXT_QBUF and EXT_DQBUF

On EXT_QBUF, userspace must tell the driver about existent memory buffers and the
corresponding DMA-fd or Userptr.

Capture: driver fills plane information, informing in which memory buffer each plane
	 was placed (Or should this be pre-determined by userspace?)

Output: userspace fills plane information, informing in which memory buffer each
	plane was placed (Or should this be pre-determined by the driver?)

For MMAP
-----------------------
userspace performs EXT_CREATE_BUF ioctl to reserve a buffer "index" range in
that mode, to be used in EXT_QBUF and EXT_DQBUF

Should the API allow userspace to select how many memory buffers it wants?
(maybe not)

userspace performs EXT_QUERY_MMAP_BUF to get the mmap offset/cookie and length
for each memory buffer.

On EXT_QBUF, userspace doesn't need to fill membuf information. Should the
mmap offset and length be filled by the kernel and returned to userspace here
as well? I'm leaning towards: no.

If the answer is no, then here is my proposal:
----------------------------------------------

/* If MMAP, drivers decide how many memory buffers to allocate */
int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_buffer *argp )

/* Returns -EINVAL if not MMAP */
int ioctl( int fd, VIDIOC_EXT_MMAP_QUERYBUF, struct v4l2_ext_mmap_querybuf *argp )

/* userspace fills v4l2_ext_buffer.membufs if DMA-fd or Userptr, leave it zero for MMAP
 * Should userspace also fill v4l2_ext_buffer.planes?
 */
int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )

/* v4l2_ext_buffer.membufs is set to zero by the driver */
int ioctl( int fd, VIDIOC_EXT_DBUF, struct v4l2_ext_buffer *argp )

(I omitted reserved fields below)

struct v4l2_ext_create_buffers {
	__u32				index;
	__u32				count;
	__u32				memory;
	__u32				capabilities;
	struct v4l2_ext_pix_format	format;
};

struct v4l2_ext_mmap_membuf {
	__u32 offset;
	__u32 length;
}

struct v4l2_ext_mmap_querybuf {
	__u32 index;
	struct v4l2_ext_mmap_membuf membufs[VIDEO_MAX_PLANES];
}

struct v4l2_ext_membuf {
	__u32 memory;
	union {
		__u64 userptr;
		__s32 dmabuf_fd;
	} m;
	// Can't we just remove the union and "memory" field, and the non-zero
	// is the one we should use?
};

struct v4l2_ext_plane {
	__u32 membuf_index;
	__u32 offset;
	__u32 bytesused;
};

struct v4l2_ext_buffer {
	__u32 index;
	__u32 type;
	__u32 field;
	__u32 sequence;
	__u64 flags;
	__u64 timestamp;
	struct v4l2_ext_membuf membufs[VIDEO_MAX_PLANES];
	struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
	__s32 request_fd;
};


Thanks,
Helen

> 
> Best regards.
> Tomasz
> 
>>
>>>>>>>> +    __s32 request_fd;
>>>>>>>> +    __u32 reserved[9];
>>>>>>>> +};
>>>>>>>
>>>>>>> Brainstorming:
>>>>>>>
>>>>>>> Some ideas I have to make it easier to support mid stream resolution/colorimetry
>>>>>>> changes:
>>>>>>>
>>>>>>> Adding width and height would support resolution changes (requires the use of
>>>>>>> CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
>>>>>>> information is provided here, then there are no race conditions.
>>>>>>>
>>>>>>> Same for adding the colorimetry fields here, this too can change on the fly (esp.
>>>>>>> with HDMI), so reporting this information here avoids race conditions as well.
>>>>>>
>>>>>> Right, do you think this is something we can discuss later in a different RFC?
>>>>>> So we can have a better view on how dynamic resolution change would be used?
>>>>>>
>>>>>> We can add more reserved fields or maybe try to do something to what has been
>>>>>> discussed in about extensible system calls [1]
>>>>>>
>>>>>> [1] https://lwn.net/Articles/830666/
>>>>>>
>>>>>>>
>>>>>>> And thirdly, I would like to have a __u64 boot_timestamp field containing the
>>>>>>> CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
>>>>>>> is that for m2m devices it is just copied and that for other devices it can have
>>>>>>> different meanings depending on the timestamp buffer flags.
>>>>>>>
>>>>>>> There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
>>>>>>> to add support for this. That way you know exactly when the driver was finished with
>>>>>>> the buffer and that helps in detecting missed frames or instrumentation.
>>>>>>
>>>>>> I don't mind adding it. Does it make sense to have both timestamp and boot_timestamp?
>>>>>>
>>>>>
>>>>> I think this is quite independent from the ext API work. AFAIR there
>>>>> was an RFC to request the timestamp source from the userspace by the
>>>>> flags field in QBUF, which would work with the existing API as well,
>>>>> or it wasn't posted in the end?
>>>
>>> It's not about selecting a specific clock source. I think that option 4 as described
>>> below would work for that.
>>>
>>> This problem I'm describing here is specific to m2m devices where the timestamp is
>>> either just passed through untouched, or it is used as an identifier for a buffer
>>> for use with stateless decoders.
>>>
>>> In both cases you cannot use the timestamp as a proper timestamp that tells you when
>>> the buffer was marked done by the driver. So this is about adding a second timestamp
>>> field (timestamp_done or something like that). Whether this would be hardcoded as using
>>> CLOCK_BOOTTIME or uses the same clock source as selected through a control is something
>>> that can be discussed, but since it does require a new field I believe this is part of
>>> this proposal.
>>
>> I'm probably lacking m2m knowledge here.
>> timestamp_done would be when the driver set the buffer as done, what about the other timestamp?
>> I would like to rename it to make it clear what it means, maybe image_timestamp?
>> Where, in a capture device, image_timestamp would be the timestamp from the hardware when
>> it captured that specific frame, and in a decoder, it would be the timestamp of that frame
>> when it got encoded?
>> Is this correct?
>>
>> Thanks
>> Helen
>>
>>>
>>> Regards,
>>>
>>>       Hans
>>>
>>>>
>>>> I was recalling the discussions we had regarding this:
>>>>
>>>> 1.
>>>>     This first attempt in the uvc driver is to use a specific kernel parameter for that case:
>>>>     https://patchwork.kernel.org/patch/10644887/
>>>>     The conclusion that the support should be in the core API and not driver specific.
>>>>
>>>> 2.
>>>>     Then an attempt to add global v4l2 support was sent with the Mediatek patch series:
>>>>     https://patchwork.linuxtv.org/patch/60878/
>>>>     The major problem is that clock type should be something selectable by userspace, and
>>>>     not pre-defined by the driver.
>>>>
>>>> 3.
>>>>     Another idea was to use the 'flags' field in the structs v4l2_requestbuffers and
>>>>     v4l2_create_buffers.
>>>>     But this field was removed in
>>>>     129134e5415d ("media: media/v4l2: remove V4L2_FLAG_MEMORY_NON_CONSISTENT flag")
>>>>     The major concern with this approach was with the uAPI, since it doesn't make much
>>>>     sense to select a clock when creating buffers.
>>>>
>>>> 4.
>>>>     Another suggestion by Nicolas Dufresne was to add this as a menu control so that userspace
>>>>     can choose the clock for the timestamps from a given list, the enum in the list can also match
>>>>     the clocks ids.
>>>>     We would need to add a new buf flag in struct v4l2_buffer, like V4L2_BUF_FLAG_TIMESTAMP_OTHER,
>>>>     which would be "as specified through controls ...."
>>>>
>>>>
>>>> So my current question is, should we have both __u32 timestamp and __u32 boottimestamp?
>>>> Or should we have a mechanism that allows switching from one to the other and use
>>>> a single field? And if this mechanism should be implemented in both APIs? Can this be
>>>> defined later?
>>>>
>>>>
>>>> Please, let me know your thoughts.
>>>>
>>>> Thanks,
>>>> Helen
>>>>
>>>>>
>>>>>>>
>>>>>>>> +
>>>>>>>>  #ifndef __KERNEL__
>>>>>>>>  /**
>>>>>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>>>>>>>> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
>>>>>>>>      __u32                   reserved[6];
>>>>>>>>  };
>>>>>>>>
>>>>>>>> +/**
>>>>>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>>>>>>>> + * @index:  on return, index of the first created buffer
>>>>>>>> + * @count:  entry: number of requested buffers,
>>>>>>>> + *          return: number of created buffers
>>>>>>>> + * @memory: enum v4l2_memory; buffer memory type
>>>>>>>> + * @capabilities: capabilities of this buffer type.
>>>>>>>> + * @format: frame format, for which buffers are requested
>>>>>>>> + * @flags:  additional buffer management attributes (ignored unless the
>>>>>>>> + *          queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>>>>>>>> + *          and configured for MMAP streaming I/O).
>>>>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>>>>>> + */
>>>>>>>> +struct v4l2_ext_create_buffers {
>>>>>>>> +    __u32                           index;
>>>>>>>> +    __u32                           count;
>>>>>>>> +    __u32                           memory;
>>>>>>>> +    __u32                           capabilities;
>>>>>>>> +    struct v4l2_ext_pix_format      format;
>>>>>>>
>>>>>>> The reality is that the only field that is ever used in the original v4l2_format
>>>>>>> struct is sizeimage. So this can be replaced with:
>>>>>>>
>>>>>>>       __u32                           plane_size[VIDEO_MAX_PLANES];
>>>>>>>
>>>>>>> (the field name I picked is debatable, but you get the idea)
>>>>>>>
>>>>>>> The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
>>>>>>> is needed for the current format. The original idea of using struct v4l2_format
>>>>>>> was that drivers would use the full format information to calculate the
>>>>>>> memory size, but that was just much too complicated to implement and nobody
>>>>>>> ever used that. Only the sizeimage field was ever used.
>>>>>>
>>>>>> Right, I'll update this in next version, This should simplify things.
>>>>>>
>>>>>
>>>>> I think this might need a bit more discussion. How would the userspace
>>>>> know what size is enough for the desired resolution? The hardware
>>>>> and/or drivers often have various alignment/padding restrictions,
>>>>> which might not be easy to guess for the userspace.
>>>>>
>>>>> Also I don't quite understand what's so complicated in handling the
>>>>> full format, or at least the most important parts of it. The
>>>>> implementation of TRY_FMT/S_FMT, which exists in every driver, should
>>>>> already be able to calculate the right plane sizes.
>>>>>
>>>>> Best regards,
>>>>> Tomasz
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Helen
>>>>>>
>>>>>>>
>>>>>>>> +    __u32                           flags;
>>>>>>>> +    __u32 reserved[5];
>>>>>>>> +};
>>>>>>>> +
>>>>>>>>  /*
>>>>>>>>   *  I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>>>>>>>   *
>>>>>>>> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
>>>>>>>>  #define VIDIOC_G_EXT_PIX_FMT        _IOWR('V', 104, struct v4l2_ext_pix_format)
>>>>>>>>  #define VIDIOC_S_EXT_PIX_FMT        _IOWR('V', 105, struct v4l2_ext_pix_format)
>>>>>>>>  #define VIDIOC_TRY_EXT_PIX_FMT      _IOWR('V', 106, struct v4l2_ext_pix_format)
>>>>>>>> +#define VIDIOC_EXT_CREATE_BUFS      _IOWR('V', 107, struct v4l2_ext_create_buffers)
>>>>>>>> +#define VIDIOC_EXT_QUERYBUF _IOWR('V', 108, struct v4l2_ext_buffer)
>>>>>>>> +#define VIDIOC_EXT_QBUF             _IOWR('V', 109, struct v4l2_ext_buffer)
>>>>>>>> +#define VIDIOC_EXT_DQBUF    _IOWR('V', 110, struct v4l2_ext_buffer)
>>>>>>>> +#define VIDIOC_EXT_PREPARE_BUF      _IOWR('V', 111, struct v4l2_ext_buffer)
>>>>>>>>
>>>>>>>>  /* Reminder: when adding new ioctls please add support for them to
>>>>>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>       Hans
>>>>>>>
>>>
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-15 14:36                 ` Helen Koike
@ 2020-12-16  3:13                   ` Tomasz Figa
  2020-12-17 13:19                     ` Helen Koike
  0 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-12-16  3:13 UTC (permalink / raw)
  To: Helen Koike
  Cc: Hans Verkuil, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

On Tue, Dec 15, 2020 at 11:37 PM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi Tomasz,
>
> On 12/14/20 7:46 AM, Tomasz Figa wrote:
> > On Fri, Dec 4, 2020 at 4:52 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>
> >> Hi,
> >>
> >> Please see my 2 points below (about v4l2_ext_buffer and another about timestamp).
> >>
> >> On 12/3/20 12:11 PM, Hans Verkuil wrote:
> >>> On 23/11/2020 18:40, Helen Koike wrote:
> >>>>
> >>>>
> >>>> On 11/23/20 12:46 PM, Tomasz Figa wrote:
> >>>>> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>>>
> >>>>>> Hi Hans,
> >>>>>>
> >>>>>> Thank you for your review.
> >>>>>>
> >>>>>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
> >>>>>>> Hi Helen,
> >>>>>>>
> >>>>>>> Again I'm just reviewing the uAPI.
> >>>>>>>
> >>>>>>> On 04/08/2020 21:29, Helen Koike wrote:
> >>>>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>>>>>
> >>>>>>>> Those extended buffer ops have several purpose:
> >>>>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
> >>>>>>>>    the number of ns elapsed since 1970
> >>>>>>>> 2/ Unify single/multiplanar handling
> >>>>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
> >>>>>>>>    to support the case where a single buffer object is storing all
> >>>>>>>>    planes data, each one being placed at a different offset
> >>>>>>>>
> >>>>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
> >>>>>>>> these new objects.
> >>>>>>>>
> >>>>>>>> The core takes care of converting new ioctls requests to old ones
> >>>>>>>> if the driver does not support the new hooks, and vice versa.
> >>>>>>>>
> >>>>>>>> Note that the timecode field is gone, since there doesn't seem to be
> >>>>>>>> in-kernel users. We can be added back in the reserved area if needed or
> >>>>>>>> use the Request API to collect more metadata information from the
> >>>>>>>> frame.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> >>>>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
> >>>>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>>>>>>> ---
> >>>>>>>> Changes in v5:
> >>>>>>>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
> >>>>>>>> - return mem_offset to struct v4l2_ext_plane
> >>>>>>>> - change sizes and reorder fields to avoid holes in the struct and make
> >>>>>>>>   it the same for 32 and 64 bits
> >>>>>>>>
> >>>>>>>> Changes in v4:
> >>>>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> >>>>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> >>>>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
> >>>>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
> >>>>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
> >>>>>>>> - Remove num_planes field from struct v4l2_ext_buffer
> >>>>>>>> - Add flags field to struct v4l2_ext_create_buffers
> >>>>>>>> - Reformulate struct v4l2_ext_plane
> >>>>>>>> - Fix some bugs caught by v4l2-compliance
> >>>>>>>> - Rebased on top of media/master (post 5.8-rc1)
> >>>>>>>>
> >>>>>>>> Changes in v3:
> >>>>>>>> - Rebased on top of media/master (post 5.4-rc1)
> >>>>>>>>
> >>>>>>>> Changes in v2:
> >>>>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
> >>>>>>>>   later on
> >>>>>>>> ---
> >>>>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
> >>>>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
> >>>>>>>>  include/media/v4l2-ioctl.h           |  26 ++
> >>>>>>>>  include/uapi/linux/videodev2.h       |  90 +++++++
> >>>>>>>>  4 files changed, 476 insertions(+), 22 deletions(-)
> >>>>>>>>
> >>>>>>>
> >>>>>>> <snip>
> >>>>>>>
> >>>>>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> >>>>>>>> index 7123c6a4d9569..334cafdd2be97 100644
> >>>>>>>> --- a/include/uapi/linux/videodev2.h
> >>>>>>>> +++ b/include/uapi/linux/videodev2.h
> >>>>>>>> @@ -996,6 +996,41 @@ struct v4l2_plane {
> >>>>>>>>      __u32                   reserved[11];
> >>>>>>>>  };
> >>>>>>>>
> >>>>>>>> +/**
> >>>>>>>> + * struct v4l2_ext_plane - extended plane buffer info
> >>>>>>>> + * @buffer_length:  size of the entire buffer in bytes, should fit
> >>>>>>>> + *                  @offset + @plane_length
> >>>>>>>> + * @plane_length:   size of the plane in bytes.
> >>>>>>>> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
> >>>>>>>> + *                  that should be passed to mmap() called on the video node.
> >>>>>>>> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
> >>>>>>>> + *                  to this plane.
> >>>>>>>> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
> >>>>>>>> + *                  associated with this plane.
> >>>>>>>> + * @offset:         offset in the memory buffer where the plane starts.
> >>>>>>>> + * @memory:         enum v4l2_memory; the method, in which the actual video
> >>>>>>>> + *                  data is passed
> >>>>>>>> + * @reserved:               extra space reserved for future fields, must be set to 0.
> >>>>>>>> + *
> >>>>>>>> + *
> >>>>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
> >>>>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
> >>>>>>>> + * Each plane can reside in a separate memory buffer, or even in
> >>>>>>>> + * a completely separate memory node (e.g. in embedded devices).
> >>>>>>>> + */
> >>>>>>>> +struct v4l2_ext_plane {
> >>>>>>>> +    __u32 buffer_length;
> >>>>>>>> +    __u32 plane_length;
> >>>>>>>> +    union {
> >>>>>>>> +            __u32 mem_offset;
> >>>>>>>> +            __u64 userptr;
> >>>>>>>> +            __s32 dmabuf_fd;
> >>>>>>>> +    } m;
> >>>>>>>> +    __u32 offset;
> >>>>>>>
> >>>>>>> I'd rename this plane_offset. I think some reordering would make this struct easier
> >>>>>>> to understand:
> >>>>>>>
> >>>>>>> struct v4l2_ext_plane {
> >>>>>>>       __u32 buffer_length;
> >>>>>>>       __u32 plane_offset;
> >>>>>>>       __u32 plane_length;
> >>>>>>>       __u32 memory;
> >>>>>>>       union {
> >>>>>>>               __u32 mem_offset;
> >>>>>>>               __u64 userptr;
> >>>>>>>               __s32 dmabuf_fd;
> >>>>>>>       } m;
> >>>>>>>       __u32 reserved[4];
> >>>>>>> };
> >>>>>>>
> >>>>>>>> +    __u32 memory;
> >>>>>>>> +    __u32 reserved[4];
> >>>>>>>> +};
> >>>>>>
> >>>>>> Ok, I'll apply this to the next version.
> >>>>>>
> >>>>>>>
> >>>>>>> What is not clear is how to tell the different between a single buffer containing
> >>>>>>> multiple planes, and using a separate buffer per plane. E.g. what would this look
> >>>>>>> like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
> >>>>>>> V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
> >>>>>>> planes are also combined in a single buffer?
> >>>>>>>
> >>>>>>> I would guess that the m union is set to 0 if the plane is part of the buffer
> >>>>>>> defined in the previous plane?
> >>>>>>
> >>>>>> The difference would be if m are equal or differ between planes, example:
> >>>>>>
> >>>>>> For V4L2_PIX_FMT_YVU420:
> >>>>>>
> >>>>>>     Y:
> >>>>>>         plane_offset = 0
> >>>>>>         m.dmabuf_fd = 3
> >>>>>>     Cb:
> >>>>>>         plane_offset = 300
> >>>>>>         m.dmabuf_fd = 3
> >>>>>>     Cr:
> >>>>>>         plane_offset = 375
> >>>>>>         m.dmabuf_fd = 3
> >>>>>>
> >>>>>> For V4L2_PIX_FMT_YVU420M:
> >>>>>>
> >>>>>>     Y:
> >>>>>>         plane_offset = 0
> >>>>>>         m.dmabuf_fd = 4
> >>>>>>     Cb:
> >>>>>>         plane_offset = 0
> >>>>>>         m.dmabuf_fd = 5
> >>>>>>     Cr:
> >>>>>>         plane_offset = 0
> >>>>>>         m.dmabuf_fd = 6
> >>>>>>
> >>>>>>
> >>>>>> Does it make sense?
> >>>>>>
> >>>>>
> >>>>> Actually all the 3 file descriptors can still point to the same
> >>>>> buffer, because they might have been dup()ed. The kernel needs to
> >>>>> resolve the file descriptors into struct dma_buf and then check
> >>>>> whether it's one or more buffers.
> >>>>
> >>>> Right, thanks for this.
> >>>>
> >>>>>
> >>>>> In fact, dup()ed FD for each plane is quite a common case in other
> >>>>> APIs, e.g. EGL, but current V4L2 API can't handle it. In Chromium we
> >>>>> basically work around it by assuming that if we receive a buffer for a
> >>>>> V4L2 device that only supports non-M formats, then we can safely
> >>>>> ignore all but first FD. The new API gives the ability to handle the
> >>>>> case properly, with full validation by the kernel.
> >>>>>
> >>>>>>>
> >>>>>>>> +
> >>>>>>>>  /**
> >>>>>>>>   * struct v4l2_buffer - video buffer info
> >>>>>>>>   * @index:  id number of the buffer
> >>>>>>>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
> >>>>>>>>      };
> >>>>>>>>  };
> >>>>>>>>
> >>>>>>>> +/**
> >>>>>>>> + * struct v4l2_ext_buffer - extended video buffer info
> >>>>>>>> + * @index:  id number of the buffer
> >>>>>>>> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
> >>>>>>>> + * @flags:  buffer informational flags
> >>>>>>>> + * @field:  enum v4l2_field; field order of the image in the buffer
> >>>>>>>> + * @timestamp:      frame timestamp
> >>>>>>>> + * @sequence:       sequence count of this frame
> >>>>>>>> + * @planes: per-plane buffer information
> >>>>>>>> + * @request_fd:     fd of the request that this buffer should use
> >>>>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
> >>>>>>>> + *
> >>>>>>>> + * Contains data exchanged by application and driver using one of the Streaming
> >>>>>>>> + * I/O methods.
> >>>>>>>> + */
> >>>>>>>> +struct v4l2_ext_buffer {
> >>>>>>>> +    __u32 index;
> >>>>>>>> +    __u32 type;
> >>>>>>>> +    __u32 field;
> >>>>>>>> +    __u32 sequence;
> >>>>>>>> +    __u64 flags;
> >>>>>>>> +    __u64 timestamp;
> >>>>>>>> +    struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> >>
> >> I would like your opinion on the following:
> >>
> >> We have two concepts here
> >> * memory buffers (that belongs to a frame buffer object aka v4l2_ext_buffer)
> >> * color components/planes (that we need to indicate to userspace where the planes
> >>   are located, which buffer and which offset inside the buffer).
> >>
> >> A v4l2_ext_buffer can be reused to a different format if it fits the image
> >> (which is checked in QBUF time, by .buf_prepare() callback).
> >>
> >> Which means that, the information regarding where each color component is placed
> >> just make sense after the buffer is queued.
> >>
> >> So if userspace calls EXT_QUERYBUF, and the buffer isn't queued, the color component
> >> information doesn't make sense.
> >>
> >> One option is to fill in the plane information according to the current
> >> configured format, but only the information returned from EXT_QBUF is guaranteed
> >> to be the correct one.
> >>
> >> Another options is to split struct v4l2_ext_plane in two:
> >>
> >> struct v4l2_ext_membuffer {
> >>         __u32 memory;
> >>         union {
> >>                 __u32 mmap_offset;
> >>                 __u64 userptr;
> >>                 __s32 dmabuf_fd;
> >>         } m;
> >>         _u32 buffer_length;
> >> }
> >>
> >> struct v4l2_ext_plane {
> >>         /*
> >>          * memory buffer where this plane belongs, index is the position in of
> >>          * membuffers[] in struct v4l2_ext_buffer below
> >>          */
> >>         unsigned int membuf_index;
> >>         _u32 plane_length;
> >>         _u32 plane_offset;
> >> }
> >>
> >> Then we would have
> >>
> >> struct v4l2_ext_buffer {
> >>         ... <snip>
> >>         struct v4l2_ext_membuffer membuffers[VIDEO_MAX_PLANES];
> >>         struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> >>         ... <snip>
> >> }
> >>
> >> Where planes would only be filled by the core only if the buffer is
> >> queued (i.e. "locked" to a given format).
> >>
> >> This also avoids having several planes with different dmabuf_fd that are dup()ed,
> >> since we'll have an entry per memory buffer.
> >> Which also avoids the following:
> >> If we are working with a single membuf for all planes for instance, vb2 would need
> >> to know how many planes (let's say there are 3) and repeat the mem buffer information
> >> 3 times, and if userspace changes to a pixelformat with 2 color components, we would
> >> repeat 2 times with the same information. And we wouldn't have this issue
> >> if we split both information.
> >>
> >> I was also assuming that once the buffer is queued, userspace can't modify
> >> the configured format (I need to check this, but make sense to me due to how
> >> .buf_prepare() works).
> >>
> >> What do you think? Does it make sense?
> >>
> >
> > How about making querybuf have its own structure that describes only
> > the buffer, as it is allocated? I.e.
>
> Make sense to me to add a struct v4l2_ext_mmap_querybuf, so we don't need
> to inform color information in this ioctl.
>
> We still need a way to tell which memory buffer a color plane was placed.
>
> I believe this must be in v4l2_ext_buffer, since userspace needs to know
> on DQBUF time where to find each plane/color component.
>
> So we need a way to enumerate memory buffers, and have planes pointing
> to them in someway.
>
> Just to be clear, the way I'm seeing this is:
> A buffer object can be composed by one or more memory buffers (membufs).
> Color planes can be placed randomly in any memory buffer, drivers (capture)
> and userspace (output) will decide where these planes will be placed.
>
> Or this doesn't make sense and we can assume that or all planes will be placed
> in a single memory buffer, or each will be placed in a different memory buffer?
> To avoid cases where we have 2 membufs and 3 planes?
>

We have two cases here:

1) MMAP - the placement of color planes is controlled by the kernel.
In practice, we've ever had only 2 cases:
    a) for non-M formats all the color planes go to the same memory buffer,
    b) for M formats, each color plane has its own memory buffer.

2) DMA-buf and userptr - the placement is fully controlled by the application.

Would it make sense to just define the behavior of MMAP buffers as per
the two cases, 1a and 1b, above and get rid of memory buffer
information from DQBUF?

> Note: we probably should standardize terminology through the docs, structs
>       and functions to differentiate color planes, memory buffer and buffer
>       objects.

Note that the existing terminology is: color planes, memory planes and
buffers. I think it might be desirable to preserve it to avoid further
confusion with all the old and new terms mixing with each other.

>
> >
> > struct v4l2_ext_mmap_plane {
>
> I would rename this to v4l2_ext_mmap_membuf to not confuse with color planes.
>
> >         __u32 mmap_offset;
> >         __u32 length;
> > }
> >
> > struct v4l2_ext_querybuf {
>
> I would rename this to v4l2_ext_mmap_querybuf, since it only make
> sense to mmap, and the ioctl can be called VIDIOC_EXT_QUERY_MMAP_BUF,
> where it would return -EINVAL if the index is not a mmaped buffers.
>
> >         /* ... */
> >         struct v4l2_ext_mmap_plane planes[VIDEO_MAX_PLANES];
>
> Then:
>          struct v4l2_ext_mmap_membuf membufs[VIDEO_MAX_PLANES];
>
> > };
> >
> > Moreover, the ioctl would be only valid if the queue is operating in
> > MMAP mode, because otherwise it doesn't provide any useful information
> > - the userspace should already know what userptr or DMA-buf was
> > associated with the buffer. In fact, returning the DMA-buf FD from
> > QBUF time is confusing, because the userspace code might have already
> > closed that FD (it can rely on other way of referencing the buffer).
>
> so I guess we should zero those fields in QBUF.
>

Do you mean DQBUF?

> >
> > WDYT?
>
> Please check my proposal below and see what you think.
>
>
> For DMA-buf and Userptr
> -----------------------
> userspace performs EXT_CREATE_BUF ioctl to reserve a buffer "index" range in
> that mode, to be used in EXT_QBUF and EXT_DQBUF
>
> On EXT_QBUF, userspace must tell the driver about existent memory buffers and the
> corresponding DMA-fd or Userptr.
>
> Capture: driver fills plane information, informing in which memory buffer each plane
>          was placed (Or should this be pre-determined by userspace?)

This has to be fully controlled by the userspace, since it's the
userspace that allocates the buffers and has its own expectations
about their layout.

>
> Output: userspace fills plane information, informing in which memory buffer each
>         plane was placed (Or should this be pre-determined by the driver?)
>
> For MMAP
> -----------------------
> userspace performs EXT_CREATE_BUF ioctl to reserve a buffer "index" range in
> that mode, to be used in EXT_QBUF and EXT_DQBUF
>
> Should the API allow userspace to select how many memory buffers it wants?
> (maybe not)

I think it does allow that - it accepts the v4l2_ext_format struct.

>
> userspace performs EXT_QUERY_MMAP_BUF to get the mmap offset/cookie and length
> for each memory buffer.
>
> On EXT_QBUF, userspace doesn't need to fill membuf information. Should the
> mmap offset and length be filled by the kernel and returned to userspace here
> as well? I'm leaning towards: no.

Yeah, based on my comment above, I think the answer should be no.

>
> If the answer is no, then here is my proposal:
> ----------------------------------------------
>
> /* If MMAP, drivers decide how many memory buffers to allocate */
> int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_buffer *argp )
>
> /* Returns -EINVAL if not MMAP */
> int ioctl( int fd, VIDIOC_EXT_MMAP_QUERYBUF, struct v4l2_ext_mmap_querybuf *argp )
>
> /* userspace fills v4l2_ext_buffer.membufs if DMA-fd or Userptr, leave it zero for MMAP
>  * Should userspace also fill v4l2_ext_buffer.planes?
>  */
> int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
>
> /* v4l2_ext_buffer.membufs is set to zero by the driver */
> int ioctl( int fd, VIDIOC_EXT_DBUF, struct v4l2_ext_buffer *argp )
>
> (I omitted reserved fields below)
>
> struct v4l2_ext_create_buffers {
>         __u32                           index;
>         __u32                           count;
>         __u32                           memory;
>         __u32                           capabilities;
>         struct v4l2_ext_pix_format      format;
> };
>
> struct v4l2_ext_mmap_membuf {
>         __u32 offset;
>         __u32 length;
> }
>
> struct v4l2_ext_mmap_querybuf {
>         __u32 index;
>         struct v4l2_ext_mmap_membuf membufs[VIDEO_MAX_PLANES];
> }
>
> struct v4l2_ext_membuf {
>         __u32 memory;
>         union {
>                 __u64 userptr;
>                 __s32 dmabuf_fd;
>         } m;
>         // Can't we just remove the union and "memory" field, and the non-zero
>         // is the one we should use?

I think that would lead to an equivalent result in this case. That
said, I'm not sure if there would be any significant enough benefit to
justify moving away from the current convention. Having the memory
field might also make the structure a bit less error prone, e.g.
resilient to missing memset().

> };
>
> struct v4l2_ext_plane {
>         __u32 membuf_index;
>         __u32 offset;
>         __u32 bytesused;
> };
>
> struct v4l2_ext_buffer {
>         __u32 index;
>         __u32 type;
>         __u32 field;
>         __u32 sequence;
>         __u64 flags;
>         __u64 timestamp;
>         struct v4l2_ext_membuf membufs[VIDEO_MAX_PLANES];
>         struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];

Do we actually need this split into membufs and planes here? After
all, all we want to pass to the kernel here is in what buffer the
plane is in.

struct v4l2_ext_plane {
        __u32 memory;
        union {
                __u32 membuf_index;
                __u64 userptr;
                __s32 dmabuf_fd;
        } m;
        __u32 offset;
        __u32 bytesused;
};

Also, we might not even need membuf_index, if we strictly define the
color plane placement for MMAP as per my comment above.

Best regards,
Tomasz

>         __s32 request_fd;
> };
>
>
> Thanks,
> Helen
>
> >
> > Best regards.
> > Tomasz
> >
> >>
> >>>>>>>> +    __s32 request_fd;
> >>>>>>>> +    __u32 reserved[9];
> >>>>>>>> +};
> >>>>>>>
> >>>>>>> Brainstorming:
> >>>>>>>
> >>>>>>> Some ideas I have to make it easier to support mid stream resolution/colorimetry
> >>>>>>> changes:
> >>>>>>>
> >>>>>>> Adding width and height would support resolution changes (requires the use of
> >>>>>>> CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
> >>>>>>> information is provided here, then there are no race conditions.
> >>>>>>>
> >>>>>>> Same for adding the colorimetry fields here, this too can change on the fly (esp.
> >>>>>>> with HDMI), so reporting this information here avoids race conditions as well.
> >>>>>>
> >>>>>> Right, do you think this is something we can discuss later in a different RFC?
> >>>>>> So we can have a better view on how dynamic resolution change would be used?
> >>>>>>
> >>>>>> We can add more reserved fields or maybe try to do something to what has been
> >>>>>> discussed in about extensible system calls [1]
> >>>>>>
> >>>>>> [1] https://lwn.net/Articles/830666/
> >>>>>>
> >>>>>>>
> >>>>>>> And thirdly, I would like to have a __u64 boot_timestamp field containing the
> >>>>>>> CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
> >>>>>>> is that for m2m devices it is just copied and that for other devices it can have
> >>>>>>> different meanings depending on the timestamp buffer flags.
> >>>>>>>
> >>>>>>> There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
> >>>>>>> to add support for this. That way you know exactly when the driver was finished with
> >>>>>>> the buffer and that helps in detecting missed frames or instrumentation.
> >>>>>>
> >>>>>> I don't mind adding it. Does it make sense to have both timestamp and boot_timestamp?
> >>>>>>
> >>>>>
> >>>>> I think this is quite independent from the ext API work. AFAIR there
> >>>>> was an RFC to request the timestamp source from the userspace by the
> >>>>> flags field in QBUF, which would work with the existing API as well,
> >>>>> or it wasn't posted in the end?
> >>>
> >>> It's not about selecting a specific clock source. I think that option 4 as described
> >>> below would work for that.
> >>>
> >>> This problem I'm describing here is specific to m2m devices where the timestamp is
> >>> either just passed through untouched, or it is used as an identifier for a buffer
> >>> for use with stateless decoders.
> >>>
> >>> In both cases you cannot use the timestamp as a proper timestamp that tells you when
> >>> the buffer was marked done by the driver. So this is about adding a second timestamp
> >>> field (timestamp_done or something like that). Whether this would be hardcoded as using
> >>> CLOCK_BOOTTIME or uses the same clock source as selected through a control is something
> >>> that can be discussed, but since it does require a new field I believe this is part of
> >>> this proposal.
> >>
> >> I'm probably lacking m2m knowledge here.
> >> timestamp_done would be when the driver set the buffer as done, what about the other timestamp?
> >> I would like to rename it to make it clear what it means, maybe image_timestamp?
> >> Where, in a capture device, image_timestamp would be the timestamp from the hardware when
> >> it captured that specific frame, and in a decoder, it would be the timestamp of that frame
> >> when it got encoded?
> >> Is this correct?
> >>
> >> Thanks
> >> Helen
> >>
> >>>
> >>> Regards,
> >>>
> >>>       Hans
> >>>
> >>>>
> >>>> I was recalling the discussions we had regarding this:
> >>>>
> >>>> 1.
> >>>>     This first attempt in the uvc driver is to use a specific kernel parameter for that case:
> >>>>     https://patchwork.kernel.org/patch/10644887/
> >>>>     The conclusion that the support should be in the core API and not driver specific.
> >>>>
> >>>> 2.
> >>>>     Then an attempt to add global v4l2 support was sent with the Mediatek patch series:
> >>>>     https://patchwork.linuxtv.org/patch/60878/
> >>>>     The major problem is that clock type should be something selectable by userspace, and
> >>>>     not pre-defined by the driver.
> >>>>
> >>>> 3.
> >>>>     Another idea was to use the 'flags' field in the structs v4l2_requestbuffers and
> >>>>     v4l2_create_buffers.
> >>>>     But this field was removed in
> >>>>     129134e5415d ("media: media/v4l2: remove V4L2_FLAG_MEMORY_NON_CONSISTENT flag")
> >>>>     The major concern with this approach was with the uAPI, since it doesn't make much
> >>>>     sense to select a clock when creating buffers.
> >>>>
> >>>> 4.
> >>>>     Another suggestion by Nicolas Dufresne was to add this as a menu control so that userspace
> >>>>     can choose the clock for the timestamps from a given list, the enum in the list can also match
> >>>>     the clocks ids.
> >>>>     We would need to add a new buf flag in struct v4l2_buffer, like V4L2_BUF_FLAG_TIMESTAMP_OTHER,
> >>>>     which would be "as specified through controls ...."
> >>>>
> >>>>
> >>>> So my current question is, should we have both __u32 timestamp and __u32 boottimestamp?
> >>>> Or should we have a mechanism that allows switching from one to the other and use
> >>>> a single field? And if this mechanism should be implemented in both APIs? Can this be
> >>>> defined later?
> >>>>
> >>>>
> >>>> Please, let me know your thoughts.
> >>>>
> >>>> Thanks,
> >>>> Helen
> >>>>
> >>>>>
> >>>>>>>
> >>>>>>>> +
> >>>>>>>>  #ifndef __KERNEL__
> >>>>>>>>  /**
> >>>>>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
> >>>>>>>> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
> >>>>>>>>      __u32                   reserved[6];
> >>>>>>>>  };
> >>>>>>>>
> >>>>>>>> +/**
> >>>>>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
> >>>>>>>> + * @index:  on return, index of the first created buffer
> >>>>>>>> + * @count:  entry: number of requested buffers,
> >>>>>>>> + *          return: number of created buffers
> >>>>>>>> + * @memory: enum v4l2_memory; buffer memory type
> >>>>>>>> + * @capabilities: capabilities of this buffer type.
> >>>>>>>> + * @format: frame format, for which buffers are requested
> >>>>>>>> + * @flags:  additional buffer management attributes (ignored unless the
> >>>>>>>> + *          queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
> >>>>>>>> + *          and configured for MMAP streaming I/O).
> >>>>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
> >>>>>>>> + */
> >>>>>>>> +struct v4l2_ext_create_buffers {
> >>>>>>>> +    __u32                           index;
> >>>>>>>> +    __u32                           count;
> >>>>>>>> +    __u32                           memory;
> >>>>>>>> +    __u32                           capabilities;
> >>>>>>>> +    struct v4l2_ext_pix_format      format;
> >>>>>>>
> >>>>>>> The reality is that the only field that is ever used in the original v4l2_format
> >>>>>>> struct is sizeimage. So this can be replaced with:
> >>>>>>>
> >>>>>>>       __u32                           plane_size[VIDEO_MAX_PLANES];
> >>>>>>>
> >>>>>>> (the field name I picked is debatable, but you get the idea)
> >>>>>>>
> >>>>>>> The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
> >>>>>>> is needed for the current format. The original idea of using struct v4l2_format
> >>>>>>> was that drivers would use the full format information to calculate the
> >>>>>>> memory size, but that was just much too complicated to implement and nobody
> >>>>>>> ever used that. Only the sizeimage field was ever used.
> >>>>>>
> >>>>>> Right, I'll update this in next version, This should simplify things.
> >>>>>>
> >>>>>
> >>>>> I think this might need a bit more discussion. How would the userspace
> >>>>> know what size is enough for the desired resolution? The hardware
> >>>>> and/or drivers often have various alignment/padding restrictions,
> >>>>> which might not be easy to guess for the userspace.
> >>>>>
> >>>>> Also I don't quite understand what's so complicated in handling the
> >>>>> full format, or at least the most important parts of it. The
> >>>>> implementation of TRY_FMT/S_FMT, which exists in every driver, should
> >>>>> already be able to calculate the right plane sizes.
> >>>>>
> >>>>> Best regards,
> >>>>> Tomasz
> >>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Helen
> >>>>>>
> >>>>>>>
> >>>>>>>> +    __u32                           flags;
> >>>>>>>> +    __u32 reserved[5];
> >>>>>>>> +};
> >>>>>>>> +
> >>>>>>>>  /*
> >>>>>>>>   *  I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
> >>>>>>>>   *
> >>>>>>>> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
> >>>>>>>>  #define VIDIOC_G_EXT_PIX_FMT        _IOWR('V', 104, struct v4l2_ext_pix_format)
> >>>>>>>>  #define VIDIOC_S_EXT_PIX_FMT        _IOWR('V', 105, struct v4l2_ext_pix_format)
> >>>>>>>>  #define VIDIOC_TRY_EXT_PIX_FMT      _IOWR('V', 106, struct v4l2_ext_pix_format)
> >>>>>>>> +#define VIDIOC_EXT_CREATE_BUFS      _IOWR('V', 107, struct v4l2_ext_create_buffers)
> >>>>>>>> +#define VIDIOC_EXT_QUERYBUF _IOWR('V', 108, struct v4l2_ext_buffer)
> >>>>>>>> +#define VIDIOC_EXT_QBUF             _IOWR('V', 109, struct v4l2_ext_buffer)
> >>>>>>>> +#define VIDIOC_EXT_DQBUF    _IOWR('V', 110, struct v4l2_ext_buffer)
> >>>>>>>> +#define VIDIOC_EXT_PREPARE_BUF      _IOWR('V', 111, struct v4l2_ext_buffer)
> >>>>>>>>
> >>>>>>>>  /* Reminder: when adding new ioctls please add support for them to
> >>>>>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
> >>>>>>>>
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>>       Hans
> >>>>>>>
> >>>
> >

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-16  3:13                   ` Tomasz Figa
@ 2020-12-17 13:19                     ` Helen Koike
  2020-12-21  3:13                       ` Tomasz Figa
  0 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-12-17 13:19 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Hans Verkuil, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

Hi Tomasz,

Thanks for your comments, I have a few questions below.

On 12/16/20 12:13 AM, Tomasz Figa wrote:
> On Tue, Dec 15, 2020 at 11:37 PM Helen Koike <helen.koike@collabora.com> wrote:
>>
>> Hi Tomasz,
>>
>> On 12/14/20 7:46 AM, Tomasz Figa wrote:
>>> On Fri, Dec 4, 2020 at 4:52 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Please see my 2 points below (about v4l2_ext_buffer and another about timestamp).
>>>>
>>>> On 12/3/20 12:11 PM, Hans Verkuil wrote:
>>>>> On 23/11/2020 18:40, Helen Koike wrote:
>>>>>>
>>>>>>
>>>>>> On 11/23/20 12:46 PM, Tomasz Figa wrote:
>>>>>>> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>>>>
>>>>>>>> Hi Hans,
>>>>>>>>
>>>>>>>> Thank you for your review.
>>>>>>>>
>>>>>>>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
>>>>>>>>> Hi Helen,
>>>>>>>>>
>>>>>>>>> Again I'm just reviewing the uAPI.
>>>>>>>>>
>>>>>>>>> On 04/08/2020 21:29, Helen Koike wrote:
>>>>>>>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>>>>>>
>>>>>>>>>> Those extended buffer ops have several purpose:
>>>>>>>>>> 1/ Fix y2038 issues by converting the timestamp into an u64 counting
>>>>>>>>>>    the number of ns elapsed since 1970
>>>>>>>>>> 2/ Unify single/multiplanar handling
>>>>>>>>>> 3/ Add a new start offset field to each v4l2 plane buffer info struct
>>>>>>>>>>    to support the case where a single buffer object is storing all
>>>>>>>>>>    planes data, each one being placed at a different offset
>>>>>>>>>>
>>>>>>>>>> New hooks are created in v4l2_ioctl_ops so that drivers can start using
>>>>>>>>>> these new objects.
>>>>>>>>>>
>>>>>>>>>> The core takes care of converting new ioctls requests to old ones
>>>>>>>>>> if the driver does not support the new hooks, and vice versa.
>>>>>>>>>>
>>>>>>>>>> Note that the timecode field is gone, since there doesn't seem to be
>>>>>>>>>> in-kernel users. We can be added back in the reserved area if needed or
>>>>>>>>>> use the Request API to collect more metadata information from the
>>>>>>>>>> frame.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>>>>>>>> Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
>>>>>>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>>>>>>> ---
>>>>>>>>>> Changes in v5:
>>>>>>>>>> - migrate memory from v4l2_ext_buffer to v4l2_ext_plane
>>>>>>>>>> - return mem_offset to struct v4l2_ext_plane
>>>>>>>>>> - change sizes and reorder fields to avoid holes in the struct and make
>>>>>>>>>>   it the same for 32 and 64 bits
>>>>>>>>>>
>>>>>>>>>> Changes in v4:
>>>>>>>>>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>>>>>>>>>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>>>>>>>>>> - Drop VIDIOC_EXT_EXPBUF, since the only difference from VIDIOC_EXPBUF
>>>>>>>>>> was that with VIDIOC_EXT_EXPBUF we could export multiple planes at once.
>>>>>>>>>> I think we can add this later, so I removed it from this RFC to simplify it.
>>>>>>>>>> - Remove num_planes field from struct v4l2_ext_buffer
>>>>>>>>>> - Add flags field to struct v4l2_ext_create_buffers
>>>>>>>>>> - Reformulate struct v4l2_ext_plane
>>>>>>>>>> - Fix some bugs caught by v4l2-compliance
>>>>>>>>>> - Rebased on top of media/master (post 5.8-rc1)
>>>>>>>>>>
>>>>>>>>>> Changes in v3:
>>>>>>>>>> - Rebased on top of media/master (post 5.4-rc1)
>>>>>>>>>>
>>>>>>>>>> Changes in v2:
>>>>>>>>>> - Add reserved space to v4l2_ext_buffer so that new fields can be added
>>>>>>>>>>   later on
>>>>>>>>>> ---
>>>>>>>>>>  drivers/media/v4l2-core/v4l2-dev.c   |  29 ++-
>>>>>>>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 353 +++++++++++++++++++++++++--
>>>>>>>>>>  include/media/v4l2-ioctl.h           |  26 ++
>>>>>>>>>>  include/uapi/linux/videodev2.h       |  90 +++++++
>>>>>>>>>>  4 files changed, 476 insertions(+), 22 deletions(-)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <snip>
>>>>>>>>>
>>>>>>>>>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>>>>>>>>> index 7123c6a4d9569..334cafdd2be97 100644
>>>>>>>>>> --- a/include/uapi/linux/videodev2.h
>>>>>>>>>> +++ b/include/uapi/linux/videodev2.h
>>>>>>>>>> @@ -996,6 +996,41 @@ struct v4l2_plane {
>>>>>>>>>>      __u32                   reserved[11];
>>>>>>>>>>  };
>>>>>>>>>>
>>>>>>>>>> +/**
>>>>>>>>>> + * struct v4l2_ext_plane - extended plane buffer info
>>>>>>>>>> + * @buffer_length:  size of the entire buffer in bytes, should fit
>>>>>>>>>> + *                  @offset + @plane_length
>>>>>>>>>> + * @plane_length:   size of the plane in bytes.
>>>>>>>>>> + * @mem_offset:             If V4L2_MEMORY_MMAP is used, then it can be a "cookie"
>>>>>>>>>> + *                  that should be passed to mmap() called on the video node.
>>>>>>>>>> + * @userptr:                when memory is V4L2_MEMORY_USERPTR, a userspace pointer pointing
>>>>>>>>>> + *                  to this plane.
>>>>>>>>>> + * @dmabuf_fd:              when memory is V4L2_MEMORY_DMABUF, a userspace file descriptor
>>>>>>>>>> + *                  associated with this plane.
>>>>>>>>>> + * @offset:         offset in the memory buffer where the plane starts.
>>>>>>>>>> + * @memory:         enum v4l2_memory; the method, in which the actual video
>>>>>>>>>> + *                  data is passed
>>>>>>>>>> + * @reserved:               extra space reserved for future fields, must be set to 0.
>>>>>>>>>> + *
>>>>>>>>>> + *
>>>>>>>>>> + * Buffers consist of one or more planes, e.g. an YCbCr buffer with two planes
>>>>>>>>>> + * can have one plane for Y, and another for interleaved CbCr components.
>>>>>>>>>> + * Each plane can reside in a separate memory buffer, or even in
>>>>>>>>>> + * a completely separate memory node (e.g. in embedded devices).
>>>>>>>>>> + */
>>>>>>>>>> +struct v4l2_ext_plane {
>>>>>>>>>> +    __u32 buffer_length;
>>>>>>>>>> +    __u32 plane_length;
>>>>>>>>>> +    union {
>>>>>>>>>> +            __u32 mem_offset;
>>>>>>>>>> +            __u64 userptr;
>>>>>>>>>> +            __s32 dmabuf_fd;
>>>>>>>>>> +    } m;
>>>>>>>>>> +    __u32 offset;
>>>>>>>>>
>>>>>>>>> I'd rename this plane_offset. I think some reordering would make this struct easier
>>>>>>>>> to understand:
>>>>>>>>>
>>>>>>>>> struct v4l2_ext_plane {
>>>>>>>>>       __u32 buffer_length;
>>>>>>>>>       __u32 plane_offset;
>>>>>>>>>       __u32 plane_length;
>>>>>>>>>       __u32 memory;
>>>>>>>>>       union {
>>>>>>>>>               __u32 mem_offset;
>>>>>>>>>               __u64 userptr;
>>>>>>>>>               __s32 dmabuf_fd;
>>>>>>>>>       } m;
>>>>>>>>>       __u32 reserved[4];
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>>> +    __u32 memory;
>>>>>>>>>> +    __u32 reserved[4];
>>>>>>>>>> +};
>>>>>>>>
>>>>>>>> Ok, I'll apply this to the next version.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> What is not clear is how to tell the different between a single buffer containing
>>>>>>>>> multiple planes, and using a separate buffer per plane. E.g. what would this look
>>>>>>>>> like for V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_YUV420M and a theoretical variant of
>>>>>>>>> V4L2_PIX_FMT_YUV420M where the luma plane has its own buffer and the two chroma
>>>>>>>>> planes are also combined in a single buffer?
>>>>>>>>>
>>>>>>>>> I would guess that the m union is set to 0 if the plane is part of the buffer
>>>>>>>>> defined in the previous plane?
>>>>>>>>
>>>>>>>> The difference would be if m are equal or differ between planes, example:
>>>>>>>>
>>>>>>>> For V4L2_PIX_FMT_YVU420:
>>>>>>>>
>>>>>>>>     Y:
>>>>>>>>         plane_offset = 0
>>>>>>>>         m.dmabuf_fd = 3
>>>>>>>>     Cb:
>>>>>>>>         plane_offset = 300
>>>>>>>>         m.dmabuf_fd = 3
>>>>>>>>     Cr:
>>>>>>>>         plane_offset = 375
>>>>>>>>         m.dmabuf_fd = 3
>>>>>>>>
>>>>>>>> For V4L2_PIX_FMT_YVU420M:
>>>>>>>>
>>>>>>>>     Y:
>>>>>>>>         plane_offset = 0
>>>>>>>>         m.dmabuf_fd = 4
>>>>>>>>     Cb:
>>>>>>>>         plane_offset = 0
>>>>>>>>         m.dmabuf_fd = 5
>>>>>>>>     Cr:
>>>>>>>>         plane_offset = 0
>>>>>>>>         m.dmabuf_fd = 6
>>>>>>>>
>>>>>>>>
>>>>>>>> Does it make sense?
>>>>>>>>
>>>>>>>
>>>>>>> Actually all the 3 file descriptors can still point to the same
>>>>>>> buffer, because they might have been dup()ed. The kernel needs to
>>>>>>> resolve the file descriptors into struct dma_buf and then check
>>>>>>> whether it's one or more buffers.
>>>>>>
>>>>>> Right, thanks for this.
>>>>>>
>>>>>>>
>>>>>>> In fact, dup()ed FD for each plane is quite a common case in other
>>>>>>> APIs, e.g. EGL, but current V4L2 API can't handle it. In Chromium we
>>>>>>> basically work around it by assuming that if we receive a buffer for a
>>>>>>> V4L2 device that only supports non-M formats, then we can safely
>>>>>>> ignore all but first FD. The new API gives the ability to handle the
>>>>>>> case properly, with full validation by the kernel.
>>>>>>>
>>>>>>>>>
>>>>>>>>>> +
>>>>>>>>>>  /**
>>>>>>>>>>   * struct v4l2_buffer - video buffer info
>>>>>>>>>>   * @index:  id number of the buffer
>>>>>>>>>> @@ -1057,6 +1092,33 @@ struct v4l2_buffer {
>>>>>>>>>>      };
>>>>>>>>>>  };
>>>>>>>>>>
>>>>>>>>>> +/**
>>>>>>>>>> + * struct v4l2_ext_buffer - extended video buffer info
>>>>>>>>>> + * @index:  id number of the buffer
>>>>>>>>>> + * @type:   V4L2_BUF_TYPE_VIDEO_CAPTURE or V4L2_BUF_TYPE_VIDEO_OUTPUT
>>>>>>>>>> + * @flags:  buffer informational flags
>>>>>>>>>> + * @field:  enum v4l2_field; field order of the image in the buffer
>>>>>>>>>> + * @timestamp:      frame timestamp
>>>>>>>>>> + * @sequence:       sequence count of this frame
>>>>>>>>>> + * @planes: per-plane buffer information
>>>>>>>>>> + * @request_fd:     fd of the request that this buffer should use
>>>>>>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>>>>>>>> + *
>>>>>>>>>> + * Contains data exchanged by application and driver using one of the Streaming
>>>>>>>>>> + * I/O methods.
>>>>>>>>>> + */
>>>>>>>>>> +struct v4l2_ext_buffer {
>>>>>>>>>> +    __u32 index;
>>>>>>>>>> +    __u32 type;
>>>>>>>>>> +    __u32 field;
>>>>>>>>>> +    __u32 sequence;
>>>>>>>>>> +    __u64 flags;
>>>>>>>>>> +    __u64 timestamp;
>>>>>>>>>> +    struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>>>
>>>> I would like your opinion on the following:
>>>>
>>>> We have two concepts here
>>>> * memory buffers (that belongs to a frame buffer object aka v4l2_ext_buffer)
>>>> * color components/planes (that we need to indicate to userspace where the planes
>>>>   are located, which buffer and which offset inside the buffer).
>>>>
>>>> A v4l2_ext_buffer can be reused to a different format if it fits the image
>>>> (which is checked in QBUF time, by .buf_prepare() callback).
>>>>
>>>> Which means that, the information regarding where each color component is placed
>>>> just make sense after the buffer is queued.
>>>>
>>>> So if userspace calls EXT_QUERYBUF, and the buffer isn't queued, the color component
>>>> information doesn't make sense.
>>>>
>>>> One option is to fill in the plane information according to the current
>>>> configured format, but only the information returned from EXT_QBUF is guaranteed
>>>> to be the correct one.
>>>>
>>>> Another options is to split struct v4l2_ext_plane in two:
>>>>
>>>> struct v4l2_ext_membuffer {
>>>>         __u32 memory;
>>>>         union {
>>>>                 __u32 mmap_offset;
>>>>                 __u64 userptr;
>>>>                 __s32 dmabuf_fd;
>>>>         } m;
>>>>         _u32 buffer_length;
>>>> }
>>>>
>>>> struct v4l2_ext_plane {
>>>>         /*
>>>>          * memory buffer where this plane belongs, index is the position in of
>>>>          * membuffers[] in struct v4l2_ext_buffer below
>>>>          */
>>>>         unsigned int membuf_index;
>>>>         _u32 plane_length;
>>>>         _u32 plane_offset;
>>>> }
>>>>
>>>> Then we would have
>>>>
>>>> struct v4l2_ext_buffer {
>>>>         ... <snip>
>>>>         struct v4l2_ext_membuffer membuffers[VIDEO_MAX_PLANES];
>>>>         struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>>>         ... <snip>
>>>> }
>>>>
>>>> Where planes would only be filled by the core only if the buffer is
>>>> queued (i.e. "locked" to a given format).
>>>>
>>>> This also avoids having several planes with different dmabuf_fd that are dup()ed,
>>>> since we'll have an entry per memory buffer.
>>>> Which also avoids the following:
>>>> If we are working with a single membuf for all planes for instance, vb2 would need
>>>> to know how many planes (let's say there are 3) and repeat the mem buffer information
>>>> 3 times, and if userspace changes to a pixelformat with 2 color components, we would
>>>> repeat 2 times with the same information. And we wouldn't have this issue
>>>> if we split both information.
>>>>
>>>> I was also assuming that once the buffer is queued, userspace can't modify
>>>> the configured format (I need to check this, but make sense to me due to how
>>>> .buf_prepare() works).
>>>>
>>>> What do you think? Does it make sense?
>>>>
>>>
>>> How about making querybuf have its own structure that describes only
>>> the buffer, as it is allocated? I.e.
>>
>> Make sense to me to add a struct v4l2_ext_mmap_querybuf, so we don't need
>> to inform color information in this ioctl.
>>
>> We still need a way to tell which memory buffer a color plane was placed.
>>
>> I believe this must be in v4l2_ext_buffer, since userspace needs to know
>> on DQBUF time where to find each plane/color component.
>>
>> So we need a way to enumerate memory buffers, and have planes pointing
>> to them in someway.
>>
>> Just to be clear, the way I'm seeing this is:
>> A buffer object can be composed by one or more memory buffers (membufs).
>> Color planes can be placed randomly in any memory buffer, drivers (capture)
>> and userspace (output) will decide where these planes will be placed.
>>
>> Or this doesn't make sense and we can assume that or all planes will be placed
>> in a single memory buffer, or each will be placed in a different memory buffer?
>> To avoid cases where we have 2 membufs and 3 planes?
>>
> 
> We have two cases here:
> 
> 1) MMAP - the placement of color planes is controlled by the kernel.
> In practice, we've ever had only 2 cases:
>     a) for non-M formats all the color planes go to the same memory buffer,
>     b) for M formats, each color plane has its own memory buffer.
> 
> 2) DMA-buf and userptr - the placement is fully controlled by the application.
> 
> Would it make sense to just define the behavior of MMAP buffers as per
> the two cases, 1a and 1b, above and get rid of memory buffer
> information from DQBUF?

Make sense to me.

> 
>> Note: we probably should standardize terminology through the docs, structs
>>       and functions to differentiate color planes, memory buffer and buffer
>>       objects.
> 
> Note that the existing terminology is: color planes, memory planes and
> buffers. I think it might be desirable to preserve it to avoid further
> confusion with all the old and new terms mixing with each other.

Right, We can keep this terminology yes.
the confusing part is that sometimes the word "planes" is used standalone.


> 
>>
>>>
>>> struct v4l2_ext_mmap_plane {
>>
>> I would rename this to v4l2_ext_mmap_membuf to not confuse with color planes.
>>
>>>         __u32 mmap_offset;
>>>         __u32 length;
>>> }
>>>
>>> struct v4l2_ext_querybuf {
>>
>> I would rename this to v4l2_ext_mmap_querybuf, since it only make
>> sense to mmap, and the ioctl can be called VIDIOC_EXT_QUERY_MMAP_BUF,
>> where it would return -EINVAL if the index is not a mmaped buffers.
>>
>>>         /* ... */
>>>         struct v4l2_ext_mmap_plane planes[VIDEO_MAX_PLANES];
>>
>> Then:
>>          struct v4l2_ext_mmap_membuf membufs[VIDEO_MAX_PLANES];
>>
>>> };
>>>
>>> Moreover, the ioctl would be only valid if the queue is operating in
>>> MMAP mode, because otherwise it doesn't provide any useful information
>>> - the userspace should already know what userptr or DMA-buf was
>>> associated with the buffer. In fact, returning the DMA-buf FD from
>>> QBUF time is confusing, because the userspace code might have already
>>> closed that FD (it can rely on other way of referencing the buffer).
>>
>> so I guess we should zero those fields in QBUF.
>>
> 
> Do you mean DQBUF?

Yes.

> 
>>>
>>> WDYT?
>>
>> Please check my proposal below and see what you think.
>>
>>
>> For DMA-buf and Userptr
>> -----------------------
>> userspace performs EXT_CREATE_BUF ioctl to reserve a buffer "index" range in
>> that mode, to be used in EXT_QBUF and EXT_DQBUF
>>
>> On EXT_QBUF, userspace must tell the driver about existent memory buffers and the
>> corresponding DMA-fd or Userptr.
>>
>> Capture: driver fills plane information, informing in which memory buffer each plane
>>          was placed (Or should this be pre-determined by userspace?)
> 
> This has to be fully controlled by the userspace, since it's the
> userspace that allocates the buffers and has its own expectations
> about their layout.

Ok.

> 
>>
>> Output: userspace fills plane information, informing in which memory buffer each
>>         plane was placed (Or should this be pre-determined by the driver?)
>>
>> For MMAP
>> -----------------------
>> userspace performs EXT_CREATE_BUF ioctl to reserve a buffer "index" range in
>> that mode, to be used in EXT_QBUF and EXT_DQBUF
>>
>> Should the API allow userspace to select how many memory buffers it wants?
>> (maybe not)
> 
> I think it does allow that - it accepts the v4l2_ext_format struct.

hmmm, I thought v4l2_ext_format would describe color planes, and not memory planes.
Should it describe memory planes instead? Since planes are defined by the pixelformat.
But is this information relevant to ext_{set/get/try} format?

> 
>>
>> userspace performs EXT_QUERY_MMAP_BUF to get the mmap offset/cookie and length
>> for each memory buffer.
>>
>> On EXT_QBUF, userspace doesn't need to fill membuf information. Should the
>> mmap offset and length be filled by the kernel and returned to userspace here
>> as well? I'm leaning towards: no.
> 
> Yeah, based on my comment above, I think the answer should be no.
> 
>>
>> If the answer is no, then here is my proposal:
>> ----------------------------------------------
>>
>> /* If MMAP, drivers decide how many memory buffers to allocate */
>> int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_buffer *argp )
>>
>> /* Returns -EINVAL if not MMAP */
>> int ioctl( int fd, VIDIOC_EXT_MMAP_QUERYBUF, struct v4l2_ext_mmap_querybuf *argp )
>>
>> /* userspace fills v4l2_ext_buffer.membufs if DMA-fd or Userptr, leave it zero for MMAP
>>  * Should userspace also fill v4l2_ext_buffer.planes?
>>  */
>> int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
>>
>> /* v4l2_ext_buffer.membufs is set to zero by the driver */
>> int ioctl( int fd, VIDIOC_EXT_DBUF, struct v4l2_ext_buffer *argp )
>>
>> (I omitted reserved fields below)
>>
>> struct v4l2_ext_create_buffers {
>>         __u32                           index;
>>         __u32                           count;
>>         __u32                           memory;
>>         __u32                           capabilities;
>>         struct v4l2_ext_pix_format      format;
>> };
>>
>> struct v4l2_ext_mmap_membuf {
>>         __u32 offset;
>>         __u32 length;
>> }
>>
>> struct v4l2_ext_mmap_querybuf {
>>         __u32 index;
>>         struct v4l2_ext_mmap_membuf membufs[VIDEO_MAX_PLANES];
>> }
>>
>> struct v4l2_ext_membuf {
>>         __u32 memory;
>>         union {
>>                 __u64 userptr;
>>                 __s32 dmabuf_fd;
>>         } m;
>>         // Can't we just remove the union and "memory" field, and the non-zero
>>         // is the one we should use?
> 
> I think that would lead to an equivalent result in this case. That
> said, I'm not sure if there would be any significant enough benefit to
> justify moving away from the current convention. Having the memory
> field might also make the structure a bit less error prone, e.g.
> resilient to missing memset().
> 
>> };
>>
>> struct v4l2_ext_plane {
>>         __u32 membuf_index;
>>         __u32 offset;
>>         __u32 bytesused;
>> };
>>
>> struct v4l2_ext_buffer {
>>         __u32 index;
>>         __u32 type;
>>         __u32 field;
>>         __u32 sequence;
>>         __u64 flags;
>>         __u64 timestamp;
>>         struct v4l2_ext_membuf membufs[VIDEO_MAX_PLANES];
>>         struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> 
> Do we actually need this split into membufs and planes here? After
> all, all we want to pass to the kernel here is in what buffer the
> plane is in.

You are right, we don't.

> 
> struct v4l2_ext_plane {
>         __u32 memory;

Should we design the API to allow a buffer to contain multiple memory planes
of different types? Lets say one memplane is DMA-fd, the other is userptr.
If the answer is yes, then struct v4l2_ext_create_buffers requires some changes.
If not, then there is no need a "memory" field per memory plane in a buffer.

>         union {
>                 __u32 membuf_index;
>                 __u64 userptr;
>                 __s32 dmabuf_fd;
>         } m;
>         __u32 offset;
>         __u32 bytesused;

We also need userptr_length right?

> };
> 
> Also, we might not even need membuf_index, if we strictly define the
> color plane placement for MMAP as per my comment above.

Ack.

Thanks,
Helen

> 
> Best regards,
> Tomasz
> 
>>         __s32 request_fd;
>> };
>>
>>
>> Thanks,
>> Helen
>>
>>>
>>> Best regards.
>>> Tomasz
>>>
>>>>
>>>>>>>>>> +    __s32 request_fd;
>>>>>>>>>> +    __u32 reserved[9];
>>>>>>>>>> +};
>>>>>>>>>
>>>>>>>>> Brainstorming:
>>>>>>>>>
>>>>>>>>> Some ideas I have to make it easier to support mid stream resolution/colorimetry
>>>>>>>>> changes:
>>>>>>>>>
>>>>>>>>> Adding width and height would support resolution changes (requires the use of
>>>>>>>>> CREATE_BUFS to ensure the allocated buffers are large enough, of course). If that
>>>>>>>>> information is provided here, then there are no race conditions.
>>>>>>>>>
>>>>>>>>> Same for adding the colorimetry fields here, this too can change on the fly (esp.
>>>>>>>>> with HDMI), so reporting this information here avoids race conditions as well.
>>>>>>>>
>>>>>>>> Right, do you think this is something we can discuss later in a different RFC?
>>>>>>>> So we can have a better view on how dynamic resolution change would be used?
>>>>>>>>
>>>>>>>> We can add more reserved fields or maybe try to do something to what has been
>>>>>>>> discussed in about extensible system calls [1]
>>>>>>>>
>>>>>>>> [1] https://lwn.net/Articles/830666/
>>>>>>>>
>>>>>>>>>
>>>>>>>>> And thirdly, I would like to have a __u64 boot_timestamp field containing the
>>>>>>>>> CLOCK_BOOTTIME of when the vb2_buffer_done() was called. The problem with 'timestamp'
>>>>>>>>> is that for m2m devices it is just copied and that for other devices it can have
>>>>>>>>> different meanings depending on the timestamp buffer flags.
>>>>>>>>>
>>>>>>>>> There also have been requests for CLOCK_BOOTTIME support, so this might be a good time
>>>>>>>>> to add support for this. That way you know exactly when the driver was finished with
>>>>>>>>> the buffer and that helps in detecting missed frames or instrumentation.
>>>>>>>>
>>>>>>>> I don't mind adding it. Does it make sense to have both timestamp and boot_timestamp?
>>>>>>>>
>>>>>>>
>>>>>>> I think this is quite independent from the ext API work. AFAIR there
>>>>>>> was an RFC to request the timestamp source from the userspace by the
>>>>>>> flags field in QBUF, which would work with the existing API as well,
>>>>>>> or it wasn't posted in the end?
>>>>>
>>>>> It's not about selecting a specific clock source. I think that option 4 as described
>>>>> below would work for that.
>>>>>
>>>>> This problem I'm describing here is specific to m2m devices where the timestamp is
>>>>> either just passed through untouched, or it is used as an identifier for a buffer
>>>>> for use with stateless decoders.
>>>>>
>>>>> In both cases you cannot use the timestamp as a proper timestamp that tells you when
>>>>> the buffer was marked done by the driver. So this is about adding a second timestamp
>>>>> field (timestamp_done or something like that). Whether this would be hardcoded as using
>>>>> CLOCK_BOOTTIME or uses the same clock source as selected through a control is something
>>>>> that can be discussed, but since it does require a new field I believe this is part of
>>>>> this proposal.
>>>>
>>>> I'm probably lacking m2m knowledge here.
>>>> timestamp_done would be when the driver set the buffer as done, what about the other timestamp?
>>>> I would like to rename it to make it clear what it means, maybe image_timestamp?
>>>> Where, in a capture device, image_timestamp would be the timestamp from the hardware when
>>>> it captured that specific frame, and in a decoder, it would be the timestamp of that frame
>>>> when it got encoded?
>>>> Is this correct?
>>>>
>>>> Thanks
>>>> Helen
>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>       Hans
>>>>>
>>>>>>
>>>>>> I was recalling the discussions we had regarding this:
>>>>>>
>>>>>> 1.
>>>>>>     This first attempt in the uvc driver is to use a specific kernel parameter for that case:
>>>>>>     https://patchwork.kernel.org/patch/10644887/
>>>>>>     The conclusion that the support should be in the core API and not driver specific.
>>>>>>
>>>>>> 2.
>>>>>>     Then an attempt to add global v4l2 support was sent with the Mediatek patch series:
>>>>>>     https://patchwork.linuxtv.org/patch/60878/
>>>>>>     The major problem is that clock type should be something selectable by userspace, and
>>>>>>     not pre-defined by the driver.
>>>>>>
>>>>>> 3.
>>>>>>     Another idea was to use the 'flags' field in the structs v4l2_requestbuffers and
>>>>>>     v4l2_create_buffers.
>>>>>>     But this field was removed in
>>>>>>     129134e5415d ("media: media/v4l2: remove V4L2_FLAG_MEMORY_NON_CONSISTENT flag")
>>>>>>     The major concern with this approach was with the uAPI, since it doesn't make much
>>>>>>     sense to select a clock when creating buffers.
>>>>>>
>>>>>> 4.
>>>>>>     Another suggestion by Nicolas Dufresne was to add this as a menu control so that userspace
>>>>>>     can choose the clock for the timestamps from a given list, the enum in the list can also match
>>>>>>     the clocks ids.
>>>>>>     We would need to add a new buf flag in struct v4l2_buffer, like V4L2_BUF_FLAG_TIMESTAMP_OTHER,
>>>>>>     which would be "as specified through controls ...."
>>>>>>
>>>>>>
>>>>>> So my current question is, should we have both __u32 timestamp and __u32 boottimestamp?
>>>>>> Or should we have a mechanism that allows switching from one to the other and use
>>>>>> a single field? And if this mechanism should be implemented in both APIs? Can this be
>>>>>> defined later?
>>>>>>
>>>>>>
>>>>>> Please, let me know your thoughts.
>>>>>>
>>>>>> Thanks,
>>>>>> Helen
>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>>> +
>>>>>>>>>>  #ifndef __KERNEL__
>>>>>>>>>>  /**
>>>>>>>>>>   * v4l2_timeval_to_ns - Convert timeval to nanoseconds
>>>>>>>>>> @@ -2523,6 +2585,29 @@ struct v4l2_create_buffers {
>>>>>>>>>>      __u32                   reserved[6];
>>>>>>>>>>  };
>>>>>>>>>>
>>>>>>>>>> +/**
>>>>>>>>>> + * struct v4l2_ext_create_buffers - VIDIOC_EXT_CREATE_BUFS argument
>>>>>>>>>> + * @index:  on return, index of the first created buffer
>>>>>>>>>> + * @count:  entry: number of requested buffers,
>>>>>>>>>> + *          return: number of created buffers
>>>>>>>>>> + * @memory: enum v4l2_memory; buffer memory type
>>>>>>>>>> + * @capabilities: capabilities of this buffer type.
>>>>>>>>>> + * @format: frame format, for which buffers are requested
>>>>>>>>>> + * @flags:  additional buffer management attributes (ignored unless the
>>>>>>>>>> + *          queue has V4L2_BUF_CAP_SUPPORTS_MMAP_CACHE_HINTS capability
>>>>>>>>>> + *          and configured for MMAP streaming I/O).
>>>>>>>>>> + * @reserved:       extra space reserved for future fields, must be set to 0
>>>>>>>>>> + */
>>>>>>>>>> +struct v4l2_ext_create_buffers {
>>>>>>>>>> +    __u32                           index;
>>>>>>>>>> +    __u32                           count;
>>>>>>>>>> +    __u32                           memory;
>>>>>>>>>> +    __u32                           capabilities;
>>>>>>>>>> +    struct v4l2_ext_pix_format      format;
>>>>>>>>>
>>>>>>>>> The reality is that the only field that is ever used in the original v4l2_format
>>>>>>>>> struct is sizeimage. So this can be replaced with:
>>>>>>>>>
>>>>>>>>>       __u32                           plane_size[VIDEO_MAX_PLANES];
>>>>>>>>>
>>>>>>>>> (the field name I picked is debatable, but you get the idea)
>>>>>>>>>
>>>>>>>>> The main purpose of CREATE_BUFS is to add new buffers with larger sizes than
>>>>>>>>> is needed for the current format. The original idea of using struct v4l2_format
>>>>>>>>> was that drivers would use the full format information to calculate the
>>>>>>>>> memory size, but that was just much too complicated to implement and nobody
>>>>>>>>> ever used that. Only the sizeimage field was ever used.
>>>>>>>>
>>>>>>>> Right, I'll update this in next version, This should simplify things.
>>>>>>>>
>>>>>>>
>>>>>>> I think this might need a bit more discussion. How would the userspace
>>>>>>> know what size is enough for the desired resolution? The hardware
>>>>>>> and/or drivers often have various alignment/padding restrictions,
>>>>>>> which might not be easy to guess for the userspace.
>>>>>>>
>>>>>>> Also I don't quite understand what's so complicated in handling the
>>>>>>> full format, or at least the most important parts of it. The
>>>>>>> implementation of TRY_FMT/S_FMT, which exists in every driver, should
>>>>>>> already be able to calculate the right plane sizes.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Tomasz
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Helen
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> +    __u32                           flags;
>>>>>>>>>> +    __u32 reserved[5];
>>>>>>>>>> +};
>>>>>>>>>> +
>>>>>>>>>>  /*
>>>>>>>>>>   *  I O C T L   C O D E S   F O R   V I D E O   D E V I C E S
>>>>>>>>>>   *
>>>>>>>>>> @@ -2626,6 +2711,11 @@ struct v4l2_create_buffers {
>>>>>>>>>>  #define VIDIOC_G_EXT_PIX_FMT        _IOWR('V', 104, struct v4l2_ext_pix_format)
>>>>>>>>>>  #define VIDIOC_S_EXT_PIX_FMT        _IOWR('V', 105, struct v4l2_ext_pix_format)
>>>>>>>>>>  #define VIDIOC_TRY_EXT_PIX_FMT      _IOWR('V', 106, struct v4l2_ext_pix_format)
>>>>>>>>>> +#define VIDIOC_EXT_CREATE_BUFS      _IOWR('V', 107, struct v4l2_ext_create_buffers)
>>>>>>>>>> +#define VIDIOC_EXT_QUERYBUF _IOWR('V', 108, struct v4l2_ext_buffer)
>>>>>>>>>> +#define VIDIOC_EXT_QBUF             _IOWR('V', 109, struct v4l2_ext_buffer)
>>>>>>>>>> +#define VIDIOC_EXT_DQBUF    _IOWR('V', 110, struct v4l2_ext_buffer)
>>>>>>>>>> +#define VIDIOC_EXT_PREPARE_BUF      _IOWR('V', 111, struct v4l2_ext_buffer)
>>>>>>>>>>
>>>>>>>>>>  /* Reminder: when adding new ioctls please add support for them to
>>>>>>>>>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>>       Hans
>>>>>>>>>
>>>>>
>>>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-17 13:19                     ` Helen Koike
@ 2020-12-21  3:13                       ` Tomasz Figa
  2020-12-23 12:04                         ` Helen Koike
  0 siblings, 1 reply; 51+ messages in thread
From: Tomasz Figa @ 2020-12-21  3:13 UTC (permalink / raw)
  To: Helen Koike
  Cc: Hans Verkuil, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

On Thu, Dec 17, 2020 at 10:20 PM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi Tomasz,
>
> Thanks for your comments, I have a few questions below.
>
> On 12/16/20 12:13 AM, Tomasz Figa wrote:
> > On Tue, Dec 15, 2020 at 11:37 PM Helen Koike <helen.koike@collabora.com> wrote:
> >>
> >> Hi Tomasz,
> >>
> >> On 12/14/20 7:46 AM, Tomasz Figa wrote:
> >>> On Fri, Dec 4, 2020 at 4:52 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> Please see my 2 points below (about v4l2_ext_buffer and another about timestamp).
> >>>>
> >>>> On 12/3/20 12:11 PM, Hans Verkuil wrote:
> >>>>> On 23/11/2020 18:40, Helen Koike wrote:
> >>>>>>
> >>>>>>
> >>>>>> On 11/23/20 12:46 PM, Tomasz Figa wrote:
> >>>>>>> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>>>>>
> >>>>>>>> Hi Hans,
> >>>>>>>>
> >>>>>>>> Thank you for your review.
> >>>>>>>>
> >>>>>>>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
> >>>>>>>>> Hi Helen,
> >>>>>>>>>
> >>>>>>>>> Again I'm just reviewing the uAPI.
> >>>>>>>>>
> >>>>>>>>> On 04/08/2020 21:29, Helen Koike wrote:
[snip]
> >
> >>
> >> Output: userspace fills plane information, informing in which memory buffer each
> >>         plane was placed (Or should this be pre-determined by the driver?)
> >>
> >> For MMAP
> >> -----------------------
> >> userspace performs EXT_CREATE_BUF ioctl to reserve a buffer "index" range in
> >> that mode, to be used in EXT_QBUF and EXT_DQBUF
> >>
> >> Should the API allow userspace to select how many memory buffers it wants?
> >> (maybe not)
> >
> > I think it does allow that - it accepts the v4l2_ext_format struct.
>
> hmmm, I thought v4l2_ext_format would describe color planes, and not memory planes.
> Should it describe memory planes instead? Since planes are defined by the pixelformat.
> But is this information relevant to ext_{set/get/try} format?
>

Good point. I ended up assuming the current convention, where giving
an M format would imply num_memory_planes == num_color_planes and
non-M format num_memory_planes == 1. Sounds like we might want
something like a flags field and that could have bits defined to
select that. I think it would actually be useful for S_FMT as well,
because that's what REQBUFS would use.

> >
> >>
> >> userspace performs EXT_QUERY_MMAP_BUF to get the mmap offset/cookie and length
> >> for each memory buffer.
> >>
> >> On EXT_QBUF, userspace doesn't need to fill membuf information. Should the
> >> mmap offset and length be filled by the kernel and returned to userspace here
> >> as well? I'm leaning towards: no.
> >
> > Yeah, based on my comment above, I think the answer should be no.
> >
> >>
> >> If the answer is no, then here is my proposal:
> >> ----------------------------------------------
> >>
> >> /* If MMAP, drivers decide how many memory buffers to allocate */
> >> int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_buffer *argp )
> >>
> >> /* Returns -EINVAL if not MMAP */
> >> int ioctl( int fd, VIDIOC_EXT_MMAP_QUERYBUF, struct v4l2_ext_mmap_querybuf *argp )
> >>
> >> /* userspace fills v4l2_ext_buffer.membufs if DMA-fd or Userptr, leave it zero for MMAP
> >>  * Should userspace also fill v4l2_ext_buffer.planes?
> >>  */
> >> int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
> >>
> >> /* v4l2_ext_buffer.membufs is set to zero by the driver */
> >> int ioctl( int fd, VIDIOC_EXT_DBUF, struct v4l2_ext_buffer *argp )
> >>
> >> (I omitted reserved fields below)
> >>
> >> struct v4l2_ext_create_buffers {
> >>         __u32                           index;
> >>         __u32                           count;
> >>         __u32                           memory;
> >>         __u32                           capabilities;
> >>         struct v4l2_ext_pix_format      format;
> >> };
> >>
> >> struct v4l2_ext_mmap_membuf {
> >>         __u32 offset;
> >>         __u32 length;
> >> }
> >>
> >> struct v4l2_ext_mmap_querybuf {
> >>         __u32 index;
> >>         struct v4l2_ext_mmap_membuf membufs[VIDEO_MAX_PLANES];
> >> }
> >>
> >> struct v4l2_ext_membuf {
> >>         __u32 memory;
> >>         union {
> >>                 __u64 userptr;
> >>                 __s32 dmabuf_fd;
> >>         } m;
> >>         // Can't we just remove the union and "memory" field, and the non-zero
> >>         // is the one we should use?
> >
> > I think that would lead to an equivalent result in this case. That
> > said, I'm not sure if there would be any significant enough benefit to
> > justify moving away from the current convention. Having the memory
> > field might also make the structure a bit less error prone, e.g.
> > resilient to missing memset().
> >
> >> };
> >>
> >> struct v4l2_ext_plane {
> >>         __u32 membuf_index;
> >>         __u32 offset;
> >>         __u32 bytesused;
> >> };
> >>
> >> struct v4l2_ext_buffer {
> >>         __u32 index;
> >>         __u32 type;
> >>         __u32 field;
> >>         __u32 sequence;
> >>         __u64 flags;
> >>         __u64 timestamp;
> >>         struct v4l2_ext_membuf membufs[VIDEO_MAX_PLANES];
> >>         struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> >
> > Do we actually need this split into membufs and planes here? After
> > all, all we want to pass to the kernel here is in what buffer the
> > plane is in.
>
> You are right, we don't.
>
> >
> > struct v4l2_ext_plane {
> >         __u32 memory;
>
> Should we design the API to allow a buffer to contain multiple memory planes
> of different types? Lets say one memplane is DMA-fd, the other is userptr.
> If the answer is yes, then struct v4l2_ext_create_buffers requires some changes.
> If not, then there is no need a "memory" field per memory plane in a buffer.
>

That's a good question. I haven't seen any practical need to do that.
Moreover, I suspect that the API might be going towards the DMA-buf
centric model, with DMA-buf heaps getting upstream acceptance, so
maybe we would be fine moving the memory field to the buffer struct
indeed.

> >         union {
> >                 __u32 membuf_index;
> >                 __u64 userptr;
> >                 __s32 dmabuf_fd;
> >         } m;
> >         __u32 offset;
> >         __u32 bytesused;
>
> We also need userptr_length right?

Is it actually needed? The length of the plane is determined by the
current format. I can only see as it being an extra sanity check
before accessing the process memory, but is it necessary? I think I
want to hear others's opinion on this.

[snip]

Best regards,
Tomasz

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-21  3:13                       ` Tomasz Figa
@ 2020-12-23 12:04                         ` Helen Koike
  2021-01-08 10:00                           ` Tomasz Figa
  0 siblings, 1 reply; 51+ messages in thread
From: Helen Koike @ 2020-12-23 12:04 UTC (permalink / raw)
  To: Tomasz Figa
  Cc: Hans Verkuil, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

Hi Tomasz,

On 12/21/20 12:13 AM, Tomasz Figa wrote:
> On Thu, Dec 17, 2020 at 10:20 PM Helen Koike <helen.koike@collabora.com> wrote:
>>
>> Hi Tomasz,
>>
>> Thanks for your comments, I have a few questions below.
>>
>> On 12/16/20 12:13 AM, Tomasz Figa wrote:
>>> On Tue, Dec 15, 2020 at 11:37 PM Helen Koike <helen.koike@collabora.com> wrote:
>>>>
>>>> Hi Tomasz,
>>>>
>>>> On 12/14/20 7:46 AM, Tomasz Figa wrote:
>>>>> On Fri, Dec 4, 2020 at 4:52 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Please see my 2 points below (about v4l2_ext_buffer and another about timestamp).
>>>>>>
>>>>>> On 12/3/20 12:11 PM, Hans Verkuil wrote:
>>>>>>> On 23/11/2020 18:40, Helen Koike wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 11/23/20 12:46 PM, Tomasz Figa wrote:
>>>>>>>>> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Hans,
>>>>>>>>>>
>>>>>>>>>> Thank you for your review.
>>>>>>>>>>
>>>>>>>>>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
>>>>>>>>>>> Hi Helen,
>>>>>>>>>>>
>>>>>>>>>>> Again I'm just reviewing the uAPI.
>>>>>>>>>>>
>>>>>>>>>>> On 04/08/2020 21:29, Helen Koike wrote:
> [snip]
>>>
>>>>
>>>> Output: userspace fills plane information, informing in which memory buffer each
>>>>         plane was placed (Or should this be pre-determined by the driver?)
>>>>
>>>> For MMAP
>>>> -----------------------
>>>> userspace performs EXT_CREATE_BUF ioctl to reserve a buffer "index" range in
>>>> that mode, to be used in EXT_QBUF and EXT_DQBUF
>>>>
>>>> Should the API allow userspace to select how many memory buffers it wants?
>>>> (maybe not)
>>>
>>> I think it does allow that - it accepts the v4l2_ext_format struct.
>>
>> hmmm, I thought v4l2_ext_format would describe color planes, and not memory planes.
>> Should it describe memory planes instead? Since planes are defined by the pixelformat.
>> But is this information relevant to ext_{set/get/try} format?
>>
> 
> Good point. I ended up assuming the current convention, where giving
> an M format would imply num_memory_planes == num_color_planes and
> non-M format num_memory_planes == 1. Sounds like we might want
> something like a flags field and that could have bits defined to
> select that. I think it would actually be useful for S_FMT as well,
> because that's what REQBUFS would use.

Would this flag select between memory and color planes?
I didn't understand how this flag would be useful to S_FMT, could you
please clarify?

Thanks
Helen

> 
>>>
>>>>
>>>> userspace performs EXT_QUERY_MMAP_BUF to get the mmap offset/cookie and length
>>>> for each memory buffer.
>>>>
>>>> On EXT_QBUF, userspace doesn't need to fill membuf information. Should the
>>>> mmap offset and length be filled by the kernel and returned to userspace here
>>>> as well? I'm leaning towards: no.
>>>
>>> Yeah, based on my comment above, I think the answer should be no.
>>>
>>>>
>>>> If the answer is no, then here is my proposal:
>>>> ----------------------------------------------
>>>>
>>>> /* If MMAP, drivers decide how many memory buffers to allocate */
>>>> int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_buffer *argp )
>>>>
>>>> /* Returns -EINVAL if not MMAP */
>>>> int ioctl( int fd, VIDIOC_EXT_MMAP_QUERYBUF, struct v4l2_ext_mmap_querybuf *argp )
>>>>
>>>> /* userspace fills v4l2_ext_buffer.membufs if DMA-fd or Userptr, leave it zero for MMAP
>>>>  * Should userspace also fill v4l2_ext_buffer.planes?
>>>>  */
>>>> int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
>>>>
>>>> /* v4l2_ext_buffer.membufs is set to zero by the driver */
>>>> int ioctl( int fd, VIDIOC_EXT_DBUF, struct v4l2_ext_buffer *argp )
>>>>
>>>> (I omitted reserved fields below)
>>>>
>>>> struct v4l2_ext_create_buffers {
>>>>         __u32                           index;
>>>>         __u32                           count;
>>>>         __u32                           memory;
>>>>         __u32                           capabilities;
>>>>         struct v4l2_ext_pix_format      format;
>>>> };
>>>>
>>>> struct v4l2_ext_mmap_membuf {
>>>>         __u32 offset;
>>>>         __u32 length;
>>>> }
>>>>
>>>> struct v4l2_ext_mmap_querybuf {
>>>>         __u32 index;
>>>>         struct v4l2_ext_mmap_membuf membufs[VIDEO_MAX_PLANES];
>>>> }
>>>>
>>>> struct v4l2_ext_membuf {
>>>>         __u32 memory;
>>>>         union {
>>>>                 __u64 userptr;
>>>>                 __s32 dmabuf_fd;
>>>>         } m;
>>>>         // Can't we just remove the union and "memory" field, and the non-zero
>>>>         // is the one we should use?
>>>
>>> I think that would lead to an equivalent result in this case. That
>>> said, I'm not sure if there would be any significant enough benefit to
>>> justify moving away from the current convention. Having the memory
>>> field might also make the structure a bit less error prone, e.g.
>>> resilient to missing memset().
>>>
>>>> };
>>>>
>>>> struct v4l2_ext_plane {
>>>>         __u32 membuf_index;
>>>>         __u32 offset;
>>>>         __u32 bytesused;
>>>> };
>>>>
>>>> struct v4l2_ext_buffer {
>>>>         __u32 index;
>>>>         __u32 type;
>>>>         __u32 field;
>>>>         __u32 sequence;
>>>>         __u64 flags;
>>>>         __u64 timestamp;
>>>>         struct v4l2_ext_membuf membufs[VIDEO_MAX_PLANES];
>>>>         struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
>>>
>>> Do we actually need this split into membufs and planes here? After
>>> all, all we want to pass to the kernel here is in what buffer the
>>> plane is in.
>>
>> You are right, we don't.
>>
>>>
>>> struct v4l2_ext_plane {
>>>         __u32 memory;
>>
>> Should we design the API to allow a buffer to contain multiple memory planes
>> of different types? Lets say one memplane is DMA-fd, the other is userptr.
>> If the answer is yes, then struct v4l2_ext_create_buffers requires some changes.
>> If not, then there is no need a "memory" field per memory plane in a buffer.
>>
> 
> That's a good question. I haven't seen any practical need to do that.
> Moreover, I suspect that the API might be going towards the DMA-buf
> centric model, with DMA-buf heaps getting upstream acceptance, so
> maybe we would be fine moving the memory field to the buffer struct
> indeed.
> 
>>>         union {
>>>                 __u32 membuf_index;
>>>                 __u64 userptr;
>>>                 __s32 dmabuf_fd;
>>>         } m;
>>>         __u32 offset;
>>>         __u32 bytesused;
>>
>> We also need userptr_length right?
> 
> Is it actually needed? The length of the plane is determined by the
> current format. I can only see as it being an extra sanity check
> before accessing the process memory, but is it necessary? I think I
> want to hear others's opinion on this.
> 
> [snip]
> 
> Best regards,
> Tomasz
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 2/7] media: v4l2: Add extended buffer operations
  2020-12-23 12:04                         ` Helen Koike
@ 2021-01-08 10:00                           ` Tomasz Figa
  0 siblings, 0 replies; 51+ messages in thread
From: Tomasz Figa @ 2021-01-08 10:00 UTC (permalink / raw)
  To: Helen Koike
  Cc: Hans Verkuil, Mauro Carvalho Chehab, Hans Verkuil,
	Laurent Pinchart, Sakari Ailus, Linux Media Mailing List,
	Boris Brezillon, Hirokazu Honda, Nicolas Dufresne, Brian Starkey,
	kernel, Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov

On Wed, Dec 23, 2020 at 9:04 PM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi Tomasz,
>
> On 12/21/20 12:13 AM, Tomasz Figa wrote:
> > On Thu, Dec 17, 2020 at 10:20 PM Helen Koike <helen.koike@collabora.com> wrote:
> >>
> >> Hi Tomasz,
> >>
> >> Thanks for your comments, I have a few questions below.
> >>
> >> On 12/16/20 12:13 AM, Tomasz Figa wrote:
> >>> On Tue, Dec 15, 2020 at 11:37 PM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>
> >>>> Hi Tomasz,
> >>>>
> >>>> On 12/14/20 7:46 AM, Tomasz Figa wrote:
> >>>>> On Fri, Dec 4, 2020 at 4:52 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> Please see my 2 points below (about v4l2_ext_buffer and another about timestamp).
> >>>>>>
> >>>>>> On 12/3/20 12:11 PM, Hans Verkuil wrote:
> >>>>>>> On 23/11/2020 18:40, Helen Koike wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 11/23/20 12:46 PM, Tomasz Figa wrote:
> >>>>>>>>> On Tue, Nov 24, 2020 at 12:08 AM Helen Koike <helen.koike@collabora.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi Hans,
> >>>>>>>>>>
> >>>>>>>>>> Thank you for your review.
> >>>>>>>>>>
> >>>>>>>>>> On 9/9/20 9:27 AM, Hans Verkuil wrote:
> >>>>>>>>>>> Hi Helen,
> >>>>>>>>>>>
> >>>>>>>>>>> Again I'm just reviewing the uAPI.
> >>>>>>>>>>>
> >>>>>>>>>>> On 04/08/2020 21:29, Helen Koike wrote:
> > [snip]
> >>>
> >>>>
> >>>> Output: userspace fills plane information, informing in which memory buffer each
> >>>>         plane was placed (Or should this be pre-determined by the driver?)
> >>>>
> >>>> For MMAP
> >>>> -----------------------
> >>>> userspace performs EXT_CREATE_BUF ioctl to reserve a buffer "index" range in
> >>>> that mode, to be used in EXT_QBUF and EXT_DQBUF
> >>>>
> >>>> Should the API allow userspace to select how many memory buffers it wants?
> >>>> (maybe not)
> >>>
> >>> I think it does allow that - it accepts the v4l2_ext_format struct.
> >>
> >> hmmm, I thought v4l2_ext_format would describe color planes, and not memory planes.
> >> Should it describe memory planes instead? Since planes are defined by the pixelformat.
> >> But is this information relevant to ext_{set/get/try} format?
> >>
> >
> > Good point. I ended up assuming the current convention, where giving
> > an M format would imply num_memory_planes == num_color_planes and
> > non-M format num_memory_planes == 1. Sounds like we might want
> > something like a flags field and that could have bits defined to
> > select that. I think it would actually be useful for S_FMT as well,
> > because that's what REQBUFS would use.
>
> Would this flag select between memory and color planes?
> I didn't understand how this flag would be useful to S_FMT, could you
> please clarify?

I mean a flag that decides the plane layout between the 2 possible
options (all planes in their own buffers at offsets 0 vs all planes in
one buffer one after another), rather than giving too much flexibility
for MMAP buffers, which isn't necessary any way, because DMABUF can be
used if more flexibility is needed.

Best regards,
Tomasz

>
> Thanks
> Helen
>
> >
> >>>
> >>>>
> >>>> userspace performs EXT_QUERY_MMAP_BUF to get the mmap offset/cookie and length
> >>>> for each memory buffer.
> >>>>
> >>>> On EXT_QBUF, userspace doesn't need to fill membuf information. Should the
> >>>> mmap offset and length be filled by the kernel and returned to userspace here
> >>>> as well? I'm leaning towards: no.
> >>>
> >>> Yeah, based on my comment above, I think the answer should be no.
> >>>
> >>>>
> >>>> If the answer is no, then here is my proposal:
> >>>> ----------------------------------------------
> >>>>
> >>>> /* If MMAP, drivers decide how many memory buffers to allocate */
> >>>> int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_buffer *argp )
> >>>>
> >>>> /* Returns -EINVAL if not MMAP */
> >>>> int ioctl( int fd, VIDIOC_EXT_MMAP_QUERYBUF, struct v4l2_ext_mmap_querybuf *argp )
> >>>>
> >>>> /* userspace fills v4l2_ext_buffer.membufs if DMA-fd or Userptr, leave it zero for MMAP
> >>>>  * Should userspace also fill v4l2_ext_buffer.planes?
> >>>>  */
> >>>> int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
> >>>>
> >>>> /* v4l2_ext_buffer.membufs is set to zero by the driver */
> >>>> int ioctl( int fd, VIDIOC_EXT_DBUF, struct v4l2_ext_buffer *argp )
> >>>>
> >>>> (I omitted reserved fields below)
> >>>>
> >>>> struct v4l2_ext_create_buffers {
> >>>>         __u32                           index;
> >>>>         __u32                           count;
> >>>>         __u32                           memory;
> >>>>         __u32                           capabilities;
> >>>>         struct v4l2_ext_pix_format      format;
> >>>> };
> >>>>
> >>>> struct v4l2_ext_mmap_membuf {
> >>>>         __u32 offset;
> >>>>         __u32 length;
> >>>> }
> >>>>
> >>>> struct v4l2_ext_mmap_querybuf {
> >>>>         __u32 index;
> >>>>         struct v4l2_ext_mmap_membuf membufs[VIDEO_MAX_PLANES];
> >>>> }
> >>>>
> >>>> struct v4l2_ext_membuf {
> >>>>         __u32 memory;
> >>>>         union {
> >>>>                 __u64 userptr;
> >>>>                 __s32 dmabuf_fd;
> >>>>         } m;
> >>>>         // Can't we just remove the union and "memory" field, and the non-zero
> >>>>         // is the one we should use?
> >>>
> >>> I think that would lead to an equivalent result in this case. That
> >>> said, I'm not sure if there would be any significant enough benefit to
> >>> justify moving away from the current convention. Having the memory
> >>> field might also make the structure a bit less error prone, e.g.
> >>> resilient to missing memset().
> >>>
> >>>> };
> >>>>
> >>>> struct v4l2_ext_plane {
> >>>>         __u32 membuf_index;
> >>>>         __u32 offset;
> >>>>         __u32 bytesused;
> >>>> };
> >>>>
> >>>> struct v4l2_ext_buffer {
> >>>>         __u32 index;
> >>>>         __u32 type;
> >>>>         __u32 field;
> >>>>         __u32 sequence;
> >>>>         __u64 flags;
> >>>>         __u64 timestamp;
> >>>>         struct v4l2_ext_membuf membufs[VIDEO_MAX_PLANES];
> >>>>         struct v4l2_ext_plane planes[VIDEO_MAX_PLANES];
> >>>
> >>> Do we actually need this split into membufs and planes here? After
> >>> all, all we want to pass to the kernel here is in what buffer the
> >>> plane is in.
> >>
> >> You are right, we don't.
> >>
> >>>
> >>> struct v4l2_ext_plane {
> >>>         __u32 memory;
> >>
> >> Should we design the API to allow a buffer to contain multiple memory planes
> >> of different types? Lets say one memplane is DMA-fd, the other is userptr.
> >> If the answer is yes, then struct v4l2_ext_create_buffers requires some changes.
> >> If not, then there is no need a "memory" field per memory plane in a buffer.
> >>
> >
> > That's a good question. I haven't seen any practical need to do that.
> > Moreover, I suspect that the API might be going towards the DMA-buf
> > centric model, with DMA-buf heaps getting upstream acceptance, so
> > maybe we would be fine moving the memory field to the buffer struct
> > indeed.
> >
> >>>         union {
> >>>                 __u32 membuf_index;
> >>>                 __u64 userptr;
> >>>                 __s32 dmabuf_fd;
> >>>         } m;
> >>>         __u32 offset;
> >>>         __u32 bytesused;
> >>
> >> We also need userptr_length right?
> >
> > Is it actually needed? The length of the plane is determined by the
> > current format. I can only see as it being an extra sanity check
> > before accessing the process memory, but is it necessary? I think I
> > want to hear others's opinion on this.
> >
> > [snip]
> >
> > Best regards,
> > Tomasz
> >

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v5 7/7] media: docs: add documentation for the Extended API
  2020-11-20 13:20         ` Hans Verkuil
@ 2021-01-14 18:04           ` Helen Koike
  0 siblings, 0 replies; 51+ messages in thread
From: Helen Koike @ 2021-01-14 18:04 UTC (permalink / raw)
  To: Hans Verkuil, Tomasz Figa
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Laurent Pinchart,
	Sakari Ailus, Linux Media Mailing List, Hirokazu Honda,
	Nicolas Dufresne, Brian Starkey, kernel, Boris Brezillon,
	Neil Armstrong, Linux Kernel Mailing List, Fritz Koenig,
	Maxime Jourdan, Stanimir Varbanov



On 11/20/20 10:20 AM, Hans Verkuil wrote:
> On 20/11/2020 13:40, Tomasz Figa wrote:
>> On Fri, Nov 20, 2020 at 9:24 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>>
>>> On 20/11/2020 12:06, Tomasz Figa wrote:z
>>>> Hi Helen,
>>>>
>>>> On Tue, Aug 04, 2020 at 04:29:39PM -0300, Helen Koike wrote:
>>>>> Add documentation and update references in current documentation for the
>>>>> Extended API.
>>>>>
>>>>
>>>> Thank you for the patch. Please see my comments inline.
>>>>
>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>> ---
>>>>> Changes in v5:
>>>>> - new patch
>>>>>
>>>>>  .../userspace-api/media/v4l/buffer.rst        |   5 +
>>>>>  .../userspace-api/media/v4l/common.rst        |   1 +
>>>>>  .../userspace-api/media/v4l/dev-capture.rst   |   5 +
>>>>>  .../userspace-api/media/v4l/dev-output.rst    |   5 +
>>>>>  .../userspace-api/media/v4l/ext-api.rst       | 107 +++++++++
>>>>>  .../userspace-api/media/v4l/format.rst        |  16 +-
>>>>>  .../userspace-api/media/v4l/user-func.rst     |   5 +
>>>>>  .../media/v4l/vidioc-ext-create-bufs.rst      |  95 ++++++++
>>>>>  .../media/v4l/vidioc-ext-prepare-buf.rst      |  62 ++++++
>>>>>  .../media/v4l/vidioc-ext-qbuf.rst             | 204 ++++++++++++++++++
>>>>>  .../media/v4l/vidioc-ext-querybuf.rst         |  79 +++++++
>>>>>  .../media/v4l/vidioc-g-ext-pix-fmt.rst        | 117 ++++++++++
>>>>>  12 files changed, 697 insertions(+), 4 deletions(-)
>>>>>  create mode 100644 Documentation/userspace-api/media/v4l/ext-api.rst
>>>>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>>>>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>>>>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>>>>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>>>>>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>>>>>
>>>>> diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst
>>>>> index 57e752aaf414a..c832bedd64e4c 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/buffer.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/buffer.rst
>>>>> @@ -27,6 +27,11 @@ such as pointers and sizes for each plane, are stored in
>>>>>  struct :c:type:`v4l2_plane` instead. In that case,
>>>>>  struct :c:type:`v4l2_buffer` contains an array of plane structures.
>>>>>
>>>>> +.. note::
>>>>> +
>>>>> +    The :ref:`ext_api` version can also be used, and it is
>>>>> +    preferable when applicable.
>>>>
>>>> Would rephrasing this as below making a bit more definitive?
>>>>
>>>>       For modern applications, these operations are replaced by their
>>>>       :ref:`ext_api` counterparts, which should be used instead.
>>>
>>> You can't say that, since especially in the beginning userspace will be running
>>> on older kernels that do not support this.
>>>
>>> This will work: "should be used instead, if supported by the driver."
>>>
>>
>> With the wrappers that the patches provide, all drivers would support
>> the new API, so this boils down to the kernel version only, not
>> specific drivers.
>>
>> Agreed, though, that application developers must be made aware that
>> the new API is only available in new kernels and old API must be used
>> for compatibility with old kernels.
> 
> Is the conversion layer fully independent of vb2? We still have older
> drivers that do not use vb2, and I'm not sure if the extended buffer
> operations work with those drivers as well. (Sorry, it's been a while
> since I last looked at this series, so I can't remember).

The conversion layer should be independent of vb2 (please check v6, I think
it is cleaner).

> 
>>
>>>>
>>>>> +
>>>>>  Dequeued video buffers come with timestamps. The driver decides at which
>>>>>  part of the frame and with which clock the timestamp is taken. Please
>>>>>  see flags in the masks ``V4L2_BUF_FLAG_TIMESTAMP_MASK`` and
>>>>> diff --git a/Documentation/userspace-api/media/v4l/common.rst b/Documentation/userspace-api/media/v4l/common.rst
>>>>> index 7d81c58a13cd7..3430e0bdad667 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/common.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/common.rst
>>>>> @@ -59,6 +59,7 @@ applicable to all devices.
>>>>>      ext-ctrls-detect
>>>>>      fourcc
>>>>>      format
>>>>> +    ext-api
>>>>>      planar-apis
>>>>>      selection-api
>>>>>      crop
>>>>> diff --git a/Documentation/userspace-api/media/v4l/dev-capture.rst b/Documentation/userspace-api/media/v4l/dev-capture.rst
>>>>> index 44d3094093ab6..5077639787d92 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/dev-capture.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/dev-capture.rst
>>>>> @@ -102,6 +102,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>>>>>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>>>>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>>>>>
>>>>> +.. note::
>>>>> +
>>>>> +    The :ref:`ext_api` version can also be used, and it is
>>>>> +    preferable when applicable.
>>>>> +
>>>>>
>>>>>  Reading Images
>>>>>  ==============
>>>>> diff --git a/Documentation/userspace-api/media/v4l/dev-output.rst b/Documentation/userspace-api/media/v4l/dev-output.rst
>>>>> index e4f2a1d8b0fc7..f8f40c708e49f 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/dev-output.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/dev-output.rst
>>>>> @@ -99,6 +99,11 @@ and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl, even if :ref:`VIDIOC_S_FMT <VIDIOC
>>>>>  requests and always returns default parameters as :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` does.
>>>>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is optional.
>>>>>
>>>>> +.. note::
>>>>> +
>>>>> +    The :ref:`ext_api` version can also be used, and it is
>>>>> +    preferable when applicable.
>>>>> +
>>>>>
>>>>>  Writing Images
>>>>>  ==============
>>>>> diff --git a/Documentation/userspace-api/media/v4l/ext-api.rst b/Documentation/userspace-api/media/v4l/ext-api.rst
>>>>> new file mode 100644
>>>>> index 0000000000000..da2a82960d22f
>>>>> --- /dev/null
>>>>> +++ b/Documentation/userspace-api/media/v4l/ext-api.rst
>>>>> @@ -0,0 +1,107 @@
>>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>>> +.. document under the terms of the GNU Free Documentation License,
>>>>> +.. Version 1.1 or any later version published by the Free Software
>>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>>> +..
>>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>>> +
>>>>> +.. _ext_api:
>>>>> +
>>>>> +*************
>>>>> +Extendend API
>>>>> +*************
>>>>> +
>>>>> +Introduction
>>>>> +============
>>>>> +
>>>>> +The Extended Format API was conceived to extend V4L2 capabilities and
>>>>> +to simplify certain mechanisms.
>>>>> +It unifies single- vs multi- planar handling,
>>>>> +brings the concept of pixelformat + modifiers, allows planes to be placed
>>>>> +in any offset inside a buffer and let userspace to change colorspace
>>>>> +attributes if supported by the driver.
>>>>
>>>> How about simplifying this paragraph a bit:
>>>>
>>>> The Extended Format API unifies single- and multi- planar format handling
>>>> and provides a base for further functional extensions, such as format
>>>> modifiers, arbitrary plane offsets, colorspace configuration.
>>>
>>> Actually, I prefer Helen's version.
>>>
>>
>> To me it's a lot of text to read to get the same information, but I
>> won't insist.
>>
>>>>
>>>>> +
>>>>> +Data format negotiation and buffer handling works very similar to the classical
>>>>> +version, thus in this document we'll just mention the main differences.
>>>>
>>>> nit: "we will" or maybe even just "we mention only the main differences".

Applied for v6.

>>>>
>>>>> +
>>>>> +Data Format Negotiation
>>>>> +=======================
>>>>> +
>>>>> +The API replaces the classical ioctls:
>>>>> +
>>>>> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>`, :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`,
>>>>> +:ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>`
>>>>> +(with :c:type:`v4l2_format` as the main parameter).
>>>>> +
>>>>> +By the extended versions:
>>>>> +
>>>>> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
>>>>> +:ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`,
>>>>> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
>>>>> +(with :c:type:`v4l2_ext_pix_format` as the main parameter).
>>>>> +
>>>>> +For CAPTURE and OUTPUT queues only.
>>>>> +
>>>>> +The ``type`` field of struct :c:type:`v4l2_ext_pix_format` only accepts
>>>>> +``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or ``V4L2_BUF_TYPE_VIDEO_OUTPUT``, and it
>>>>> +supports multiplanar format through a combination of ``pixelfomat`` and
>>>>> +``modifier`` fields.
>>>>> +
>>>>> +Only the single-planar variants of the pixel formats are valid in the
>>>>> +``pixelformat`` field.
>>>>
>>>> Some formats have only multi-planar variants, like NV12MT. What should we
>>>> do about them? Since we have to keep those defines to support the legacy
>>>> API anyway, maybe we can just accept both variants and treat them equally?
>>>
>>> Helen mentioned that as well in a reply to patch 1/7. I'll have a think about
>>> that myself and reply to that post because this is a good question.
>>>
>>
>> I'm thinking about it as well. :)
>>
>>>>
>>>>> +To support multi-planar, a modifier should be set in the ``modifier`` field to
>>>>> +provide more information regarding the memory layout of pixels and number of
>>>>> +planes.
>>>>> +
>>>>> +The ``plane_fmt`` field is an array which holds information by plane using
>>>>> +the :c:type:`v4l2_plane_ext_pix_format`. When this struct is filled, its
>>>>> +``sizeimage`` field should be non-zero for all valid planes for a given
>>>>> +``pixelformat`` + ``modifier`` combination, and zeroed for the invalid ones.
>>>>> +
>>>>> +Colospace attributes are not read-only as in the classical version, i.e, they
>>>>> +can be set by application and drivers will adjust accordingly depending on what
>>>>> +is supported by the underlying hardware.
>>>>> +
>>>>> +Buffers
>>>>> +=======
>>>>> +
>>>>> +The API replaces the classical ioctls:
>>>>> +
>>>>> +:ref:`VIDIOC_CREATE_BUFS <VIDIOC_CREATE_BUFS>`
>>>>> +(with :c:type:`v4l2_create_buffers` as the main parameter),
>>>>> +:ref:`VIDIOC_QUERYBUF <VIDIOC_QUERYBUF>`, :ref:`VIDIOC_QBUF <VIDIOC_QBUF>`,
>>>>> +:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`, :ref:`VIDIOC_PREPARE_BUF <VIDIOC_PREPARE_BUF>`
>>>>> +(with :c:type:`v4l2_buffer` as the main parameter)
>>>>> +
>>>>> +By the extended versions:
>>>>> +
>>>>> +:ref:`VIDIOC_EXT_CREATE_BUFS <VIDIOC_EXT_CREATE_BUFS>`
>>>>> +(with :c:type:`v4l2_ext_create_buffers` as the main parameter),
>>>>> +:ref:`VIDIOC_EXT_QUERYBUF <VIDIOC_EXT_QUERYBUF>`,
>>>>> +:ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>`,
>>>>> +:ref:`VIDIOC_EXT_DQBUF <VIDIOC_EXT_QBUF>`,
>>>>> +:ref:`VIDIOC_EXT_PREPARE_BUF <VIDIOC_EXT_PREPARE_BUF>`
>>>>> +(with :c:type:`v4l2_ext_buffer` as the main parameter)
>>>>> +
>>>>> +The basic difference between :c:type:`v4l2_create_buffers` and
>>>>> +:c:type:`v4l2_ext_create_buffers` is that the later have a
>>>>> +:c:type:`v4l2_ext_pix_format` as the type of the ``format`` field.
>>>>> +
>>>>> +Comparing :c:type:`v4l2_ext_buffer` with :c:type:`v4l2_buffer`, in the
>>>>> +extended version there is a unification of the single-/multi- planar handling
>>>>> +through the ``planes`` field of type :c:type:`v4l2_ext_plane`.
>>>>> +
>>>>> +The :c:type:`v4l2_ext_plane also allows planes to be placed in a given offset
>>>>> +inside a buffer.
>>>>> +Planes can be placed in different locations inside the same buffer, or in
>>>>> +different buffers.
>>>>> +
>>>>> +
>>>>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>>>>> +   :functions: v4l2_ext_plane
>>>>> +
>>>>> +
>>>>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>>>>> +   :functions: v4l2_ext_buffer
>>>>> diff --git a/Documentation/userspace-api/media/v4l/format.rst b/Documentation/userspace-api/media/v4l/format.rst
>>>>> index e47fc0505727c..b96d26f26793c 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/format.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/format.rst
>>>>> @@ -28,13 +28,19 @@ format and the driver selects and reports the best the hardware can do
>>>>>  to satisfy the request. Of course applications can also just query the
>>>>>  current selection.
>>>>>
>>>>> -A single mechanism exists to negotiate all data formats using the
>>>>> -aggregate struct :c:type:`v4l2_format` and the
>>>>> +There are two mechanism to negociate data formats:
>>>>
>>>> negotiate

Ack, updated for v6.

>>>>
>>>>> +
>>>>> +The first one is using the aggregate struct :c:type:`v4l2_format` and the
>>>>>  :ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and
>>>>>  :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctls. Additionally the
>>>>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` ioctl can be used to examine
>>>>>  what the hardware *could* do, without actually selecting a new data
>>>>> -format. The data formats supported by the V4L2 API are covered in the
>>>>> +format.
>>>>> +
>>>>> +The second is through the :ref:`ext_api`, please refer to its documentation
>>>>> +for more information.
>>>>> +
>>>>
>>>> Should we reverse the order to prefer the new API?

Ack, updated for v6.

>>>>
>>>>> +The data formats supported by the V4L2 API are covered in the
>>>>>  respective device section in :ref:`devices`. For a closer look at
>>>>>  image formats see :ref:`pixfmt`.
>>>>>
>>>>> @@ -71,7 +77,9 @@ earlier versions of V4L2. Switching the logical stream or returning into
>>>>>  *may* support a switch using :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>`.
>>>>>
>>>>>  All drivers exchanging data with applications must support the
>>>>> -:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl. Implementation of the
>>>>> +:ref:`VIDIOC_G_FMT <VIDIOC_G_FMT>` and :ref:`VIDIOC_S_FMT <VIDIOC_G_FMT>` ioctl
>>>>> +or the Extended respective versions (TODO: link).
>>>>
>>>> TODO? :)

Thanks!

>>>>
>>>>> +Implementation of the
>>>>>  :ref:`VIDIOC_TRY_FMT <VIDIOC_G_FMT>` is highly recommended but optional.
>>>>>
>>>>>
>>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
>>>>> index bf77c842718e5..67b254812791c 100644
>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst
>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
>>>>> @@ -20,6 +20,7 @@ Function Reference
>>>>>      func-close
>>>>>      func-ioctl
>>>>>      vidioc-create-bufs
>>>>> +    vidioc-ext-create-bufs
>>>>>      vidioc-cropcap
>>>>>      vidioc-dbg-g-chip-info
>>>>>      vidioc-dbg-g-register
>>>>> @@ -48,6 +49,7 @@ Function Reference
>>>>>      vidioc-g-ext-ctrls
>>>>>      vidioc-g-fbuf
>>>>>      vidioc-g-fmt
>>>>> +    vidioc-g-ext-pix-fmt
>>>>>      vidioc-g-frequency
>>>>>      vidioc-g-input
>>>>>      vidioc-g-jpegcomp
>>>>> @@ -62,8 +64,11 @@ Function Reference
>>>>>      vidioc-log-status
>>>>>      vidioc-overlay
>>>>>      vidioc-prepare-buf
>>>>> +    vidioc-ext-prepare-buf
>>>>>      vidioc-qbuf
>>>>> +    vidioc-ext-qbuf
>>>>>      vidioc-querybuf
>>>>> +    vidioc-ext-querybuf
>>>>>      vidioc-querycap
>>>>>      vidioc-queryctrl
>>>>>      vidioc-query-dv-timings
>>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>>>>> new file mode 100644
>>>>> index 0000000000000..67f2c541e4d79
>>>>> --- /dev/null
>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-create-bufs.rst
>>>>> @@ -0,0 +1,95 @@
>>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>>> +.. document under the terms of the GNU Free Documentation License,
>>>>> +.. Version 1.1 or any later version published by the Free Software
>>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>>> +..
>>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>>> +
>>>>
>>>> TODO? :)

Thanks!

>>>>
>>>>> +.. _VIDIOC_EXT_CREATE_BUFS:
>>>>> +
>>>>> +****************************
>>>>> +ioctl VIDIOC_EXT_CREATE_BUFS
>>>>> +****************************
>>>>> +
>>>>> +Name
>>>>> +====
>>>>> +
>>>>> +VIDIOC_EXT_CREATE_BUFS - Create buffers for Memory Mapped or User Pointer or DMA Buffer I/O
>>>>> +
>>>>> +
>>>>> +Synopsis
>>>>> +========
>>>>> +
>>>>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_CREATE_BUFS, struct v4l2_ext_create_buffers *argp )
>>>>> +    :name: VIDIOC_EXT_CREATE_BUFS
>>>>> +
>>>>> +
>>>>> +Arguments
>>>>> +=========
>>>>> +
>>>>> +``fd``
>>>>> +    File descriptor returned by :ref:`open() <func-open>`.
>>>>> +
>>>>> +``argp``
>>>>> +    Pointer to struct :c:type:`v4l2_ext_create_buffers`.
>>>>> +
>>>>> +
>>>>> +Description
>>>>> +===========
>>>>> +
>>>>> +This ioctl is used to create buffers for :ref:`memory mapped <mmap>`
>>>>> +or :ref:`user pointer <userp>` or :ref:`DMA buffer <dmabuf>` I/O.
>>>>> +This ioctl can be called multiple times to
>>>>> +create buffers of different sizes.
>>>>> +
>>>>> +To allocate the device buffers applications must initialize the relevant
>>>>> +fields of the struct :c:type:`v4l2_ext_create_buffers` structure. The
>>>>> +``count`` field must be set to the number of requested buffers, the
>>>>> +``memory`` field specifies the requested I/O method and the ``reserved``
>>>>> +array must be zeroed.
>>>>> +
>>>>> +The ``format`` field specifies the image format that the buffers must be
>>>>> +able to handle. The application has to fill in this struct
>>>>> +:c:type:`v4l2_format`. Usually this will be done using the
>>>>> +:ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` or
>>>>> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctls to ensure that the
>>>>> +requested format is supported by the driver.
>>>>> +The driver may return an error if the size(s) are not supported by the
>>>>> +hardware (usually because they are too small).
>>>>> +
>>>>> +The driver can create a memory buffer per plane, or a single memory buffer
>>>>> +containing all the planes, with a minimum size according to the size
>>>>> +defined by the ``format.plane_fmt[i].sizeimage`` field.
>>>>> +Usually if the ``format.plane_fmt[i].sizeimage``
>>>>> +field is less than the minimum required for the given format, then an
>>>>> +error will be returned since drivers will typically not allow this. If
>>>>> +it is larger, then the value will be used as-is. In other words, the
>>>>> +driver may reject the requested size, but if it is accepted the driver
>>>>> +will use it unchanged.
>>>>> +
>>>>> +When the ioctl is called with a pointer to this structure the driver
>>>>> +will attempt to allocate up to the requested number of buffers and store
>>>>> +the actual number allocated and the starting index in the ``count`` and
>>>>> +the ``index`` fields respectively. On return ``count`` can be smaller
>>>>> +than the number requested.
>>>>
>>>> The 'capabilities' and 'flags' fields seem to go undocumented.

I removed create_bufs for v6 for now, since it is only useful for MMAP, and if
we are going torwards a DMA-fd centric API, then this is low priority and we
can define it later. Please check v6 and let me know what you think.

>>>>
>>>>> +
>>>>> +
>>>>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>>>>> +   :functions: v4l2_ext_create_buffers
>>>>> +
>>>>> +
>>>>> +Return Value
>>>>> +============
>>>>> +
>>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>>> +appropriately. The generic error codes are described at the
>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>>> +
>>>>> +ENOMEM
>>>>> +    No memory to allocate buffers for :ref:`memory mapped <mmap>` I/O.
>>>>> +
>>>>> +EINVAL
>>>>> +    The buffer type (``format.type`` field), requested I/O method
>>>>> +    (``memory``) or format (``format`` field) is not valid.
>>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>>>>> new file mode 100644
>>>>> index 0000000000000..88e4b57121254
>>>>> --- /dev/null
>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-prepare-buf.rst
>>>>> @@ -0,0 +1,62 @@
>>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>>> +.. document under the terms of the GNU Free Documentation License,
>>>>> +.. Version 1.1 or any later version published by the Free Software
>>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>>> +..
>>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>>> +
>>>>> +.. _VIDIOC_EXT_PREPARE_BUF:
>>>>> +
>>>>> +****************************
>>>>> +ioctl VIDIOC_EXT_PREPARE_BUF
>>>>> +****************************
>>>>> +
>>>>> +Name
>>>>> +====
>>>>> +
>>>>> +VIDIOC_EXT_PREPARE_BUF - Prepare a buffer for I/O
>>>>> +
>>>>> +
>>>>> +Synopsis
>>>>> +========
>>>>> +
>>>>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_PREPARE_BUF, struct v4l2_ext_buffer *argp )
>>>>> +    :name: VIDIOC_EXT_PREPARE_BUF
>>>>> +
>>>>> +
>>>>> +Arguments
>>>>> +=========
>>>>> +
>>>>> +``fd``
>>>>> +    File descriptor returned by :ref:`open() <func-open>`.
>>>>> +
>>>>> +``argp``
>>>>> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
>>>>> +
>>>>> +
>>>>> +Description
>>>>> +===========
>>>>> +
>>>>> +Applications can optionally call the :ref:`VIDIOC_EXT_PREPARE_BUF` ioctl to
>>>>> +pass ownership of the buffer to the driver before actually enqueuing it,
>>>>> +using the :ref:`VIDIOC_EXT_QBUF <VIDIOC_EXT_QBUF>` ioctl, and to prepare it for future I/O. Such
>>>>> +preparations may include cache invalidation or cleaning. Performing them
>>>>> +in advance saves time during the actual I/O.
>>>>> +
>>>>> +
>>>>> +Return Value
>>>>> +============
>>>>> +
>>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>>> +appropriately. The generic error codes are described at the
>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>>> +
>>>>> +EBUSY
>>>>> +    File I/O is in progress.
>>>>> +
>>>>> +EINVAL
>>>>> +    The buffer ``type`` is not supported, or the ``index`` is out of
>>>>> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
>>>>> +    ``length`` are invalid.
>>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>>>>> new file mode 100644
>>>>> index 0000000000000..083e106cf6bad
>>>>> --- /dev/null
>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-qbuf.rst
>>>>> @@ -0,0 +1,204 @@
>>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>>> +.. document under the terms of the GNU Free Documentation License,
>>>>> +.. Version 1.1 or any later version published by the Free Software
>>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>>> +..
>>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>>> +
>>>>> +.. _VIDIOC_EXT_QBUF:
>>>>> +
>>>>> +***************************************
>>>>> +ioctl VIDIOC_EXT_QBUF, VIDIOC_EXT_DQBUF
>>>>> +***************************************
>>>>> +
>>>>> +Name
>>>>> +====
>>>>> +
>>>>> +VIDIOC_EXT_QBUF - VIDIOC_EXT_DQBUF - Exchange a buffer with the driver
>>>>> +
>>>>> +
>>>>> +Synopsis
>>>>> +========
>>>>> +
>>>>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QBUF, struct v4l2_ext_buffer *argp )
>>>>> +    :name: VIDIOC_EXT_QBUF
>>>>> +
>>>>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_DQBUF, struct v4l2_ext_buffer *argp )
>>>>> +    :name: VIDIOC_EXT_DQBUF
>>>>> +
>>>>> +
>>>>> +Arguments
>>>>> +=========
>>>>> +
>>>>> +``fd``
>>>>> +    File descriptor returned by :ref:`open() <func-open>`.
>>>>> +
>>>>> +``argp``
>>>>> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
>>>>> +
>>>>> +
>>>>> +Description
>>>>> +===========
>>>>> +
>>>>> +Applications call the ``VIDIOC_EXT_QBUF`` ioctl to enqueue an empty
>>>>> +(capturing) or filled (output) buffer in the driver's incoming queue.
>>>>> +The semantics depend on the selected I/O method.
>>>>> +
>>>>> +To enqueue a buffer applications set the ``type`` field of a struct
>>>>> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
>>>>> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``.
>>>>> +Applications must also set the ``index`` field. Valid index numbers
>>>>> +range from zero to the number of buffers allocated with
>>>>> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
>>>>> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
>>>>> +one. The contents of the struct :c:type:`v4l2_ext_buffer` returned
>>>>
>>>> I think this might not be entirely accurate. One can call
>>>> VIDIOC_EXT_CREATE_BUFS multiple times. Moreover, I don't think we've
>>>> prohibited VIDIOC_REQBUFS for the ext API, or have we?

As per above, I removed VIDIOC_EXT_CREATE_BUFS for v6.

>>>>
>>>>> +by a :ref:`VIDIOC_EXT_QUERYBUF` ioctl will do as well.
>>>>> +When the buffer is intended for output (``type`` is
>>>>> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``) applications must also initialize the
>>>>> +``timestamp`` fields and the ``planes[i].plane_length`` for each valid plane,
>>>>> +and invalid ones must be set as zero.
>>>>> +see :ref:`buffer` for details. Applications must also set ``flags`` to 0. The
>>>>> +``reserved`` field must be set to 0.
>>>>> +
>>>>> +To enqueue a :ref:`memory mapped <mmap>` buffer applications set the
>>>>> +``planes[i].memory`` field to ``V4L2_MEMORY_MMAP`` in all the valid planes,
>>>>> +and invalid ones must be set as zero.
>>>>> +When ``VIDIOC_EXT_QBUF`` is called
>>>>> +with a pointer to this structure the driver sets the
>>>>> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_QUEUED`` flags and clears
>>>>> +the ``V4L2_BUF_FLAG_DONE`` flag in the ``flags`` field, or it returns an
>>>>> +``EINVAL`` error code.
>>>>> +
>>>>> +To enqueue a :ref:`user pointer <userp>` buffer applications set the
>>>>> +``planes[i].memory`` field to ``V4L2_MEMORY_USERPTR`` in all the valid planes,
>>>>> +and invalid ones must be set as zero, the ``planes[i].m.userptr`` field to the
>>>>> +address of the buffer,``planes[i].buffer_length`` to the size of the memory
>>>>> +buffer, ``planes[i].plane_length`` to the size that should be used by the plane,
>>>>> +and ``planes[i].offset`` of the plane in the memory buffer.
>>>>
>>>> Does offset make sense here if an arbitrary pointer can be given for the
>>>> start of the plane?

You are right, it doesn't, but offset can be set to non-zero in dqueue time,
in case drivers want to skip a header in the buffer.
Make sense?

>>>>
>>>>> +
>>>>> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure
>>>>> +the driver sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
>>>>> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
>>>>> +``flags`` field, or it returns an error code. This ioctl locks the
>>>>
>>>> This seems to be a copy of the text few lines above.

It is not quite the same, the flags that are set or cleared are slighly different.
The one a few lines above has V4L2_BUF_FLAG_MAPPED set, and here it is cleared.
Maybe we can show a table instead, showing which ones are set and which are
cleared, to make it easier to read.

>>>>
>>>>> +memory pages of the buffer in physical memory, they cannot be swapped
>>>>> +out to disk. Buffers remain locked until dequeued, until the
>>>>> +:ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
>>>>> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
>>>>> +device is closed.
>>>>
>>>> Why would CREATE_BUFS affect existing buffers?

This was a mistake, fixed in v6.

>>>>
>>>>> +
>>>>> +To enqueue a :ref:`DMABUF <dmabuf>` buffer applications set the
>>>>> +``planes[i].memory`` field to ``V4L2_MEMORY_DMABUF`` in all the valid planes,
>>>>> +and invalid ones must be set as zero, the ``planes[i].m.fd`` field to a
>>>>> +file descriptor associated with a DMABUF buffer.,``planes[i].buffer_length`` to
>>>>> +the size of the memory buffer, ``planes[i].plane_length`` to the size that
>>>>
>>>> What is the buffer_length field needed for? DMA-bufs by definition are
>>>> kernel objects and the kernel should already know their sizes.
>>>
>>> See this discussion:
>>>
>>> https://patchwork.linuxtv.org/project/linux-media/patch/gh5kef5bkeel3o6b2dkgc2dfagu9klj4c0@4ax.com/
>>>
>>
>> I briefly read through it and I don't see what the conclusion was.
>> Generally I'd still prefer if the need to manually specify the buffer
>> size was eliminated, because that imposes unnecessary burden on the
>> userspace for the majority of cases. It could still remain valid for
>> userspace that wants to restrict the buffer size as it was mentioned
>> in that thread, though.
> 
> That's really the open issue: if userspace sets length to 0, then it should
> always be replaced by the dmabuf size, that's a no-brainer. But if it is non-zero,
> what do you do then? Your input would be appreciated.

I removed the length field for v6, please check.

> 
>>
>>>>
>>>>> +should be used by the plane, and ``planes[i].offset`` of the plane in the memory buffer.
>>>>> +When ``VIDIOC_EXT_QBUF`` is called with a pointer to this structure the driver
>>>>> +sets the ``V4L2_BUF_FLAG_QUEUED`` flag and clears the
>>>>> +``V4L2_BUF_FLAG_MAPPED`` and ``V4L2_BUF_FLAG_DONE`` flags in the
>>>>> +``flags`` field, or it returns an error code. This ioctl locks the
>>>>
>>>> Ditto.

Ditto.

>>>>
>>>>> +buffer. Locking a buffer means passing it to a driver for a hardware
>>>>> +access (usually DMA). If an application accesses (reads/writes) a locked
>>>>> +buffer then the result is undefined. Buffers remain locked until
>>>>> +dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
>>>>> +:ref:`VIDIOC_EXT_CREATE_BUFS` ioctl is called, or until the
>>>>> +device is closed.
>>>>
>>>> Ditto.

Ditto.

>>>>
>>>>> +
>>>>> +The ``request_fd`` field can be used with the ``VIDIOC_EXT_QBUF`` ioctl to specify
>>>>> +the file descriptor of a :ref:`request <media-request-api>`, if requests are
>>>>> +in use. Setting it means that the buffer will not be passed to the driver
>>>>> +until the request itself is queued. Also, the driver will apply any
>>>>> +settings associated with the request for this buffer. This field will
>>>>> +be ignored unless the ``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
>>>>> +If the device does not support requests, then ``EBADR`` will be returned.
>>>>> +If requests are supported but an invalid request file descriptor is given,
>>>>> +then ``EINVAL`` will be returned.
>>>>> +
>>>>> +.. caution::
>>>>> +   It is not allowed to mix queuing requests with queuing buffers directly.
>>>>> +   ``EBUSY`` will be returned if the first buffer was queued directly and
>>>>> +   then the application tries to queue a request, or vice versa. After
>>>>> +   closing the file descriptor, calling
>>>>> +   :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or calling :ref:`VIDIOC_EXT_CREATE_BUFS`
>>>>> +   the check for this will be reset.
>>>>> +
>>>>> +   For :ref:`memory-to-memory devices <mem2mem>` you can specify the
>>>>> +   ``request_fd`` only for output buffers, not for capture buffers. Attempting
>>>>> +   to specify this for a capture buffer will result in an ``EBADR`` error.
>>>>> +
>>>>> +Applications call the ``VIDIOC_EXT_DQBUF`` ioctl to dequeue a filled
>>>>> +(capturing) or displayed (output) buffer from the driver's outgoing
>>>>
>>>> "displayed" is quite inaccurate given that the majority of use cases for
>>>> OUTPUT in V4L2 now are m2m devices. Maybe "consumed"?

Right, I copy&pasted from vidioc-qbuf.rst, so I guess we should update there too.

>>>>
>>>>> +queue. They just set the ``type``, ``planes[i].memory`` and ``reserved`` fields of
>>>>
>>>> Why does the application have to set the memory field? It should be already
>>>> known by the kernel which owns the buffer at the given time.

Ack, does this apply to EXT_QBUF too? Since userspace needs to switch the memory type
in advance with reqbufs, then the kernel already knows the memory type at queue time
too.

>>>>
>>>>> +a struct :c:type:`v4l2_ext_buffer` as above, when
>>>>> +``VIDIOC_EXT_DQBUF`` is called with a pointer to this structure the driver
>>>>> +fills the remaining fields or returns an error code. The driver may also
>>>>> +set ``V4L2_BUF_FLAG_ERROR`` in the ``flags`` field. It indicates a
>>>>> +non-critical (recoverable) streaming error. In such case the application
>>>>> +may continue as normal, but should be aware that data in the dequeued
>>>>> +buffer might be corrupted. When using the multi-planar API, the planes
>>>>> +array must be passed in as well.
>>>>> +
>>>>> +If the application sets the ``memory`` field to ``V4L2_MEMORY_DMABUF`` to
>>>>> +dequeue a :ref:`DMABUF <dmabuf>` buffer, the driver fills the ``m.fd`` field
>>>>> +with a file descriptor numerically the same as the one given to ``VIDIOC_EXT_QBUF``
>>>>> +when the buffer was enqueued. No new file descriptor is created at dequeue time
>>>>> +and the value is only for the application convenience.
>>>>
>>>> Honestly, I don't like this aspect of the current API. It kind of implies
>>>> that the value returned is valid, but the file descriptor might have been
>>>> already closed. Could we just remove this and explicitly state that for
>>>> DMA-buf, it is the user space responsibility to track the mapping between
>>>> V4L2 indexes and its own buffer objects?

I can set to zero, and I would do this for all memory types, not only dma, just for
consistency.
But wouldn't it be usefull if applications want to re-queue the buffer
without the need to re-assign the pointers?

>>>>
>>>>> +
>>>>> +By default ``VIDIOC_EXT_DQBUF`` blocks when no buffer is in the outgoing
>>>>> +queue. When the ``O_NONBLOCK`` flag was given to the
>>>>> +:ref:`open() <func-open>` function, ``VIDIOC_EXT_DQBUF`` returns
>>>>> +immediately with an ``EAGAIN`` error code when no buffer is available.
>>>>> +
>>>>> +
>>>>> +.. kernel-doc:: include/uapi/linux/videodev2.h
>>>>> +   :functions: v4l2_ext_buffers
>>>>> +
>>>>> +
>>>>> +Return Value
>>>>> +============
>>>>> +
>>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>>> +appropriately. The generic error codes are described at the
>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>>> +
>>>>> +EAGAIN
>>>>> +    Non-blocking I/O has been selected using ``O_NONBLOCK`` and no
>>>>> +    buffer was in the outgoing queue.
>>>>> +
>>>>> +EINVAL
>>>>> +    The buffer ``type`` is not supported, or the ``index`` is out of
>>>>> +    bounds, or no buffers have been allocated yet, or the ``userptr`` or
>>>>> +    ``length`` are invalid, or the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was
>>>>
>>>> There is no 'length' now.

Ack, removed.

>>>>
>>>>> +    set but the the given ``request_fd`` was invalid, or ``m.fd`` was
>>>>> +    an invalid DMABUF file descriptor.
>>>>> +
>>>>> +EIO
>>>>> +    ``VIDIOC_EXT_DQBUF`` failed due to an internal error. Can also indicate
>>>>> +    temporary problems like signal loss.
>>>
>>> Drop the last sentence, -EIO should never be used for temporary problems like
>>> that. EIO really means that the DMA engine crashed and userspace will have to call
>>> STREAMOFF and try to restart the DMA with STREAMON.
>>>
>>
>> I suspect the users of the new API wouldn't actually have to deal with
>> a signal loss, but if not, what should be the behavior then?
> 
> The typical behavior is to send out the SOURCE_CHANGE event and stall streaming,
> waiting for a valid signal.
> 
> That said, this is something I will dig into a bit deeper for the tegra HDMI
> support patch series next week, since this might be hard to do for that hardware.
> 
> But regardless of what is chosen for signal loss, EIO is wrong. Return values
> as are documented for VIDIOC_QUERY_DV_TIMINGS would be much more appropriate for this.

I'm dropping the last sentense for v6.

> 
>>
>>>>> +
>>>>> +    .. note::
>>>>> +
>>>>> +       The driver might dequeue an (empty) buffer despite returning
>>>>> +       an error, or even stop capturing. Reusing such buffer may be unsafe
>>>>> +       though and its details (e.g. ``index``) may not be returned either.
>>>
>>> The first part of this note is rather vague. I'm not sure what it meant with this.
>>>
>>>>> +       It is recommended that drivers indicate recoverable errors by setting
>>>>> +       the ``V4L2_BUF_FLAG_ERROR`` and returning 0 instead. In that case the
>>>>> +       application should be able to safely reuse the buffer and continue
>>>>> +       streaming.
>>>>
>>>> What is this for? It sounds like some legacy behavior, which maybe we could
>>>> drop with this new API?
>>>
>>> No, V4L2_BUF_FLAG_ERROR is really needed. Some hardware can really have
>>> intermittent failures causing video frames to be corrupt. And yes, that's
>>> typically due to hardware bugs. It usually either happens when you start
>>> streaming or just intermittent, but it is not severe enough to stop streaming
>>> completely (that's the -EIO error due to a call to vb2_queue_error).
>>>
>>> It's also used when you stop streaming and pending buffers are returned to
>>> userspace. They are marked as ERROR since they don't contain any video, or
>>> only partial video (if the DMA was stopped midway a frame).
>>>
>>
>> Sorry, I meant to insert that comment below the first half. The second
>> half is fine.
> 
> Ah, that makes much more sense :-)

I'm removing the first half for v6 then.

> 
>>
>>>>
>>>>> +
>>>>> +EPIPE
>>>>> +    ``VIDIOC_EXT_DQBUF`` returns this on an empty capture queue for mem2mem
>>>>> +    codecs if a buffer with the ``V4L2_BUF_FLAG_LAST`` was already
>>>>> +    dequeued and no new buffers are expected to become available.
>>>>> +
>>>>> +EBADR
>>>>> +    The ``V4L2_BUF_FLAG_REQUEST_FD`` flag was set but the device does not
>>>>> +    support requests for the given buffer type, or
>>>>> +    the ``V4L2_BUF_FLAG_REQUEST_FD`` flag was not set but the device requires
>>>>> +    that the buffer is part of a request.
>>>>> +
>>>>> +EBUSY
>>>>> +    The first buffer was queued via a request, but the application now tries
>>>>> +    to queue it directly, or vice versa (it is not permitted to mix the two
>>>>> +    APIs).
>>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>>>>> new file mode 100644
>>>>> index 0000000000000..f2a12017253f6
>>>>> --- /dev/null
>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-ext-querybuf.rst
>>>>> @@ -0,0 +1,79 @@
>>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>>> +.. document under the terms of the GNU Free Documentation License,
>>>>> +.. Version 1.1 or any later version published by the Free Software
>>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>>> +..
>>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>>> +
>>>>> +.. _VIDIOC_EXT_QUERYBUF:
>>>>> +
>>>>> +*************************
>>>>> +ioctl VIDIOC_EXT_QUERYBUF
>>>>> +*************************
>>>>> +
>>>>> +Name
>>>>> +====
>>>>> +
>>>>> +VIDIOC_EXT_QUERYBUF - Query the status of a buffer
>>>>> +
>>>>> +
>>>>> +Synopsis
>>>>> +========
>>>>> +
>>>>> +.. c:function:: int ioctl( int fd, VIDIOC_EXT_QUERYBUF, struct v4l2_ext_buffer *argp )
>>>>> +    :name: VIDIOC_EXT_QUERYBUF
>>>>> +
>>>>> +
>>>>> +Arguments
>>>>> +=========
>>>>> +
>>>>> +``fd``
>>>>> +    File descriptor returned by :ref:`open() <func-open>`.
>>>>> +
>>>>> +``argp``
>>>>> +    Pointer to struct :c:type:`v4l2_ext_buffer`.
>>>>> +
>>>>> +
>>>>> +Description
>>>>> +===========
>>>>> +
>>>>> +This ioctl is part of the :ref:`streaming <mmap>` I/O method. It can
>>>>> +be used to query the status of a buffer at any time after buffers have
>>>>> +been allocated with the :ref:`VIDIOC_EXT_CREATE_BUFS` ioctl.
>>>>> +
>>>>> +Applications set the ``type`` field of a struct
>>>>> +:c:type:`v4l2_ext_buffer` to the same buffer type as was
>>>>> +previously used with struct :c:type:`v4l2_ext_pix_format` ``type``,
>>>>> +and the ``index`` field. Valid index numbers range from zero to the
>>>>> +number of buffers allocated with
>>>>> +:ref:`VIDIOC_EXT_CREATE_BUFS` (struct
>>>>> +:c:type:`v4l2_ext_create_buffers` ``count``) minus
>>>>> +one. The ``reserved`` field must be set to 0.
>>>>> +
>>>>> +In the ``flags`` field the ``V4L2_BUF_FLAG_MAPPED``,
>>>>> +``V4L2_BUF_FLAG_PREPARED``, ``V4L2_BUF_FLAG_QUEUED`` and
>>>>> +``V4L2_BUF_FLAG_DONE`` flags will be valid. The ``planes.memory`` fields will be
>>>>> +set to the current I/O method for each plane.
>>>>> +
>>>>> +For every valid plane, an entry in ``planes`` will be filled, and zeroed for
>>>>> +invalid ones. ``planes[i].buffer_length`` is the size of the memory buffer
>>>>> +which contains the plane, ``planes[i].plane_length`` is the length of the plane,
>>>>
>>>> What is the length of the plane here? It seems like depending on the
>>>> context, the field can mean size of the payload (for filled buffer) or
>>>> total space for the payload (for empty buffer), which is a bit confusing.

I removed ext_querybuf for v6.

>>>>
>>>>> +and ``planes[i].offset` is where the plane is placed in the memory buffer.
>>>>> +
>>>>> +The size of the ``planes`` array can be calculated by the number of sequential
>>>>> +planes with ``planes[i].buffer_length`` that differs from zero up to the max
>>>>> +size of the array.
>>>>
>>>> Not sure if it's just me, but "size of array" sounds to me like the
>>>> total memory size of the array, i.e. the max number of elements, which
>>>> is fixed for this struct. How about
>>>>
>>>>   "The entries of the planes array are valid up to the last entry with
>>>>   non-zero buffer_length or the last entry of the array, whichever is
>>>>   first."
>>>>
>>>> ?

Sounds better indeed, but I removed ext_querybuf for v6.

>>>>
>>>>> +
>>>>> +
>>>>> +Return Value
>>>>> +============
>>>>> +
>>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>>> +appropriately. The generic error codes are described at the
>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>>> +
>>>>> +EINVAL
>>>>> +    The buffer ``type`` is not supported, or the ``index`` is out of
>>>>> +    bounds.
>>>>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>>>>> new file mode 100644
>>>>> index 0000000000000..008e6c98a88a5
>>>>> --- /dev/null
>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-g-ext-pix-fmt.rst
>>>>> @@ -0,0 +1,117 @@
>>>>> +.. Permission is granted to copy, distribute and/or modify this
>>>>> +.. document under the terms of the GNU Free Documentation License,
>>>>> +.. Version 1.1 or any later version published by the Free Software
>>>>> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
>>>>> +.. and no Back-Cover Texts. A copy of the license is included at
>>>>> +.. Documentation/userspace-api/media/fdl-appendix.rst.
>>>>> +..
>>>>> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
>>>>> +
>>>>> +.. _VIDIOC_G_EXT_PIX_FMT:
>>>>> +
>>>>> +************************************************************************
>>>>> +ioctl VIDIOC_G_EXT_PIX_FMT, VIDIOC_S_EXT_PIX_FMT, VIDIOC_TRY_EXT_PIX_FMT
>>>>> +************************************************************************
>>>>> +
>>>>> +Name
>>>>> +====
>>>>> +
>>>>> +VIDIOC_G_EXT_PIX_FMT - VIDIOC_S_EXT_PIX_FMT - VIDIOC_TRY_EXT_PIX_FMT - Get or set the data format, try a format
>>>>> +
>>>>> +
>>>>> +Synopsis
>>>>> +========
>>>>> +
>>>>> +.. c:function:: int ioctl( int fd, VIDIOC_G_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
>>>>> +    :name: VIDIOC_G_EXT_PIX_FMT
>>>>> +
>>>>> +.. c:function:: int ioctl( int fd, VIDIOC_S_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
>>>>> +    :name: VIDIOC_S_EXT_PIX_FMT
>>>>> +
>>>>> +.. c:function:: int ioctl( int fd, VIDIOC_TRY_EXT_PIX_FMT, struct v4l2_ext_pix_format *argp )
>>>>> +    :name: VIDIOC_TRY_EXT_PIX_FMT
>>>>> +
>>>>> +Arguments
>>>>> +=========
>>>>> +
>>>>> +``fd``
>>>>> +    File descriptor returned by :ref:`open() <func-open>`.
>>>>> +
>>>>> +``argp``
>>>>> +    Pointer to struct :c:type:`v4l2_ext_pix_format`.
>>>>> +
>>>>> +
>>>>> +Description
>>>>> +===========
>>>>> +
>>>>> +These ioctls are used to negotiate the format of data (typically image
>>>>> +format) exchanged between driver and application.
>>>>> +
>>>>> +To query the current parameters applications set the ``type`` field of a
>>>>> +struct :c:type:`v4l2_ext_pix_format` to  ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or
>>>>> +``V4L2_BUF_TYPE_VIDEO_OUTPUT``, all the other types are invalid in this API,
>>>>> +and multiplanar is supported through modifiers.
>>>>> +
>>>>> +When the application calls the
>>>>> +:ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with a pointer to this
>>>>> +structure the driver fills the other members.
>>>>> +When the requested buffer type is not supported drivers return
>>>>> +an ``EINVAL`` error code.
>>>>> +
>>>>> +To change the current format parameters applications initialize all
>>>>> +the fields in the struct.
>>>>> +For details see the documentation of the various devices types in
>>>>> +:ref:`devices`. Good practice is to query the current parameters
>>>>> +first, and to modify only those parameters not suitable for the
>>>>> +application. When the application calls the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl with
>>>>> +a pointer to a struct :c:type:`v4l2_ext_pix_format` structure the driver
>>>>> +checks and adjusts the parameters against hardware abilities. Drivers
>>>>> +should not return an error code unless the ``type`` field is invalid,
>>>>> +this is a mechanism to fathom device capabilities and to approach
>>>>> +parameters acceptable for both the application and driver. On success
>>>>> +the driver may program the hardware, allocate resources and generally
>>>>> +prepare for data exchange. Finally the :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl returns
>>>>> +the current format parameters as :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` does. Very simple,
>>>>> +inflexible devices may even ignore all input and always return the
>>>>> +default parameters. However all V4L2 devices exchanging data with the
>>>>> +application must implement the :ref:`VIDIOC_G_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` and :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>`
>>>>> +ioctl. When the requested buffer type is not supported drivers return an
>>>>> +EINVAL error code on a :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` attempt. When I/O is already in
>>>>> +progress or the resource is not available for other reasons drivers
>>>>> +return the ``EBUSY`` error code.
>>>>> +
>>>>> +The :ref:`VIDIOC_TRY_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` ioctl is equivalent to :ref:`VIDIOC_S_EXT_PIX_FMT <VIDIOC_G_EXT_PIX_FMT>` with one
>>>>> +exception: it does not change driver state. It can also be called at any
>>>>> +time, never returning ``EBUSY``. This function is provided to negotiate
>>>>> +parameters, to learn about hardware limitations, without disabling I/O
>>>>> +or possibly time consuming hardware preparations. Although strongly
>>>>> +recommended drivers are not required to implement this ioctl.
>>>>> +
>>>>
>>>> Could we make this mandatory to avoid ambiguities in the API? In
>>>
>>> Yes, please!

I updated the text for v6.
The only problem is if userspace uses EXT_TRY_FMT to a legacy driver through
a conversion layer, and the driver doesn't implement TRY_FMT, do we need a dummy
hook for those cases?

Thanks
Helen

>>>
>>>> general, most drivers implemented it anyway, without much added
>>>> complexity, because one can then simplify S_FMT to TRY_FMT + state
>>>> update.
>>>
>>> Ideally we would also fix the ambiguous situation regarding TRY/S_FMT
>>> and unsupported pixelformats: some drivers will replace unsupported
>>> pixelformats with a valid pixelformat, others will return an error.
>>>
>>> v4l2-compliance warns about drivers that return an error and point to
>>> this very old thread:
>>>
>>> http://www.mail-archive.com/linux-media@vger.kernel.org/msg56550.html
>>>
>>> However, I'm not really sure if we can resolve this ambiguity. I believe
>>> that it is only older webcam drivers that return an error.
>>
>> I guess we can at least enforce a specific behavior in the new API,
>> before we get compatibility obligations. :)
>>
>> Best regards,
>> Tomasz
>>
> 
> Regards,
> 
> 	Hans
> 

^ permalink raw reply	[flat|nested] 51+ messages in thread

end of thread, other threads:[~2021-01-14 18:06 UTC | newest]

Thread overview: 51+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-04 19:29 [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
2020-08-04 19:29 ` [PATCH v5 1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more) Helen Koike
2020-08-14  7:49   ` Alexandre Courbot
2020-11-19 16:23     ` Helen Koike
2020-09-09 11:41   ` Hans Verkuil
2020-09-14  2:14     ` Helen Koike
2020-10-02 19:49   ` Tomasz Figa
2020-11-14 14:21     ` Helen Koike
2020-11-19  5:45       ` Tomasz Figa
2020-11-19 10:08         ` Helen Koike
2020-11-19 13:43           ` Helen Koike
2020-12-14 10:19             ` Tomasz Figa
2020-08-04 19:29 ` [PATCH v5 2/7] media: v4l2: Add extended buffer operations Helen Koike
2020-08-14  7:49   ` Alexandre Courbot
2020-09-09 12:27   ` Hans Verkuil
2020-11-23 15:08     ` Helen Koike
2020-11-23 15:46       ` Tomasz Figa
2020-11-23 17:40         ` Helen Koike
2020-12-03 15:11           ` Hans Verkuil
2020-12-03 19:52             ` Helen Koike
2020-12-14 10:46               ` Tomasz Figa
2020-12-15 14:36                 ` Helen Koike
2020-12-16  3:13                   ` Tomasz Figa
2020-12-17 13:19                     ` Helen Koike
2020-12-21  3:13                       ` Tomasz Figa
2020-12-23 12:04                         ` Helen Koike
2021-01-08 10:00                           ` Tomasz Figa
2020-12-14 10:38             ` Tomasz Figa
2020-11-20 11:14   ` Tomasz Figa
2020-11-23 20:33     ` Helen Koike
2020-12-14 10:36       ` Tomasz Figa
2020-12-14 13:23         ` Helen Koike
2020-12-15  9:03           ` Tomasz Figa
2020-08-04 19:29 ` [PATCH v5 3/7] media: videobuf2: Expose helpers to implement the _ext_fmt and _ext_buf hooks Helen Koike
2020-12-14  8:52   ` Tomasz Figa
2020-12-14 12:29     ` Helen Koike
2020-08-04 19:29 ` [PATCH v5 4/7] media: mediabus: Add helpers to convert a ext_pix format to/from a mbus_fmt Helen Koike
2020-08-14  7:49   ` Alexandre Courbot
2020-08-04 19:29 ` [PATCH v5 5/7] media: vivid: Convert the capture and output drivers to EXT_FMT/EXT_BUF Helen Koike
2020-08-04 19:29 ` [PATCH v5 6/7] media: vimc: Implement the ext_fmt and ext_buf hooks Helen Koike
2020-08-04 19:29 ` [PATCH v5 7/7] media: docs: add documentation for the Extended API Helen Koike
2020-08-14  7:49   ` Alexandre Courbot
2020-11-19 10:28   ` Helen Koike
2020-11-20 11:06   ` Tomasz Figa
2020-11-20 12:24     ` Hans Verkuil
2020-11-20 12:40       ` Tomasz Figa
2020-11-20 13:20         ` Hans Verkuil
2021-01-14 18:04           ` Helen Koike
2020-08-04 19:34 ` [PATCH v5 0/7] media: v4l2: Add extended fmt and buffer ioctls Helen Koike
2020-08-14  7:49 ` Alexandre Courbot
2020-11-27 15:06 ` Helen Koike

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).